The most important Python script I ever wrote - en - Twincloud's Youtube Subtitle Extractor

I used to Lo Monday afternoons at work as Not only was it the beginning of the

week but it was also the time I had set aside to spend a couple of hours what I

could only describe as the most mind-numbing task I had to do in an

otherwise interesting job data entry it was a sort of task that didn't take long

enough to complain about although I did obviously but was tedious enough to

Warrant me thinking that there must be a better way to do this I can't seriously

have to sit here for hours and hours my first approach was to try and palm it

off onto someone else that failed quickly so with that feeling of it

shouldn't be this hard I carried on spending my Mondays copying and pasting

and copying and pasting and not always getting it right fast forward a few

months and my journey into python began I was looking for something new to get

into my age and failing reflexes were condemning me to lower and lower ranks

in Counter Strike and with my dream of going pro fading away I thought I'd

apply myself to something else whilst looking for learning resources I came

across automate the boring stuff by alswager it came highly recommended and

I found the topics extremely enticing so I got it and started working through the

examples whilst it wasn't the be and endle it was my Catalyst into my Rush B

approach to learning as it really opened my eyes as to what was possible with

python and programming in general after all you don't know what you don't know

but now I did know know I was hurted I felt like a lot was possible a lot I saw

how easy it was to read and write CSV files take data from one file and add it

to another once I saw that you could control a browser my my mind was blown

it sent me down a path of ideas and was convinced that within a few months I'd

be able to automate all my work and I'd spend my 9 to-5 drinking coffee and

boring my colleagues of with stories of how I was once in a top 20 ukcs Clan so

off I went down the rabbit hole with beginner's Vigor and I started to climb

that initial peak of the done in querer curve with my sights firmly on a

particular Monday afternoon task but of course it wasn't all that easy I fell

into every single trap and snake pit I wanted to give up but I didn't I took my

limited knowledge banged my head against a few walls copy stack Overflow and

eventually hack something together some Jacky ass code that would just about do

the task for me just but it did they did it more quickly more accurately and even

better than I could even when it inevitably crashed this would become the

most important automation script I ever wrote and in the rest of this video I

want to share you share with you a better more modern example of what I did

so hopefully you can replicate it in some way and find your own path into

automation with python I really think you'll benefit from trying to do this

yourself first if you've never done this before go to playright use the

documentation copy the example and build upon it and try to build a script that

will visit a set of order numbers from a list and go to each page and try and

pull some arbitrary data from that page do that first and then come back and see

how I did it and you'll be able to compare and maybe pick up some ideas or

maybe yours will be better than [Music]

mine so the first thing that we want to do is we want to have a look at the site

that want to automate because we need to sort of figure out the best way that

we're going to do this obviously we need to log in which is actually demo and

demo for the username and password then we click log in then we're going to have

this box that's going to come up that we're going to need to deal with so

that's worth keeping in mind then we want to come over to the sales the order

section you can see all the orders have this order ID now if I click on one

you'll notice that the URL changes the middle part change

with the user token and then we have this and with the order ID parameter out

here so now if I change this to three it changes the order so this means now I

know how I'm going to Loop through all the orders when I import them and we'll

just pull some arbitary information off this page simulate trying to get

something like maybe we'll grab the flat shipping rate thing here or something

like that to automate you know what we may or may not want to do from this um

from this from this from this project so now I've got that open I'm going to come

over to my edit my uh folder where I'm working and I'm going to create a new

virtual environment d mvnv v MV this is how I do it you can use um virtual M if

you want to make sure you do use a virtual environment for this we need to

activate it I'm going to type act which is my shortcut otherwise it's Source VMV

bin activate um and then I'm going to do pip three install and we're going to be

using playright um I actually started with uh selenium um playright wasn't as

widely available then but this is a much better option nowadays I think for this

sort of thing although selenium will work just as well I'm just going to

create a couple of new windows just my workflow if you're using vs code or

something you obviously don't need to do anything like this you can just use vs

code and use the terminal at the bottom so now I'm going to create a new file

I'm going to call this main.py and we're going to open that

with my code editor here I'm going to come out of full screen because we want

to go over here I'm going to go to playright dodev have a look at the

website we're going to go to get started and get started library and right here

we can literally copy all of this and paste this into our code to get us

started so I'm going to move this window off onto another screen just so I have

this one open only I'm going to save this and I'm going to come and I'm going

to run my main.py file and we can see that it works now

that's because I already have the chromium installed if this is the first

time you're going to want to do play right install chromium you can do

install and install all of them but I would just recommend using chromium you

don't need to use the other browsers this one will work just fine save

yourself some hard drive space so make sure you do that step if this is the

first time that you're working with this so now that I can see that that works I

want to wa one change up here and we're going to do

headless is equal to false we want to see the browser when we run it and now

when we do so we can see that that works worked just fine so the first thing that

we want to do is we want to come back and grab the actual URL that we want to

work with which was this one here and it was the admin page cool so let's copy

this and we'll put this into here instead of this URL there we go I'm

going to remove this as well and whilst we're working I'm going to import in

time this is just an easy way I find to um put some sleeps in so I can kind of

really see what's going on cool so let's save that and I'm going

to run this now on the openart website we can see that it's popped up here and

I have the logins Pages etc etc and because I had that sleep it's done cool

so now I'm going to bring the my actual browser back over and I'm going to

minimize the code a little bit hopefully you can still read it and what we want

to do is we want to open up the inspect element tool I'm going to move this to

the bottom and we'll move it over here like

this so what we want to do is we want to

actually now use the um use playright to type the username and password in for us

and uh then hit submit so we do page Dot and I'm going to do get by placeholder

in this instance and this is going to find the element for us there's a few

different ways you can do find element but because we have a placeholder value

here I'm going to do it like that so username like

so I'm going to do fill to fill it in with the word demo I'm going to copy

this line and we'll do the same for password let's save and now we'll do uh

time sorry we'll do time dot sleep just temporarily so I can show

you and let's go ahead and move this back over and run it again and we should

hopefully now see that the demo has been filled into that browser on the right

hand side which it has perfect so we can now go ahead and do clicking that button

so we want to do page. get by and I'm going to do get by roll in this instance

and thank you to my LSP which makes it quite difficult to see it is button and

the name is equal to

here we can do dot click so hopefully you can see how you can start to um

click on things on the page so if I go back to this there's a button

button and the text uh was login I think name was login um I don't see the name

on this one this might work anyway let's try

it and we're going to make the Sleep five just for the moment and now

let's see if we can click that button we can this is going to give us this Cloud

faving verifying we're human this is just this probably won't happen for you

it's probably not going to be an issue it just happens on this website for

whatever reason so now that we're here what I'm going to do just to make sure

we get through that I'm going to make this 15 um just to make sure that we're

not getting stuck here and we do actually get to the next page although

we do have that URL you can see up there which has our good token in okay so that

does work perfect so now we just need to get through that um next thing here so

I'm going to let's make this 10 for the moment and now again I'm going to do

page. get roll get by roll and again it's a

button and we want to select this um button here that comes up let me just

log back in over here this one so what I'm going to do is I'm going to hover

over this little cross mark here we can see that we have this this button here

and I'm going to go ahead and do copy

selector and put this in here I don't really like the look of the selector but

what I can actually see here is that this is going to be um the end button

now quite often when you do copy the selector it should look it might not

look like this it might look a little bit bit neater depending

on um yeah see we we're not we're not we're not in the right place okay so I

think this one will work if I do nth one do click because this is probably this

will be the first index of the button so I'm just going to check that that will

work for this instance and we'll do do time. sleep

again so we should now be able to log in just wait for the uh verification

thing to happen this this is strange and do we click on the

button no yes we did click on the button you can see it worked so this did work

um again you'll have to play around with the selectors for the buttons you want

to click on just to make sure that you're in the right place and we'll do

this one now sleep by one and we'll chop this one down to five and that should be

good for us at the moment so now we're actually loading up that page and we're

inside the main sort of bit so what I want to do from here is I want to get

our page uh we'll call this order no call this page

URL and that's going to be equal to page.url like

this and uh no it should be like that and I'm going to print out the page URL

and I'm just going to show you how I go about constructing the URL for this

order page um gu it's slightly slightly complely different depending on like how

things go about okay so here we go here's the URL now if we compare that

to our order page URL where we can actually get the orders let's move them

side by side you can see we have this root common dashboard and here it's rout

sale SL order and then at the end of it we actually get info and then the order

at the end so what I want to do is I want to come over here and I want to say

sale page it's going to be equal to our page.

URL if I can type which I cannot I'm going do do replace so we want to

replace the part here root common dashboard in fact we just need to

replace this part don't we I want to replace

that with this after the

root this like so okay that should be good so I'm going to move

this back over so now we have they're going to have a decent URL constructed

for us we can actually start to um let's move that there uh we don't need that

now let's do um for order in orders I'm going to sort that out in just a second

we'll do page. go to because we want to go to a new URL that's going to be our

sale page URL so let's go ahead and let's put put a quick orders list in uh

3,000 3001

3,2 so we can Loop through these and but but this is going to give us that URL

but we need to add on the end of it are what was it

order ID equals cool so we can put that in

and then I'm going to do plus the string of the order this is going to be our

full URL for that set that sales page so we can go there so I'm going to do time

do sleep is equal to two so we should have now two seconds on each of those

pages when it loads up when we when we get in and we just need to put the Amper

sand in there like that so now this should work okay so we can see we're on

pay order 3000 3001 3002 and we can see that that's how we're going to be able

to Loop through these orders now from here we want to grab some information so

I'll just I will go for let's just get the shipping method uh this could be

anything you could be inputting data you could be collecting it from anything

like that um but to do that we're going to do page. query

selector and this is going to be give us access to um the sort of the CSS

selector on this page so I'm going to come over here and we're going to grab

this and this is now going to be um one of these here so input shipping method

like so so this is going to get us to this this element then we want to get to

this one here the option value uh the option one the second one down so to do

that we can get uh this needs to be an ID so this needs to be

hash then after that we want to go option but we want the nth child

child of two I believe should get us there so now we have this shipping

method what we want to do is we want to say if shipping method because you know

some orders might not have it then we'll do

print shipping method. text

text content and we're also going to now

append this to a results list so we call this

results and we'll do results.

append and we'll put in a twole so we'll do

order and the shipping method do text content then after our browser closes

we'll just print out our results so this looks pretty good so far

I'm going to save and I'm going to run this and hopefully we can have pull out

all those shipping method uh text information now I think think on this

website they're all the same so you know we're just going to get flat export or

whatever each time flat shipping rate but you can see it's coming through like

so and then at the end we have our list of tupes with each one available in it

and as I said before you can easily go ahead and do anything else you want to

do on this page we could you get any bits more bits more of information or if

you wanted to add and add something there's a comment you could type into

this box by finding the selector for this box

like this and type text in here and then add that would work too no not a problem

um so there's quite a lot you can do with that the one the last thing I want

to show you is is we're going to import CSV and we're going to do uh with and

we'll have an orders. CSV file CSV and we'll have this is read as

F we'll have our reader is equal to CSV do reader and then we'll have our

orders it's going to be equal to uh order index of zero for order in

reader like so I believe what have I done wrong here with

open cool so now I have uh the opportunity now I can create an orders.

CSV file and we can open it up here and now

we can have our 3,000 3001 3002

33 let's go for that save this and close this

buffer so now what we're doing is we're just going to import our orders from a

CSV file which you know obviously if you were doing this for your job like I was

I had a massive long CSV file so I just imported it I just put them all into the

orders. CSV in the main folder with this and I would just set this code running

and instead of printing the results I would um export them to CSV or whatever

and um yeah that would work so let's just run it let's just uh run it one

more time and we'll just see and double check

this all works fine now this is a little bit fickle so we're we're going for

direct selectors which means you know if something changes it might not work but

you'll find as you go along that you'll need to tweak it you'll find where bits

stop working um maybe certain orders don't have that information on so you'll

need to change the selectors I'm using time. sleep Which is far from ideal but

it does work there's all sorts of Weights that you can use for um

playright so if we look here it's waiting for events waiting for State all

this sort of stuff Alo waiting so it's worth looking into that and uh and

choosing and sprucing this up a bit but I really wanted to show you sort of the

most basic version and sort of how you can really easily do stuff like this

across websites um for your job or for whatever you need to do and this is 38

lines of code and this could be less or certainly much better in some cases but

hopefully you've enjoyed this video if you have like comment subscribe it

really does make a difference to me um come and join the Discord we got big

Community there have a chat and come talk to me um yeah thanks for watching

if you've enjoyed this and you want to know a bit more about Automation and how

you can get playright to actually write some of this code for you I'm going to

direct you to my other video right here