Twincloud's Youtube Subtitle Extractor
09:47
if you're still writing a single script for your web scraper goes to URLs Loops
through other ones pulls different data from different places and relies on that
single script that Loops through I really think you're missing out and I
would go as far as saying is your code is going to be pretty useless when it
comes to stability and scaling up which are in my opinion the two hardest things
in web scraping so what's the answer to this and why do I need to change my code
if it's already working well the answer is a q system this is very important and
it opens up so many doors of stability and scalability for you so if you think
your scraper works fine as it is what happens if you end up scraping a site
you find a load of URLs one fails and your whole whole code Falls over so
maybe you've written retries in but you're still going to have a problem
you're not going to know where you got to you might lose all the data that
you've gathered so far you might end up having to rerun the whole thing this
video is sponsored by proxy scrape Friends of the channel and the proxies I
use myself whenever I'm scraping data I always use the proxy as I know then it's
going to work in my production environment plus when it comes to
scaling up like we're going to be doing here with the Q and worker system it
becomes even more important with proxy scrape there's 10 million plus proxies
in the pool to use or with unlimited concurrent sessions and from countries
all over the globe enabling us to scrape quickly and efficiently if you're just
starting out I'd recommend looking into the residential proxies as these are
going to give you the best option for beating any bot protection and I would
recommend choosing countries that are appropriate to the site you're scraping
and matching your own country where possible I've also been having great
success with the mobile proxies although these have a bit more of a niche use
case and don't forget you can auto rotate or keep the same IP for a certain
amount of time depending on your use case either way it's still only one line
of code to add to your project and we can let proxy scrape handle the rest
from there also any traffic you purchase is yours to use whenever as it doesn't
ever expire so if this all sounds good to you go ahead and check out proxy
scrape at the link in the description below now let's get back to the video
with this Q system we can push everything into our redis Q in this
instance and then we can pull URLs from it with our workers what this means is
that if we pull a URL and we fail to get the data for whatever reason we can then
write code that's going to push that URL back into the queue maybe it goes into a
priority Quee or maybe it goes into a holding queue that notifies us later but
what it actually what it means is that our code does not fall over and even if
our one of our workers does fall over we can restart it or we have other workers
running so map provides us a massive boost in stability and resilience when
web scraping the second thing that is super important and really really
worthwhile in this instance to use a que is scalability as I described we pull
URLs from our que to our workers to pull the data from the URL from the website
we can run as many workers as we see fit and not only can we run as many workers
as we see fit to extract data we can also do the same when it comes to
passing it decouples our code gives us great separation of concerns different
parts of our program handle different things on their own and that is their
sole job so let's talk about why we're going to be using reddis as I mentioned
earlier well first of all reddis is extremely easy to set up and run you can
run it on your local machine just by installing it you can run it through
Docker you can do one-click installs on VPS um Cloud servers to run it it's very
very easy to use it integrates into python very well super simple it's also
very very quick we don't have to worry about it's all in memory so there's no
writing to databases or anything like that simply going to save our URL into
memory it helps with persistence too imagine if all of our code Falls over
for some reason assuming that our reddish server is not on the same server
say server went down with our code it's going to hold all those URLs exactly
where we were at so we can resume right away it also going to allow us to
control duplicates as well although that's possibly less important in this
case so let's talk about architecture and what our code may look like so
initially you want to have some kind of way to get our URLs into our redis Q now
that depends on the site that you're scraping and how you want to manage that
maybe you have a list already of csvs that have come you know you know from a
client or that are important to you that you need to check that makes it nice and
easy because we can just run those and push them into our queue but if you
don't maybe you have a uh sitemap that you want to scrape you can run your
sitemap scraper to then pull those URLs out and push them into your redis Q once
you have a queue full of URLs we're going to run a couple of different
workers the first one is solely to extract the data from that URL it's
going to pull the queue it's going to check every however however of we want
to check maybe you know 5 seconds or something and it's going to check if if
there's URLs in that queue if there are it's going to pop one out which means
it's going to be removed as it pulls it in so it can't be picked up by any any
other workers and then it's going to extract the data from that URL from that
website and then we're going to choose what we're going to do with it we can
run multiple workers that will all work together and pull as many URLs as you
want so you can see how you can infinitely scale this across and not
only can you scale this locally across but you also scale this across multiple
servers or multiple machines worldwide to give yourself a very very wide net
although without going too far into it be careful when it comes to using too
many too many requests uh and stay away from accidentally or intentionally dsing
the website obviously scrape with care in mind finally we can have a passing
worker so let's say we have our redis Q which has our URLs in our extraction
workers extract the data and then they push it back into another storage bin
you could store them on your reddis instance if you wanted to I've done that
before but generally speaking it's going to say it's going to take up memory you
know if we're scraping lots of pages you want to push them somewhere else so I'd
recommend a separate database like mongod DB or something and just store
the plane text in there for the HTML for the site or whatever it is you've pulled
out from there we can then run workers that will check whatever our place is on
my mongodb or whatever and pull that information out pass it and then send it
back onto somewhere else as a complete uh scraped and passed piece of data but
because this is all separated out we can work on these separately we can decide
where we need to implement different parts of our code with different
features if we need certain things to pass or maybe we want to build something
else there's a different way of passing or something similar to that it's easily
done because our whole code is separated out we don't need to start sticking
things into loops and working that out there which is going to be a massive
mess there's three kind of main mistakes I think that people might fall into when
they start building Q systems uh and I definitely did a couple of these myself
and the first one is trying to store too much in reddis I mentioned this just a
second ago but if you try and push all your data back into your redis database
as soon as you start to scale up you know it's stored in memory you're going
to you going to have a potential to you know use up too much memory and have an
issue so store the scraped data separately somewhere else again use
reddis solely as your url Q uh and build it out from there and the second one is
not having some kind of monitoring system built as I mentioned reddis is
really um easy to work with in Python so I would have something separate that
monitors your cues and you can really see what's going on it's very easy to
write simple piece of code that you can just run uh and check every now and
again to see how many like URLs are in your Q the state of it Etc like that and
the third thing is building your extraction workers to do too much you
simply want them to take a URL from the queue and send on the actual scraped
data down the pipeline the only thing you want to do in there logic wise is if
you get a 403 or Another Bad status code and then push the URL back don't try
doing retries in there don't try and handle anything else don't try any
passing unless you know it's very very small but even then I wouldn't bother I
would move that out because as soon as you add that extra complex extra
complexity into what should be a very very singular uh task orientated piece
of code you're going to have extra fail points where you're going to be going
back into things where you shouldn't be so storing too much not monitoring and
trying to get your extraction workers to do too much is the main three things I
think that people will fall over on so overall I think if you're not using your
queue you probably should be but a q alone isn't going to save you but a
well-designed scraping system system built around that queue I think will I
think it's very important web scraping at scale is very difficult and you know
just having a q system like this will just make it easier to get going
although it's not going to be the be all and end all we all know all of the other
issues that are around web scraping and extracting the data but I think you know
it's not just a nice to have it's pretty mandatory for buildings reliable uh
scalable web scraping code so let me know what you think do you use a queue
do you use red for queue do you have some kind of other you know smaller
local queue I know python has queuing systems in it let me know what your
thoughts are down below and watch this video next so my YouTube session time
watch grows and my channel grows with it thanks