Use THIS to stay JUST under rate limits with Async - en - Twincloud's Youtube Subtitle Extractor

so I came across an interesting problem this week that I wanted to share with

you on a project that I've been working on that requests data from two separate

locations in this case two different apis these apis both had different rate

limits so I found that when I was trying to maximize my throughput and get as

much information from it from the rate limit as possible I was either way under

one or way over the other so I rewrote my code a little bit and I thought I can

put async in here I have enough requests available but then I found that it was

too much so we needed a solution to say that was quicker than synchronous one by

one requests but not as much as a full uncontrolled async request that was

making too many requests and causing me to get blocked and then have to remake

or redo those requests so interestingly enough for this project I actually used

flask which I think you might find a little bit unusual given the fact that

they've got fast API and everything these days so if you're interested in to

know why I use flask let me leave me a comment and I'll do a video on that so I

found AIO limiter which actually lets you control that you can give in certain

rates it works on a leaky bucket algorithm that will then allow those

rates through to the server as you go and it was super easy to put in so I

wanted to show you in this video how I went about that and how you can use it

in your code and projects as well so you can really control async requests and

you can actually get as many requests done as you are allowed to do so what I

did is I created a demo server this is written in go and I followed along with

these two links here to show me how to implement the rate limit here so what I

did is I put a ray limit on this endpoint and I created a couple of

scripts to request data from it so I did put a sleep time in the endpoint there

on the go server just so I could mimic a little bit of network or lag obviously

because this is running on my Local Host we can see that the synchronous version

where the requests happen one after the other never gets near the limit and it

actually never sees any errors but it takes quite a while to run then we'll

find that the async version which is obviously much faster because we can

make requests without having to wait for the response each time so you'll see

here just how much of a difference async makes when we don't have to actually

weigh and we can actually make requests instead of waiting that whole second to

get the data back synchronous took like 17 minutes which was really boring this

one took just 10 and a half seconds but we did get 106 errors which means we

went over our rate limit 106 times and we would need to do something with that

this code can execute thousands of requests per second and it's easy to

write and that will smash through any API rate limit that you might have and

will easily get you blocked from any website now I wrote the code and go on

the server to actually send this error back when we are over the limit and this

is just to sim symbolize like whatever you might find a 429 too many requests

or something along those lines depending on what data you're requesting from

where so you can see we've got errors with this one here so what I'm going to

do now is I'm actually going to add in the limiter into this code here and we

can see how we can tailor it to make sure that we get as many requests done

as possible without going over that limit so here you can see the async code

that we just ran took 10 and a half seconds 100 or plus errors it's pretty

straightforward we have our main function that makes the collects all the

tasks together and we can see here we're requesting and using our async and away

so to actually control this a bit better we can use our AI limiter that's pretty

easy to put in so I'm just going to do from AIO limiter we're going to import

in our async limiter class here now from what we can do from here is we need to

actually change our function a little bit so this one is now going to become

async with limiter because we need to pass in our limiter class here so I'm

going to call this limiter we'll put that here and then we'll get rid of this

and we'll make a new async def function async with limiter here so we can

actually tell our code what to do and we need to then in indent these ones here

so it's indent this and indent this one as well and that was terrible and Bim

skills so that's it for that case uh we need to actually construct our limiter

here so I'm going to come to our main function and above this I'm going to

create our rate limit and this is where we can configure what we actually want

to do so I'm going to start with a hundred this is going to be our burst

amount of requests so this can be really useful if you know that you have say two

or three hundred requests per minute you can make and you only need to make a

couple of hundred you can burst them all in one go and have to and then wait

literally one second or even that to get all the data back from the API that you

want I'm then going to put 0.1 and this means we're going to do a hundred in

burst and then we're going to control it at 0.1 seconds I think this is going to

work for the rate limit I set up on the server which I think was like 200 or

something like that so now we've done that we can then go ahead and put our

rate limit in here because we need to add that in to tell this that we want to

use the rate limit that we have just created rate limit there we go so let's

save this open up my terminal I need to activate my virtual

environment go back into this async limit folder and now let's run our time

on Pi on our example async again with our limiter it's going to take a bit

longer because obviously we are now waiting and not sending as many requests

but it's going to be much quicker than our 17 minutes that's for sure and we

can tweak this depending on what we're doing so there we go took 12 seconds and

we got 999 successes which was all of the requests we made and so we're only

two seconds slower than the full async but we get none of the errors back there

so as you can see it's pretty simple to implement I thought it worked really

well we were able to actually really easily tailor how many requests we want

with that leaky bucket algorithm we can actually burst a load of requests in and

make use of that first limit on the API take that data and then make use of the

limit on the second API to pull the rest of the information that I needed for

this project I found that this was a really good way of doing it and it

worked really well in this instance so if you've enjoyed this video it'd be

great if you could like and comment maybe subscribe to my channel for more

stuff like this more web scraping content and if you found this

interesting about async here but you want to learn how to actually create

async code from your own synchronous code using something like httpx then

you're going to want to watch this video right here where I could do exactly that