How to Make 2500 HTTP Requests in 2 Seconds with Async & Await - en

so when you run your code it will wait for each line to be executed or each

block of code to execute before the next one can happen now if you put something

in there that requires a bit of waiting time like you're waiting on a server or

a response from something external your program's just gonna be sat there doing

nothing now there's a couple of ways around this we can use threading and we

can use async i'm going to try and show you a few examples using a request to a

server so i can show you how it all goes a bit quicker so on the screen i have a

synchronous version this basically just goes out to this url sends a number uh

one to 201 to two to two and a half thousand and then gives us a response

back now as you can see uh when i ran this it took 5.4 seconds so this is what

i'm actually

using as my demo it is a server on the local host on my machine here on my

network so we're actually not getting an awful

lot of delay in the actual response but that's

just going to further compound so you can really see the difference in time

taken and then you could extrapolate that out when you actually have to wait

for a response back ignore this one we're just hitting this

endpoint which is just returning an id a number

so when i run this code here it's going to go through each line line by line

when it gets the response back you can see it's printing it out to the screen

here and it took 5.85 seconds slightly slower than the time i ran it just a

minute ago so this is all well and good but what

happens if we have thousands and thousands of urls that we need to check

on this is going to take forever so this the met the first option is to

use uh threading so i'm going to be using here in this example we're using

the concurrent futures

we're using the thread pool executor which is basically going to allow us to

use extra threads and it's going to

speed it up ever so slightly so generally speaking threads and

multi-threading like this it's better for if you're trying to work in parallel

so if your code is constantly doing something you're not waiting for

anything external then this is probably going to be the best way for you but if

you're dealing with any network requests then you bob you are going to want to

use async and i'll show you that in just a second so let me just run this one

quick you can see it looks slightly different maybe a bit quicker and this

is going to come through 3.84 seconds so slightly faster than that one there this

is fairly easy to use and fairly simple to set up all you need is a function

that does your thing for you and a list in this case because i'm using the map

there are different ways that you can use this but this is how you would use

it if you were making network requests so the third option is async now this is

slightly more complicated because there is more to it we need to have an extra

function here so we can see that this is our main function that actually makes

the request then we have one here that creates all the tasks we are using async

io here and it will await for the response for us so this is basically

like sending out all of your requests in one go sort of and it will manage and

wait for each of the responses to come back so it will basically

remove any of that time spent waiting in your code

so i'm going to run this and we'll see that it will take literally about a

second 1.37 seconds so it took 1.34 last time we got all of the data back within

a list so when would you want to use any of these versions well

if you're trying to request data from the same server over and over again

you're going to find that you're going to get rate limited so you aren't

generally going to be able to use this async version but you can work it

through because if for example the rate limit on your server that you're trying

to get to is 300 calls per minute and you only need to make 200 calls you can

asynchronously send 200 and get all the responses back in a flash whereas in if

you're doing it synchronously you'd have to send 200 actuals two or 300 single

requests which could take a couple of minutes

the other alternative is if you have lots of urls that you need to visit for

lots of different sites lots of different servers you can then use your

async code to request them all in one go and get the responses back and then

handle it that way so there'll be a link to my github down below with these

examples in and if you're interested in how you might use these in a different

kind of way i think you're going to enjoy this video right here