Web Scraping: HTML Tables with Python - en - Twincloud's Youtube Subtitle Extractor

- welcome John here today's video is all about scraping HTML tables this kind of

builds on my last video where I did the basics of web scraping and this actually

came about from a reddit post that I answered in our learn Python or someone

was asking for help scraping the Premier League table I

answered that I'm gonna showcase that answer for you here now today we want to

expand on my previous video which was the basics of web scraping and we're

going to be looking at how to scrape information out of HTML tables now

there's a good chance that if you're looking for information to scrape from

the web that is going to be stored in a table so learning how to get that is

really valuable skill the website that we're going to be scraping is the

premier league table and after the weekends games live for still top of the

league so the first thing that we would want to do is as always to view the

source so we can see what we're dealing with so let's get that up here now and

you can see we've got all the information here a good thing that I

like to do to start with is just by searching in the HTML for the opening

part of a table tag which you can just see down here so if we search for that

we find that we get one match which is really good which means we know there's

only one table on this website and hopefully that's the one that we want to

find that we want to get our information out of and we can see that it is here

and if we look down we can see this team name stone to appear and the points and

what not so we know there's only one table which is really good if there was

multiple tables we could use indexing or so or maybe they have different classes

to find that information and get that out but in this case we've only got one

and we can see that it has this class name here which is going to be really

important to us so if we go to our scripts and start we are going to need

to import the libraries that we need to use and as last time as request and

beautifulsoup let's get those imported

like that there we go and as always I like to set my URL as variable and let's

get that from a website here and it's this let's put that in there okay great

so now we need to go ahead and use requests to get to download that

information for us so as before it was R is equal to requests gets and then our

URL and I never spell requests right there we go and to check that we have a

good response for that we can print out our dot and then the status code of that

so hopefully this will give us a 200 response yes there we go so we know

we're getting something back the next thing we want to do is to get that

information that we've just downloaded and pass it with beautifulsoup as before

they soup is equal to beautiful soup and then we pass in our dot text like that

and this time we're going to remember to use the HTML parser to avoid the errors

the warnings that come up when you run it so let's print the soup out and see

what we go okay and just flash by their folks panelist you can see we have

indeed got all of the HTML here okay let's get rid of that the next thing I

want to do is to actually find the table itself so if we go back to this source

we can see that the table has a class of this standing table blah blah blah blah

so if we copy that and we come back here and we say that the league table is

equal to soup dot find because that will find the table element in the HTML and

we say we are looking for a table and the table has a class again class has

the underscore on the end when we're searching for it this way and beautiful

sweet because it's not the Python thing that is a class itself and then we pass

in the standing table slash table called bore which is the class of the table we

were looking for so if we save that we can print out

league table and we can just verify that we're actually getting back information

which we are just expand this a bit you can see here we go this is all the table

information so now that we've accessed the the HTML table itself we need to

pick through all of the different parts and get out the information that we want

so to do that we're going to want to loop loop through it eventually but

let's do a bit of testing first to find the actual parts of the information so

where are we looking here so inside our table the first part that is in two is

in the body and then a row okay so what we need to do is we would need to use

beautifulsoup to first look inside the table tag and then the body tag and then

inside that find the row class and then these there's TD which is basically a

cell of the table so let's get going on that so if we start our loop with four

team in league table and we want to find all of the T body which was where the

information was so what we're doing here is we're gonna store everything we find

in this variable and we're looking in here which is where we've just got the

actual league table from and we're finding the T or D class which is here

so this is the sort of the next step down if you like so that all the

information from the table that we were interested in is in the body it's not in

the table head we aren't interested in any of this we're only interested in

what's in here so if we then do its call it rose for sake of anything else is

team so now we're going to look in in what we've stored in this variable dot

find all and then the rows themselves okay so what we're doing here is we're

searching for inside we've looked for this and then in that we've looked for

that we've stored that in this very and then in that variable we're looking

for the row so we're now down here so we've looked in here and now we're

searching for all of these in here all of this table row so we can see it comes

all the way down so now we're actually going to access this little block of the

table code here and get that information out so to do that again you have to live

through that we need to find inside that four row in rows let's call it our team

is equal to Rho dot find our TD and what's the class for this one it's here

we go table standing table cell and that should give us the position but we might

as well go straight for the name and let's find this class like that so let's

see what that gives us see if we print there PL table now what do we get the

team sorry I should look through all the team names right it did but we didn't do

a dot txt at the end so all that we're getting is the full HTML data so if we

do at the end of this dot txt and we run it again we should get all the text

information out of it there we go it looks like it has some extra padding

around it so we can always call dot strip as well to remove that padding the

whitespace and hopefully we should get a long list of names there we go so now we

have a long list of names in the order they come through in the table one two

three four five six seven see arsanov seventh it's not good it's not good

so using this this method here we can now find out for example the next part

of information so maybe we want to know the points that each team has so we can

find that so the easiest way to do that is if we look here and we look when we

know that this team here they've lost 37 points so that we know

that it's this TD class here that has the points in it now we can't just

access that by calling the class because you can see that the class here isn't

unique there's at least two three or so that have the same name but what we can

do is we can actually use in index to get that number of the class down here

so if we look inside our TR which is where we are there are 1 2 3 4 5 6 7 8 9

10 10 or so so if we go and remember it's a 0 and X so 0 1 2 3 4 5 6 7 8 9 9

so if we do let's move this down PL points is equal to Rho dot find TD and

if we put the class in and the class was standing tables so let's see what we get

out standing table so so if we find that we'll see that we've got actually these

ones have a different class so here's this one that's the first one 0 that's 1

2 so this looks like this might be 3 so let's give that a go

and we'll do 3 and again we're going to call dot txt so we only get the text

information out of that so now if we do that and PL points at the same time

hopefully if we get the right information no we missed something here

what did we miss I think we need to do find all and that will allow us to find

every single one where is this with fine we were only searching for the first one

there we go now I can see straight away that I've got the wrong index so I

thought I've only found the number 12 which must be something else

here we go tell so it actually does count these classes so we do need to go

and use number nine like I originally said and that should give us the actual

points value okay so now we've successfully used finding and we've gone

through the table so we found our table up here using the class tag we search

for the table in the HTML in our soup and then in that table variable we've

searched to loop through every team in the body so we have found all the rows

and then we've found the cell that has the name of the team and we've called

dot txt and dot strip to get that right down to just the text value and then

we've used the same principle to find the points for the team we've used the

class and we found that it was the ninth one remember ace is zero so zero to nine

has the point's value in it and again we've called dot text to get there and

then we've printed that all to the terminal so hopefully that helps you

when you're looking through HTML you find that all your informations in the

table and you can use this and apply this to yourself and and get the table

information out that you're looking for it's actually it is quite simple when

you know how you've just got to make sure you you check out the HTML and you

see where you are and you follow the tags all the way through and don't

forget if there is no unique identifier for your classes that you can use the

indexing here to get to the specific one that you like that you're looking for

just remember that you need to use fine when there's load when there's multiple

multiple one sorry so you can actually use the indexing to get out so hopefully

that's been helpful to you guys and let me know if there's any other questions

in the comments or if you're confused about something hopefully I can clear it

up Jess bye