Twincloud's Youtube Subtitle Extractor
09:56
Okay, look at this. I've got a basic spider here. I didn't
change anything. This is straight from the box. I did a thousand requests,
which is what I actually asked for, so it stopped on that rather than a limit.
756 items, no errors, full logs, and it only took half an hour. I mean, that's
just mind-blowing. So, coming up this morning, I'll have a really interesting
call. It's with a guy called Ian from Zyte, and he is the chief product
officer over there, and they have a new Scrapy product that we're going to talk
about. And the one thing that makes this particularly is that it's AI backed.
Definitely when AI came out originally and sort of hit the forefront with
ChatGPT, I couldn't really see how we could use that as web scrapers and data
extractors because the problems that we had couldn't really
be solved by that sort of a tool. But, the main problem I always thought about
that was the fact that the LLMs are heavy on compute and heavy on compute
cost. So, it's going to be interesting to see how they get around that.
This video is sponsored by Zyte. They gave me access to the AI tool and asked
me for my genuine feedback as well as to share it with you guys. What I want to
know is the the motivation behind creating this this tool.
Our motivation is that we think that web scraping is too slow.
That really both the challenge of getting set up for new sites and what
What does that look like when there's multiple sites to set up for each and
every one? There's the challenge of maintenance.
And when we speak to engineering teams, they often say, "We're under pressure.
We've got to add 100 site. We are maintaining the ones that we've already
got. Adding 100 is difficult." Uh and so, our motivation is how can we
make this quicker? Pretty interesting because it's it's rather than trying to
make it easier to scrape, it's trying to get you to a base where you can start
the rest of your project quicker. So, what we've looked to do is produce a
solution which has automation where the automation does the routine
the for the most common types of website, like products. For the most
common data points in a schema, we've got about 20. For the most common
crawling strategies, like whole sites or category or search keywords, we can get
you to data immediately. But, then to give people control. That's good to know
that it's still going to be open source, or at least the spider part is, and
you'll be able to actually take that and build upon it and then
have your own sort of spider that you can put into their system, host, and
then run. Kind of makes sense, too. Do you think that part of it might
creating all that automation to to take that big chunk of work out of the start
makes this different to other generic AI, I'm going to use that term loosely,
web scraping solutions where people are like just asking
ChatGPT for an example as an example. Because this is a more controlled use
case, we've been able to develop models that are specific to this. And so, we
have a machine learning model for product type websites. We have another
for articles, for jobs. And because they're much more focused, they're more
accurate for that task, but they're also a lot cheaper. To be honest, that the
the spider itself is relatively simple. So, what it That's doing the crawling
part or or the controlling of the crawling. And so, that's then calling
Zyte API, which is solving the bans, which is allowing people to retrieve the
HTML in the format they want, whether they need it rendered or not, do they
need to do uh browser actions like scripting, um it's the machine learning
of the parsing of of turning the HTML into JSON.
Uh but, uh it's it's controlled by a relatively simple Scrapy spider.
Uh and and typically, what we find is that if people want to make a
modification, like add a filter to a crawl, it's a couple of lines of code.
But, it's also quite a nice way for people to get started because they can
start something where they can see code and they can see what it does. Yeah. Uh
and so, modifying that is very straightforward. Only scenario we find
where this isn't a great solution is the very very largest sites where the
cost of setup and maintenance is dwarfed by the volume of requests. In those
scenarios, the um it's worth it to create a spider because
running static code will be the cheapest solution. In general, um for anyone who
is looking to set up sites where they're not yet in the billions of requests per
month, it's merely up to tens of millions, we're finding it's a great
solution. All right, so I need to know, is this going to change the landscape
forever or is this going to make me redundant? So, what I'm going to do, I'm
going to take two hours, I'm going to build two spiders, and I'm going to
compare myself to the AI. Okay, so here's the results for the two
spiders. Here's mine. I got 1,634 items, and it took uh 1,250 seconds,
which is about 20 minutes. Um I got a decent amount of data points. So, on the
left-hand side here is the overview for the AI spider job. Now, this one took
about an hour. I thought this was interesting because this was much slower
than the other sites that I did in my test. We did get all of the data points
though that I was expecting to get, in fact, more than I pulled um because I
didn't want to parse all of that information. I would definitely check
with Zyte as to why this one took an hour to run. Slightly slower. We had no
issues. It It did it all there, and given the fact that mine took an hour to
write, we are definitely in the positive here. So, we should have a quick look at
the items. You can see here's all the data points and all the items of which
you can download and export in these formats if you want to. And there are
full logs and stats as well. So, the stats are going to be similar to the
ones here. As you can see, request counts, etc., etc. Everything that you
could want to see from your spider comes back to you here. And here's the results
from the other spider. I got about the same amount of items, 1547, and I think
I got 1590 or 15 66. Mine took about 37 minutes to run that, and this one was 26
minutes. And that's because the AI was deciding when to use browser rendering
and when not, whereas I was forcing browser rendering all the time, which
caused mine to be a bit slower. Plus, it took me about an hour to write as well.
So, you know, this is much much quicker. Again, let's have a quick look at the
stats over here just so you can see everything and any all the information
here that you want to know about general spider stats, the logs as well, and the
items here. Very similar to the item data that I pulled out. You can see I
have it here with the SKU. Slightly different uh format, but it's all here
available. You can see all of the information. And again, you can download
as you need here. All in all, very very positive experience with this one. Given
how easy it was to create these, all I had to do was put in the URL and
configure two or three settings and set it to run. The fact that it managed to
get all of these data points out in such a short amount of time with very very
little setup, default out of the box, is very impressive. Now, there's a lot to
be said about getting to that point so you can actually build upon it if you
need to, but in these cases, I wouldn't have to. So, the time saved would be
huge, especially considering this was just over two use cases. If I had to do
that tenfold, the amount of time I would save by using the AI spider is
astronomical. So, what are my thoughts about this, and
has it changed how I feel about AI in this space? Well, I think the tool kind
of speaks for itself. It's pretty incredible. The fact that you can go
from nothing to data with just one URL that quickly is really impressive. The
fact that it handle it handles all the bans through the Zyte API as well, it's
just an extension of that. I think it makes it a really really great package.
I've used it quite a lot now over the last week or so since I recorded the
rest of this video, and I've been very impressed. I think that it's important
to know that tools like this are supposed to be supplementary to your
work, not supposed to take over what you do. Whilst it does take out quite a lot
of the knowledge, it does mean for me, I think, that I can provide a better
service to my clients by being able to give them the data they need in much
less time. And I think I'm just going to, honestly, adapt to it and work with
it and build upon it myself, and then use it to enhance what the services that
I can deliver. How do I feel about AI in this space? Still mixed. One thing that
I think is important is that this is a much smaller model. This isn't the large
language models that we would associate with AI. This is a very different thing.
It's a much more along the machine learning type of uh style here where
it's been fed data, specific data. It's been trained on lots of different
specific sites to pull out information, and I think that is definitely the right
place to be in when we're talking about this sort of thing, AI/ML in web
scraping. Using it in conjunction with other tools to pull out the data that
you need to save yourself a load of time passing out a load of data. If you want
to try this tool for yourself, there'll be a link in the description, and
there'll be a code as well for you to use to get you sta- to get you going,
and hopefully, you enjoy it and use it and like it as much as I did. So, yeah,
once again, thanks to Zyte for this opportunity, and thank you very much for
watching. Join the Discord, like, comment, subscribe. It all makes a lot
of difference to me. Thank you very much. I'll see you in the next one.