How to crawl a large list of urls? [closed]

I have a massive list of urls. It looks something like this:

And continues about 400,000 times. The intent is to find a specific string of text in all of the pages. Does anyone know how I would go about doing this?

1 Answer

I have no experience with this project but googling I did find one called Scrapy. It's written in Python and is discussed in several places, such as these:

• on StackOverflow - , where it's suggested that you run Scrapy as a daemon, scrapyd.
• Develop your first web crawler in Python Scrapy
• Scrapy example - quotesbot

The 2nd link above shows this example as well:

from scrapy.spiders import CrawlSpider, Rule from scrapy.linkextractors import LinkExtractor class ElectronicsSpider(CrawlSpider): name = "electronics" allowed_domains = [""] start_urls = [ ' ' ' ] rules = ( Rule(LinkExtractor(allow=(), restrict_css=('.pageNextPrev',)), callback="parse_item", follow=True),) def parse_item(self, response): print('Processing..' + response.url) 

You could amend the above to either include all your URLs or put them into a separate file, and read that file in through Python.

If I were you I'd be inclined to start with the 3rd link, quotesbot, since this project sounds eerily similar to what you're ultimately trying to accomplish.