Learn More
Resend Email
4
aarfa
7y

What is the best way to scrape websites?

question

scraping
Favorite
Ranter
Comments
  • 5
    7y
    Python + BeautifulSoup
  • 1
    7y
    I've enjoyed PhantomJS/CasperJS but the project is essentially dead now, I know a ton of people use Selenium, and headless chrome is becoming a popular solution.
  • 1
    7y
    An implementation of curl and an HTML parser
  • 0
    7y
    Greasemonkey and nodejs
  • 1
    7y
    Don't mind me, I'm just waiting to shit on the person who's going to recommend regex 👀
  • 0
    7y
    Sites not using JS : Requests + BeautifulSoup

    if site using JS add :
    - Chromedriver + Selenium
    - Scrapy Splash (one beautiful piece of work)

    Or use scrapy, a crappy lib but okay
  • 0
    7y
    @justin-tamblyn

    When there are counter measures, you often need to run your script through a proxy to change up your IP address or you'll just get blocked.
Related Rants
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment