Ask HN: What is the easiest way to scrape websites today?

by stevofolife on 6/9/2024, 1:56:43 AM with 5 comments
With the rise of large language models (LLMs), what’s the easiest way to scrape the web today? Additionally, what’s the simplest method to automate interactions with a website?

  • by fumanchu36 on 6/10/2024, 7:12:57 AM

    The easiest way to scrape websites today in my opinion also features a simple way to automate interactions with websites. The scraping browser integrates with the 3 major headless browsers (Selenium, Playwright, Puppeteer) and lets you interact with your target site’s HTML in a multi-step manner to extract the data you need. Try it here https://bit.ly/3STY9ON

  • by eimrine on 6/9/2024, 2:56:31 AM

    Maybe some headless browser and a lot of proxies? I don't see how neural networks affect downloading some webpages.

  • by gbertb on 6/9/2024, 6:36:05 PM

    check out https://spider.cloud - llm-friendly markdown support, crawl dozens of urls in secs, proxy and headless chrome etc

  • by JSDevOps on 6/9/2024, 6:42:50 PM

    If you are relying on scraping you are doing it wrong

  • by throwaway211 on 6/9/2024, 2:17:19 AM

    LLMs lower the bar for being correct.

    You don't need to scrape sites if you dictate your truth.

    Therefore read some psychology papers about how people create their realities. Then make stuff up; what gets believed is going to be mostly correct in your audience's reality, which should feed back into your model. The cost of getting stuff wrong is profit if you exploit it. Let your audience filter do it for you, and who are you to dictate what's correct for someone else anyway? Make it sound catchy and convincing for a variety of tastes. Generate a few short video creators with different personas like Street Fighter characters from yesteryear that viewers can project themselves on to.