Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I use browsertrix-crawler[0] for crawling and it does well on JS heavy sites since it uses a real browser to request pages. Even has options to load browser profiles so you can crawl while being authenticated on sites.

[0] https://github.com/webrecorder/browsertrix-crawler



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: