THE DUDE’S FIELD GUIDE
Open-source Browser & scraping
Browser and scraping tools automate web interactions or extract structured information. Choose between full browser automation and lighter document parsing based on the target page.
How to choose
Check JavaScript rendering, session handling and rate controls. Respect site access rules and avoid collecting information you do not need.
Repositories to explore
Ordered by observed stars. This is a directory sample, not a best-to-worst verdict. Individual briefs show observation dates and source links.
| Project and purpose | Code license | Setup | Observed stars |
|---|---|---|---|
| firecrawl/firecrawl Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥 | AGPL-3.0 | Developer library | 202,5892026-11-17 |
| browser-use/browser-use Connect an AI agent to a web browser. | MIT | Developer tool | 121,3092026-11-17 |
| D4Vinci/Scrapling 🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: and follow here for daily tips and tricks: | BSD-3-Clause | Developer library | 103,9152026-11-17 |
| microsoft/playwright Make browser tasks repeatable. | Apache-2.0 | Developer tool | 100,4272026-11-17 |
| puppeteer/puppeteer Control a browser with JavaScript. | Apache-2.0 | Developer tool | 95,8532026-11-17 |
| unclecode/crawl4ai Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers… | Apache-2.0 | Self-hosted service | 87,4802026-11-17 |
| scrapy/scrapy Scrapy, a fast high-level web crawling & scraping framework for Python. | BSD-3-Clause | Developer library | 65,4502026-11-17 |
| JCodesMore/ai-website-cloner-template Clone any website with one command using AI coding agents | MIT | See setup guide | 39,3682026-11-17 |
| dgtlmoon/changedetection.io Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—… | Apache-2.0 | See setup guide | 35,8672026-11-17 |
| SeleniumHQ/selenium Automate browser tests across your stack. | Apache-2.0 | Developer tool | 34,5942026-11-17 |
| CloakHQ/CloakBrowser Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed. | MIT | See setup guide | 32,9542026-11-17 |
| apify/crawlee Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other… | Apache-2.0 | Developer library | 26,4822026-11-17 |
| browserbase/stagehand The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more. | MIT | Developer library | 26,0832026-11-17 |
| lexmount/moli Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust | Apache-2.0 | See setup guide | 23,9212026-11-17 |
| Evil0ctal/Douyin_TikTok_Download_API 🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents,… | Apache-2.0 | MCP server | 21,4682026-11-17 |
| getmaxun/maxun Turn any website into a structured API. Extract, automate, search and monitor the web. | AGPL-3.0 | See setup guide | 18,4982026-11-17 |
| MODSetter/SurfSense Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: | Apache-2.0 | See setup guide | 16,8932026-11-17 |
| yusufkaraaslan/Skill_Seekers Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection | MIT | See setup guide | 15,3292026-11-17 |
| nanobrowser/nanobrowser Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks. | Apache-2.0 | See setup guide | 14,3342026-11-17 |
| seleniumbase/SeleniumBase 📊 Blazing fast Python framework for web crawling, testing, scraping, and stealth. Supports pytest. Bypasses bot-detection with CDP Mode. Compatible with Playwright. | MIT | Developer library | 13,2662026-11-17 |
| jo-inc/camofox-browser Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement. | MIT | See setup guide | 12,3532026-11-17 |
| jhy/jsoup jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety. | MIT | See setup guide | 11,4182026-11-17 |
| NopeCHALLC/nopecha-extension Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more. | MIT | See setup guide | 11,1342026-11-17 |
| pinchtab/pinchtab High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard. | MIT | See setup guide | 10,5882026-11-17 |
| apify/crawlee-python Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand.… | Apache-2.0 | Developer library | 9,6772026-11-17 |
Is open source the same as free to run?
No. The code license and operating costs are separate. Check model terms, hardware, hosting and paid services in each project’s documentation.
Read our sourcing and ranking methodology →