Back to MCP & Tools
Development & Testing
Web Scraping (Crawlee)
Professional web data collection — Crawlee and Playwright for real sites, not just APIs.
Claude Code Cursor Codex Windsurf
Overview
Crawlee (from the Apify team) is the professional scraping library: URL queues, retries, proxies, and built-in Playwright for JavaScript-heavy sites. Agents write complete scrapers with correct selectors and handle pagination and lazy loading.
Important: respect target sites' terms and robots.txt and collect only permitted data — the agent can add polite rate limiting automatically.
Important: respect target sites' terms and robots.txt and collect only permitted data — the agent can add polite rate limiting automatically.
Who Is This For?
- Collecting permitted competitor or pricing data
- Building datasets from public sites
- Scraping JavaScript sites with Playwright
- Scheduled monitoring of page changes
Installation
Setup for Claude Code
npm install crawlee playwrightExplore AI Tools
Discover the best AI tools that complement your skills
Read AI & Design Articles
Tips and trends in the world of design and AI
Related Servers
Development & Testing
Linear MCP Server
Manage Linear issues, projects, and workflows directly from your AI coding agent without leaving the terminal.
Claude Code Cursor
Development & Testing Playwright MCP
Automate browser interactions and run end-to-end tests through the Model Context Protocol, enabling AI agents to verify UI behavior in real browsers.
Claude Code Cursor Copilot
Development & Testing Jest Test Runner
Run, debug, and analyze Jest test suites directly from your AI agent. Quickly identify failing tests and get suggested fixes.
Claude Code Codex Copilot