đ Crawl4AI Cloud API â Closed Beta (Launching Soon)
\n
Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.
\n
đ Apply here for early access \nWeâll be onboarding in phases and working closely with early users.\nLimited slots.
\n\n
Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data pipelines. Fully open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.
\n
\n
Enjoy using Crawl4AI? Consider becoming a sponsor to support ongoing development and community growth!
\n
\n
đ AI Assistant Skill Now Available!
\n
\n
đ¤ Crawl4AI Skill for Claude & AI Assistants
\n
Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:
\n Works with Claude, Cursor, Windsurf, and other AI coding assistants. Import the .zip file into your AI assistant's skill/knowledge system.\n
\n
\n\n
đŻ New: Adaptive Web Crawling
\n
Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your query.
Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n # Create an instance of AsyncWebCrawler\n asyncwith AsyncWebCrawler() as crawler:\n # Run the crawler on a URL\n result = await crawler.arun(url=\"https://crawl4ai.com\")\n\n # Print the extracted content\n print(result.markdown)\n\n# Run the async main function\nasyncio.run(main())\n
\n\n
Video Tutorial
\n
\n \n
\n\n\n
What Does Crawl4AI Do?
\n
Crawl4AI is a feature-rich crawler and scraper that aims to:
\n
1.âGenerate Clean Markdown: Perfect for RAG pipelines or direct ingestion into LLMs. \n2.âStructured Extraction: Parse repeated patterns with CSS, XPath, or LLM-based extraction. \n3.âAdvanced Browser Control: Hooks, proxies, stealth modes, session re-useâfine-grained control. \n4.âHigh Performance: Parallel crawling, chunk-based extraction, real-time use cases. \n5.âOpen Source: No forced API keys, no paywallsâeveryone can access their data.
\n
Core Philosophies:\n- Democratize Data: Free to use, transparent, and highly configurable. \n- LLM Friendly: Minimally processed, well-structured text, images, and metadata, so AI models can easily consume it.
\n\n
Documentation Structure
\n
To help you get started, weâve organized our docs into clear sections:
\n
\n
Setup & Installation \n Basic instructions to install Crawl4AI via pip or Docker.
\n
Quick Start \n A hands-on introduction showing how to do your first crawl, generate Markdown, and do a simple extraction.
\n
Core \n Deeper guides on single-page crawling, advanced browser/crawler parameters, content filtering, and caching.
\n
Advanced \n Explore link & media handling, lazy loading, hooking & authentication, proxies, session management, and more.
\n
Extraction \n Detailed references for no-LLM (CSS, XPath) vs. LLM-based strategies, chunking, and clustering approaches.
\n
API Reference \n Find the technical specifics of each class and method, including AsyncWebCrawler, arun(), and CrawlResult.
\n
\n
Throughout these sections, youâll find code samples you can copy-paste into your environment. If something is missing or unclear, raise an issue or PR.
\n\n
How You Can Support
\n
\n
Star & Fork: If you find Crawl4AI helpful, star the repo on GitHub or fork it to add your own features.
\n
File Issues: Encounter a bug or missing feature? Let us know by filing an issue, so we can improve.
\n
Pull Requests: Whether itâs a small fix, a big feature, or better docsâcontributions are always welcome.
\n
Join Discord: Come chat about web scraping, crawling tips, or AI workflows with the community.
\n
Spread the Word: Mention Crawl4AI in your blog posts, talks, or on social media.
\n
\n
Our mission: to empower everyoneâstudents, researchers, entrepreneurs, data scientistsâto access, parse, and shape the worldâs data with speed, cost-efficiency, and creative freedom.
đ Crawl4AI Cloud API â Closed Beta (Launching Soon)
\n
Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.
\n
đ Apply here for early access \nWeâll be onboarding in phases and working closely with early users.\nLimited slots.
\n\n
Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data pipelines. Fully open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.
\n
\n
Enjoy using Crawl4AI? Consider becoming a sponsor to support ongoing development and community growth!
\n
\n
đ AI Assistant Skill Now Available!
\n
\n
đ¤ Crawl4AI Skill for Claude & AI Assistants
\n
Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:
\n Works with Claude, Cursor, Windsurf, and other AI coding assistants. Import the .zip file into your AI assistant's skill/knowledge system.\n
\n
\n\n
đŻ New: Adaptive Web Crawling
\n
Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your query.
Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n # Create an instance of AsyncWebCrawler\n asyncwith AsyncWebCrawler() as crawler:\n # Run the crawler on a URL\n result = await crawler.arun(url=\"https://crawl4ai.com\")\n\n # Print the extracted content\n print(result.markdown)\n\n# Run the async main function\nasyncio.run(main())\n
\n\n
Video Tutorial
\n
\n \n
\n\n\n
What Does Crawl4AI Do?
\n
Crawl4AI is a feature-rich crawler and scraper that aims to:
\n
1.âGenerate Clean Markdown: Perfect for RAG pipelines or direct ingestion into LLMs. \n2.âStructured Extraction: Parse repeated patterns with CSS, XPath, or LLM-based extraction. \n3.âAdvanced Browser Control: Hooks, proxies, stealth modes, session re-useâfine-grained control. \n4.âHigh Performance: Parallel crawling, chunk-based extraction, real-time use cases. \n5.âOpen Source: No forced API keys, no paywallsâeveryone can access their data.
\n
Core Philosophies:\n- Democratize Data: Free to use, transparent, and highly configurable. \n- LLM Friendly: Minimally processed, well-structured text, images, and metadata, so AI models can easily consume it.
\n\n
Documentation Structure
\n
To help you get started, weâve organized our docs into clear sections:
\n
\n
Setup & Installation \n Basic instructions to install Crawl4AI via pip or Docker.
\n
Quick Start \n A hands-on introduction showing how to do your first crawl, generate Markdown, and do a simple extraction.
\n
Core \n Deeper guides on single-page crawling, advanced browser/crawler parameters, content filtering, and caching.
\n
Advanced \n Explore link & media handling, lazy loading, hooking & authentication, proxies, session management, and more.
\n
Extraction \n Detailed references for no-LLM (CSS, XPath) vs. LLM-based strategies, chunking, and clustering approaches.
\n
API Reference \n Find the technical specifics of each class and method, including AsyncWebCrawler, arun(), and CrawlResult.
\n
\n
Throughout these sections, youâll find code samples you can copy-paste into your environment. If something is missing or unclear, raise an issue or PR.
\n\n
How You Can Support
\n
\n
Star & Fork: If you find Crawl4AI helpful, star the repo on GitHub or fork it to add your own features.
\n
File Issues: Encounter a bug or missing feature? Let us know by filing an issue, so we can improve.
\n
Pull Requests: Whether itâs a small fix, a big feature, or better docsâcontributions are always welcome.
\n
Join Discord: Come chat about web scraping, crawling tips, or AI workflows with the community.
\n
Spread the Word: Mention Crawl4AI in your blog posts, talks, or on social media.
\n
\n
Our mission: to empower everyoneâstudents, researchers, entrepreneurs, data scientistsâto access, parse, and shape the worldâs data with speed, cost-efficiency, and creative freedom.
Welcome to the Crawl4AI project! As an open-source library for web crawling and AI integration, we value contributions from the community. This guide explains our branching strategy, how to contribute effectively, and the overall release process. Our goal is to maintain a stable, collaborative environment where bug fixes, features, and improvements can be integrated smoothly while allowing for experimental development.
\n
We follow a GitFlow-inspired workflow to ensure predictability and quality. Releases occur approximately every two weeks, with a focus on semantic versioning, comprehensive documentation, and user-friendly updates.
\n
Core Branches
\n
\n
main: The stable branch containing production-ready code. It's always identical to the latest released version and is tagged for releases. Do not submit PRs directly here.
\n
develop: The primary integration branch for ongoing development. This is where all contributions (bug fixes, minor features, documentation updates) are merged. Submit your pull requests targeting this branch.
\n
next: Reserved for the lead maintainer (Unclecode) to experiment with major features, refactors, or cutting-edge changes. These are merged into develop when ready.
\n
release/vX.Y.Z: Temporary branches created from develop for final release preparations (e.g., version bumps, demos, release notes). These are short-lived and deleted after the release.
\n
\n
Contributor Workflow
\n
We encourage contributions of all kinds: bug fixes, new features, documentation improvements, tests, or even Docker enhancements. Follow these steps to contribute:
\n\n
Fork the Repository: Create your own fork on GitHub.
\n
\n
Create a Branch: Base your work on the develop branch.
\n
git checkout develop\ngit checkout -b feature/your-feature-name # Or bugfix/your-bugfix-name\n
\n
\n
\n
Make Changes:
\n
\n
Implement your feature or fix.
\n
If updating documentation (e.g., README.md, mkdocs.yml, or docs/blog/), ensure version references are consistent (e.g., update site_name in mkdocs.yml to reflect the upcoming version if relevant).
\n
For Docker-related changes (e.g., Dockerfile, docker-compose.yml, or docs/md_v2/core/docker-deployment.md), test locally and include build instructions in your PR description.
\n
Add tests if applicable (run pytest to verify).
\n
Follow code style guidelines (use black for formatting).
\n
\n
\n
Commit and Push:
\n
Use descriptive commit messages (e.g., \"Fix: Resolve issue with async crawling\").
\n
Push to your fork: git push origin feature/your-feature-name.
\n
\n
\n
Submit a Pull Request:
\n
Target the develop branch.
\n
Provide a clear description: What does it do? Link to any issues. Include screenshots or code examples if helpful.
\n
If your change affects documentation or Docker, mention how it aligns with the version (e.g., \"Updates Docker docs for v0.7.0 compatibility\").
\n
We'll review and merge approved PRs into develop.
\n
\n
\n
Discuss Large Changes: For major features or experimental ideas, open an issue first to align with the project's direction.
\n\n
If your PR involves breaking changes, include a migration guide in the description.
\n
Lead Maintainer's Workflow (For Reference)
\n
\n
The lead maintainer (Unclecode) uses the next branch for isolated experimental work.
\n
Features from next are periodically merged into develop (via rebase and merge) to keep everything in sync.
\n
This isolation ensures your contributions aren't disrupted by ongoing major changes.
\n
\n
Release Process (High-Level Overview)
\n
Releases happen bi-weekly to ship improvements regularly. As a contributor, your merged changes in develop will be included in the next release unless specified otherwise. Here's a summary of what happens:
\n
\n
Preparation: A temporary release/vX.Y.Z branch is created from develop. Any ready features from next are merged here.
\n
Final Updates:
\n
Version bump in code (e.g., __version__.py).
\n
Creation of a demo script in examples/ to showcase new features.
\n
Writing release notes in docs/blog/ (personal \"I\" voice from Unclecode, with code examples, impacts, and migration guides if needed).
\n
Documentation updates: README.md (highlights, version refs), mkdocs.yml (site_name with version), docs/blog/index.md (add new release), and copying notes to docs/md_v2/blog/releases/.
\n
Docker updates: Dockerfile (version arg), docker-compose.yml, deploy/docker/README.md, and docs/md_v2/core/docker-deployment.md. A release candidate image (e.g., X.Y.Z-r1) is built and tested.
\n
\n
\n
Testing and Merge: Full tests run; changes committed and merged to main with a tag.
\n
Publication: Tagged release on GitHub (with notes), publish to PyPI, and push Docker images (stable and latest after testing).
\n
Sync: Back-merge to develop and reset next for the next cycle.
\n
\n
Semantic versioning is used: MAJOR for breaking changes, MINOR for features, PATCH for fixes. Pre-releases (e.g., -rc1) may be used for testing.
\n
If your contribution requires Docker testing or affects docs, it may be part of this stepâfeel free to suggest updates in your PR.
\n
Benefits of This Approach
\n
\n
Stability: main is always reliable for users.
\n
Collaboration: Fixed PR target (develop) makes contributing straightforward.
\n
Isolation: Experimental work in next doesn't block team progress.
\n
User-Focused: Releases include demos, detailed notes, and updated docs/Docker for easy adoption.
\n
Predictability: Bi-weekly cadence keeps the project active.
\n
\n
Checklist for Contributors
\n
Before submitting a PR:
\n
\n
[ ] Based on and targeting develop.
\n
[ ] Tests pass (pytest).
\n
[ ] Docs updated if needed (e.g., version refs in mkdocs.yml, Docker files).
\n
[ ] No breaking changes without a migration guide.
\n
[ ] Descriptive title and description.
\n
\n
Common Issues
\n
\n
Merge Conflicts: Rebase your branch on latest develop before PR.
\n
Docker Builds: Test multi-arch (amd64/arm64) locally if changing Dockerfile.
\n
Version Consistency: Ensure any version mentions match semantic rules.
\n
\n
Communication
\n
\n
Open issues for discussions or bugs.
\n
Join our Discord (link in README) for real-time help.
\n
After releases, announcements go to GitHub, Discord, and social media.
\n
\n
Thanks for contributing to Crawl4AI â we appreciate your help in making it better!
While the default adaptive crawling configuration works well for most use cases, understanding the underlying strategies and scoring mechanisms allows you to fine-tune the crawler for specific domains and requirements.
\n
The Three-Layer Scoring System
\n
1. Coverage Score
\n
Coverage measures how comprehensively your knowledge base covers the query terms and related concepts.
# For technical documentation with specific terminology\nconfig = AdaptiveConfig(\n confidence_threshold=0.85, # Require high coverage\n top_k_links=5 # Cast wider net\n)\n\n# For general topics with synonyms\nconfig = AdaptiveConfig(\n confidence_threshold=0.6, # Lower threshold\n top_k_links=2 # More focused\n)\n
\n
2. Consistency Score
\n
Consistency evaluates whether the information across pages is coherent and non-contradictory.
\n
How It Works
\n\n
Extracts key statements from each document
\n
Compares statements across documents
\n
Measures agreement vs. contradiction
\n
Returns normalized score (0-1)
\n\n
Practical Impact
\n
\n
High consistency (>0.8): Information is reliable and coherent
\n
Medium consistency (0.5-0.8): Some variation, but generally aligned
\n
Low consistency (<0.5): Conflicting information, need more sources
\n
\n
3. Saturation Score
\n
Saturation detects when new pages stop providing novel information.
\n
Detection Algorithm
\n
# Tracks new unique terms per page\nnew_terms_page_1 = 50\nnew_terms_page_2 = 30# 60% of first\nnew_terms_page_3 = 15# 50% of second\nnew_terms_page_4 = 5# 33% of third\n# Saturation detected: rapidly diminishing returns\n
\n
Configuration
\n
config = AdaptiveConfig(\n min_gain_threshold=0.1 # Stop if <10% new information\n)\n
Rationale:\n- Lower threshold (articles often repeat information)\n- Higher gain threshold (avoid duplicate stories)\n- More links per page (explore different perspectives)
Rationale:\n- Very high threshold for completeness\n- Many pages allowed for thorough research\n- Very low gain threshold to capture references
\n
Performance Optimization
\n
Memory Management
\n
# For large crawls, use streaming\nconfig = AdaptiveConfig(\n max_pages=100,\n save_state=True,\n state_path=\"large_crawl.json\"\n)\n\n# Periodically clean state\niflen(state.knowledge_base) > 1000:\n # Keep only the top 500 most relevant docs\n top_content = adaptive.get_relevant_content(top_k=500)\n keep_indices = {d[\"index\"] for d in top_content}\n state.knowledge_base = [\n doc for i, doc inenumerate(state.knowledge_base) if i in keep_indices\n ]\n
\n
Parallel Processing
\n
# Use multiple start points\nstart_urls = [\n \"https://docs.example.com/intro\",\n \"https://docs.example.com/api\",\n \"https://docs.example.com/guides\"\n]\n\n# Crawl in parallel\ntasks = [\n adaptive.digest(url, query)\n for url in start_urls\n]\nresults = await asyncio.gather(*tasks)\n
classHybridStrategy(CrawlStrategy):\n def__init__(self):\n self.strategies = [\n TechnicalDocStrategy(),\n SemanticSimilarityStrategy(),\n URLPatternStrategy()\n ]\n\n defcalculate_confidence(self, state: CrawlState) -> float:\n # Weighted combination of strategies\n scores = [s.calculate_confidence(state) for s in self.strategies]\n weights = [0.5, 0.3, 0.2]\n returnsum(s * w for s, w inzip(scores, weights))\n
\n
Best Practices
\n
1. Start Conservative
\n
Begin with default settings and adjust based on results:
\n
# Start with defaults\nresult = await adaptive.digest(url, query)\n\n# Analyze and adjust\nif adaptive.confidence < 0.7:\n config.max_pages += 10\n config.confidence_threshold -= 0.1\n
\n
2. Monitor Resource Usage
\n
import psutil\n\n# Check memory before large crawls\nmemory_percent = psutil.virtual_memory().percent\nif memory_percent > 80:\n config.max_pages = min(config.max_pages, 20)\n
\n
3. Use Domain Knowledge
\n
# For API documentation\nif \"api\" in start_url:\n config.top_k_links =2# APIs have clear structure\n\n# For blogs\nif \"blog\" in start_url:\n config.min_gain_threshold =0.2# Avoid similar posts\n
\n
4. Validate Results
\n
# Always validate the knowledge base\nrelevant_content = adaptive.get_relevant_content(top_k=10)\n\n# Check coverage\nquery_terms = set(query.lower().split())\ncovered_terms = set()\n\nfor doc in relevant_content:\n content_lower = doc['content'].lower()\n for term in query_terms:\n if term in content_lower:\n covered_terms.add(term)\n\ncoverage_ratio = len(covered_terms) / len(query_terms)\nprint(f\"Query term coverage: {coverage_ratio:.0%}\")\n
Prerequisites \n- You have a basic grasp of AsyncWebCrawler Basics \n- You know how to run or configure your Python environment with Playwright installed
\n
\n\n
1. Proxy Usage
\n
If you need to route your crawl traffic through a proxyâwhether for IP rotation, geo-testing, or privacyâCrawl4AI supports it via BrowserConfig.proxy_config.
Key Points \n- proxy_config expects a dict with server and optional auth credentials. \n- Many commercial proxies provide an HTTP/HTTPS âgatewayâ server that you specify in server. \n- If your proxy doesnât need auth, omit username/password.
\n\n
2. Capturing PDFs & Screenshots
\n
Sometimes you need a visual record of a page or a PDF âprintout.â Crawl4AI can do both in one pass:
\n
import os, asyncio\nfrom base64 import b64decode\nfrom crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig\n\nasyncdefmain():\n run_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n screenshot=True,\n pdf=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://en.wikipedia.org/wiki/List_of_common_misconceptions\",\n config=run_config\n )\n if result.success:\n print(f\"Screenshot data present: {result.screenshot isnotNone}\")\n print(f\"PDF data present: {result.pdf isnotNone}\")\n\n if result.screenshot:\n print(f\"[OK] Screenshot captured, size: {len(result.screenshot)} bytes\")\n withopen(\"wikipedia_screenshot.png\", \"wb\") as f:\n f.write(b64decode(result.screenshot))\n else:\n print(\"[WARN] Screenshot data is None.\")\n\n if result.pdf:\n print(f\"[OK] PDF captured, size: {len(result.pdf)} bytes\")\n withopen(\"wikipedia_page.pdf\", \"wb\") as f:\n f.write(result.pdf)\n else:\n print(\"[WARN] PDF data is None.\")\n\n else:\n print(\"[ERROR]\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Why PDF + Screenshot? \n- Large or complex pages can be slow or error-prone with âtraditionalâ full-page screenshots. \n- Exporting a PDF is more reliable for very long pages. Crawl4AI automatically converts the first PDF page into an image if you request both.
\n
Relevant Parameters \n- pdf=True: Exports the current page as a PDF (base64-encoded in result.pdf). \n- screenshot=True: Creates a screenshot (base64-encoded in result.screenshot). \n- scroll_delay: Controls the delay (seconds) between scroll steps when taking a full-page screenshot of a tall page. Defaults to 0.2. Increase for pages with slow-loading assets. \n- scan_full_page or advanced hooking can further refine how the crawler captures content.
\n\n
3. Handling SSL Certificates
\n
If you need to verify or export a siteâs SSL certificateâfor compliance, debugging, or data analysisâCrawl4AI can fetch it during the crawl:
\n
import asyncio, os\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n tmp_dir = os.path.join(os.getcwd(), \"tmp\")\n os.makedirs(tmp_dir, exist_ok=True)\n\n config = CrawlerRunConfig(\n fetch_ssl_certificate=True,\n cache_mode=CacheMode.BYPASS\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=\"https://example.com\", config=config)\n\n if result.success and result.ssl_certificate:\n cert = result.ssl_certificate\n print(\"\\nCertificate Information:\")\n print(f\"Issuer (CN): {cert.issuer.get('CN', '')}\")\n print(f\"Valid until: {cert.valid_until}\")\n print(f\"Fingerprint: {cert.fingerprint}\")\n\n # Export in multiple formats:\n cert.to_json(os.path.join(tmp_dir, \"certificate.json\"))\n cert.to_pem(os.path.join(tmp_dir, \"certificate.pem\"))\n cert.to_der(os.path.join(tmp_dir, \"certificate.der\"))\n\n print(\"\\nCertificate exported to JSON/PEM/DER in 'tmp' folder.\")\n else:\n print(\"[ERROR] No certificate or crawl failed.\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Key Points \n- fetch_ssl_certificate=True triggers certificate retrieval. \n- result.ssl_certificate includes methods (to_json, to_pem, to_der) for saving in various formats (handy for server config, Java keystores, etc.).
\n\n
4. Custom Headers
\n
Sometimes you need to set custom headers (e.g., language preferences, authentication tokens, or specialized user-agent strings). You can do this in multiple ways:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n # Option 1: Set headers at the crawler strategy level\n crawler1 = AsyncWebCrawler(\n # The underlying strategy can accept headers in its constructor\n crawler_strategy=None# We'll override below for clarity\n )\n crawler1.crawler_strategy.update_user_agent(\"MyCustomUA/1.0\")\n crawler1.crawler_strategy.set_custom_headers({\n \"Accept-Language\": \"fr-FR,fr;q=0.9\"\n })\n result1 = await crawler1.arun(\"https://www.example.com\")\n print(\"Example 1 result success:\", result1.success)\n\n # Option 2: Pass headers directly to `arun()`\n crawler2 = AsyncWebCrawler()\n result2 = await crawler2.arun(\n url=\"https://www.example.com\",\n headers={\"Accept-Language\": \"es-ES,es;q=0.9\"}\n )\n print(\"Example 2 result success:\", result2.success)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Notes \n- Some sites may react differently to certain headers (e.g., Accept-Language). \n- If you need advanced user-agent randomization or client hints, see Identity-Based Crawling (Anti-Bot) or use UserAgentGenerator.
\n\n
5. Session Persistence & Local Storage
\n
Crawl4AI can preserve cookies and localStorage so you can continue where you left offâideal for logging into sites or skipping repeated auth flows.
\n
5.1 storage_state
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n storage_dict = {\n \"cookies\": [\n {\n \"name\": \"session\",\n \"value\": \"abcd1234\",\n \"domain\": \"example.com\",\n \"path\": \"/\",\n \"expires\": 1699999999.0,\n \"httpOnly\": False,\n \"secure\": False,\n \"sameSite\": \"None\"\n }\n ],\n \"origins\": [\n {\n \"origin\": \"https://example.com\",\n \"localStorage\": [\n {\"name\": \"token\", \"value\": \"my_auth_token\"}\n ]\n }\n ]\n }\n\n # Provide the storage state as a dictionary to start \"already logged in\"\n asyncwith AsyncWebCrawler(\n headless=True,\n storage_state=storage_dict\n ) as crawler:\n result = await crawler.arun(\"https://example.com/protected\")\n if result.success:\n print(\"Protected page content length:\", len(result.html))\n else:\n print(\"Failed to crawl protected page\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
5.2 Exporting & Reusing State
\n
You can sign in once, export the browser context, and reuse it laterâwithout re-entering credentials.
\n
\n
await context.storage_state(path=\"my_storage.json\"): Exports cookies, localStorage, etc. to a file.
\n
Provide storage_state=\"my_storage.json\" on subsequent runs to skip the login step.
Crawl4AI supports respecting robots.txt rules with efficient caching:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Enable robots.txt checking in config\n config = CrawlerRunConfig(\n check_robots_txt=True# Will check and respect robots.txt rules\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n \"https://example.com\",\n config=config\n )\n\n ifnot result.success and result.status_code == 403:\n print(\"Access denied by robots.txt\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Key Points\n- Robots.txt files are cached locally for efficiency\n- Cache is stored in ~/.crawl4ai/robots/robots_cache.db\n- Cache has a default TTL of 7 days\n- If robots.txt can't be fetched, crawling is allowed\n- Returns 403 status code if URL is disallowed
\n\n
Putting It All Together
\n
Hereâs a snippet that combines multiple âadvancedâ features (proxy, PDF, screenshot, SSL, custom headers, and session reuse) into one run. Normally, youâd tailor each setting to your projectâs needs.
\n
import os, asyncio\nfrom base64 import b64decode\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n # 1. Browser config with proxy + headless\n browser_cfg = BrowserConfig(\n proxy_config={\n \"server\": \"http://proxy.example.com:8080\",\n \"username\": \"myuser\",\n \"password\": \"mypass\",\n },\n headless=True,\n )\n\n # 2. Crawler config with PDF, screenshot, SSL, custom headers, and ignoring caches\n crawler_cfg = CrawlerRunConfig(\n pdf=True,\n screenshot=True,\n fetch_ssl_certificate=True,\n cache_mode=CacheMode.BYPASS,\n headers={\"Accept-Language\": \"en-US,en;q=0.8\"},\n storage_state=\"my_storage.json\", # Reuse session from a previous sign-in\n verbose=True,\n )\n\n # 3. Crawl\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\n url = \"https://secure.example.com/protected\", \n config=crawler_cfg\n )\n\n if result.success:\n print(\"[OK] Crawled the secure page. Links found:\", len(result.links.get(\"internal\", [])))\n\n # Save PDF & screenshot\n if result.pdf:\n withopen(\"result.pdf\", \"wb\") as f:\n f.write(b64decode(result.pdf))\n if result.screenshot:\n withopen(\"result.png\", \"wb\") as f:\n f.write(b64decode(result.screenshot))\n\n # Check SSL cert\n if result.ssl_certificate:\n print(\"SSL Issuer CN:\", result.ssl_certificate.issuer.get(\"CN\", \"\"))\n else:\n print(\"[ERROR]\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n\n
7. Anti-Bot Features (Stealth Mode & Undetected Browser)
\n
Crawl4AI provides two powerful features to bypass bot detection:
\n
7.1 Stealth Mode
\n
Stealth mode uses playwright-stealth to modify browser fingerprints and behaviors. Enable it with a simple flag:
PDF & Screenshot capturing for large or critical pages
\n
SSL Certificate retrieval & exporting
\n
Custom Headers for language or specialized requests
\n
Session Persistence via storage state
\n
Robots.txt Compliance
\n
Anti-Bot Features (Stealth Mode & Undetected Browser)
\n
\n
With these power tools, you can build robust scraping workflows that mimic real user behavior, handle secure sites, capture detailed snapshots, manage sessions across multiple runs, and bypass bot detectionâstreamlining your entire data collection pipeline.
\n
Note: In future versions, we may enable stealth mode and undetected browser by default. For now, users should explicitly enable these features when needed.
When crawling sites protected by anti-bot systems (Akamai, Cloudflare, PerimeterX, DataDome, Imperva, etc.), requests often get blocked with CAPTCHAs, 403 responses, or empty pages. Crawl4AI provides a layered retry and fallback system that automatically detects blocking and escalates through multiple strategies until content is retrieved.
\n
How Detection Works
\n
After each crawl attempt, Crawl4AI inspects the HTTP status code and HTML content for known anti-bot signals:
Detection uses structural HTML markers (specific element IDs, script sources, form actions) rather than generic keywords to minimize false positives. A normal page that happens to mention \"CAPTCHA\" or \"Cloudflare\" in its content will not be flagged.
\n
When all attempts fail and blocking is still detected, the result is returned with success=False and error_message describing the block reason.
\n
Configuration Options
\n
All anti-bot retry options live on CrawlerRunConfig:
\n
\n\n
\n
Parameter
\n
Type
\n
Default
\n
Description
\n
\n\n\n
\n
proxy_config
\n
ProxyConfig, list[ProxyConfig], or None
\n
None
\n
Single proxy or ordered list of proxies to try. Each retry round iterates through the full list. Use \"direct\" or ProxyConfig.DIRECT in a list to explicitly try without a proxy.
\n
\n
\n
max_retries
\n
int
\n
0
\n
Number of retry rounds when blocking is detected. 0 = no retries.
\n
\n
\n
fallback_fetch_function
\n
async (str) -> str
\n
None
\n
Async function called as last resort. Takes URL, returns raw HTML.
\n
\n\n
\n
Escalation Chain
\n
Each retry round tries every proxy in proxy_config in order. If all rounds are exhausted and the page is still blocked, the fallback fetch function is called as a last resort.
\n
For each round (1 + max_retries rounds):\n 1. Try proxy_config[0] (or direct if proxy_config isNone)\n 2. If blocked â try proxy_config[1]\n 3. If blocked â try proxy_config[2]\n 4. ... continue through all proxies\n 5. If any attempt succeeds â done\n\nIf all rounds exhausted and still blocked:\n 6. Call fallback_fetch_function(url) â process returned HTML\n
\n
Worst-case attempts before the fetch function: (1 + max_retries) x len(proxy_config)
\n
Crawl Stats
\n
Every crawl result includes a crawl_stats dict with detailed attempt tracking:
Retry the crawl up to 3 times when blocking is detected. Useful when blocks are intermittent or IP-based.
\n
from crawl4ai import AsyncWebCrawler\nfrom crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig\n\nasyncwith AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=CrawlerRunConfig(max_retries=3),\n )\n
\n
Single Proxy
\n
Pass a single ProxyConfig â it's used on every attempt. Same behavior as always.
\n
from crawl4ai.async_configs importProxyConfig\n\nconfig= CrawlerRunConfig(\n max_retries=2,\n proxy_config=ProxyConfig(\n server=\"http://proxy.example.com:8080\",\n username=\"user\",\n password=\"pass\",\n ),\n)\n
\n
Direct-First, Then Proxies
\n
Try without a proxy first, then escalate to proxies if blocked. Use ProxyConfig.DIRECT (or the string \"direct\") in the list to represent a no-proxy attempt.
With this setup, each round tries the datacenter proxy first, then the residential proxy. With max_retries=1, worst case is 2 rounds x 2 proxies = 4 attempts.
\n
Fallback Fetch Function
\n
When all browser-based attempts fail, call a custom async function as a last resort. This function receives the URL and must return raw HTML as a string. The returned HTML is processed through the normal pipeline (markdown generation, extraction, etc.).
\n
This is useful when you have access to a scraping API, a pre-fetched cache, or any other source of HTML.
\n
import aiohttp\n\nasyncdefmy_scraping_api(url: str) -> str:\n \"\"\"Fetch HTML via an external scraping API.\"\"\"\n asyncwith aiohttp.ClientSession() as session:\n asyncwith session.get(\n \"https://api.my-scraping-service.com/fetch\",\n params={\"url\": url, \"format\": \"html\"},\n headers={\"Authorization\": \"Bearer MY_TOKEN\"},\n ) as resp:\n if resp.status == 200:\n returnawait resp.text()\n raise RuntimeError(f\"API error: {resp.status}\")\n\nconfig = CrawlerRunConfig(\n max_retries=1,\n fallback_fetch_function=my_scraping_api,\n)\n
\n
The function can do anything â call an API, read from a database, return cached HTML, or make a simple HTTP request with a different library. Crawl4AI does not care how the HTML is obtained.
\n
Full Escalation (All Features Combined)
\n
This example combines every layer: stealth mode, a list of proxies tried in order, retries, and a final fetch function.
\n
import aiohttp\nfrom crawl4ai import AsyncWebCrawler\nfrom crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, ProxyConfig\n\n# Last-resort: fetch HTML via an external service\nasyncdefexternal_fetch(url: str) -> str:\n asyncwith aiohttp.ClientSession() as session:\n asyncwith session.post(\n \"https://api.my-service.com/scrape\",\n json={\"url\": url, \"render_js\": True},\n headers={\"Authorization\": \"Bearer MY_TOKEN\"},\n ) as resp:\n returnawait resp.text()\n\nbrowser_config = BrowserConfig(\n headless=True,\n enable_stealth=True,\n)\n\ncrawl_config = CrawlerRunConfig(\n magic=True,\n wait_until=\"load\",\n max_retries=2,\n\n # Proxies tried in order â cheapest first\n proxy_config=[\n ProxyConfig(\n server=\"http://datacenter-proxy.example.com:8080\",\n username=\"user\",\n password=\"pass\",\n ),\n ProxyConfig(\n server=\"http://residential-proxy.example.com:9090\",\n username=\"user\",\n password=\"pass\",\n ),\n ],\n\n # Last resort â called after all retries and proxies are exhausted\n fallback_fetch_function=external_fetch,\n)\n\nasyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://protected-site.com/products\",\n config=crawl_config,\n )\n\n if result.success:\n print(f\"Got {len(result.markdown.raw_markdown)} chars of markdown\")\n print(f\"Resolved by: {result.crawl_stats['resolved_by']}\")\n print(f\"Attempts: {result.crawl_stats['attempts']}\")\n else:\n print(f\"All attempts failed: {result.error_message}\")\n
\n
What happens step by step:
\n
\n\n
\n
Round
\n
Attempt
\n
What runs
\n
\n\n\n
\n
1
\n
1
\n
Datacenter proxy â blocked
\n
\n
\n
1
\n
2
\n
Residential proxy â blocked
\n
\n
\n
2
\n
1
\n
Datacenter proxy â blocked
\n
\n
\n
2
\n
2
\n
Residential proxy â blocked
\n
\n
\n
3
\n
1
\n
Datacenter proxy â blocked
\n
\n
\n
3
\n
2
\n
Residential proxy â blocked
\n
\n
\n
-
\n
-
\n
external_fetch(url) called â returns HTML
\n
\n\n
\n
That's up to 6 browser attempts + 1 function call before giving up.
\n
Tips
\n
\n
Start with max_retries=0 and a fallback_fetch_function if you just want a safety net without burning time on retries.
\n
Order proxies cheapest-first â datacenter proxies before residential, residential before premium.
\n
Combine with stealth mode â BrowserConfig(enable_stealth=True) and CrawlerRunConfig(magic=True) reduce the chance of being blocked in the first place.
\n
wait_until=\"load\" is important for anti-bot sites â the default domcontentloaded can return before the anti-bot sensor finishes.
\n
Check crawl_stats to understand what happened â how many attempts, which proxy worked, whether the fallback function was needed.
Weâre excited to announce a Crawl Dispatcher module that can handle thousands of crawling tasks simultaneously. By efficiently managing system resources (memory, CPU, network), this dispatcher ensures high-performance data extraction at scale. It also provides real-time monitoring of each crawlerâs status, memory usage, and overall progress.
\n
Stay tunedâthis feature is coming soon in an upcoming release of Crawl4AI! For the latest news, keep an eye on our changelogs and follow @unclecode on X.
\n
Below is a sample of how the dispatcherâs performance monitor might look in action:
\n
\n
We canât wait to bring you this streamlined, scalable approach to multi-URL crawlingâwatch this space for updates!
This guide explains how to use Crawl4AI to handle file downloads during crawling. You'll learn how to trigger downloads, specify download locations, and access downloaded files.
\n
Enabling Downloads
\n
To enable downloads, set the accept_downloads parameter in the BrowserConfig object and pass it to the crawler.
\n
from crawl4ai.async_configs import BrowserConfig, AsyncWebCrawler\n\nasyncdefmain():\n config = BrowserConfig(accept_downloads=True) # Enable downloads globally\n asyncwith AsyncWebCrawler(config=config) as crawler:\n # ... your crawling logic ...\n\nasyncio.run(main())\n
\n
Specifying Download Location
\n
Specify the download directory using the downloads_path attribute in the BrowserConfig object. If not provided, Crawl4AI defaults to creating a \"downloads\" directory inside the .crawl4ai folder in your home directory.
\n
from crawl4ai.async_configs import BrowserConfig\nimport os\n\ndownloads_path = os.path.join(os.getcwd(), \"my_downloads\") # Custom download path\nos.makedirs(downloads_path, exist_ok=True)\n\nconfig = BrowserConfig(accept_downloads=True, downloads_path=downloads_path)\n\nasyncdefmain():\n asyncwith AsyncWebCrawler(config=config) as crawler:\n result = await crawler.arun(url=\"https://example.com\")\n # ...\n
\n
Triggering Downloads
\n
Downloads are typically triggered by user interactions on a web page, such as clicking a download button. Use js_code in CrawlerRunConfig to simulate these actions and wait_for to allow sufficient time for downloads to start.
\n
from crawl4ai.async_configs import CrawlerRunConfig\n\nconfig = CrawlerRunConfig(\n js_code=\"\"\"\n const downloadLink = document.querySelector('a[href$=\".exe\"]');\n if (downloadLink) {\n downloadLink.click();\n }\n \"\"\",\n wait_for=5# Wait 5 seconds for the download to start\n)\n\nresult = await crawler.arun(url=\"https://www.python.org/downloads/\", config=config)\n
\n
Accessing Downloaded Files
\n
The downloaded_files attribute of the CrawlResult object contains paths to downloaded files.
\n
if result.downloaded_files:\n print(\"Downloaded files:\")\n for file_path in result.downloaded_files:\n print(f\"- {file_path}\")\n file_size = os.path.getsize(file_path)\n print(f\"- File size: {file_size} bytes\")\nelse:\n print(\"No files downloaded.\")\n
\n
Example: Downloading Multiple Files
\n
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig\nimport os\nfrom pathlib import Path\n\nasyncdefdownload_multiple_files(url: str, download_path: str):\n config = BrowserConfig(accept_downloads=True, downloads_path=download_path)\n asyncwith AsyncWebCrawler(config=config) as crawler:\n run_config = CrawlerRunConfig(\n js_code=\"\"\"\n const downloadLinks = document.querySelectorAll('a[download]');\n for (const link of downloadLinks) {\n link.click();\n // Delay between clicks\n await new Promise(r => setTimeout(r, 2000)); \n }\n \"\"\",\n wait_for=10# Wait for all downloads to start\n )\n result = await crawler.arun(url=url, config=run_config)\n\n if result.downloaded_files:\n print(\"Downloaded files:\")\n for file in result.downloaded_files:\n print(f\"- {file}\")\n else:\n print(\"No files downloaded.\")\n\n# Usage\ndownload_path = os.path.join(Path.home(), \".crawl4ai\", \"downloads\")\nos.makedirs(download_path, exist_ok=True)\n\nasyncio.run(download_multiple_files(\"https://www.python.org/downloads/windows/\", download_path))\n
\n
Important Considerations
\n
\n
Browser Context: Downloads are managed within the browser context. Ensure js_code correctly targets the download triggers on the webpage.
\n
Timing: Use wait_for in CrawlerRunConfig to manage download timing.
\n
Error Handling: Handle errors to manage failed downloads or incorrect paths gracefully.
\n
Security: Scan downloaded files for potential security threats before use.
\n
\n
This revised guide ensures consistency with the Crawl4AI codebase by using BrowserConfig and CrawlerRunConfig for all download-related configurations. Let me know if further adjustments are needed!
Crawl4AIâs hooks let you customize the crawler at specific points in the pipeline:
\n
1.âon_browser_created â After browser creation. \n2.âon_page_context_created â After a new context & page are created. \n3.âbefore_goto â Just before navigating to a page. \n4.âafter_goto â Right after navigation completes. \n5.âon_user_agent_updated â Whenever the user agent changes. \n6.âon_execution_started â Once custom JavaScript execution begins. \n7.âbefore_retrieve_html â Just before the crawler retrieves final HTML. \n8.âbefore_return_html â Right before returning the HTML content.
\n
Important: Avoid heavy tasks in on_browser_created since you donât yet have a page context. If you need to log in, do so in on_page_context_created.
\n
\n
note \"Important Hook Usage Warning\"\n Avoid Misusing Hooks: Do not manipulate page objects in the wrong hook or at the wrong time, as it can crash the pipeline or produce incorrect results. A common mistake is attempting to handle authentication prematurelyâsuch as creating or closing pages in on_browser_created.
\n
Use the Right Hook for Auth: If you need to log in or set tokens, use on_page_context_created. This ensures you have a valid page/context to work with, without disrupting the main crawling flow.
\n
Identity-Based Crawling: For robust auth, consider identity-based crawling (or passing a session ID) to preserve state. Run your initial login steps in a separate, well-defined process, then feed that session to your main crawlârather than shoehorning complex authentication into early hooks. Check out Identity-Based Crawling for more details.
\n
Be Cautious: Overwriting or removing elements in the wrong hook can compromise the final crawl. Keep hooks focused on smaller tasks (like route filters, custom headers), and let your main logic (crawling, data extraction) proceed normally.
\n
\n
Below is an example demonstration.
\n\n
Example: Using Hooks in AsyncWebCrawler
\n
import asyncio\nimport json\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom playwright.async_api import Page, BrowserContext\n\nasyncdefmain():\n print(\"đ Hooks Example: Demonstrating recommended usage\")\n\n # 1) Configure the browser\n browser_config = BrowserConfig(\n headless=True,\n verbose=True\n )\n\n # 2) Configure the crawler run\n crawler_run_config = CrawlerRunConfig(\n js_code=\"window.scrollTo(0, document.body.scrollHeight);\",\n wait_for=\"body\",\n cache_mode=CacheMode.BYPASS\n )\n\n # 3) Create the crawler instance\n crawler = AsyncWebCrawler(config=browser_config)\n\n #\n # Define Hook Functions\n #\n\n asyncdefon_browser_created(browser, **kwargs):\n # Called once the browser instance is created (but no pages or contexts yet)\n print(\"[HOOK] on_browser_created - Browser created successfully!\")\n # Typically, do minimal setup here if needed\n return browser\n\n asyncdefon_page_context_created(page: Page, context: BrowserContext, **kwargs):\n # Called right after a new page + context are created (ideal for auth or route config).\n print(\"[HOOK] on_page_context_created - Setting up page & context.\")\n\n # Example 1: Route filtering (e.g., block images)\n asyncdefroute_filter(route):\n if route.request.resource_type == \"image\":\n print(f\"[HOOK] Blocking image request: {route.request.url}\")\n await route.abort()\n else:\n await route.continue_()\n\n await context.route(\"**\", route_filter)\n\n # Example 2: (Optional) Simulate a login scenario\n # (We do NOT create or close pages here, just do quick steps if needed)\n # e.g., await page.goto(\"https://example.com/login\")\n # e.g., await page.fill(\"input[name='username']\", \"testuser\")\n # e.g., await page.fill(\"input[name='password']\", \"password123\")\n # e.g., await page.click(\"button[type='submit']\")\n # e.g., await page.wait_for_selector(\"#welcome\")\n # e.g., await context.add_cookies([...])\n # Then continue\n\n # Example 3: Adjust the viewport\n await page.set_viewport_size({\"width\": 1080, \"height\": 600})\n return page\n\n asyncdefbefore_goto(\n page: Page, context: BrowserContext, url: str, **kwargs\n ):\n # Called before navigating to each URL.\n print(f\"[HOOK] before_goto - About to navigate: {url}\")\n # e.g., inject custom headers\n await page.set_extra_http_headers({\n \"Custom-Header\": \"my-value\"\n })\n return page\n\n asyncdefafter_goto(\n page: Page, context: BrowserContext, \n url: str, response, **kwargs\n ):\n # Called after navigation completes.\n print(f\"[HOOK] after_goto - Successfully loaded: {url}\")\n # e.g., wait for a certain element if we want to verify\n try:\n await page.wait_for_selector('.content', timeout=1000)\n print(\"[HOOK] Found .content element!\")\n except:\n print(\"[HOOK] .content not found, continuing anyway.\")\n return page\n\n asyncdefon_user_agent_updated(\n page: Page, context: BrowserContext, \n user_agent: str, **kwargs\n ):\n # Called whenever the user agent updates.\n print(f\"[HOOK] on_user_agent_updated - New user agent: {user_agent}\")\n return page\n\n asyncdefon_execution_started(page: Page, context: BrowserContext, **kwargs):\n # Called after custom JavaScript execution begins.\n print(\"[HOOK] on_execution_started - JS code is running!\")\n return page\n\n asyncdefbefore_retrieve_html(page: Page, context: BrowserContext, **kwargs):\n # Called before final HTML retrieval.\n print(\"[HOOK] before_retrieve_html - We can do final actions\")\n # Example: Scroll again\n await page.evaluate(\"window.scrollTo(0, document.body.scrollHeight);\")\n return page\n\n asyncdefbefore_return_html(\n page: Page, context: BrowserContext, html: str, **kwargs\n ):\n # Called just before returning the HTML in the result.\n print(f\"[HOOK] before_return_html - HTML length: {len(html)}\")\n return page\n\n #\n # Attach Hooks\n #\n\n crawler.crawler_strategy.set_hook(\"on_browser_created\", on_browser_created)\n crawler.crawler_strategy.set_hook(\n \"on_page_context_created\", on_page_context_created\n )\n crawler.crawler_strategy.set_hook(\"before_goto\", before_goto)\n crawler.crawler_strategy.set_hook(\"after_goto\", after_goto)\n crawler.crawler_strategy.set_hook(\n \"on_user_agent_updated\", on_user_agent_updated\n )\n crawler.crawler_strategy.set_hook(\n \"on_execution_started\", on_execution_started\n )\n crawler.crawler_strategy.set_hook(\n \"before_retrieve_html\", before_retrieve_html\n )\n crawler.crawler_strategy.set_hook(\n \"before_return_html\", before_return_html\n )\n\n await crawler.start()\n\n # 4) Run the crawler on an example page\n url = \"https://example.com\"\n result = await crawler.arun(url, config=crawler_run_config)\n\n if result.success:\n print(\"\\nCrawled URL:\", result.url)\n print(\"HTML length:\", len(result.html))\n else:\n print(\"Error:\", result.error_message)\n\n await crawler.close()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n
Hook Lifecycle Summary
\n
1.âon_browser_created: \n - Browser is up, but no pages or contexts yet. \n - Light setup onlyâdonât try to open or close pages here (that belongs in on_page_context_created).
\n
2.âon_page_context_created: \n - Perfect for advanced auth or route blocking. \n - You have a page + context ready but havenât navigated to the target URL yet.
\n
3.âbefore_goto: \n - Right before navigation. Typically used for setting custom headers or logging the target URL.
\n
4.âafter_goto: \n - After page navigation is done. Good place for verifying content or waiting on essential elements.
\n
5.âon_user_agent_updated: \n - Whenever the user agent changes (for stealth or different UA modes).
\n
6.âon_execution_started: \n - If you set js_code or run custom scripts, this runs once your JS is about to start.
\n
7.âbefore_retrieve_html: \n - Just before the final HTML snapshot is taken. Often you do a final scroll or lazy-load triggers here.
\n
8.âbefore_return_html: \n - The last hook before returning HTML to the CrawlResult. Good for logging HTML length or minor modifications.
\n\n
When to Handle Authentication
\n
Recommended: Use on_page_context_created if you need to:
\n
\n
Navigate to a login page or fill forms
\n
Set cookies or localStorage tokens
\n
Block resource routes to avoid ads
\n
\n
This ensures the newly created context is under your control beforearun() navigates to the main URL.
\n\n
Additional Considerations
\n
\n
Session Management: If you want multiple arun() calls to reuse a single session, pass session_id= in your CrawlerRunConfig. Hooks remain the same.
\n
Performance: Hooks can slow down crawling if they do heavy tasks. Keep them concise.
\n
Error Handling: If a hook fails, the overall crawl might fail. Catch exceptions or handle them gracefully.
\n
Concurrency: If you run arun_many(), each URL triggers these hooks in parallel. Ensure your hooks are thread/async-safe.
\n
\n\n
Conclusion
\n
Hooks provide fine-grained control over:
\n
\n
Browser creation (light tasks only)
\n
Page and context creation (auth, route blocking)
\n
Navigation phases
\n
Final HTML retrieval
\n
\n
Follow the recommended usage:\n- Login or advanced tasks in on_page_context_created \n- Custom headers or logs in before_goto / after_goto \n- Scrolling or final checks in before_retrieve_html / before_return_html
Crawl4AI empowers you to navigate and interact with the web using your authentic digital identity, ensuring youâre recognized as a human and not mistaken for a bot. This tutorial covers:
\n
1.âManaged Browsers â The recommended approach for persistent profiles and identity-based crawling. \n2.âMagic Mode â A simplified fallback solution for quick automation without persistent identity.
\n\n
1. Managed Browsers: Your Digital Identity Solution
\n
Managed Browsers let developers create and use persistent browser profiles. These profiles store local storage, cookies, and other session data, letting you browse as your real selfâcomplete with logins, preferences, and cookies.
\n
Key Benefits
\n
\n
Authentic Browsing Experience: Retain session data and browser fingerprints as though youâre a normal user.
\n
Effortless Configuration: Once you log in or solve CAPTCHAs in your chosen data directory, you can re-run crawls without repeating those steps.
\n
Empowered Data Access: If you can see the data in your own browser, you can automate its retrieval with your genuine identity.
\n
\n\n
Below is a partial update to your Managed Browsers tutorial, specifically the section about creating a user-data directory using Playwrightâs Chromium binary rather than a system-wide Chrome/Edge. Weâll show how to locate that binary and launch it with a --user-data-dir argument to set up your profile. You can then point BrowserConfig.user_data_dir to that folder for subsequent crawls.
\n\n
Creating a User Data Directory (Command-Line Approach via Playwright)
\n
If you installed Crawl4AI (which installs Playwright under the hood), you already have a Playwright-managed Chromium on your system. Follow these steps to launch that Chromium from your command line, specifying a custom data directory:
\n
1.âFind the Playwright Chromium binary:\n - On most systems, installed browsers go under a ~/.cache/ms-playwright/ folder or similar path. \n - To see an overview of installed browsers, run:\n
python -m playwright install --dry-run\n
\n or\n
playwright install --dry-run\n
\n (depending on your environment). This shows where Playwright keeps Chromium.\n
\n on Linux, or a corresponding folder on macOS/Windows.
\n
\n
2.âLaunch the Playwright Chromium binary with a custom user-data directory:\n
# Linux example\n~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome \\\n --user-data-dir=/home/<you>/my_chrome_profile\n
\n
# macOS example (Playwrightâs internal binary)\n~/Library/Caches/ms-playwright/chromium-1234/chrome-mac/Chromium.app/Contents/MacOS/Chromium \\\n --user-data-dir=/Users/<you>/my_chrome_profile\n
\n
# Windows example (PowerShell/cmd)\n\"C:\\Users\\<you>\\AppData\\Local\\ms-playwright\\chromium-1234\\chrome-win\\chrome.exe\" ^\n --user-data-dir=\"C:\\Users\\<you>\\my_chrome_profile\"\n
\n
Replace the path with the actual subfolder indicated in your ms-playwright cache structure. \n - This opens a fresh Chromium with your new or existing data folder. \n - Log into any sites or configure your browser the way you want. \n - Close when doneâyour profile data is saved in that folder.
\n
3.âUse that folder in BrowserConfig.user_data_dir:\n
\n - Next time you run your code, it reuses that folderâpreserving your session data, cookies, local storage, etc.\n\n
Creating a Profile Using the Crawl4AI CLI (Easiest)
\n
If you prefer a guided, interactive setup, use the built-in CLI to create and manage persistent browser profiles.
\n
1.â Launch the profile manager:\n
crwl profiles\n
\n
2.â Choose \"Create new profile\" and enter a profile name. A Chromium window opens so you can log in to sites and configure settings. When finished, return to the terminal and press q to save the profile.
\n
3.â Profiles are saved under ~/.crawl4ai/profiles/<profile_name> (for example: /home/<you>/.crawl4ai/profiles/test_profile_1) along with a storage_state.json for cookies and session data.
\n
4.â Optionally, choose \"List profiles\" in the CLI to view available profiles and their paths.
\n
5.â Use the saved path with BrowserConfig.user_data_dir:\n
from crawl4ai import AsyncWebCrawler, BrowserConfig\n\nprofile_path = \"/home/<you>/.crawl4ai/profiles/test_profile_1\"\n\nbrowser_config = BrowserConfig(\n headless=True,\n use_managed_browser=True,\n user_data_dir=profile_path,\n browser_type=\"chromium\",\n)\n\nasyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(url=\"https://example.com/private\")\n
\n
The CLI also supports listing and deleting profiles, and even testing a crawl directly from the menu.
\n\n
3. Using Managed Browsers in Crawl4AI
\n
Once you have a data directory with your session data, pass it to BrowserConfig:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n # 1) Reference your persistent data directory\n browser_config = BrowserConfig(\n headless=True, # 'True' for automated runs\n verbose=True,\n use_managed_browser=True, # Enables persistent browser strategy\n browser_type=\"chromium\",\n user_data_dir=\"/path/to/my-chrome-profile\"\n )\n\n # 2) Standard crawl config\n crawl_config = CrawlerRunConfig(\n wait_for=\"css:.logged-in-content\"\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(url=\"https://example.com/private\", config=crawl_config)\n if result.success:\n print(\"Successfully accessed private data with your identity!\")\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Workflow
\n
1.âLogin externally (via CLI or your normal Chrome with --user-data-dir=...). \n2.âClose that browser. \n3.âUse the same folder in user_data_dir= in Crawl4AI. \n4.âCrawl â The site sees your identity as if youâre the same user who just logged in.
\n\n
4. Magic Mode: Simplified Automation
\n
If you donât need a persistent profile or identity-based approach, Magic Mode offers a quick way to simulate human-like browsing without storing long-term data.
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=CrawlerRunConfig(\n magic=True, # Simplifies a lot of interaction\n remove_overlay_elements=True,\n page_timeout=60000\n )\n )\n
\n
Magic Mode:
\n
\n
Simulates a user-like experience
\n
Randomizes user agent & navigator
\n
Randomizes interactions & timings
\n
Masks automation signals
\n
Attempts pop-up handling
\n
\n
But itâs no substitute for true user-based sessions if you want a fully legitimate identity-based solution.
\n\n
5. Comparing Managed Browsers vs. Magic Mode
\n
\n\n
\n
Feature
\n
Managed Browsers
\n
Magic Mode
\n
\n\n\n
\n
Session Persistence
\n
Full localStorage/cookies retained in user_data_dir
\n
No persistent data (fresh each run)
\n
\n
\n
Genuine Identity
\n
Real user profile with full rights & preferences
\n
Emulated user-like patterns, but no actual identity
\n
\n
\n
Complex Sites
\n
Best for login-gated sites or heavy config
\n
Simple tasks, minimal login or config needed
\n
\n
\n
Setup
\n
External creation of user_data_dir, then use in Crawl4AI
\n
Single-line approach (magic=True)
\n
\n
\n
Reliability
\n
Extremely consistent (same data across runs)
\n
Good for smaller tasks, can be less stable
\n
\n\n
\n\n
6. Using the BrowserProfiler Class
\n
Crawl4AI provides a dedicated BrowserProfiler class for managing browser profiles, making it easy to create, list, and delete profiles for identity-based browsing.
\n
Creating and Managing Profiles with BrowserProfiler
\n
The BrowserProfiler class offers a comprehensive API for browser profile management:
\n
import asyncio\nfrom crawl4ai import BrowserProfiler\n\nasyncdefmanage_profiles():\n # Create a profiler instance\n profiler = BrowserProfiler()\n\n # Create a profile interactively - opens a browser window\n profile_path = await profiler.create_profile(\n profile_name=\"my-login-profile\"# Optional: name your profile\n )\n\n print(f\"Profile saved at: {profile_path}\")\n\n # List all available profiles\n profiles = profiler.list_profiles()\n\n for profile in profiles:\n print(f\"Profile: {profile['name']}\")\n print(f\" Path: {profile['path']}\")\n print(f\" Created: {profile['created']}\")\n print(f\" Browser type: {profile['type']}\")\n\n # Get a specific profile path by name\n specific_profile = profiler.get_profile_path(\"my-login-profile\")\n\n # Delete a profile when no longer needed\n success = profiler.delete_profile(\"old-profile-name\")\n\nasyncio.run(manage_profiles())\n
\n
How profile creation works:\n1. A browser window opens for you to interact with\n2. You log in to websites, set preferences, etc.\n3. When you're done, press 'q' in the terminal to close the browser\n4. The profile is saved in the Crawl4AI profiles directory\n5. You can use the returned path with BrowserConfig.user_data_dir
\n
Interactive Profile Management
\n
The BrowserProfiler also offers an interactive management console that guides you through profile creation, listing, and deletion:
\n
import asyncio\nfrom crawl4ai import BrowserProfiler, AsyncWebCrawler, BrowserConfig\n\n# Define a function to use a profile for crawling\nasyncdefcrawl_with_profile(profile_path, url):\n browser_config = BrowserConfig(\n headless=True,\n use_managed_browser=True,\n user_data_dir=profile_path\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(url)\n return result\n\nasyncdefmain():\n # Create a profiler instance\n profiler = BrowserProfiler()\n\n # Launch the interactive profile manager\n # Passing the crawl function as a callback adds a \"crawl with profile\" option\n await profiler.interactive_manager(crawl_callback=crawl_with_profile)\n\nasyncio.run(main())\n
\n
Legacy Methods
\n
For backward compatibility, the previous methods on ManagedBrowser are still available, but they delegate to the new BrowserProfiler class:
\n
from crawl4ai.browser_manager import ManagedBrowser\n\n# These methods still work but use BrowserProfiler internally\nprofiles = ManagedBrowser.list_profiles()\n
\n
Complete Example
\n
See the full example in docs/examples/identity_based_browsing.py for a complete demonstration of creating and using profiles for authenticated browsing using the new BrowserProfiler class.
\n\n
7. Locale, Timezone, and Geolocation Control
\n
In addition to using persistent profiles, Crawl4AI supports customizing your browser's locale, timezone, and geolocation settings. These features enhance your identity-based browsing experience by allowing you to control how websites perceive your location and regional settings.
\n
Setting Locale and Timezone
\n
You can set the browser's locale and timezone through CrawlerRunConfig:
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncwithAsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=CrawlerRunConfig(\n # Set browser locale (language and region formatting)\n locale=\"fr-FR\", # French (France)\n\n # Set browser timezone\n timezone_id=\"Europe/Paris\",\n\n # Other normal options...\n magic=True,\n page_timeout=60000\n )\n )\n
\n
How it works:\n- locale affects language preferences, date formats, number formats, etc.\n- timezone_id affects JavaScript's Date object and time-related functionality\n- These settings are applied when creating the browser context and maintained throughout the session
\n
Configuring Geolocation
\n
Control the GPS coordinates reported by the browser's geolocation API:
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, GeolocationConfig\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://maps.google.com\", # Or any location-aware site\n config=CrawlerRunConfig(\n # Configure precise GPS coordinates\n geolocation=GeolocationConfig(\n latitude=48.8566, # Paris coordinates\n longitude=2.3522,\n accuracy=100# Accuracy in meters (optional)\n ),\n\n # This site will see you as being in Paris\n page_timeout=60000\n )\n )\n
\n
Important notes:\n- When geolocation is specified, the browser is automatically granted permission to access location\n- Websites using the Geolocation API will receive the exact coordinates you specify\n- This affects map services, store locators, delivery services, etc.\n- Combined with the appropriate locale and timezone_id, you can create a fully consistent location profile
\n
Combining with Managed Browsers
\n
These settings work perfectly with managed browsers for a complete identity solution:
Combining persistent profiles with precise geolocation and region settings gives you complete control over your digital identity.
\n
8. Summary
\n
\n
Create your user-data directory either:
\n
By launching Chrome/Chromium externally with --user-data-dir=/some/path
\n
Or by using the built-in BrowserProfiler.create_profile() method
\n
Or through the interactive interface with profiler.interactive_manager()
\n
Log in or configure sites as needed, then close the browser
\n
Reference that folder in BrowserConfig(user_data_dir=\"...\") + use_managed_browser=True
\n
Customize identity aspects with locale, timezone_id, and geolocation
\n
List and reuse profiles with BrowserProfiler.list_profiles()
\n
Manage your profiles with the dedicated BrowserProfiler class
\n
Enjoy persistent sessions that reflect your real identity
\n
If you only need quick, ephemeral automation, Magic Mode might suffice
\n
\n
Recommended: Always prefer a Managed Browser for robust, identity-based crawling and simpler interactions with complex sites. Use Magic Mode for quick tasks or prototypes where persistent data is unnecessary.
\n
With these approaches, you preserve your authentic browsing environment, ensuring the site sees you exactly as a normal userâno repeated logins or wasted time.
Many websites now load images lazily as you scroll. If you need to ensure they appear in your final crawl (and in result.media), consider:
\n
1.âwait_for_images=True â Wait for images to fully load. \n2.âscan_full_page â Force the crawler to scroll the entire page, triggering lazy loads. \n3.âscroll_delay â Add small delays between scroll steps.
\n
Note: If the site requires multiple âLoad Moreâ triggers or complex interactions, see the Page Interaction docs. For sites with virtual scrolling (Twitter/Instagram style), see the Virtual Scroll docs.
\n
Example: Ensuring Lazy Images Appear
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig\nfrom crawl4ai.async_configs import CacheMode\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # Force the crawler to wait until images are fully loaded\n wait_for_images=True,\n\n # Option 1: If you want to automatically scroll the page to load images\n scan_full_page=True, # Tells the crawler to try scrolling the entire page\n scroll_delay=0.5, # Delay (seconds) between scroll steps\n\n # Option 2: If the site uses a 'Load More' or JS triggers for images,\n # you can also specify js_code or wait_for logic here.\n\n cache_mode=CacheMode.BYPASS,\n verbose=True\n )\n\n asyncwith AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:\n result = await crawler.arun(\"https://www.example.com/gallery\", config=config)\n\n if result.success:\n images = result.media.get(\"images\", [])\n print(\"Images found:\", len(images))\n for i, img inenumerate(images[:5]):\n print(f\"[Image {i}] URL: {img['src']}, Score: {img.get('score','N/A')}\")\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Explanation:
\n
\n
wait_for_images=True \n The crawler tries to ensure images have finished loading before finalizing the HTML.
\n
scan_full_page=True \n Tells the crawler to attempt scrolling from top to bottom. Each scroll step helps trigger lazy loading.
\n
scroll_delay=0.5 \n Pause half a second between each scroll step. Helps the site load images before continuing.
\n
\n
When to Use:
\n
\n
Lazy-Loading: If images appear only when the user scrolls into view, scan_full_page + scroll_delay helps the crawler see them.
\n
Heavier Pages: If a page is extremely long, be mindful that scanning the entire page can be slow. Adjust scroll_delay or the max scroll steps as needed.
\n
\n\n
Combining with Other Link & Media Filters
\n
You can still combine lazy-load logic with the usual exclude_external_images, exclude_domains, or link filtration:
\n
config = CrawlerRunConfig(\n wait_for_images=True,\n scan_full_page=True,\n scroll_delay=0.5,\n\n # Filter out external images if you only want local ones\n exclude_external_images=True,\n\n # Exclude certain domains for links\n exclude_domains=[\"spammycdn.com\"],\n)\n
\n
This approach ensures you see all images from the main domain while ignoring external ones, and the crawler physically scrolls the entire page so that lazy-loading triggers.
\n\n
Tips & Troubleshooting
\n
1.âLong Pages \n - Setting scan_full_page=True on extremely long or infinite-scroll pages can be resource-intensive. \n - Consider using hooks or specialized logic to load specific sections or âLoad Moreâ triggers repeatedly.
\n
2.âMixed Image Behavior \n - Some sites load images in batches as you scroll. If youâre missing images, increase your scroll_delay or call multiple partial scrolls in a loop with JS code or hooks.
\n
3.âCombining with Dynamic Wait \n - If the site has a placeholder that only changes to a real image after a certain event, you might do wait_for=\"css:img.loaded\" or a custom JS wait_for.
\n
4.âCaching \n - If cache_mode is enabled, repeated crawls might skip some network fetches. If you suspect caching is missing new images, set cache_mode=CacheMode.BYPASS for fresh fetches.
\n\n
With lazy-loading support, wait_for_images, and scan_full_page settings, you can capture the entire gallery or feed of images you expectâeven if the site only loads them as the user scrolls. Combine these with the standard media filtering and domain exclusion for a complete link & media handling strategy.
Heads Up: Crawl4AI supports advanced dispatchers for parallel or throttled crawling, providing dynamic rate limiting and memory usage checks. The built-in arun_many() function uses these dispatchers to handle concurrency efficiently.
\n
\n
1. Introduction
\n
When crawling many URLs:
\n
\n
Basic: Use arun() in a loop (simple but less efficient)
\n
Better: Use arun_many(), which efficiently handles multiple URLs with proper concurrency control
\n
Best: Customize dispatcher behavior for your specific needs (memory management, rate limits, etc.)
\n
\n
Why Dispatchers?
\n
\n
Adaptive: Memory-based dispatchers can pause or slow down based on system resources
\n
Rate-limiting: Built-in rate limiting with exponential backoff for 429/503 responses
\n
Real-time Monitoring: Live dashboard of ongoing tasks, memory usage, and performance
\n
Flexibility: Choose between memory-adaptive or semaphore-based concurrency
\n
\n\n
2. Core Components
\n
2.1 Rate Limiter
\n
classRateLimiter:\n def__init__(\n # Random delay range between requests\n base_delay: Tuple[float, float] = (1.0, 3.0), \n\n # Maximum backoff delay\n max_delay: float = 60.0, \n\n # Retries before giving up\n max_retries: int = 3, \n\n # Status codes triggering backoff\n rate_limit_codes: List[int] = [429, 503] \n )\n
\n
Hereâs the revised and simplified explanation of the RateLimiter, focusing on constructor parameters and adhering to your markdown style and mkDocs guidelines.
\n
RateLimiter Constructor Parameters
\n
The RateLimiter is a utility that helps manage the pace of requests to avoid overloading servers or getting blocked due to rate limits. It operates internally to delay requests and handle retries but can be configured using its constructor parameters.
\n
Parameters of the RateLimiter constructor:
\n
1.âbase_delay (Tuple[float, float], default: (1.0, 3.0)) \nââThe range for a random delay (in seconds) between consecutive requests to the same domain.
\n
\n
A random delay is chosen between base_delay[0] and base_delay[1] for each request.
\n
This prevents sending requests at a predictable frequency, reducing the chances of triggering rate limits.
\n
\n
Example: \nIf base_delay = (2.0, 5.0), delays could be randomly chosen as 2.3s, 4.1s, etc.
\n\n
2.âmax_delay (float, default: 60.0) \nââThe maximum allowable delay when rate-limiting errors occur.
\n
\n
When servers return rate-limit responses (e.g., 429 or 503), the delay increases exponentially with jitter.
\n
The max_delay ensures the delay doesnât grow unreasonably high, capping it at this value.
\n
\n
Example: \nFor a max_delay = 30.0, even if backoff calculations suggest a delay of 45s, it will cap at 30s.
\n\n
3.âmax_retries (int, default: 3) \nââThe maximum number of retries for a request if rate-limiting errors occur.
\n
\n
After encountering a rate-limit response, the RateLimiter retries the request up to this number of times.
\n
If all retries fail, the request is marked as failed, and the process continues.
\n
\n
Example: \nIf max_retries = 3, the system retries a failed request three times before giving up.
\n\n
4.ârate_limit_codes (List[int], default: [429, 503]) \nââA list of HTTP status codes that trigger the rate-limiting logic.
\n
\n
These status codes indicate the server is overwhelmed or actively limiting requests.
\n
You can customize this list to include other codes based on specific server behavior.
\n
\n
Example: \nIf rate_limit_codes = [429, 503, 504], the crawler will back off on these three error codes.
\n\n
How to Use the RateLimiter:
\n
Hereâs an example of initializing and using a RateLimiter in your project:
\n
from crawl4ai import RateLimiter\n\n# Create a RateLimiter with custom settings\nrate_limiter = RateLimiter(\n base_delay=(2.0, 4.0), # Random delay between 2-4 seconds\n max_delay=30.0, # Cap delay at 30 seconds\n max_retries=5, # Retry up to 5 times on rate-limiting errors\n rate_limit_codes=[429, 503]# Handle these HTTP status codes\n)\n\n# RateLimiter will handle delays and retries internally\n# No additional setup is required for its operation\n
\n
The RateLimiter integrates seamlessly with dispatchers like MemoryAdaptiveDispatcher and SemaphoreDispatcher, ensuring requests are paced correctly without user intervention. Its internal mechanisms manage delays and retries to avoid overwhelming servers while maximizing efficiency.
\n
2.2 Crawler Monitor
\n
The CrawlerMonitor provides real-time visibility into crawling operations:
\n
from crawl4ai import CrawlerMonitor, DisplayMode\nmonitor = CrawlerMonitor(\n # Maximum rows in live display\n max_visible_rows=15, \n\n # DETAILED or AGGREGATED view\n display_mode=DisplayMode.DETAILED \n)\n
\n
Display Modes:
\n\n
DETAILED: Shows individual task status, memory usage, and timing
\n
AGGREGATED: Displays summary statistics and overall progress
\n\n\n
3. Available Dispatchers
\n
3.1 MemoryAdaptiveDispatcher (Default)
\n
Automatically manages concurrency based on system memory usage:
\n
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher\n\ndispatcher = MemoryAdaptiveDispatcher(\n memory_threshold_percent=90.0, # Pause if memory exceeds this\n check_interval=1.0, # How often to check memory\n max_session_permit=10, # Maximum concurrent tasks\n rate_limiter=RateLimiter( # Optional rate limiting\n base_delay=(1.0, 2.0),\n max_delay=30.0,\n max_retries=2\n ),\n monitor=CrawlerMonitor( # Optional monitoring\n max_visible_rows=15,\n display_mode=DisplayMode.DETAILED\n )\n)\n
\n
Constructor Parameters:
\n
1.âmemory_threshold_percent (float, default: 90.0) \nââSpecifies the memory usage threshold (as a percentage). If system memory usage exceeds this value, the dispatcher pauses crawling to prevent system overload.
\n
2.âcheck_interval (float, default: 1.0) \nââThe interval (in seconds) at which the dispatcher checks system memory usage.
\n
3.âmax_session_permit (int, default: 10) \nââThe maximum number of concurrent crawling tasks allowed. This ensures resource limits are respected while maintaining concurrency.
\n
4.âmemory_wait_timeout (float, default: 600.0)\nââOptional timeout (in seconds). If memory usage exceeds memory_threshold_percent for longer than this duration, a MemoryError is raised.
\n
5.ârate_limiter (RateLimiter, default: None) \nââOptional rate-limiting logic to avoid server-side blocking (e.g., for handling 429 or 503 errors). See RateLimiter for details.
\n
6.âmonitor (CrawlerMonitor, default: None) \nââOptional monitoring for real-time task tracking and performance insights. See CrawlerMonitor for details.
\n\n
3.2 SemaphoreDispatcher
\n
Provides simple concurrency control with a fixed limit:
1.âmax_session_permit (int, default: 20) \nââThe maximum number of concurrent crawling tasks allowed, irrespective of semaphore slots.
\n
2.ârate_limiter (RateLimiter, default: None) \nââOptional rate-limiting logic to avoid overwhelming servers. See RateLimiter for details.
\n
3.âmonitor (CrawlerMonitor, default: None) \nââOptional monitoring for tracking task progress and resource usage. See CrawlerMonitor for details.
\n\n
4. Usage Examples
\n
4.1 Batch Processing (Default)
\n
asyncdefcrawl_batch():\n browser_config = BrowserConfig(headless=True, verbose=False)\n run_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n stream=False# Default: get all results at once\n )\n\n dispatcher = MemoryAdaptiveDispatcher(\n memory_threshold_percent=70.0,\n check_interval=1.0,\n max_session_permit=10,\n monitor=CrawlerMonitor(\n display_mode=DisplayMode.DETAILED\n )\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n # Get all results at once\n results = await crawler.arun_many(\n urls=urls,\n config=run_config,\n dispatcher=dispatcher\n )\n\n # Process all results after completion\n for result in results:\n if result.success:\n await process_result(result)\n else:\n print(f\"Failed to crawl {result.url}: {result.error_message}\")\n
\n
Review: \n- Purpose: Executes a batch crawl with all URLs processed together after crawling is complete. \n- Dispatcher: Uses MemoryAdaptiveDispatcher to manage concurrency and system memory. \n- Stream: Disabled (stream=False), so all results are collected at once for post-processing. \n- Best Use Case: When you need to analyze results in bulk rather than individually during the crawl.
\n\n
4.2 Streaming Mode
\n
asyncdefcrawl_streaming():\n browser_config = BrowserConfig(headless=True, verbose=False)\n run_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n stream=True# Enable streaming mode\n )\n\n dispatcher = MemoryAdaptiveDispatcher(\n memory_threshold_percent=70.0,\n check_interval=1.0,\n max_session_permit=10,\n monitor=CrawlerMonitor(\n display_mode=DisplayMode.DETAILED\n )\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n # Process results as they become available\n asyncfor result inawait crawler.arun_many(\n urls=urls,\n config=run_config,\n dispatcher=dispatcher\n ):\n if result.success:\n # Process each result immediately\n await process_result(result)\n else:\n print(f\"Failed to crawl {result.url}: {result.error_message}\")\n
\n
Review: \n- Purpose: Enables streaming to process results as soon as theyâre available. \n- Dispatcher: Uses MemoryAdaptiveDispatcher for concurrency and memory management. \n- Stream: Enabled (stream=True), allowing real-time processing during crawling. \n- Best Use Case: When you need to act on results immediately, such as for real-time analytics or progressive data storage.
Review: \n- Purpose: Uses SemaphoreDispatcher to limit concurrency with a fixed number of slots. \n- Dispatcher: Configured with a semaphore to control parallel crawling tasks. \n- Rate Limiter: Prevents servers from being overwhelmed by pacing requests. \n- Best Use Case: When you want precise control over the number of concurrent requests, independent of system memory.
\n\n
4.4 Robots.txt Consideration
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n urls = [\n \"https://example1.com\",\n \"https://example2.com\",\n \"https://example3.com\"\n ]\n\n config = CrawlerRunConfig(\n cache_mode=CacheMode.ENABLED,\n check_robots_txt=True, # Will respect robots.txt for each URL\n semaphore_count=3# Max concurrent requests\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n asyncfor result in crawler.arun_many(urls, config=config):\n if result.success:\n print(f\"Successfully crawled {result.url}\")\n elif result.status_code == 403and\"robots.txt\"in result.error_message:\n print(f\"Skipped {result.url} - blocked by robots.txt\")\n else:\n print(f\"Failed to crawl {result.url}: {result.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Review: \n- Purpose: Ensures compliance with robots.txt rules for ethical and legal web crawling. \n- Configuration: Set check_robots_txt=True to validate each URL against robots.txt before crawling. \n- Dispatcher: Handles requests with concurrency limits (semaphore_count=3). \n- Best Use Case: When crawling websites that strictly enforce robots.txt policies or for responsible crawling practices.
for result in results:\n if result.success:\n dr = result.dispatch_result\n print(f\"URL: {result.url}\")\n print(f\"Memory: {dr.memory_usage:.1f}MB\")\n print(f\"Duration: {dr.end_time - dr.start_time}\")\n
\n
6. URL-Specific Configurations
\n
When crawling diverse content types, you often need different configurations for different URLs. For example:\n- PDFs need specialized extraction\n- Blog pages benefit from content filtering\n- Dynamic sites need JavaScript execution\n- API endpoints need JSON parsing
\n
6.1 Basic URL Pattern Matching
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode\nfrom crawl4ai.processors.pdf import PDFContentScrapingStrategy\nfrom crawl4ai.extraction_strategy import JsonCssExtractionStrategy\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefcrawl_mixed_content():\n # Configure different strategies for different content\n configs = [\n # PDF files - specialized extraction\n CrawlerRunConfig(\n url_matcher=\"*.pdf\",\n scraping_strategy=PDFContentScrapingStrategy()\n ),\n\n # Blog/article pages - content filtering\n CrawlerRunConfig(\n url_matcher=[\"*/blog/*\", \"*/article/*\"],\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.48)\n )\n ),\n\n # Dynamic pages - JavaScript execution\n CrawlerRunConfig(\n url_matcher=lambda url: 'github.com'in url,\n js_code=\"window.scrollTo(0, 500);\"\n ),\n\n # API endpoints - JSON extraction\n CrawlerRunConfig(\n url_matcher=lambda url: 'api'in url or url.endswith('.json'),\n # Custome settings for JSON extraction\n ),\n\n # Default config for everything else\n CrawlerRunConfig() # No url_matcher means it matches ALL URLs (fallback)\n ]\n\n # Mixed URLs\n urls = [\n \"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf\",\n \"https://blog.python.org/\",\n \"https://github.com/microsoft/playwright\",\n \"https://httpbin.org/json\",\n \"https://example.com/\"\n ]\n\n asyncwith AsyncWebCrawler() as crawler:\n results = await crawler.arun_many(\n urls=urls,\n config=configs # Pass list of configs\n )\n\n for result in results:\n print(f\"{result.url}: {len(result.markdown)} chars\")\n
\n
6.2 Advanced Pattern Matching
\n
Important: A CrawlerRunConfig without url_matcher (or with url_matcher=None) matches ALL URLs. This makes it perfect as a default/fallback configuration.
\n
The url_matcher parameter supports three types of patterns:
\n
Glob Patterns (Strings)
\n
# Simple patterns\n\"*.pdf\"# Any PDF file\n\"*/api/*\"# Any URL with /api/ in path\n\"https://*.example.com/*\"# Subdomain matching\n\"*://example.com/blog/*\"# Any protocol\n
\n
Custom Functions
\n
# Complex logic with lambdas\nlambda url: url.startswith('https://') and'secure'in url\nlambda url: len(url) > 50and url.count('/') > 5\nlambda url: any(domain in url for domain in ['api.', 'data.', 'feed.'])\n
\n
Mixed Lists with AND/OR Logic
\n
# Combine multiple conditions\nCrawlerRunConfig(\n url_matcher=[\n \"https://*\", # Must be HTTPS\n lambda url: 'internal'in url, # Must contain 'internal'\n lambda url: not url.endswith('.pdf') # Must not be PDF\n ],\n match_mode=MatchMode.AND # ALL conditions must match\n)\n
Crawl4AI can capture all network requests and browser console messages during a crawl, which is invaluable for debugging, security analysis, or understanding page behavior.
\n
Configuration
\n
To enable network and console capturing, use these configuration options:
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\n# Enable both network request capture and console message capture\nconfig = CrawlerRunConfig(\n capture_network_requests=True, # Capture all network requests and responses\n capture_console_messages=True# Capture all browser console output\n)\n
\n
Example Usage
\n
import asyncio\nimport json\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Enable both network request capture and console message capture\n config = CrawlerRunConfig(\n capture_network_requests=True,\n capture_console_messages=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=config\n )\n\n if result.success:\n # Analyze network requests\n if result.network_requests:\n print(f\"Captured {len(result.network_requests)} network events\")\n\n # Count request types\n request_count = len([r for r in result.network_requests if r.get(\"event_type\") == \"request\"])\n response_count = len([r for r in result.network_requests if r.get(\"event_type\") == \"response\"])\n failed_count = len([r for r in result.network_requests if r.get(\"event_type\") == \"request_failed\"])\n\n print(f\"Requests: {request_count}, Responses: {response_count}, Failed: {failed_count}\")\n\n # Find API calls\n api_calls = [r for r in result.network_requests \n if r.get(\"event_type\") == \"request\"and\"api\"in r.get(\"url\", \"\")]\n if api_calls:\n print(f\"Detected {len(api_calls)} API calls:\")\n for call in api_calls[:3]: # Show first 3\n print(f\" - {call.get('method')}{call.get('url')}\")\n\n # Analyze console messages\n if result.console_messages:\n print(f\"Captured {len(result.console_messages)} console messages\")\n\n # Group by type\n message_types = {}\n for msg in result.console_messages:\n msg_type = msg.get(\"type\", \"unknown\")\n message_types[msg_type] = message_types.get(msg_type, 0) + 1\n\n print(\"Message types:\", message_types)\n\n # Show errors (often the most important)\n errors = [msg for msg in result.console_messages if msg.get(\"type\") == \"error\"]\n if errors:\n print(f\"Found {len(errors)} console errors:\")\n for err in errors[:2]: # Show first 2\n print(f\" - {err.get('text', '')[:100]}\")\n\n # Export all captured data to a file for detailed analysis\n withopen(\"network_capture.json\", \"w\") as f:\n json.dump({\n \"url\": result.url,\n \"network_requests\": result.network_requests or [],\n \"console_messages\": result.console_messages or []\n }, f, indent=2)\n\n print(\"Exported detailed capture data to network_capture.json\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Captured Data Structure
\n
Network Requests
\n
The result.network_requests contains a list of dictionaries, each representing a network event with these common fields:
\n
\n\n
\n
Field
\n
Description
\n
\n\n\n
\n
event_type
\n
Type of event: \"request\", \"response\", or \"request_failed\"
Ad/Tracker Analysis: Detect and catalog advertising or tracking calls
\n\n
This capability is especially valuable for complex sites with heavy JavaScript, single-page applications, or when you need to understand the exact communication happening between a browser and servers.
Crawl4AI provides specialized strategies for handling and extracting content from PDF files. These strategies allow you to seamlessly integrate PDF processing into your crawling workflows, whether the PDFs are hosted online or stored locally.
\n
PDFCrawlerStrategy
\n
Overview
\n
PDFCrawlerStrategy is an implementation of AsyncCrawlerStrategy designed specifically for PDF documents. Instead of interpreting the input URL as an HTML webpage, this strategy treats it as a pointer to a PDF file. It doesn't perform deep crawling or HTML parsing itself but rather prepares the PDF source for a dedicated PDF scraping strategy. Its primary role is to identify the PDF source (web URL or local file) and pass it along the processing pipeline in a way that AsyncWebCrawler can handle.
\n
When to Use
\n
Use PDFCrawlerStrategy when you need to:\n- Process PDF files using the AsyncWebCrawler.\n- Handle PDFs from both web URLs (e.g., https://example.com/document.pdf) and local file paths (e.g., file:///path/to/your/document.pdf).\n- Integrate PDF content extraction into a unified CrawlResult object, allowing consistent handling of PDF data alongside web page data.
\n
Key Methods and Their Behavior
\n
\n
__init__(self, logger: AsyncLogger = None):
\n
Initializes the strategy.
\n
logger: An optional AsyncLogger instance (from crawl4ai.async_logger) for logging purposes.
This method is called by the AsyncWebCrawler during the arun process.
\n
It takes the url (which should point to a PDF) and creates a minimal AsyncCrawlResponse.
\n
The html attribute of this response is typically empty or a placeholder, as the actual PDF content processing is deferred to the PDFContentScrapingStrategy (or a similar PDF-aware scraping strategy).
\n
It sets response_headers to indicate \"application/pdf\" and status_code to 200.
\n
\n
\n
async close(self):
\n
A method for cleaning up any resources used by the strategy. For PDFCrawlerStrategy, this is usually minimal.
Enables asynchronous context management for the strategy, allowing it to be used with async with.
\n
\n
\n
\n
Example Usage
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy\n\nasyncdefmain():\n # Initialize the PDF crawler strategy\n pdf_crawler_strategy = PDFCrawlerStrategy()\n\n # PDFCrawlerStrategy is typically used in conjunction with PDFContentScrapingStrategy\n # The scraping strategy handles the actual PDF content extraction\n pdf_scraping_strategy = PDFContentScrapingStrategy()\n run_config = CrawlerRunConfig(scraping_strategy=pdf_scraping_strategy)\n\n asyncwith AsyncWebCrawler(crawler_strategy=pdf_crawler_strategy) as crawler:\n # Example with a remote PDF URL\n pdf_url = \"https://arxiv.org/pdf/2310.06825.pdf\"# A public PDF from arXiv\n\n print(f\"Attempting to process PDF: {pdf_url}\")\n result = await crawler.arun(url=pdf_url, config=run_config)\n\n if result.success:\n print(f\"Successfully processed PDF: {result.url}\")\n print(f\"Metadata Title: {result.metadata.get('title', 'N/A')}\")\n # Further processing of result.markdown, result.media, etc.\n # would be done here, based on what PDFContentScrapingStrategy extracts.\n if result.markdown andhasattr(result.markdown, 'raw_markdown'):\n print(f\"Extracted text (first 200 chars): {result.markdown.raw_markdown[:200]}...\")\n else:\n print(\"No markdown (text) content extracted.\")\n else:\n print(f\"Failed to process PDF: {result.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Pros and Cons
\n
Pros:\n- Enables AsyncWebCrawler to handle PDF sources directly using familiar arun calls.\n- Provides a consistent interface for specifying PDF sources (URLs or local paths).\n- Abstracts the source handling, allowing a separate scraping strategy to focus on PDF content parsing.
\n
Cons:\n- Does not perform any PDF data extraction itself; it strictly relies on a compatible scraping strategy (like PDFContentScrapingStrategy) to process the PDF.\n- Has limited utility on its own; most of its value comes from being paired with a PDF-specific content scraping strategy.
\n\n
PDFContentScrapingStrategy
\n
Overview
\n
PDFContentScrapingStrategy is an implementation of ContentScrapingStrategy designed to extract text, metadata, and optionally images from PDF documents. It is intended to be used in conjunction with a crawler strategy that can provide it with a PDF source, such as PDFCrawlerStrategy. This strategy uses the NaivePDFProcessorStrategy internally to perform the low-level PDF parsing.
\n
When to Use
\n
Use PDFContentScrapingStrategy when your AsyncWebCrawler (often configured with PDFCrawlerStrategy) needs to:\n- Extract textual content page by page from a PDF document.\n- Retrieve standard metadata embedded within the PDF (e.g., title, author, subject, creation date, page count).\n- Optionally, extract images contained within the PDF pages. These images can be saved to a local directory or made available for further processing.\n- Produce a ScrapingResult that can be converted into a CrawlResult, making PDF content accessible in a manner similar to HTML web content (e.g., text in result.markdown, metadata in result.metadata).
\n
Key Configuration Attributes
\n
When initializing PDFContentScrapingStrategy, you can configure its behavior using the following attributes:\n- extract_images: bool = False: If True, the strategy will attempt to extract images from the PDF.\n- save_images_locally: bool = False: If True (and extract_images is also True), extracted images will be saved to disk in the image_save_dir. If False, image data might be available in another form (e.g., base64, depending on the underlying processor) but not saved as separate files by this strategy.\n- image_save_dir: str = None: Specifies the directory where extracted images should be saved if save_images_locally is True. If None, a default or temporary directory might be used.\n- batch_size: int = 4: Defines how many PDF pages are processed in a single batch. This can be useful for managing memory when dealing with very large PDF documents.\n- logger: AsyncLogger = None: An optional AsyncLogger instance for logging.
Initializes the strategy with configurations for image handling, batch processing, and logging. It sets up an internal NaivePDFProcessorStrategy instance which performs the actual PDF parsing.
The asynchronous version of scrap. Under the hood, it typically runs the synchronous scrap method in a separate thread using asyncio.to_thread to avoid blocking the event loop.
\n
\n
\n
_get_pdf_path(self, url: str) -> str:
\n
A private helper method to manage PDF file access. If the url is remote (http/https), it downloads the PDF to a temporary local file and returns its path. If url indicates a local file (file:// or a direct path), it resolves and returns the local path.
\n
\n
\n
\n
Example Usage
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy\nimport os # For creating image directory\n\nasyncdefmain():\n # Define the directory for saving extracted images\n image_output_dir = \"./my_pdf_images\"\n os.makedirs(image_output_dir, exist_ok=True)\n\n # Configure the PDF content scraping strategy\n # Enable image extraction and specify where to save them\n pdf_scraping_cfg = PDFContentScrapingStrategy(\n extract_images=True,\n save_images_locally=True,\n image_save_dir=image_output_dir,\n batch_size=2# Process 2 pages at a time for demonstration\n )\n\n # The PDFCrawlerStrategy is needed to tell AsyncWebCrawler how to \"crawl\" a PDF\n pdf_crawler_cfg = PDFCrawlerStrategy()\n\n # Configure the overall crawl run\n run_cfg = CrawlerRunConfig(\n scraping_strategy=pdf_scraping_cfg # Use our PDF scraping strategy\n )\n\n # Initialize the crawler with the PDF-specific crawler strategy\n asyncwith AsyncWebCrawler(crawler_strategy=pdf_crawler_cfg) as crawler:\n pdf_url = \"https://arxiv.org/pdf/2310.06825.pdf\"# Example PDF\n\n print(f\"Starting PDF processing for: {pdf_url}\")\n result = await crawler.arun(url=pdf_url, config=run_cfg)\n\n if result.success:\n print(\"\\n--- PDF Processing Successful ---\")\n print(f\"Processed URL: {result.url}\")\n\n print(\"\\n--- Metadata ---\")\n for key, value in result.metadata.items():\n print(f\" {key.replace('_', ' ').title()}: {value}\")\n\n if result.markdown andhasattr(result.markdown, 'raw_markdown'):\n print(f\"\\n--- Extracted Text (Markdown Snippet) ---\")\n print(result.markdown.raw_markdown[:500].strip() + \"...\")\n else:\n print(\"\\nNo text (markdown) content extracted.\")\n\n if result.media and result.media.get(\"images\"):\n print(f\"\\n--- Image Extraction ---\")\n print(f\"Extracted {len(result.media['images'])} image(s).\")\n for i, img_info inenumerate(result.media[\"images\"][:2]): # Show info for first 2 images\n print(f\" Image {i+1}:\")\n print(f\" Page: {img_info.get('page')}\")\n print(f\" Format: {img_info.get('format', 'N/A')}\")\n if img_info.get('path'):\n print(f\" Saved at: {img_info.get('path')}\")\n else:\n print(\"\\nNo images were extracted (or extract_images was False).\")\n else:\n print(f\"\\n--- PDF Processing Failed ---\")\n print(f\"Error: {result.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Pros and Cons
\n
Pros:\n- Provides a comprehensive way to extract text, metadata, and (optionally) images from PDF documents.\n- Handles both remote PDFs (via URL) and local PDF files.\n- Configurable image extraction allows saving images to disk or accessing their data.\n- Integrates smoothly with the CrawlResult object structure, making PDF-derived data accessible in a way consistent with web-scraped data.\n- The batch_size parameter can help in managing memory consumption when processing large or numerous PDF pages.
\n
Cons:\n- Extraction quality and performance can vary significantly depending on the PDF's complexity, encoding, and whether it's image-based (scanned) or text-based.\n- Image extraction can be resource-intensive (both CPU and disk space if save_images_locally is true).\n- Relies on NaivePDFProcessorStrategy internally, which might have limitations with very complex layouts, encrypted PDFs, or forms compared to more sophisticated PDF parsing libraries. Scanned PDFs will not yield text unless an OCR step is performed (which is not part of this strategy by default).\n- Link extraction from PDFs can be basic and depends on how hyperlinks are embedded in the document.
This guide covers proxy configuration and security features in Crawl4AI, including SSL certificate analysis and proxy rotation strategies.
\n
Understanding Proxy Configuration
\n
Crawl4AI recommends configuring proxies per request through CrawlerRunConfig.proxy_config. This gives you precise control, enables rotation strategies, and keeps examples simple enough to copy, paste, and run.
\n
Basic Proxy Setup
\n
Configure proxies that apply to each crawl operation:
CrawlerRunConfig.proxy_config keeps each request self-contained, so swapping proxies or rotation strategies is just a matter of building a new run configuration.
\n
\n
Supported Proxy Formats
\n
The ProxyConfig.from_string() method supports multiple formats:
Load proxies from environment variables for easy configuration:
\n
import os\nfrom crawl4ai import ProxyConfig, CrawlerRunConfig\n\n# Set environment variable\nos.environ[\"PROXIES\"] = \"ip1:port1:user1:pass1,ip2:port2:user2:pass2,ip3:port3\"\n\n# Load all proxies\nproxies = ProxyConfig.from_env()\nprint(f\"Loaded {len(proxies)} proxies\")\n\n# Use first proxy\nif proxies:\n run_config = CrawlerRunConfig(proxy_config=proxies[0])\n
\n
Rotating Proxies
\n
Crawl4AI supports automatic proxy rotation to distribute requests across multiple proxy servers. Rotation is applied per request using a rotation strategy on CrawlerRunConfig.
\n
Proxy Rotation (recommended)
\n
import asyncio\nimport re\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig\nfrom crawl4ai.proxy_strategy import RoundRobinProxyStrategy\n\nasyncdefmain():\n # Load proxies from environment\n proxies = ProxyConfig.from_env()\n ifnot proxies:\n print(\"No proxies found! Set PROXIES environment variable.\")\n return\n\n # Create rotation strategy\n proxy_strategy = RoundRobinProxyStrategy(proxies)\n\n # Configure per-request with proxy rotation\n browser_config = BrowserConfig(headless=True, verbose=False)\n run_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n proxy_rotation_strategy=proxy_strategy,\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n urls = [\"https://httpbin.org/ip\"] * (len(proxies) * 2) # Test each proxy twice\n\n print(f\"đ Testing {len(proxies)} proxies with rotation...\")\n results = await crawler.arun_many(urls=urls, config=run_config)\n\n for i, result inenumerate(results):\n if result.success:\n # Extract IP from response\n ip_match = re.search(r'(?:[0-9]{1,3}\\.){3}[0-9]{1,3}', result.html)\n if ip_match:\n detected_ip = ip_match.group(0)\n proxy_index = i % len(proxies)\n expected_ip = proxies[proxy_index].ip\n\n print(f\"â Request {i+1}: Proxy {proxy_index+1} -> IP {detected_ip}\")\n if detected_ip == expected_ip:\n print(\" đŻ IP matches proxy configuration\")\n else:\n print(f\" â ď¸ IP mismatch (expected {expected_ip})\")\n else:\n print(f\"â Request {i+1}: Could not extract IP from response\")\n else:\n print(f\"â Request {i+1}: Failed - {result.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
SSL Certificate Analysis
\n
Combine proxy usage with SSL certificate inspection for enhanced security analysis. SSL certificate fetching is configured per request via CrawlerRunConfig.
\n
Per-Request SSL Certificate Analysis
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nrun_config = CrawlerRunConfig(\n proxy_config={\n \"server\": \"http://proxy.example.com:8080\",\n \"username\": \"user\",\n \"password\": \"pass\",\n },\n fetch_ssl_certificate=True, # Enable SSL certificate analysis for this request\n)\n\n\nasyncdefmain():\n browser_config = BrowserConfig()\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(url=\"https://example.com\", config=run_config)\n\n if result.success:\n print(f\"â Crawled via proxy: {result.url}\")\n\n # Analyze SSL certificate\n if result.ssl_certificate:\n cert = result.ssl_certificate\n print(\"đ SSL Certificate Info:\")\n print(f\" Issuer: {cert.issuer}\")\n print(f\" Subject: {cert.subject}\")\n print(f\" Valid until: {cert.valid_until}\")\n print(f\" Fingerprint: {cert.fingerprint}\")\n\n # Export certificate\n cert.to_json(\"certificate.json\")\n print(\"đž Certificate exported to certificate.json\")\n else:\n print(\"â ď¸ No SSL certificate information available\")\n\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Security Best Practices
\n
1. Proxy Rotation for Anonymity
\n
from crawl4ai import CrawlerRunConfig, ProxyConfig\nfrom crawl4ai.proxy_strategy import RoundRobinProxyStrategy\n\n# Use multiple proxies to avoid IP blocking\nproxies = ProxyConfig.from_env(\"PROXIES\")\nstrategy = RoundRobinProxyStrategy(proxies)\n\n# Configure rotation per request (recommended)\nrun_config = CrawlerRunConfig(proxy_rotation_strategy=strategy)\n\n# For a fixed proxy across all requests, just reuse the same run_config instance\nstatic_run_config = run_config\n
\n
2. SSL Certificate Verification
\n
from crawl4ai import CrawlerRunConfig\n\n# Always verify SSL certificates when possible\n# Per-request (affects specific requests)\nrun_config = CrawlerRunConfig(fetch_ssl_certificate=True)\n
\n
3. Environment Variable Security
\n
# Use environment variables for sensitive proxy credentials\n# Avoid hardcoding usernames/passwords in code\nexport PROXIES=\"ip1:port1:user1:pass1,ip2:port2:user2:pass2\"\n
\n
4. SOCKS5 for Enhanced Security
\n
from crawl4ai import CrawlerRunConfig\n\n# Prefer SOCKS5 proxies for better protocol support\nrun_config = CrawlerRunConfig(proxy_config=\"socks5://proxy.example.com:1080\")\n
\n
Migration from Deprecated proxy Parameter
\n
\n
\"Deprecation Notice\"\n The legacy proxy argument on BrowserConfig is deprecated. Configure proxies through CrawlerRunConfig.proxy_config so each request fully describes its network settings.
\n
\n
# Old (deprecated) approach\n# from crawl4ai import BrowserConfig\n# browser_config = BrowserConfig(proxy_config=\"http://proxy.example.com:8080\")\n\n# New (preferred) approach\nfrom crawl4ai import CrawlerRunConfig\nrun_config = CrawlerRunConfig(proxy_config=\"http://proxy.example.com:8080\")\n
Session management in Crawl4AI is a powerful feature that allows you to maintain state across multiple requests, making it particularly suitable for handling complex multi-step crawling tasks. It enables you to reuse the same browser tab (or page object) across sequential actions and crawls, which is beneficial for:
\n
\n
Performing JavaScript actions before and after crawling.
\n
Executing multiple sequential crawls faster without needing to reopen tabs or allocate memory repeatedly.
\n
\n
Note: This feature is designed for sequential workflows and is not suitable for parallel operations.
\n\n
Basic Session Usage
\n
Use BrowserConfig and CrawlerRunConfig to maintain state with a session_id:
\n
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig\n\nasyncwithAsyncWebCrawler() as crawler:\n session_id = \"my_session\"\n\n # Define configurations\n config1 = CrawlerRunConfig(\n url=\"https://example.com/page1\", session_id=session_id\n )\n config2 = CrawlerRunConfig(\n url=\"https://example.com/page2\", session_id=session_id\n )\n\n # First request\n result1 = await crawler.arun(config=config1)\n\n # Subsequent request using the same session\n result2 = await crawler.arun(config=config2)\n\n # Clean up when done\n await crawler.crawler_strategy.kill_session(session_id)\n
\n\n
Dynamic Content with Sessions
\n
Here's an example of crawling GitHub commits across multiple pages while preserving session state:
This example shows:\n1. Reusing the same session_id across multiple requests.\n2. Executing JavaScript to load more content dynamically.\n3. Properly closing the session to free resources.
\n\n
Advanced Technique 1: Custom Execution Hooks
\n
\n
Warning: You might feel confused by the end of the next few examples đ , so make sure you are comfortable with the order of the parts before you start this.
\n
\n
Use custom hooks to handle complex scenarios, such as waiting for content to load dynamically:
The SSLCertificate class encapsulates an SSL certificateâs data and allows exporting it in various formats (PEM, DER, JSON, or text). Itâs used within Crawl4AI whenever you set fetch_ssl_certificate=True in your CrawlerRunConfig.
\n
1. Overview
\n
Location: crawl4ai/ssl_certificate.py
\n
classSSLCertificate:\n \"\"\"\n Represents an SSL certificate with methods to export in various formats.\n\n Main Methods:\n - from_url(url, timeout=10)\n - from_file(file_path)\n - from_binary(binary_data)\n - to_json(filepath=None)\n - to_pem(filepath=None)\n - to_der(filepath=None)\n ...\n\n Common Properties:\n - issuer\n - subject\n - valid_from\n - valid_until\n - fingerprint\n \"\"\"\n
\n
Typical Use Case
\n\n
You enable certificate fetching in your crawl by:\n
1.âTimeout: SSLCertificate.from_url internally uses a default 10s socket connect and wraps SSL. \n2.âBinary Form: The certificate is loaded in ASN.1 (DER) form, then re-parsed by OpenSSL.crypto. \n3.âValidation: This does not validate the certificate chain or trust store. It only fetches and parses. \n4.âIntegration: Within Crawl4AI, you typically just set fetch_ssl_certificate=True in CrawlerRunConfig; the final resultâs ssl_certificate is automatically built. \n5.âExport: If you need to store or analyze a cert, the to_json and to_pem are quite universal.
\n\n
Summary
\n
\n
SSLCertificate is a convenience class for capturing and exporting the TLS certificate from your crawled site(s).
\n
Common usage is in the CrawlResult.ssl_certificate field, accessible after setting fetch_ssl_certificate=True.
\n
Offers quick access to essential certificate details (issuer, subject, fingerprint) and is easy to export (PEM, DER, JSON) for further analysis or server usage.
\n
\n
Use it whenever you need insight into a siteâs certificate or require some form of cryptographic or compliance check.
You're willing to trade some performance for better evasion
\n
\n
Best Practice: Progressive Enhancement
\n\n
Start with: Regular browser + Stealth mode
\n
If blocked: Switch to Undetected browser
\n
If still blocked: Combine Undetected browser + Stealth mode
\n\n
Stealth Mode
\n
Stealth mode is the simpler anti-bot solution that works with both regular and undetected browsers:
\n
from crawl4ai import AsyncWebCrawler, BrowserConfig\n\n# Enable stealth mode with regular browser\nbrowser_config = BrowserConfig(\n enable_stealth=True, # Simple flag to enable\n headless=False# Better for avoiding detection\n)\n\nasyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\"https://example.com\")\n
\n
What Stealth Mode Does:
\n
\n
Removes navigator.webdriver flag
\n
Modifies browser fingerprints
\n
Emulates realistic plugin behavior
\n
Adjusts navigator properties
\n
Fixes common automation leaks
\n
\n
Undetected Browser Mode
\n
For sites with sophisticated bot detection that stealth mode can't bypass, use the undetected browser adapter:
\n
Key Features
\n
\n
Drop-in Replacement: Uses the same API as regular browser mode
\n
Enhanced Stealth: Built-in patches to evade common detection methods
\n
Browser Adapter Pattern: Seamlessly switch between regular and undetected modes
\n
Automatic Installation: crawl4ai-setup installs all necessary browser dependencies
\n
\n
Quick Start
\n
import asyncio\nfrom crawl4ai import (\n AsyncWebCrawler, \n BrowserConfig, \n CrawlerRunConfig,\n UndetectedAdapter\n)\nfrom crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy\n\nasyncdefmain():\n # Create the undetected adapter\n undetected_adapter = UndetectedAdapter()\n\n # Create browser config\n browser_config = BrowserConfig(\n headless=False, # Headless mode can be detected easier\n verbose=True,\n )\n\n # Create the crawler strategy with undetected adapter\n crawler_strategy = AsyncPlaywrightCrawlerStrategy(\n browser_config=browser_config,\n browser_adapter=undetected_adapter\n )\n\n # Create the crawler with our custom strategy\n asyncwith AsyncWebCrawler(\n crawler_strategy=crawler_strategy,\n config=browser_config\n ) as crawler:\n # Your crawling code here\n result = await crawler.arun(\n url=\"https://example.com\",\n config=CrawlerRunConfig()\n )\n print(result.markdown[:500])\n\nasyncio.run(main())\n
\n
Combining Both Features
\n
For maximum evasion, combine stealth mode with undetected browser:
\n
from crawl4ai import AsyncWebCrawler, BrowserConfig, UndetectedAdapter\nfrom crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy\n\n# Create browser config with stealth enabled\nbrowser_config = BrowserConfig(\n enable_stealth=True, # Enable stealth mode\n headless=False\n)\n\n# Create undetected adapter\nadapter = UndetectedAdapter()\n\n# Create strategy with both features\nstrategy = AsyncPlaywrightCrawlerStrategy(\n browser_config=browser_config,\n browser_adapter=adapter\n)\n\nasyncwith AsyncWebCrawler(\n crawler_strategy=strategy,\n config=browser_config\n) as crawler:\n result = await crawler.arun(\"https://protected-site.com\")\n
\n
Examples
\n
Example 1: Basic Stealth Mode
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdeftest_stealth_mode():\n # Simple stealth mode configuration\n browser_config = BrowserConfig(\n enable_stealth=True,\n headless=False\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://bot.sannysoft.com\",\n config=CrawlerRunConfig(screenshot=True)\n )\n\n if result.success:\n print(\"â Successfully accessed bot detection test site\")\n # Save screenshot to verify detection results\n if result.screenshot:\n import base64\n withopen(\"stealth_test.png\", \"wb\") as f:\n f.write(base64.b64decode(result.screenshot))\n print(\"â Screenshot saved - check for green (passed) tests\")\n\nasyncio.run(test_stealth_mode())\n
\n
Example 2: Undetected Browser Mode
\n
import asyncio\nfrom crawl4ai import (\n AsyncWebCrawler,\n BrowserConfig,\n CrawlerRunConfig,\n UndetectedAdapter\n)\nfrom crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy\n\n\nasyncdefmain():\n # Create browser config\n browser_config = BrowserConfig(\n headless=False,\n verbose=True,\n )\n\n # Create the undetected adapter\n undetected_adapter = UndetectedAdapter()\n\n # Create the crawler strategy with the undetected adapter\n crawler_strategy = AsyncPlaywrightCrawlerStrategy(\n browser_config=browser_config,\n browser_adapter=undetected_adapter\n )\n\n # Create the crawler with our custom strategy\n asyncwith AsyncWebCrawler(\n crawler_strategy=crawler_strategy,\n config=browser_config\n ) as crawler:\n # Configure the crawl\n crawler_config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter()\n ),\n capture_console_messages=True, # Test adapter console capture\n )\n\n # Test on a site that typically detects bots\n print(\"Testing undetected adapter...\")\n result: CrawlResult = await crawler.arun(\n url=\"https://www.helloworld.org\", \n config=crawler_config\n )\n\n print(f\"Status: {result.status_code}\")\n print(f\"Success: {result.success}\")\n print(f\"Console messages captured: {len(result.console_messages or [])}\")\n print(f\"Markdown content (first 500 chars):\\n{result.markdown.raw_markdown[:500]}\")\n\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Browser Adapter Pattern
\n
The undetected browser support is implemented using an adapter pattern, allowing seamless switching between different browser implementations:
# Use selective undetected mode only for protected sites\nif is_protected_site(url):\n adapter = UndetectedAdapter()\nelse:\n adapter = PlaywrightAdapter() # Default adapter\n
\n
Future Plans
\n
Note: In future versions of Crawl4AI, we may enable stealth mode and undetected browser by default to provide better out-of-the-box success rates. For now, users should explicitly enable these features when needed.
\n
Conclusion
\n
Crawl4AI provides flexible anti-bot solutions:
\n\n
Start Simple: Use regular browser + stealth mode for most sites
\n
Escalate if Needed: Switch to undetected browser for sophisticated protection
\n
Combine for Maximum Effect: Use both features together when facing the toughest challenges
\n\n
Remember:\n- Always respect robots.txt and website terms of service\n- Use appropriate delays to avoid overwhelming servers\n- Consider the performance trade-offs of each approach\n- Test progressively to find the minimum necessary evasion level
Modern websites increasingly use virtual scrolling (also called windowed rendering or viewport rendering) to handle large datasets efficiently. This technique only renders visible items in the DOM, replacing content as users scroll. Popular examples include Twitter's timeline, Instagram's feed, and many data tables.
\n
Crawl4AI's Virtual Scroll feature automatically detects and handles these scenarios, ensuring you capture all content, not just what's initially visible.
\n
Understanding Virtual Scroll
\n
The Problem
\n
Traditional infinite scroll appends new content to existing content. Virtual scroll replaces content to maintain performance:
Without proper handling, crawlers only capture the currently visible items, missing the rest of the content.
\n
Three Scrolling Scenarios
\n
Crawl4AI's Virtual Scroll detects and handles three scenarios:
\n\n
No Change - Content doesn't update on scroll (static page or end reached)
\n
Content Appended - New items added to existing ones (traditional infinite scroll)
\n
Content Replaced - Items replaced with new ones (true virtual scroll)
\n\n
Only scenario 3 requires special handling, which Virtual Scroll automates.
\n
Basic Usage
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig\n\n# Configure virtual scroll\nvirtual_config = VirtualScrollConfig(\n container_selector=\"#feed\", # CSS selector for scrollable container\n scroll_count=20, # Number of scrolls to perform\n scroll_by=\"container_height\", # How much to scroll each time\n wait_after_scroll=0.5# Wait time (seconds) after each scroll\n)\n\n# Use in crawler configuration\nconfig = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n)\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=\"https://example.com\", config=config)\n # result.html contains ALL items from the virtual scroll\n
\n
Configuration Parameters
\n
VirtualScrollConfig
\n
\n\n
\n
Parameter
\n
Type
\n
Default
\n
Description
\n
\n\n\n
\n
container_selector
\n
str
\n
Required
\n
CSS selector for the scrollable container
\n
\n
\n
scroll_count
\n
int
\n
10
\n
Maximum number of scrolls to perform
\n
\n
\n
scroll_by
\n
str or int
\n
\"container_height\"
\n
Scroll amount per step
\n
\n
\n
wait_after_scroll
\n
float
\n
0.5
\n
Seconds to wait after each scroll
\n
\n\n
\n
Scroll By Options
\n
\n
\"container_height\" - Scroll by the container's visible height
\n
\"page_height\" - Scroll by the viewport height
\n
500 (integer) - Scroll by exact pixel amount
\n
\n
Real-World Examples
\n
Twitter-like Timeline
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig, BrowserConfig\n\nasyncdefcrawl_twitter_timeline():\n # Twitter replaces tweets as you scroll\n virtual_config = VirtualScrollConfig(\n container_selector=\"[data-testid='primaryColumn']\",\n scroll_count=30,\n scroll_by=\"container_height\",\n wait_after_scroll=1.0# Twitter needs time to load\n )\n\n browser_config = BrowserConfig(headless=True) # Set to False to watch it work\n config = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://twitter.com/search?q=AI\",\n config=config\n )\n\n # Extract tweet count\n import re\n tweets = re.findall(r'data-testid=\"tweet\"', result.html)\n print(f\"Captured {len(tweets)} tweets\")\n
\n
Instagram Grid
\n
asyncdefcrawl_instagram_grid():\n # Instagram uses virtualized grid for performance\n virtual_config = VirtualScrollConfig(\n container_selector=\"article\", # Main feed container\n scroll_count=50, # More scrolls for grid layout\n scroll_by=800, # Fixed pixel scrolling\n wait_after_scroll=0.8\n )\n\n config = CrawlerRunConfig(\n virtual_scroll_config=virtual_config,\n screenshot=True# Capture final state\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.instagram.com/explore/tags/photography/\",\n config=config\n )\n\n # Count posts\n posts = result.html.count('class=\"post\"')\n print(f\"Captured {posts} posts from virtualized grid\")\n
\n
Mixed Content (News Feed)
\n
Some sites mix static and virtualized content:
\n
asyncdefcrawl_mixed_feed():\n # Featured articles stay, regular articles virtualize\n virtual_config = VirtualScrollConfig(\n container_selector=\".main-feed\",\n scroll_count=25,\n scroll_by=\"container_height\",\n wait_after_scroll=0.5\n )\n\n config = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.example.com\",\n config=config\n )\n\n # Featured articles remain throughout\n featured = result.html.count('class=\"featured-article\"')\n regular = result.html.count('class=\"regular-article\"')\n\n print(f\"Featured (static): {featured}\")\n print(f\"Regular (virtualized): {regular}\")\n
\n
Virtual Scroll vs scan_full_page
\n
Both features handle dynamic content, but serve different purposes:
\n
\n\n
\n
Feature
\n
Virtual Scroll
\n
scan_full_page
\n
\n\n\n
\n
Purpose
\n
Capture content that's replaced during scroll
\n
Load content that's appended during scroll
\n
\n
\n
Use Case
\n
Twitter, Instagram, virtual tables
\n
Traditional infinite scroll, lazy-loaded images
\n
\n
\n
DOM Behavior
\n
Replaces elements
\n
Adds elements
\n
\n
\n
Memory Usage
\n
Efficient (merges content)
\n
Can grow large
\n
\n
\n
Configuration
\n
Requires container selector
\n
Works on full page
\n
\n\n
\n
When to Use Which?
\n
Use Virtual Scroll when:\n- Content disappears as you scroll (Twitter timeline)\n- DOM element count stays relatively constant\n- You need ALL items from a virtualized list\n- Container-based scrolling (not full page)
\n
Use scan_full_page when:\n- Content accumulates as you scroll\n- Images load lazily\n- Simple \"load more\" behavior\n- Full page scrolling
\n
Combining with Extraction
\n
Virtual Scroll works seamlessly with extraction strategies:
\n
from crawl4ai import LLMExtractionStrategy, LLMConfig\n\n# Define extraction schema\nschema = {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\", \n \"properties\": {\n \"author\": {\"type\": \"string\"},\n \"content\": {\"type\": \"string\"},\n \"timestamp\": {\"type\": \"string\"}\n }\n }\n}\n\n# Configure both virtual scroll and extraction\nconfig = CrawlerRunConfig(\n virtual_scroll_config=VirtualScrollConfig(\n container_selector=\"#timeline\",\n scroll_count=20\n ),\n extraction_strategy=LLMExtractionStrategy(\n llm_config=LLMConfig(provider=\"openai/gpt-4o-mini\"),\n schema=schema\n )\n)\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=\"...\", config=config)\n\n # Extracted data from ALL scrolled content\n import json\n posts = json.loads(result.extracted_content)\n print(f\"Extracted {len(posts)} posts from virtual scroll\")\n
\n
Performance Tips
\n\n
\n
Container Selection: Be specific with selectors. Using the correct container improves performance.
\n
\n
\n
Scroll Count: Start conservative and increase as needed:\n
# Start with fewer scrolls\nvirtual_config = VirtualScrollConfig(\n container_selector=\"#feed\",\n scroll_count=10 # Test with 10, increase if needed\n)\n
\n
\n
\n
Wait Times: Adjust based on site speed:\n
# Fast sites\nwait_after_scroll=0.2\n\n# Slower sites or heavy content\nwait_after_scroll=1.5\n
\n
\n
\n
Debug Mode: Set headless=False to watch scrolling:\n
browser_config = BrowserConfig(headless=False)\nasyncwith AsyncWebCrawler(config=browser_config) as crawler:\n # Watch the scrolling happen\n
\n
\n\n
How It Works Internally
\n\n
Detection Phase: Scrolls and compares HTML to detect behavior
\n
Capture Phase: For replaced content, stores HTML chunks at each position
\n
Merge Phase: Combines all chunks, removing duplicates based on text content
\n
Result: Complete HTML with all unique items
\n\n
The deduplication uses normalized text (lowercase, no spaces/symbols) to ensure accurate merging without false positives.
\n
Error Handling
\n
Virtual Scroll handles errors gracefully:
\n
# If container not found or scrolling fails\nresult = await crawler.arun(url=\"...\", config=config)\n\nif result.success:\n # Virtual scroll worked or wasn't needed\n print(f\"Captured {len(result.html)} characters\")\nelse:\n # Crawl failed entirely\n print(f\"Error: {result.error_message}\")\n
\n
If the container isn't found, crawling continues normally without virtual scroll.
\n
Complete Example
\n
See our comprehensive example that demonstrates:\n- Twitter-like feeds\n- Instagram grids \n- Traditional infinite scroll\n- Mixed content scenarios\n- Performance comparisons
\n
# Run the examples\ncd docs/examples\npython virtual_scroll_example.py\n
\n
The example includes a local test server with different scrolling behaviors for experimentation.
The AdaptiveCrawler class implements intelligent web crawling that automatically determines when sufficient information has been gathered to answer a query. It uses a three-layer scoring system to evaluate coverage, consistency, and saturation.
Returns:\n- coverage: Query term coverage score\n- consistency: Information consistency score \n- saturation: Content saturation score\n- confidence: Overall confidence score
\n
is_sufficient
\n
Boolean indicating whether sufficient information has been gathered.
\n
@property\ndefis_sufficient(self) -> bool\n
\n
state
\n
Access to the current crawl state.
\n
@property\ndefstate(self) -> CrawlState\n
\n
Methods
\n
get_relevant_content()
\n
Retrieve the most relevant content from the knowledge base.
\n
defget_relevant_content(\n self,\n top_k: int = 5\n) -> List[Dict[str, Any]]\n
\n
Parameters
\n
\n
top_k (int): Number of top relevant documents to return (default: 5)
\n
\n
Returns
\n
List of dictionaries containing:\n- url: The URL of the page\n- content: The page content\n- score: Relevance score\n- metadata: Additional page metadata
path (Union[str, Path]): Path to JSONL file to import
\n
\n
Configuration
\n
The AdaptiveConfig class controls the behavior of adaptive crawling:
\n
@dataclass\nclassAdaptiveConfig:\n confidence_threshold: float = 0.8# Stop when confidence reaches this\n max_pages: int = 50# Maximum pages to crawl\n top_k_links: int = 5# Links to follow per page\n min_gain_threshold: float = 0.1# Minimum expected gain to continue\n save_state: bool = False# Auto-save crawl state\n state_path: Optional[str] = None# Path for state persistence\n
In Crawl4AIâs latest configuration model, nearly all parameters that once went directly to arun() are now part of CrawlerRunConfig.âWhen calling arun(), you provide:
Below is an organized look at the parameters that can go inside CrawlerRunConfig, divided by their functional areas.âFor Browser settings (e.g., headless, browser_type), see BrowserConfig.
\n\n
1.âCore Usage
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n run_config = CrawlerRunConfig(\n verbose=True, # Detailed logging\n cache_mode=CacheMode.ENABLED, # Use normal read/write cache\n check_robots_txt=True, # Respect robots.txt rules\n # ...âother parameters\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=run_config\n )\n\n # Check if blocked by robots.txt\n ifnot result.success and result.status_code == 403:\n print(f\"Error: {result.error_message}\")\n
\n
Key Fields:\n- verbose=True logs each crawl step.â \n- cache_mode decides how to read/write the local crawl cache.
\n\n
2.âCache Control
\n
cache_mode (default: CacheMode.ENABLED) \nUse a built-in enum from CacheMode:
\n
\n
ENABLED: Normal cachingâreads if available, writes if missing.
\n
DISABLED: No cachingâalways refetch pages.
\n
READ_ONLY: Reads from cache only; no new writes.
\n
WRITE_ONLY: Writes to cache but doesnât read existing data.
\n
BYPASS: Skips reading cache for this crawl (though it might still write if set up that way).
CacheMode.BYPASS â Skip cache entirely; always fetch fresh, write result to cache.
\n
CacheMode.DISABLED â No caching at all; don't read or write.
\n
CacheMode.WRITE_ONLY â Never read from cache, but write results.
\n
CacheMode.READ_ONLY â Read from cache if available, never write.
\n
\n\n
3.âContent Processing & Selection
\n
3.1 Text Processing
\n
run_config = CrawlerRunConfig(\n word_count_threshold=10, # Ignore text blocks <10 words\n only_text=False, # If True, tries to remove non-text elements\n keep_data_attributes=False# Keep or discard data-* attributes\n)\n
\n
3.2 Content Selection
\n
run_config = CrawlerRunConfig(\n css_selector=\".main-content\", # Focus on .main-content region only\n excluded_tags=[\"form\", \"nav\"], # Remove entire tag blocks\n remove_forms=True, # Specifically strip <form> elements\n remove_overlay_elements=True, # Attempt to remove modals/popups\n)\n
\n
3.3 Link Handling
\n
run_config = CrawlerRunConfig(\n exclude_external_links=True, # Remove external links from final content\n exclude_social_media_links=True, # Remove links to known social sites\n exclude_domains=[\"ads.example.com\"], # Exclude links to these domains\n exclude_social_media_domains=[\"facebook.com\",\"twitter.com\"], # Extend the default list\n)\n
\n
3.4 Media Filtering
\n
run_config = CrawlerRunConfig(\n exclude_external_images=True# Strip images from other domains\n)\n
\n\n
4.âPage Navigation & Timing
\n
4.1 Basic Browser Flow
\n
run_config = CrawlerRunConfig(\n wait_for=\"css:.dynamic-content\", # Wait for .dynamic-content\n delay_before_return_html=2.0, # Wait 2s before capturing final HTML\n page_timeout=60000, # Navigation & script timeout (ms)\n)\n
\nIf re-used in subsequent arun() calls, the same tab/page context is continued (helpful for multi-step tasks or stateful browsing).\n\n
6.âScreenshot, PDF & Media Options
\n
run_config = CrawlerRunConfig(\n screenshot=True, # Grab a screenshot as base64\n screenshot_wait_for=1.0, # Wait 1s before capturing\n pdf=True, # Also produce a PDF\n image_description_min_word_threshold=5, # If analyzing alt text\n image_score_threshold=3, # Filter out low-score images\n)\n
\nWhere they appear:\n- result.screenshot â Base64 screenshot string.\n- result.pdf â Byte array with PDF data.\n\n
7.âExtraction Strategy
\n
For advanced data extraction (CSS/LLM-based), set extraction_strategy:
1.âCrawling the main content region, ignoring external links.â \n2.âRunning JavaScript to click â.show-moreâ.â \n3.âWaiting for â.loaded-blockâ to appear.â \n4.âGenerating a screenshot & PDF of the final page.â \n5.âExtracting repeated âarticle.postâ elements with a CSS-based extraction strategy.
\n\n
9.âBest Practices
\n
1.âUse BrowserConfig for global browser settings (headless, user agent).â \n2.âUse CrawlerRunConfig to handle the specific crawl needs: content filtering, caching, JS, screenshot, extraction, etc.â \n3.âKeep your parameters consistent in run configsâespecially if youâre part of a large codebase with multiple crawls.â \n4.âLimit large concurrency (semaphore_count) if the site or your system canât handle it.â \n5.âFor dynamic pages, set js_code or scan_full_page so you load all content.
\n\n
10.âConclusion
\n
All parameters that used to be direct arguments to arun() now belong in CrawlerRunConfig.âThis approach:
\n
\n
Makes code clearer and more maintainable.â
\n
Minimizes confusion about which arguments affect global vs.âper-crawl behavior.â
\n
Allows you to create reusable config objects for different pages or tasks.
Note: This function is very similar to arun() but focused on concurrent or batch crawling.âIf youâre unfamiliar with arun() usage, please read that doc first, then review this for differences.
\n
\n
Function Signature
\n
asyncdefarun_many(\n urls: Union[List[str], List[Any]],\n config: Optional[Union[CrawlerRunConfig, List[CrawlerRunConfig]]] = None,\n dispatcher: Optional[BaseDispatcher] = None,\n ...\n) -> RunManyReturn:\n \"\"\"\n Crawl multiple URLs concurrently or in batches.\n\n :param urls: A list of URLs (or tasks) to crawl.\n :param config: (Optional) Either:\n - A single `CrawlerRunConfig` applying to all URLs\n - A list of `CrawlerRunConfig` objects with url_matcher patterns\n :param dispatcher: (Optional) A concurrency controller (e.g.âMemoryAdaptiveDispatcher).\n ...\n :return: RunManyReturn containing either a list of `CrawlResult` objects or an async generator if streaming is enabled.\n \"\"\"\n
\n
Differences from arun()
\n
1.âMultiple URLs:
\n
\n
Instead of crawling a single URL, you pass a list of them (strings or tasks).â
\n
The function returns RunManyReturn which contains either a list of CrawlResult or an async generator if streaming is enabled.
If omitted, a default dispatcher (like MemoryAdaptiveDispatcher) is used internally.â
\n
Dispatchers handle concurrency, rate limiting, and memory-based adaptive throttling (see Multi-URL Crawling).
\n
\n
3.âStreaming Support:
\n
\n
Enable streaming by setting stream=True in your CrawlerRunConfig.
\n
When streaming, use async for to process results as they become available.
\n
Ideal for processing large numbers of URLs without waiting for all to complete.
\n
\n
4.âParallel Execution**:
\n
\n
arun_many() can run multiple requests concurrently under the hood.â
\n
Each CrawlResult might also include a dispatch_result with concurrency details (like memory usage, start/end times).
\n
\n
Basic Example (Batch Mode)
\n
# Minimal usage: The default dispatcher will be used\nresults = await crawler.arun_many(\n urls=[\"https://site1.com\", \"https://site2.com\"],\n config=CrawlerRunConfig(stream=False) # Default behavior\n)\n\nfor res in results:\n if res.success:\n print(res.url, \"crawled OK!\")\n else:\n print(\"Failed:\", res.url, \"-\", res.error_message)\n
\n
Streaming Example
\n
config = CrawlerRunConfig(\n stream=True, # Enable streaming mode\n cache_mode=CacheMode.BYPASS\n)\n\n# Process results as they complete\nasyncfor result inawait crawler.arun_many(\n urls=[\"https://site1.com\", \"https://site2.com\", \"https://site3.com\"],\n config=config\n):\n if result.success:\n print(f\"Just completed: {result.url}\")\n # Process each result immediately\n process_result(result)\n
Instead of using one config for all URLs, provide a list of configs with url_matcher patterns:
\n
from crawl4ai import CrawlerRunConfig, MatchMode\nfrom crawl4ai.processors.pdf import PDFContentScrapingStrategy\nfrom crawl4ai.extraction_strategy import JsonCssExtractionStrategy\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\n# PDF files - specialized extraction\npdf_config = CrawlerRunConfig(\n url_matcher=\"*.pdf\",\n scraping_strategy=PDFContentScrapingStrategy()\n)\n\n# Blog/article pages - content filtering\nblog_config = CrawlerRunConfig(\n url_matcher=[\"*/blog/*\", \"*/article/*\", \"*python.org*\"],\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.48)\n )\n)\n\n# Dynamic pages - JavaScript execution\ngithub_config = CrawlerRunConfig(\n url_matcher=lambda url: 'github.com'in url,\n js_code=\"window.scrollTo(0, 500);\"\n)\n\n# API endpoints - JSON extraction\napi_config = CrawlerRunConfig(\n url_matcher=lambda url: 'api'in url or url.endswith('.json'),\n # Custome settings for JSON extraction\n)\n\n# Default fallback config\ndefault_config = CrawlerRunConfig() # No url_matcher means it never matches except as fallback\n\n# Pass the list of configs - first match wins!\nresults = await crawler.arun_many(\n urls=[\n \"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf\", # â pdf_config\n \"https://blog.python.org/\", # â blog_config\n \"https://github.com/microsoft/playwright\", # â github_config\n \"https://httpbin.org/json\", # â api_config\n \"https://example.com/\"# â default_config\n ],\n config=[pdf_config, blog_config, github_config, api_config, default_config]\n)\n
\n
URL Matching Features:\n- String patterns: \"*.pdf\", \"*/blog/*\", \"*python.org*\"\n- Function matchers: lambda url: 'api' in url\n- Mixed patterns: Combine strings and functions with MatchMode.OR or MatchMode.AND\n- First match wins: Configs are evaluated in order
\n
Key Points:\n- Each URL is processed by the same or separate sessions, depending on the dispatcherâs strategy.\n- dispatch_result in each CrawlResult (if using concurrency) can hold memory and timing info.â \n- If you need to handle authentication or session IDs, pass them in each individual task or within your run config.\n- Important: Always include a default config (without url_matcher) as the last item if you want to handle all URLs. Otherwise, unmatched URLs will fail.
\n
Return Value
\n
Returns a RunManyReturn object which contains either a list of CrawlResult objects, or an async generator if streaming is enabled.âYou can iterate to check result.success or read each itemâs extracted_content, markdown, or dispatch_result.
\n\n
Dispatcher Reference
\n
\n
MemoryAdaptiveDispatcher: Dynamically manages concurrency based on system memory usage.â
\n
SemaphoreDispatcher: Fixed concurrency limit, simpler but less adaptive.â
1.âLarge Lists: If you pass thousands of URLs, be mindful of memory or rate-limits.âA dispatcher can help.â
\n
2.âSession Reuse: If you need specialized logins or persistent contexts, ensure your dispatcher or tasks handle sessions accordingly.â
\n
3.âError Handling: Each CrawlResult might fail for different reasonsâalways check result.success or the error_message before proceeding.
\n\n
Conclusion
\n
Use arun_many() when you want to crawl multiple URLs simultaneously or in controlled parallel tasks.âIf you need advanced concurrency features (like memory-based adaptive throttling or complex rate-limiting), provide a dispatcher.âEach result is a standard CrawlResult, possibly augmented with concurrency stats (dispatch_result) for deeper inspection.âFor more details on concurrency logic and dispatchers, see the Advanced Multi-URL Crawling docs.
The AsyncWebCrawler is the core class for asynchronous web crawling in Crawl4AI.âYou typically create it once, optionally customize it with a BrowserConfig (e.g., headless, user agent), then run multiple arun() calls with different CrawlerRunConfig objects.
\n
Recommended usage:
\n
1.âCreate a BrowserConfig for global browser settings.â
3.âUse the crawler in an async context manager (async with) or manage start/close manually.â
\n
4.âCallarun(url, config=crawler_run_config) for each page you want.
\n\n
1.âConstructor Overview
\n
classAsyncWebCrawler:\n def__init__(\n self,\n crawler_strategy: Optional[AsyncCrawlerStrategy] = None,\n config: Optional[BrowserConfig] = None,\n always_bypass_cache: bool = False, # deprecated\n always_by_pass_cache: Optional[bool] = None, # also deprecated\n base_directory: str = ...,\n thread_safe: bool = False,\n **kwargs,\n ):\n \"\"\"\n Create an AsyncWebCrawler instance.\n\n Args:\n crawler_strategy: \n (Advanced) Provide a custom crawler strategy if needed.\n config: \n A BrowserConfig object specifying how the browser is set up.\n always_bypass_cache: \n (Deprecated) Use CrawlerRunConfig.cache_mode instead.\n base_directory: \n Folder for storing caches/logs (if relevant).\n thread_safe: \n If True, attempts some concurrency safeguards.âUsually False.\n **kwargs: \n Additional legacy or debugging parameters.\n \"\"\"\n )\n\n### Typical Initialization\n\n```python\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig\n\nbrowser_cfg = BrowserConfig(\n browser_type=\"chromium\",\n headless=True,\n verbose=True\n)\n\ncrawler = AsyncWebCrawler(config=browser_cfg)\n
\n
Notes:
\n
\n
Legacy parameters like always_bypass_cache remain for backward compatibility, but prefer to set caching in CrawlerRunConfig.
\n
\n\n
2.âLifecycle: Start/Close or Context Manager
\n
2.1 Context Manager (Recommended)
\n
asyncwithAsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\"https://example.com\")\n # The crawler automatically starts/closes resources\n
\n
When the async with block ends, the crawler cleans up (closes the browser, etc.).
You pass a CrawlerRunConfig object that sets up everything about a crawlâcontent filtering, caching, session reuse, JS code, screenshots, etc.
\n
import asyncio\nfrom crawl4ai import CrawlerRunConfig, CacheMode\n\nrun_cfg = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n css_selector=\"main.article\",\n word_count_threshold=10,\n screenshot=True\n)\n\nasyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\"https://example.com/news\", config=run_cfg)\n print(\"Crawled HTML length:\", len(result.cleaned_html))\n if result.screenshot:\n print(\"Screenshot base64 length:\", len(result.screenshot))\n
\n
3.2 Legacy Parameters Still Accepted
\n
For backward compatibility, arun() can still accept direct arguments like css_selector=..., word_count_threshold=..., etc., but we strongly advise migrating them into a CrawlerRunConfig.
\n\n
4.âBatch Processing: arun_many()
\n
asyncdefarun_many(\n self,\n urls: List[str],\n config: Optional[CrawlerRunConfig] = None,\n # Legacy parameters maintained for backwards compatibility...\n) -> RunManyReturn:\n \"\"\"\n Process multiple URLs with intelligent rate limiting and resource monitoring.\n \"\"\"\n
\n
4.1 Resource-Aware Crawling
\n
The arun_many() method now uses an intelligent dispatcher that:
\n
\n
Monitors system memory usage
\n
Implements adaptive rate limiting
\n
Provides detailed progress monitoring
\n
Manages concurrent crawls efficiently
\n
\n
4.2 Example Usage
\n
Check page Multi-url Crawling for a detailed example of how to use arun_many().
\n
### 4.3 Key Features\n\n1.â**Rate Limiting**\n\n - Automatic delay between requests\n - Exponential backoff on rate limit detection\n - Domain-specific rate limiting\n - Configurable retry strategy\n\n2.â**Resource Monitoring**\n\n - Memory usage tracking\n - Adaptive concurrency based on system load\n - Automatic pausing when resources are constrained\n\n3.â**Progress Monitoring**\n\n - Detailed or aggregated progress display\n - Real-time status updates\n - Memory usage statistics\n\n4.â**Error Handling**\n\n - Graceful handling of rate limits\n - Automatic retries with backoff\n - Detailed error reporting\n\n---\n\n## 5.â`CrawlResult` Output\n\nEach `arun()` returns a **`CrawlResult`** containing:\n\n- `url`: Final URL (if redirected).\n- `html`: Original HTML.\n- `cleaned_html`: Sanitized HTML.\n- `markdown_v2`: Removed in v0.5. Accessing it raises `AttributeError`; use `markdown`.\n- `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies).\n- `screenshot`, `pdf`: If screenshots/PDF requested.\n- `media`, `links`: Information about discovered images/links.\n- `success`, `error_message`: Status info.\n\nFor details, see [CrawlResult doc](./crawl-result.md).\n\n---\n\n## 6.âQuick Example\n\nBelow is an example hooking it all together:\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom crawl4ai import JsonCssExtractionStrategy\nimport json\n\nasyncdefmain():\n # 1.âBrowser config\n browser_cfg = BrowserConfig(\n browser_type=\"firefox\",\n headless=False,\n verbose=True\n )\n\n # 2.âRun config\n schema = {\n \"name\": \"Articles\",\n \"baseSelector\": \"article.post\",\n \"fields\": [\n {\n \"name\": \"title\", \n \"selector\": \"h2\", \n \"type\": \"text\"\n },\n {\n \"name\": \"url\", \n \"selector\": \"a\", \n \"type\": \"attribute\", \n \"attribute\": \"href\"\n }\n ]\n }\n\n run_cfg = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n extraction_strategy=JsonCssExtractionStrategy(schema),\n word_count_threshold=15,\n remove_overlay_elements=True,\n wait_for=\"css:.post\"# Wait for posts to appear\n )\n\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\n url=\"https://example.com/blog\",\n config=run_cfg\n )\n\n if result.success:\n print(\"Cleaned HTML length:\", len(result.cleaned_html))\n if result.extracted_content:\n articles = json.loads(result.extracted_content)\n print(\"Extracted articles:\", articles[:2])\n else:\n print(\"Error:\", result.error_message)\n\nasyncio.run(main())\n
\n
Explanation:
\n
\n
We define a BrowserConfig with Firefox, no headless, and verbose=True.â
\n
We define a CrawlerRunConfig that bypasses cache, uses a CSS extraction schema, has a word_count_threshold=15, etc.â
\n
We pass them to AsyncWebCrawler(config=...) and arun(url=..., config=...).
\n
\n\n
7.âBest Practices & Migration Notes
\n
1.âUseBrowserConfig for global settings about the browserâs environment.â \n2.âUseCrawlerRunConfig for per-crawl logic (caching, content filtering, extraction strategies, wait conditions).â \n3.âAvoid legacy parameters like css_selector or word_count_threshold directly in arun().âInstead:
arun(url, config=CrawlerRunConfig) is the main method for single-page crawls.â
\n
arun_many(urls, config=CrawlerRunConfig) handles concurrency across multiple URLs.â
\n
For advanced lifecycle control, use start() and close() explicitly.â
\n
\n
Migration:
\n
\n
If you used AsyncWebCrawler(browser_type=\"chromium\", css_selector=\"...\"), move browser settings to BrowserConfig(...) and content/crawl logic to CrawlerRunConfig(...).
\n
\n
This modular approach ensures your code is clean, scalable, and easy to maintain.âFor any advanced or rarely used parameters, see the BrowserConfig docs.
Notes:\n- Works with input, textarea elements\n- Faster than selecting all and deleting\n- Triggers appropriate change events
\n\n
SET <selector> \"<value>\"
\n
Set the value of an input field directly.
\n
Syntax:\n
SET `<selector>`\"<value>\"\n
\n
Parameters:\n- selector - CSS selector for input element (string in backticks)\n- value - Value to set (string in quotes)
\n
Examples:\n
SET `#email`\"user@example.com\"\nSET `#age`\"25\"\nSET `textarea#message`\"Hello, this is a test message.\"\n
\n
Notes:\n- Directly sets value without typing animation\n- Faster than TYPE for long text\n- Triggers change and input events
\n
đ Control Flow Commands
\n
Add conditional logic and loops to your scripts.
\n
IF (EXISTS <selector>) THEN <command>
\n
Execute command if element exists.
\n
Syntax:\n
IF (EXISTS `<selector>`) THEN <command>\n
\n
Parameters:\n- selector - CSS selector to check (string in backticks)\n- command - Command to execute if condition is true
\n
Examples:\n
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept-cookies`\nIF (EXISTS `#popup-modal`) THEN CLICK `.close-button`\nIF (EXISTS `.error-message`) THEN RELOAD\n
\n
Notes:\n- Checks for element existence at time of execution\n- Does not wait for element to appear\n- Can be combined with ELSE
\n\n
IF (EXISTS <selector>) THEN <command> ELSE <command>
\n
Execute command based on element existence.
\n
Syntax:\n
IF (EXISTS `<selector>`) THEN <command> ELSE <command>\n
\n
Parameters:\n- selector - CSS selector to check (string in backticks)\n- First command - Execute if condition is true\n- Second command - Execute if condition is false
\n
Examples:\n
IF (EXISTS `.user-menu`) THEN CLICK `.logout` ELSE CLICK `.login`\nIF (EXISTS `.loading`) THEN WAIT 5 ELSE CLICK `#continue`\n
\n
Notes:\n- Exactly one command will be executed\n- Useful for handling different page states\n- Commands must be on same line
\n\n
IF (NOT EXISTS <selector>) THEN <command>
\n
Execute command if element does not exist.
\n
Syntax:\n
IF (NOT EXISTS `<selector>`) THEN <command>\n
\n
Parameters:\n- selector - CSS selector to check (string in backticks)\n- command - Command to execute if element doesn't exist
\n
Examples:\n
IF (NOT EXISTS `.logged-in`) THEN GO /login\nIF (NOT EXISTS `.results`) THEN CLICK `#search-button`\n
\n
Notes:\n- Inverse of EXISTS condition\n- Useful for error handling\n- Can check for missing required elements
\n\n
IF (<javascript>) THEN <command>
\n
Execute command based on JavaScript condition.
\n
Syntax:\n
IF (`<javascript>`) THEN <command>\n
\n
Parameters:\n- javascript - JavaScript expression that returns boolean (string in backticks)\n- command - Command to execute if condition is true
\n
Examples:\n
IF (`window.innerWidth < 768`) THEN CLICK `.mobile-menu`\nIF (`document.readyState === \"complete\"`) THEN CLICK `#start`\nIF (`localStorage.getItem(\"user\")`) THEN GO /dashboard\n
\n
Notes:\n- JavaScript executes in browser context\n- Must return boolean value\n- Access to all browser APIs and globals
\n\n
REPEAT (<command>, <count>)
\n
Repeat a command a specific number of times.
\n
Syntax:\n
REPEAT (<command>, <count>)\n
\n
Parameters:\n- command - Command to repeat\n- count - Number of times to repeat (number)
Notes:\n- Full access to browser JavaScript APIs\n- Useful for custom logic and debugging\n- Return values are not captured\n- Be careful with security implications
\n
đ Comments and Documentation
\n
# <comment>
\n
Add comments to scripts for documentation.
\n
Syntax:\n
# <comment text>\n
\n
Examples:\n
# This script logs into the application\n# Step 1: Navigate to login page\nGO /login\n\n# Step 2: Fill credentials\nTYPE \"user@example.com\"\n
\n
Notes:\n- Comments are ignored during execution\n- Useful for documentation and debugging\n- Can appear anywhere in script\n- Supports multi-line documentation blocks
\n
đ§ Procedures (Advanced)
\n
Define reusable command sequences.
\n
PROC <name> ... ENDPROC
\n
Define a reusable procedure.
\n
Syntax:\n
PROC <name>\n <commands>\nENDPROC\n
\n
Parameters:\n- name - Procedure name (alphanumeric, underscore)\n- commands - Commands to include in procedure
\n
Examples:\n
PROC login\n CLICK `#email`\n TYPE $email\n CLICK `#password`\n TYPE $password\n CLICK `#submit`\nENDPROC\n\nPROC handle_popups\n IF (EXISTS `.cookie-banner`) THEN CLICK `.accept`\n IF (EXISTS `.newsletter-modal`) THEN CLICK `.close`\nENDPROC\n
\n
Notes:\n- Procedures must be defined before use\n- Support nested command structures\n- Variables are shared with main script scope
\n\n
<procedure_name>
\n
Call a defined procedure.
\n
Syntax:\n
<procedure_name>\n
\n
Examples:\n
# Defineprocedurefirst\nPROC setup\n GO /login\n WAIT `#form` 5\nENDPROC\n\n# Callprocedure\nsetup\nlogin\n
\n
Notes:\n- Procedure must be defined before calling\n- Can be called multiple times\n- No parameters supported (use variables instead)
\n
Error Handling Best Practices
\n
1. Always Use Waits
\n
# Bad - element might not be ready\nCLICK `#button`\n\n# Good - wait for element first\nWAIT `#button` 5\nCLICK `#button`\n
\n
2. Handle Optional Elements
\n
# Check before interacting\nIF (EXISTS `.popup`) THEN CLICK `.close`\nIF (EXISTS `.cookie-banner`) THEN CLICK `.accept`\n\n# Then proceed with main flow\nCLICK `#main-action`\n
\n
3. Use Descriptive Variables
\n
# Set up reusable data\nSETVAR admin_email = \"admin@company.com\"\nSETVAR test_password = \"TestPass123!\"\nSETVAR staging_url = \"https://staging.example.com\"\n\n# Use throughout script\nGO $staging_url\nTYPE $admin_email\n
\n
4. Add Debugging Information
\n
# Log progress\nEVAL `console.log(\"Starting login process\")`\nGO /login\n\n# Verify page state\nIF (`document.title.includes(\"Login\")`) THEN EVAL `console.log(\"On login page\")`\n\n# Continue with login\nTYPE $username\n
\n
Common Patterns
\n
Login Flow
\n
# Complete login automation\nSETVAR email = \"user@example.com\"\nSETVAR password = \"mypassword\"\n\nGO /login\nWAIT `#login-form`5\n\n# Handle optional cookie banner\nIF (EXISTS `.cookie-banner`) THEN CLICK `.accept-cookies`\n\n# Fill and submit form\nCLICK `#email`\nTYPE $email\nPRESS Tab\nTYPE $password\nCLICK `button[type=\"submit\"]`\n\n# Wait for redirect\nWAIT `.dashboard`10\n
\n
Infinite Scroll
\n
# Load all content with infinite scroll\nGO /products\n\n# Scroll and load more content\nREPEAT (SCROLL DOWN 500, `document.querySelector(\".load-more\")`)\n\n# Alternative: Fixed number of scrolls\nREPEAT (SCROLL DOWN 800, 10)\nWAIT 2\n
\n
Form Validation
\n
# Handle form with validation\nSET `#email`\"invalid-email\"\nCLICK `#submit`\n\n# Check for validation error\nIF (EXISTS `.error-email`) THEN SET `#email`\"valid@example.com\"\n\n# Retry submission\nCLICK `#submit`\nWAIT `.success-message`5\n
The CrawlResult class encapsulates everything returned after a single crawl operation. It provides the raw or processed content, details on links and media, plus optional metadata (like screenshots, PDFs, or extracted JSON).
What: The page's HTTP status code (e.g., 200, 404). When the page was reached via redirect, this is the status code of the first response in the redirect chain (e.g., 301 or 302).\nUsage:\n
if result.status_code == 404:\n print(\"Page not found!\")\n
\n
1.4 redirected_status_code(Optional[int])
\n
What: The HTTP status code of the final redirect destination. For a 302â200 redirect, status_code is 302 and redirected_status_code is 200. None for non-HTTP requests (raw HTML, local files).\nUsage:\n
if result.status_code in (301, 302) and result.redirected_status_code == 200:\n print(f\"Redirected to {result.redirected_url} (OK)\")\n
\n
1.5 error_message(Optional[str])
\n
What: If success=False, a textual description of the failure. \nUsage:\n
if not result.success:\n print(\"Error:\", result.error_message)\n
\n
1.5 session_id(Optional[str])
\n
What: The ID used for reusing a browser context across multiple calls. \nUsage:\n
# If you used session_id=\"login_session\" in CrawlerRunConfig, see it here:\nprint(\"Session:\", result.session_id)\n
\n
1.6 response_headers(Optional[dict])
\n
What: Final HTTP response headers. \nUsage:\n
if result.response_headers:\n print(\"Server:\", result.response_headers.get(\"Server\", \"Unknown\"))\n
\n
1.7 ssl_certificate(Optional[SSLCertificate])
\n
What: If fetch_ssl_certificate=True in your CrawlerRunConfig, result.ssl_certificate contains a SSLCertificate object describing the site's certificate. You can export the cert in multiple formats (PEM/DER/JSON) or access its properties like issuer, \n subject, valid_from, valid_until, etc. \nUsage:\n
if result.ssl_certificate:\n print(\"Issuer:\", result.ssl_certificate.issuer)\n
\n\n
2. Raw / Cleaned Content
\n
2.1 html(str)
\n
What: The original unmodified HTML from the final page load. \nUsage:\n
# Possibly large\nprint(len(result.html))\n
\n
2.2 cleaned_html(Optional[str])
\n
What: A sanitized HTML versionâscripts, styles, or excluded tags are removed based on your CrawlerRunConfig. \nUsage:\n
print(result.cleaned_html[:500]) # Show a snippet\n
\n\n
3. Markdown Fields
\n
3.1 The Markdown Generation Approach
\n
Crawl4AI can convert HTMLâMarkdown, optionally including:
\n
\n
Raw markdown
\n
Links as citations (with a references section)
\n
Fit markdown if a content filter is used (like Pruning or BM25)
\n
\n
MarkdownGenerationResult includes:\n- raw_markdown(str): The full HTMLâMarkdown conversion. \n- markdown_with_citations(str): Same markdown, but with link references as academic-style citations. \n- references_markdown(str): The reference list or footnotes at the end. \n- fit_markdown(Optional[str]): If content filtering (Pruning/BM25) was applied, the filtered \"fit\" text. \n- fit_html(Optional[str]): The HTML that led to fit_markdown.
\n
Usage:\n
if result.markdown:\n md_res = result.markdown\n print(\"Raw MD:\", md_res.raw_markdown[:300])\n print(\"Citations MD:\", md_res.markdown_with_citations[:300])\n print(\"References:\", md_res.references_markdown)\n if md_res.fit_markdown:\n print(\"Pruned text:\", md_res.fit_markdown[:300])\n
\nImportant: \"Fit\" content (in fit_markdown/fit_html) exists in result.markdown, only if you used a filter (like PruningContentFilter or BM25ContentFilter) within a MarkdownGenerationStrategy.\n\n
4. Media & Links
\n
4.1 media(Dict[str, List[Dict]])
\n
What: Contains info about discovered images, videos, or audio. Typically keys: \"images\", \"videos\", \"audios\". \nCommon Fields in each item:
\n
\n
src(str): Media URL
\n
alt or title(str): Descriptive text
\n
score(float): Relevance score if the crawler's heuristic found it \"important\"
\n
desc or description(Optional[str]): Additional context extracted from surrounding text
\n
\n
Usage:\n
images = result.media.get(\"images\", [])\nfor img in images:\n if img.get(\"score\", 0) > 5:\n print(\"High-value image:\", img[\"src\"])\n
\n
4.2 links(Dict[str, List[Dict]])
\n
What: Holds internal and external link data. Usually two keys: \"internal\" and \"external\". \nCommon Fields:
\n
\n
href(str): The link target
\n
text(str): Link text
\n
title(str): Title attribute
\n
context(str): Surrounding text snippet
\n
domain(str): If external, the domain
\n
\n
Usage:\n
for link in result.links[\"internal\"]:\n print(f\"Internal link to {link['href']} with text {link['text']}\")\n
\n\n
5. Additional Fields
\n
5.1 extracted_content(Optional[str])
\n
What: If you used extraction_strategy (CSS, LLM, etc.), the structured output (JSON). \nUsage:\n
if result.extracted_content:\n data = json.loads(result.extracted_content)\n print(data)\n
\n
5.2 downloaded_files(Optional[List[str]])
\n
What: If accept_downloads=True in your BrowserConfig + downloads_path, lists local file paths for downloaded items. \nUsage:\n
if result.downloaded_files:\n for file_path in result.downloaded_files:\n print(\"Downloaded:\", file_path)\n
\n
5.3 screenshot(Optional[str])
\n
What: Base64-encoded screenshot if screenshot=True in CrawlerRunConfig. \nUsage:\n
import base64\nif result.screenshot:\n withopen(\"page.png\", \"wb\") as f:\n f.write(base64.b64decode(result.screenshot))\n
\n
5.4 pdf(Optional[bytes])
\n
What: Raw PDF bytes if pdf=True in CrawlerRunConfig. \nUsage:\n
if result.pdf:\n withopen(\"page.pdf\", \"wb\") as f:\n f.write(result.pdf)\n
\n
5.5 mhtml(Optional[str])
\n
What: MHTML snapshot of the page if capture_mhtml=True in CrawlerRunConfig. MHTML (MIME HTML) format preserves the entire web page with all its resources (CSS, images, scripts, etc.) in a single file. \nUsage:\n
if result.mhtml:\n withopen(\"page.mhtml\", \"w\", encoding=\"utf-8\") as f:\n f.write(result.mhtml)\n
\n
5.6 metadata(Optional[dict])
\n
What: Page-level metadata if discovered (title, description, OG data, etc.). \nUsage:\n
if result.metadata:\n print(\"Title:\", result.metadata.get(\"title\"))\n print(\"Author:\", result.metadata.get(\"author\"))\n
\n\n
6. dispatch_result (optional)
\n
A DispatchResult object providing additional concurrency and resource usage information when crawling URLs in parallel (e.g., via arun_many() with custom dispatchers). It contains:
\n
\n
task_id: A unique identifier for the parallel task.
\n
memory_usage (float): The memory (in MB) used at the time of completion.
\n
peak_memory (float): The peak memory usage (in MB) recorded during the task's execution.
\n
start_time / end_time (datetime): Time range for this crawling task.
\n
error_message (str): Any dispatcher- or concurrency-related error encountered.
\n
\n
# Example usage:\nfor result in results:\n if result.success and result.dispatch_result:\n dr = result.dispatch_result\n print(f\"URL: {result.url}, Task ID: {dr.task_id}\")\n print(f\"Memory: {dr.memory_usage:.1f} MB (Peak: {dr.peak_memory:.1f} MB)\")\n print(f\"Duration: {dr.end_time - dr.start_time}\")\n
\n
\n
Note: This field is typically populated when using arun_many(...) alongside a dispatcher (e.g., MemoryAdaptiveDispatcher or SemaphoreDispatcher). If no concurrency or dispatcher is used, dispatch_result may remain None.
\n
\n\n
7. Network Requests & Console Messages
\n
When you enable network and console message capturing in CrawlerRunConfig using capture_network_requests=True and capture_console_messages=True, the CrawlResult will include these fields:
What: A list of dictionaries containing information about all network requests, responses, and failures captured during the crawl.\nStructure:\n- Each item has an event_type field that can be \"request\", \"response\", or \"request_failed\".\n- Request events include url, method, headers, post_data, resource_type, and is_navigation_request.\n- Response events include url, status, status_text, headers, and request_timing.\n- Failed request events include url, method, resource_type, and failure_text.\n- All events include a timestamp field.
\n
Usage:\n
if result.network_requests:\n # Count different types of events\n requests = [r for r in result.network_requests if r.get(\"event_type\") == \"request\"]\n responses = [r for r in result.network_requests if r.get(\"event_type\") == \"response\"]\n failures = [r for r in result.network_requests if r.get(\"event_type\") == \"request_failed\"]\n\n print(f\"Captured {len(requests)} requests, {len(responses)} responses, and {len(failures)} failures\")\n\n # Analyze API calls\n api_calls = [r for r in requests if\"api\"in r.get(\"url\", \"\")]\n\n # Identify failed resources\n for failure in failures:\n print(f\"Failed to load: {failure.get('url')} - {failure.get('failure_text')}\")\n
What: A list of dictionaries containing all browser console messages captured during the crawl.\nStructure:\n- Each item has a type field indicating the message type (e.g., \"log\", \"error\", \"warning\", etc.).\n- The text field contains the actual message text.\n- Some messages include location information (URL, line, column).\n- All messages include a timestamp field.
\n
Usage:\n
if result.console_messages:\n # Count messages by type\n message_types = {}\n for msg in result.console_messages:\n msg_type = msg.get(\"type\", \"unknown\")\n message_types[msg_type] = message_types.get(msg_type, 0) + 1\n\n print(f\"Message type counts: {message_types}\")\n\n # Display errors (which are usually most important)\n for msg in result.console_messages:\n if msg.get(\"type\") == \"error\":\n print(f\"Error: {msg.get('text')}\")\n
\n
These fields provide deep visibility into the page's network activity and browser console, which is invaluable for debugging, security analysis, and understanding complex web applications.
asyncdefhandle_result(result: CrawlResult):\n ifnot result.success:\n print(\"Crawl error:\", result.error_message)\n return\n\n # Basic info\n print(\"Crawled URL:\", result.url)\n print(\"Status code:\", result.status_code)\n\n # HTML\n print(\"Original HTML size:\", len(result.html))\n print(\"Cleaned HTML size:\", len(result.cleaned_html or\"\"))\n\n # Markdown output\n if result.markdown:\n print(\"Raw Markdown:\", result.markdown.raw_markdown[:300])\n print(\"Citations Markdown:\", result.markdown.markdown_with_citations[:300])\n if result.markdown.fit_markdown:\n print(\"Fit Markdown:\", result.markdown.fit_markdown[:200])\n\n # Media & Links\n if\"images\"in result.media:\n print(\"Image count:\", len(result.media[\"images\"]))\n if\"internal\"in result.links:\n print(\"Internal link count:\", len(result.links[\"internal\"]))\n\n # Extraction strategy result\n if result.extracted_content:\n print(\"Structured data:\", result.extracted_content)\n\n # Screenshot/PDF/MHTML\n if result.screenshot:\n print(\"Screenshot length:\", len(result.screenshot))\n if result.pdf:\n print(\"PDF bytes length:\", len(result.pdf))\n if result.mhtml:\n print(\"MHTML length:\", len(result.mhtml))\n\n # Network and console capturing\n if result.network_requests:\n print(f\"Network requests captured: {len(result.network_requests)}\")\n # Analyze request types\n req_types = {}\n for req in result.network_requests:\n if\"resource_type\"in req:\n req_types[req[\"resource_type\"]] = req_types.get(req[\"resource_type\"], 0) + 1\n print(f\"Resource types: {req_types}\")\n\n if result.console_messages:\n print(f\"Console messages captured: {len(result.console_messages)}\")\n # Count by message type\n msg_types = {}\n for msg in result.console_messages:\n msg_types[msg.get(\"type\", \"unknown\")] = msg_types.get(msg.get(\"type\", \"unknown\"), 0) + 1\n print(f\"Message types: {msg_types}\")\n
\n\n
9. Key Points & Future
\n
1.âDeprecated legacy properties of CrawlResult \n - markdown_v2 - Removed in v0.5 and now raises AttributeError. Use result.markdown instead.\n - fit_markdown and fit_html - No longer top-level properties in v0.5. Use result.markdown.fit_markdown and result.markdown.fit_html.
\n
2.âFit Content \n - fit_markdown and fit_html appear in MarkdownGenerationResult, only if you used a content filter (like PruningContentFilter or BM25ContentFilter) inside your MarkdownGenerationStrategy or set them directly. \n - If no filter is used, they remain None.
\n
3.âReferences & Citations \n - If you enable link citations in your DefaultMarkdownGenerator (options={\"citations\": True}), youâll see markdown_with_citations plus a references_markdown block. This helps large language models or academic-like referencing.
\n
4.âLinks & Media \n - links[\"internal\"] and links[\"external\"] group discovered anchors by domain. \n - media[\"images\"] / [\"videos\"] / [\"audios\"] store extracted media elements with optional scoring or context.
\n
5.âError Cases \n - If success=False, check error_message (e.g., timeouts, invalid URLs). \n - status_code might be None if we failed before an HTTP response.
\n
Use CrawlResult to glean all final outputs and feed them into your data pipelines, AI models, or archives. With the synergy of a properly configured BrowserConfig and CrawlerRunConfig, the crawler can produce robust, structured results here in CrawlResult.
The digest() method is the primary interface for adaptive web crawling. It intelligently crawls websites starting from a given URL, guided by a query, and automatically determines when sufficient information has been gathered.
Description: The starting URL for the crawl. This should be a valid HTTP/HTTPS URL that serves as the entry point for information gathering.
\n
\n
query
\n
\n
Type: str
\n
Required: Yes
\n
Description: The search query that guides the crawling process. This should contain key terms related to the information you're seeking. The crawler uses this to evaluate relevance and determine which links to follow.
\n
\n
resume_from
\n
\n
Type: Optional[Union[str, Path]]
\n
Default: None
\n
Description: Path to a previously saved crawl state file. When provided, the crawler resumes from the saved state instead of starting fresh.
\n
\n
Return Value
\n
Returns a CrawlState object containing:
\n
\n
crawled_urls (Set[str]): All URLs that have been crawled
\n
knowledge_base (List[CrawlResult]): Collection of crawled pages with content
\n
pending_links (List[Link]): Links discovered but not yet crawled
\n
metrics (Dict[str, float]): Performance and quality metrics
\n
query (str): The original query
\n
Additional statistical information for scoring
\n
\n
How It Works
\n
The digest() method implements an intelligent crawling algorithm:
\n\n
Initial Crawl: Starts from the provided URL
\n
Link Analysis: Evaluates all discovered links for relevance
\n
Scoring: Uses three metrics to assess information sufficiency:
\n
Coverage: How well the query terms are covered
\n
Consistency: Information coherence across pages
\n
Saturation: Diminishing returns detection
\n
Adaptive Selection: Chooses the most promising links to follow
\n
Stopping Decision: Automatically stops when confidence threshold is reached
config = AdaptiveConfig(\n confidence_threshold=0.9, # Require high confidence\n max_pages=30, # Allow more pages\n top_k_links=3 # Follow top 3 links per page\n)\n\nadaptive = AdaptiveCrawler(crawler, config=config)\n\nstate = await adaptive.digest(\n start_url=\"https://api.example.com/docs\",\n query=\"authentication endpoints rate limits\"\n)\n
\n
Resuming a Previous Crawl
\n
# First crawl - may be interrupted\nstate1 = await adaptive.digest(\n start_url=\"https://example.com\",\n query=\"machine learning algorithms\"\n)\n\n# Save state (if not auto-saved)\nstate1.save(\"ml_crawl_state.json\")\n\n# Later, resume from saved state\nstate2 = await adaptive.digest(\n start_url=\"https://example.com\",\n query=\"machine learning algorithms\",\n resume_from=\"ml_crawl_state.json\"\n)\n
Multiple Concepts: Combine related concepts for comprehensive coverage\n
query = \"rest api pagination sorting filtering\"\n
\n
\n\n
Performance Considerations
\n
\n
Initial URL: Choose a page with good navigation (e.g., documentation index)
\n
Query Length: 3-8 terms typically work best
\n
Link Density: Sites with clear navigation crawl more efficiently
\n
Caching: Enable caching for repeated crawls of the same domain
\n
\n
Error Handling
\n
try:\n state = await adaptive.digest(\n start_url=\"https://example.com\",\n query=\"search terms\"\n )\nexcept Exception as e:\n print(f\"Crawl failed: {e}\")\n # State is auto-saved if save_state=True in config\n
\n
Stopping Conditions
\n
The crawl stops when any of these conditions are met:
\n\n
Confidence Threshold: Reached the configured confidence level
\n
Page Limit: Crawled the maximum number of pages
\n
Diminishing Returns: Expected information gain below threshold
\n
No Relevant Links: No promising links remain to follow
Which browser engine to use. \"chromium\" is typical for many sites, \"firefox\" or \"webkit\" for specialized tests.
\n
\n
\n
headless
\n
bool (default: True)
\n
Headless means no visible UI. False is handy for debugging.
\n
\n
\n
browser_mode
\n
str (default: \"dedicated\")
\n
How browser is initialized: \"dedicated\" (new instance), \"builtin\" (CDP background), \"custom\" (explicit CDP), \"docker\" (container).
\n
\n
\n
use_managed_browser
\n
bool (default: False)
\n
Launch browser via CDP for advanced control. Set automatically based on browser_mode.
\n
\n
\n
cdp_url
\n
str (default: None)
\n
Chrome DevTools Protocol endpoint URL (e.g., \"ws://localhost:9222/devtools/browser/\"). Set automatically based on browser_mode.
\n
\n
\n
debugging_port
\n
int (default: 9222)
\n
Port for browser debugging protocol.
\n
\n
\n
host
\n
str (default: \"localhost\")
\n
Host for browser connection.
\n
\n
\n
viewport_width
\n
int (default: 1080)
\n
Initial page width (in px). Useful for testing responsive layouts.
\n
\n
\n
viewport_height
\n
int (default: 600)
\n
Initial page height (in px).
\n
\n
\n
viewport
\n
dict (default: None)
\n
Viewport dimensions dict. If set, overrides viewport_width and viewport_height.
\n
\n
\n
device_scale_factor
\n
float (default: 1.0)
\n
Device pixel ratio for rendering. Use 2.0 for Retina-quality screenshots. Higher values produce larger images and use more memory.
\n
\n
\n
proxy
\n
str (deprecated)
\n
Deprecated. Use proxy_config instead. If set, it will be auto-converted internally.
\n
\n
\n
proxy_config
\n
ProxyConfig or dict (default: None)
\n
For advanced or multi-proxy needs, specify ProxyConfig object or dict like {\"server\": \"...\", \"username\": \"...\", \"password\": \"...\"}.
\n
\n
\n
use_persistent_context
\n
bool (default: False)
\n
If True, uses a persistent browser context (keep cookies, sessions across runs). Also sets use_managed_browser=True.
\n
\n
\n
user_data_dir
\n
str or None (default: None)
\n
Directory to store user data (profiles, cookies). Must be set if you want permanent sessions.
\n
\n
\n
chrome_channel
\n
str (default: \"chromium\")
\n
Chrome channel to launch (e.g., \"chrome\", \"msedge\"). Only for browser_type=\"chromium\". Auto-set to empty for Firefox/WebKit.
\n
\n
\n
channel
\n
str (default: \"chromium\")
\n
Alias for chrome_channel.
\n
\n
\n
accept_downloads
\n
bool (default: False)
\n
Whether to allow file downloads. Requires downloads_path if True.
\n
\n
\n
downloads_path
\n
str or None (default: None)
\n
Directory to store downloaded files.
\n
\n
\n
storage_state
\n
str or dict or None (default: None)
\n
In-memory storage state (cookies, localStorage) to restore browser state.
\n
\n
\n
ignore_https_errors
\n
bool (default: True)
\n
If True, continues despite invalid certificates (common in dev/staging).
\n
\n
\n
java_script_enabled
\n
bool (default: True)
\n
Disable if you want no JS overhead, or if only static content is needed.
\n
\n
\n
sleep_on_close
\n
bool (default: False)
\n
Add a small delay when closing browser (can help with cleanup issues).
\n
\n
\n
cookies
\n
list (default: [])
\n
Pre-set cookies, each a dict like {\"name\": \"session\", \"value\": \"...\", \"url\": \"...\"}.
\n
\n
\n
headers
\n
dict (default: {})
\n
Extra HTTP headers for every request, e.g. {\"Accept-Language\": \"en-US\"}.
\n
\n
\n
user_agent
\n
str (default: Chrome-based UA)
\n
Your custom user agent string.
\n
\n
\n
user_agent_mode
\n
str (default: \"\")
\n
Set to \"random\" to randomize user agent from a pool (helps with bot detection).
\n
\n
\n
user_agent_generator_config
\n
dict (default: {})
\n
Configuration dict for user agent generation when user_agent_mode=\"random\".
\n
\n
\n
text_mode
\n
bool (default: False)
\n
If True, tries to disable images/other heavy content for speed.
\n
\n
\n
light_mode
\n
bool (default: False)
\n
Disables some background features for performance gains.
\n
\n
\n
avoid_ads
\n
bool (default: False)
\n
If True, blocks requests to common ad/tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level.
\n
\n
\n
avoid_css
\n
bool (default: False)
\n
If True, blocks loading of CSS files (.css, .less, .scss, .sass) for faster, leaner crawls when only text content is needed.
\n
\n
\n
extra_args
\n
list (default: [])
\n
Additional flags for the underlying browser process, e.g. [\"--disable-extensions\"].
\n
\n
\n
enable_stealth
\n
bool (default: False)
\n
Enable playwright-stealth mode to bypass bot detection. Cannot be used with browser_mode=\"builtin\".
\n
\n\n
\n
Tips:\n- Set headless=False to visually debug how pages load or how interactions proceed. \n- If you need authentication storage or repeated sessions, consider use_persistent_context=True and specify user_data_dir. \n- For large pages, you might need a bigger viewport_width and viewport_height to handle dynamic content.
\n\n
2.âCrawlerRunConfig â Controlling Each Crawl
\n
While BrowserConfig sets up the environment, CrawlerRunConfig details how each crawl operation should behave: caching, content filtering, link or domain blocking, timeouts, JavaScript code, etc.
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nrun_cfg = CrawlerRunConfig(\n wait_for=\"css:.main-content\",\n word_count_threshold=15,\n excluded_tags=[\"nav\", \"footer\"],\n exclude_external_links=True,\n stream=True, # Enable streaming for arun_many()\n)\n
\n
2.1 Parameter Highlights
\n
We group them by category.
\n
A) Content Processing
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
word_count_threshold
\n
int (default: ~200)
\n
Skips text blocks below X words. Helps ignore trivial sections.
\n
\n
\n
extraction_strategy
\n
ExtractionStrategy (default: None)
\n
If set, extracts structured data (CSS-based, LLM-based, etc.).
\n
\n
\n
chunking_strategy
\n
ChunkingStrategy (default: RegexChunking())
\n
Strategy to chunk content before extraction. Can be customized for different chunking approaches.
\n
\n
\n
markdown_generator
\n
MarkdownGenerationStrategy (None)
\n
If you want specialized markdown output (citations, filtering, chunking, etc.). Can be customized with options such as content_source parameter to select the HTML input source ('cleaned_html', 'raw_html', or 'fit_html').
\n
\n
\n
css_selector
\n
str (None)
\n
Retains only the part of the page matching this selector. Affects the entire extraction process.
\n
\n
\n
target_elements
\n
List[str] (None)
\n
List of CSS selectors for elements to focus on for markdown generation and data extraction, while still processing the entire page for links, media, etc. Provides more flexibility than css_selector.
GPS coordinates configuration. Use GeolocationConfig(latitude=..., longitude=..., accuracy=...).
\n
\n
\n
fetch_ssl_certificate
\n
bool (False)
\n
If True, fetches and includes SSL certificate information in the result.
\n
\n
\n
proxy_config
\n
ProxyConfig, list[ProxyConfig], or None (None)
\n
Proxy configuration for this specific crawl. Pass a single proxy or an ordered list of proxies to try. See Anti-Bot & Fallback.
\n
\n
\n
proxy_rotation_strategy
\n
ProxyRotationStrategy (None)
\n
Strategy for rotating proxies during crawl operations.
\n
\n
\n
max_retries
\n
int (0)
\n
Number of retry rounds when anti-bot blocking is detected. Each round tries all proxies in proxy_config.
\n
\n
\n
fallback_fetch_function
\n
async (str) -> str or None (None)
\n
Async function called as last resort after all retries are exhausted. Takes URL, returns raw HTML. See Anti-Bot & Fallback.
\n
\n\n
\n\n
C) Caching & Session
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
cache_mode
\n
CacheMode or None
\n
Controls how caching is handled (ENABLED, BYPASS, DISABLED, etc.). If None, typically defaults to ENABLED.
\n
\n
\n
session_id
\n
str or None
\n
Assign a unique ID to reuse a single browser session across multiple arun() calls.
\n
\n
\n
bypass_cache
\n
bool (False)
\n
Deprecated. If True, acts like CacheMode.BYPASS. Use cache_mode instead.
\n
\n
\n
disable_cache
\n
bool (False)
\n
Deprecated. If True, acts like CacheMode.DISABLED. Use cache_mode instead.
\n
\n
\n
no_cache_read
\n
bool (False)
\n
Deprecated. If True, acts like CacheMode.WRITE_ONLY (writes cache but never reads). Use cache_mode instead.
\n
\n
\n
no_cache_write
\n
bool (False)
\n
Deprecated. If True, acts like CacheMode.READ_ONLY (reads cache but never writes). Use cache_mode instead.
\n
\n
\n
shared_data
\n
dict or None (None)
\n
Shared data to be passed between hooks and accessible across crawl operations.
\n
\n\n
\n
Use these for controlling whether you read or write from a local content cache. Handy for large batch crawls or repeated site visits.
\n\n
D) Page Navigation & Timing
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
wait_until
\n
str (domcontentloaded)
\n
Condition for navigation to \"complete\". Often \"networkidle\" or \"domcontentloaded\".
\n
\n
\n
page_timeout
\n
int (60000 ms)
\n
Timeout for page navigation or JS steps. Increase for slow sites.
\n
\n
\n
wait_for
\n
str or None
\n
Wait for a CSS (\"css:selector\") or JS (\"js:() => bool\") condition before content extraction.
\n
\n
\n
wait_for_timeout
\n
int or None (None)
\n
Specific timeout in ms for the wait_for condition. If None, uses page_timeout.
\n
\n
\n
wait_for_images
\n
bool (False)
\n
Wait for images to load before finishing. Slows down if you only want text.
\n
\n
\n
delay_before_return_html
\n
float (0.1)
\n
Additional pause (seconds) before final HTML is captured. Good for last-second updates.
\n
\n
\n
check_robots_txt
\n
bool (False)
\n
Whether to check and respect robots.txt rules before crawling. If True, caches robots.txt for efficiency.
\n
\n
\n
mean_delay and max_range
\n
float (0.1, 0.3)
\n
If you call arun_many(), these define random delay intervals between crawls, helping avoid detection or rate limits.
\n
\n
\n
semaphore_count
\n
int (5)
\n
Max concurrency for arun_many(). Increase if you have resources for parallel crawls.
\n
\n\n
\n\n
E) Page Interaction
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
js_code
\n
str or list[str] (None)
\n
JavaScript to run afterwait_for and delay_before_return_html, on the fully-loaded page. E.g. \"document.querySelector('button')?.click();\".
\n
\n
\n
js_code_before_wait
\n
str or list[str] (None)
\n
JavaScript to run beforewait_for. Use for triggering loading that wait_for then checks (e.g. clicking a tab, then waiting for its content).
\n
\n
\n
c4a_script
\n
str or list[str] (None)
\n
C4A script that compiles to JavaScript. Alternative to writing raw JS.
\n
\n
\n
js_only
\n
bool (False)
\n
If True, indicates we're reusing an existing session and only applying JS. No full reload.
\n
\n
\n
ignore_body_visibility
\n
bool (True)
\n
Skip checking if <body> is visible. Usually best to keep True.
\n
\n
\n
scan_full_page
\n
bool (False)
\n
If True, auto-scroll the page to load dynamic content (infinite scroll).
\n
\n
\n
scroll_delay
\n
float (0.2)
\n
Delay between scroll steps when scanning the full page (scan_full_page=True) or capturing full-page screenshots.
\n
\n
\n
max_scroll_steps
\n
int or None (None)
\n
Maximum number of scroll steps during full page scan. If None, scrolls until entire page is loaded.
\n
\n
\n
process_iframes
\n
bool (False)
\n
Inlines iframe content for single-page extraction.
\n
\n
\n
flatten_shadow_dom
\n
bool (False)
\n
Flattens Shadow DOM content into the light DOM before HTML capture. Resolves slots, strips shadow-scoped styles, and force-opens closed shadow roots. Essential for sites built with Web Components (Stencil, Lit, Shoelace, etc.).
\n
\n
\n
remove_overlay_elements
\n
bool (False)
\n
Removes potential modals/popups blocking the main content.
\n
\n
\n
remove_consent_popups
\n
bool (False)
\n
Removes GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, TrustArc, Quantcast, Didomi, Sourcepoint, FundingChoices, etc.). Tries clicking \"Accept All\" first, then falls back to DOM removal.
\n
\n
\n
simulate_user
\n
bool (False)
\n
Simulate user interactions (mouse movements) to avoid bot detection.
\n
\n
\n
override_navigator
\n
bool (False)
\n
Override navigator properties in JS for stealth.
\n
\n
\n
magic
\n
bool (False)
\n
Automatic handling of popups/consent banners. Experimental.
\n
\n
\n
adjust_viewport_to_content
\n
bool (False)
\n
Resizes viewport to match page content height.
\n
\n\n
\n
If your page is a single-page app with repeated JS updates, set js_only=True in subsequent calls, plus a session_id for reusing the same tab.
\n\n
F) Media Handling
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
screenshot
\n
bool (False)
\n
Capture a screenshot (base64) in result.screenshot.
\n
\n
\n
screenshot_wait_for
\n
float or None
\n
Extra wait time before the screenshot.
\n
\n
\n
screenshot_height_threshold
\n
int (~20000)
\n
If the page is taller than this, alternate screenshot strategies are used.
\n
\n
\n
force_viewport_screenshot
\n
bool (False)
\n
If True, always captures a viewport-only screenshot regardless of page height. Faster and smaller than full-page screenshots.
\n
\n
\n
pdf
\n
bool (False)
\n
If True, returns a PDF in result.pdf.
\n
\n
\n
capture_mhtml
\n
bool (False)
\n
If True, captures an MHTML snapshot of the page in result.mhtml. MHTML includes all page resources (CSS, images, etc.) in a single file.
\n
\n
\n
image_description_min_word_threshold
\n
int (~50)
\n
Minimum words for an image's alt text or description to be considered valid.
\n
\n
\n
image_score_threshold
\n
int (~3)
\n
Filter out low-scoring images. The crawler scores images by relevance (size, context, etc.).
\n
\n
\n
exclude_external_images
\n
bool (False)
\n
Exclude images from other domains.
\n
\n
\n
exclude_all_images
\n
bool (False)
\n
If True, excludes all images from processing (both internal and external).
\n
\n
\n
table_score_threshold
\n
int (7)
\n
Minimum score threshold for processing a table. Lower values include more tables.
\n
\n
\n
table_extraction
\n
TableExtractionStrategy (DefaultTableExtraction)
\n
Strategy for table extraction. Defaults to DefaultTableExtraction with configured threshold.
\n
\n\n
\n\n
G) Link/Domain Handling
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
exclude_social_media_domains
\n
list (e.g. Facebook/Twitter)
\n
A default list can be extended. Any link to these domains is removed from final output.
\n
\n
\n
exclude_external_links
\n
bool (False)
\n
Removes all links pointing outside the current domain.
\n
\n
\n
exclude_social_media_links
\n
bool (False)
\n
Strips links specifically to social sites (like Facebook or Twitter).
\n
\n
\n
exclude_domains
\n
list ([])
\n
Provide a custom list of domains to exclude (like [\"ads.com\", \"trackers.io\"]).
\n
\n
\n
exclude_internal_links
\n
bool (False)
\n
If True, excludes internal links from the results.
\n
\n
\n
score_links
\n
bool (False)
\n
If True, calculates intrinsic quality scores for all links using URL structure, text quality, and contextual metrics.
\n
\n
\n
preserve_https_for_internal_links
\n
bool (False)
\n
If True, preserves HTTPS scheme for internal links even when the server redirects to HTTP. Useful for security-conscious crawling.
\n
\n\n
\n
Use these for link-level content filtering (often to keep crawls âinternalâ or to remove spammy domains).
\n\n
H) Debug, Logging & Network Monitoring
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
verbose
\n
bool (True)
\n
Prints logs detailing each step of crawling, interactions, or errors.
\n
\n
\n
log_console
\n
bool (False)
\n
Logs the page's JavaScript console output if you want deeper JS debugging.
\n
\n
\n
capture_network_requests
\n
bool (False)
\n
If True, captures network requests made by the page in result.captured_requests.
\n
\n
\n
capture_console_messages
\n
bool (False)
\n
If True, captures console messages from the page in result.console_messages.
\n
\n\n
\n\n
I) Connection & HTTP Parameters
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
method
\n
str (\"GET\")
\n
HTTP method to use when using AsyncHTTPCrawlerStrategy (e.g., \"GET\", \"POST\").
\n
\n
\n
stream
\n
bool (False)
\n
If True, enables streaming mode for arun_many() to process URLs as they complete rather than waiting for all.
\n
\n
\n
url
\n
str or None (None)
\n
URL for this specific config. Not typically set directly but used internally for URL-specific configurations.
\n
\n
\n
user_agent
\n
str or None (None)
\n
Custom User-Agent string for this crawl. Can override browser-level user agent.
\n
\n
\n
user_agent_mode
\n
str or None (None)
\n
Set to \"random\" to randomize user agent. Can override browser-level setting.
\n
\n
\n
user_agent_generator_config
\n
dict ({})
\n
Configuration for user agent generation when user_agent_mode=\"random\".
\n
\n\n
\n\n
J) Virtual Scroll Configuration
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
virtual_scroll_config
\n
VirtualScrollConfig or dict (None)
\n
Configuration for handling virtualized scrolling on sites like Twitter/Instagram where content is replaced rather than appended.
\n
\n\n
\n
When sites use virtual scrolling (content replaced as you scroll), use VirtualScrollConfig:
\n
from crawl4ai import VirtualScrollConfig\n\nvirtual_config = VirtualScrollConfig(\n container_selector=\"#timeline\", # CSS selector for scrollable container\n scroll_count=30, # Number of times to scroll\n scroll_by=\"container_height\", # How much to scroll: \"container_height\", \"page_height\", or pixels (e.g. 500)\n wait_after_scroll=0.5 # Seconds to wait after each scroll for content to load\n)\n\nconfig = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n)\n
\n
VirtualScrollConfig Parameters:
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
container_selector
\n
str (required)
\n
CSS selector for the scrollable container (e.g., \"#feed\", \".timeline\")
\n
\n
\n
scroll_count
\n
int (10)
\n
Maximum number of scrolls to perform
\n
\n
\n
scroll_by
\n
str or int (\"container_height\")
\n
Scroll amount: \"container_height\", \"page_height\", or pixels (e.g., 500)
\n
\n
\n
wait_after_scroll
\n
float (0.5)
\n
Time in seconds to wait after each scroll for new content to load
\n
\n\n
\n
When to use Virtual Scroll vs scan_full_page:\n- Use virtual_scroll_config when content is replaced during scroll (Twitter, Instagram)\n- Use scan_full_page when content is appended during scroll (traditional infinite scroll)
Pattern(s) to match URLs against. Can be: string (glob), function, or list of mixed types. None means match ALL URLs
\n
\n
\n
match_mode
\n
MatchMode (MatchMode.OR)
\n
How to combine multiple matchers in a list: MatchMode.OR (any match) or MatchMode.AND (all must match)
\n
\n\n
\n
The url_matcher parameter enables URL-specific configurations when used with arun_many():
\n
from crawl4ai import CrawlerRunConfig, MatchMode\nfrom crawl4ai.processors.pdf import PDFContentScrapingStrategy\nfrom crawl4ai.extraction_strategy import JsonCssExtractionStrategy\n\n# Simple string pattern (glob-style)\npdf_config = CrawlerRunConfig(\n url_matcher=\"*.pdf\",\n scraping_strategy=PDFContentScrapingStrategy()\n)\n\n# Multiple patterns with OR logic (default)\nblog_config = CrawlerRunConfig(\n url_matcher=[\"*/blog/*\", \"*/article/*\", \"*/news/*\"],\n match_mode=MatchMode.OR # Any pattern matches\n)\n\n# Function matcher\napi_config = CrawlerRunConfig(\n url_matcher=lambda url: 'api'in url or url.endswith('.json'),\n # Other settings like extraction_strategy\n)\n\n# Mixed: String + Function with AND logic\ncomplex_config = CrawlerRunConfig(\n url_matcher=[\n lambda url: url.startswith('https://'), # Must be HTTPS\n \"*.org/*\", # Must be .org domain\n lambda url: 'docs'in url # Must contain 'docs'\n ],\n match_mode=MatchMode.AND # ALL conditions must match\n)\n\n# Combined patterns and functions with AND logic\nsecure_docs = CrawlerRunConfig(\n url_matcher=[\"https://*\", lambda url: '.doc'in url],\n match_mode=MatchMode.AND # Must be HTTPS AND contain .doc\n)\n\n# Default config - matches ALL URLs\ndefault_config = CrawlerRunConfig() # No url_matcher = matches everything\n
\n
UrlMatcher Types:\n- None (default): When url_matcher is None or not set, the config matches ALL URLs\n- String patterns: Glob-style patterns like \"*.pdf\", \"*/api/*\", \"https://*.example.com/*\"\n- Functions: lambda url: bool - Custom logic for complex matching\n- Lists: Mix strings and functions, combined with MatchMode.OR or MatchMode.AND
\n
Important Behavior:\n- When passing a list of configs to arun_many(), URLs are matched against each config's url_matcher in order. First match wins!\n- If no config matches a URL and there's no default config (one without url_matcher), the URL will fail with \"No matching configuration found\"\n- Always include a default config as the last item if you want to handle all URLs
\n\n
L) Advanced Crawling Features
\n
\n\n
\n
Parameter
\n
Type / Default
\n
What It Does
\n
\n\n\n
\n
deep_crawl_strategy
\n
DeepCrawlStrategy or None (None)
\n
Strategy for deep/recursive crawling. Enables automatic link following and multi-level site crawling.
\n
\n
\n
link_preview_config
\n
LinkPreviewConfig or dict or None (None)
\n
Configuration for link head extraction and scoring. Fetches and scores link metadata without full page loads.
\n
\n
\n
experimental
\n
dict or None (None)
\n
Dictionary for experimental/beta features not yet integrated into main parameters. Use with caution.
\n
\n\n
\n
Deep Crawl Strategy enables automatic site exploration by following links according to defined rules. Useful for sitemap generation or comprehensive site archiving.
\n
Link Preview Config allows efficient link discovery and scoring by fetching only the <head> section of linked pages, enabling smart crawl prioritization without the overhead of full page loads.
\n
Experimental parameters are features in beta testing. They may change or be removed in future versions. Check documentation for currently available experimental features.
\n\n
2.2 Helper Methods
\n
Both BrowserConfig and CrawlerRunConfig provide a clone() method to create modified copies:
\n
# Create a base configuration\nbase_config = CrawlerRunConfig(\n cache_mode=CacheMode.ENABLED,\n word_count_threshold=200\n)\n\n# Create variations using clone()\nstream_config = base_config.clone(stream=True)\nno_cache_config = base_config.clone(\n cache_mode=CacheMode.BYPASS,\n stream=True\n)\n
\n
The clone() method is particularly useful when you need slightly different configurations for different use cases, without modifying the original config.
Both config classes support class-level default overrides. When deploying in a server or cloud context, this eliminates the need to pass the same parameters at every call site.
from crawl4ai import BrowserConfig, CrawlerRunConfig\n\n# Set once at application startup\nBrowserConfig.set_defaults(\n cache_cdp_connection=True,\n cdp_close_delay=0,\n create_isolated_context=True,\n)\nCrawlerRunConfig.set_defaults(verbose=False)\n\n# All new instances inherit the class defaults\ncfg1 = BrowserConfig(cdp_url=\"ws://localhost:9222\")\n# â cache_cdp_connection=True, cdp_close_delay=0\n\ncfg2 = BrowserConfig(cdp_url=\"ws://localhost:9222\", cache_cdp_connection=False)\n# â cache_cdp_connection=False (explicit value wins)\n\n# Inspect current defaults\nBrowserConfig.get_defaults()\n# â {\"cache_cdp_connection\": True, \"cdp_close_delay\": 0, \"create_isolated_context\": True}\n\n# Remove a single default\nBrowserConfig.reset_defaults(\"cdp_close_delay\")\n\n# Remove all defaults\nBrowserConfig.reset_defaults()\n
\n
API Reference:
\n
\n\n
\n
Method
\n
Signature
\n
Description
\n
\n\n\n
\n
set_defaults
\n
set_defaults(**kwargs)
\n
Set class-level defaults for new instances. Raises ValueError if any key is not a valid __init__ parameter.
\n
\n
\n
get_defaults
\n
get_defaults() â dict
\n
Return a deep copy of the current class-level defaults.
\n
\n
\n
reset_defaults
\n
reset_defaults(*names)
\n
With no args, clears all defaults. With args, removes only the named defaults.
\n
\n\n
\n
Notes:\n- Defaults are independent per class â BrowserConfig.set_defaults() has no effect on CrawlerRunConfig.\n- Mutable values (lists, dicts) are deep-copied on storage and on each instance creation, so instances do not share objects.\n- clone(), dump()/load(), and from_kwargs() all work correctly with class defaults â serialized data is self-contained and independent of the current class defaults.\n- Defaults are stored in memory for the lifetime of the process. They are not persisted to disk.
LLMConfig is useful to pass LLM provider config to strategies and functions that rely on LLMs to do extraction, filtering, schema generation etc. Currently it can be used in the following -
\n\n
LLMExtractionStrategy
\n
LLMContentFilter
\n
JsonCssExtractionStrategy.generate_schema
\n
JsonXPathExtractionStrategy.generate_schema
\n
AdaptiveConfig.embedding_llm_config (embedding model for adaptive crawling)
\n
AdaptiveConfig.query_llm_config (chat completion model for query expansion in adaptive crawling)
1.Optional. When not provided explicitly, api_token will be read from environment variables based on provider. For example: If a gemini model is passed as provider then,\"GEMINI_API_KEY\" will be read from environment variables 2. API token of LLM provider eg: api_token = \"gsk_1ClHGGJ7Lpn4WGybR7vNWGdyb3FY7zXEw3SCiy0BAVM9lL8CQv\" 3. Environment variable - use with prefix \"env:\" eg:api_token = \"env: GROQ_API_KEY\"
\n
API token to use for the given provider
\n
\n
\n
base_url
\n
Optional. Custom API endpoint
\n
If your provider has a custom endpoint
\n
\n
\n
backoff_base_delay
\n
Optional. int(default: 2)
\n
Seconds to wait before the first retry when the provider throttles a request.
\n
\n
\n
backoff_max_attempts
\n
Optional. int(default: 3)
\n
Total tries (initial call + retries) before surfacing an error.
\n
\n
\n
backoff_exponential_factor
\n
Optional. int(default: 2)
\n
Multiplier that increases the wait time for each retry (delay = base_delay * factor^attempt).
UseBrowserConfig for global browser settings: engine, headless, proxy, user agent.
\n
UseCrawlerRunConfig for each crawlâs context: how to filter content, handle caching, wait for dynamic elements, or run JS.
\n
Pass both configs to AsyncWebCrawler (the BrowserConfig) and then to arun() (the CrawlerRunConfig).
\n
UseLLMConfig for LLM provider configurations that can be used across all extraction, filtering, schema generation, and adaptive crawling tasks. Can be used in - LLMExtractionStrategy, LLMContentFilter, JsonCssExtractionStrategy.generate_schema, JsonXPathExtractionStrategy.generate_schema, and AdaptiveConfig (embedding_llm_config / query_llm_config)
\n
\n
# Create a modified copywith the clone() method\nstream_cfg = run_cfg.clone(\n stream=True,\n cache_mode=CacheMode.BYPASS\n)\n\n# Orset project-wide defaults once at startup\nBrowserConfig.set_defaults(headless=True, text_mode=True)\nCrawlerRunConfig.set_defaults(cache_mode=CacheMode.BYPASS)\n
This documentation covers the API reference for extraction and chunking strategies in Crawl4AI.
\n
Extraction Strategies
\n
All extraction strategies inherit from the base ExtractionStrategy class and implement two key methods:\n- extract(url: str, html: str) -> List[Dict[str, Any]]\n- run(url: str, sections: List[str]) -> List[Dict[str, Any]]
\n
LLMExtractionStrategy
\n
Used for extracting structured data using Language Models.
\n
LLMExtractionStrategy(\n # Required Parameters\n provider: str = DEFAULT_PROVIDER, # LLM provider (e.g., \"ollama/llama2\")\n api_token: Optional[str] = None, # API token\n\n # Extraction Configuration\n instruction: str = None, # Custom extraction instruction\n schema: Dict = None, # Pydantic model schema for structured data\n extraction_type: str = \"block\", # \"block\" or \"schema\"\n\n # Chunking Parameters\n chunk_token_threshold: int = 4000, # Maximum tokens per chunk\n overlap_rate: float = 0.1, # Overlap between chunks\n word_token_rate: float = 0.75, # Word to token conversion rate\n apply_chunking: bool = True, # Enable/disable chunking\n\n # API Configuration\n base_url: str = None, # Base URL for API\n extra_args: Dict = {}, # Additional provider arguments\n verbose: bool = False# Enable verbose logging\n)\n
\n
RegexExtractionStrategy
\n
Used for fast pattern-based extraction of common entities using regular expressions.
\n
RegexExtractionStrategy(\n # Pattern Configuration\n pattern: IntFlag = RegexExtractionStrategy.Nothing, # Bit flags of built-in patterns to use\n custom: Optional[Dict[str, str]] = None, # Custom pattern dictionary {label: regex}\n\n # Input Format\n input_format: str = \"fit_html\", # \"html\", \"markdown\", \"text\" or \"fit_html\"\n)\n\n# Built-in Patterns as Bit Flags\nRegexExtractionStrategy.Email # Email addresses\nRegexExtractionStrategy.PhoneIntl # International phone numbers \nRegexExtractionStrategy.PhoneUS # US-format phone numbers\nRegexExtractionStrategy.Url # HTTP/HTTPS URLs\nRegexExtractionStrategy.IPv4 # IPv4 addresses\nRegexExtractionStrategy.IPv6 # IPv6 addresses\nRegexExtractionStrategy.Uuid # UUIDs\nRegexExtractionStrategy.Currency # Currency values (USD, EUR, etc)\nRegexExtractionStrategy.Percentage # Percentage values\nRegexExtractionStrategy.Number # Numeric values\nRegexExtractionStrategy.DateIso # ISO format dates\nRegexExtractionStrategy.DateUS # US format dates\nRegexExtractionStrategy.Time24h # 24-hour format times\nRegexExtractionStrategy.PostalUS # US postal codes\nRegexExtractionStrategy.PostalUK # UK postal codes\nRegexExtractionStrategy.HexColor # HTML hex color codes\nRegexExtractionStrategy.TwitterHandle # Twitter handles\nRegexExtractionStrategy.Hashtag # Hashtags\nRegexExtractionStrategy.MacAddr # MAC addresses\nRegexExtractionStrategy.Iban # International bank account numbers\nRegexExtractionStrategy.CreditCard # Credit card numbers\nRegexExtractionStrategy.All # All available patterns\n
\n
CosineStrategy
\n
Used for content similarity-based extraction and clustering.
\n
CosineStrategy(\n # Content Filtering\n semantic_filter: str = None, # Topic/keyword filter\n word_count_threshold: int = 10, # Minimum words per cluster\n sim_threshold: float = 0.3, # Similarity threshold\n\n # Clustering Parameters\n max_dist: float = 0.2, # Maximum cluster distance\n linkage_method: str = 'ward', # Clustering method\n top_k: int = 3, # Top clusters to return\n\n # Model Configuration\n model_name: str = 'sentence-transformers/all-MiniLM-L6-v2', # Embedding model\n\n verbose: bool = False# Enable verbose logging\n)\n
\n
JsonCssExtractionStrategy
\n
Used for CSS selector-based structured data extraction.
\n
JsonCssExtractionStrategy(\n schema: Dict[str, Any], # Extraction schema\n verbose: bool = False# Enable verbose logging\n)\n\n# Schema Structure\nschema = {\n \"name\": str, # Schema name\n \"baseSelector\": str, # Base CSS selector\n \"fields\": [ # List of fields to extract\n {\n \"name\": str, # Field name\n \"selector\": str, # CSS selector\n \"type\": str, # Field type: \"text\", \"attribute\", \"html\", \"regex\"\n \"attribute\": str, # For type=\"attribute\"\n \"pattern\": str, # For type=\"regex\"\n \"transform\": str, # Optional: \"lowercase\", \"uppercase\", \"strip\"\n \"default\": Any, # Default value if extraction fails\n \"source\": str, # Optional: navigate to sibling first, e.g. \"+ tr\"\n }\n ]\n}\n
\n
Chunking Strategies
\n
All chunking strategies inherit from ChunkingStrategy and implement the chunk(text: str) -> list method.
from crawl4ai.chunking_strategy import OverlappingWindowChunking\nfrom crawl4ai import LLMConfig\n\n# Create chunking strategy\nchunker = OverlappingWindowChunking(\n window_size=500, # 500 words per chunk\n overlap=50# 50 words overlap\n)\n\n# Use with extraction strategy\nstrategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"ollama/llama2\"),\n chunking_strategy=chunker\n)\n\nresult = await crawler.arun(\n url=\"https://example.com/long-article\",\n extraction_strategy=strategy\n)\n
\n
Best Practices
\n\n
Choose the Right Strategy
\n
Use RegexExtractionStrategy for common data types like emails, phones, URLs, dates
\n
Use JsonCssExtractionStrategy for well-structured HTML with consistent patterns
\n
Use LLMExtractionStrategy for complex, unstructured content requiring reasoning
\n
\n
Use CosineStrategy for content similarity and clustering
\n
\n
\n
Strategy Selection Guide\n
Is the target data a common type (email/phone/date/URL)? \nâ RegexExtractionStrategy\n\nDoes the page have consistent HTML structure?\nâ JsonCssExtractionStrategy or JsonXPathExtractionStrategy\n\nIs the data semantically complex or unstructured?\nâ LLMExtractionStrategy\n\nNeed to find content similar toa specific topic?\nâ CosineStrategy\n
\n
\n
\n
Optimize Chunking\n
# For long documents\nstrategy = LLMExtractionStrategy(\n chunk_token_threshold=2000, # Smaller chunks\n overlap_rate=0.1 # 10% overlap\n)\n
\n
\n
\n
Combine Strategies for Best Performance\n
# First pass: Extract structure with CSS\ncss_strategy = JsonCssExtractionStrategy(product_schema)\ncss_result = await crawler.arun(url, config=CrawlerRunConfig(extraction_strategy=css_strategy))\nproduct_data = json.loads(css_result.extracted_content)\n\n# Second pass: Extract specific fields with regex\ndescriptions = [product[\"description\"] for product in product_data]\nregex_strategy = RegexExtractionStrategy(\n pattern=RegexExtractionStrategy.Email | RegexExtractionStrategy.PhoneUS,\n custom={\"dimension\": r\"\\d+x\\d+x\\d+ (?:cm|in)\"}\n)\n\n# Process descriptions with regex\nfor text in descriptions:\n matches = regex_strategy.extract(\"\", text) # Direct extraction\n
\n
\n
\n
Handle Errors\n
try:\n result = await crawler.arun(\n url=\"https://example.com\",\n extraction_strategy=strategy\n )\n if result.success:\n content = json.loads(result.extracted_content)\nexcept Exception as e:\n print(f\"Extraction failed: {e}\")\n
Welcome to the Crawl4AI Apps Hub - your gateway to interactive tools and demos that make web scraping more intuitive and powerful.
\n\n\n
\n
đ ď¸ Interactive Tools for Modern Web Scraping
\n
\nOur apps are designed to make Crawl4AI more accessible and powerful. Whether you're learning browser automation, designing extraction strategies, or building complex scrapers, these tools provide visual, interactive ways to work with Crawl4AI's features.\n
\n
\n\n
đŻ Available Apps
\n
\n\n
\n Available\n
đ¨ C4A-Script Interactive Editor
\n
\n A visual, block-based programming environment for creating browser automation scripts. Perfect for beginners and experts alike!\n
OPrompt for AI Coding Assistant: Create an Interactive LLM Context Builder Page
\n
Objective:
\n
Your task is to create an interactive HTML webpage with JavaScript functionality that allows users to select and combine different crawl4ai LLM context files into a single downloadable Markdown (.md) file. This tool will empower users to craft tailored context for their AI assistants based on their specific needs.
\n
Core Functionality:
\n\n
Display crawl4ai Components: The page will list all available crawl4ai documentation components.
\n
Select Context Types: For each component, users can select which types of context they want to include:
\n
Memory (API facts)
\n
Reasoning (How-to/why)
\n
Examples (Code snippets)\n(All should be selected by default for each initially selected component).
\n
\n
\n
Special \"Aggregate\" Contexts: Include options for special, pre-combined contexts:
\n
\"Vibe Coding\" (a curated mix for general AI prompting)
\n
\"All Library Context\" (a comprehensive aggregation of all memory, reasoning, and examples for the entire library).
\n
\n
\n
Fetch and Concatenate: When the user clicks a \"Download Combined Context\" button:
\n
The JavaScript will fetch the content of all selected Markdown files from the server (from a predefined folder, e.g., /llmtxt/).
\n
It will concatenate the content of these files into a single string.
\n
\n
\n
Client-Side Download: The concatenated content will be offered to the user as a download (e.g., custom_crawl4ai_context.md).
\n\n
Input/Assumptions:
\n
\n
Context Files Location: All individual context Markdown files are located on the server in a publicly accessible folder named llmtxt/.
\n
File Naming Convention: Files follow the pattern: crawl4ai_{{component_name}}_[memory|reasoning|examples]_content.llm.md.
\n
{{component_name}} can contain underscores (e.g., deep_crawling, config_objects).
\n
The special contexts will have names like crawl4ai_vibe_content.llm.md and crawl4ai_all_content.llm.md.
\n
\n
\n
Component List: You will be provided with a list of crawl4ai components. For this implementation, use the following list:
Introduction: Briefly explain the purpose of the tool (from the USING_LLM_CONTEXTS.md content you helped draft: \"Supercharging Your AI Assistant...\").
\n
Selection Area:
\n
Special Aggregate Contexts (Radio Buttons or Prominent Checkboxes):
Behavior: Selecting one of these might disable individual component selections (or vice-versa) to avoid redundancy, or simply add them to the list. Consider user experience here. A simple approach is that if an aggregate is selected, it's the only thing downloaded.
\n
\n
\n
Individual Component Selection (Table or List of Checkboxes):
\n
A section titled \"Select Individual Components & Context Types:\"
\n
For each component in the provided list:
\n
A master checkbox for the component itself (e.g., [ ] Core Functionality). Selected by default.
\n
Nested checkboxes (indented or grouped) for context types, enabled only if the parent component is checked:
\n
[x] Memory (API Facts)
\n
[x] Reasoning (How-to/Why)
\n
[x] Examples (Code Snippets)\n(These three sub-checkboxes should be selected by default if the parent component is selected).
\n
\n
\n
\n
\n
\n
\n
\n
\n
Action Button:
\n
A button: \"Generate & Download Combined Context\"
\n
\n
\n
Status/Feedback Area: (Optional, but good UX)
\n
Display messages like \"Fetching files...\", \"Combining context...\", \"Download starting...\" or error messages.
\n
\n
\n
\n
\n\n
Final Output:
\n
\n
A single HTML file (e.g., interactive_context_builder.html).
\n
Associated JavaScript code (can be inline within <script> tags or in a separate .js file).
\n
Associated CSS code (can be inline within <style> tags or in a separate .css file).
\n
\n
This interactive tool will greatly enhance the user experience for crawl4ai developers looking to leverage your specialized LLM contexts. Please ensure the JavaScript is robust and provides good user feedback.
\n\n
This prompt should give your AI coding assistant a very clear set of requirements and guidelines for building the interactive context builder. Remember to provide it with the list of components as mentioned in the \"Input/Assumptions\" section.
\n",
"success": true,
"error": "",
"crawled_at": 1786439010.7467635,
"is_shell": false
},
{
"url": "https://docs.crawl4ai.com/apps/llmtxt/why/",
"title": "Supercharging Your AI Assistant: My Journey to Better LLM Contexts for crawl4ai - Crawl4AI Documentation (v0.9.x)",
"content": "\n
Supercharging Your AI Assistant: My Journey to Better LLM Contexts for crawl4ai
\n
When I started diving deep into using AI coding assistants with my own libraries, particularly crawl4ai, I quickly realized that the common approach to providing context via a simple llm.txt or even a beefed-up README.md just wasn't cutting it. This document explains the problems I encountered and how I've tried to create a more effective system for crawl4ai, allowing you (and your AI assistant) to get precisely the right information.
\n
My Frustration with Standard llm.txt Files
\n
My experience with generic llm.txt files for complex libraries like crawl4ai revealed several pain points:
\n\n
\n
Information Overload & Lost Focus: I found that when I threw a massive, monolithic context file at an LLM, it often struggled. The sheer volume of information seemed to dilute its focus. If I asked a specific question about a niche feature, the LLM might get sidetracked by more prominent but currently irrelevant parts of the library. It felt like trying to find a single sentence in a thousand-page novel â the information was there, but not always accessible or prioritized correctly by the AI.
\n
\n
\n
The \"What\" Without the \"How\" or \"Why\": Most llm.txt files I encountered were essentially API dumps â a list of functions, classes, and parameters. This is the \"what\" of a library. But to truly use a library effectively, especially one as flexible as crawl4ai, you need the \"how\" (idiomatic usage patterns, best practices for common tasks) and the \"why\" (the design rationale behind certain features). Without this, I noticed my AI assistant would often generate syntactically correct but practically inefficient or non-idiomatic code. It was guessing the intent and the best way to use the library, and those guesses weren't always right.
\n
\n
\n
No Guidance on \"Thinking\" Like an Expert: A static list of facts doesn't teach an LLM the art of using the library. It doesn't convey the trade-offs an experienced developer considers, the common pitfalls they've learned to avoid, or the clever ways to combine features to solve complex problems. I wanted my AI assistant to not just recall an API, but to help me reason about the best way to build a solution with crawl4ai.
I've always admired how libraries like Lodash or jQuery (in its modular days) allowed developers to pick and choose only the parts they needed, resulting in smaller, more focused bundles. This idea of modularity and selective inclusion resonated deeply with me as I thought about LLM context. Why force-feed an LLM the entire library's details when I'm only working on a specific component or task?
\n
This led me to develop a new approach for crawl4ai: multi-dimensional, modular contexts.
\n
Instead of one giant llm.txt, I've broken down the crawl4ai documentation into:
\n\n
Logical Components: Context is organized around the major functional areas of the library (e.g., Core, Data Extraction, Deep Crawling, Markdown Generation, etc.). This allows you to select context relevant only to the task at hand.
\n
Three Dimensions of Context for Each Component:
\n
_memory.md (Foundational Memory): This is the \"what.\" It contains the precise, factual information about the component's public API, data structures, configuration objects, parameters, and method signatures. It's the detailed, unambiguous reference.
\n
_reasoning.md (Reasoning & Problem-Solving Framework): This is the \"how\" and \"why.\" It includes design principles, common task workflows with decision guides, best practices, anti-patterns, illustrative code examples solving real problems, and explanations of trade-offs. It aims to guide the LLM in \"thinking\" like an expert crawl4ai user.
\n
_examples.md (Practical Code Examples): This is pure \"show-me-the-code.\" It's a collection of runnable snippets demonstrating various ways to use the component's features and configurations, with minimal explanatory text. Itâs for quickly seeing different patterns in action.
\n
\n
\n\n
The Goal:\nMy aim is to provide you with a flexible system. You can give your AI assistant:\n* Just the memory files for quick API lookups.\n* The reasoning files (perhaps with memory) for help designing solutions.\n* The examples files for seeing practical implementations.\n* A combination of these across one or more components tailored to your specific task.\n* Or, for broader understanding, special aggregate contexts like the \"Vibe Coding\" context or the \"All Library Context.\"
\n
By providing these structured, multi-faceted contexts, I hope to significantly improve the quality and relevance of the assistance you get when using AI to code with crawl4ai. The following sections will guide you on how to select and use these different context files.
Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package, use it with Docker, or run it as a local server.
đĄ After installation with \"torch\", \"transformer\", or \"all\" options, it's recommended to run the following CLI command to load the required models:
\n
crawl4ai-download-models\n
\n
This is optional but will boost the performance and speed of the crawler. You only need to do this once after installation.
\n
Playwright Installation Note for Ubuntu
\n
If you encounter issues with Playwright installation on Ubuntu, you may need to install additional dependencies:
Docker support for Crawl4AI is currently in progress and will be available soon. This will allow you to run Crawl4AI in a containerized environment, ensuring consistency across different systems.
\n
Option 3: Local Server Installation
\n
For those who prefer to run Crawl4AI as a local server, instructions will be provided once the Docker implementation is complete.
\n
Verifying Your Installation
\n
After installation, you can verify that Crawl4AI is working correctly by running a simple Python script:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n result = await crawler.arun(url=\"https://www.example.com\")\n print(result.markdown[:500]) # Print first 500 characters\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
This script should successfully crawl the example website and print the first 500 characters of the extracted content.
\n
Getting Help
\n
If you encounter any issues during installation or usage, please check the documentation or raise an issue on the GitHub repository.
\n- Performs OS-level checks (e.g., missing libs on Linux)\n- Confirms your environment is ready to crawl\n
2.2 Diagnostics
\n
crawl4ai-doctor\n
\n- Check Python version compatibility\n- Verify Playwright installation\n- Inspect environment variables or library conflicts\nIf any issues arise, follow its suggestions (e.g., installing additional system packages) and re-run crawl4ai-setup.\n
3. Verifying Installation: A Simple Crawl (Skip this step if you already run crawl4ai-doctor)
\n
Below is a minimal Python script demonstrating a basic crawl. It uses our new BrowserConfig and CrawlerRunConfig for clarity, though no custom settings are passed in this example:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.example.com\",\n )\n print(result.markdown[:300]) # Show the first 300 characters of extracted text\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- A headless browser session loads example.com\n- Crawl4AI returns ~300 characters of markdown. \nIf errors occur, rerun crawl4ai-doctor or manually ensure Playwright is installed correctly.\n
docker pull unclecode/crawl4ai:basic\ndocker run -p 11235:11235 unclecode/crawl4ai:basic\n
\nYou can then make POST requests to http://localhost:11235/crawl to perform crawls.âProduction usage is discouraged until our new Docker approach is ready (planned in Jan or Feb 2025).\n
6. Local Server Mode (Legacy)
\n
Summary
\n
1.âInstall with pip install crawl4ai and run crawl4ai-setup.\n2.âDiagnose with crawl4ai-doctor if you see errors.\n3.âVerify by crawling example.com with minimal BrowserConfig + CrawlerRunConfig.
\n
Quick Start
\n
Getting Started with Crawl4AI
\n\n
Run your first crawl using minimal configuration.
\n
Experiment with a simple CSS-based extraction strategy.
\n
Crawl a dynamic page that loads content via JavaScript.
\n\n
1. Introduction
\n
\n
An asynchronous crawler, AsyncWebCrawler.
\n
Configurable browser and run settings via BrowserConfig and CrawlerRunConfig.
\n
Automatic HTML-to-Markdown conversion via DefaultMarkdownGenerator (supports optional filters).
\n
Multiple extraction strategies (LLM-based or âtraditionalâ CSS/XPath-based).
\n
\n
2. Your First Crawl
\n
Hereâs a minimal Python script that creates an AsyncWebCrawler, fetches a webpage, and prints the first 300 characters of its Markdown output:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\")\n print(result.markdown[:300]) # Print first 300 chars\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- AsyncWebCrawler launches a headless browser (Chromium by default).\n- It fetches https://example.com.\n- Crawl4AI automatically converts the HTML into Markdown.\n
3. Basic Configuration (Light Introduction)
\n
1.âBrowserConfig: Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.). \n2.âCrawlerRunConfig: Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.).\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n browser_conf = BrowserConfig(headless=True) # or False to see the browser\n run_conf = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS\n )\n\n asyncwith AsyncWebCrawler(config=browser_conf) as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=run_conf\n )\n print(result.markdown)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
\n
IMPORTANT: By default cache mode is set to CacheMode.BYPASS to have fresh content. Set CacheMode.ENABLED to enable caching.
\n
\n
4. Generating Markdown Output
\n
\n
result.markdown:
\n
result.markdown.fit_markdown: \n The same content after applying any configured content filter (e.g., PruningContentFilter).
\n
\n
Example: Using a Filter with DefaultMarkdownGenerator
\nNote: If you do not specify a content filter or markdown generator, youâll typically see only the raw Markdown. PruningContentFilter may adds around 50ms in processing time. Weâll dive deeper into these strategies in a dedicated Markdown Generation tutorial.\n
5. Simple Data Extraction (CSS-based)
\n
from crawl4ai import JsonCssExtractionStrategy\nfrom crawl4ai import LLMConfig\n\n# Generate a schema (one-time cost)\nhtml = \"<div class='product'><h2>Gaming Laptop</h2><span class='price'>$999.99</span></div>\"\n\n# Using OpenAI (requires API token)\nschema = JsonCssExtractionStrategy.generate_schema(\n html,\n llm_config = LLMConfig(provider=\"openai/gpt-4o\",api_token=\"your-openai-token\") # Required for OpenAI\n)\n\n# Or using Ollama (open source, no token needed)\nschema = JsonCssExtractionStrategy.generate_schema(\n html,\n llm_config = LLMConfig(provider=\"ollama/llama3.3\", api_token=None) # Not needed for Ollama\n)\n\n# Use the schema for fast, repeated extractions\nstrategy = JsonCssExtractionStrategy(schema)\n
\n- Great for repetitive page structures (e.g., item listings, articles).\n- No AI usage or costs.\n- The crawler returns a JSON string you can parse or store.\n- For sites where data is split across sibling elements (e.g. Hacker News), use the \"source\" field key to navigate to a sibling before extracting: {\"name\": \"score\", \"selector\": \"span.score\", \"type\": \"text\", \"source\": \"+ tr\"}.\n
\n
Tips: You can pass raw HTML to the crawler instead of a URL. To do so, prefix the HTML with raw://.
\n- Automatic stopping: Stops when sufficient information is gathered\n- Intelligent link selection: Follows only relevant links\n- Confidence scoring: Know how complete your information is\n
8. Multi-URL Concurrency (Preview)
\n
If you need to crawl multiple URLs in parallel, you can use arun_many(). By default, Crawl4AI employs a MemoryAdaptiveDispatcher, automatically adjusting concurrency based on system resources. Hereâs a quick glimpse:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefquick_parallel_example():\n urls = [\n \"https://example.com/page1\",\n \"https://example.com/page2\",\n \"https://example.com/page3\"\n ]\n\n run_conf = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n stream=True# Enable streaming mode\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n # Stream results as they complete\n asyncfor result inawait crawler.arun_many(urls, config=run_conf):\n if result.success:\n print(f\"[OK] {result.url}, length: {len(result.markdown.raw_markdown)}\")\n else:\n print(f\"[ERROR] {result.url} => {result.error_message}\")\n\n # Or get all results at once (default behavior)\n run_conf = run_conf.clone(stream=False)\n results = await crawler.arun_many(urls, config=run_conf)\n for res in results:\n if res.success:\n print(f\"[OK] {res.url}, length: {len(res.markdown.raw_markdown)}\")\n else:\n print(f\"[ERROR] {res.url} => {res.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(quick_parallel_example())\n
\n1. Streaming mode (stream=True): Process results as they become available using async for\n2. Batch mode (stream=False): Wait for all results to complete\n
8. Dynamic Content Example
\n
Some sites require multiple âpage clicksâ or dynamic JavaScript updates. Below is an example showing how to click a âNext Pageâ button and wait for new commits to load on GitHub, using BrowserConfig and CrawlerRunConfig:\n
\n- BrowserConfig(headless=False): We want to watch it click âNext Page.â \n- CrawlerRunConfig(...): We specify the extraction strategy, pass session_id to reuse the same page. \n- js_code and wait_for are used for subsequent pages (page > 0) to click the âNextâ button and wait for new commits to load. \n- js_only=True indicates weâre not re-navigating but continuing the existing session. \n- Finally, we call kill_session() to clean up the page and browser session.\n
9. Next Steps
\n\n
Performed a basic crawl and printed Markdown.
\n
Used content filters with a markdown generator.
\n
Extracted JSON via CSS or LLM strategies.
\n
Handled dynamic pages with JavaScript triggers.
\n\n
Core API
\n
AsyncWebCrawler
\n
The AsyncWebCrawler is the core class for asynchronous web crawling in Crawl4AI.âYou typically create it once, optionally customize it with a BrowserConfig (e.g., headless, user agent), then run multiple arun() calls with different CrawlerRunConfig objects.\n1.âCreate a BrowserConfig for global browser settings.â \n2.âInstantiateAsyncWebCrawler(config=browser_config).â \n3.âUse the crawler in an async context manager (async with) or manage start/close manually.â \n4.âCallarun(url, config=crawler_run_config) for each page you want.
\n
1.âConstructor Overview
\n
classAsyncWebCrawler:\n def__init__(\n self,\n crawler_strategy: Optional[AsyncCrawlerStrategy] = None,\n config: Optional[BrowserConfig] = None,\n always_bypass_cache: bool = False, # deprecated\n always_by_pass_cache: Optional[bool] = None, # also deprecated\n base_directory: str = ...,\n thread_safe: bool = False,\n **kwargs,\n ):\n \"\"\"\n Create an AsyncWebCrawler instance.\n\n Args:\n crawler_strategy: \n (Advanced) Provide a custom crawler strategy if needed.\n config: \n A BrowserConfig object specifying how the browser is set up.\n always_bypass_cache: \n (Deprecated) Use CrawlerRunConfig.cache_mode instead.\n base_directory: \n Folder for storing caches/logs (if relevant).\n thread_safe: \n If True, attempts some concurrency safeguards.âUsually False.\n **kwargs: \n Additional legacy or debugging parameters.\n \"\"\"\n )\n\n### Typical Initialization\n\n```python\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig\nbrowser_cfg = BrowserConfig(\n browser_type=\"chromium\",\n headless=True,\n verbose=True\ncrawler = AsyncWebCrawler(config=browser_cfg)\n
\n
Notes:
\n
\n
Legacy parameters like always_bypass_cache remain for backward compatibility, but prefer to set caching in CrawlerRunConfig.
\n
\n\n
2.âLifecycle: Start/Close or Context Manager
\n
2.1 Context Manager (Recommended)
\n
asyncwithAsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\"https://example.com\")\n # The crawler automatically starts/closes resources\n
\n
When the async with block ends, the crawler cleans up (closes the browser, etc.).
You pass a CrawlerRunConfig object that sets up everything about a crawlâcontent filtering, caching, session reuse, JS code, screenshots, etc.
\n
import asyncio\nfrom crawl4ai import CrawlerRunConfig, CacheMode\nrun_cfg = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n css_selector=\"main.article\",\n word_count_threshold=10,\n screenshot=True\nasyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\"https://example.com/news\", config=run_cfg)\n
\n
3.2 Legacy Parameters Still Accepted
\n
For backward compatibility, arun() can still accept direct arguments like css_selector=..., word_count_threshold=..., etc., but we strongly advise migrating them into a CrawlerRunConfig.
The arun_many() method now uses an intelligent dispatcher that:
\n
\n
Monitors system memory usage
\n
Implements adaptive rate limiting
\n
Provides detailed progress monitoring
\n
Manages concurrent crawls efficiently
\n
\n
4.2 Example Usage
\n
Check page Multi-url Crawling for a detailed example of how to use arun_many().
\n
### 4.3 Key Features\n1.â**Rate Limiting**\n - Automatic delay between requests\n - Exponential backoff on rate limit detection\n - Domain-specific rate limiting\n - Configurable retry strategy\n2.â**Resource Monitoring**\n - Memory usage tracking\n - Adaptive concurrency based on system load\n - Automatic pausing when resources are constrained\n3.â**Progress Monitoring**\n - Detailed or aggregated progress display\n - Real-time status updates\n - Memory usage statistics\n4.â**Error Handling**\n - Graceful handling of rate limits\n - Automatic retries with backoff\n - Detailed error reporting\n## 5.â`CrawlResult` Output\nEach `arun()` returns a **`CrawlResult`** containing:\n- `url`: Final URL (if redirected).\n- `html`: Original HTML.\n- `cleaned_html`: Sanitized HTML.\n- `markdown_v2`: Removed in v0.5. Accessing it raises `AttributeError`. Use `markdown`.\n- `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies).\n- `screenshot`, `pdf`: If screenshots/PDF requested.\n- `media`, `links`: Information about discovered images/links.\n- `success`, `error_message`: Status info.\n## 6.âQuick Example\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom crawl4ai import JsonCssExtractionStrategy\nimport json\n\nasyncdefmain():\n # 1.âBrowser config\n browser_cfg = BrowserConfig(\n browser_type=\"firefox\",\n headless=False,\n verbose=True\n )\n\n # 2.âRun config\n schema = {\n \"name\": \"Articles\",\n \"baseSelector\": \"article.post\",\n \"fields\": [\n {\n \"name\": \"title\", \n \"selector\": \"h2\", \n \"type\": \"text\"\n },\n {\n \"name\": \"url\", \n \"selector\": \"a\", \n \"type\": \"attribute\", \n \"attribute\": \"href\"\n }\n ]\n }\n\n run_cfg = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n extraction_strategy=JsonCssExtractionStrategy(schema),\n word_count_threshold=15,\n remove_overlay_elements=True,\n wait_for=\"css:.post\"# Wait for posts to appear\n )\n\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\n url=\"https://example.com/blog\",\n config=run_cfg\n )\n\n if result.success:\n print(\"Cleaned HTML length:\", len(result.cleaned_html))\n if result.extracted_content:\n articles = json.loads(result.extracted_content)\n print(\"Extracted articles:\", articles[:2])\n else:\n print(\"Error:\", result.error_message)\n\nasyncio.run(main())\n
\n- We define a BrowserConfig with Firefox, no headless, and verbose=True.â \n- We define a CrawlerRunConfig that bypasses cache, uses a CSS extraction schema, has a word_count_threshold=15, etc.â \n- We pass them to AsyncWebCrawler(config=...) and arun(url=..., config=...).\n
7.âBest Practices & Migration Notes
\n
1.âUseBrowserConfig for global settings about the browserâs environment.â \n2.âUseCrawlerRunConfig for per-crawl logic (caching, content filtering, extraction strategies, wait conditions).â \n3.âAvoid legacy parameters like css_selector or word_count_threshold directly in arun().âInstead:\n
arun(url, config=CrawlerRunConfig) is the main method for single-page crawls.â
\n
arun_many(urls, config=CrawlerRunConfig) handles concurrency across multiple URLs.â
\n
For advanced lifecycle control, use start() and close() explicitly.â
\n
If you used AsyncWebCrawler(browser_type=\"chromium\", css_selector=\"...\"), move browser settings to BrowserConfig(...) and content/crawl logic to CrawlerRunConfig(...).
\n
\n
arun() Parameter Guide (New Approach)
\n
In Crawl4AIâs latest configuration model, nearly all parameters that once went directly to arun() are now part of CrawlerRunConfig.âWhen calling arun(), you provide:\n
\nBelow is an organized look at the parameters that can go inside CrawlerRunConfig, divided by their functional areas.âFor Browser settings (e.g., headless, browser_type), see BrowserConfig.\n
1.âCore Usage
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n run_config = CrawlerRunConfig(\n verbose=True, # Detailed logging\n cache_mode=CacheMode.ENABLED, # Use normal read/write cache\n check_robots_txt=True, # Respect robots.txt rules\n # ...âother parameters\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=run_config\n )\n\n # Check if blocked by robots.txt\n ifnot result.success and result.status_code == 403:\n print(f\"Error: {result.error_message}\")\n
\n- verbose=True logs each crawl step.â \n- cache_mode decides how to read/write the local crawl cache.\n
2.âCache Control
\n
cache_mode (default: CacheMode.ENABLED) \nUse a built-in enum from CacheMode:\n- ENABLED: Normal cachingâreads if available, writes if missing.\n- DISABLED: No cachingâalways refetch pages.\n- READ_ONLY: Reads from cache only; no new writes.\n- WRITE_ONLY: Writes to cache but doesnât read existing data.\n- BYPASS: Skips reading cache for this crawl (though it might still write if set up that way).\n
\n- CacheMode.BYPASS â Skip cache entirely; always fetch fresh, write result to cache.\n- CacheMode.DISABLED â No caching at all; don't read or write.\n- CacheMode.WRITE_ONLY â Never read from cache, but write results.\n- CacheMode.READ_ONLY â Read from cache if available, never write.\n
3.âContent Processing & Selection
\n
3.1 Text Processing
\n
run_config = CrawlerRunConfig(\n word_count_threshold=10, # Ignore text blocks <10 words\n only_text=False, # If True, tries to remove non-text elements\n keep_data_attributes=False# Keep or discard data-* attributes\n)\n
\n
3.2 Content Selection
\n
run_config = CrawlerRunConfig(\n css_selector=\".main-content\", # Focus on .main-content region only\n excluded_tags=[\"form\", \"nav\"], # Remove entire tag blocks\n remove_forms=True, # Specifically strip <form> elements\n remove_overlay_elements=True, # Attempt to remove modals/popups\n)\n
\n
3.3 Link Handling
\n
run_config = CrawlerRunConfig(\n exclude_external_links=True, # Remove external links from final content\n exclude_social_media_links=True, # Remove links to known social sites\n exclude_domains=[\"ads.example.com\"], # Exclude links to these domains\n exclude_social_media_domains=[\"facebook.com\",\"twitter.com\"], # Extend the default list\n)\n
\n
3.4 Media Filtering
\n
run_config = CrawlerRunConfig(\n exclude_external_images=True# Strip images from other domains\n)\n
\n
4.âPage Navigation & Timing
\n
4.1 Basic Browser Flow
\n
run_config = CrawlerRunConfig(\n wait_for=\"css:.dynamic-content\", # Wait for .dynamic-content\n delay_before_return_html=2.0, # Wait 2s before capturing final HTML\n page_timeout=60000, # Navigation & script timeout (ms)\n)\n
\n- wait_for: \n - \"css:selector\" or \n - \"js:() => boolean\" \n e.g.âjs:() => document.querySelectorAll('.item').length > 10.\n- mean_delay & max_range: define random delays for arun_many() calls.â \n- semaphore_count: concurrency limit when crawling multiple URLs.\n
\n- js_code can be a single string or a list of strings.â \n- js_only=True means âIâm continuing in the same session with new JS steps, no new full navigation.â\n
\nIf re-used in subsequent arun() calls, the same tab/page context is continued (helpful for multi-step tasks or stateful browsing).\n
6.âScreenshot, PDF & Media Options
\n
run_config = CrawlerRunConfig(\n screenshot=True, # Grab a screenshot as base64\n screenshot_wait_for=1.0, # Wait 1s before capturing\n pdf=True, # Also produce a PDF\n image_description_min_word_threshold=5, # If analyzing alt text\n image_score_threshold=3, # Filter out low-score images\n)\n
\n- result.screenshot â Base64 screenshot string.\n- result.pdf â Byte array with PDF data.\n
7.âExtraction Strategy
\n
For advanced data extraction (CSS/LLM-based), set extraction_strategy:\n
\n1.âCrawling the main content region, ignoring external links.â \n2.âRunning JavaScript to click â.show-moreâ.â \n3.âWaiting for â.loaded-blockâ to appear.â \n4.âGenerating a screenshot & PDF of the final page.â \n
9.âBest Practices
\n
1.âUse BrowserConfig for global browser settings (headless, user agent).â \n2.âUse CrawlerRunConfig to handle the specific crawl needs: content filtering, caching, JS, screenshot, extraction, etc.â \n4.âLimit large concurrency (semaphore_count) if the site or your system canât handle it.â \n5.âFor dynamic pages, set js_code or scan_full_page so you load all content.
\n
10.âConclusion
\n
All parameters that used to be direct arguments to arun() now belong in CrawlerRunConfig.âThis approach:\n- Makes code clearer and more maintainable.â
\n
arun_many(...) Reference
\n
\n
Note: This function is very similar to arun() but focused on concurrent or batch crawling.âIf youâre unfamiliar with arun() usage, please read that doc first, then review this for differences.
\n
\n
Function Signature
\n
asyncdefarun_many(\n urls: Union[List[str], List[Any]],\n config: Optional[Union[CrawlerRunConfig, List[CrawlerRunConfig]]] = None,\n dispatcher: Optional[BaseDispatcher] = None,\n ...\n) -> Union[List[CrawlResult], AsyncGenerator[CrawlResult, None]]:\n \"\"\"\n Crawl multiple URLs concurrently or in batches.\n\n :param urls: A list of URLs (or tasks) to crawl.\n :param config: (Optional) Either:\n - A single `CrawlerRunConfig` applying to all URLs\n - A list of `CrawlerRunConfig` objects with url_matcher patterns\n :param dispatcher: (Optional) A concurrency controller (e.g.âMemoryAdaptiveDispatcher).\n ...\n :return: Either a list of `CrawlResult` objects, or an async generator if streaming is enabled.\n \"\"\"\n
\n
Differences from arun()
\n
1.âMultiple URLs: \n - Instead of crawling a single URL, you pass a list of them (strings or tasks).â \n - The function returns either a list of CrawlResult or an async generator if streaming is enabled.\n2.âConcurrency & Dispatchers: \n - dispatcher param allows advanced concurrency control.â \n - If omitted, a default dispatcher (like MemoryAdaptiveDispatcher) is used internally.â \n3.âStreaming Support: \n - Enable streaming by setting stream=True in your CrawlerRunConfig.\n - When streaming, use async for to process results as they become available.\n4.âParallel Execution: \n - arun_many() can run multiple requests concurrently under the hood.â \n - Each CrawlResult might also include a dispatch_result** with concurrency details (like memory usage, start/end times).
\n
Basic Example (Batch Mode)
\n
# Minimal usage: The default dispatcher will be used\nresults = await crawler.arun_many(\n urls=[\"https://site1.com\", \"https://site2.com\"],\n config=CrawlerRunConfig(stream=False) # Default behavior\n)\n\nfor res in results:\n if res.success:\n print(res.url, \"crawled OK!\")\n else:\n print(\"Failed:\", res.url, \"-\", res.error_message)\n
\n
Streaming Example
\n
config = CrawlerRunConfig(\n stream=True, # Enable streaming mode\n cache_mode=CacheMode.BYPASS\n)\n\n# Process results as they complete\nasyncfor result inawait crawler.arun_many(\n urls=[\"https://site1.com\", \"https://site2.com\", \"https://site3.com\"],\n config=config\n):\n if result.success:\n print(f\"Just completed: {result.url}\")\n # Process each result immediately\n process_result(result)\n
Instead of using one config for all URLs, provide a list of configs with url_matcher patterns:\n
from crawl4ai import CrawlerRunConfig, MatchMode\nfrom crawl4ai.processors.pdf import PDFContentScrapingStrategy\nfrom crawl4ai.extraction_strategy import JsonCssExtractionStrategy\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\n# PDF files - specialized extraction\npdf_config = CrawlerRunConfig(\n url_matcher=\"*.pdf\",\n scraping_strategy=PDFContentScrapingStrategy()\n)\n\n# Blog/article pages - content filtering\nblog_config = CrawlerRunConfig(\n url_matcher=[\"*/blog/*\", \"*/article/*\", \"*python.org*\"],\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.48)\n )\n)\n\n# Dynamic pages - JavaScript execution\ngithub_config = CrawlerRunConfig(\n url_matcher=lambda url: 'github.com'in url,\n js_code=\"window.scrollTo(0, 500);\"\n)\n\n# API endpoints - JSON extraction\napi_config = CrawlerRunConfig(\n url_matcher=lambda url: 'api'in url or url.endswith('.json'),\n # Custome settings for JSON extraction\n)\n\n# Default fallback config\ndefault_config = CrawlerRunConfig() # No url_matcher means it never matches except as fallback\n\n# Pass the list of configs - first match wins!\nresults = await crawler.arun_many(\n urls=[\n \"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf\", # â pdf_config\n \"https://blog.python.org/\", # â blog_config\n \"https://github.com/microsoft/playwright\", # â github_config\n \"https://httpbin.org/json\", # â api_config\n \"https://example.com/\"# â default_config\n ],\n config=[pdf_config, blog_config, github_config, api_config, default_config]\n)\n
\n- String patterns: \"*.pdf\", \"*/blog/*\", \"*python.org*\"\n- Function matchers: lambda url: 'api' in url\n- Mixed patterns: Combine strings and functions with MatchMode.OR or MatchMode.AND\n- First match wins: Configs are evaluated in order\n- dispatch_result in each CrawlResult (if using concurrency) can hold memory and timing info.â \n- Important: Always include a default config (without url_matcher) as the last item if you want to handle all URLs. Otherwise, unmatched URLs will fail.\n
Return Value
\n
Either a list of CrawlResult objects, or an async generator if streaming is enabled.âYou can iterate to check result.success or read each itemâs extracted_content, markdown, or dispatch_result.
\n
Dispatcher Reference
\n
\n
MemoryAdaptiveDispatcher: Dynamically manages concurrency based on system memory usage.â
\n
SemaphoreDispatcher: Fixed concurrency limit, simpler but less adaptive.â
\n
\n
Common Pitfalls
\n
3.âError Handling: Each CrawlResult might fail for different reasonsâalways check result.success or the error_message before proceeding.
\n
Conclusion
\n
Use arun_many() when you want to crawl multiple URLs simultaneously or in controlled parallel tasks.âIf you need advanced concurrency features (like memory-based adaptive throttling or complex rate-limiting), provide a dispatcher.âEach result is a standard CrawlResult, possibly augmented with concurrency stats (dispatch_result) for deeper inspection.âFor more details on concurrency logic and dispatchers, see the Advanced Multi-URL Crawling docs.
\n
CrawlResult Reference
\n
The CrawlResult class encapsulates everything returned after a single crawl operation. It provides the raw or processed content, details on links and media, plus optional metadata (like screenshots, PDFs, or extracted JSON).\nLocation: crawl4ai/crawler/models.py (for reference)\n
if result.status_code == 404:\n print(\"Page not found!\")\n
\n
1.4 error_message(Optional[str])
\n
What: If success=False, a textual description of the failure. \n
if not result.success:\n print(\"Error:\", result.error_message)\n
\n
1.5 session_id(Optional[str])
\n
# If you used session_id=\"login_session\" in CrawlerRunConfig, see it here:\nprint(\"Session:\", result.session_id)\n
\n
1.6 response_headers(Optional[dict])
\n
if result.response_headers:\n print(\"Server:\", result.response_headers.get(\"Server\", \"Unknown\"))\n
\n
1.7 ssl_certificate(Optional[SSLCertificate])
\n
What: If fetch_ssl_certificate=True in your CrawlerRunConfig, result.ssl_certificate contains a SSLCertificate object describing the site's certificate. You can export the cert in multiple formats (PEM/DER/JSON) or access its properties like issuer, \n subject, valid_from, valid_until, etc. \n
if result.ssl_certificate:\n print(\"Issuer:\", result.ssl_certificate.issuer)\n
\n
2. Raw / Cleaned Content
\n
2.1 html(str)
\n
# Possibly large\nprint(len(result.html))\n
\n
2.2 cleaned_html(Optional[str])
\n
What: A sanitized HTML versionâscripts, styles, or excluded tags are removed based on your CrawlerRunConfig. \n
print(result.cleaned_html[:500]) # Show a snippet\n
\n
3. Markdown Fields
\n
3.1 The Markdown Generation Approach
\n
\n
Raw markdown
\n
Links as citations (with a references section)
\n
Fit markdown if a content filter is used (like Pruning or BM25)\nMarkdownGenerationResult includes:
\n
raw_markdown(str): The full HTMLâMarkdown conversion.
\n
markdown_with_citations(str): Same markdown, but with link references as academic-style citations.
\n
references_markdown(str): The reference list or footnotes at the end.
\n
fit_markdown(Optional[str]): If content filtering (Pruning/BM25) was applied, the filtered \"fit\" text.
\n
fit_html(Optional[str]): The HTML that led to fit_markdown.\n
if result.markdown:\n md_res = result.markdown\n print(\"Raw MD:\", md_res.raw_markdown[:300])\n print(\"Citations MD:\", md_res.markdown_with_citations[:300])\n print(\"References:\", md_res.references_markdown)\n if md_res.fit_markdown:\n print(\"Pruned text:\", md_res.fit_markdown[:300])\n
\nImportant: \"Fit\" content (in fit_markdown/fit_html) exists in result.markdown, only if you used a filter (like PruningContentFilter or BM25ContentFilter) within a MarkdownGenerationStrategy.\n
4. Media & Links
\n
4.1 media(Dict[str, List[Dict]])
\n
What: Contains info about discovered images, videos, or audio. Typically keys: \"images\", \"videos\", \"audios\". \n- src(str): Media URL \n- alt or title(str): Descriptive text \n- score(float): Relevance score if the crawler's heuristic found it \"important\" \n- desc or description(Optional[str]): Additional context extracted from surrounding text \n
images = result.media.get(\"images\", [])\nfor img in images:\n if img.get(\"score\", 0) > 5:\n print(\"High-value image:\", img[\"src\"])\n
\n
4.2 links(Dict[str, List[Dict]])
\n
What: Holds internal and external link data. Usually two keys: \"internal\" and \"external\". \n- href(str): The link target \n- text(str): Link text \n- title(str): Title attribute \n- context(str): Surrounding text snippet \n- domain(str): If external, the domain\n
for link in result.links[\"internal\"]:\n print(f\"Internal link to {link['href']} with text {link['text']}\")\n
\n
5. Additional Fields
\n
5.1 extracted_content(Optional[str])
\n
What: If you used extraction_strategy (CSS, LLM, etc.), the structured output (JSON). \n
if result.extracted_content:\n data = json.loads(result.extracted_content)\n print(data)\n
\n
5.2 downloaded_files(Optional[List[str]])
\n
What: If accept_downloads=True in your BrowserConfig + downloads_path, lists local file paths for downloaded items. \n
if result.downloaded_files:\n for file_path in result.downloaded_files:\n print(\"Downloaded:\", file_path)\n
\n
5.3 screenshot(Optional[str])
\n
What: Base64-encoded screenshot if screenshot=True in CrawlerRunConfig. \n
import base64\nif result.screenshot:\n withopen(\"page.png\", \"wb\") as f:\n f.write(base64.b64decode(result.screenshot))\n
\n
5.4 pdf(Optional[bytes])
\n
What: Raw PDF bytes if pdf=True in CrawlerRunConfig. \n
if result.pdf:\n withopen(\"page.pdf\", \"wb\") as f:\n f.write(result.pdf)\n
\n
5.5 mhtml(Optional[str])
\n
What: MHTML snapshot of the page if capture_mhtml=True in CrawlerRunConfig. MHTML (MIME HTML) format preserves the entire web page with all its resources (CSS, images, scripts, etc.) in a single file. \n
if result.mhtml:\n withopen(\"page.mhtml\", \"w\", encoding=\"utf-8\") as f:\n f.write(result.mhtml)\n
\n
5.6 metadata(Optional[dict])
\n
if result.metadata:\n print(\"Title:\", result.metadata.get(\"title\"))\n print(\"Author:\", result.metadata.get(\"author\"))\n
\n
6. dispatch_result (optional)
\n
A DispatchResult object providing additional concurrency and resource usage information when crawling URLs in parallel (e.g., via arun_many() with custom dispatchers). It contains:\n- task_id: A unique identifier for the parallel task.\n- memory_usage (float): The memory (in MB) used at the time of completion.\n- peak_memory (float): The peak memory usage (in MB) recorded during the task's execution.\n- start_time / end_time (datetime): Time range for this crawling task.\n- error_message (str): Any dispatcher- or concurrency-related error encountered.\n
# Example usage:\nfor result in results:\n if result.success and result.dispatch_result:\n dr = result.dispatch_result\n print(f\"URL: {result.url}, Task ID: {dr.task_id}\")\n print(f\"Memory: {dr.memory_usage:.1f} MB (Peak: {dr.peak_memory:.1f} MB)\")\n print(f\"Duration: {dr.end_time - dr.start_time}\")\n
\n
\n
Note: This field is typically populated when using arun_many(...) alongside a dispatcher (e.g., MemoryAdaptiveDispatcher or SemaphoreDispatcher). If no concurrency or dispatcher is used, dispatch_result may remain None.
\n
\n
7. Network Requests & Console Messages
\n
When you enable network and console message capturing in CrawlerRunConfig using capture_network_requests=True and capture_console_messages=True, the CrawlResult will include these fields:
Each item has an event_type field that can be \"request\", \"response\", or \"request_failed\".
\n
Request events include url, method, headers, post_data, resource_type, and is_navigation_request.
\n
Response events include url, status, status_text, headers, and request_timing.
\n
Failed request events include url, method, resource_type, and failure_text.
\n
All events include a timestamp field.\n
if result.network_requests:\n # Count different types of events\n requests = [r for r in result.network_requests if r.get(\"event_type\") == \"request\"]\n responses = [r for r in result.network_requests if r.get(\"event_type\") == \"response\"]\n failures = [r for r in result.network_requests if r.get(\"event_type\") == \"request_failed\"]\n\n print(f\"Captured {len(requests)} requests, {len(responses)} responses, and {len(failures)} failures\")\n\n # Analyze API calls\n api_calls = [r for r in requests if\"api\"in r.get(\"url\", \"\")]\n\n # Identify failed resources\n for failure in failures:\n print(f\"Failed to load: {failure.get('url')} - {failure.get('failure_text')}\")\n
Each item has a type field indicating the message type (e.g., \"log\", \"error\", \"warning\", etc.).
\n
The text field contains the actual message text.
\n
Some messages include location information (URL, line, column).
\n
All messages include a timestamp field.\n
if result.console_messages:\n # Count messages by type\n message_types = {}\n for msg in result.console_messages:\n msg_type = msg.get(\"type\", \"unknown\")\n message_types[msg_type] = message_types.get(msg_type, 0) + 1\n\n print(f\"Message type counts: {message_types}\")\n\n # Display errors (which are usually most important)\n for msg in result.console_messages:\n if msg.get(\"type\") == \"error\":\n print(f\"Error: {msg.get('text')}\")\n
\n
\n
8. Example: Accessing Everything
\n
asyncdefhandle_result(result: CrawlResult):\n ifnot result.success:\n print(\"Crawl error:\", result.error_message)\n return\n\n # Basic info\n print(\"Crawled URL:\", result.url)\n print(\"Status code:\", result.status_code)\n\n # HTML\n print(\"Original HTML size:\", len(result.html))\n print(\"Cleaned HTML size:\", len(result.cleaned_html or\"\"))\n\n # Markdown output\n if result.markdown:\n print(\"Raw Markdown:\", result.markdown.raw_markdown[:300])\n print(\"Citations Markdown:\", result.markdown.markdown_with_citations[:300])\n if result.markdown.fit_markdown:\n print(\"Fit Markdown:\", result.markdown.fit_markdown[:200])\n\n # Media & Links\n if\"images\"in result.media:\n print(\"Image count:\", len(result.media[\"images\"]))\n if\"internal\"in result.links:\n print(\"Internal link count:\", len(result.links[\"internal\"]))\n\n # Extraction strategy result\n if result.extracted_content:\n print(\"Structured data:\", result.extracted_content)\n\n # Screenshot/PDF/MHTML\n if result.screenshot:\n print(\"Screenshot length:\", len(result.screenshot))\n if result.pdf:\n print(\"PDF bytes length:\", len(result.pdf))\n if result.mhtml:\n print(\"MHTML length:\", len(result.mhtml))\n\n # Network and console capturing\n if result.network_requests:\n print(f\"Network requests captured: {len(result.network_requests)}\")\n # Analyze request types\n req_types = {}\n for req in result.network_requests:\n if\"resource_type\"in req:\n req_types[req[\"resource_type\"]] = req_types.get(req[\"resource_type\"], 0) + 1\n print(f\"Resource types: {req_types}\")\n\n if result.console_messages:\n print(f\"Console messages captured: {len(result.console_messages)}\")\n # Count by message type\n msg_types = {}\n for msg in result.console_messages:\n msg_types[msg.get(\"type\", \"unknown\")] = msg_types.get(msg.get(\"type\", \"unknown\"), 0) + 1\n print(f\"Message types: {msg_types}\")\n
\n
9. Key Points & Future
\n
1.âDeprecated legacy properties of CrawlResult \n - markdown_v2 - Removed in v0.5. Accessing it raises AttributeError. Use markdown.\n - fit_markdown and fit_html - Removed as top-level CrawlResult properties in v0.5. Use result.markdown.fit_markdown and result.markdown.fit_html.\n2.âFit Content \n - fit_markdown and fit_html appear in MarkdownGenerationResult, only if you used a content filter (like PruningContentFilter or BM25ContentFilter) inside your MarkdownGenerationStrategy or set them directly. \n - If no filter is used, they remain None.\n3.âReferences & Citations \n - If you enable link citations in your DefaultMarkdownGenerator (options={\"citations\": True}), youâll see markdown_with_citations plus a references_markdown block. This helps large language models or academic-like referencing.\n4.âLinks & Media \n - links[\"internal\"] and links[\"external\"] group discovered anchors by domain. \n - media[\"images\"] / [\"videos\"] / [\"audios\"] store extracted media elements with optional scoring or context.\n5.âError Cases \n - If success=False, check error_message (e.g., timeouts, invalid URLs). \n - status_code might be None if we failed before an HTTP response.\nUse CrawlResult to glean all final outputs and feed them into your data pipelines, AI models, or archives. With the synergy of a properly configured BrowserConfig and CrawlerRunConfig, the crawler can produce robust, structured results here in CrawlResult.
Crawl4AI's flexibility stems from two key classes:\n1. BrowserConfig â Dictates how the browser is launched and behaves (e.g., headless or visible, proxy, user agent). \n2. CrawlerRunConfig â Dictates how each crawl operates (e.g., caching, extraction, timeouts, JavaScript code to run, etc.). \n3. LLMConfig - Dictates how LLM providers are configured. (model, api token, base url, temperature etc.)\nIn most examples, you create oneBrowserConfig for the entire crawler session, then pass a fresh or re-used CrawlerRunConfig whenever you call arun(). This tutorial shows the most commonly used parameters. If you need advanced or rarely used fields, see the Configuration Parameters.
Some sites behave differently with smaller or bigger viewports.
\n
verbose:
\n
If True, prints extra logs.
\n
Handy for debugging.
\n
use_persistent_context:
\n
If True, uses a persistent browser profile, storing cookies/local storage across runs.
\n
Typically also set user_data_dir to point to a folder.
\n
cookies & headers:
\n
E.g. cookies=[{\"name\": \"session\", \"value\": \"abc123\", \"domain\": \"example.com\"}].
\n
user_agent:
\n
Custom User-Agent string. If None, a default is used.
\n
You can also set user_agent_mode=\"random\" for randomization (if you want to fight bot detection).
\n
text_mode & light_mode:
\n
text_mode=True disables images, possibly speeding up text-only crawls.
\n
light_mode=True turns off certain background features for performance.
\n
avoid_ads & avoid_css:
\n
avoid_ads=True blocks requests to common ad and tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level. Reduces network overhead and memory usage.
\n
avoid_css=True blocks loading of CSS files (.css, .less, .scss, .sass), useful when you only need text content and want faster, leaner crawls.
\n
Both default to False (opt-in). Can be combined with each other and with text_mode.
\n
\n
\n
extra_args:
\n
Additional flags for the underlying browser.
\n
E.g. [\"--disable-extensions\"].
\n
\n
\n
enable_stealth:
\n
If True, enables stealth mode using playwright-stealth.
\n
Modifies browser fingerprints to avoid basic bot detection.
\n
Default is False. Recommended for sites with bot protection.
\n
\n
\n\n
Helper Methods
\n
Both configuration classes provide a clone() method to create modified copies:\n
# Create a base browser config\nbase_browser = BrowserConfig(\n browser_type=\"chromium\",\n headless=True,\n text_mode=True\n)\n\n# Create a visible browser config for debugging\ndebug_browser = base_browser.clone(\n headless=False,\n verbose=True\n)\n
\n
from crawl4ai import AsyncWebCrawler, BrowserConfig\n\nbrowser_conf = BrowserConfig(\n browser_type=\"firefox\",\n headless=False,\n text_mode=True\n)\n\nasyncwith AsyncWebCrawler(config=browser_conf) as crawler:\n result = await crawler.arun(\"https://example.com\")\n print(result.markdown[:300])\n
geolocation: GPS coordinates via GeolocationConfig(latitude=48.8566, longitude=2.3522)
\n
verbose:
\n
Logs additional runtime details.
\n
Overlaps with the browser's verbosity if also set to True in BrowserConfig.
\n
enable_rate_limiting:
\n
If True, enables rate limiting for batch processing.
\n
Requires rate_limit_config to be set.
\n
memory_threshold_percent:
\n
The memory threshold (as a percentage) to monitor.
\n
If exceeded, the crawler will pause or slow down.
\n
\n
\n
check_interval:
\n
The interval (in seconds) to check system resources.
\n
Affects how often memory and CPU usage are monitored.
\n
\n
\n
max_session_permit:
\n
The maximum number of concurrent crawl sessions.
\n
Helps prevent overwhelming the system.
\n
\n
\n
url_matcher & match_mode:
\n
Enable URL-specific configurations when used with arun_many().
\n
Set url_matcher to patterns (glob, function, or list) to match specific URLs.
\n
Use match_mode (OR/AND) to control how multiple patterns combine.
\n
\n
\n
display_mode:
\n
The display mode for progress information (DETAILED, BRIEF, etc.).
\n
Affects how much information is printed during the crawl.
\n
\n
\n\n
Helper Methods
\n
The clone() method is particularly useful for creating variations of your crawler configuration:\n
# Create a base configuration\nbase_config = CrawlerRunConfig(\n cache_mode=CacheMode.ENABLED,\n word_count_threshold=200,\n wait_until=\"networkidle\"\n)\n\n# Create variations for different use cases\nstream_config = base_config.clone(\n stream=True, # Enable streaming mode\n cache_mode=CacheMode.BYPASS\n)\n\ndebug_config = base_config.clone(\n page_timeout=120000, # Longer timeout for debugging\n verbose=True\n)\n
\nThe clone() method:\n- Creates a new instance with all the same settings\n- Updates only the specified parameters\n- Leaves the original configuration unchanged\n- Perfect for creating variations without repeating all parameters\n
3. LLMConfig Essentials
\n
Key fields to note
\n\n
provider:
\n
Which LLM provider to use.
\n
Possible values are \"ollama/llama3\",\"groq/llama3-70b-8192\",\"groq/llama3-8b-8192\", \"openai/gpt-4o-mini\" ,\"openai/gpt-4o\",\"openai/o1-mini\",\"openai/o1-preview\",\"openai/o3-mini\",\"openai/o3-mini-high\",\"anthropic/claude-3-haiku-20240307\",\"anthropic/claude-3-opus-20240229\",\"anthropic/claude-3-sonnet-20240229\",\"anthropic/claude-3-5-sonnet-20240620\",\"gemini/gemini-pro\",\"gemini/gemini-1.5-pro\",\"gemini/gemini-2.0-flash\",\"gemini/gemini-2.0-flash-exp\",\"gemini/gemini-2.0-flash-lite-preview-02-05\",\"deepseek/deepseek-chat\" (default: \"openai/gpt-4o-mini\")
\n
api_token:
\n
Optional. When not provided explicitly, api_token will be read from environment variables based on provider. For example: If a gemini model is passed as provider then,\"GEMINI_API_KEY\" will be read from environment variables
\n
API token of LLM provider eg: api_token = \"gsk_1ClHGGJ7Lpn4WGybR7vNWGdyb3FY7zXEw3SCiy0BAVM9lL8CQv\"
\n
Environment variable - use with prefix \"env:\" eg:api_token = \"env: GROQ_API_KEY\"
\n
\n
\n
base_url:
\n
\n
If your provider has a custom endpoint
\n
\n
\n
Backoff controls(optional):
\n
\n
backoff_base_delay(default 2 seconds) â how long to pause before the first retry if the provider rate-limits you.
\n
backoff_max_attempts(default 3) â total tries for the same prompt (initial call + retries).
\n
backoff_exponential_factor(default 2) â how quickly the pause grows between retries. A factor of 2 yields waits like 2s â 4s â 8s.
\n
Because these plug into Crawl4AIâs retry helper, every LLM strategy automatically follows the pacing you define here.\n
In a typical scenario, you define oneBrowserConfig for your crawler session, then create one or moreCrawlerRunConfig & LLMConfig depending on each call's needs:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig, LLMContentFilter, DefaultMarkdownGenerator\nfrom crawl4ai import JsonCssExtractionStrategy\n\nasyncdefmain():\n # 1) Browser config: headless, bigger viewport, no proxy\n browser_conf = BrowserConfig(\n headless=True,\n viewport_width=1280,\n viewport_height=720\n )\n\n # 2) Example extraction strategy\n schema = {\n \"name\": \"Articles\",\n \"baseSelector\": \"div.article\",\n \"fields\": [\n {\"name\": \"title\", \"selector\": \"h2\", \"type\": \"text\"},\n {\"name\": \"link\", \"selector\": \"a\", \"type\": \"attribute\", \"attribute\": \"href\"}\n ]\n }\n extraction = JsonCssExtractionStrategy(schema)\n\n # 3) Example LLM content filtering\n\n gemini_config = LLMConfig(\n provider=\"gemini/gemini-1.5-pro\", \n api_token = \"env:GEMINI_API_TOKEN\"\n )\n\n # Initialize LLM filter with specific instruction\n filter = LLMContentFilter(\n llm_config=gemini_config, # or your preferred provider\n instruction=\"\"\"\n Focus on extracting the core educational content.\n Include:\n - Key concepts and explanations\n - Important code examples\n - Essential technical details\n Exclude:\n - Navigation elements\n - Sidebars\n - Footer content\n Format the output as clean markdown with proper code blocks and headers.\n \"\"\",\n chunk_token_threshold=500, # Adjust based on your needs\n verbose=True\n )\n\n md_generator = DefaultMarkdownGenerator(\n content_filter=filter,\n options={\"ignore_links\": True}\n )\n\n # 4) Crawler run config: skip cache, use extraction\n run_conf = CrawlerRunConfig(\n markdown_generator=md_generator,\n extraction_strategy=extraction,\n cache_mode=CacheMode.BYPASS,\n )\n\n asyncwith AsyncWebCrawler(config=browser_conf) as crawler:\n # 4) Execute the crawl\n result = await crawler.arun(url=\"https://example.com/news\", config=run_conf)\n\n if result.success:\n print(\"Extracted content:\", result.extracted_content)\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
Which browser engine to use. \"chromium\" is typical for many sites, \"firefox\" or \"webkit\" for specialized tests.
\n
\n
\n
headless
\n
bool (default: True)
\n
Headless means no visible UI. False is handy for debugging.
\n
\n
\n
viewport_width
\n
int (default: 1080)
\n
Initial page width (in px). Useful for testing responsive layouts.
\n
\n
\n
viewport_height
\n
int (default: 600)
\n
Initial page height (in px).
\n
\n
\n
proxy
\n
str (deprecated)
\n
Deprecated. Use proxy_config instead. If set, it will be auto-converted internally.
\n
\n
\n
proxy_config
\n
dict (default: None)
\n
For advanced or multi-proxy needs, specify details like {\"server\": \"...\", \"username\": \"...\", ...}.
\n
\n
\n
use_persistent_context
\n
bool (default: False)
\n
If True, uses a persistent browser context (keep cookies, sessions across runs). Also sets use_managed_browser=True.
\n
\n
\n
user_data_dir
\n
str or None (default: None)
\n
Directory to store user data (profiles, cookies). Must be set if you want permanent sessions.
\n
\n
\n
ignore_https_errors
\n
bool (default: True)
\n
If True, continues despite invalid certificates (common in dev/staging).
\n
\n
\n
java_script_enabled
\n
bool (default: True)
\n
Disable if you want no JS overhead, or if only static content is needed.
\n
\n
\n
cookies
\n
list (default: [])
\n
Pre-set cookies, each a dict like {\"name\": \"session\", \"value\": \"...\", \"url\": \"...\"}.
\n
\n
\n
headers
\n
dict (default: {})
\n
Extra HTTP headers for every request, e.g. {\"Accept-Language\": \"en-US\"}.
\n
\n
\n
user_agent
\n
str (default: Chrome-based UA)
\n
Your custom or random user agent. user_agent_mode=\"random\" can shuffle it.
\n
\n
\n
light_mode
\n
bool (default: False)
\n
Disables some background features for performance gains.
\n
\n
\n
text_mode
\n
bool (default: False)
\n
If True, tries to disable images/other heavy content for speed.
\n
\n
\n
use_managed_browser
\n
bool (default: False)
\n
For advanced âmanagedâ interactions (debugging, CDP usage). Typically set automatically if persistent context is on.
\n
\n
\n
extra_args
\n
list (default: [])
\n
Additional flags for the underlying browser process, e.g. [\"--disable-extensions\"].
\n
\n
\n
- Set headless=False to visually debug how pages load or how interactions proceed.
\n
\n
\n
\n
\n
- If you need authentication storage or repeated sessions, consider use_persistent_context=True and specify user_data_dir.
\n
\n
\n
\n
\n
- For large pages, you might need a bigger viewport_width and viewport_height to handle dynamic content.
\n
\n
\n
\n
\n
# 2.âCrawlerRunConfig â Controlling Each Crawl
\n
\n
\n
\n
\n
While BrowserConfig sets up the environment, CrawlerRunConfig details how each crawl operation should behave: caching, content filtering, link or domain blocking, timeouts, JavaScript code, etc.
\n
\n
\n
\n
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nrun_cfg = CrawlerRunConfig(\n wait_for=\"css:.main-content\",\n word_count_threshold=15,\n excluded_tags=[\"nav\", \"footer\"],\n exclude_external_links=True,\n stream=True, # Enable streaming for arun_many()\n)\n
Skips text blocks below X words. Helps ignore trivial sections.
\n
\n
\n
extraction_strategy
\n
ExtractionStrategy (default: None)
\n
If set, extracts structured data (CSS-based, LLM-based, etc.).
\n
\n
\n
markdown_generator
\n
MarkdownGenerationStrategy (None)
\n
If you want specialized markdown output (citations, filtering, chunking, etc.). Can be customized with options such as content_source parameter to select the HTML input source ('cleaned_html', 'raw_html', or 'fit_html').
\n
\n
\n
css_selector
\n
str (None)
\n
Retains only the part of the page matching this selector. Affects the entire extraction process.
\n
\n
\n
target_elements
\n
List[str] (None)
\n
List of CSS selectors for elements to focus on for markdown generation and data extraction, while still processing the entire page for links, media, etc. Provides more flexibility than css_selector.
JavaScript to run afterwait_for and delay_before_return_html, on the fully-loaded page. E.g. \"document.querySelector('button')?.click();\".
\n
\n
\n
js_code_before_wait
\n
str or list[str] (None)
\n
JavaScript to run beforewait_for. Use for triggering loading that wait_for then checks.
\n
\n
\n
js_only
\n
bool (False)
\n
If True, indicates we're reusing an existing session and only applying JS. No full reload.
\n
\n
\n
ignore_body_visibility
\n
bool (True)
\n
Skip checking if <body> is visible. Usually best to keep True.
\n
\n
\n
scan_full_page
\n
bool (False)
\n
If True, auto-scroll the page to load dynamic content (infinite scroll).
\n
\n
\n
scroll_delay
\n
float (0.2)
\n
Delay between scroll steps when scanning the full page (scan_full_page=True) or capturing full-page screenshots.
\n
\n
\n
process_iframes
\n
bool (False)
\n
Inlines iframe content for single-page extraction.
\n
\n
\n
flatten_shadow_dom
\n
bool (False)
\n
Flattens Shadow DOM content into the light DOM before HTML capture. Resolves slots, strips shadow-scoped styles, and force-opens closed shadow roots. Essential for sites built with Web Components.
\n
\n
\n
remove_overlay_elements
\n
bool (False)
\n
Removes potential modals/popups blocking the main content.
\n
\n
\n
remove_consent_popups
\n
bool (False)
\n
Removes GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, TrustArc, Quantcast, Didomi, Sourcepoint, FundingChoices, etc.). Tries clicking \"Accept All\" first, then falls back to DOM removal.
\n
\n
\n
simulate_user
\n
bool (False)
\n
Simulate user interactions (mouse movements) to avoid bot detection.
\n
\n
\n
override_navigator
\n
bool (False)
\n
Override navigator properties in JS for stealth.
\n
\n
\n
magic
\n
bool (False)
\n
Automatic handling of popups/consent banners. Experimental.
\n
\n
\n
adjust_viewport_to_content
\n
bool (False)
\n
Resizes viewport to match page content height.
\n
\n
\n
If your page is a single-page app with repeated JS updates, set js_only=True in subsequent calls, plus a session_id for reusing the same tab.
Configuration for handling virtualized scrolling on sites like Twitter/Instagram where content is replaced rather than appended.
\n
\n
\n
When sites use virtual scrolling (content replaced as you scroll), use VirtualScrollConfig:
\n
\n
\n
\n
\n
from crawl4ai import VirtualScrollConfig\n\nvirtual_config = VirtualScrollConfig(\n container_selector=\"#timeline\", # CSS selector for scrollable container\n scroll_count=30, # Number of times to scroll\n scroll_by=\"container_height\", # How much to scroll: \"container_height\", \"page_height\", or pixels (e.g. 500)\n wait_after_scroll=0.5 # Seconds to wait after each scroll for content to load\n)\n\nconfig = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n)\n
Pattern(s) to match URLs against. Can be: string (glob), function, or list of mixed types. None means match ALL URLs
\n
\n
\n
match_mode
\n
MatchMode (MatchMode.OR)
\n
How to combine multiple matchers in a list: MatchMode.OR (any match) or MatchMode.AND (all must match)
\n
\n
\n
The url_matcher parameter enables URL-specific configurations when used with arun_many():
\n
\n
\n
\n
\n
from crawl4ai import CrawlerRunConfig, MatchMode\nfrom crawl4ai.processors.pdf import PDFContentScrapingStrategy\nfrom crawl4ai.extraction_strategy import JsonCssExtractionStrategy\n\n# Simple string pattern (glob-style)\npdf_config = CrawlerRunConfig(\n url_matcher=\"*.pdf\",\n scraping_strategy=PDFContentScrapingStrategy()\n)\n\n# Multiple patterns with OR logic (default)\nblog_config = CrawlerRunConfig(\n url_matcher=[\"*/blog/*\", \"*/article/*\", \"*/news/*\"],\n match_mode=MatchMode.OR # Any pattern matches\n)\n\n# Function matcher\napi_config = CrawlerRunConfig(\n url_matcher=lambda url: 'api'in url or url.endswith('.json'),\n # Other settings like extraction_strategy\n)\n\n# Mixed: String + Function with AND logic\ncomplex_config = CrawlerRunConfig(\n url_matcher=[\n lambda url: url.startswith('https://'), # Must be HTTPS\n \"*.org/*\", # Must be .org domain\n lambda url: 'docs'in url # Must contain 'docs'\n ],\n match_mode=MatchMode.AND # ALL conditions must match\n)\n\n# Combined patterns and functions with AND logic\nsecure_docs = CrawlerRunConfig(\n url_matcher=[\"https://*\", lambda url: '.doc'in url],\n match_mode=MatchMode.AND # Must be HTTPS AND contain .doc\n)\n\n# Default config - matches ALL URLs\ndefault_config = CrawlerRunConfig() # No url_matcher = matches everything\n
\n
\n
\n
\n
\n
UrlMatcher Types:
\n
\n
\n
\n
\n
- None (default): When url_matcher is None or not set, the config matches ALL URLs
\n
\n
\n
\n
\n
- String patterns: Glob-style patterns like \"*.pdf\", \"*/api/*\", \"https://*.example.com/*\"
- Lists: Mix strings and functions, combined with MatchMode.OR or MatchMode.AND
\n
\n
\n
\n
\n
Important Behavior:
\n
\n
\n
\n
\n
- When passing a list of configs to arun_many(), URLs are matched against each config's url_matcher in order. First match wins!
\n
\n
\n
\n
\n
- If no config matches a URL and there's no default config (one without url_matcher), the URL will fail with \"No matching configuration found\"
\n
\n
\n
\n
\n
Both BrowserConfig and CrawlerRunConfig provide a clone() method to create modified copies:
\n
\n
\n
\n
\n
# Create a base configuration\nbase_config = CrawlerRunConfig(\n cache_mode=CacheMode.ENABLED,\n word_count_threshold=200\n)\n\n# Create variations using clone()\nstream_config = base_config.clone(stream=True)\nno_cache_config = base_config.clone(\n cache_mode=CacheMode.BYPASS,\n stream=True\n)\n
\n
\n
\n
\n
\n
The clone() method is particularly useful when you need slightly different configurations for different use cases, without modifying the original config.
1.Optional. When not provided explicitly, api_token will be read from environment variables based on provider. For example: If a gemini model is passed as provider then,\"GEMINI_API_KEY\" will be read from environment variables 2. API token of LLM provider eg: api_token = \"gsk_1ClHGGJ7Lpn4WGybR7vNWGdyb3FY7zXEw3SCiy0BAVM9lL8CQv\" 3. Environment variable - use with prefix \"env:\" eg:api_token = \"env: GROQ_API_KEY\"
- UseBrowserConfig for global browser settings: engine, headless, proxy, user agent.
\n
\n
\n
\n
\n
- UseCrawlerRunConfig for each crawlâs context: how to filter content, handle caching, wait for dynamic elements, or run JS.
\n
\n
\n
\n
\n
- Pass both configs to AsyncWebCrawler (the BrowserConfig) and then to arun() (the CrawlerRunConfig).
\n
\n
\n
\n
\n
- UseLLMConfig for LLM provider configurations that can be used across all extraction, filtering, schema generation tasks. Can be used in - LLMExtractionStrategy, LLMContentFilter, JsonCssExtractionStrategy.generate_schema & JsonXPathExtractionStrategy.generate_schema
\n
\n
\n
\n
\n
# Create a modified copywith the clone() method\nstream_cfg = run_cfg.clone(\n stream=True,\n cache_mode=CacheMode.BYPASS\n)\n
\n
\n
\n
\n\n
\n
Crawling Patterns
\n
Simple Crawling
\n
Basic Usage
\n
Set up a simple crawl using BrowserConfig and CrawlerRunConfig:\n
The arun() method returns a CrawlResult object with several useful properties. Here's a quick overview (see CrawlResult for complete details):\n
config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.6),\n options={\"ignore_links\": True}\n )\n)\n\nresult = await crawler.arun(\n url=\"https://example.com\",\n config=config\n)\n\n# Different content formats\nprint(result.html) # Raw HTML\nprint(result.cleaned_html) # Cleaned HTML\nprint(result.markdown.raw_markdown) # Raw markdown from cleaned html\nprint(result.markdown.fit_markdown) # Most relevant content in markdown\n\n# Check success status\nprint(result.success) # True if crawl succeeded\nprint(result.status_code) # HTTP status code (e.g., 200, 404)\n\n# Access extracted media and links\nprint(result.media) # Dictionary of found media (images, videos, audio)\nprint(result.links) # Dictionary of internal and external links\n
\n
Adding Basic Options
\n
Customize your crawl using CrawlerRunConfig:\n
run_config = CrawlerRunConfig(\n word_count_threshold=10, # Minimum words per content block\n exclude_external_links=True, # Remove external links\n remove_overlay_elements=True, # Remove popups/modals\n process_iframes=True# Process iframe content\n)\n\nresult = await crawler.arun(\n url=\"https://example.com\",\n config=run_config\n)\n
browser_config = BrowserConfig(verbose=True)\n\nasyncwithAsyncWebCrawler(config=browser_config) as crawler:\n run_config = CrawlerRunConfig()\n result = await crawler.arun(url=\"https://example.com\", config=run_config)\n
\n
Complete Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\nfrom crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n browser_config = BrowserConfig(verbose=True)\n run_config = CrawlerRunConfig(\n # Content filtering\n word_count_threshold=10,\n excluded_tags=['form', 'header'],\n exclude_external_links=True,\n\n # Content processing\n process_iframes=True,\n remove_overlay_elements=True,\n\n # Cache control\n cache_mode=CacheMode.ENABLED # Use cache if available\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=run_config\n )\n\n if result.success:\n # Print clean content\n print(\"Content:\", result.markdown[:500]) # First 500 chars\n\n # Process images\n for image in result.media[\"images\"]:\n print(f\"Found image: {image['src']}\")\n\n # Process links\n for link in result.links[\"internal\"]:\n print(f\"Internal link: {link['href']}\")\n\n else:\n print(f\"Crawl failed: {result.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Content Processing
\n
Markdown Generation Basics
\n\n
How to configure the Default Markdown Generator
\n
The difference between raw markdown (result.markdown) and filtered markdown (fit_markdown)
\n
\n
You know how to configure CrawlerRunConfig.
\n
\n
\n
\n\n
1. Quick Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefmain():\n config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator()\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n\n if result.success:\n print(\"Raw Markdown Output:\\n\")\n print(result.markdown) # The unfiltered markdown from the page\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- CrawlerRunConfig( markdown_generator = DefaultMarkdownGenerator() ) instructs Crawl4AI to convert the final HTML into markdown at the end of each crawl. \n- The resulting markdown is accessible via result.markdown.\n
2. How Markdown Generation Works
\n
2.1 HTML-to-Text Conversion (Forked & Modified)
\n
\n
Preserves headings, code blocks, bullet points, etc.
\n
Removes extraneous tags (scripts, styles) that donât add meaningful content.
\n
Can optionally generate references for links or skip them altogether.
\n
\n
2.2 Link Citations & References
\n
By default, the generator can convert <a href=\"...\"> elements into [text][1] citations, then place the actual links at the bottom of the document. This is handy for research workflows that demand references in a structured manner.
\n
2.3 Optional Content Filters
\n
3. Configuring the Default Markdown Generator
\n
You can tweak the output by passing an options dict to DefaultMarkdownGenerator. For example:\n
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Example: ignore all links, don't escape HTML, and wrap text at 80 characters\n md_generator = DefaultMarkdownGenerator(\n options={\n \"ignore_links\": True,\n \"escape_html\": False,\n \"body_width\": 80\n }\n )\n\n config = CrawlerRunConfig(\n markdown_generator=md_generator\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com/docs\", config=config)\n if result.success:\n print(\"Markdown:\\n\", result.markdown[:500]) # Just a snippet\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n import asyncio\n asyncio.run(main())\n
\nSome commonly used options:\n- ignore_links (bool): Whether to remove all hyperlinks in the final markdown. \n- ignore_images (bool): Remove all ![image]() references. \n- escape_html (bool): Turn HTML entities into text (default is often True). \n- body_width (int): Wrap text at N characters. 0 or None means no wrapping. \n- skip_internal_links (bool): If True, omit #localAnchors or internal links referencing the same page. \n- include_sup_sub (bool): Attempt to handle <sup> / <sub> in a more readable way.\n
4. Selecting the HTML Source for Markdown Generation
\n
The content_source parameter allows you to control which HTML content is used as input for markdown generation. This gives you flexibility in how the HTML is processed before conversion to markdown.\n
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Option 1: Use the raw HTML directly from the webpage (before any processing)\n raw_md_generator = DefaultMarkdownGenerator(\n content_source=\"raw_html\",\n options={\"ignore_links\": True}\n )\n\n # Option 2: Use the cleaned HTML (after scraping strategy processing - default)\n cleaned_md_generator = DefaultMarkdownGenerator(\n content_source=\"cleaned_html\", # This is the default\n options={\"ignore_links\": True}\n )\n\n # Option 3: Use preprocessed HTML optimized for schema extraction\n fit_md_generator = DefaultMarkdownGenerator(\n content_source=\"fit_html\",\n options={\"ignore_links\": True}\n )\n\n # Use one of the generators in your crawler config\n config = CrawlerRunConfig(\n markdown_generator=raw_md_generator # Try each of the generators\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n if result.success:\n print(\"Markdown:\\n\", result.markdown.raw_markdown[:500])\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n import asyncio\n asyncio.run(main())\n
\n
HTML Source Options
\n
\n
\"cleaned_html\" (default): Uses the HTML after it has been processed by the scraping strategy. This HTML is typically cleaner and more focused on content, with some boilerplate removed.
\n
\"raw_html\": Uses the original HTML directly from the webpage, before any cleaning or processing. This preserves more of the original content, but may include navigation bars, ads, footers, and other elements that might not be relevant to the main content.
\n
\"fit_html\": Uses HTML preprocessed for schema extraction. This HTML is optimized for structured data extraction and may have certain elements simplified or removed.
\n
\n
When to Use Each Option
\n
\n
Use \"cleaned_html\" (default) for most cases where you want a balance of content preservation and noise removal.
\n
Use \"raw_html\" when you need to preserve all original content, or when the cleaning process is removing content you actually want to keep.
\n
Use \"fit_html\" when working with structured data or when you need HTML that's optimized for schema extraction.
\n- user_query: The term you want to focus on. BM25 tries to keep only content blocks relevant to that query. \n- bm25_threshold: Raise it to keep fewer blocks; lower it to keep more. \n- use_stemming(default True): Whether to apply stemming to the query and content.\n- language (str): Language for stemming (default: 'english').\n
5.2 PruningContentFilter
\n
If you donât have a specific query, or if you just want a robust âjunk remover,â use PruningContentFilter. It analyzes text density, link density, HTML structure, and known patterns (like ânav,â âfooterâ) to systematically prune extraneous or repetitive sections.\n
from crawl4ai.content_filter_strategy import PruningContentFilter\n\nprune_filter = PruningContentFilter(\n threshold=0.5,\n threshold_type=\"fixed\", # or \"dynamic\"\n min_word_threshold=50\n)\n
\n- threshold: Score boundary. Blocks below this score get removed. \n- threshold_type: \n - \"fixed\": Straight comparison (score >= threshold keeps the block). \n - \"dynamic\": The filter adjusts threshold in a data-driven manner. \n- min_word_threshold: Discard blocks under N words as likely too short or unhelpful.\n- You want a broad cleanup without a user query. \n
5.3 LLMContentFilter
\n
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, LLMConfig, DefaultMarkdownGenerator\nfrom crawl4ai.content_filter_strategy import LLMContentFilter\n\nasyncdefmain():\n # Initialize LLM filter with specific instruction\n filter = LLMContentFilter(\n llm_config = LLMConfig(provider=\"openai/gpt-4o\",api_token=\"your-api-token\"), #or use environment variable\n instruction=\"\"\"\n Focus on extracting the core educational content.\n Include:\n - Key concepts and explanations\n - Important code examples\n - Essential technical details\n Exclude:\n - Navigation elements\n - Sidebars\n - Footer content\n Format the output as clean markdown with proper code blocks and headers.\n \"\"\",\n chunk_token_threshold=4096, # Adjust based on your needs\n verbose=True\n )\n md_generator = DefaultMarkdownGenerator(\n content_filter=filter,\n options={\"ignore_links\": True}\n )\n config = CrawlerRunConfig(\n markdown_generator=md_generator,\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n print(result.markdown.fit_markdown) # Filtered markdown content\n
\n- Chunk Processing: Handles large documents by processing them in chunks (controlled by chunk_token_threshold)\n- Parallel Processing: For better performance, use smaller chunk_token_threshold (e.g., 2048 or 4096) to enable parallel processing of content chunks\n1. Exact Content Preservation:\n
filter = LLMContentFilter(\n instruction=\"\"\"\n Extract the main educational content while preserving its original wording and substance completely.\n 1. Maintain the exact language and terminology\n 2. Keep all technical explanations and examples intact\n 3. Preserve the original flow and structure\n 4. Remove only clearly irrelevant elements like navigation menus and ads\n \"\"\",\n chunk_token_threshold=4096\n)\n
\n2. Focused Content Extraction:\n
filter = LLMContentFilter(\n instruction=\"\"\"\n Focus on extracting specific types of content:\n - Technical documentation\n - Code examples\n - API references\n Reformat the content into clear, well-structured markdown\n \"\"\",\n chunk_token_threshold=4096\n)\n
\n
\n
Performance Tip: Set a smaller chunk_token_threshold (e.g., 2048 or 4096) to enable parallel processing of content chunks. The default value is infinity, which processes the entire content as a single chunk.
\n
\n
6. Using Fit Markdown
\n
When a content filter is active, the library produces two forms of markdown inside result.markdown:\n1.âraw_markdown: The full unfiltered markdown. \n2.âfit_markdown: A âfitâ version where the filter has removed or trimmed noisy segments.\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\n\nasyncdefmain():\n config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.6),\n options={\"ignore_links\": True}\n )\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://news.example.com/tech\", config=config)\n if result.success:\n print(\"Raw markdown:\\n\", result.markdown)\n\n # If a filter is used, we also have .fit_markdown:\n md_object = result.markdown # or your equivalent\n print(\"Filtered markdown:\\n\", md_object.fit_markdown)\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
7. The MarkdownGenerationResult Object
\n
If your library stores detailed markdown output in an object like MarkdownGenerationResult, youâll see fields such as:\n- raw_markdown: The direct HTML-to-markdown transformation (no filtering). \n- markdown_with_citations: A version that moves links to reference-style footnotes. \n- references_markdown: A separate string or section containing the gathered references. \n- fit_markdown: The filtered markdown if you used a content filter. \n- fit_html: The corresponding HTML snippet used to generate fit_markdown (helpful for debugging or advanced usage).\n
md_obj = result.markdown # your libraryâs naming may vary\nprint(\"RAW:\\n\", md_obj.raw_markdown)\nprint(\"CITED:\\n\", md_obj.markdown_with_citations)\nprint(\"REFERENCES:\\n\", md_obj.references_markdown)\nprint(\"FIT:\\n\", md_obj.fit_markdown)\n
\n- You can supply raw_markdown to an LLM if you want the entire text. \n- Or feed fit_markdown into a vector database to reduce token usage. \n- references_markdown can help you keep track of link provenance.\n
8. Combining Filters (BM25 + Pruning) in Two Passes
\n
You might want to prune out noisy boilerplate first (with PruningContentFilter), and then rank whatâs left against a user query (with BM25ContentFilter). You donât have to crawl the page twice. Instead:\n1.âFirst pass: Apply PruningContentFilter directly to the raw HTML from result.html (the crawlerâs downloaded HTML). \n2.âSecond pass: Take the pruned HTML (or text) from step 1, and feed it into BM25ContentFilter, focusing on a user query.
\n
Two-Pass Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter\nfrom bs4 import BeautifulSoup\n\nasyncdefmain():\n # 1. Crawl with minimal or no markdown generator, just get raw HTML\n config = CrawlerRunConfig(\n # If you only want raw HTML, you can skip passing a markdown_generator\n # or provide one but focus on .html in this example\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com/tech-article\", config=config)\n\n ifnot result.success ornot result.html:\n print(\"Crawl failed or no HTML content.\")\n return\n\n raw_html = result.html\n\n # 2. First pass: PruningContentFilter on raw HTML\n pruning_filter = PruningContentFilter(threshold=0.5, min_word_threshold=50)\n\n # filter_content returns a list of \"text chunks\" or cleaned HTML sections\n pruned_chunks = pruning_filter.filter_content(raw_html)\n # This list is basically pruned content blocks, presumably in HTML or text form\n\n # For demonstration, let's combine these chunks back into a single HTML-like string\n # or you could do further processing. It's up to your pipeline design.\n pruned_html = \"\\n\".join(pruned_chunks)\n\n # 3. Second pass: BM25ContentFilter with a user query\n bm25_filter = BM25ContentFilter(\n user_query=\"machine learning\",\n bm25_threshold=1.2,\n language=\"english\"\n )\n\n # returns a list of text chunks\n bm25_chunks = bm25_filter.filter_content(pruned_html) \n\n ifnot bm25_chunks:\n print(\"Nothing matched the BM25 query after pruning.\")\n return\n\n # 4. Combine or display final results\n final_text = \"\\n---\\n\".join(bm25_chunks)\n\n print(\"==== PRUNED OUTPUT (first pass) ====\")\n print(pruned_html[:500], \"... (truncated)\") # preview\n\n print(\"\\n==== BM25 OUTPUT (second pass) ====\")\n print(final_text[:500], \"... (truncated)\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Whatâs Happening?
\n
1.âRaw HTML: We crawl once and store the raw HTML in result.html. \n4.âBM25ContentFilter: We feed the pruned string into BM25ContentFilter with a user query. This second pass further narrows the content to chunks relevant to âmachine learning.â\nNo Re-Crawling: We used raw_html from the first pass, so thereâs no need to run arun() againâno second network request.
\n
Tips & Variations
\n
\n
Plain Text vs. HTML: If your pruned output is mostly text, BM25 can still handle it; just keep in mind it expects a valid string input. If you supply partial HTML (like \"<p>some text</p>\"), it will parse it as HTML.
\n
Adjust Thresholds: If you see too much or too little text in step one, tweak threshold=0.5 or min_word_threshold=50. Similarly, bm25_threshold=1.2 can be raised/lowered for more or fewer chunks in step two.
\n
\n
One-Pass Combination?
\n
9. Common Pitfalls & Tips
\n
1.âNo Markdown Output? \n2.âPerformance Considerations \n - Very large pages with multiple filters can be slower. Consider cache_mode to avoid re-downloading. \n3.âTake Advantage of fit_markdown \n4.âAdjusting html2text Options \n - If you see lots of raw HTML slipping into the text, turn on escape_html. \n - If code blocks look messy, experiment with mark_code or handle_code_in_pre.
\n
10. Summary & Next Steps
\n
\n
Configure the DefaultMarkdownGenerator with HTML-to-text options.
\n
Select different HTML sources using the content_source parameter.
\n
Distinguish between raw and filtered markdown (fit_markdown).
\n
Leverage the MarkdownGenerationResult object to handle different forms of output (citations, references, etc.).
\n
\n
Fit Markdown with Pruning & BM25
\n
1. How âFit Markdownâ Works
\n
1.1 The content_filter
\n
In CrawlerRunConfig, you can specify a content_filter to shape how content is pruned or ranked before final markdown generation. A filterâs logic is applied before or during the HTMLâMarkdown process, producing:\n- result.markdown.raw_markdown (unfiltered)\n- result.markdown.fit_markdown (filtered or âfitâ version)\n- result.markdown.fit_html (the corresponding HTML snippet that produced fit_markdown)
\n
1.2 Common Filters
\n
2. PruningContentFilter
\n
2.1 Usage Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefmain():\n # Step 1: Create a pruning filter\n prune_filter = PruningContentFilter(\n # Lower â more content retained, higher â more content pruned\n threshold=0.45, \n # \"fixed\" or \"dynamic\"\n threshold_type=\"dynamic\", \n # Ignore nodes with <5 words\n min_word_threshold=5 \n )\n\n # Step 2: Insert it into a Markdown Generator\n md_generator = DefaultMarkdownGenerator(content_filter=prune_filter)\n\n # Step 3: Pass it to CrawlerRunConfig\n config = CrawlerRunConfig(\n markdown_generator=md_generator\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", \n config=config\n )\n\n if result.success:\n # 'fit_markdown' is your pruned content, focusing on \"denser\" text\n print(\"Raw Markdown length:\", len(result.markdown.raw_markdown))\n print(\"Fit Markdown length:\", len(result.markdown.fit_markdown))\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
2.2 Key Parameters
\n
\n
min_word_threshold (int): If a block has fewer words than this, itâs pruned.
\n
threshold_type (str):
\n
\"fixed\" â each node must exceed threshold (0â1).
\n
\"dynamic\" â node scoring adjusts according to tag type, text/link density, etc.
\n
threshold (float, default ~0.48): The base or âanchorâ cutoff.
\n
Link density â Penalizes sections that are mostly links.
\n
Tag importance â e.g., an <article> or <p> might be more important than a <div>.
\n
\n
3. BM25ContentFilter
\n
3.1 Usage Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_filter_strategy import BM25ContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefmain():\n # 1) A BM25 filter with a user query\n bm25_filter = BM25ContentFilter(\n user_query=\"startup fundraising tips\",\n # Adjust for stricter or looser results\n bm25_threshold=1.2 \n )\n\n # 2) Insert into a Markdown Generator\n md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)\n\n # 3) Pass to crawler config\n config = CrawlerRunConfig(\n markdown_generator=md_generator\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", \n config=config\n )\n if result.success:\n print(\"Fit Markdown (BM25 query-based):\")\n print(result.markdown.fit_markdown)\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
3.2 Parameters
\n
\n
user_query (str, optional): E.g. \"machine learning\". If blank, the filter tries to glean a query from page metadata.
\n
bm25_threshold (float, default 1.0):
\n
Higher â fewer chunks but more relevant.
\n
Lower â more inclusive.
\n
In more advanced scenarios, you might see parameters like language, case_sensitive, or priority_tags to refine how text is tokenized or weighted.
\n
\n
\n
\n
4. Accessing the âFitâ Output
\n
After the crawl, your âfitâ content is found in result.markdown.fit_markdown. \n
\nIf the content filter is BM25, you might see additional logic or references in fit_markdown that highlight relevant segments. If itâs Pruning, the text is typically well-cleaned but not necessarily matched to a query.\n
\n1. The crawlerâs excluded_tags are removed from the HTML first. \n3. The final âfitâ content is generated in result.markdown.fit_markdown.\n
7. Custom Filters
\n
If you need a different approach (like a specialized ML model or site-specific heuristics), you can create a new class inheriting from RelevantContentFilter and implement filter_content(html). Then inject it into your markdown generator:\n
from crawl4ai.content_filter_strategy import RelevantContentFilter\n\nclassMyCustomFilter(RelevantContentFilter):\n deffilter_content(self, html, min_word_threshold=None):\n # parse HTML, implement custom logic\n return [block for block in ... if ... some condition...]\n
\n1. Subclass RelevantContentFilter. \n2. Implement filter_content(...). \n3. Use it in your DefaultMarkdownGenerator(content_filter=MyCustomFilter(...)).\n
8. Final Thoughts
\n
\n
Summaries: Quickly get the important text from a cluttered page.
\n
Search: Combine with BM25 to produce content relevant to a query.
\n
BM25ContentFilter: Perfect for query-based extraction or searching.
\n
Combine with excluded_tags, exclude_external_links, word_count_threshold to refine your final âfitâ text.
\n
Fit markdown ends up in result.markdown.fit_markdown; eventually result.markdown.fit_markdown in future versions.
\n
Last Updated: 2025-01-01
\n
\n
Content Selection
\n
Crawl4AI provides multiple ways to select, filter, and refine the content from your crawls. Whether you need to target a specific CSS region, exclude entire tags, filter out external links, or remove certain domains and images, CrawlerRunConfig offers a wide range of parameters.
\n
1. CSS-Based Selection
\n
There are two ways to select content from a page: using css_selector or the more flexible target_elements.
\n
1.1 Using css_selector
\n
A straightforward way to limit your crawl results to a certain region of the page is css_selector in CrawlerRunConfig:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # e.g., first 30 items from Hacker News\n css_selector=\".athing:nth-child(-n+30)\" \n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com/newest\", \n config=config\n )\n print(\"Partial HTML length:\", len(result.cleaned_html))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\nResult: Only elements matching that selector remain in result.cleaned_html.\n
1.2 Using target_elements
\n
The target_elements parameter provides more flexibility by allowing you to target multiple elements for content extraction while preserving the entire page context for other features:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # Target article body and sidebar, but not other content\n target_elements=[\"article.main-content\", \"aside.sidebar\"]\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/blog-post\", \n config=config\n )\n print(\"Markdown focused on target elements\")\n print(\"Links from entire page still available:\", len(result.links.get(\"internal\", [])))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\nKey difference: With target_elements, the markdown generation and structural data extraction focus on those elements, but other page elements (like links, images, and tables) are still extracted from the entire page. This gives you fine-grained control over what appears in your markdown content while preserving full page context for link analysis and media collection.\n
2. Content Filtering & Exclusions
\n
2.1 Basic Overview
\n
config = CrawlerRunConfig(\n # Content thresholds\n word_count_threshold=10, # Minimum words per block\n\n # Tag exclusions\n excluded_tags=['form', 'header', 'footer', 'nav'],\n\n # Link filtering\n exclude_external_links=True, \n exclude_social_media_links=True,\n # Block entire domains\n exclude_domains=[\"adtrackers.com\", \"spammynews.org\"], \n exclude_social_media_domains=[\"facebook.com\", \"twitter.com\"],\n\n # Media filtering\n exclude_external_images=True\n)\n
\n- word_count_threshold: Ignores text blocks under X words. Helps skip trivial blocks like short nav or disclaimers. \n- excluded_tags: Removes entire tags (<form>, <header>, <footer>, etc.). \n- Link Filtering: \n - exclude_external_links: Strips out external links and may remove them from result.links. \n - exclude_social_media_links: Removes links pointing to known social media domains. \n - exclude_domains: A custom list of domains to block if discovered in links. \n - exclude_social_media_domains: A curated list (override or add to it) for social media sites. \n- Media Filtering: \n - exclude_external_images: Discards images not hosted on the same domain as the main page (or its subdomains).\nBy default in case you set exclude_social_media_links=True, the following social media domains are excluded:\n
Sites built with Web Components (Stencil, Lit, Shoelace, Angular Elements, etc.) render content inside Shadow DOM â an encapsulated sub-tree invisible to page.content(). Set flatten_shadow_dom=True to extract it:\n
config = CrawlerRunConfig(\n flatten_shadow_dom=True,\n wait_until=\"load\",\n delay_before_return_html=3.0, # give components time to hydrate\n)\n
\nWhen enabled, Crawl4AI also injects an init script that force-opens closed shadow roots. The flattener resolves <slot> projections and strips shadow-scoped <style> tags, producing clean HTML for the downstream scraping/markdown pipeline.\n
Execution order: flatten_shadow_dom runs right before HTML capture, after all waits and JS execution:\n
import asyncio\nimport json\nfrom pydantic import BaseModel, Field\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\n\nclassArticleData(BaseModel):\n headline: str\n summary: str\n\nasyncdefmain():\n llm_strategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"openai/gpt-4\",api_token=\"sk-YOUR_API_KEY\")\n schema=ArticleData.schema(),\n extraction_type=\"schema\",\n instruction=\"Extract 'headline' and a short 'summary' from the content.\"\n )\n\n config = CrawlerRunConfig(\n exclude_external_links=True,\n word_count_threshold=20,\n extraction_strategy=llm_strategy\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=\"https://news.ycombinator.com\", config=config)\n article = json.loads(result.extracted_content)\n print(article)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- Filters out external links (exclude_external_links=True). \n- Ignores very short text blocks (word_count_threshold=20). \n- Passes the final HTML to your LLM strategy for an AI-driven parse.\n
\n- CSS scoping with #main-content. \n- Multiple exclude_ parameters to remove domains, external images, etc. \n- A JsonCssExtractionStrategy to parse repeated article blocks.\n
6. Scraping Modes
\n
Crawl4AI uses LXMLWebScrapingStrategy (LXML-based) as the default scraping strategy for HTML content processing. This strategy offers excellent performance, especially for large HTML documents.\nNote: For backward compatibility, WebScrapingStrategy is still available as an alias for LXMLWebScrapingStrategy.\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LXMLWebScrapingStrategy\n\nasyncdefmain():\n # Default configuration already uses LXMLWebScrapingStrategy\n config = CrawlerRunConfig()\n\n # Or explicitly specify it if desired\n config_explicit = CrawlerRunConfig(\n scraping_strategy=LXMLWebScrapingStrategy()\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\", \n config=config\n )\n
\nYou can also create your own custom scraping strategy by inheriting from ContentScrapingStrategy. The strategy must return a ScrapingResult object with the following structure:\n
from crawl4ai import ContentScrapingStrategy, ScrapingResult, MediaItem, Media, Link, Links\n\nclassCustomScrapingStrategy(ContentScrapingStrategy):\n defscrap(self, url: str, html: str, **kwargs) -> ScrapingResult:\n # Implement your custom scraping logic here\n return ScrapingResult(\n cleaned_html=\"<html>...</html>\", # Cleaned HTML content\n success=True, # Whether scraping was successful\n media=Media(\n images=[ # List of images found\n MediaItem(\n src=\"https://example.com/image.jpg\",\n alt=\"Image description\",\n desc=\"Surrounding text\",\n score=1,\n type=\"image\",\n group_id=1,\n format=\"jpg\",\n width=800\n )\n ],\n videos=[], # List of videos (same structure as images)\n audios=[] # List of audio files (same structure as images)\n ),\n links=Links(\n internal=[ # List of internal links\n Link(\n href=\"https://example.com/page\",\n text=\"Link text\",\n title=\"Link title\",\n base_domain=\"example.com\"\n )\n ],\n external=[] # List of external links (same structure)\n ),\n metadata={ # Additional metadata\n \"title\": \"Page Title\",\n \"description\": \"Page description\"\n }\n )\n\n asyncdefascrap(self, url: str, html: str, **kwargs) -> ScrapingResult:\n # For simple cases, you can use the sync version\n returnawait asyncio.to_thread(self.scrap, url, html, **kwargs)\n
\n
Performance Considerations
\n
\n
Fast processing of large HTML documents (especially >100KB)
\n
Efficient memory usage
\n
Good handling of well-formed HTML
\n
Robust table detection and extraction
\n
\n
Backward Compatibility
\n
For users upgrading from earlier versions:\n- WebScrapingStrategy is now an alias for LXMLWebScrapingStrategy\n- Existing code using WebScrapingStrategy will continue to work without modification\n- No changes are required to your existing code
\n
7. Combining CSS Selection Methods
\n
You can combine css_selector and target_elements in powerful ways to achieve fine-grained control over your output:\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n # Target specific content but preserve page context\n config = CrawlerRunConfig(\n # Focus markdown on main content and sidebar\n target_elements=[\"#main-content\", \".sidebar\"],\n\n # Global filters applied to entire page\n excluded_tags=[\"nav\", \"footer\", \"header\"],\n exclude_external_links=True,\n\n # Use basic content thresholds\n word_count_threshold=15,\n\n cache_mode=CacheMode.BYPASS\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/article\",\n config=config\n )\n\n print(f\"Content focuses on specific elements, but all links still analyzed\")\n print(f\"Internal links: {len(result.links.get('internal', []))}\")\n print(f\"External links: {len(result.links.get('external', []))}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- Links, images and other page data still give you the full context of the page\n- Content filtering still applies globally\n
8. Conclusion
\n
By mixing target_elements or css_selector scoping, content filtering parameters, and advanced extraction strategies, you can precisely choose which data to keep. Key parameters in CrawlerRunConfig for content selection include:\n1. target_elements â Array of CSS selectors to focus markdown generation and data extraction, while preserving full page context for links and media.\n2. css_selector â Basic scoping to an element or region for all extraction processes. \n3. word_count_threshold â Skip short blocks. \n4. excluded_tags â Remove entire HTML tags. \n5. exclude_external_links, exclude_social_media_links, exclude_domains â Filter out unwanted links or domains. \n6. exclude_external_images â Remove images from external sources. \n7. process_iframes â Merge iframe content if needed.
\n
Page Interaction
\n\n
Click âLoad Moreâ buttons
\n
Fill forms and submit them
\n
Wait for elements or data to appear
\n
Reuse sessions across multiple steps
\n\n
1. JavaScript Execution
\n
Basic Execution
\n
js_code in CrawlerRunConfig accepts either a single JS string or a list of JS snippets. \n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Single JS command\n config = CrawlerRunConfig(\n js_code=\"window.scrollTo(0, document.body.scrollHeight);\"\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", # Example site\n config=config\n )\n print(\"Crawled length:\", len(result.cleaned_html))\n\n # Multiple commands\n js_commands = [\n \"window.scrollTo(0, document.body.scrollHeight);\",\n # 'More' link on Hacker News\n \"document.querySelector('a.morelink')?.click();\", \n ]\n config = CrawlerRunConfig(js_code=js_commands)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", # Another pass\n config=config\n )\n print(\"After scroll+click, length:\", len(result.cleaned_html))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\nRelevant CrawlerRunConfig params:\n- js_code: A string or list of strings with JavaScript to run after the page loads.\n- js_only: If set to True on subsequent calls, indicates weâre continuing an existing session without a new full navigation. \n- session_id: If you want to keep the same page across multiple calls, specify an ID.\n
2. Wait Conditions
\n
2.1 CSS-Based Waiting
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # Wait for at least 30 items on Hacker News\n wait_for=\"css:.athing:nth-child(30)\" \n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\",\n config=config\n )\n print(\"We have at least 30 items loaded!\")\n # Rough check\n print(\"Total items in HTML:\", result.cleaned_html.count(\"athing\")) \n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- wait_for=\"css:...\": Tells the crawler to wait until that CSS selector is present.\n
2.2 JavaScript-Based Waiting
\n
For more complex conditions (e.g., waiting for content length to exceed a threshold), prefix js::\n
wait_condition = \"\"\"() => {\n const items = document.querySelectorAll('.athing');\n return items.length > 50; // Wait for at least 51 items\n}\"\"\"\n\nconfig = CrawlerRunConfig(wait_for=f\"js:{wait_condition}\")\n
\nBehind the Scenes: Crawl4AI keeps polling the JS function until it returns true or a timeout occurs.\n
3. Handling Dynamic Content
\n
3.1 Load More Example (Hacker News âMoreâ Link)
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Step 1: Load initial Hacker News page\n config = CrawlerRunConfig(\n wait_for=\"css:.athing:nth-child(30)\"# Wait for 30 items\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\",\n config=config\n )\n print(\"Initial items loaded.\")\n\n # Step 2: Let's scroll and click the \"More\" link\n load_more_js = [\n \"window.scrollTo(0, document.body.scrollHeight);\",\n # The \"More\" link at page bottom\n \"document.querySelector('a.morelink')?.click();\" \n ]\n\n next_page_conf = CrawlerRunConfig(\n js_code=load_more_js,\n wait_for=\"\"\"js:() => {\n return document.querySelectorAll('.athing').length > 30;\n }\"\"\",\n # Mark that we do not re-navigate, but run JS in the same session:\n js_only=True,\n session_id=\"hn_session\"\n )\n\n # Re-use the same crawler session\n result2 = await crawler.arun(\n url=\"https://news.ycombinator.com\", # same URL but continuing session\n config=next_page_conf\n )\n total_items = result2.cleaned_html.count(\"athing\")\n print(\"Items after load-more:\", total_items)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- session_id=\"hn_session\": Keep the same page across multiple calls to arun().\n- js_only=True: Weâre not performing a full reload, just applying JS in the existing page.\n- wait_for with js:: Wait for item count to grow beyond 30.\n
3.2 Form Interaction
\n
If the site has a search or login form, you can fill fields and submit them with js_code. For instance, if GitHub had a local search form:\n
1.âpage_timeout (ms): Overall page load or script execution time limit. \n2.âdelay_before_return_html (seconds): Wait an extra moment before capturing the final HTML. \n3.âmean_delay & max_range: If you call arun_many() with multiple URLs, these add a random pause between each request.\n
Below is a simplified script that does multiple âLoad Moreâ clicks on GitHubâs TypeScript commits page. It re-uses the same session to accumulate new commits each time. The code includes the relevant CrawlerRunConfig parameters youâd rely on.\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasyncdefmulti_page_commits():\n browser_cfg = BrowserConfig(\n headless=False, # Visible for demonstration\n verbose=True\n )\n session_id = \"github_ts_commits\"\n\n base_wait = \"\"\"js:() => {\n const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');\n return commits.length > 0;\n }\"\"\"\n\n # Step 1: Load initial commits\n config1 = CrawlerRunConfig(\n wait_for=base_wait,\n session_id=session_id,\n cache_mode=CacheMode.BYPASS,\n # Not using js_only yet since it's our first load\n )\n\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\n url=\"https://github.com/microsoft/TypeScript/commits/main\",\n config=config1\n )\n print(\"Initial commits loaded. Count:\", result.cleaned_html.count(\"commit\"))\n\n # Step 2: For subsequent pages, we run JS to click 'Next Page' if it exists\n js_next_page = \"\"\"\n const selector = 'a[data-testid=\"pagination-next-button\"]';\n const button = document.querySelector(selector);\n if (button) button.click();\n \"\"\"\n\n # Wait until new commits appear\n wait_for_more = \"\"\"js:() => {\n const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');\n if (!window.firstCommit && commits.length>0) {\n window.firstCommit = commits[0].textContent;\n return false;\n }\n // If top commit changes, we have new commits\n const topNow = commits[0]?.textContent.trim();\n return topNow && topNow !== window.firstCommit;\n }\"\"\"\n\n for page inrange(2): # let's do 2 more \"Next\" pages\n config_next = CrawlerRunConfig(\n session_id=session_id,\n js_code=js_next_page,\n wait_for=wait_for_more,\n js_only=True, # We're continuing from the open tab\n cache_mode=CacheMode.BYPASS\n )\n result2 = await crawler.arun(\n url=\"https://github.com/microsoft/TypeScript/commits/main\",\n config=config_next\n )\n print(f\"Page {page+2} commits count:\", result2.cleaned_html.count(\"commit\"))\n\n # Optionally kill session\n await crawler.crawler_strategy.kill_session(session_id)\n\nasyncdefmain():\n await multi_page_commits()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n- session_id: Keep the same page open. \n- js_code + wait_for + js_only=True: We do partial refreshes, waiting for new commits to appear. \n- cache_mode=CacheMode.BYPASS ensures we always see fresh data each step.\n
6. Combine Interaction with Extraction
\n
Once dynamic content is loaded, you can attach an extraction_strategy (like JsonCssExtractionStrategy or LLMExtractionStrategy). For example:\n
\nWhen done, check result.extracted_content for the JSON.\n
7. Relevant CrawlerRunConfig Parameters
\n
Below are the key interaction-related parameters in CrawlerRunConfig. For a full list, see Configuration Parameters.\n- js_code: JavaScript to run after initial load. \n- js_only: If True, no new page navigationâonly JS in the existing session. \n- wait_for: CSS (\"css:...\") or JS (\"js:...\") expression to wait for. \n- session_id: Reuse the same page across calls. \n- cache_mode: Whether to read/write from the cache or bypass. \n- remove_overlay_elements: Remove certain popups automatically.\n- remove_consent_popups: Remove GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, Didomi, etc.).\n- simulate_user, override_navigator, magic: Anti-bot or \"human-like\" interactions.
\n
8. Conclusion
\n
1.âExecute JavaScript for scrolling, clicks, or form filling. \n2.âWait for CSS or custom JS conditions before capturing data. \n4. Combine with structured extraction for dynamic sites.
\n
9. Virtual Scrolling
\n
For sites that use virtual scrolling (where content is replaced rather than appended as you scroll, like Twitter or Instagram), Crawl4AI provides a dedicated VirtualScrollConfig:\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig\n\nasyncdefcrawl_twitter_timeline():\n # Configure virtual scroll for Twitter-like feeds\n virtual_config = VirtualScrollConfig(\n container_selector=\"[data-testid='primaryColumn']\", # Twitter's main column\n scroll_count=30, # Scroll 30 times\n scroll_by=\"container_height\", # Scroll by container height each time\n wait_after_scroll=1.0# Wait 1 second after each scroll\n )\n\n config = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://twitter.com/search?q=AI\",\n config=config\n )\n # result.html now contains ALL tweets from the virtual scroll\n
\n
Virtual Scroll vs JavaScript Scrolling
\n
\n\n
\n
Feature
\n
Virtual Scroll
\n
JS Code Scrolling
\n
\n\n\n
\n
Use Case
\n
Content replaced during scroll
\n
Content appended or simple scroll
\n
\n
\n
Configuration
\n
VirtualScrollConfig object
\n
js_code with scroll commands
\n
\n
\n
Automatic Merging
\n
Yes - merges all unique content
\n
No - captures final state only
\n
\n
\n
Best For
\n
Twitter, Instagram, virtual tables
\n
Traditional pages, load more buttons
\n
\n\n
\n
Link & Media
\n\n
Extract links (internal, external) from crawled pages
\n
Filter or exclude specific domains (e.g., social media or custom domains)
You're crawling image-heavy pages that cause memory issues
\n
You want to focus only on text content
\n
Configure your crawler to exclude or prioritize certain images\nBelow is a revised version of the Link Extraction and Media Extraction sections that includes example data structures showing how links and media items are stored in CrawlResult. Feel free to adjust any field names or descriptions to match your actual output.
\n\n
1. Link Extraction
\n
1.1 result.links
\n
When you call arun() or arun_many() on a URL, Crawl4AI automatically extracts links and stores them in the links field of CrawlResult. By default, the crawler tries to distinguish internal links (same domain) from external links (different domains).\n
from crawl4ai import AsyncWebCrawler\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://www.example.com\")\n if result.success:\n internal_links = result.links.get(\"internal\", [])\n external_links = result.links.get(\"external\", [])\n print(f\"Found {len(internal_links)} internal links.\")\n print(f\"Found {len(internal_links)} external links.\")\n print(f\"Found {len(result.media)} media items.\")\n\n # Each link is typically a dictionary with fields like:\n # { \"href\": \"...\", \"text\": \"...\", \"title\": \"...\", \"base_domain\": \"...\" }\n if internal_links:\n print(\"Sample Internal Link:\", internal_links[0])\n else:\n print(\"Crawl failed:\", result.error_message)\n
\n- href: The raw hyperlink URL. \n- text: The link text (if any) within the <a> tag. \n- title: The title attribute of the link (if present). \n- base_domain: The domain extracted from href. Helpful for filtering or grouping by domain.\n
2. Advanced Link Head Extraction & Scoring
\n
Ever wanted to not just extract links, but also get the actual content (title, description, metadata) from those linked pages? And score them for relevance? This is exactly what Link Head Extraction does - it fetches the <head> section from each discovered link and scores them using multiple algorithms.
\n
2.1 Why Link Head Extraction?
\n\n
Fetching head content from each link (title, description, meta tags)
\n
Combining scores intelligently to give you a final relevance ranking
\n\n
2.2 Complete Working Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai import LinkPreviewConfig\n\nasyncdefextract_link_heads_example():\n \"\"\"\n Complete example showing link head extraction with scoring.\n This will crawl a documentation site and extract head content from internal links.\n \"\"\"\n\n # Configure link head extraction\n config = CrawlerRunConfig(\n # Enable link head extraction with detailed configuration\n link_preview_config=LinkPreviewConfig(\n include_internal=True, # Extract from internal links\n include_external=False, # Skip external links for this example\n max_links=10, # Limit to 10 links for demo\n concurrency=5, # Process 5 links simultaneously\n timeout=10, # 10 second timeout per link\n query=\"API documentation guide\", # Query for contextual scoring\n score_threshold=0.3, # Only include links scoring above 0.3\n verbose=True# Show detailed progress\n ),\n # Enable intrinsic scoring (URL quality, text relevance)\n score_links=True,\n # Keep output clean\n only_text=True,\n verbose=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n # Crawl a documentation site (great for testing)\n result = await crawler.arun(\"https://docs.python.org/3/\", config=config)\n\n if result.success:\n print(f\"â Successfully crawled: {result.url}\")\n print(f\"đ Page title: {result.metadata.get('title', 'No title')}\")\n\n # Access links (now enhanced with head data and scores)\n internal_links = result.links.get(\"internal\", [])\n external_links = result.links.get(\"external\", [])\n\n print(f\"\\nđ Found {len(internal_links)} internal links\")\n print(f\"đ Found {len(external_links)} external links\")\n\n # Count links with head data\n links_with_head = [link for link in internal_links \n if link.get(\"head_data\") isnotNone]\n print(f\"đ§ Links with head data extracted: {len(links_with_head)}\")\n\n # Show the top 3 scoring links\n print(f\"\\nđ Top 3 Links with Full Scoring:\")\n for i, link inenumerate(links_with_head[:3]):\n print(f\"\\n{i+1}. {link['href']}\")\n print(f\" Link Text: '{link.get('text', 'No text')[:50]}...'\")\n\n # Show all three score types\n intrinsic = link.get('intrinsic_score')\n contextual = link.get('contextual_score') \n total = link.get('total_score')\n\n if intrinsic isnotNone:\n print(f\" đ Intrinsic Score: {intrinsic:.2f}/10.0 (URL quality & context)\")\n if contextual isnotNone:\n print(f\" đŻ Contextual Score: {contextual:.3f} (BM25 relevance to query)\")\n if total isnotNone:\n print(f\" â Total Score: {total:.3f} (combined final score)\")\n\n # Show extracted head data\n head_data = link.get(\"head_data\", {})\n if head_data:\n title = head_data.get(\"title\", \"No title\")\n description = head_data.get(\"meta\", {}).get(\"description\", \"No description\")\n\n print(f\" đ° Title: {title[:60]}...\")\n if description:\n print(f\" đ Description: {description[:80]}...\")\n\n # Show extraction status\n status = link.get(\"head_extraction_status\", \"unknown\")\n print(f\" â Extraction Status: {status}\")\n else:\n print(f\"â Crawl failed: {result.error_message}\")\n\n# Run the example\nif __name__ == \"__main__\":\n asyncio.run(extract_link_heads_example())\n
\n
â Successfully crawled:https://docs.python.org/3/\nđPage title:3.13.5Documentation\nđFound53internallinks\nđFound1externallinks\nđ§ Links with head data extracted:10\n\nđTop 3 Links with Full Scoring:\n\n1.https://docs.python.org/3.15/\n Link Text:'Python 3.15 (in development)...'\n đIntrinsic Score:4.17/10.0(URLquality&context)\n đŻContextual Score:1.000(BM25relevancetoquery)\n âTotal Score:5.917(combinedfinalscore)\n đ°Title:3.15.0a0Documentation...\n đDescription:TheofficialPythondocumentation...\n â Extraction Status:valid\n
\n
2.3 Configuration Deep Dive
\n
The LinkPreviewConfig class supports these options:\n
from crawl4ai import LinkPreviewConfig\n\nlink_preview_config = LinkPreviewConfig(\n # BASIC SETTINGS\n verbose=True, # Show detailed logs (recommended for learning)\n\n # LINK FILTERING\n include_internal=True, # Include same-domain links\n include_external=True, # Include different-domain links\n max_links=50, # Maximum links to process (prevents overload)\n\n # PATTERN FILTERING\n include_patterns=[ # Only process links matching these patterns\n \"*/docs/*\", \n \"*/api/*\", \n \"*/reference/*\"\n ],\n exclude_patterns=[ # Skip links matching these patterns\n \"*/login*\",\n \"*/admin*\"\n ],\n\n # PERFORMANCE SETTINGS\n concurrency=10, # How many links to process simultaneously\n timeout=5, # Seconds to wait per link\n\n # RELEVANCE SCORING\n query=\"machine learning API\", # Query for BM25 contextual scoring\n score_threshold=0.3, # Only include links above this score\n)\n
\n
2.4 Understanding the Three Score Types
\n
# High intrinsic score indicators:\n# â Clean URL structure (docs.python.org/api/reference)\n# â Meaningful link text (\"API Reference Guide\")\n# â Relevant to page context\n# â Not buried deep in navigation\n\n# Low intrinsic score indicators:\n# â Random URLs (site.com/x7f9g2h)\n# â No link text or generic text (\"Click here\")\n# â Unrelated to page content\n
\nOnly available when you provide a query. Uses BM25 algorithm against head content:\n
# Example: query = \"machine learning tutorial\"\n# High contextual score: Link to \"Complete Machine Learning Guide\"\n# Low contextual score: Link to \"Privacy Policy\"\n
\n
# When both scores available: (intrinsic * 0.3) + (contextual * 0.7)\n# When only intrinsic: uses intrinsic score\n# When only contextual: uses contextual score\n# When neither: not calculated\n
\n
2.5 Practical Use Cases
\n
asyncdefresearch_assistant():\n config = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n include_internal=True,\n include_external=True,\n include_patterns=[\"*/docs/*\", \"*/tutorial/*\", \"*/guide/*\"],\n query=\"machine learning neural networks\",\n max_links=20,\n score_threshold=0.5, # Only high-relevance links\n verbose=True\n ),\n score_links=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://scikit-learn.org/\", config=config)\n\n if result.success:\n # Get high-scoring links\n good_links = [link for link in result.links.get(\"internal\", [])\n if link.get(\"total_score\", 0) > 0.7]\n\n print(f\"đŻ Found {len(good_links)} highly relevant links:\")\n for link in good_links[:5]:\n print(f\"â {link['total_score']:.3f} - {link['href']}\")\n print(f\" {link.get('head_data', {}).get('title', 'No title')}\")\n
\n
asyncdefapi_discovery():\n config = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n include_internal=True,\n include_patterns=[\"*/api/*\", \"*/reference/*\"],\n exclude_patterns=[\"*/deprecated/*\"],\n max_links=100,\n concurrency=15,\n verbose=False# Clean output\n ),\n score_links=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://docs.example-api.com/\", config=config)\n\n if result.success:\n api_links = result.links.get(\"internal\", [])\n\n # Group by endpoint type\n endpoints = {}\n for link in api_links:\n if link.get(\"head_data\"):\n title = link[\"head_data\"].get(\"title\", \"\")\n if\"GET\"in title:\n endpoints.setdefault(\"GET\", []).append(link)\n elif\"POST\"in title:\n endpoints.setdefault(\"POST\", []).append(link)\n\n for method, links in endpoints.items():\n print(f\"\\n{method} Endpoints ({len(links)}):\")\n for link in links[:3]:\n print(f\" ⢠{link['href']}\")\n
\n
asyncdefquality_analysis():\n config = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n include_internal=True,\n max_links=200,\n concurrency=20,\n ),\n score_links=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://your-website.com/\", config=config)\n\n if result.success:\n links = result.links.get(\"internal\", [])\n\n # Analyze intrinsic scores\n scores = [link.get('intrinsic_score', 0) for link in links]\n avg_score = sum(scores) / len(scores) if scores else0\n\n print(f\"đ Link Quality Analysis:\")\n print(f\" Average intrinsic score: {avg_score:.2f}/10.0\")\n print(f\" High quality links (>7.0): {len([s for s in scores if s > 7.0])}\")\n print(f\" Low quality links (<3.0): {len([s for s in scores if s < 3.0])}\")\n\n # Find problematic links\n bad_links = [link for link in links \n if link.get('intrinsic_score', 0) < 2.0]\n\n if bad_links:\n print(f\"\\nâ ď¸ Links needing attention:\")\n for link in bad_links[:5]:\n print(f\" {link['href']} (score: {link.get('intrinsic_score', 0):.1f})\")\n
\n
2.6 Performance Tips
\n\n
Start Small: Begin with max_links: 10 to understand the feature
\n
Use Patterns: Filter with include_patterns to focus on relevant sections
\n
Adjust Concurrency: Higher concurrency = faster but more resource usage
\n
Set Timeouts: Use timeout: 5 to prevent hanging on slow sites
\n
Use Score Thresholds: Filter out low-quality links with score_threshold
\n\n
2.7 Troubleshooting
\n
# Check your configuration:\nconfig = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n verbose=True# â Enable to see what's happening\n )\n)\n
\n
# Make sure scoring is enabled:\nconfig = CrawlerRunConfig(\n score_links=True, # â Enable intrinsic scoring\n link_preview_config=LinkPreviewConfig(\n query=\"your search terms\"# â For contextual scoring\n )\n)\n
Some websites contain hundreds of third-party or affiliate links. You can filter out certain domains at crawl time by configuring the crawler. The most relevant parameters in CrawlerRunConfig are:\n- exclude_external_links: If True, discard any link pointing outside the root domain. \n- exclude_social_media_domains: Provide a list of social media platforms (e.g., [\"facebook.com\", \"twitter.com\"]) to exclude from your crawl. \n- exclude_social_media_links: If True, automatically skip known social platforms. \n- exclude_domains: Provide a list of custom domains you want to exclude (e.g., [\"spammyads.com\", \"tracker.net\"]).
\n
3.1 Example: Excluding External & Social Media Links
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n crawler_cfg = CrawlerRunConfig(\n exclude_external_links=True, # No links outside primary domain\n exclude_social_media_links=True# Skip recognized social media domains\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n \"https://www.example.com\",\n config=crawler_cfg\n )\n if result.success:\n print(\"[OK] Crawled:\", result.url)\n print(\"Internal links count:\", len(result.links.get(\"internal\", [])))\n print(\"External links count:\", len(result.links.get(\"external\", []))) \n # Likely zero external links in this scenario\n else:\n print(\"[ERROR]\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
3.2 Example: Excluding Specific Domains
\n
If you want to let external links in, but specifically exclude a domain (e.g., suspiciousads.com), do this:\n
By default, Crawl4AI collects images, audio and video URLs it finds on the page. These are stored in result.media, a dictionary keyed by media type (e.g., images, videos, audio).\nNote: Tables have been moved from result.media[\"tables\"] to the new result.tables format for better organization and direct access.\n
if result.success:\n # Get images\n images_info = result.media.get(\"images\", [])\n print(f\"Found {len(images_info)} images in total.\")\n for i, img inenumerate(images_info[:3]): # Inspect just the first 3\n print(f\"[Image {i}] URL: {img['src']}\")\n print(f\" Alt text: {img.get('alt', '')}\")\n print(f\" Score: {img.get('score')}\")\n print(f\" Description: {img.get('desc', '')}\\n\")\n
\n
result.media = {\n \"images\": [\n {\n \"src\": \"https://cdn.prod.website-files.com/.../Group%2089.svg\",\n \"alt\": \"coding school for kids\",\n \"desc\": \"Trial Class Degrees degrees All Degrees AI Degree Technology ...\",\n \"score\": 3,\n \"type\": \"image\",\n \"group_id\": 0,\n \"format\": None,\n \"width\": None,\n \"height\": None\n },\n # ...\n ],\n \"videos\": [\n # Similar structure but with video-specific fields\n ],\n \"audio\": [\n # Similar structure but with audio-specific fields\n ],\n}\n
\n- src: The media URL (e.g., image source) \n- alt: The alt text for images (if present) \n- desc: A snippet of nearby text or a short description (optional) \n- score: A heuristic relevance score if youâre using content-scoring features \n- width, height: If the crawler detects dimensions for the image/video \n- type: Usually \"image\", \"video\", or \"audio\" \n- group_id: If youâre grouping related media items, the crawler might assign an ID \n
screenshot: Set to True if you want a full-page screenshot stored as base64 in result.screenshot.
\n
pdf: Set to True if you want a PDF version of the page in result.pdf.
\n
capture_mhtml: Set to True if you want an MHTML snapshot of the page in result.mhtml. This format preserves the entire web page with all its resources (CSS, images, scripts) in a single file, making it perfect for archiving or offline viewing.
\n
wait_for_images: If True, attempts to wait until images are fully loaded before final extraction.\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n crawler_cfg = CrawlerRunConfig(\n capture_mhtml=True# Enable MHTML capture\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=crawler_cfg)\n\n if result.success and result.mhtml:\n # Save the MHTML snapshot to a file\n withopen(\"example.mhtml\", \"w\", encoding=\"utf-8\") as f:\n f.write(result.mhtml)\n print(\"MHTML snapshot saved to example.mhtml\")\n else:\n print(\"Failed to capture MHTML:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
It captures the complete page state including all resources
\n
It can be opened in most modern browsers for offline viewing
\n
It preserves the page exactly as it appeared during crawling
\n
It's a single file, making it easy to store and transfer
\n
\n
5. Putting It All Together: Link & Media Filtering
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n # Suppose we want to keep only internal links, remove certain domains, \n # and discard external images from the final crawl data.\n crawler_cfg = CrawlerRunConfig(\n exclude_external_links=True,\n exclude_domains=[\"spammyads.com\"],\n exclude_social_media_links=True, # skip Twitter, Facebook, etc.\n exclude_external_images=True, # keep only images from main domain\n wait_for_images=True, # ensure images are loaded\n verbose=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://www.example.com\", config=crawler_cfg)\n\n if result.success:\n print(\"[OK] Crawled:\", result.url)\n\n # 1. Links\n in_links = result.links.get(\"internal\", [])\n ext_links = result.links.get(\"external\", [])\n print(\"Internal link count:\", len(in_links))\n print(\"External link count:\", len(ext_links)) # should be zero with exclude_external_links=True\n\n # 2. Images\n images = result.media.get(\"images\", [])\n print(\"Images found:\", len(images))\n\n # Let's see a snippet of these images\n for i, img inenumerate(images[:3]):\n print(f\" - {img['src']} (alt={img.get('alt','')}, score={img.get('score','N/A')})\")\n else:\n print(\"[ERROR] Failed to crawl. Reason:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
6. Common Pitfalls & Tips
\n
1.âConflicting Flags: \n - exclude_external_links=True but then also specifying exclude_social_media_links=True is typically fine, but understand that the first setting already discards all external links. The second becomes somewhat redundant. \n - exclude_external_images=True but want to keep some external images? Currently no partial domain-based setting for images, so you might need a custom approach or hook logic.\n2.âRelevancy Scores: \n - If your version of Crawl4AI or your scraping strategy includes an img[\"score\"], itâs typically a heuristic based on size, position, or content analysis. Evaluate carefully if you rely on it.\n3.âPerformance: \n4.âSocial Media Lists: \n - exclude_social_media_links=True typically references an internal list of known social domains like Facebook, Twitter, LinkedIn, etc. If you need to add or remove from that list, look for library settings or a local config file (depending on your version).
\n
Extraction Strategies
\n
Extracting JSON (No LLM)
\n\n
Schema-based extraction with CSS or XPath selectors via JsonCssExtractionStrategy and JsonXPathExtractionStrategy
\n
Regular expression extraction with RegexExtractionStrategy for fast pattern matching
\n
Faster & Cheaper: No API calls or GPU overhead.
\n\n
1. Intro to Schema-Based Extraction
\n\n
Nested or list types for repeated or hierarchical structures.
\n\n
2. Simple Example: Crypto Prices
\n
Let's begin with a simple schema-based extraction using the JsonCssExtractionStrategy. Below is a snippet that extracts cryptocurrency prices from a site (similar to the legacy Coinbase example). Notice we don't call any LLM:\n
import json\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\nfrom crawl4ai import JsonCssExtractionStrategy\n\nasyncdefextract_crypto_prices():\n # 1. Define a simple extraction schema\n schema = {\n \"name\": \"Crypto Prices\",\n \"baseSelector\": \"div.crypto-row\", # Repeated elements\n \"fields\": [\n {\n \"name\": \"coin_name\",\n \"selector\": \"h2.coin-name\",\n \"type\": \"text\"\n },\n {\n \"name\": \"price\",\n \"selector\": \"span.coin-price\",\n \"type\": \"text\"\n }\n ]\n }\n\n # 2. Create the extraction strategy\n extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n\n # 3. Set up your crawler config (if needed)\n config = CrawlerRunConfig(\n # e.g., pass js_code or wait_for if the page is dynamic\n # wait_for=\"css:.crypto-row:nth-child(20)\"\n cache_mode = CacheMode.BYPASS,\n extraction_strategy=extraction_strategy,\n )\n\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n # 4. Run the crawl and extraction\n result = await crawler.arun(\n url=\"https://example.com/crypto-prices\",\n\n config=config\n )\n\n ifnot result.success:\n print(\"Crawl failed:\", result.error_message)\n return\n\n # 5. Parse the extracted JSON\n data = json.loads(result.extracted_content)\n print(f\"Extracted {len(data)} coin entries\")\n print(json.dumps(data[0], indent=2) if data else\"No data found\")\n\nasyncio.run(extract_crypto_prices())\n
\n- baseSelector: Tells us where each \"item\" (crypto row) is. \n- fields: Two fields (coin_name, price) using simple CSS selectors. \n- Each field defines a type (e.g., text, attribute, html, regex, etc.).\n
XPath Example with raw:// HTML
\n
Below is a short example demonstrating XPath extraction plus the raw:// scheme. We'll pass a dummy HTML directly (no network request) and define the extraction strategy in CrawlerRunConfig.\n
import json\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai import JsonXPathExtractionStrategy\n\nasyncdefextract_crypto_prices_xpath():\n # 1. Minimal dummy HTML with some repeating rows\n dummy_html = \"\"\"\n <html>\n <body>\n <div class='crypto-row'>\n <h2 class='coin-name'>Bitcoin</h2>\n <span class='coin-price'>$28,000</span>\n </div>\n <div class='crypto-row'>\n <h2 class='coin-name'>Ethereum</h2>\n <span class='coin-price'>$1,800</span>\n </div>\n </body>\n </html>\n \"\"\"\n\n # 2. Define the JSON schema (XPath version)\n schema = {\n \"name\": \"Crypto Prices via XPath\",\n \"baseSelector\": \"//div[@class='crypto-row']\",\n \"fields\": [\n {\n \"name\": \"coin_name\",\n \"selector\": \".//h2[@class='coin-name']\",\n \"type\": \"text\"\n },\n {\n \"name\": \"price\",\n \"selector\": \".//span[@class='coin-price']\",\n \"type\": \"text\"\n }\n ]\n }\n\n # 3. Place the strategy in the CrawlerRunConfig\n config = CrawlerRunConfig(\n extraction_strategy=JsonXPathExtractionStrategy(schema, verbose=True)\n )\n\n # 4. Use raw:// scheme to pass dummy_html directly\n raw_url = f\"raw://{dummy_html}\"\n\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n result = await crawler.arun(\n url=raw_url,\n config=config\n )\n\n ifnot result.success:\n print(\"Crawl failed:\", result.error_message)\n return\n\n data = json.loads(result.extracted_content)\n print(f\"Extracted {len(data)} coin rows\")\n if data:\n print(\"First item:\", data[0])\n\nasyncio.run(extract_crypto_prices_xpath())\n
\n1. JsonXPathExtractionStrategy is used instead of JsonCssExtractionStrategy. \n2. baseSelector and each field's \"selector\" use XPath instead of CSS. \n3. raw:// lets us pass dummy_html with no real network requestâhandy for local testing. \n4. Everything (including the extraction strategy) is in CrawlerRunConfig. \nThat's how you keep the config self-contained, illustrate XPath usage, and demonstrate the raw scheme for direct HTML inputâall while avoiding the old approach of passing extraction_strategy directly to arun().\n
\n- Nested vs. List: \n - type: \"nested\" means a single sub-object (like details). \n - type: \"list\" means multiple items that are simple dictionaries or single text fields. \n - type: \"nested_list\" means repeated complex objects (like products or reviews).\n- Base Fields: We can extract attributes from the container element via \"baseFields\". For instance, \"data_cat_id\" might be data-cat-id=\"elect123\". \n- Transforms: We can also define a transform if we want to lower/upper case, strip whitespace, or even run a custom function.\n
Running the Extraction
\n
import json\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai import JsonCssExtractionStrategy\n\necommerce_schema = {\n # ... the advanced schema from above ...\n}\n\nasyncdefextract_ecommerce_data():\n strategy = JsonCssExtractionStrategy(ecommerce_schema, verbose=True)\n\n config = CrawlerRunConfig()\n\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n result = await crawler.arun(\n url=\"https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html\",\n extraction_strategy=strategy,\n config=config\n )\n\n ifnot result.success:\n print(\"Crawl failed:\", result.error_message)\n return\n\n # Parse the JSON output\n data = json.loads(result.extracted_content)\n print(json.dumps(data, indent=2) if data else\"No data found.\")\n\nasyncio.run(extract_ecommerce_data())\n
\nIf all goes well, you get a structured JSON array with each \"category,\" containing an array of products. Each product includes details, features, reviews, etc. All of that without an LLM.\n
4. RegexExtractionStrategy - Fast Pattern-Based Extraction
\n
Crawl4AI now offers a powerful new zero-LLM extraction strategy: RegexExtractionStrategy. This strategy provides lightning-fast extraction of common data types like emails, phone numbers, URLs, dates, and more using pre-compiled regular expressions.
\n
Key Features
\n
\n
Zero LLM Dependency: Extracts data without any AI model calls
\n
Blazing Fast: Uses pre-compiled regex patterns for maximum performance
\n
Built-in Patterns: Includes ready-to-use patterns for common data types
\n
\n
Simple Example: Extracting Common Entities
\n
import json\nimport asyncio\nfrom crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n RegexExtractionStrategy\n)\n\nasyncdefextract_with_regex():\n # Create a strategy using built-in patterns for URLs and currencies\n strategy = RegexExtractionStrategy(\n pattern = RegexExtractionStrategy.Url | RegexExtractionStrategy.Currency\n )\n\n config = CrawlerRunConfig(extraction_strategy=strategy)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=config\n )\n\n if result.success:\n data = json.loads(result.extracted_content)\n for item in data[:5]: # Show first 5 matches\n print(f\"{item['label']}: {item['value']}\")\n print(f\"Total matches: {len(data)}\")\n\nasyncio.run(extract_with_regex())\n
\n
Available Built-in Patterns
\n
RegexExtractionStrategy provides these common patterns as IntFlag attributes for easy combining:\n
# Use individual patterns\nstrategy = RegexExtractionStrategy(pattern=RegexExtractionStrategy.Email)\n\n# Combine multiple patterns\nstrategy = RegexExtractionStrategy(\n pattern = (\n RegexExtractionStrategy.Email | \n RegexExtractionStrategy.PhoneUS | \n RegexExtractionStrategy.Url\n )\n)\n\n# Use all available patterns\nstrategy = RegexExtractionStrategy(pattern=RegexExtractionStrategy.All)\n
\n- Email - Email addresses\n- PhoneIntl - International phone numbers\n- PhoneUS - US-format phone numbers\n- Url - HTTP/HTTPS URLs\n- IPv4 - IPv4 addresses\n- IPv6 - IPv6 addresses\n- Uuid - UUIDs\n- Currency - Currency values (USD, EUR, etc.)\n- Percentage - Percentage values\n- Number - Numeric values\n- DateIso - ISO format dates\n- DateUS - US format dates\n- Time24h - 24-hour format times\n- PostalUS - US postal codes\n- PostalUK - UK postal codes\n- HexColor - HTML hex color codes\n- TwitterHandle - Twitter handles\n- Hashtag - Hashtags\n- MacAddr - MAC addresses\n- Iban - International bank account numbers\n- CreditCard - Credit card numbers\n
Custom Pattern Example
\n
import json\nimport asyncio\nfrom crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n RegexExtractionStrategy\n)\n\nasyncdefextract_prices():\n # Define a custom pattern for US Dollar prices\n price_pattern = {\"usd_price\": r\"\\$\\s?\\d{1,3}(?:,\\d{3})*(?:\\.\\d{2})?\"}\n\n # Create strategy with custom pattern\n strategy = RegexExtractionStrategy(custom=price_pattern)\n config = CrawlerRunConfig(extraction_strategy=strategy)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.example.com/products\",\n config=config\n )\n\n if result.success:\n data = json.loads(result.extracted_content)\n for item in data:\n print(f\"Found price: {item['value']}\")\n\nasyncio.run(extract_prices())\n
\n
LLM-Assisted Pattern Generation
\n
import json\nimport asyncio\nfrom pathlib import Path\nfrom crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n RegexExtractionStrategy,\n LLMConfig\n)\n\nasyncdefextract_with_generated_pattern():\n cache_dir = Path(\"./pattern_cache\")\n cache_dir.mkdir(exist_ok=True)\n pattern_file = cache_dir / \"price_pattern.json\"\n\n # 1. Generate or load pattern\n if pattern_file.exists():\n pattern = json.load(pattern_file.open())\n print(f\"Using cached pattern: {pattern}\")\n else:\n print(\"Generating pattern via LLM...\")\n\n # Configure LLM\n llm_config = LLMConfig(\n provider=\"openai/gpt-4o-mini\",\n api_token=\"env:OPENAI_API_KEY\",\n )\n\n # Get sample HTML for context\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com/products\")\n html = result.markdown.fit_html\n\n # Generate pattern (one-time LLM usage)\n pattern = RegexExtractionStrategy.generate_pattern(\n label=\"price\",\n html=html,\n query=\"Product prices in USD format\",\n llm_config=llm_config,\n )\n\n # Cache pattern for future use\n json.dump(pattern, pattern_file.open(\"w\"), indent=2)\n\n # 2. Use pattern for extraction (no LLM calls)\n strategy = RegexExtractionStrategy(custom=pattern)\n config = CrawlerRunConfig(extraction_strategy=strategy)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/products\",\n config=config\n )\n\n if result.success:\n data = json.loads(result.extracted_content)\n for item in data[:10]:\n print(f\"Extracted: {item['value']}\")\n print(f\"Total matches: {len(data)}\")\n\nasyncio.run(extract_with_generated_pattern())\n
\n1. Use an LLM once to generate a highly optimized regex for your specific site\n2. Save the pattern to disk for reuse \n3. Extract data using only regex (no further LLM calls) in production\n
Extraction Results Format
\n
The RegexExtractionStrategy returns results in a consistent format:\n
\n- url: The source URL\n- label: The pattern name that matched (e.g., \"email\", \"phone_us\")\n- value: The extracted text\n- span: The start and end positions in the source content\n
5. Why \"No LLM\" Is Often Better
\n
6. Base Element Attributes & Additional Fields
\n
It's easy to extract attributes (like href, src, or data-xxx) from your base or nested elements using:\n
\nYou can define them in baseFields (extracted from the main container element) or in each field's sub-lists. This is especially helpful if you need an item's link or ID stored in the parent <div>.\n
7. Putting It All Together: Larger Example
\n
Consider a blog site. We have a schema that extracts the URL from each post card (via baseFields with an \"attribute\": \"href\"), plus the title, date, summary, and author:\n
\nThen run with JsonCssExtractionStrategy(schema) to get an array of blog post objects, each with \"post_url\", \"title\", \"date\", \"summary\", \"author\".\n
8. Tips & Best Practices
\n\n
Test your schema on partial HTML or a test page before a big crawl.
\n
Combine with JS Execution if the site loads content dynamically. You can pass js_code or wait_for in CrawlerRunConfig.
\n
Look at Logs when verbose=True: if your selectors are off or your schema is malformed, it'll often show warnings.
\n
Use baseFields if you need attributes from the container element (e.g., href, data-id), especially for the \"parent\" item.
\n
Consider Using Regex First: For simple data types like emails, URLs, and dates, RegexExtractionStrategy is often the fastest approach.
\n\n
9. Schema Generation Utility
\n\n
You're dealing with a new website structure and want a quick starting point
\n
You need to extract complex nested data structures
\n
You want to avoid the learning curve of CSS/XPath selector syntax
\n\n
Using the Schema Generator
\n
The schema generator is available as a static method on both JsonCssExtractionStrategy and JsonXPathExtractionStrategy. You can choose between OpenAI's GPT-4 or the open-source Ollama for schema generation:\n
from crawl4ai import JsonCssExtractionStrategy, JsonXPathExtractionStrategy\nfrom crawl4ai import LLMConfig\n\n# Sample HTML with product information\nhtml = \"\"\"\n<div class=\"product-card\">\n <h2 class=\"title\">Gaming Laptop</h2>\n <div class=\"price\">$999.99</div>\n <div class=\"specs\">\n <ul>\n <li>16GB RAM</li>\n <li>1TB SSD</li>\n </ul>\n </div>\n</div>\n\"\"\"\n\n# Option 1: Using OpenAI (requires API token)\ncss_schema = JsonCssExtractionStrategy.generate_schema(\n html,\n schema_type=\"css\", \n llm_config = LLMConfig(provider=\"openai/gpt-4o\",api_token=\"your-openai-token\")\n)\n\n# Option 2: Using Ollama (open source, no token needed)\nxpath_schema = JsonXPathExtractionStrategy.generate_schema(\n html,\n schema_type=\"xpath\",\n llm_config = LLMConfig(provider=\"ollama/llama3.3\", api_token=None) # Not needed for Ollama\n)\n\n# Use the generated schema for fast, repeated extractions\nstrategy = JsonCssExtractionStrategy(css_schema)\n
\n
Token Usage Tracking
\n
generate_schema may make multiple LLM calls internally (field inference, generation, validation retries). Track total token consumption by passing a TokenUsage accumulator:\n
from crawl4ai.models import TokenUsage\n\nusage = TokenUsage()\nschema = JsonCssExtractionStrategy.generate_schema(\n url=\"https://example.com/products\",\n query=\"extract product name and price\",\n usage=usage,\n)\nprint(f\"Total tokens: {usage.total_tokens}\")\n
\nThe usage parameter is optional and fully backward-compatible. Both generate_schema (sync) and agenerate_schema (async) support it.\n
LLM Provider Options
\n\n
OpenAI GPT-4 (openai/gpt4o)
\n
Default provider
\n
Requires an API token
\n
Generally provides more accurate schemas
\n
Set via environment variable: OPENAI_API_KEY
\n
Ollama (ollama/llama3.3)
\n
Open source alternative
\n
No API token required
\n
Self-hosted option
\n
Good for development and testing
\n\n
Benefits of Schema Generation
\n
Best Practices
\n\n
Choose Provider Wisely:
\n
Use OpenAI for production-quality schemas
\n
Use Ollama for development, testing, or when you need a self-hosted solution
\n\n
10. Conclusion
\n
With Crawl4AI's LLM-free extraction strategies - JsonCssExtractionStrategy, JsonXPathExtractionStrategy, and now RegexExtractionStrategy - you can build powerful pipelines that:\n- Scrape any consistent site for structured data. \n- Support nested objects, repeating lists, or pattern-based extraction. \n- Scale to thousands of pages quickly and reliably.\n- Use RegexExtractionStrategy for fast extraction of common data types like emails, phones, URLs, dates, etc.\n- Use JsonCssExtractionStrategy or JsonXPathExtractionStrategy for structured data with clear HTML patterns
\n
Extracting JSON (LLM)
\n
Important: LLM-based extraction can be slower and costlier than schema-based approaches. If your page data is highly structured, consider using JsonCssExtractionStrategy or JsonXPathExtractionStrategy first. But if you need AI to interpret or reorganize content, read on!
\nCrawl4AI uses a âprovider stringâ (e.g., \"openai/gpt-4o\", \"ollama/llama2.0\", \"aws/titan\") to identify your LLM.âAny model that LiteLLM supports is fair game. You just provide:\n- provider: The <provider>/<model_name> identifier (e.g., \"openai/gpt-4\", \"ollama/llama2\", \"huggingface/google-flan\", etc.). \n- api_token: If needed (for OpenAI, HuggingFace, etc.); local models or Ollama might not require it. \n- base_url (optional): If your provider has a custom endpoint. \n
3. How LLM Extraction Works
\n
3.1 Flow
\n
1.âChunking (optional): The HTML or markdown is split into smaller segments if itâs very long (based on chunk_token_threshold, overlap, etc.). \n2.âPrompt Construction: For each chunk, the library forms a prompt that includes your instruction (and possibly schema or examples). \n4.âCombining: The results from each chunk are merged and parsed into JSON.
\n
3.2 extraction_type
\n
\n
\"schema\": The model tries to return JSON conforming to your Pydantic-based schema.
\n
\"block\": The model returns freeform text, or smaller JSON structures, which the library collects. \nFor structured data, \"schema\" is recommended. You provide schema=YourPydanticModel.model_json_schema().
\n
\n
4. Key Parameters
\n
Below is an overview of important LLM extraction parameters. All are typically set inside LLMExtractionStrategy(...). You then put that strategy in your CrawlerRunConfig(..., extraction_strategy=...).\n1.âllmConfig (LlmConfig): e.g., \"openai/gpt-4\", \"ollama/llama2\". \n2.âschema (dict): A JSON schema describing the fields you want. Usually generated by YourModel.model_json_schema(). \n3.âextraction_type (str): \"schema\" or \"block\". \n4.âinstruction (str): Prompt text telling the LLM what you want extracted. E.g., âExtract these fields as a JSON array.â \n5.âchunk_token_threshold (int): Maximum tokens per chunk. If your content is huge, you can break it up for the LLM. \n6.âoverlap_rate (float): Overlap ratio between adjacent chunks. E.g., 0.1 means 10% of each chunk is repeated to preserve context continuity. \n7.âapply_chunking (bool): Set True to chunk automatically. If you want a single pass, set False. \n8.âinput_format (str): Determines which crawler result is passed to the LLM. Options include: \n - \"markdown\": The raw markdown (default). \n - \"fit_markdown\": The filtered âfitâ markdown if you used a content filter. \n - \"html\": The cleaned or raw HTML. \n9.âextra_args (dict): Additional LLM parameters like temperature, max_tokens, top_p, etc. \n10.âshow_usage(): A method you can call to print out usage info (token usage per chunk, total cost if known). \n
extraction_strategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"openai/gpt-4\", api_token=\"YOUR_OPENAI_KEY\"),\n schema=MyModel.model_json_schema(),\n extraction_type=\"schema\",\n instruction=\"Extract a list of items from the text with 'name' and 'price' fields.\",\n chunk_token_threshold=1200,\n overlap_rate=0.1,\n apply_chunking=True,\n input_format=\"html\",\n extra_args={\"temperature\":0.1, \"max_tokens\":1000},\n verbose=True\n)\n
\n
5. Putting It in CrawlerRunConfig
\n
Important: In Crawl4AI, all strategy definitions should go inside the CrawlerRunConfig, not directly as a param in arun(). Hereâs a full example:\n
import os\nimport asyncio\nimport json\nfrom pydantic import BaseModel, Field\nfrom typing importList\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\n\nclassProduct(BaseModel):\n name: str\n price: str\n\nasyncdefmain():\n # 1. Define the LLM extraction strategy\n llm_strategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"openai/gpt-4o-mini\", api_token=os.getenv('OPENAI_API_KEY')),\n schema=Product.schema_json(), # Or use model_json_schema()\n extraction_type=\"schema\",\n instruction=\"Extract all product objects with 'name' and 'price' from the content.\",\n chunk_token_threshold=1000,\n overlap_rate=0.0,\n apply_chunking=True,\n input_format=\"markdown\", # or \"html\", \"fit_markdown\"\n extra_args={\"temperature\": 0.0, \"max_tokens\": 800}\n )\n\n # 2. Build the crawler config\n crawl_config = CrawlerRunConfig(\n extraction_strategy=llm_strategy,\n cache_mode=CacheMode.BYPASS\n )\n\n # 3. Create a browser config if needed\n browser_cfg = BrowserConfig(headless=True)\n\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n # 4. Let's say we want to crawl a single page\n result = await crawler.arun(\n url=\"https://example.com/products\",\n config=crawl_config\n )\n\n if result.success:\n # 5. The extracted content is presumably JSON\n data = json.loads(result.extracted_content)\n print(\"Extracted items:\", data)\n\n # 6. Show usage stats\n llm_strategy.show_usage() # prints token usage\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
6. Chunking Details
\n
6.1 chunk_token_threshold
\n
If your page is large, you might exceed your LLMâs context window.âchunk_token_threshold sets the approximate max tokens per chunk. The library calculates wordâtoken ratio using word_token_rate (often ~0.75 by default). If chunking is enabled (apply_chunking=True), the text is split into segments.
\n
6.2 overlap_rate
\n
To keep context continuous across chunks, we can overlap them. E.g., overlap_rate=0.1 means each subsequent chunk includes 10% of the previous chunkâs text. This is helpful if your needed info might straddle chunk boundaries.
\n
6.3 Performance & Parallelism
\n
7. Input Format
\n
By default, LLMExtractionStrategy uses input_format=\"markdown\", meaning the crawlerâs final markdown is fed to the LLM. You can change to:\n- html: The cleaned HTML or raw HTML (depending on your crawler config) goes into the LLM. \n- fit_markdown: If you used, for instance, PruningContentFilter, the âfitâ version of the markdown is used. This can drastically reduce tokens if you trust the filter. \n- markdown: Standard markdown output from the crawlerâs markdown_generator.\nThis setting is crucial: if the LLM instructions rely on HTML tags, pick \"html\". If you prefer a text-based approach, pick \"markdown\".\n
LLMExtractionStrategy(\n # ...\n input_format=\"html\", # Instead of \"markdown\" or \"fit_markdown\"\n)\n
\n
8. Token Usage & Show Usage
\n
\n
usages (list): token usage per chunk or call.
\n
total_usage: sum of all chunk calls.
\n
show_usage(): prints a usage report (if the provider returns usage data).\n
llm_strategy = LLMExtractionStrategy(...)\n# ...\nllm_strategy.show_usage()\n# e.g. âTotal usage: 1241 tokens across 2 chunk callsâ\n
\n
\n
9. Example: Building a Knowledge Graph
\n
Below is a snippet combining LLMExtractionStrategy with a Pydantic schema for a knowledge graph. Notice how we pass an instruction telling the model what to parse.\n
\n- extraction_type=\"schema\" ensures we get JSON fitting our KnowledgeGraph. \n- input_format=\"html\" means we feed HTML to the model. \n- instruction guides the model to output a structured knowledge graph. \n
10. Best Practices & Caveats
\n
4.âSchema Strictness: \"schema\" extraction tries to parse the model output as JSON. If the model returns invalid JSON, partial extraction might happen, or you might get an error.
\n
11. Conclusion
\n
\n
Put your LLM strategy in CrawlerRunConfig.
\n
Use input_format to pick which form (markdown, HTML, fit_markdown) the LLM sees.
\n
Tweak chunk_token_threshold, overlap_rate, and apply_chunking to handle large content efficiently.
\n
Monitor token usage with show_usage().\nIf your siteâs data is consistent or repetitive, consider JsonCssExtractionStrategy first for speed and simplicity. But if you need an AI-driven approach, LLMExtractionStrategy offers a flexible, multi-provider solution for extracting structured JSON from any website.\n1.âExperiment with Different Providers
\n
Try switching the provider (e.g., \"ollama/llama2\", \"openai/gpt-4o\", etc.) to see differences in speed, accuracy, or cost.
\n
Pass different extra_args like temperature, top_p, and max_tokens to fine-tune your results.\n2.âPerformance Tuning
\n
If pages are large, tweak chunk_token_threshold, overlap_rate, or apply_chunking to optimize throughput.
\n
Check the usage logs with show_usage() to keep an eye on token consumption and identify potential bottlenecks.\n3.âValidate Outputs
\n
If using extraction_type=\"schema\", parse the LLMâs JSON with a Pydantic model for a final validation step. \n4.âExplore Hooks & Automation
\n
\n
Advanced Features
\n
Session Management
\n
\n
Performing JavaScript actions before and after crawling.\nUse BrowserConfig and CrawlerRunConfig to maintain state with a session_id:\n
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig\n\nasyncwithAsyncWebCrawler() as crawler:\n session_id = \"my_session\"\n\n # Define configurations\n config1 = CrawlerRunConfig(\n url=\"https://example.com/page1\", session_id=session_id\n )\n config2 = CrawlerRunConfig(\n url=\"https://example.com/page2\", session_id=session_id\n )\n\n # First request\n result1 = await crawler.arun(config=config1)\n\n # Subsequent request using the same session\n result2 = await crawler.arun(config=config2)\n\n # Clean up when done\n await crawler.crawler_strategy.kill_session(session_id)\n
\n1. Reusing the same session_id across multiple requests.\n2. Executing JavaScript to load more content dynamically.\n3. Properly closing the session to free resources.\n
\n1.âAuthentication Flows: Login and interact with secured pages.\n2.âPagination Handling: Navigate through multiple pages.\n3.âForm Submissions: Fill forms, submit, and process results.\n4.âMulti-step Processes: Complete workflows that span multiple actions.\n
Hooks & Auth in AsyncWebCrawler
\n
1.âon_browser_created â After browser creation. \n2.âon_page_context_created â After a new context & page are created. \n3.âbefore_goto â Just before navigating to a page. \n4.âafter_goto â Right after navigation completes. \n5.âon_user_agent_updated â Whenever the user agent changes. \n6.âon_execution_started â Once custom JavaScript execution begins. \n7.âbefore_retrieve_html â Just before the crawler retrieves final HTML. \n8.âbefore_return_html â Right before returning the HTML content.\nImportant: Avoid heavy tasks in on_browser_created since you donât yet have a page context. If you need to log in, do so in on_page_context_created.
\n
\n
note \"Important Hook Usage Warning\"\n Avoid Misusing Hooks: Do not manipulate page objects in the wrong hook or at the wrong time, as it can crash the pipeline or produce incorrect results. A common mistake is attempting to handle authentication prematurelyâsuch as creating or closing pages in on_browser_created. \n Use the Right Hook for Auth: If you need to log in or set tokens, use on_page_context_created. This ensures you have a valid page/context to work with, without disrupting the main crawling flow.
\n
\n
Example: Using Hooks in AsyncWebCrawler
\n
import asyncio\nimport json\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\nfrom playwright.async_api import Page, BrowserContext\n\nasyncdefmain():\n print(\"đ Hooks Example: Demonstrating recommended usage\")\n\n # 1) Configure the browser\n browser_config = BrowserConfig(\n headless=True,\n verbose=True\n )\n\n # 2) Configure the crawler run\n crawler_run_config = CrawlerRunConfig(\n js_code=\"window.scrollTo(0, document.body.scrollHeight);\",\n wait_for=\"body\",\n cache_mode=CacheMode.BYPASS\n )\n\n # 3) Create the crawler instance\n crawler = AsyncWebCrawler(config=browser_config)\n\n #\n # Define Hook Functions\n #\n\n asyncdefon_browser_created(browser, **kwargs):\n # Called once the browser instance is created (but no pages or contexts yet)\n print(\"[HOOK] on_browser_created - Browser created successfully!\")\n # Typically, do minimal setup here if needed\n return browser\n\n asyncdefon_page_context_created(page: Page, context: BrowserContext, **kwargs):\n # Called right after a new page + context are created (ideal for auth or route config).\n print(\"[HOOK] on_page_context_created - Setting up page & context.\")\n\n # Example 1: Route filtering (e.g., block images)\n asyncdefroute_filter(route):\n if route.request.resource_type == \"image\":\n print(f\"[HOOK] Blocking image request: {route.request.url}\")\n await route.abort()\n else:\n await route.continue_()\n\n await context.route(\"**\", route_filter)\n\n # Example 2: (Optional) Simulate a login scenario\n # (We do NOT create or close pages here, just do quick steps if needed)\n # e.g., await page.goto(\"https://example.com/login\")\n # e.g., await page.fill(\"input[name='username']\", \"testuser\")\n # e.g., await page.fill(\"input[name='password']\", \"password123\")\n # e.g., await page.click(\"button[type='submit']\")\n # e.g., await page.wait_for_selector(\"#welcome\")\n # e.g., await context.add_cookies([...])\n # Then continue\n\n # Example 3: Adjust the viewport\n await page.set_viewport_size({\"width\": 1080, \"height\": 600})\n return page\n\n asyncdefbefore_goto(\n page: Page, context: BrowserContext, url: str, **kwargs\n ):\n # Called before navigating to each URL.\n print(f\"[HOOK] before_goto - About to navigate: {url}\")\n # e.g., inject custom headers\n await page.set_extra_http_headers({\n \"Custom-Header\": \"my-value\"\n })\n return page\n\n asyncdefafter_goto(\n page: Page, context: BrowserContext, \n url: str, response, **kwargs\n ):\n # Called after navigation completes.\n print(f\"[HOOK] after_goto - Successfully loaded: {url}\")\n # e.g., wait for a certain element if we want to verify\n try:\n await page.wait_for_selector('.content', timeout=1000)\n print(\"[HOOK] Found .content element!\")\n except:\n print(\"[HOOK] .content not found, continuing anyway.\")\n return page\n\n asyncdefon_user_agent_updated(\n page: Page, context: BrowserContext, \n user_agent: str, **kwargs\n ):\n # Called whenever the user agent updates.\n print(f\"[HOOK] on_user_agent_updated - New user agent: {user_agent}\")\n return page\n\n asyncdefon_execution_started(page: Page, context: BrowserContext, **kwargs):\n # Called after custom JavaScript execution begins.\n print(\"[HOOK] on_execution_started - JS code is running!\")\n return page\n\n asyncdefbefore_retrieve_html(page: Page, context: BrowserContext, **kwargs):\n # Called before final HTML retrieval.\n print(\"[HOOK] before_retrieve_html - We can do final actions\")\n # Example: Scroll again\n await page.evaluate(\"window.scrollTo(0, document.body.scrollHeight);\")\n return page\n\n asyncdefbefore_return_html(\n page: Page, context: BrowserContext, html: str, **kwargs\n ):\n # Called just before returning the HTML in the result.\n print(f\"[HOOK] before_return_html - HTML length: {len(html)}\")\n return page\n\n #\n # Attach Hooks\n #\n\n crawler.crawler_strategy.set_hook(\"on_browser_created\", on_browser_created)\n crawler.crawler_strategy.set_hook(\n \"on_page_context_created\", on_page_context_created\n )\n crawler.crawler_strategy.set_hook(\"before_goto\", before_goto)\n crawler.crawler_strategy.set_hook(\"after_goto\", after_goto)\n crawler.crawler_strategy.set_hook(\n \"on_user_agent_updated\", on_user_agent_updated\n )\n crawler.crawler_strategy.set_hook(\n \"on_execution_started\", on_execution_started\n )\n crawler.crawler_strategy.set_hook(\n \"before_retrieve_html\", before_retrieve_html\n )\n crawler.crawler_strategy.set_hook(\n \"before_return_html\", before_return_html\n )\n\n await crawler.start()\n\n # 4) Run the crawler on an example page\n url = \"https://example.com\"\n result = await crawler.arun(url, config=crawler_run_config)\n\n if result.success:\n print(\"\\nCrawled URL:\", result.url)\n print(\"HTML length:\", len(result.html))\n else:\n print(\"Error:\", result.error_message)\n\n await crawler.close()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Hook Lifecycle Summary
\n
1.âon_browser_created: \n - Browser is up, but no pages or contexts yet. \n - Light setup onlyâdonât try to open or close pages here (that belongs in on_page_context_created).\n2.âon_page_context_created: \n - Perfect for advanced auth or route blocking. \n3.âbefore_goto: \n4.âafter_goto: \n5.âon_user_agent_updated: \n - Whenever the user agent changes (for stealth or different UA modes).\n6.âon_execution_started: \n - If you set js_code or run custom scripts, this runs once your JS is about to start.\n7.âbefore_retrieve_html: \n8.âbefore_return_html: \n - The last hook before returning HTML to the CrawlResult. Good for logging HTML length or minor modifications.
\n
When to Handle Authentication
\n
Recommended: Use on_page_context_created if you need to:\n- Navigate to a login page or fill forms\n- Set cookies or localStorage tokens\n- Block resource routes to avoid ads\nThis ensures the newly created context is under your control beforearun() navigates to the main URL.
\n
Additional Considerations
\n
\n
Session Management: If you want multiple arun() calls to reuse a single session, pass session_id= in your CrawlerRunConfig. Hooks remain the same.
\n
Concurrency: If you run arun_many(), each URL triggers these hooks in parallel. Ensure your hooks are thread/async-safe.
\n
\n
Conclusion
\n
\n
Browser creation (light tasks only)
\n
Page and context creation (auth, route blocking)
\n
Navigation phases
\n
Final HTML retrieval
\n
Login or advanced tasks in on_page_context_created
\n
Custom headers or logs in before_goto / after_goto
\n
Scrolling or final checks in before_retrieve_html / before_return_html
\n
\n\n
Quick Reference
\n
Core Imports
\n
from crawl4ai importAsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig\nfrom crawl4ai.extraction_strategyimportLLMExtractionStrategy, JsonCssExtractionStrategy, CosineStrategy\n
\n
Basic Pattern
\n
asyncwithAsyncWebCrawler() as crawler:\n result = await crawler.arun(url=\"https://example.com\")\n print(result.markdown)\n
\n
Advanced Pattern
\n
browser_config = BrowserConfig(headless=True, viewport_width=1920)\ncrawler_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n wait_for=\"css:.content\",\n screenshot=True,\n pdf=True\n)\nstrategy = LLMExtractionStrategy(\n llm_config=LLMConfig(provider=\"openai/gpt-4\", api_token=\"your-openai-token\"),\n instruction=\"Extract products with name and price\"\n)\n\nasyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=crawler_config,\n extraction_strategy=strategy\n )\n
Traditional web crawlers follow predetermined patterns, crawling pages blindly without knowing when they've gathered enough information. Adaptive Crawling changes this paradigm by introducing intelligence into the crawling process.
\n
Think of it like research: when you're looking for information, you don't read every book in the library. You stop when you've found sufficient information to answer your question. That's exactly what Adaptive Crawling does for web scraping.
\n
Key Concepts
\n
The Problem It Solves
\n
When crawling websites for specific information, you face two challenges:\n1. Under-crawling: Stopping too early and missing crucial information\n2. Over-crawling: Wasting resources by crawling irrelevant pages
\n
Adaptive Crawling solves both by using a three-layer scoring system that determines when you have \"enough\" information.
\n
How It Works
\n
The AdaptiveCrawler uses three metrics to measure information sufficiency:
\n
\n
Coverage: How well your collected pages cover the query terms
\n
Consistency: Whether the information is coherent across pages
\n
Saturation: Detecting when new pages aren't adding new information
\n
\n
When these metrics indicate sufficient information has been gathered, crawling stops automatically.
\n
Quick Start
\n
Basic Usage
\n
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler\n\nasyncdefmain():\n asyncwith AsyncWebCrawler() as crawler:\n # Create an adaptive crawler (config is optional)\n adaptive = AdaptiveCrawler(crawler)\n\n # Start crawling with a query\n result = await adaptive.digest(\n start_url=\"https://docs.python.org/3/\",\n query=\"async context managers\"\n )\n\n # View statistics\n adaptive.print_stats()\n\n # Get the most relevant content\n relevant_pages = adaptive.get_relevant_content(top_k=5)\n for page in relevant_pages:\n print(f\"- {page['url']} (score: {page['score']:.2f})\")\n
\n
Configuration Options
\n
from crawl4ai import AdaptiveConfig\n\nconfig = AdaptiveConfig(\n confidence_threshold=0.8, # Stopwhen80% confident (default: 0.7)\n max_pages=30, # Maximum pages to crawl (default: 20)\n top_k_links=5, # Links to follow per page (default: 3)\n min_gain_threshold=0.05 # Minimum expected gain tocontinue (default: 0.1)\n)\n\nadaptive = AdaptiveCrawler(crawler, config)\n
\n
Crawling Strategies
\n
Adaptive Crawling supports two distinct strategies for determining information sufficiency:
\n
Statistical Strategy (Default)
\n
The statistical strategy uses pure information theory and term-based analysis:
\n
\n
Fast and efficient - No API calls or model loading
\n
Term-based coverage - Analyzes query term presence and distribution
\n
No external dependencies - Works offline
\n
Best for: Well-defined queries with specific terminology
\n
\n
# Default configuration uses statistical strategy\nconfig = AdaptiveConfig(\n strategy=\"statistical\", # This is the default\n confidence_threshold=0.8\n)\n
\n
Embedding Strategy
\n
The embedding strategy uses semantic embeddings for deeper understanding:
\n
\n
Semantic understanding - Captures meaning beyond exact term matches
Gap-driven selection - Identifies semantic gaps in knowledge
\n
Validation-based stopping - Uses held-out queries to validate coverage
\n
Best for: Complex queries, ambiguous topics, conceptual understanding
\n
\n
# Configure embedding strategy with local embeddings\nconfig = AdaptiveConfig(\n strategy=\"embedding\",\n embedding_model=\"sentence-transformers/all-MiniLM-L6-v2\", # Default\n n_query_variations=10, # Generate 10 query variations\n embedding_min_confidence_threshold=0.1# Stop if completely irrelevant\n)\n\n# With separate LLM configs for embeddings and query expansion (recommended)\nfrom crawl4ai import LLMConfig\n\nconfig = AdaptiveConfig(\n strategy=\"embedding\",\n # Embedding model â used for text-to-vector calls\n embedding_llm_config=LLMConfig(\n provider='openai/text-embedding-3-small',\n api_token='your-api-key'\n ),\n # Query model â used for chat completion (query expansion)\n query_llm_config=LLMConfig(\n provider='openai/gpt-4o-mini',\n api_token='your-api-key'\n )\n)\n\n# Alternative: Dictionary format (backward compatible)\nconfig = AdaptiveConfig(\n strategy=\"embedding\",\n embedding_llm_config={\n 'provider': 'openai/text-embedding-3-small',\n 'api_token': 'your-api-key'\n },\n query_llm_config={\n 'provider': 'openai/gpt-4o-mini',\n 'api_token': 'your-api-key'\n }\n)\n
\n
\n
Note: The embedding strategy makes two types of API calls that need different model types:\n- Embedding calls (text â vector) require an embedding model like text-embedding-3-small\n- Query expansion (chat completion) requires a chat model like gpt-4o-mini
\n
Use embedding_llm_config for the embedding model and query_llm_config for the chat model. If query_llm_config is not set, it falls back to embedding_llm_config for backward compatibility.
\n
\n
Strategy Comparison
\n
\n\n
\n
Feature
\n
Statistical
\n
Embedding
\n
\n\n\n
\n
Speed
\n
Very fast
\n
Moderate (API calls)
\n
\n
\n
Cost
\n
Free
\n
Depends on provider
\n
\n
\n
Accuracy
\n
Good for exact terms
\n
Excellent for concepts
\n
\n
\n
Dependencies
\n
None
\n
Embedding model/API
\n
\n
\n
Query Understanding
\n
Literal
\n
Semantic
\n
\n
\n
Best Use Case
\n
Technical docs, specific terms
\n
Research, broad topics
\n
\n\n
\n
Embedding Strategy Configuration
\n
The embedding strategy offers fine-tuned control through several parameters:
\n
config = AdaptiveConfig(\n strategy=\"embedding\",\n\n # Model configuration\n embedding_model=\"sentence-transformers/all-MiniLM-L6-v2\",\n embedding_llm_config=None, # Use for API-based embeddings (embedding model)\n query_llm_config=None, # Use for query expansion (chat completion model)\n\n # Query expansion\n n_query_variations=10, # Number of query variations to generate\n\n # Coverage parameters\n embedding_coverage_radius=0.2, # Distance threshold for coverage\n embedding_k_exp=3.0, # Exponential decay factor (higher = stricter)\n\n # Stopping criteria\n embedding_min_relative_improvement=0.1, # Min improvement to continue\n embedding_validation_min_score=0.3, # Min validation score\n embedding_min_confidence_threshold=0.1, # Below this = irrelevant\n\n # Link selection\n embedding_overlap_threshold=0.85, # Similarity for deduplication\n\n # Display confidence mapping\n embedding_quality_min_confidence=0.7, # Min displayed confidence\n embedding_quality_max_confidence=0.95# Max displayed confidence\n)\n
\n
Handling Irrelevant Queries
\n
The embedding strategy can detect when a query is completely unrelated to the content:
\n
# This will stop quickly with low confidence\nresult = await adaptive.digest(\n start_url=\"https://docs.python.org/3/\",\n query=\"how to cook pasta\"# Irrelevant to Python docs\n)\n\n# Check if query was irrelevant\nif result.metrics.get('is_irrelevant', False):\n print(\"Query is unrelated to the content!\")\n
\n
When to Use Adaptive Crawling
\n
Perfect For:
\n
\n
Research Tasks: Finding comprehensive information about a topic
\n
Question Answering: Gathering sufficient context to answer specific queries
\n
Knowledge Base Building: Creating focused datasets for AI/ML applications
\n
Competitive Intelligence: Collecting complete information about specific products/features
\n
\n
Not Recommended For:
\n
\n
Full Site Archiving: When you need every page regardless of content
\n
Structured Data Extraction: When targeting specific, known page patterns
\n
Real-time Monitoring: When you need continuous updates
\n
\n
Understanding the Output
\n
Confidence Score
\n
The confidence score (0-1) indicates how sufficient the gathered information is:\n- 0.0-0.3: Insufficient information, needs more crawling\n- 0.3-0.6: Partial information, may answer basic queries\n- 0.6-0.7: Good coverage, can answer most queries\n- 0.7-1.0: Excellent coverage, comprehensive information
# Export collected pages to JSONL\nadaptive.export_knowledge_base(\"knowledge_base.jsonl\")\n\n# Import into another session\nnew_adaptive = AdaptiveCrawler(crawler)\nawait new_adaptive.import_knowledge_base(\"knowledge_base.jsonl\")\n
\n
Best Practices
\n
1. Query Formulation
\n
\n
Use specific, descriptive queries
\n
Include key terms you expect to find
\n
Avoid overly broad queries
\n
\n
2. Threshold Tuning
\n
\n
Start with default (0.7) for general use
\n
Lower to 0.5-0.6 for exploratory crawling
\n
Raise to 0.8+ for exhaustive coverage
\n
\n
3. Performance Optimization
\n
\n
Use appropriate max_pages limits
\n
Adjust top_k_links based on site structure
\n
Enable caching for repeat crawls
\n
\n
4. Link Selection
\n
\n
The crawler prioritizes links based on:
\n
Relevance to query
\n
Expected information gain
\n
URL structure and depth
\n
\n
Examples
\n
Research Assistant
\n
# Gather information about a programming concept\nresult = await adaptive.digest(\n start_url=\"https://realpython.com\",\n query=\"python decorators implementation patterns\"\n)\n\n# Get the most relevant excerpts\nfor doc in adaptive.get_relevant_content(top_k=3):\n print(f\"\\nFrom: {doc['url']}\")\n print(f\"Relevance: {doc['score']:.2%}\")\n print(doc['content'][:500] + \"...\")\n
\n
Knowledge Base Builder
\n
# Build a focused knowledge base about machine learning\nqueries =[\n \"supervised learning algorithms\",\n \"neural network architectures\", \n \"model evaluation metrics\"\n]\n\nfor query in queries:\n await adaptive.digest(\n start_url=\"https://scikit-learn.org/stable/\",\n query=query\n )\n\n# Export combined knowledge base\nadaptive.export_knowledge_base(\"ml_knowledge.jsonl\")\n
Q: How is this different from traditional crawling?\nA: Traditional crawling follows fixed patterns (BFS/DFS). Adaptive crawling makes intelligent decisions about which links to follow and when to stop based on information gain.
\n
Q: Can I use this with JavaScript-heavy sites?\nA: Yes! AdaptiveCrawler inherits all capabilities from AsyncWebCrawler, including JavaScript execution.
\n
Q: How does it handle large websites?\nA: The algorithm naturally limits crawling to relevant sections. Use max_pages as a safety limit.
\n
Q: Can I customize the scoring algorithms?\nA: Advanced users can implement custom strategies. See Adaptive Strategies.
Crawl4AI's flexibility stems from two key classes:
\n\n
BrowserConfig â Dictates how the browser is launched and behaves (e.g., headless or visible, proxy, user agent).
\n
CrawlerRunConfig â Dictates how each crawl operates (e.g., caching, extraction, timeouts, JavaScript code to run, etc.).
\n
LLMConfig - Dictates how LLM providers are configured. (model, api token, base url, temperature etc.)
\n\n
In most examples, you create oneBrowserConfig for the entire crawler session, then pass a fresh or re-used CrawlerRunConfig whenever you call arun(). This tutorial shows the most commonly used parameters. If you need advanced or rarely used fields, see the Configuration Parameters.
\n\n
1. BrowserConfig Essentials
\n
classBrowserConfig:\n def__init__(\n browser_type=\"chromium\",\n headless=True,\n browser_mode=\"dedicated\",\n use_managed_browser=False,\n cdp_url=None,\n debugging_port=9222,\n host=\"localhost\",\n proxy_config=None,\n viewport_width=1080,\n viewport_height=600,\n verbose=True,\n use_persistent_context=False,\n user_data_dir=None,\n cookies=None,\n headers=None,\n user_agent=(\n # \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) AppleWebKit/537.36 \"\n # \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 \"\n # \"(KHTML, like Gecko) Chrome/116.0.5845.187 Safari/604.1 Edg/117.0.2045.47\"\n \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/116.0.0.0 Safari/537.36\"\n ),\n user_agent_mode=\"\",\n text_mode=False,\n light_mode=False,\n extra_args=None,\n enable_stealth=False,\n # ... other advanced parameters omitted here\n ):\n ...\n
\n
Key Fields to Note
\n
1.â browser_type \n - Options: \"chromium\", \"firefox\", or \"webkit\". \n - Defaults to \"chromium\". \n - If you need a different engine, specify it here.
\n
2.â headless \n - True: Runs the browser in headless mode (invisible browser). \n - False: Runs the browser in visible mode, which helps with debugging.
\n
3.â browser_mode \n - Determines how the browser should be initialized:\n - \"dedicated\" (default): Creates a new browser instance each time\n - \"builtin\": Uses the builtin CDP browser running in background\n - \"custom\": Uses explicit CDP settings provided in cdp_url\n - \"docker\": Runs browser in Docker container with isolation
\n
4.â use_managed_browser & cdp_url \n - use_managed_browser=True: Launch browser using Chrome DevTools Protocol (CDP) for advanced control\n - cdp_url: URL for CDP endpoint (e.g., \"ws://localhost:9222/devtools/browser/\")\n - Automatically set based on browser_mode
\n
5.â debugging_port & host \n - debugging_port: Port for browser debugging protocol (default: 9222)\n - host: Host for browser connection (default: \"localhost\")
\n
6.â proxy_config \n - A ProxyConfig object or dictionary with fields like: \n
7.â viewport_width & viewport_height\n - The initial window size.\n - Some sites behave differently with smaller or bigger viewports.
\n
8.â device_scale_factor\n - Controls the device pixel ratio (DPR) for rendering. Default is 1.0.\n - Set to 2.0 for Retina-quality screenshots (e.g., a 1920Ă1080 viewport produces 3840Ă2160 images).\n - Higher values increase screenshot size and rendering time proportionally.
\n
9.â verbose \n - If True, prints extra logs. \n - Handy for debugging.
\n
9.â use_persistent_context \n - If True, uses a persistent browser profile, storing cookies/local storage across runs. \n - Typically also set user_data_dir to point to a folder.
\n
10.â cookies & headers \n - If you want to start with specific cookies or add universal HTTP headers to the browser context, set them here. \n - E.g. cookies=[{\"name\": \"session\", \"value\": \"abc123\", \"domain\": \"example.com\"}].
\n
11.â user_agent & user_agent_mode \n - user_agent: Custom User-Agent string. If None, a default is used. \n - user_agent_mode: Set to \"random\" for randomization (helps fight bot detection).
\n
12.â text_mode & light_mode\n - text_mode=True disables images, possibly speeding up text-only crawls.\n - light_mode=True turns off certain background features for performance.
\n
13.â avoid_ads & avoid_css\n - avoid_ads=True blocks requests to common ad and tracker domains (Google Analytics, DoubleClick, Facebook, Hotjar, etc.) at the browser context level. Reduces network overhead and memory usage.\n - avoid_css=True blocks loading of CSS files (.css, .less, .scss, .sass), useful when you only need text content and want faster, leaner crawls.\n - Both default to False (opt-in). Can be combined with each other and with text_mode.
\n
14.â extra_args \n - Additional flags for the underlying browser. \n - E.g. [\"--disable-extensions\"].
\n
15.â enable_stealth\n - If True, enables stealth mode using playwright-stealth.\n - Modifies browser fingerprints to avoid basic bot detection.\n - Default is False. Recommended for sites with bot protection.
\n
Helper Methods
\n
Both configuration classes provide a clone() method to create modified copies:
\n
# Create a base browser config\nbase_browser = BrowserConfig(\n browser_type=\"chromium\",\n headless=True,\n text_mode=True\n)\n\n# Create a visible browser config for debugging\ndebug_browser = base_browser.clone(\n headless=False,\n verbose=True\n)\n
\n
Class-Level Defaults
\n
Both BrowserConfig and CrawlerRunConfig support class-level default overrides via set_defaults(). This is useful in server/cloud deployments where every config instance needs the same base settings â set them once at startup instead of repeating at every call site.
\n
from crawl4ai import BrowserConfig, CrawlerRunConfig\n\n# At application startup â one time\nBrowserConfig.set_defaults(\n cache_cdp_connection=True,\n cdp_close_delay=0,\n create_isolated_context=True,\n)\nCrawlerRunConfig.set_defaults(verbose=False)\n\n# Every new instance automatically inherits those defaults\ncfg = BrowserConfig(cdp_url=\"ws://localhost:9222\")\n# â cache_cdp_connection=True, cdp_close_delay=0, create_isolated_context=True\n\n# Explicit values still win\ncfg = BrowserConfig(cdp_url=\"ws://localhost:9222\", cache_cdp_connection=False)\n# â cache_cdp_connection=False (explicitoverrides the classdefault)\n
\n
Available methods (on both BrowserConfig and CrawlerRunConfig):
\n
\n\n
\n
Method
\n
Description
\n
\n\n\n
\n
set_defaults(**kwargs)
\n
Set class-level defaults. Invalid parameter names raise ValueError.
\n
\n
\n
get_defaults()
\n
Return a copy of the current class-level defaults.
\n
\n
\n
reset_defaults()
\n
Clear all class-level defaults.
\n
\n
\n
reset_defaults(\"param1\", \"param2\")
\n
Clear only the named defaults.
\n
\n\n
\n
\n
Note: Class defaults are independent per class â BrowserConfig.set_defaults() does not affect CrawlerRunConfig, and vice versa. Defaults are stored in memory and apply for the lifetime of the process.
\n
\n
Minimal Example:
\n
from crawl4ai import AsyncWebCrawler, BrowserConfig\n\nbrowser_conf = BrowserConfig(\n browser_type=\"firefox\",\n headless=False,\n text_mode=True\n)\n\nasyncwith AsyncWebCrawler(config=browser_conf) as crawler:\n result = await crawler.arun(\"https://example.com\")\n print(result.markdown[:300])\n
1.â word_count_threshold: \n - The minimum word count before a block is considered. \n - If your site has lots of short paragraphs or items, you can lower it.
\n
2.â extraction_strategy: \n - Where you plug in JSON-based extraction (CSS, LLM, etc.). \n - If None, no structured extraction is done (only raw/cleaned HTML + markdown).
\n
3.â chunking_strategy: \n - Strategy to chunk content before extraction. \n - Defaults to RegexChunking(). Can be customized for different chunking approaches.
\n
4.â markdown_generator: \n - E.g., DefaultMarkdownGenerator(...), controlling how HTMLâMarkdown conversion is done. \n - If None, a default approach is used.
6.â js_code, js_code_before_wait, & c4a_script:\n - js_code: JavaScript to run afterwait_for completes â on the fully-loaded page.\n - js_code_before_wait: JavaScript to run beforewait_for â for triggering loading that wait_for then checks.\n - c4a_script: C4A script that compiles to JavaScript.\n - Great for \"Load More\" buttons or user interactions.
\n
7.â wait_for: \n - A CSS or JS expression to wait for before extracting content. \n - Common usage: wait_for=\"css:.main-loaded\" or wait_for=\"js:() => window.loaded === true\".
\n
8.â flatten_shadow_dom:\n - If True, flattens Shadow DOM content into the light DOM before HTML capture.\n - Essential for sites built with Web Components (Stencil, Lit, Shoelace, etc.).\n - Also force-opens closed shadow roots. See Flattening Shadow DOM.
\n
9.â screenshot, pdf, & capture_mhtml:\n - If True, captures a screenshot, PDF, or MHTML snapshot after the page is fully loaded.\n - The results go to result.screenshot (base64), result.pdf (bytes), or result.mhtml (string).\n - Use force_viewport_screenshot=True to capture only the visible viewport instead of the full page. This is faster and produces smaller images when you don't need a full-page screenshot.
\n
9.â Location Parameters: \n - locale: Browser's locale (e.g., \"en-US\", \"fr-FR\") for language preferences\n - timezone_id: Browser's timezone (e.g., \"America/New_York\", \"Europe/Paris\")\n - geolocation: GPS coordinates via GeolocationConfig(latitude=48.8566, longitude=2.3522)\n - See Identity Based Crawling
\n
10.â Proxy Configuration:\n - proxy_config: Single ProxyConfig or list[ProxyConfig] â proxies tried in order. Pass a list for automatic escalation.\n - proxy_rotation_strategy: Strategy for rotating proxies during crawls
\n
11.â Anti-Bot Retry & Fallback (see Anti-Bot & Fallback):\n - max_retries: Number of retry rounds when blocking is detected (default: 0). Each round tries all proxies in proxy_config.\n - fallback_fetch_function: Async function called as last resort â takes URL, returns raw HTML
\n
12.â Page Interaction Parameters:\n - scan_full_page: If True, scroll through the entire page to load all content\n - wait_until: Condition to wait for when navigating (e.g., \"domcontentloaded\", \"networkidle\")\n - page_timeout: Timeout in milliseconds for page operations (default: 60000)\n - delay_before_return_html: Delay in seconds before retrieving final HTML.
\n
13.â url_matcher & match_mode: \n - Enable URL-specific configurations when used with arun_many().\n - Set url_matcher to patterns (glob, function, or list) to match specific URLs.\n - Use match_mode (OR/AND) to control how multiple patterns combine.\n - See URL-Specific Configurations for examples.
\n
13.â verbose: \n - Logs additional runtime details. \n - Overlaps with the browser's verbosity if also set to True in BrowserConfig.
\n
14.â stream: \n - If True, enables streaming mode for arun_many() to process URLs as they complete.\n - Allows handling results incrementally instead of waiting for all URLs to finish.
\n
Helper Methods
\n
The clone() method is particularly useful for creating variations of your crawler configuration:
\n
# Create a base configuration\nbase_config = CrawlerRunConfig(\n cache_mode=CacheMode.ENABLED,\n word_count_threshold=200,\n wait_until=\"networkidle\"\n)\n\n# Create variations for different use cases\nstream_config = base_config.clone(\n stream=True, # Enable streaming mode\n cache_mode=CacheMode.BYPASS\n)\n\ndebug_config = base_config.clone(\n page_timeout=120000, # Longer timeout for debugging\n verbose=True\n)\n
\n
The clone() method:\n- Creates a new instance with all the same settings\n- Updates only the specified parameters\n- Leaves the original configuration unchanged\n- Perfect for creating variations without repeating all parameters
\n\n
3. LLMConfig Essentials
\n
Key fields to note
\n
1.â provider: \n- Which LLM provider to use. \n- Possible values are \"ollama/llama3\",\"groq/llama3-70b-8192\",\"groq/llama3-8b-8192\", \"openai/gpt-4o-mini\" ,\"openai/gpt-4o\",\"openai/o1-mini\",\"openai/o1-preview\",\"openai/o3-mini\",\"openai/o3-mini-high\",\"anthropic/claude-3-haiku-20240307\",\"anthropic/claude-3-opus-20240229\",\"anthropic/claude-3-sonnet-20240229\",\"anthropic/claude-3-5-sonnet-20240620\",\"gemini/gemini-pro\",\"gemini/gemini-1.5-pro\",\"gemini/gemini-2.0-flash\",\"gemini/gemini-2.0-flash-exp\",\"gemini/gemini-2.0-flash-lite-preview-02-05\",\"deepseek/deepseek-chat\" (default: \"openai/gpt-4o-mini\")
\n
2.â api_token: \n - Optional. When not provided explicitly, api_token will be read from environment variables based on provider. For example: If a gemini model is passed as provider then,\"GEMINI_API_KEY\" will be read from environment variables \n - API token of LLM provider eg: api_token = \"gsk_1ClHGGJ7Lpn4WGybR7vNWGdyb3FY7zXEw3SCiy0BAVM9lL8CQv\"\n - Environment variable - use with prefix \"env:\" eg:api_token = \"env: GROQ_API_KEY\"
\n
3.â base_url: \n - If your provider has a custom endpoint
\n
4.â Retry/backoff controls(optional): \n - backoff_base_delay(default 2 seconds) â base delay inserted before the first retry when the provider returns a rate-limit response. \n - backoff_max_attempts(default 3) â total number of attempts (initial call plus retries) before the request is surfaced as an error. \n - backoff_exponential_factor(default 2) â growth rate for the retry delay (delay = base_delay * factor^attempt). \n - These values are forwarded to the shared perform_completion_with_backoff helper, ensuring every strategy that consumes your LLMConfig honors the same throttling policy.
In a typical scenario, you define oneBrowserConfig for your crawler session, then create one or moreCrawlerRunConfig & LLMConfig depending on each call's needs:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig, LLMContentFilter, DefaultMarkdownGenerator\nfrom crawl4ai import JsonCssExtractionStrategy\n\nasyncdefmain():\n # 1) Browser config: headless, bigger viewport, no proxy\n browser_conf = BrowserConfig(\n headless=True,\n viewport_width=1280,\n viewport_height=720\n )\n\n # 2) Example extraction strategy\n schema = {\n \"name\": \"Articles\",\n \"baseSelector\": \"div.article\",\n \"fields\": [\n {\"name\": \"title\", \"selector\": \"h2\", \"type\": \"text\"},\n {\"name\": \"link\", \"selector\": \"a\", \"type\": \"attribute\", \"attribute\": \"href\"}\n ]\n }\n extraction = JsonCssExtractionStrategy(schema)\n\n # 3) Example LLM content filtering\n\n gemini_config = LLMConfig(\n provider=\"gemini/gemini-1.5-pro\", \n api_token = \"env:GEMINI_API_TOKEN\"\n )\n\n # Initialize LLM filter with specific instruction\n filter = LLMContentFilter(\n llm_config=gemini_config, # or your preferred provider\n instruction=\"\"\"\n Focus on extracting the core educational content.\n Include:\n - Key concepts and explanations\n - Important code examples\n - Essential technical details\n Exclude:\n - Navigation elements\n - Sidebars\n - Footer content\n Format the output as clean markdown with proper code blocks and headers.\n \"\"\",\n chunk_token_threshold=500, # Adjust based on your needs\n verbose=True\n )\n\n md_generator = DefaultMarkdownGenerator(\n content_filter=filter,\n options={\"ignore_links\": True}\n )\n\n # 4) Crawler run config: skip cache, use extraction\n run_conf = CrawlerRunConfig(\n markdown_generator=md_generator,\n extraction_strategy=extraction,\n cache_mode=CacheMode.BYPASS,\n )\n\n asyncwith AsyncWebCrawler(config=browser_conf) as crawler:\n # 4) Execute the crawl\n result = await crawler.arun(url=\"https://example.com/news\", config=run_conf)\n\n if result.success:\n print(\"Extracted content:\", result.extracted_content)\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n
5. Next Steps
\n
For a detailed list of available parameters (including advanced ones), see:
BrowserConfig, CrawlerRunConfig and LLMConfig give you straightforward ways to define:
\n
\n
Which browser to launch, how it should run, and any proxy or user agent needs.
\n
How each crawl should behaveâcaching, timeouts, JavaScript code, extraction strategies, etc.
\n
Which LLM provider to use, api token, temperature and base url for custom endpoints
\n
\n
Use them together for clear, maintainable code, and when you need more specialized behavior, check out the advanced parameters in the reference docs. Happy crawling!
C4A-Script is a powerful, human-readable domain-specific language (DSL) designed for web automation and interaction. Think of it as a simplified programming language that anyone can read and write, perfect for automating repetitive web tasks, testing user interfaces, or creating interactive demos.
\n
Why C4A-Script?
\n
Simple Syntax, Powerful Results\n
# Navigate and interact in plain English\nGO https://example.com\nWAIT `#search-box` 5\nTYPE \"Hello World\"\nCLICK `button[type=\"submit\"]`\n
\n
Visual Programming Support\nC4A-Script comes with a built-in Blockly visual editor, allowing you to create scripts by dragging and dropping blocks - no coding experience required!
\n
Perfect for:\n- UI Testing: Automate user interaction flows\n- Demo Creation: Build interactive product demonstrations \n- Data Entry: Automate form filling and submissions\n- Testing Workflows: Validate complex user journeys\n- Training: Teach web automation without code complexity
\n
Getting Started: Your First Script
\n
Let's create a simple script that searches for something on a website:
\n
# My first C4A-Script\nGO https://duckduckgo.com\n\n# Wait for the search box to appear\nWAIT `input[name=\"q\"]` 10\n\n# Type our search query\nTYPE\"Crawl4AI\"\n\n# Press Enter to search\nPRESS Enter\n\n# Wait for results\nWAIT `.results` 5\n
\n
That's it! In just a few lines, you've automated a complete search workflow.
\n
Interactive Tutorial & Live Demo
\n
Want to learn by doing? We've got you covered:
\n
đ Live Demo - Try C4A-Script in your browser right now!
The tutorial includes a Flask-based web interface with:\n- Live Code Editor with syntax highlighting\n- Visual Blockly Editor for drag-and-drop programming\n- Recording Mode to capture your actions and generate scripts\n- Timeline View to see and edit your automation steps
\n
# Clone and navigate to the tutorial\ncd docs/examples/c4a_script/tutorial/\n\n# Install dependencies\npip install -r requirements.txt\n\n# Launch the tutorial server\npython server.py\n\n# Open http://localhost:8000 in your browser\n
\n
Core Concepts
\n
Commands and Syntax
\n
C4A-Script uses simple, English-like commands. Each command does one specific thing:
\n
# Comments start with #\nCOMMAND parameter1 parameter2\n\n# Most commands use CSS selectors in backticks\nCLICK `#submit-button`\n\n# Text content goes in quotes\nTYPE\"Hello, World!\"\n\n# Numbers are used directly\nWAIT 3\n
\n
Selectors: Finding Elements
\n
C4A-Script uses CSS selectors to identify elements on the page:
\n
# By ID\nCLICK `#login-button`\n\n# By class\nCLICK `.submit-btn`\n\n# By attribute\nCLICK `button[type=\"submit\"]`\n\n# By accessible attributes\nCLICK `button[aria-label=\"Search\"][title=\"Search\"]`\n\n# Complex selectors\nCLICK `.form-container input[name=\"email\"]`\n
\n
Variables and Dynamic Content
\n
Store and reuse values with variables:
\n
# Set a variable\nSETVAR username = \"john@example.com\"\nSETVAR password = \"secret123\"\n\n# Use variables (prefix with $)\nTYPE $username\nPRESS Tab\nTYPE $password\n
\n
Command Categories
\n
đ§ Navigation Commands
\n
Move around the web like a user would:
\n
\n\n
\n
Command
\n
Purpose
\n
Example
\n
\n\n\n
\n
GO
\n
Navigate to URL
\n
GO https://example.com
\n
\n
\n
RELOAD
\n
Refresh current page
\n
RELOAD
\n
\n
\n
BACK
\n
Go back in history
\n
BACK
\n
\n
\n
FORWARD
\n
Go forward in history
\n
FORWARD
\n
\n\n
\n
âąď¸ Wait Commands
\n
Ensure elements are ready before interacting:
\n
\n\n
\n
Command
\n
Purpose
\n
Example
\n
\n\n\n
\n
WAIT
\n
Wait for time/element/text
\n
WAIT 3 or WAIT \\#element` 10`
\n
\n\n
\n
đąď¸ Mouse Commands
\n
Click, drag, and move like a human:
\n
\n\n
\n
Command
\n
Purpose
\n
Example
\n
\n\n\n
\n
CLICK
\n
Click element or coordinates
\n
CLICK \\button`orCLICK 100 200`
\n
\n
\n
DOUBLE_CLICK
\n
Double-click element
\n
DOUBLE_CLICK \\.item``
\n
\n
\n
RIGHT_CLICK
\n
Right-click element
\n
RIGHT_CLICK \\#menu``
\n
\n
\n
SCROLL
\n
Scroll in direction
\n
SCROLL DOWN 500
\n
\n
\n
DRAG
\n
Drag from point to point
\n
DRAG 100 100 500 300
\n
\n\n
\n
â¨ď¸ Keyboard Commands
\n
Type text and press keys naturally:
\n
\n\n
\n
Command
\n
Purpose
\n
Example
\n
\n\n\n
\n
TYPE
\n
Type text or variable
\n
TYPE \"Hello\" or TYPE $username
\n
\n
\n
PRESS
\n
Press special keys
\n
PRESS Tab or PRESS Enter
\n
\n
\n
CLEAR
\n
Clear input field
\n
CLEAR \\#search``
\n
\n
\n
SET
\n
Set input value directly
\n
SET \\#email` \"user@example.com\"`
\n
\n\n
\n
đ Control Flow
\n
Add logic and repetition to your scripts:
\n
\n\n
\n
Command
\n
Purpose
\n
Example
\n
\n\n\n
\n
IF
\n
Conditional execution
\n
IF (EXISTS \\#popup`) THEN CLICK `#close``
\n
\n
\n
REPEAT
\n
Loop commands
\n
REPEAT (SCROLL DOWN 300, 5)
\n
\n\n
\n
đž Variables & Advanced
\n
Store data and execute custom code:
\n
\n\n
\n
Command
\n
Purpose
\n
Example
\n
\n\n\n
\n
SETVAR
\n
Create variable
\n
SETVAR email = \"test@example.com\"
\n
\n
\n
EVAL
\n
Execute JavaScript
\n
EVAL \\console.log('Hello')``
\n
\n\n
\n
Real-World Examples
\n
Example 1: Login Flow
\n
# Complete login automation\nGO https://myapp.com/login\n\n# Wait for page to load\nWAIT `#login-form` 5\n\n# Fill credentials\nCLICK `#email`\nTYPE\"user@example.com\"\nPRESS Tab\nTYPE\"mypassword\"\n\n# Submit form\nCLICK `button[type=\"submit\"]`\n\n# Wait for dashboard\nWAIT `.dashboard` 10\n
\n
Example 2: E-commerce Shopping
\n
# Shopping automation with variables\nSETVAR product = \"laptop\"\nSETVAR budget = \"1000\"\n\nGO https://shop.example.com\nWAIT `#search-box`3\n\n# Search for product\nTYPE $product\nPRESS Enter\nWAIT `.product-list`5\n\n# Filter by price\nCLICK `.price-filter`\nSET `#max-price` $budget\nCLICK `.apply-filters`\n\n# Select first result\nWAIT `.product-item`3\nCLICK `.product-item:first-child`\n
\n
Example 3: Form Automation with Conditions
\n
# Smart form filling with error handling\nGO https://forms.example.com\n\n# Check if user is already logged in\nIF (EXISTS `.user-menu`) THEN GO https://forms.example.com/new\nIF (NOT EXISTS `.user-menu`) THEN CLICK `#login-link`\n\n# Fill form\nWAIT `#contact-form`5\nSET `#name`\"John Doe\"\nSET `#email`\"john@example.com\"\nSET `#message`\"Hello from C4A-Script!\"\n\n# Handle popup if it appears\nIF (EXISTS `.cookie-banner`) THEN CLICK `.accept-cookies`\n\n# Submit\nCLICK `#submit-button`\nWAIT `.success-message`10\n
\n
Visual Programming with Blockly
\n
C4A-Script includes a powerful visual programming interface built on Google Blockly. Perfect for:
\n
\n
Non-programmers who want to create automation
\n
Rapid prototyping of automation workflows
\n
Educational environments for teaching automation concepts
\n
Collaborative development where visual representation helps communication
\n
\n
Features:
\n
\n
Drag & Drop Interface: Build scripts by connecting blocks
\n
Real-time Sync: Changes in visual mode instantly update the text script
\n
Smart Block Types: Blocks are categorized by function (Navigation, Actions, etc.)
The tutorial interface includes a recording feature that watches your browser interactions and automatically generates C4A-Script commands:
\n\n
Click \"Record\" in the tutorial interface
\n
Perform actions in the browser preview
\n
Watch as C4A-Script commands are generated in real-time
\n
Edit and refine the generated script
\n\n
Error Handling and Debugging
\n
C4A-Script provides clear error messages and debugging information:
\n
# Use comments for debugging\n# This will wait up to 10 seconds for the element\nWAIT `#slow-loading-element` 10\n\n# Check if element exists before clicking\nIF (EXISTS `#optional-button`) THEN CLICK `#optional-button`\n\n# Use EVAL for custom debugging\nEVAL `console.log(\"Current page title:\", document.title)`\n
\n
Integration with Crawl4AI
\n
C4A-Script integrates seamlessly with Crawl4AI's web crawling capabilities:
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\n# Use C4A-Script for interaction before crawling\nscript = \"\"\"\nGO https://example.com\nCLICK `#load-more-content`\nWAIT `.dynamic-content` 5\n\"\"\"\n\nconfig = CrawlerRunConfig(\n js_code=script,\n wait_for=\".dynamic-content\"\n)\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n print(result.markdown)\n
\n
Best Practices
\n
1. Always Wait for Elements
\n
# Bad: Clicking immediately\nCLICK `#button`\n\n# Good: Wait for element to appear\nWAIT `#button` 5\nCLICK `#button`\n
\n
2. Use Descriptive Comments
\n
# Login to user account\nGO https://myapp.com/login\nWAIT `#login-form`5\n\n# Enter credentials\nTYPE \"user@example.com\"\nPRESS Tab\nTYPE \"password123\"\n\n# Submit and wait for redirect\nCLICK `#submit-button`\nWAIT `.dashboard`10\n
\n
3. Handle Variable Conditions
\n
# Handle different page states\nIF (EXISTS `.cookie-banner`) THEN CLICK `.accept-cookies`\nIF (EXISTS `.popup-modal`) THEN CLICK `.close-modal`\n\n# Proceed with main workflow\nCLICK `#main-action`\n
C4A-Script makes web automation accessible to everyone. Whether you're a developer automating tests, a designer creating interactive demos, or a business user streamlining repetitive tasks, C4A-Script has the tools you need.
\n
Start automating today - your future self will thank you! đ
Starting from version 0.5.0, Crawl4AI introduces a new caching system that replaces the old boolean flags with a more intuitive CacheMode enum. This change simplifies cache control and makes the behavior more predictable.
\n
Old vs New Approach
\n
Old Way (Deprecated)
\n
The old system used multiple boolean flags:\n- bypass_cache: Skip cache entirely\n- disable_cache: Disable all caching\n- no_cache_read: Don't read from cache\n- no_cache_write: Don't write to cache
\n
New Way (Recommended)
\n
The new system uses a single CacheMode enum:\n- CacheMode.ENABLED: Normal caching (read/write)\n- CacheMode.DISABLED: No caching at all\n- CacheMode.READ_ONLY: Only read from cache\n- CacheMode.WRITE_ONLY: Only write to cache\n- CacheMode.BYPASS: Skip cache for this operation
\n
Migration Example
\n
Old Code (Deprecated)
\n
from crawl4ai import AsyncWebCrawler\n\nasyncdefold_code(crawler: AsyncWebCrawler):\n # Legacy `bypass_cache` / `disable_cache` / `no_cache_read` / `no_cache_write`\n # were removed in v0.5+. This example no longer applies:\n result = await crawler.arun(\n url=\"https://www.nbcnews.com/business\",\n # cache_mode is the only cache option now.\n )\n print(len(result.markdown))\n
\n
New Code (Recommended)
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CacheMode\nfrom crawl4ai.async_configs import CrawlerRunConfig\n\nasyncdefuse_proxy():\n # Use CacheMode in CrawlerRunConfig\n config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS) \n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n result = await crawler.arun(\n url=\"https://www.nbcnews.com/business\",\n config=config # Pass the configuration object\n )\n print(len(result.markdown))\n\nasyncdefmain():\n await use_proxy()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
# Using config file\ncrwl https://example.com -B browser.yml\n\n# Using direct parameters\ncrwl https://example.com -b \"headless=true,viewport_width=1280,user_agent_mode=random\"\n
# Using config file\ncrwl https://example.com -C crawler.yml\n\n# Using direct parameters\ncrwl https://example.com -c \"css_selector=#main,delay_before_return_html=2,scan_full_page=true\"\n
# extract_llm.yml\ntype:\"llm\"\nprovider:\"openai/gpt-4\"\ninstruction:\"Extract all articles with their titles and links\"\napi_token:\"your-token\"\nparams:\n temperature: 0.3\n max_tokens: 1000\n
\n\n
// llm_schema.json\n{\n \"title\":\"Article\",\n \"type\":\"object\",\n \"properties\":{\n \"title\":{\n \"type\":\"string\",\n \"description\":\"The title of the article\"\n },\n \"link\":{\n \"type\":\"string\",\n \"description\":\"URL to the full article\"\n }\n }\n}\n
\n
Advanced Features
\n
LLM Q&A
\n
Ask questions about crawled content:
\n
# Simple question\ncrwl https://example.com -q \"What is the main topic discussed?\"\n\n# View content then ask questions\ncrwl https://example.com -o markdown # See content first\ncrwl https://example.com -q \"Summarize the key points\"\ncrwl https://example.com -q \"What are the conclusions?\"\n\n# Combined with advanced crawling\ncrwl https://example.com \\\n -B browser.yml \\\n -c \"css_selector=article,scan_full_page=true\" \\\n -q \"What are the pros and cons mentioned?\"\n
\n
First-time setup:\n- Prompts for LLM provider and API token\n- Saves configuration in ~/.crawl4ai/global.yml\n- Supports various providers (openai/gpt-4, anthropic/claude-3-sonnet, etc.)\n- For case of ollama you do not need to provide API token.\n- See LiteLLM Providers for full list
# First crawl and view\ncrwl https://example.com -o markdown\n\n# Then ask questions\ncrwl https://example.com -q \"What are the main points?\"\ncrwl https://example.com -q \"Summarize the conclusions\"\n
\n
\n\n
Best Practices & Tips
\n\n
Configuration Management:
\n
Keep common configurations in YAML files
\n
Use CLI parameters for quick overrides
\n
\n
Store sensitive data (API tokens) in ~/.crawl4ai/global.yml
\n
\n
\n
Performance Optimization:
\n
\n
Use --bypass-cache for fresh content
\n
Enable scan_full_page for infinite scroll pages
\n
\n
Adjust delay_before_return_html for dynamic content
\n
\n
\n
Content Extraction:
\n
\n
Use CSS extraction for structured content
\n
Use LLM extraction for unstructured content
\n
\n
Combine with filters for focused results
\n
\n
\n
Q&A Workflow:
\n
\n
View content first with -o markdown
\n
Ask specific questions
\n
Use broader context with appropriate selectors
\n\n
Recap
\n
The Crawl4AI CLI provides:\n- Flexible configuration via files and parameters\n- Multiple extraction strategies (CSS, XPath, LLM)\n- Content filtering and optimization\n- Interactive Q&A capabilities\n- Various output formats
Crawl4AI provides multiple ways to select, filter, and refine the content from your crawls. Whether you need to target a specific CSS region, exclude entire tags, filter out external links, or remove certain domains and images, CrawlerRunConfig offers a wide range of parameters.
\n
Below, we show how to configure these parameters and combine them for precise control.
\n\n
1. CSS-Based Selection
\n
There are two ways to select content from a page: using css_selector or the more flexible target_elements.
\n
1.1 Using css_selector
\n
A straightforward way to limit your crawl results to a certain region of the page is css_selector in CrawlerRunConfig:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # e.g., first 30 items from Hacker News\n css_selector=\".athing:nth-child(-n+30)\" \n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com/newest\", \n config=config\n )\n print(\"Partial HTML length:\", len(result.cleaned_html))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Result: Only elements matching that selector remain in result.cleaned_html.
\n
1.2 Using target_elements
\n
The target_elements parameter provides more flexibility by allowing you to target multiple elements for content extraction while preserving the entire page context for other features:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # Target article body and sidebar, but not other content\n target_elements=[\"article.main-content\", \"aside.sidebar\"]\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/blog-post\", \n config=config\n )\n print(\"Markdown focused on target elements\")\n print(\"Links from entire page still available:\", len(result.links.get(\"internal\", [])))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Key difference: With target_elements, the markdown generation and structural data extraction focus on those elements, but other page elements (like links, images, and tables) are still extracted from the entire page. This gives you fine-grained control over what appears in your markdown content while preserving full page context for link analysis and media collection.
\n\n
2. Content Filtering & Exclusions
\n
2.1 Basic Overview
\n
config = CrawlerRunConfig(\n # Content thresholds\n word_count_threshold=10, # Minimum words per block\n\n # Tag exclusions\n excluded_tags=['form', 'header', 'footer', 'nav'],\n\n # Link filtering\n exclude_external_links=True, \n exclude_social_media_links=True,\n # Block entire domains\n exclude_domains=[\"adtrackers.com\", \"spammynews.org\"], \n exclude_social_media_domains=[\"facebook.com\", \"twitter.com\"],\n\n # Media filtering\n exclude_external_images=True\n)\n
\n
Explanation:
\n
\n
word_count_threshold: Ignores text blocks under X words. Helps skip trivial blocks like short nav or disclaimers.
Sites built with Web Components (Stencil, Lit, Shoelace, Angular Elements, etc.) render content inside Shadow DOM â an encapsulated sub-tree that is invisible to normal page serialization. The browser renders it on screen, but page.content() never includes it.
\n
Set flatten_shadow_dom=True to walk all shadow trees, resolve <slot> projections, and produce a single flat HTML document:
\n
config = CrawlerRunConfig(\n # Flatten shadow DOM into the main document\n flatten_shadow_dom=True,\n # Give web components time to hydrate\n wait_until=\"load\",\n delay_before_return_html=3.0,\n)\n
\n
Full example â crawling a product page where specs live inside shadow roots:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n flatten_shadow_dom=True,\n wait_until=\"load\",\n delay_before_return_html=3.0,\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011\",\n config=config,\n )\n # Without flatten_shadow_dom: ~1 KB of markdown (breadcrumbs only)\n # With flatten_shadow_dom: ~33 KB (full product specs, downloads, etc.)\n print(len(result.markdown.raw_markdown))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
When flatten_shadow_dom=True is set, Crawl4AI also injects an init script that force-opens closed shadow roots (by patching Element.prototype.attachShadow), so even components that use mode: 'closed' become accessible.
\n
\n
Tip: Web components need JavaScript to run before they render content (a process called hydration). Use wait_until=\"load\" and a delay_before_return_html of 2â5 seconds to ensure components are fully hydrated before flattening.
You can combine content selection with a more advanced extraction strategy. For instance, a CSS-based or LLM-based extraction strategy can run on the filtered HTML.
import asyncio\nimport json\nfrom pydantic import BaseModel, Field\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\n\nclassArticleData(BaseModel):\n headline: str\n summary: str\n\nasyncdefmain():\n llm_strategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"openai/gpt-4\",api_token=\"sk-YOUR_API_KEY\")\n schema=ArticleData.schema(),\n extraction_type=\"schema\",\n instruction=\"Extract 'headline' and a short 'summary' from the content.\"\n )\n\n config = CrawlerRunConfig(\n exclude_external_links=True,\n word_count_threshold=20,\n extraction_strategy=llm_strategy\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=\"https://news.ycombinator.com\", config=config)\n article = json.loads(result.extracted_content)\n print(article)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Here, the crawler:
\n
\n
Filters out external links (exclude_external_links=True).
\n
Ignores very short text blocks (word_count_threshold=20).
\n
Passes the final HTML to your LLM strategy for an AI-driven parse.
\n
\n\n
5. Comprehensive Example
\n
Below is a short function that unifies CSS selection, exclusion logic, and a pattern-based extraction, demonstrating how you can fine-tune your final data:
Why This Works:\n- CSS scoping with #main-content. \n- Multiple exclude_ parameters to remove domains, external images, etc. \n- A JsonCssExtractionStrategy to parse repeated article blocks.
\n\n
6. Scraping Modes
\n
Crawl4AI uses LXMLWebScrapingStrategy (LXML-based) as the default scraping strategy for HTML content processing. This strategy offers excellent performance, especially for large HTML documents.
\n
Note: For backward compatibility, WebScrapingStrategy is still available as an alias for LXMLWebScrapingStrategy.
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LXMLWebScrapingStrategy\n\nasyncdefmain():\n # Default configuration already uses LXMLWebScrapingStrategy\n config = CrawlerRunConfig()\n\n # Or explicitly specify it if desired\n config_explicit = CrawlerRunConfig(\n scraping_strategy=LXMLWebScrapingStrategy()\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\", \n config=config\n )\n
\n
You can also create your own custom scraping strategy by inheriting from ContentScrapingStrategy. The strategy must return a ScrapingResult object with the following structure:
\n
from crawl4ai import ContentScrapingStrategy, ScrapingResult, MediaItem, Media, Link, Links\n\nclassCustomScrapingStrategy(ContentScrapingStrategy):\n defscrap(self, url: str, html: str, **kwargs) -> ScrapingResult:\n # Implement your custom scraping logic here\n return ScrapingResult(\n cleaned_html=\"<html>...</html>\", # Cleaned HTML content\n success=True, # Whether scraping was successful\n media=Media(\n images=[ # List of images found\n MediaItem(\n src=\"https://example.com/image.jpg\",\n alt=\"Image description\",\n desc=\"Surrounding text\",\n score=1,\n type=\"image\",\n group_id=1,\n format=\"jpg\",\n width=800\n )\n ],\n videos=[], # List of videos (same structure as images)\n audios=[] # List of audio files (same structure as images)\n ),\n links=Links(\n internal=[ # List of internal links\n Link(\n href=\"https://example.com/page\",\n text=\"Link text\",\n title=\"Link title\",\n base_domain=\"example.com\"\n )\n ],\n external=[] # List of external links (same structure)\n ),\n metadata={ # Additional metadata\n \"title\": \"Page Title\",\n \"description\": \"Page description\"\n }\n )\n\n asyncdefascrap(self, url: str, html: str, **kwargs) -> ScrapingResult:\n # For simple cases, you can use the sync version\n returnawait asyncio.to_thread(self.scrap, url, html, **kwargs)\n
\n
Performance Considerations
\n
The LXML strategy provides excellent performance, particularly when processing large HTML documents, offering up to 10-20x faster processing compared to BeautifulSoup-based approaches.
\n
Benefits of LXML strategy:\n- Fast processing of large HTML documents (especially >100KB)\n- Efficient memory usage\n- Good handling of well-formed HTML\n- Robust table detection and extraction
\n
Backward Compatibility
\n
For users upgrading from earlier versions:\n- WebScrapingStrategy is now an alias for LXMLWebScrapingStrategy\n- Existing code using WebScrapingStrategy will continue to work without modification\n- No changes are required to your existing code
\n\n
7. Combining CSS Selection Methods
\n
You can combine css_selector and target_elements in powerful ways to achieve fine-grained control over your output:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n # Target specific content but preserve page context\n config = CrawlerRunConfig(\n # Focus markdown on main content and sidebar\n target_elements=[\"#main-content\", \".sidebar\"],\n\n # Global filters applied to entire page\n excluded_tags=[\"nav\", \"footer\", \"header\"],\n exclude_external_links=True,\n\n # Use basic content thresholds\n word_count_threshold=15,\n\n cache_mode=CacheMode.BYPASS\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/article\",\n config=config\n )\n\n print(f\"Content focuses on specific elements, but all links still analyzed\")\n print(f\"Internal links: {len(result.links.get('internal', []))}\")\n print(f\"External links: {len(result.links.get('external', []))}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
This approach gives you the best of both worlds:\n- Markdown generation and content extraction focus on the elements you care about\n- Links, images and other page data still give you the full context of the page\n- Content filtering still applies globally
\n
8. Conclusion
\n
By mixing target_elements or css_selector scoping, content filtering parameters, and advanced extraction strategies, you can precisely choose which data to keep. Key parameters in CrawlerRunConfig for content selection include:
\n\n
target_elements â Array of CSS selectors to focus markdown generation and data extraction, while preserving full page context for links and media.
\n
css_selector â Basic scoping to an element or region for all extraction processes.
\n
word_count_threshold â Skip short blocks.
\n
excluded_tags â Remove entire HTML tags.
\n
exclude_external_links, exclude_social_media_links, exclude_domains â Filter out unwanted links or domains.
\n
exclude_external_images â Remove images from external sources.
\n
process_iframes â Merge iframe content if needed.
\n\n
Combine these with structured extraction (CSS, LLM-based, or others) to build powerful crawls that yield exactly the content you want, from raw or cleaned HTML up to sophisticated JSON structures. For more detail, see Configuration Reference. Enjoy curating your data to the max!
When you call arun() on a page, Crawl4AI returns a CrawlResult object containing everything you might needâraw HTML, a cleaned version, optional screenshots or PDFs, structured extraction results, and more. This document explains those fields and how they map to different output types.
\n\n
1. The CrawlResult Model
\n
Below is the core schema. Each field captures a different aspect of the crawlâs result:
The final or actual URL crawled (in case of redirects).
\n
\n
\n
html (str)
\n
Original, unmodified page HTML. Good for debugging or custom processing.
\n
\n
\n
fit_html (Optional[str])
\n
Preprocessed HTML optimized for extraction and content filtering.
\n
\n
\n
success (bool)
\n
True if the crawl completed without major errors, else False.
\n
\n
\n
cleaned_html (Optional[str])
\n
Sanitized HTML with scripts/styles removed; can exclude tags if configured via excluded_tags etc.
\n
\n
\n
media (Dict[str, List[Dict]])
\n
Extracted media info (images, audio, etc.), each with attributes like src, alt, score, etc.
\n
\n
\n
links (Dict[str, List[Dict]])
\n
Extracted link data, split by internal and external. Each link usually has href, text, etc.
\n
\n
\n
downloaded_files (Optional[List[str]])
\n
If accept_downloads=True in BrowserConfig, this lists the filepaths of saved downloads.
\n
\n
\n
js_execution_result (Optional[Dict[str, Any]])
\n
Results from JavaScript execution during crawling.
\n
\n
\n
screenshot (Optional[str])
\n
Screenshot of the page (base64-encoded) if screenshot=True.
\n
\n
\n
pdf (Optional[bytes])
\n
PDF of the page if pdf=True.
\n
\n
\n
mhtml (Optional[str])
\n
MHTML snapshot of the page if capture_mhtml=True. Contains the full page with all resources.
\n
\n
\n
markdown (Optional[str or MarkdownGenerationResult])
\n
It holds a MarkdownGenerationResult. Over time, this will be consolidated into markdown. The generator can provide raw markdown, citations, references, and optionally fit_markdown.
\n
\n
\n
extracted_content (Optional[str])
\n
The output of a structured extraction (CSS/LLM-based) stored as JSON string or other text.
\n
\n
\n
metadata (Optional[dict])
\n
Additional info about the crawl or extracted data.
\n
\n
\n
error_message (Optional[str])
\n
If success=False, contains a short description of what went wrong.
\n
\n
\n
session_id (Optional[str])
\n
The ID of the session used for multi-page or persistent crawling.
\n
\n
\n
response_headers (Optional[dict])
\n
HTTP response headers, if captured.
\n
\n
\n
status_code (Optional[int])
\n
HTTP status code (e.g., 200 for OK).
\n
\n
\n
ssl_certificate (Optional[SSLCertificate])
\n
SSL certificate info if fetch_ssl_certificate=True.
\n
\n
\n
dispatch_result (Optional[DispatchResult])
\n
Additional concurrency and resource usage information when crawling URLs in parallel.
\n
\n
\n
redirected_url (Optional[str])
\n
The URL after any redirects (different from url which is the final URL).
\n
\n
\n
redirected_status_code (Optional[int])
\n
HTTP status code of the final redirect destination (e.g., 200). None for non-HTTP requests (raw HTML, local files).
\n
\n
\n
network_requests (Optional[List[Dict[str, Any]]])
\n
List of network requests, responses, and failures captured during the crawl if capture_network_requests=True.
\n
\n
\n
console_messages (Optional[List[Dict[str, Any]]])
\n
List of browser console messages captured during the crawl if capture_console_messages=True.
\n
\n
\n
tables (List[Dict])
\n
Table data extracted from HTML tables with structure [{headers, rows, caption, summary}].
\n
\n\n
\n\n
2. HTML Variants
\n
html: Raw HTML
\n
Crawl4AI preserves the exact HTML as result.html. Useful for:
\n
\n
Debugging page issues or checking the original content.
\n
Performing your own specialized parse if needed.
\n
\n
cleaned_html: Sanitized
\n
If you specify any cleanup or exclusion parameters in CrawlerRunConfig (like excluded_tags, remove_forms, etc.), youâll see the result here:
markdown: The current location for detailed markdown output, returning a MarkdownGenerationResult object.
\n
markdown_v2: Removed in v0.5. Accessing it now raises AttributeError; use markdown.
\n
\n
MarkdownGenerationResult Fields:
\n
\n\n
\n
Field
\n
Description
\n
\n\n\n
\n
raw_markdown
\n
The basic HTMLâMarkdown conversion.
\n
\n
\n
markdown_with_citations
\n
Markdown including inline citations that reference links at the end.
\n
\n
\n
references_markdown
\n
The references/citations themselves (if citations=True).
\n
\n
\n
fit_markdown
\n
The filtered/âfitâ markdown if a content filter was used.
\n
\n
\n
fit_html
\n
The filtered HTML that generated fit_markdown.
\n
\n\n
\n
3.2 Basic Example with a Markdown Generator
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nconfig = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator(\n options={\"citations\": True, \"body_width\": 80} # e.g. pass html2text style options\n )\n)\nresult = await crawler.arun(url=\"https://example.com\", config=config)\n\nmd_res = result.markdown # or eventually 'result.markdown'\nprint(md_res.raw_markdown[:500])\nprint(md_res.markdown_with_citations)\nprint(md_res.references_markdown)\n
\n
Note: If you use a filter like PruningContentFilter, youâll get fit_markdown and fit_html as well.
\n\n
4. Structured Extraction: extracted_content
\n
If you run a JSON-based extraction strategy (CSS, XPath, LLM, etc.), the structured data is not stored in markdownâitâs placed in result.extracted_content as a JSON string (or sometimes plain text).
Here:\n- url=\"raw://...\" passes the HTML content directly, no network requests. \n- The CSS extraction strategy populates result.extracted_content with the JSON array [{\"title\": \"...\", \"link\": \"...\"}].
\n\n
5. More Fields: Links, Media, Tables and More
\n
5.1 links
\n
A dictionary, typically with \"internal\" and \"external\" lists. Each entry might have href, text, title, etc. This is automatically captured if you havenât disabled link extraction.
\n
print(result.links[\"internal\"][:3]) # Show first 3 internal links\n
\n
5.2 media
\n
Similarly, a dictionary with \"images\", \"audio\", \"video\", etc. Each item could include src, alt, score, and more, if your crawler is set to gather them.
The tables field contains structured data extracted from HTML tables found on the crawled page. Tables are analyzed based on various criteria to determine if they are actual data tables (as opposed to layout tables), including:
\n
\n
Presence of thead and tbody sections
\n
Use of th elements for headers
\n
Column consistency
\n
Text density
\n
And other factors
\n
\n
Tables that score above the threshold (default: 7) are extracted and stored in result.tables.
\n
Accessing Table data:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.w3schools.com/html/html_tables.asp\",\n config=CrawlerRunConfig(\n table_score_threshold=7# Minimum score for table detection\n )\n )\n\n if result.success and result.tables:\n print(f\"Found {len(result.tables)} tables\")\n\n for i, table inenumerate(result.tables):\n print(f\"\\nTable {i+1}:\")\n print(f\"Caption: {table.get('caption', 'No caption')}\")\n print(f\"Headers: {table['headers']}\")\n print(f\"Rows: {len(table['rows'])}\")\n\n # Print first few rows as example\n for j, row inenumerate(table['rows'][:3]):\n print(f\" Row {j+1}: {row}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Configuring Table Extraction:
\n
You can adjust the sensitivity of the table detection algorithm with:
\n
config = CrawlerRunConfig(\n table_score_threshold=5 # Lower value = more tables detected (default: 7)\n)\n
\n
Each extracted table contains:
\n
\n
headers: Column header names
\n
rows: List of rows, each containing cell values
\n
caption: Table caption text (if available)
\n
summary: Table summary attribute (if specified)
\n
\n
Table Extraction Tips
\n
\n
Not all HTML tables are extracted - only those detected as \"data tables\" vs. layout tables.
\n
Tables with inconsistent cell counts, nested tables, or those used purely for layout may be skipped.
\n
If you're missing tables, try adjusting the table_score_threshold to a lower value (default is 7).
\n
\n
The table detection algorithm scores tables based on features like consistent columns, presence of headers, text density, and more. Tables scoring above the threshold are considered data tables worth extracting.
\n
5.4 screenshot, pdf, and mhtml
\n
If you set screenshot=True, pdf=True, or capture_mhtml=True in CrawlerRunConfig, then:
\n
\n
result.screenshot contains a base64-encoded PNG string.
\n
result.pdf contains raw PDF bytes (you can write them to a file).
\n
result.mhtml contains the MHTML snapshot of the page as a string (you can write it to a .mhtml file).
\n
\n
# Save the PDF\nwithopen(\"page.pdf\", \"wb\") as f:\n f.write(result.pdf)\n\n# Save the MHTML\nif result.mhtml:\n withopen(\"page.mhtml\", \"w\", encoding=\"utf-8\") as f:\n f.write(result.mhtml)\n
\n
The MHTML (MIME HTML) format is particularly useful as it captures the entire web page including all of its resources (CSS, images, scripts, etc.) in a single file, making it perfect for archiving or offline viewing.
\n
5.5 ssl_certificate
\n
If fetch_ssl_certificate=True, result.ssl_certificate holds details about the siteâs SSL cert, such as issuer, validity dates, etc.
\n\n
6. Accessing These Fields
\n
After you run:
\n
result = await crawler.arun(url=\"https://example.com\", config=some_config)\n
\n
Check any field:
\n
if result.success:\n print(result.status_code, result.response_headers)\n print(\"Links found:\", len(result.links.get(\"internal\", [])))\n if result.markdown:\n print(\"Markdown snippet:\", result.markdown.raw_markdown[:200])\n if result.extracted_content:\n print(\"Structured JSON:\", result.extracted_content)\nelse:\n print(\"Error:\", result.error_message)\n
\n
Deprecation: Since v0.5 markdown_v2, fit_markdown, and fit_html are removed from CrawlResult. Use result.markdown for markdown output. It holds MarkdownGenerationResult, including fit_html and fit_markdown.
\n\n
7. Next Steps
\n
\n
Markdown Generation: Dive deeper into how to configure DefaultMarkdownGenerator and various filters.
\n
Content Filtering: Learn how to use BM25ContentFilter and PruningContentFilter.
\n
Session & Hooks: If you want to manipulate the page or preserve state across multiple arun() calls, see the hooking or session docs.
\n
LLM Extraction: For complex or unstructured content requiring AI-driven parsing, check the LLM-based strategies doc.
\n
\n
Enjoy exploring all that CrawlResult offersâwhether you need raw HTML, sanitized output, markdown, or fully structured data, Crawl4AI has you covered!
One of Crawl4AI's most powerful features is its ability to perform configurable deep crawling that can explore websites beyond a single page. With fine-tuned control over crawl depth, domain boundaries, and content filtering, Crawl4AI gives you the tools to extract precisely the content you need.
\n
In this tutorial, you'll learn:
\n\n
How to set up a Basic Deep Crawler with BFS strategy
\n
Understanding the difference between streamed and non-streamed output
\n
Implementing filters and scorers to target specific content
\n
Creating advanced filtering chains for sophisticated crawls
\n
Using BestFirstCrawling for intelligent exploration prioritization
\n
Crash recovery for long-running production crawls
\n
Prefetch mode for fast URL discovery
\n\n
\n
Prerequisites \n- Youâve completed or read AsyncWebCrawler Basics to understand how to run a simple crawl. \n- You know how to configure CrawlerRunConfig.
\n
\n\n
1. Quick Example
\n
Here's a minimal code snippet that implements a basic deep crawl using the BFSDeepCrawlStrategy:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.deep_crawling import BFSDeepCrawlStrategy\nfrom crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy\n\nasyncdefmain():\n # Configure a 2-level deep crawl\n config = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(\n max_depth=2, \n include_external=False\n ),\n scraping_strategy=LXMLWebScrapingStrategy(),\n verbose=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n results = await crawler.arun(\"https://example.com\", config=config)\n\n print(f\"Crawled {len(results)} pages in total\")\n\n # Access individual results\n for result in results[:3]: # Show first 3 results\n print(f\"URL: {result.url}\")\n print(f\"Depth: {result.metadata.get('depth', 0)}\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
What's happening? \n- BFSDeepCrawlStrategy(max_depth=2, include_external=False) instructs Crawl4AI to:\n - Crawl the starting page (depth 0) plus 2 more levels\n - Stay within the same domain (don't follow external links)\n- Each result contains metadata like the crawl depth\n- Results are returned as a list after all crawling is complete
\n\n
2. Understanding Deep Crawling Strategy Options
\n
2.1 BFSDeepCrawlStrategy (Breadth-First Search)
\n
The BFSDeepCrawlStrategy uses a breadth-first approach, exploring all links at one depth before moving deeper:
\n
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy\n\n# Basic configuration\nstrategy = BFSDeepCrawlStrategy(\n max_depth=2, # Crawl initial page + 2 levels deep\n include_external=False, # Stay within the same domain\n max_pages=50, # Maximum number of pages to crawl (optional)\n score_threshold=0.3, # Minimum score for URLs to be crawled (optional)\n)\n
\n
Key parameters:\n- max_depth: Number of levels to crawl beyond the starting page\n- include_external: Whether to follow links to other domains\n- max_pages: Maximum number of pages to crawl (default: infinite)\n- score_threshold: Minimum score for URLs to be crawled (default: -inf)\n- filter_chain: FilterChain instance for URL filtering\n- url_scorer: Scorer instance for evaluating URLs
\n
2.2 DFSDeepCrawlStrategy (Depth-First Search)
\n
The DFSDeepCrawlStrategy uses a depth-first approach, explores as far down a branch as possible before backtracking.
\n
from crawl4ai.deep_crawling import DFSDeepCrawlStrategy\n\n# Basic configuration\nstrategy = DFSDeepCrawlStrategy(\n max_depth=2, # Crawl initial page + 2 levels deep\n include_external=False, # Stay within the same domain\n max_pages=30, # Maximum number of pages to crawl (optional)\n score_threshold=0.5, # Minimum score for URLs to be crawled (optional)\n)\n
\n
Key parameters:\n- max_depth: Number of levels to crawl beyond the starting page\n- include_external: Whether to follow links to other domains\n- max_pages: Maximum number of pages to crawl (default: infinite)\n- score_threshold: Minimum score for URLs to be crawled (default: -inf)\n- filter_chain: FilterChain instance for URL filtering\n- url_scorer: Scorer instance for evaluating URLs
\n
2.3 BestFirstCrawlingStrategy (âď¸ - Recommended Deep crawl strategy)
\n
For more intelligent crawling, use BestFirstCrawlingStrategy with scorers to prioritize the most relevant pages:
\n
from crawl4ai.deep_crawling import BestFirstCrawlingStrategy\nfrom crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer\n\n# Create a scorer\nscorer = KeywordRelevanceScorer(\n keywords=[\"crawl\", \"example\", \"async\", \"configuration\"],\n weight=0.7\n)\n\n# Configure the strategy\nstrategy = BestFirstCrawlingStrategy(\n max_depth=2,\n include_external=False,\n url_scorer=scorer,\n max_pages=25, # Maximum number of pages to crawl (optional)\n)\n
\n
This crawling approach:\n- Evaluates each discovered URL based on scorer criteria\n- Visits higher-scoring pages first\n- Helps focus crawl resources on the most relevant content\n- Can limit total pages crawled with max_pages\n- Does not need score_threshold as it naturally prioritizes by score
\n\n
3. Streaming vs. Non-Streaming Results
\n
Crawl4AI can return results in two modes:
\n
3.1 Non-Streaming Mode (Default)
\n
config = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1),\n stream=False # Default behavior\n)\n\nasyncwithAsyncWebCrawler() as crawler:\n # Wait for ALL results to be collected before returning\n results = await crawler.arun(\"https://example.com\", config=config)\n\n for result in results:\n process_result(result)\n
\n
When to use non-streaming mode:\n- You need the complete dataset before processing\n- You're performing batch operations on all results together\n- Crawl time isn't a critical factor
\n
3.2 Streaming Mode
\n
config = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1),\n stream=True# Enable streaming\n)\n\nasyncwith AsyncWebCrawler() as crawler:\n # Returns an async iterator\n asyncfor result inawait crawler.arun(\"https://example.com\", config=config):\n # Process each result as it becomes available\n process_result(result)\n
\n
Benefits of streaming mode:\n- Process results immediately as they're discovered\n- Start working with early results while crawling continues\n- Better for real-time applications or progressive display\n- Reduces memory pressure when handling many pages
\n\n
4. Filtering Content with Filter Chains
\n
Filters help you narrow down which pages to crawl. Combine multiple filters using FilterChain for powerful targeting.
\n
4.1 Basic URL Pattern Filter
\n
from crawl4ai.deep_crawling.filters import FilterChain, URLPatternFilter\n\n# Only follow URLs containing \"blog\" or \"docs\"\nurl_filter = URLPatternFilter(patterns=[\"*blog*\", \"*docs*\"])\n\nconfig = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(\n max_depth=1,\n filter_chain=FilterChain([url_filter])\n )\n)\n
\n
4.2 Combining Multiple Filters
\n
from crawl4ai.deep_crawling.filters import (\n FilterChain,\n URLPatternFilter,\n DomainFilter,\n ContentTypeFilter\n)\n\n# Create a chain of filters\nfilter_chain = FilterChain([\n # Only follow URLs with specific patterns\n URLPatternFilter(patterns=[\"*guide*\", \"*tutorial*\"]),\n\n # Only crawl specific domains\n DomainFilter(\n allowed_domains=[\"docs.example.com\"],\n blocked_domains=[\"old.docs.example.com\"]\n ),\n\n # Only include specific content types\n ContentTypeFilter(allowed_types=[\"text/html\"])\n])\n\nconfig = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(\n max_depth=2,\n filter_chain=filter_chain\n )\n)\n
\n
4.3 Available Filter Types
\n
Crawl4AI includes several specialized filters:
\n
\n
URLPatternFilter: Matches URL patterns using wildcard syntax
\n
DomainFilter: Controls which domains to include or exclude
\n
ContentTypeFilter: Filters based on HTTP Content-Type
\n
ContentRelevanceFilter: Uses similarity to a text query
\n
SEOFilter: Evaluates SEO elements (meta tags, headers, etc.)
\n
\n\n
5. Using Scorers for Prioritized Crawling
\n
Scorers assign priority values to discovered URLs, helping the crawler focus on the most relevant content first.
\n
5.1 KeywordRelevanceScorer
\n
from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer\nfrom crawl4ai.deep_crawling import BestFirstCrawlingStrategy\n\n# Create a keyword relevance scorer\nkeyword_scorer = KeywordRelevanceScorer(\n keywords=[\"crawl\", \"example\", \"async\", \"configuration\"],\n weight=0.7# Importance of this scorer (0.0 to 1.0)\n)\n\nconfig = CrawlerRunConfig(\n deep_crawl_strategy=BestFirstCrawlingStrategy(\n max_depth=2,\n url_scorer=keyword_scorer\n ),\n stream=True# Recommended with BestFirstCrawling\n)\n\n# Results will come in order of relevance score\nasyncwith AsyncWebCrawler() as crawler:\n asyncfor result inawait crawler.arun(\"https://example.com\", config=config):\n score = result.metadata.get(\"score\", 0)\n print(f\"Score: {score:.2f} | {result.url}\")\n
\n
How scorers work:\n- Evaluate each discovered URL before crawling\n- Calculate relevance based on various signals\n- Help the crawler make intelligent choices about traversal order
\n\n
6. Advanced Filtering Techniques
\n
6.1 SEO Filter for Quality Assessment
\n
The SEOFilter helps you identify pages with strong SEO characteristics:
\n
from crawl4ai.deep_crawling.filters import FilterChain, SEOFilter\n\n# Create an SEO filter that looks for specific keywords in page metadata\nseo_filter = SEOFilter(\n threshold=0.5, # Minimum score (0.0 to 1.0)\n keywords=[\"tutorial\", \"guide\", \"documentation\"]\n)\n\nconfig = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(\n max_depth=1,\n filter_chain=FilterChain([seo_filter])\n )\n)\n
\n
6.2 Content Relevance Filter
\n
The ContentRelevanceFilter analyzes the actual content of pages:
\n
from crawl4ai.deep_crawling.filters import FilterChain, ContentRelevanceFilter\n\n# Create a content relevance filter\nrelevance_filter = ContentRelevanceFilter(\n query=\"Web crawling and data extraction with Python\",\n threshold=0.7 # Minimum similarity score (0.0 to 1.0)\n)\n\nconfig = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(\n max_depth=1,\n filter_chain=FilterChain([relevance_filter])\n )\n)\n
\n
This filter:\n- Measures semantic similarity between query and page content\n- It's a BM25-based relevance filter using head section content
\n\n
7. Building a Complete Advanced Crawler
\n
This example combines multiple techniques for a sophisticated crawl:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy\nfrom crawl4ai.deep_crawling import BestFirstCrawlingStrategy\nfrom crawl4ai.deep_crawling.filters import (\n FilterChain,\n DomainFilter,\n URLPatternFilter,\n ContentTypeFilter\n)\nfrom crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer\n\nasyncdefrun_advanced_crawler():\n # Create a sophisticated filter chain\n filter_chain = FilterChain([\n # Domain boundaries\n DomainFilter(\n allowed_domains=[\"docs.example.com\"],\n blocked_domains=[\"old.docs.example.com\"]\n ),\n\n # URL patterns to include\n URLPatternFilter(patterns=[\"*guide*\", \"*tutorial*\", \"*blog*\"]),\n\n # Content type filtering\n ContentTypeFilter(allowed_types=[\"text/html\"])\n ])\n\n # Create a relevance scorer\n keyword_scorer = KeywordRelevanceScorer(\n keywords=[\"crawl\", \"example\", \"async\", \"configuration\"],\n weight=0.7\n )\n\n # Set up the configuration\n config = CrawlerRunConfig(\n deep_crawl_strategy=BestFirstCrawlingStrategy(\n max_depth=2,\n include_external=False,\n filter_chain=filter_chain,\n url_scorer=keyword_scorer\n ),\n scraping_strategy=LXMLWebScrapingStrategy(),\n stream=True,\n verbose=True\n )\n\n # Execute the crawl\n results = []\n asyncwith AsyncWebCrawler() as crawler:\n asyncfor result inawait crawler.arun(\"https://docs.example.com\", config=config):\n results.append(result)\n score = result.metadata.get(\"score\", 0)\n depth = result.metadata.get(\"depth\", 0)\n print(f\"Depth: {depth} | Score: {score:.2f} | {result.url}\")\n\n # Analyze the results\n print(f\"Crawled {len(results)} high-value pages\")\n print(f\"Average score: {sum(r.metadata.get('score', 0) for r in results) / len(results):.2f}\")\n\n # Group by depth\n depth_counts = {}\n for result in results:\n depth = result.metadata.get(\"depth\", 0)\n depth_counts[depth] = depth_counts.get(depth, 0) + 1\n\n print(\"Pages crawled by depth:\")\n for depth, count insorted(depth_counts.items()):\n print(f\" Depth {depth}: {count} pages\")\n\nif __name__ == \"__main__\":\n asyncio.run(run_advanced_crawler())\n
\n\n
8. Limiting and Controlling Crawl Size
\n
8.1 Using max_pages
\n
You can limit the total number of pages crawled with the max_pages parameter:
\n
# Limit to exactly 20 pages regardless of depth\nstrategy = BFSDeepCrawlStrategy(\n max_depth=3,\n max_pages=20\n)\n
\n
This feature is useful for:\n- Controlling API costs\n- Setting predictable execution times\n- Focusing on the most important content\n- Testing crawl configurations before full execution
\n
8.2 Using score_threshold
\n
For BFS and DFS strategies, you can set a minimum score threshold to only crawl high-quality pages:
\n
# Only follow links with scores above 0.4\nstrategy = DFSDeepCrawlStrategy(\n max_depth=2,\n url_scorer=KeywordRelevanceScorer(keywords=[\"api\", \"guide\", \"reference\"]),\n score_threshold=0.4 # Skip URLs with scores below this value\n)\n
\n
Note that for BestFirstCrawlingStrategy, score_threshold is not needed since pages are already processed in order of highest score first.
\n
9. Common Pitfalls & Tips
\n
1.Set realistic limits. Be cautious with max_depth values > 3, which can exponentially increase crawl size. Use max_pages to set hard limits.
\n
2.Don't neglect the scoring component. BestFirstCrawling works best with well-tuned scorers. Experiment with keyword weights for optimal prioritization.
\n
3.Be a good web citizen. Respect robots.txt. (disabled by default)
\n
4.Handle page errors gracefully. Not all pages will be accessible. Check result.status when processing results.
\n
5.Balance breadth vs. depth. Choose your strategy wisely - BFS for comprehensive coverage, DFS for deep exploration, BestFirst for focused relevance-based crawling.
\n
6.Preserve HTTPS for security. If crawling HTTPS sites that redirect to HTTP, use preserve_https_for_internal_links=True to maintain secure connections:
\n
config = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2),\n preserve_https_for_internal_links=True# Keep HTTPS even if server redirects to HTTP\n)\n
\n
This is especially useful for security-conscious crawling or when dealing with sites that support both protocols.
\n\n
10. Crash Recovery for Long-Running Crawls
\n
For production deployments, especially in cloud environments where instances can be terminated unexpectedly, Crawl4AI provides built-in crash recovery support for all deep crawl strategies.
\n
10.1 Enabling State Persistence
\n
All deep crawl strategies (BFS, DFS, Best-First) support two optional parameters:
\n
\n
resume_state: Pass a previously saved state to resume from a checkpoint
\n
on_state_change: Async callback fired after each URL is processed
\n
\n
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy\nimport json\n\n# Callback to save state after each URL\nasyncdefsave_state_to_redis(state: dict):\n await redis.set(\"crawl_state\", json.dumps(state))\n\nstrategy = BFSDeepCrawlStrategy(\n max_depth=3,\n on_state_change=save_state_to_redis, # Called after each URL\n)\n
\n
10.2 State Structure
\n
The state dictionary is JSON-serializable and contains:
import json\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.deep_crawling import BFSDeepCrawlStrategy\n\n# Load saved state (e.g., from Redis, database, or file)\nsaved_state = json.loads(await redis.get(\"crawl_state\"))\n\n# Resume crawling from where we left off\nstrategy = BFSDeepCrawlStrategy(\n max_depth=3,\n resume_state=saved_state, # Continue from checkpoint\n on_state_change=save_state_to_redis, # Keep saving progress\n)\n\nconfig = CrawlerRunConfig(deep_crawl_strategy=strategy)\n\nasyncwith AsyncWebCrawler() as crawler:\n # Will skip already-visited URLs and continue from pending queue\n results = await crawler.arun(start_url, config=config)\n
\n
10.4 Manual State Export
\n
You can export the last captured state using export_state(). Note that this requires on_state_change to be set (state is captured in the callback):
\n
import json\n\ncaptured_state = None\n\nasyncdefcapture_state(state: dict):\n global captured_state\n captured_state = state\n\nstrategy = BFSDeepCrawlStrategy(\n max_depth=2,\n on_state_change=capture_state, # Required for state capture\n)\nconfig = CrawlerRunConfig(deep_crawl_strategy=strategy)\n\nasyncwith AsyncWebCrawler() as crawler:\n results = await crawler.arun(start_url, config=config)\n\n# Get the last captured state\nstate = strategy.export_state()\nif state:\n # Save to your preferred storage\n withopen(\"crawl_checkpoint.json\", \"w\") as f:\n json.dump(state, f)\n
When resume_state=None and on_state_change=None (the defaults), there is no performance impact. State tracking only activates when you enable these features.
\n\n
11. Cancellation Support for Deep Crawls
\n
For production environments like cloud platforms, you often need to stop a running crawl mid-executionâwhether the user changed their mind, specified the wrong URL, or wants to control costs. Crawl4AI provides built-in cancellation support for all deep crawl strategies.
\n
11.1 Two Ways to Cancel
\n
Option A: Callback-based cancellation (recommended for external systems)
\n
Use should_cancel to check an external source (Redis, database, API) before each URL:
\n
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy\n\nasyncdefcheck_if_cancelled():\n # Check Redis, database, or any external source\n job = await redis.get(f\"job:{job_id}\")\n return job.get(\"status\") == \"cancelled\"\n\nstrategy = BFSDeepCrawlStrategy(\n max_depth=3,\n max_pages=1000,\n should_cancel=check_if_cancelled, # Called before each URL\n)\n
\n
Option B: Direct cancellation (for in-process control)
\n
Call cancel() directly on the strategy instance:
\n
strategy = BFSDeepCrawlStrategy(max_depth=3, max_pages=1000)\n\n# In another coroutine or thread:\nstrategy.cancel()# Thread-safe, stops before next URL\n
\n
11.2 Checking Cancellation Status
\n
Use the cancelled property to check if a crawl was cancelled:
\n
asyncwith AsyncWebCrawler() as crawler:\n results = await crawler.arun(url, config=config)\n\nif strategy.cancelled:\n print(f\"Crawl was cancelled after {len(results)} pages\")\nelse:\n print(f\"Crawl completed with {len(results)} pages\")\n
\n
11.3 State Notifications Include Cancelled Flag
\n
When using on_state_change, the state dictionary includes a cancelled field:
\n
asyncdefhandle_state(state: dict):\n if state.get(\"cancelled\"):\n print(\"Crawl was cancelled!\")\n print(f\"Crawled {state['pages_crawled']} pages before cancellation\")\n # Save state for potential resume\n await redis.set(\"crawl_state\", json.dumps(state))\n\nstrategy = BFSDeepCrawlStrategy(\n max_depth=3,\n should_cancel=check_cancelled,\n on_state_change=handle_state,\n)\n
All strategies support:\n- should_cancel callback parameter\n- cancel() method\n- cancelled property
\n\n
12. Prefetch Mode for Fast URL Discovery
\n
When you need to quickly discover URLs without full page processing, use prefetch mode. This is ideal for two-phase crawling where you first map the site, then selectively process specific pages.
\n
12.1 Enabling Prefetch Mode
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nconfig = CrawlerRunConfig(prefetch=True)\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n\n # Result contains only HTML and links - no markdown, no extraction\n print(f\"Found {len(result.links['internal'])} internal links\")\n print(f\"Found {len(result.links['external'])} external links\")\n
\n
12.2 What Gets Skipped
\n
Prefetch mode uses a fast path that bypasses heavy processing:
\n
\n\n
\n
Processing Step
\n
Normal Mode
\n
Prefetch Mode
\n
\n\n\n
\n
Fetch HTML
\n
â
\n
â
\n
\n
\n
Extract links
\n
â
\n
â (fast quick_extract_links())
\n
\n
\n
Generate markdown
\n
â
\n
â Skipped
\n
\n
\n
Content scraping
\n
â
\n
â Skipped
\n
\n
\n
Media extraction
\n
â
\n
â Skipped
\n
\n
\n
LLM extraction
\n
â
\n
â Skipped
\n
\n\n
\n
12.3 Performance Benefit
\n
\n
Normal mode: Full pipeline (~2-5 seconds per page)
\n
Prefetch mode: HTML + links only (~200-500ms per page)
\n
\n
This makes prefetch mode 5-10x faster for URL discovery.
\n
12.4 Two-Phase Crawling Pattern
\n
The most common use case is two-phase crawling:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdeftwo_phase_crawl(start_url: str):\n asyncwith AsyncWebCrawler() as crawler:\n # âââââââââââââââââââââââââââââââââââââââââââââââ\n # Phase 1: Fast discovery (prefetch mode)\n # âââââââââââââââââââââââââââââââââââââââââââââââ\n prefetch_config = CrawlerRunConfig(prefetch=True)\n discovery = await crawler.arun(start_url, config=prefetch_config)\n\n all_urls = [link[\"href\"] for link in discovery.links.get(\"internal\", [])]\n print(f\"Discovered {len(all_urls)} URLs\")\n\n # Filter to URLs you care about\n blog_urls = [url for url in all_urls if\"/blog/\"in url]\n print(f\"Found {len(blog_urls)} blog posts to process\")\n\n # âââââââââââââââââââââââââââââââââââââââââââââââ\n # Phase 2: Full processing on selected URLs only\n # âââââââââââââââââââââââââââââââââââââââââââââââ\n full_config = CrawlerRunConfig(\n # Your normal extraction settings\n word_count_threshold=100,\n remove_overlay_elements=True,\n )\n\n results = []\n for url in blog_urls:\n result = await crawler.arun(url, config=full_config)\n if result.success:\n results.append(result)\n print(f\"Processed: {url}\")\n\n return results\n\nif __name__ == \"__main__\":\n results = asyncio.run(two_phase_crawl(\"https://example.com\"))\n print(f\"Fully processed {len(results)} pages\")\n
\n
12.5 Use Cases
\n
\n
Site mapping: Quickly discover all URLs before deciding what to process
\n
Link validation: Check which pages exist without heavy processing
\n
Selective deep crawl: Prefetch to find URLs, filter by pattern, then full crawl
\n
Crawl planning: Estimate crawl size before committing resources
\n
\n\n
13. Summary & Next Steps
\n
In this Deep Crawling with Crawl4AI tutorial, you learned to:
\n
\n
Configure BFSDeepCrawlStrategy, DFSDeepCrawlStrategy, and BestFirstCrawlingStrategy
\n
Process results in streaming or non-streaming mode
\n
Apply filters to target specific content
\n
Use scorers to prioritize the most relevant pages
\n
Limit crawls with max_pages and score_threshold parameters
\n
Build a complete advanced crawler with combined techniques
\n
Implement crash recovery with resume_state and on_state_change for production deployments
\n
Cancel running crawls with should_cancel callback or cancel() method for cloud platform job management
\n
Use prefetch mode for fast URL discovery and two-phase crawling
\n
\n
With these tools, you can efficiently extract structured data from websites at scale, focusing precisely on the content you need for your specific use case.
Domain mapping goes beyond URL seeding. Instead of checking a single sitemap or index, DomainMapper combines 8 discovery sources to find every URL under a domain â including subdomains you didn't know existed.
Real-world example: For superdesign.dev, AsyncUrlSeeder found 4 URLs. DomainMapper found 171 URLs across 11 hosts â including docs, API servers, staging environments, and analytics dashboards that no sitemap listed.
\n
Quick Start
\n
import asyncio\nfrom crawl4ai import DomainMapper, DomainMapperConfig\n\nasyncdefmain():\n asyncwith DomainMapper() as mapper:\n results = await mapper.scan(\"example.com\")\n\n print(f\"Found {len(results)} URLs\")\n for r in results[:10]:\n print(f\" [{r['source']}] {r['url']}\")\n if r.get(\"head_data\", {}).get(\"title\"):\n print(f\" Title: {r['head_data']['title']}\")\n\nasyncio.run(main())\n
\n
Or via AsyncWebCrawler:
\n
from crawl4ai import AsyncWebCrawler, DomainMapperConfig\n\nasyncwithAsyncWebCrawler() as crawler:\n results = await crawler.amap_domain(\"example.com\")\n
\n
The 8 Discovery Sources
\n
DomainMapper combines these sources, each catching URLs the others miss:
\n
1. sitemap â Sitemap Discovery
\n
Checks /sitemap.xml, /sitemap_index.xml, and robots.txtSitemap: directives on every discovered host â not just the root domain.
\n
config = DomainMapperConfig(source=\"sitemap\")\n
\n
2. cc â Common Crawl
\n
Queries the Common Crawl CDX API for *.domain.tld/*, catching URLs and subdomains the web's largest public crawl has indexed.
\n
config = DomainMapperConfig(source=\"cc\")\n
\n
3. wayback â Wayback Machine
\n
Queries the Internet Archive's CDX API. Often has different coverage than Common Crawl â including historical pages that have since been removed.
\n
config = DomainMapperConfig(source=\"wayback\")\n
\n
4. crt â Certificate Transparency
\n
Queries crt.sh for SSL certificates issued to *.domain.tld. This is the single most effective subdomain discovery technique â it found 14 subdomains for superdesign.dev that no other source knew about.
\n
config = DomainMapperConfig(source=\"crt\")\n
\n
5. probe â Common Path Probing
\n
Tries ~25 well-known paths on each discovered host (/docs, /api, /login, /dashboard, /openapi.json, etc.). Combined with soft-404 detection to avoid false positives.
Parses Disallow: and Allow: lines from robots.txt. These are confirmed real paths the site acknowledges exist â often revealing admin panels, APIs, and internal tools that aren't linked anywhere.
\n
config = DomainMapperConfig(source=\"robots\")\n
\n
7. feed â RSS/Atom Feed Parsing
\n
Discovers and parses RSS/Atom feeds at common paths (/feed, /rss, /atom.xml, etc.). Feeds are curated lists of content URLs maintained by the site.
\n
config = DomainMapperConfig(source=\"feed\")\n
\n
8. homepage â Homepage Link Extraction
\n
Fetches each host's homepage via HTTP and extracts all internal links using quick_extract_links(). Also mines <link rel=\"alternate|preload|prefetch\"> tags from the <head> for additional URLs. No browser needed.
Head extraction â parallel <head> fetching for metadata (optional)
\n
BM25 scoring â relevance scoring against a query (optional)
\n\n
Soft-404 Detection
\n
Many modern SPAs return HTTP 200 for every URL â even pages that don't exist. DomainMapper detects this:
\n\n
Fingerprinting: Fetches a guaranteed-nonexistent URL (e.g., /c4ai-probe-a1b2c3d4) on each host
\n
Recording: Captures the response title and body hash
\n
Filtering: When probing real paths, compares against the fingerprint. If they match â soft-404, filtered out
\n\n
For superdesign.dev, this correctly:\n- Blocked all 25+ probe paths on app.superdesign.dev (SPA that returns 200 for everything)\n- Blocked 476 sitemap URLs from app.superdesign.dev (all rendering the same shell)\n- Kept all 19 legitimate URLs from docs.superdesign.dev
\n
# Soft-404 detection is on by default\nconfig = DomainMapperConfig(soft_404_detection=True)\n\n# Disable if you want raw results\nconfig = DomainMapperConfig(soft_404_detection=False)\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, DomainMapperConfig, CrawlerRunConfig\n\nasyncdefcrawl_all_docs():\n asyncwith AsyncWebCrawler() as crawler:\n # Step 1: Discover all URLs\n pages = await crawler.amap_domain(\"example.com\", DomainMapperConfig(\n source=\"sitemap+crt+probe+homepage\",\n extract_head=True,\n query=\"documentation tutorial guide\",\n ))\n\n # Step 2: Filter for docs\n doc_urls = [\n p[\"url\"] for p in pages\n if p.get(\"relevance_score\", 0) > 0.3\n ]\n print(f\"Found {len(doc_urls)} documentation pages\")\n\n # Step 3: Crawl them\n results = await crawler.arun_many(\n doc_urls[:50],\n config=CrawlerRunConfig(only_text=True)\n )\n for r in results:\n if r.success:\n print(f\" Crawled: {r.url}\")\n\nasyncio.run(crawl_all_docs())\n
\n
Security Audit: Find Exposed Services
\n
asyncdefaudit_domain():\n asyncwith DomainMapper() as mapper:\n results = await mapper.scan(\"company.com\", DomainMapperConfig(\n source=\"crt+probe+robots\",\n extract_head=True,\n probe_paths=[\n \"/openapi.json\", \"/swagger.json\", \"/api-docs\",\n \"/graphql\", \"/.env\", \"/debug\", \"/admin\",\n \"/phpinfo.php\", \"/server-status\",\n ],\n ))\n\n # Flag exposed services\n for r in results:\n title = r.get(\"head_data\", {}).get(\"title\", \"\")\n ifany(x in title.lower() for x in [\"swagger\", \"api\", \"admin\", \"debug\"]):\n print(f\" EXPOSED: {r['url']} â {title}\")\n
\n
Compare Subdomains Across a Domain
\n
asyncdefmap_infrastructure():\n asyncwith DomainMapper() as mapper:\n results = await mapper.scan(\"company.com\", DomainMapperConfig(\n source=\"crt+probe\",\n extract_head=False,\n ))\n\n # Group by host\n from collections import defaultdict\n by_host = defaultdict(list)\n for r in results:\n by_host[r[\"host\"]].append(r)\n\n print(f\"Discovered {len(by_host)} hosts:\")\n for host, urls insorted(by_host.items()):\n print(f\" {host}: {len(urls)} URLs\")\n
\n
Tips and Best Practices
\n\n
\n
Start with the default sources (sitemap+cc+crt+probe). Add wayback, robots, feed, and homepage if you need maximum coverage.
\n
\n
\n
Use extract_head=False for speed when you just need URL lists. Head extraction makes ~1 HTTP request per URL.
\n
\n
\n
The query parameter is powerful for finding specific content across a large domain without crawling anything.
\n
\n
\n
probe_paths is your extensibility hook â add domain-specific paths you suspect exist.
\n
\n
\n
Rate limiting matters â hits_per_sec=10 is respectful. Lower it for smaller sites, raise it for your own infrastructure.
\n
\n
\n
Soft-404 detection is critical for SPAs â without it, single-page apps flood your results with hundreds of identical shell pages.
\n
\n\n
See Also
\n
\n
URL Seeding â simpler, single-host URL discovery from sitemaps and Common Crawl
\n
Deep Crawling â follow links dynamically within pages
This page provides a comprehensive list of example scripts that demonstrate various features and capabilities of Crawl4AI. Each example is designed to showcase specific functionality, making it easier for you to understand how to implement these features in your own projects.
\n
Getting Started Examples
\n
\n\n
\n
Example
\n
Description
\n
Link
\n
\n\n\n
\n
Hello World
\n
A simple introductory example demonstrating basic usage of AsyncWebCrawler with JavaScript execution and content filtering.
A comprehensive collection of examples showcasing various features including basic crawling, content cleaning, link analysis, JavaScript execution, CSS selectors, media handling, custom hooks, proxy configuration, screenshots, and multiple extraction strategies.
NSTProxy Seamlessly integrates with crawl4ai â no setup required. Access high-performance residential, datacenter, ISP, and IPv6 proxies with smart rotation and anti-blocking technology. Starts from $0.1/GB. Use code crawl4ai for 10% off.
An extensive tutorial on deep crawling capabilities, demonstrating BFS and BestFirst strategies, stream vs. non-stream execution, filters, scorers, and advanced configurations.
Comprehensive examples for handling virtualized scrolling on sites like Twitter, Instagram. Demonstrates different scrolling scenarios with local test server.
Seamlessly integrate with CapSolver to automatically solve reCAPTCHA v2/v3, Cloudflare Turnstile / Challenges, AWS WAF and more for uninterrupted scraping and automation.
To run any of these examples, you'll need to have Crawl4AI installed:
\n
pip install crawl4ai\n
\n
Then, you can run an example script like this:
\n
python -m docs.examples.hello_world\n
\n
For examples that require additional dependencies or environment variables, refer to the comments at the top of each file.
\n
Some examples may require:\n- API keys (for LLM-based examples)\n- Docker setup (for Docker-related examples)\n- Additional dependencies (specified in the example files)
\n
Contributing New Examples
\n
If you've created an interesting example that demonstrates a unique use case or feature of Crawl4AI, we encourage you to contribute it to our examples collection. Please see our contribution guidelines for more information.
Fit Markdown is a specialized filtered version of your pageâs markdown, focusing on the most relevant content. By default, Crawl4AI converts the entire HTML into a broad raw_markdown. With fit markdown, we apply a content filter algorithm (e.g., Pruning or BM25) to remove or rank low-value sectionsâsuch as repetitive sidebars, shallow text blocks, or irrelevanciesâleaving a concise textual âcore.â
\n\n
1. How âFit Markdownâ Works
\n
1.1 The content_filter
\n
In CrawlerRunConfig, you can specify a content_filter to shape how content is pruned or ranked before final markdown generation. A filterâs logic is applied before or during the HTMLâMarkdown process, producing:
\n
\n
result.markdown.raw_markdown (unfiltered)
\n
result.markdown.fit_markdown (filtered or âfitâ version)
\n
result.markdown.fit_html (the corresponding HTML snippet that produced fit_markdown)
\n
\n
1.2 Common Filters
\n
1.âPruningContentFilter â Scores each node by text density, link density, and tag importance, discarding those below a threshold. \n2.âBM25ContentFilter â Focuses on textual relevance using BM25 ranking, especially useful if you have a specific user query (e.g., âmachine learningâ or âfood nutritionâ).
\n\n
2. PruningContentFilter
\n
Pruning discards less relevant nodes based on text density, link density, and tag importance. Itâs a heuristic-based approachâif certain sections appear too âthinâ or too âspammy,â theyâre pruned.
\n
2.1 Usage Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefmain():\n # Step 1: Create a pruning filter\n prune_filter = PruningContentFilter(\n # Lower â more content retained, higher â more content pruned\n threshold=0.45, \n # \"fixed\" or \"dynamic\"\n threshold_type=\"dynamic\", \n # Ignore nodes with <5 words\n min_word_threshold=5 \n )\n\n # Step 2: Insert it into a Markdown Generator\n md_generator = DefaultMarkdownGenerator(content_filter=prune_filter)\n\n # Step 3: Pass it to CrawlerRunConfig\n config = CrawlerRunConfig(\n markdown_generator=md_generator\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", \n config=config\n )\n\n if result.success:\n # 'fit_markdown' is your pruned content, focusing on \"denser\" text\n print(\"Raw Markdown length:\", len(result.markdown.raw_markdown))\n print(\"Fit Markdown length:\", len(result.markdown.fit_markdown))\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
2.2 Key Parameters
\n
\n
min_word_threshold (int): If a block has fewer words than this, itâs pruned.
\n
threshold_type (str):
\n
\"fixed\" â each node must exceed threshold (0â1).
\n
\"dynamic\" â node scoring adjusts according to tag type, text/link density, etc.
\n
threshold (float, default ~0.48): The base or âanchorâ cutoff.
\n
\n
Algorithmic Factors:
\n
\n
Text density â Encourages blocks that have a higher ratio of text to overall content.
\n
Link density â Penalizes sections that are mostly links.
\n
Tag importance â e.g., an <article> or <p> might be more important than a <div>.
\n
Structural context â If a node is deeply nested or in a suspected sidebar, it might be deprioritized.
\n
\n\n
3. BM25ContentFilter
\n
BM25 is a classical text ranking algorithm often used in search engines. If you have a user query or rely on page metadata to derive a query, BM25 can identify which text chunks best match that query.
\n
3.1 Usage Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_filter_strategy import BM25ContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefmain():\n # 1) A BM25 filter with a user query\n bm25_filter = BM25ContentFilter(\n user_query=\"startup fundraising tips\",\n # Adjust for stricter or looser results\n bm25_threshold=1.2 \n )\n\n # 2) Insert into a Markdown Generator\n md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)\n\n # 3) Pass to crawler config\n config = CrawlerRunConfig(\n markdown_generator=md_generator\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", \n config=config\n )\n if result.success:\n print(\"Fit Markdown (BM25 query-based):\")\n print(result.markdown.fit_markdown)\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
3.2 Parameters
\n
\n
user_query (str, optional): E.g. \"machine learning\". If blank, the filter tries to glean a query from page metadata.
\n
bm25_threshold (float, default 1.0):
\n
Higher â fewer chunks but more relevant.
\n
Lower â more inclusive.
\n
\n
\n
In more advanced scenarios, you might see parameters like language, case_sensitive, or priority_tags to refine how text is tokenized or weighted.
\n
\n\n
4. Accessing the âFitâ Output
\n
After the crawl, your âfitâ content is found in result.markdown.fit_markdown.
If the content filter is BM25, you might see additional logic or references in fit_markdown that highlight relevant segments. If itâs Pruning, the text is typically well-cleaned but not necessarily matched to a query.
The crawlerâs excluded_tags are removed from the HTML first.
\n
The content filter (Pruning, BM25, or custom) prunes or ranks the remaining text blocks.
\n
The final âfitâ content is generated in result.markdown.fit_markdown.
\n\n\n
7. Custom Filters
\n
If you need a different approach (like a specialized ML model or site-specific heuristics), you can create a new class inheriting from RelevantContentFilter and implement filter_content(html). Then inject it into your markdown generator:
\n
from crawl4ai.content_filter_strategy import RelevantContentFilter\n\nclassMyCustomFilter(RelevantContentFilter):\n deffilter_content(self, html, min_word_threshold=None):\n # parse HTML, implement custom logic\n return [block for block in ... if ... some condition...]\n
\n
Steps:
\n\n
Subclass RelevantContentFilter.
\n
Implement filter_content(...).
\n
Use it in your DefaultMarkdownGenerator(content_filter=MyCustomFilter(...)).
\n\n\n
8. Final Thoughts
\n
Fit Markdown is a crucial feature for:
\n
\n
Summaries: Quickly get the important text from a cluttered page.
\n
Search: Combine with BM25 to produce content relevant to a query.
\n
AI Pipelines: Filter out boilerplate so LLM-based extraction or summarization runs on denser text.
\n
\n
Key Points:\n- PruningContentFilter: Great if you just want the âmeatiestâ text without a user query. \n- BM25ContentFilter: Perfect for query-based extraction or searching. \n- Combine with excluded_tags, exclude_external_links, word_count_threshold to refine your final âfitâ text. \n- Fit markdown ends up in result.markdown.fit_markdown; eventually result.markdown.fit_markdown in future versions.
\n
With these tools, you can zero in on the text that truly matters, ignoring spammy or boilerplate content, and produce a concise, relevant âfit markdownâ for your AI or data pipelines. Happy pruning and searching!
This installs the core Crawl4AI library along with essential dependencies.âNo advanced features (like transformers or PyTorch) are included yet.
\n
2. Initial Setup & Diagnostics
\n
2.1 Run the Setup Command
\n
After installing, call:
\n
crawl4ai-setup\n
\n
What does it do?\n- Installs or updates required browser dependencies for both regular and undetected modes\n- Performs OS-level checks (e.g., missing libs on Linux)\n- Confirms your environment is ready to crawl
\n
2.2 Diagnostics
\n
Optionally, you can run diagnostics to confirm everything is functioning:
\n
crawl4ai-doctor\n
\n
This command attempts to:\n- Check Python version compatibility\n- Verify Playwright installation\n- Inspect environment variables or library conflicts
\n
If any issues arise, follow its suggestions (e.g., installing additional system packages) and re-run crawl4ai-setup.
\n\n
3. Verifying Installation: A Simple Crawl (Skip this step if you already run crawl4ai-doctor)
\n
Below is a minimal Python script demonstrating a basic crawl. It uses our new BrowserConfig and CrawlerRunConfig for clarity, though no custom settings are passed in this example:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.example.com\",\n )\n print(result.markdown[:300]) # Show the first 300 characters of extracted text\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Expected outcome:\n- A headless browser session loads example.com\n- Crawl4AI returns ~300 characters of markdown. \nIf errors occur, rerun crawl4ai-doctor or manually ensure Playwright is installed correctly.
\n\n
4. Advanced Installation (Optional)
\n
Warning: Only install these if you truly need them. They bring in larger dependencies, including big models, which can increase disk usage and memory load significantly.
\n
4.1 Torch, Transformers, or All
\n
\n
\n
Text Clustering (Torch) \n
pip install crawl4ai[torch]\ncrawl4ai-setup\n
\n Installs PyTorch-based features (e.g., cosine similarity or advanced semantic chunking).\n
\n Adds Hugging Face-based summarization or generation strategies.\n
\n
\n
All Features \n
pip install crawl4ai[all]\ncrawl4ai-setup\n
\n
\n
\n
(Optional) Pre-Fetching Models
\n
crawl4ai-download-models\n
\nThis step caches large models locally (if needed).âOnly do this if your workflow requires them.\n\n
5. Docker (Experimental)
\n
We provide a temporary Docker approach for testing.âItâs not stable and may break with future releases. We plan a major Docker revamp in a future stable version, 2025 Q1. If you still want to try:
\n
docker pull unclecode/crawl4ai:basic\ndocker run -p 11235:11235 unclecode/crawl4ai:basic\n
\n
You can then make POST requests to http://localhost:11235/crawl to perform crawls.âProduction usage is discouraged until our new Docker approach is ready (planned in Jan or Feb 2025).
\n\n
6. Local Server Mode (Legacy)
\n
Some older docs mention running Crawl4AI as a local server. This approach has been partially replaced by the new Docker-based prototype and upcoming stable server release. You can experiment, but expect major changes. Official local server instructions will arrive once the new Docker architecture is finalized.
\n\n
Summary
\n
1.âInstall with pip install crawl4ai and run crawl4ai-setup.\n2.âDiagnose with crawl4ai-doctor if you see errors.\n3.âVerify by crawling example.com with minimal BrowserConfig + CrawlerRunConfig.\n4.âAdvanced features (Torch, Transformers) are optionalâavoid them if you donât need them (they significantly increase resource usage).\n5.âDocker is experimentalâuse at your own risk until the stable version is released.\n6.âLocal server references in older docs are largely deprecated; a new solution is in progress.
\n
Got questions? Check GitHub issues for updates or ask the community!
This setting removes all images very early in the processing pipeline, which significantly improves memory efficiency and processing speed. This is particularly useful when:\n- You don't need image data in your results\n- You're crawling image-heavy pages that cause memory issues\n- You want to focus only on text content\n- You need to maximize crawling speeddata (especially images) in the crawl result \n4. Configure your crawler to exclude or prioritize certain images
\n
\n
Prerequisites \n- You have completed or are familiar with the AsyncWebCrawler Basics tutorial. \n- You can run Crawl4AI in your environment (Playwright, Python, etc.).
\n
\n\n
Below is a revised version of the Link Extraction and Media Extraction sections that includes example data structures showing how links and media items are stored in CrawlResult. Feel free to adjust any field names or descriptions to match your actual output.
\n\n
1. Link Extraction
\n
1.1 result.links
\n
When you call arun() or arun_many() on a URL, Crawl4AI automatically extracts links and stores them in the links field of CrawlResult. By default, the crawler tries to distinguish internal links (same domain) from external links (different domains).
\n
Basic Example:
\n
from crawl4ai import AsyncWebCrawler\n\nasyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://www.example.com\")\n if result.success:\n internal_links = result.links.get(\"internal\", [])\n external_links = result.links.get(\"external\", [])\n print(f\"Found {len(internal_links)} internal links.\")\n print(f\"Found {len(internal_links)} external links.\")\n print(f\"Found {len(result.media)} media items.\")\n\n # Each link is typically a dictionary with fields like:\n # { \"href\": \"...\", \"text\": \"...\", \"title\": \"...\", \"base_domain\": \"...\" }\n if internal_links:\n print(\"Sample Internal Link:\", internal_links[0])\n else:\n print(\"Crawl failed:\", result.error_message)\n
title: The title attribute of the link (if present).
\n
base_domain: The domain extracted from href. Helpful for filtering or grouping by domain.
\n
\n\n
2. Advanced Link Head Extraction & Scoring
\n
Ever wanted to not just extract links, but also get the actual content (title, description, metadata) from those linked pages? And score them for relevance? This is exactly what Link Head Extraction does - it fetches the <head> section from each discovered link and scores them using multiple algorithms.
\n
2.1 Why Link Head Extraction?
\n
When you crawl a page, you get hundreds of links. But which ones are actually valuable? Link Head Extraction solves this by:
\n\n
Fetching head content from each link (title, description, meta tags)
\n
Scoring links intrinsically based on URL quality, text relevance, and context
\n
Scoring links contextually using BM25 algorithm when you provide a search query
\n
Combining scores intelligently to give you a final relevance ranking
\n\n
2.2 Complete Working Example
\n
Here's a full example you can copy, paste, and run immediately:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai import LinkPreviewConfig\n\nasyncdefextract_link_heads_example():\n \"\"\"\n Complete example showing link head extraction with scoring.\n This will crawl a documentation site and extract head content from internal links.\n \"\"\"\n\n # Configure link head extraction\n config = CrawlerRunConfig(\n # Enable link head extraction with detailed configuration\n link_preview_config=LinkPreviewConfig(\n include_internal=True, # Extract from internal links\n include_external=False, # Skip external links for this example\n max_links=10, # Limit to 10 links for demo\n concurrency=5, # Process 5 links simultaneously\n timeout=10, # 10 second timeout per link\n query=\"API documentation guide\", # Query for contextual scoring\n score_threshold=0.3, # Only include links scoring above 0.3\n verbose=True# Show detailed progress\n ),\n # Enable intrinsic scoring (URL quality, text relevance)\n score_links=True,\n # Keep output clean\n only_text=True,\n verbose=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n # Crawl a documentation site (great for testing)\n result = await crawler.arun(\"https://docs.python.org/3/\", config=config)\n\n if result.success:\n print(f\"â Successfully crawled: {result.url}\")\n print(f\"đ Page title: {result.metadata.get('title', 'No title')}\")\n\n # Access links (now enhanced with head data and scores)\n internal_links = result.links.get(\"internal\", [])\n external_links = result.links.get(\"external\", [])\n\n print(f\"\\nđ Found {len(internal_links)} internal links\")\n print(f\"đ Found {len(external_links)} external links\")\n\n # Count links with head data\n links_with_head = [link for link in internal_links \n if link.get(\"head_data\") isnotNone]\n print(f\"đ§ Links with head data extracted: {len(links_with_head)}\")\n\n # Show the top 3 scoring links\n print(f\"\\nđ Top 3 Links with Full Scoring:\")\n for i, link inenumerate(links_with_head[:3]):\n print(f\"\\n{i+1}. {link['href']}\")\n print(f\" Link Text: '{link.get('text', 'No text')[:50]}...'\")\n\n # Show all three score types\n intrinsic = link.get('intrinsic_score')\n contextual = link.get('contextual_score') \n total = link.get('total_score')\n\n if intrinsic isnotNone:\n print(f\" đ Intrinsic Score: {intrinsic:.2f}/10.0 (URL quality & context)\")\n if contextual isnotNone:\n print(f\" đŻ Contextual Score: {contextual:.3f} (BM25 relevance to query)\")\n if total isnotNone:\n print(f\" â Total Score: {total:.3f} (combined final score)\")\n\n # Show extracted head data\n head_data = link.get(\"head_data\", {})\n if head_data:\n title = head_data.get(\"title\", \"No title\")\n description = head_data.get(\"meta\", {}).get(\"description\", \"No description\")\n\n print(f\" đ° Title: {title[:60]}...\")\n if description:\n print(f\" đ Description: {description[:80]}...\")\n\n # Show extraction status\n status = link.get(\"head_extraction_status\", \"unknown\")\n print(f\" â Extraction Status: {status}\")\n else:\n print(f\"â Crawl failed: {result.error_message}\")\n\n# Run the example\nif __name__ == \"__main__\":\n asyncio.run(extract_link_heads_example())\n
\n
Expected Output:\n
â Successfully crawled:https://docs.python.org/3/\nđPage title:3.13.5Documentation\nđFound53internallinks\nđFound1externallinks\nđ§ Links with head data extracted:10\n\nđTop 3 Links with Full Scoring:\n\n1.https://docs.python.org/3.15/\n Link Text:'Python 3.15 (in development)...'\n đIntrinsic Score:4.17/10.0(URLquality&context)\n đŻContextual Score:1.000(BM25relevancetoquery)\n âTotal Score:5.917(combinedfinalscore)\n đ°Title:3.15.0a0Documentation...\n đDescription:TheofficialPythondocumentation...\n â Extraction Status:valid\n
\n
2.3 Configuration Deep Dive
\n
The LinkPreviewConfig class supports these options:
\n
from crawl4ai import LinkPreviewConfig\n\nlink_preview_config = LinkPreviewConfig(\n # BASIC SETTINGS\n verbose=True, # Show detailed logs (recommended for learning)\n\n # LINK FILTERING\n include_internal=True, # Include same-domain links\n include_external=True, # Include different-domain links\n max_links=50, # Maximum links to process (prevents overload)\n\n # PATTERN FILTERING\n include_patterns=[ # Only process links matching these patterns\n \"*/docs/*\", \n \"*/api/*\", \n \"*/reference/*\"\n ],\n exclude_patterns=[ # Skip links matching these patterns\n \"*/login*\",\n \"*/admin*\"\n ],\n\n # PERFORMANCE SETTINGS\n concurrency=10, # How many links to process simultaneously\n timeout=5, # Seconds to wait per link\n\n # RELEVANCE SCORING\n query=\"machine learning API\", # Query for BM25 contextual scoring\n score_threshold=0.3, # Only include links above this score\n)\n
\n
2.4 Understanding the Three Score Types
\n
Each extracted link gets three different scores:
\n
1. Intrinsic Score (0-10) - URL and Content Quality
\n
Based on URL structure, link text quality, and page context:
\n
# High intrinsic score indicators:\n# â Clean URL structure (docs.python.org/api/reference)\n# â Meaningful link text (\"API Reference Guide\")\n# â Relevant to page context\n# â Not buried deep in navigation\n\n# Low intrinsic score indicators:\n# â Random URLs (site.com/x7f9g2h)\n# â No link text or generic text (\"Click here\")\n# â Unrelated to page content\n
\n
2. Contextual Score (0-1) - BM25 Relevance to Query
\n
Only available when you provide a query. Uses BM25 algorithm against head content:
\n
# Example: query = \"machine learning tutorial\"\n# High contextual score: Link to \"Complete Machine Learning Guide\"\n# Low contextual score: Link to \"Privacy Policy\"\n
\n
3. Total Score - Smart Combination
\n
Intelligently combines intrinsic and contextual scores with fallbacks:
\n
# When both scores available: (intrinsic * 0.3) + (contextual * 0.7)\n# When only intrinsic: uses intrinsic score\n# When only contextual: uses contextual score\n# When neither: not calculated\n
\n
2.5 Practical Use Cases
\n
Use Case 1: Research Assistant
\n
Find the most relevant documentation pages:
\n
asyncdefresearch_assistant():\n config = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n include_internal=True,\n include_external=True,\n include_patterns=[\"*/docs/*\", \"*/tutorial/*\", \"*/guide/*\"],\n query=\"machine learning neural networks\",\n max_links=20,\n score_threshold=0.5, # Only high-relevance links\n verbose=True\n ),\n score_links=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://scikit-learn.org/\", config=config)\n\n if result.success:\n # Get high-scoring links\n good_links = [link for link in result.links.get(\"internal\", [])\n if link.get(\"total_score\", 0) > 0.7]\n\n print(f\"đŻ Found {len(good_links)} highly relevant links:\")\n for link in good_links[:5]:\n print(f\"â {link['total_score']:.3f} - {link['href']}\")\n print(f\" {link.get('head_data', {}).get('title', 'No title')}\")\n
\n
Use Case 2: Content Discovery
\n
Find all API endpoints and references:
\n
asyncdefapi_discovery():\n config = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n include_internal=True,\n include_patterns=[\"*/api/*\", \"*/reference/*\"],\n exclude_patterns=[\"*/deprecated/*\"],\n max_links=100,\n concurrency=15,\n verbose=False# Clean output\n ),\n score_links=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://docs.example-api.com/\", config=config)\n\n if result.success:\n api_links = result.links.get(\"internal\", [])\n\n # Group by endpoint type\n endpoints = {}\n for link in api_links:\n if link.get(\"head_data\"):\n title = link[\"head_data\"].get(\"title\", \"\")\n if\"GET\"in title:\n endpoints.setdefault(\"GET\", []).append(link)\n elif\"POST\"in title:\n endpoints.setdefault(\"POST\", []).append(link)\n\n for method, links in endpoints.items():\n print(f\"\\n{method} Endpoints ({len(links)}):\")\n for link in links[:3]:\n print(f\" ⢠{link['href']}\")\n
\n
Use Case 3: Link Quality Analysis
\n
Analyze website structure and content quality:
\n
asyncdefquality_analysis():\n config = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n include_internal=True,\n max_links=200,\n concurrency=20,\n ),\n score_links=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://your-website.com/\", config=config)\n\n if result.success:\n links = result.links.get(\"internal\", [])\n\n # Analyze intrinsic scores\n scores = [link.get('intrinsic_score', 0) for link in links]\n avg_score = sum(scores) / len(scores) if scores else0\n\n print(f\"đ Link Quality Analysis:\")\n print(f\" Average intrinsic score: {avg_score:.2f}/10.0\")\n print(f\" High quality links (>7.0): {len([s for s in scores if s > 7.0])}\")\n print(f\" Low quality links (<3.0): {len([s for s in scores if s < 3.0])}\")\n\n # Find problematic links\n bad_links = [link for link in links \n if link.get('intrinsic_score', 0) < 2.0]\n\n if bad_links:\n print(f\"\\nâ ď¸ Links needing attention:\")\n for link in bad_links[:5]:\n print(f\" {link['href']} (score: {link.get('intrinsic_score', 0):.1f})\")\n
\n
2.6 Performance Tips
\n\n
Start Small: Begin with max_links: 10 to understand the feature
\n
Use Patterns: Filter with include_patterns to focus on relevant sections
\n
Adjust Concurrency: Higher concurrency = faster but more resource usage
\n
Set Timeouts: Use timeout: 5 to prevent hanging on slow sites
\n
Use Score Thresholds: Filter out low-quality links with score_threshold
\n\n
2.7 Troubleshooting
\n
No head data extracted?\n
# Check your configuration:\nconfig = CrawlerRunConfig(\n link_preview_config=LinkPreviewConfig(\n verbose=True# â Enable to see what's happening\n )\n)\n
\n
Scores showing as None?\n
# Make sure scoring is enabled:\nconfig = CrawlerRunConfig(\n score_links=True, # â Enable intrinsic scoring\n link_preview_config=LinkPreviewConfig(\n query=\"your search terms\"# â For contextual scoring\n )\n)\n
Some websites contain hundreds of third-party or affiliate links. You can filter out certain domains at crawl time by configuring the crawler. The most relevant parameters in CrawlerRunConfig are:
\n
\n
exclude_external_links: If True, discard any link pointing outside the root domain.
\n
exclude_social_media_domains: Provide a list of social media platforms (e.g., [\"facebook.com\", \"twitter.com\"]) to exclude from your crawl.
\n
exclude_social_media_links: If True, automatically skip known social platforms.
\n
exclude_domains: Provide a list of custom domains you want to exclude (e.g., [\"spammyads.com\", \"tracker.net\"]).
\n
\n
3.1 Example: Excluding External & Social Media Links
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n crawler_cfg = CrawlerRunConfig(\n exclude_external_links=True, # No links outside primary domain\n exclude_social_media_links=True# Skip recognized social media domains\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n \"https://www.example.com\",\n config=crawler_cfg\n )\n if result.success:\n print(\"[OK] Crawled:\", result.url)\n print(\"Internal links count:\", len(result.links.get(\"internal\", [])))\n print(\"External links count:\", len(result.links.get(\"external\", []))) \n # Likely zero external links in this scenario\n else:\n print(\"[ERROR]\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
3.2 Example: Excluding Specific Domains
\n
If you want to let external links in, but specifically exclude a domain (e.g., suspiciousads.com), do this:
This approach is handy when you still want external links but need to block certain sites you consider spammy.
\n\n
4. Media Extraction
\n
4.1 Accessing result.media
\n
By default, Crawl4AI collects images, audio and video URLs it finds on the page. These are stored in result.media, a dictionary keyed by media type (e.g., images, videos, audio).\nNote: Tables have been moved from result.media[\"tables\"] to the new result.tables format for better organization and direct access.
\n
Basic Example:
\n
if result.success:\n # Get images\n images_info = result.media.get(\"images\", [])\n print(f\"Found {len(images_info)} images in total.\")\n for i, img inenumerate(images_info[:3]): # Inspect just the first 3\n print(f\"[Image {i}] URL: {img['src']}\")\n print(f\" Alt text: {img.get('alt', '')}\")\n print(f\" Score: {img.get('score')}\")\n print(f\" Description: {img.get('desc', '')}\\n\")\n
\n
Structure Example:
\n
result.media = {\n \"images\": [\n {\n \"src\": \"https://cdn.prod.website-files.com/.../Group%2089.svg\",\n \"alt\": \"coding school for kids\",\n \"desc\": \"Trial Class Degrees degrees All Degrees AI Degree Technology ...\",\n \"score\": 3,\n \"type\": \"image\",\n \"group_id\": 0,\n \"format\": None,\n \"width\": None,\n \"height\": None\n },\n # ...\n ],\n \"videos\": [\n # Similar structure but with video-specific fields\n ],\n \"audio\": [\n # Similar structure but with audio-specific fields\n ],\n}\n
\n
Depending on your Crawl4AI version or scraping strategy, these dictionaries can include fields like:
\n
\n
src: The media URL (e.g., image source)
\n
alt: The alt text for images (if present)
\n
desc: A snippet of nearby text or a short description (optional)
\n
score: A heuristic relevance score if youâre using content-scoring features
\n
width, height: If the crawler detects dimensions for the image/video
\n
type: Usually \"image\", \"video\", or \"audio\"
\n
group_id: If youâre grouping related media items, the crawler might assign an ID
\n
\n
With these details, you can easily filter out or focus on certain images (for instance, ignoring images with very low scores or a different domain), or gather metadata for analytics.
\n
4.2 Excluding External Images
\n
If youâre dealing with heavy pages or want to skip third-party images (advertisements, for example), you can turn on:
This setting attempts to discard images from outside the primary domain, keeping only those from the site youâre crawling.
\n
4.3 Additional Media Config
\n
\n
screenshot: Set to True if you want a full-page screenshot stored as base64 in result.screenshot.
\n
pdf: Set to True if you want a PDF version of the page in result.pdf.
\n
capture_mhtml: Set to True if you want an MHTML snapshot of the page in result.mhtml. This format preserves the entire web page with all its resources (CSS, images, scripts) in a single file, making it perfect for archiving or offline viewing.
\n
wait_for_images: If True, attempts to wait until images are fully loaded before final extraction.
\n
\n
Example: Capturing Page as MHTML
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n crawler_cfg = CrawlerRunConfig(\n capture_mhtml=True# Enable MHTML capture\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=crawler_cfg)\n\n if result.success and result.mhtml:\n # Save the MHTML snapshot to a file\n withopen(\"example.mhtml\", \"w\", encoding=\"utf-8\") as f:\n f.write(result.mhtml)\n print(\"MHTML snapshot saved to example.mhtml\")\n else:\n print(\"Failed to capture MHTML:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
The MHTML format is particularly useful because:\n- It captures the complete page state including all resources\n- It can be opened in most modern browsers for offline viewing\n- It preserves the page exactly as it appeared during crawling\n- It's a single file, making it easy to store and transfer
\n\n
5. Putting It All Together: Link & Media Filtering
\n
Hereâs a combined example demonstrating how to filter out external links, skip certain domains, and exclude external images:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig\n\nasyncdefmain():\n # Suppose we want to keep only internal links, remove certain domains, \n # and discard external images from the final crawl data.\n crawler_cfg = CrawlerRunConfig(\n exclude_external_links=True,\n exclude_domains=[\"spammyads.com\"],\n exclude_social_media_links=True, # skip Twitter, Facebook, etc.\n exclude_external_images=True, # keep only images from main domain\n wait_for_images=True, # ensure images are loaded\n verbose=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://www.example.com\", config=crawler_cfg)\n\n if result.success:\n print(\"[OK] Crawled:\", result.url)\n\n # 1. Links\n in_links = result.links.get(\"internal\", [])\n ext_links = result.links.get(\"external\", [])\n print(\"Internal link count:\", len(in_links))\n print(\"External link count:\", len(ext_links)) # should be zero with exclude_external_links=True\n\n # 2. Images\n images = result.media.get(\"images\", [])\n print(\"Images found:\", len(images))\n\n # Let's see a snippet of these images\n for i, img inenumerate(images[:3]):\n print(f\" - {img['src']} (alt={img.get('alt','')}, score={img.get('score','N/A')})\")\n else:\n print(\"[ERROR] Failed to crawl. Reason:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n
6. Common Pitfalls & Tips
\n
1.âConflicting Flags: \n - exclude_external_links=True but then also specifying exclude_social_media_links=True is typically fine, but understand that the first setting already discards all external links. The second becomes somewhat redundant. \n - exclude_external_images=True but want to keep some external images? Currently no partial domain-based setting for images, so you might need a custom approach or hook logic.
\n
2.âRelevancy Scores: \n - If your version of Crawl4AI or your scraping strategy includes an img[\"score\"], itâs typically a heuristic based on size, position, or content analysis. Evaluate carefully if you rely on it.
\n
3.âPerformance: \n - Excluding certain domains or external images can speed up your crawl, especially for large, media-heavy pages. \n - If you want a âfullâ link map, do not exclude them. Instead, you can post-filter in your own code.
\n
4.âSocial Media Lists: \n - exclude_social_media_links=True typically references an internal list of known social domains like Facebook, Twitter, LinkedIn, etc. If you need to add or remove from that list, look for library settings or a local config file (depending on your version).
\n\n
Thatâs it for Link & Media Analysis! Youâre now equipped to filter out unwanted sites and zero in on the images and videos that matter for your project.
This guide will walk you through using the Crawl4AI library to crawl web pages, local HTML files, and raw HTML strings. We'll demonstrate these capabilities using a Wikipedia page as an example.
\n
Crawling a Web URL
\n
To crawl a live web page, provide the URL starting with http:// or https://, using a CrawlerRunConfig object:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig\n\nasyncdefcrawl_web():\n config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://en.wikipedia.org/wiki/apple\", \n config=config\n )\n if result.success:\n print(\"Markdown Content:\")\n print(result.markdown)\n else:\n print(f\"Failed to crawl: {result.error_message}\")\n\nasyncio.run(crawl_web())\n
\n
Crawling a Local HTML File
\n
To crawl a local HTML file, prefix the file path with file://.
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig\n\nasyncdefcrawl_local_file():\n local_file_path = \"/path/to/apple.html\"# Replace with your file path\n file_url = f\"file://{local_file_path}\"\n config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=file_url, config=config)\n if result.success:\n print(\"Markdown Content from Local File:\")\n print(result.markdown)\n else:\n print(f\"Failed to crawl local file: {result.error_message}\")\n\nasyncio.run(crawl_local_file())\n
\n
Crawling Raw HTML Content
\n
To crawl raw HTML content, prefix the HTML string with raw:.
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CacheMode\nfrom crawl4ai.async_configs import CrawlerRunConfig\n\nasyncdefcrawl_raw_html():\n raw_html = \"<html><body><h1>Hello, World!</h1></body></html>\"\n raw_html_url = f\"raw:{raw_html}\"\n config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url=raw_html_url, config=config)\n if result.success:\n print(\"Markdown Content from Raw HTML:\")\n print(result.markdown)\n else:\n print(f\"Failed to crawl raw HTML: {result.error_message}\")\n\nasyncio.run(crawl_raw_html())\n
\n\n
Complete Example
\n
Below is a comprehensive script that:
\n\n
Crawls the Wikipedia page for \"Apple.\"
\n
Saves the HTML content to a local file (apple.html).
\n
Crawls the local HTML file and verifies the markdown length matches the original crawl.
\n
Crawls the raw HTML content from the saved file and verifies consistency.
\n\n
import os\nimport sys\nimport asyncio\nfrom pathlib import Path\nfrom crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig\n\nasyncdefmain():\n wikipedia_url = \"https://en.wikipedia.org/wiki/apple\"\n script_dir = Path(__file__).parent\n html_file_path = script_dir / \"apple.html\"\n\n asyncwith AsyncWebCrawler() as crawler:\n # Step 1: Crawl the Web URL\n print(\"\\n=== Step 1: Crawling the Wikipedia URL ===\")\n web_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n result = await crawler.arun(url=wikipedia_url, config=web_config)\n\n ifnot result.success:\n print(f\"Failed to crawl {wikipedia_url}: {result.error_message}\")\n return\n\n withopen(html_file_path, 'w', encoding='utf-8') as f:\n f.write(result.html)\n web_crawl_length = len(result.markdown)\n print(f\"Length of markdown from web crawl: {web_crawl_length}\\n\")\n\n # Step 2: Crawl from the Local HTML File\n print(\"=== Step 2: Crawling from the Local HTML File ===\")\n file_url = f\"file://{html_file_path.resolve()}\"\n file_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n local_result = await crawler.arun(url=file_url, config=file_config)\n\n ifnot local_result.success:\n print(f\"Failed to crawl local file {file_url}: {local_result.error_message}\")\n return\n\n local_crawl_length = len(local_result.markdown)\n assert web_crawl_length == local_crawl_length, \"Markdown length mismatch\"\n print(\"â Markdown length matches between web and local file crawl.\\n\")\n\n # Step 3: Crawl Using Raw HTML Content\n print(\"=== Step 3: Crawling Using Raw HTML Content ===\")\n withopen(html_file_path, 'r', encoding='utf-8') as f:\n raw_html_content = f.read()\n raw_html_url = f\"raw:{raw_html_content}\"\n raw_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n raw_result = await crawler.arun(url=raw_html_url, config=raw_config)\n\n ifnot raw_result.success:\n print(f\"Failed to crawl raw HTML content: {raw_result.error_message}\")\n return\n\n raw_crawl_length = len(raw_result.markdown)\n assert web_crawl_length == raw_crawl_length, \"Markdown length mismatch\"\n print(\"â Markdown length matches between web and raw HTML crawl.\\n\")\n\n print(\"All tests passed successfully!\")\n if html_file_path.exists():\n os.remove(html_file_path)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n
Conclusion
\n
With the unified url parameter and prefix-based handling in Crawl4AI, you can seamlessly handle web URLs, local HTML files, and raw HTML content. Use CrawlerRunConfig for flexible and consistent configuration in all scenarios.
One of Crawl4AIâs core features is generating clean, structured markdown from web pages. Originally built to solve the problem of extracting only the âactualâ content and discarding boilerplate or noise, Crawl4AIâs markdown system remains one of its biggest draws for AI workflows.
\n
In this tutorial, youâll learn:
\n\n
How to configure the Default Markdown Generator
\n
How content filters (BM25 or Pruning) help you refine markdown and discard junk
\n
The difference between raw markdown (result.markdown) and filtered markdown (fit_markdown)
\n\n
\n
Prerequisites \n- Youâve completed or read AsyncWebCrawler Basics to understand how to run a simple crawl. \n- You know how to configure CrawlerRunConfig.
\n
\n\n
1. Quick Example
\n
Hereâs a minimal code snippet that uses the DefaultMarkdownGenerator with no additional filtering:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nasyncdefmain():\n config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator()\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n\n if result.success:\n print(\"Raw Markdown Output:\\n\")\n print(result.markdown) # The unfiltered markdown from the page\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Whatâs happening? \n- CrawlerRunConfig( markdown_generator = DefaultMarkdownGenerator() ) instructs Crawl4AI to convert the final HTML into markdown at the end of each crawl. \n- The resulting markdown is accessible via result.markdown.
\n\n
2. How Markdown Generation Works
\n
2.1 HTML-to-Text Conversion (Forked & Modified)
\n
Under the hood, DefaultMarkdownGenerator uses a specialized HTML-to-text approach that:
\n
\n
Preserves headings, code blocks, bullet points, etc.
\n
Removes extraneous tags (scripts, styles) that donât add meaningful content.
\n
Can optionally generate references for links or skip them altogether.
\n
\n
A set of options (passed as a dict) allows you to customize precisely how HTML converts to markdown. These map to standard html2text-like configuration plus your own enhancements (e.g., ignoring internal links, preserving certain tags verbatim, or adjusting line widths).
\n
2.2 Link Citations & References
\n
By default, the generator can convert <a href=\"...\"> elements into [text][1] citations, then place the actual links at the bottom of the document. This is handy for research workflows that demand references in a structured manner.
\n
2.3 Optional Content Filters
\n
Before or after the HTML-to-Markdown step, you can apply a content filter (like BM25 or Pruning) to reduce noise and produce a âfit_markdownââa heavily pruned version focusing on the pageâs main text. Weâll cover these filters shortly.
\n\n
3. Configuring the Default Markdown Generator
\n
You can tweak the output by passing an options dict to DefaultMarkdownGenerator. For example:
\n
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Example: ignore all links, don't escape HTML, and wrap text at 80 characters\n md_generator = DefaultMarkdownGenerator(\n options={\n \"ignore_links\": True,\n \"escape_html\": False,\n \"body_width\": 80\n }\n )\n\n config = CrawlerRunConfig(\n markdown_generator=md_generator\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com/docs\", config=config)\n if result.success:\n print(\"Markdown:\\n\", result.markdown[:500]) # Just a snippet\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n import asyncio\n asyncio.run(main())\n
\n
Some commonly used options:
\n
\n
ignore_links (bool): Whether to remove all hyperlinks in the final markdown.
\n
ignore_images (bool): Remove all ![image]() references.
\n
escape_html (bool): Turn HTML entities into text (default is often True).
\n
body_width (int): Wrap text at N characters. 0 or None means no wrapping.
\n
skip_internal_links (bool): If True, omit #localAnchors or internal links referencing the same page.
\n
include_sup_sub (bool): Attempt to handle <sup> / <sub> in a more readable way.
\n
\n
4. Selecting the HTML Source for Markdown Generation
\n
The content_source parameter allows you to control which HTML content is used as input for markdown generation. This gives you flexibility in how the HTML is processed before conversion to markdown.
\n
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Option 1: Use the raw HTML directly from the webpage (before any processing)\n raw_md_generator = DefaultMarkdownGenerator(\n content_source=\"raw_html\",\n options={\"ignore_links\": True}\n )\n\n # Option 2: Use the cleaned HTML (after scraping strategy processing - default)\n cleaned_md_generator = DefaultMarkdownGenerator(\n content_source=\"cleaned_html\", # This is the default\n options={\"ignore_links\": True}\n )\n\n # Option 3: Use preprocessed HTML optimized for schema extraction\n fit_md_generator = DefaultMarkdownGenerator(\n content_source=\"fit_html\",\n options={\"ignore_links\": True}\n )\n\n # Use one of the generators in your crawler config\n config = CrawlerRunConfig(\n markdown_generator=raw_md_generator # Try each of the generators\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n if result.success:\n print(\"Markdown:\\n\", result.markdown.raw_markdown[:500])\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n import asyncio\n asyncio.run(main())\n
\n
HTML Source Options
\n
\n
\n
\"cleaned_html\" (default): Uses the HTML after it has been processed by the scraping strategy. This HTML is typically cleaner and more focused on content, with some boilerplate removed.
\n
\n
\n
\"raw_html\": Uses the original HTML directly from the webpage, before any cleaning or processing. This preserves more of the original content, but may include navigation bars, ads, footers, and other elements that might not be relevant to the main content.
\n
\n
\n
\"fit_html\": Uses HTML preprocessed for schema extraction. This HTML is optimized for structured data extraction and may have certain elements simplified or removed.
\n
\n
\n
When to Use Each Option
\n
\n
Use \"cleaned_html\" (default) for most cases where you want a balance of content preservation and noise removal.
\n
Use \"raw_html\" when you need to preserve all original content, or when the cleaning process is removing content you actually want to keep.
\n
Use \"fit_html\" when working with structured data or when you need HTML that's optimized for schema extraction.
\n
\n\n
5. Content Filters
\n
Content filters selectively remove or rank sections of text before turning them into Markdown. This is especially helpful if your page has ads, nav bars, or other clutter you donât want.
\n
5.1 BM25ContentFilter
\n
If you have a search query, BM25 is a good choice:
user_query: The term you want to focus on. BM25 tries to keep only content blocks relevant to that query.
\n
bm25_threshold: Raise it to keep fewer blocks; lower it to keep more.
\n
use_stemming(default True): Whether to apply stemming to the query and content.
\n
language (str): Language for stemming (default: 'english').
\n
\n
No query provided? BM25 tries to glean a context from page metadata, or you can simply treat it as a scorched-earth approach that discards text with low generic score. Realistically, you want to supply a query for best results.
\n
5.2 PruningContentFilter
\n
If you donât have a specific query, or if you just want a robust âjunk remover,â use PruningContentFilter. It analyzes text density, link density, HTML structure, and known patterns (like ânav,â âfooterâ) to systematically prune extraneous or repetitive sections.
\n
from crawl4ai.content_filter_strategy import PruningContentFilter\n\nprune_filter = PruningContentFilter(\n threshold=0.5,\n threshold_type=\"fixed\", # or \"dynamic\"\n min_word_threshold=50\n)\n
\n
\n
threshold: Score boundary. Blocks below this score get removed.
\n
threshold_type:
\n
\"fixed\": Straight comparison (score >= threshold keeps the block).
\n
\"dynamic\": The filter adjusts threshold in a data-driven manner.
\n
\n
\n
min_word_threshold: Discard blocks under N words as likely too short or unhelpful.
\n
\n
When to Use PruningContentFilter \n- You want a broad cleanup without a user query. \n- The page has lots of repeated sidebars, footers, or disclaimers that hamper text extraction.
\n
5.3 LLMContentFilter
\n
For intelligent content filtering and high-quality markdown generation, you can use the LLMContentFilter. This filter leverages LLMs to generate relevant markdown while preserving the original content's meaning and structure:
\n
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, LLMConfig, DefaultMarkdownGenerator\nfrom crawl4ai.content_filter_strategy import LLMContentFilter\n\nasyncdefmain():\n # Initialize LLM filter with specific instruction\n filter = LLMContentFilter(\n llm_config = LLMConfig(provider=\"openai/gpt-4o\",api_token=\"your-api-token\"), #or use environment variable\n instruction=\"\"\"\n Focus on extracting the core educational content.\n Include:\n - Key concepts and explanations\n - Important code examples\n - Essential technical details\n Exclude:\n - Navigation elements\n - Sidebars\n - Footer content\n Format the output as clean markdown with proper code blocks and headers.\n \"\"\",\n chunk_token_threshold=4096, # Adjust based on your needs\n verbose=True\n )\n md_generator = DefaultMarkdownGenerator(\n content_filter=filter,\n options={\"ignore_links\": True}\n )\n config = CrawlerRunConfig(\n markdown_generator=md_generator,\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\", config=config)\n print(result.markdown.fit_markdown) # Filtered markdown content\n
\n
Key Features:\n- Intelligent Filtering: Uses LLMs to understand and extract relevant content while maintaining context\n- Customizable Instructions: Tailor the filtering process with specific instructions\n- Chunk Processing: Handles large documents by processing them in chunks (controlled by chunk_token_threshold)\n- Parallel Processing: For better performance, use smaller chunk_token_threshold (e.g., 2048 or 4096) to enable parallel processing of content chunks
\n
Two Common Use Cases:
\n\n
\n
Exact Content Preservation:\n
filter = LLMContentFilter(\n instruction=\"\"\"\n Extract the main educational content while preserving its original wording and substance completely.\n 1. Maintain the exact language and terminology\n 2. Keep all technical explanations and examples intact\n 3. Preserve the original flow and structure\n 4. Remove only clearly irrelevant elements like navigation menus and ads\n \"\"\",\n chunk_token_threshold=4096\n)\n
\n
\n
\n
Focused Content Extraction:\n
filter = LLMContentFilter(\n instruction=\"\"\"\n Focus on extracting specific types of content:\n - Technical documentation\n - Code examples\n - API references\n Reformat the content into clear, well-structured markdown\n \"\"\",\n chunk_token_threshold=4096\n)\n
\n
\n\n
\n
Performance Tip: Set a smaller chunk_token_threshold (e.g., 2048 or 4096) to enable parallel processing of content chunks. The default value is infinity, which processes the entire content as a single chunk.
\n
\n\n
6. Using Fit Markdown
\n
When a content filter is active, the library produces two forms of markdown inside result.markdown:
\n
1.âraw_markdown: The full unfiltered markdown. \n2.âfit_markdown: A âfitâ version where the filter has removed or trimmed noisy segments.
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\nfrom crawl4ai.content_filter_strategy import PruningContentFilter\n\nasyncdefmain():\n config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.6),\n options={\"ignore_links\": True}\n )\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://news.example.com/tech\", config=config)\n if result.success:\n print(\"Raw markdown:\\n\", result.markdown)\n\n # If a filter is used, we also have .fit_markdown:\n md_object = result.markdown # or your equivalent\n print(\"Filtered markdown:\\n\", md_object.fit_markdown)\n else:\n print(\"Crawl failed:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n
7. The MarkdownGenerationResult Object
\n
If your library stores detailed markdown output in an object like MarkdownGenerationResult, youâll see fields such as:
\n
\n
raw_markdown: The direct HTML-to-markdown transformation (no filtering).
\n
markdown_with_citations: A version that moves links to reference-style footnotes.
\n
references_markdown: A separate string or section containing the gathered references.
\n
fit_markdown: The filtered markdown if you used a content filter.
\n
fit_html: The corresponding HTML snippet used to generate fit_markdown (helpful for debugging or advanced usage).
\n
\n
Example:
\n
md_obj = result.markdown # your libraryâs naming may vary\nprint(\"RAW:\\n\", md_obj.raw_markdown)\nprint(\"CITED:\\n\", md_obj.markdown_with_citations)\nprint(\"REFERENCES:\\n\", md_obj.references_markdown)\nprint(\"FIT:\\n\", md_obj.fit_markdown)\n
\n
Why Does This Matter? \n- You can supply raw_markdown to an LLM if you want the entire text. \n- Or feed fit_markdown into a vector database to reduce token usage. \n- references_markdown can help you keep track of link provenance.
\n\n
Below is a revised section under âCombining Filters (BM25 + Pruning)â that demonstrates how you can run two passes of content filtering without re-crawling, by taking the HTML (or text) from a first pass and feeding it into the second filter. It uses real code patterns from the snippet you provided for BM25ContentFilter, which directly accepts HTML strings (and can also handle plain text with minimal adaptation).
\n\n
8. Combining Filters (BM25 + Pruning) in Two Passes
\n
You might want to prune out noisy boilerplate first (with PruningContentFilter), and then rank whatâs left against a user query (with BM25ContentFilter). You donât have to crawl the page twice. Instead:
\n
1.âFirst pass: Apply PruningContentFilter directly to the raw HTML from result.html (the crawlerâs downloaded HTML). \n2.âSecond pass: Take the pruned HTML (or text) from step 1, and feed it into BM25ContentFilter, focusing on a user query.
\n
Two-Pass Example
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter\nfrom bs4 import BeautifulSoup\n\nasyncdefmain():\n # 1. Crawl with minimal or no markdown generator, just get raw HTML\n config = CrawlerRunConfig(\n # If you only want raw HTML, you can skip passing a markdown_generator\n # or provide one but focus on .html in this example\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com/tech-article\", config=config)\n\n ifnot result.success ornot result.html:\n print(\"Crawl failed or no HTML content.\")\n return\n\n raw_html = result.html\n\n # 2. First pass: PruningContentFilter on raw HTML\n pruning_filter = PruningContentFilter(threshold=0.5, min_word_threshold=50)\n\n # filter_content returns a list of \"text chunks\" or cleaned HTML sections\n pruned_chunks = pruning_filter.filter_content(raw_html)\n # This list is basically pruned content blocks, presumably in HTML or text form\n\n # For demonstration, let's combine these chunks back into a single HTML-like string\n # or you could do further processing. It's up to your pipeline design.\n pruned_html = \"\\n\".join(pruned_chunks)\n\n # 3. Second pass: BM25ContentFilter with a user query\n bm25_filter = BM25ContentFilter(\n user_query=\"machine learning\",\n bm25_threshold=1.2,\n language=\"english\"\n )\n\n # returns a list of text chunks\n bm25_chunks = bm25_filter.filter_content(pruned_html) \n\n ifnot bm25_chunks:\n print(\"Nothing matched the BM25 query after pruning.\")\n return\n\n # 4. Combine or display final results\n final_text = \"\\n---\\n\".join(bm25_chunks)\n\n print(\"==== PRUNED OUTPUT (first pass) ====\")\n print(pruned_html[:500], \"... (truncated)\") # preview\n\n print(\"\\n==== BM25 OUTPUT (second pass) ====\")\n print(final_text[:500], \"... (truncated)\")\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Whatâs Happening?
\n
1.âRaw HTML: We crawl once and store the raw HTML in result.html. \n2.âPruningContentFilter: Takes HTML + optional parameters. It extracts blocks of text or partial HTML, removing headings/sections deemed ânoise.â It returns a list of text chunks. \n3.âCombine or Transform: We join these pruned chunks back into a single HTML-like string. (Alternatively, you could store them in a list for further logicâwhatever suits your pipeline.) \n4.âBM25ContentFilter: We feed the pruned string into BM25ContentFilter with a user query. This second pass further narrows the content to chunks relevant to âmachine learning.â
\n
No Re-Crawling: We used raw_html from the first pass, so thereâs no need to run arun() againâno second network request.
\n
Tips & Variations
\n
\n
Plain Text vs. HTML: If your pruned output is mostly text, BM25 can still handle it; just keep in mind it expects a valid string input. If you supply partial HTML (like \"<p>some text</p>\"), it will parse it as HTML.
\n
Chaining in a Single Pipeline: If your code supports it, you can chain multiple filters automatically. Otherwise, manual two-pass filtering (as shown) is straightforward.
\n
Adjust Thresholds: If you see too much or too little text in step one, tweak threshold=0.5 or min_word_threshold=50. Similarly, bm25_threshold=1.2 can be raised/lowered for more or fewer chunks in step two.
\n
\n
One-Pass Combination?
\n
If your codebase or pipeline design allows applying multiple filters in one pass, you could do so. But often itâs simplerâand more transparentâto run them sequentially, analyzing each stepâs result.
\n
Bottom Line: By manually chaining your filtering logic in two passes, you get powerful incremental control over the final content. First, remove âglobalâ clutter with Pruning, then refine further with BM25-based query relevanceâwithout incurring a second network crawl.
\n\n
9. Common Pitfalls & Tips
\n
1.âNo Markdown Output? \n - Make sure the crawler actually retrieved HTML. If the site is heavily JS-based, you may need to enable dynamic rendering or wait for elements. \n - Check if your content filter is too aggressive. Lower thresholds or disable the filter to see if content reappears.
\n
2.âPerformance Considerations \n - Very large pages with multiple filters can be slower. Consider cache_mode to avoid re-downloading. \n - If your final use case is LLM ingestion, consider summarizing further or chunking big texts.
\n
3.âTake Advantage of fit_markdown \n - Great for RAG pipelines, semantic search, or any scenario where extraneous boilerplate is unwanted. \n - Still verify the textual qualityâsome sites have crucial data in footers or sidebars.
\n
4.âAdjusting html2text Options \n - If you see lots of raw HTML slipping into the text, turn on escape_html. \n - If code blocks look messy, experiment with mark_code or handle_code_in_pre.
\n\n
10. Summary & Next Steps
\n
In this Markdown Generation Basics tutorial, you learned to:
\n
\n
Configure the DefaultMarkdownGenerator with HTML-to-text options.
\n
Select different HTML sources using the content_source parameter.
\n
Use BM25ContentFilter for query-specific extraction or PruningContentFilter for general noise removal.
\n
Distinguish between raw and filtered markdown (fit_markdown).
\n
Leverage the MarkdownGenerationResult object to handle different forms of output (citations, references, etc.).
\n
\n
Now you can produce high-quality Markdown from any website, focusing on exactly the content you needâan essential step for powering AI models, summarization pipelines, or knowledge-base queries.
Crawl4AI provides powerful features for interacting with dynamic webpages, handling JavaScript execution, waiting for conditions, and managing multi-step flows. By combining js_code, wait_for, and certain CrawlerRunConfig parameters, you can:
\n\n
Click âLoad Moreâ buttons
\n
Fill forms and submit them
\n
Wait for elements or data to appear
\n
Reuse sessions across multiple steps
\n\n
Below is a quick overview of how to do it.
\n\n
1. JavaScript Execution
\n
Basic Execution
\n
js_code in CrawlerRunConfig accepts either a single JS string or a list of JS snippets. It runs afterwait_for and delay_before_return_html â so the page is fully loaded when your code executes.
\n
Example: We'll scroll to the bottom of the page, then optionally click a \"Load More\" button.
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Single JS command\n config = CrawlerRunConfig(\n js_code=\"window.scrollTo(0, document.body.scrollHeight);\"\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", # Example site\n config=config\n )\n print(\"Crawled length:\", len(result.cleaned_html))\n\n # Multiple commands\n js_commands = [\n \"window.scrollTo(0, document.body.scrollHeight);\",\n # 'More' link on Hacker News\n \"document.querySelector('a.morelink')?.click();\", \n ]\n config = CrawlerRunConfig(js_code=js_commands)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\", # Another pass\n config=config\n )\n print(\"After scroll+click, length:\", len(result.cleaned_html))\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Relevant CrawlerRunConfig params:\n- js_code: JavaScript to run afterwait_for and delay_before_return_html complete. Runs on the fully-loaded page.\n- js_code_before_wait: JavaScript to run beforewait_for. Use when you need to trigger loading that wait_for then checks.\n- js_only: If set to True on subsequent calls, indicates we're continuing an existing session without a new full navigation.\n- session_id: If you want to keep the same page across multiple calls, specify an ID.
\n
Execution Order
\n
Understanding when your JavaScript runs relative to other pipeline steps:
\n
1. Page navigation (page.goto)\n2. js_code_before_wait â triggers loading / clicks tabs\n3. wait_for â waits for content to appear\n4. delay_before_return_html â extra safety margin\n5. js_code â runs on the fully-loaded page\n6. flatten_shadow_dom â if enabled\n7. page.content() â HTML capture\n
\n
If you need JS to trigger something and then wait for the result, use js_code_before_wait + wait_for:
\n
config = CrawlerRunConfig(\n # Click a tab first\n js_code_before_wait=\"document.querySelector('#specs-tab')?.click();\",\n # Then wait for the tab content to appear\n wait_for=\"css:#specs-panel .content\",\n)\n
\n\n
2. Wait Conditions
\n
2.1 CSS-Based Waiting
\n
Sometimes, you just want to wait for a specific element to appear. For example:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n config = CrawlerRunConfig(\n # Wait for at least 30 items on Hacker News\n wait_for=\"css:.athing:nth-child(30)\" \n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\",\n config=config\n )\n print(\"We have at least 30 items loaded!\")\n # Rough check\n print(\"Total items in HTML:\", result.cleaned_html.count(\"athing\")) \n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Key param:\n- wait_for=\"css:...\": Tells the crawler to wait until that CSS selector is present.
\n
2.2 JavaScript-Based Waiting
\n
For more complex conditions (e.g., waiting for content length to exceed a threshold), prefix js::
\n
wait_condition = \"\"\"() => {\n const items = document.querySelectorAll('.athing');\n return items.length > 50; // Wait for at least 51 items\n}\"\"\"\n\nconfig = CrawlerRunConfig(wait_for=f\"js:{wait_condition}\")\n
\n
Behind the Scenes: Crawl4AI keeps polling the JS function until it returns true or a timeout occurs.
\n\n
3. Handling Dynamic Content
\n
Many modern sites require multiple steps: scrolling, clicking âLoad More,â or updating via JavaScript. Below are typical patterns.
\n
3.1 Load More Example (Hacker News âMoreâ Link)
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefmain():\n # Step 1: Load initial Hacker News page\n config = CrawlerRunConfig(\n wait_for=\"css:.athing:nth-child(30)\"# Wait for 30 items\n )\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://news.ycombinator.com\",\n config=config\n )\n print(\"Initial items loaded.\")\n\n # Step 2: Let's scroll and click the \"More\" link\n load_more_js = [\n \"window.scrollTo(0, document.body.scrollHeight);\",\n # The \"More\" link at page bottom\n \"document.querySelector('a.morelink')?.click();\" \n ]\n\n next_page_conf = CrawlerRunConfig(\n js_code=load_more_js,\n wait_for=\"\"\"js:() => {\n return document.querySelectorAll('.athing').length > 30;\n }\"\"\",\n # Mark that we do not re-navigate, but run JS in the same session:\n js_only=True,\n session_id=\"hn_session\"\n )\n\n # Re-use the same crawler session\n result2 = await crawler.arun(\n url=\"https://news.ycombinator.com\", # same URL but continuing session\n config=next_page_conf\n )\n total_items = result2.cleaned_html.count(\"athing\")\n print(\"Items after load-more:\", total_items)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Key params:\n- session_id=\"hn_session\": Keep the same page across multiple calls to arun().\n- js_only=True: Weâre not performing a full reload, just applying JS in the existing page.\n- wait_for with js:: Wait for item count to grow beyond 30.
\n\n
3.2 Form Interaction
\n
If the site has a search or login form, you can fill fields and submit them with js_code. For instance, if GitHub had a local search form:
In reality: Replace IDs or classes with the real siteâs form selectors.
\n\n
4. Timing Control
\n
1.âpage_timeout (ms): Overall page load or script execution time limit. \n2.âdelay_before_return_html (seconds): Wait an extra moment before capturing the final HTML. \n3.âmean_delay & max_range: If you call arun_many() with multiple URLs, these add a random pause between each request.
Below is a simplified script that does multiple âLoad Moreâ clicks on GitHubâs TypeScript commits page. It re-uses the same session to accumulate new commits each time. The code includes the relevant CrawlerRunConfig parameters youâd rely on.
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasyncdefmulti_page_commits():\n browser_cfg = BrowserConfig(\n headless=False, # Visible for demonstration\n verbose=True\n )\n session_id = \"github_ts_commits\"\n\n base_wait = \"\"\"js:() => {\n const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');\n return commits.length > 0;\n }\"\"\"\n\n # Step 1: Load initial commits\n config1 = CrawlerRunConfig(\n wait_for=base_wait,\n session_id=session_id,\n cache_mode=CacheMode.BYPASS,\n # Not using js_only yet since it's our first load\n )\n\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n result = await crawler.arun(\n url=\"https://github.com/microsoft/TypeScript/commits/main\",\n config=config1\n )\n print(\"Initial commits loaded. Count:\", result.cleaned_html.count(\"commit\"))\n\n # Step 2: For subsequent pages, we run JS to click 'Next Page' if it exists\n js_next_page = \"\"\"\n const selector = 'a[data-testid=\"pagination-next-button\"]';\n const button = document.querySelector(selector);\n if (button) button.click();\n \"\"\"\n\n # Wait until new commits appear\n wait_for_more = \"\"\"js:() => {\n const commits = document.querySelectorAll('li.Box-sc-g0xbh4-0 h4');\n if (!window.firstCommit && commits.length>0) {\n window.firstCommit = commits[0].textContent;\n return false;\n }\n // If top commit changes, we have new commits\n const topNow = commits[0]?.textContent.trim();\n return topNow && topNow !== window.firstCommit;\n }\"\"\"\n\n for page inrange(2): # let's do 2 more \"Next\" pages\n config_next = CrawlerRunConfig(\n session_id=session_id,\n js_code=js_next_page,\n wait_for=wait_for_more,\n js_only=True, # We're continuing from the open tab\n cache_mode=CacheMode.BYPASS\n )\n result2 = await crawler.arun(\n url=\"https://github.com/microsoft/TypeScript/commits/main\",\n config=config_next\n )\n print(f\"Page {page+2} commits count:\", result2.cleaned_html.count(\"commit\"))\n\n # Optionally kill session\n await crawler.crawler_strategy.kill_session(session_id)\n\nasyncdefmain():\n await multi_page_commits()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Key Points:
\n
\n
session_id: Keep the same page open.
\n
js_code + wait_for + js_only=True: We do partial refreshes, waiting for new commits to appear.
\n
cache_mode=CacheMode.BYPASS ensures we always see fresh data each step.
\n
\n\n
6. Combine Interaction with Extraction
\n
Once dynamic content is loaded, you can attach an extraction_strategy (like JsonCssExtractionStrategy or LLMExtractionStrategy). For example:
When done, check result.extracted_content for the JSON.
\n\n
7. Shadow DOM Flattening
\n
Sites built with Web Components (Stencil, Lit, Shoelace, etc.) render content inside Shadow DOM â an encapsulated sub-tree that is invisible to normal page serialization. Set flatten_shadow_dom=True to extract it:
\n
config = CrawlerRunConfig(\n flatten_shadow_dom=True,\n wait_until=\"load\",\n delay_before_return_html=3.0, # give components time to hydrate\n)\n
\n
This walks all shadow trees, resolves <slot> projections, and produces flat HTML. It also force-opens closed shadow roots via an init script. For details and a full example, see Flattening Shadow DOM and shadow_dom_crawling.py.
\n\n
8. Relevant CrawlerRunConfig Parameters
\n
Below are the key interaction-related parameters in CrawlerRunConfig. For a full list, see Configuration Parameters.
\n
\n
js_code: JavaScript to run after wait_for + delay_before_return_html, on the fully-loaded page.
\n
js_code_before_wait: JavaScript to run before wait_for. For triggering loading that wait_for then checks.
\n
js_only: If True, no new page navigationâonly JS in the existing session.
\n
wait_for: CSS (\"css:...\") or JS (\"js:...\") expression to wait for.
\n
session_id: Reuse the same page across calls.
\n
cache_mode: Whether to read/write from the cache or bypass.
\n
flatten_shadow_dom: Flatten Shadow DOM content into the light DOM before capture.
\n
process_iframes: Inline iframe content into the main document.
\n
remove_overlay_elements: Remove certain popups automatically.
\n
remove_consent_popups: Remove GDPR/cookie consent popups from known CMP providers (OneTrust, Cookiebot, Didomi, etc.).
\n
simulate_user, override_navigator, magic: Anti-bot or \"human-like\" interactions.
\n
\n\n
9. Conclusion
\n
Crawl4AI's page interaction features let you:
\n
1.âExecute JavaScript for scrolling, clicks, or form filling. \n2.âWait for CSS or custom JS conditions before capturing data. \n3.âHandle multi-step flows (like âLoad Moreâ) with partial reloads or persistent sessions. \n4. Flatten Shadow DOM on Web Component sites to extract hidden content.\n5. Combine with structured extraction for dynamic sites.
\n
With these tools, you can scrape modern, interactive webpages confidently. For advanced hooking, user simulation, or in-depth config, check the API reference or related advanced docs. Happy scripting!
\n\n
10. Virtual Scrolling
\n
For sites that use virtual scrolling (where content is replaced rather than appended as you scroll, like Twitter or Instagram), Crawl4AI provides a dedicated VirtualScrollConfig:
\n
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig\n\nasyncdefcrawl_twitter_timeline():\n # Configure virtual scroll for Twitter-like feeds\n virtual_config = VirtualScrollConfig(\n container_selector=\"[data-testid='primaryColumn']\", # Twitter's main column\n scroll_count=30, # Scroll 30 times\n scroll_by=\"container_height\", # Scroll by container height each time\n wait_after_scroll=1.0# Wait 1 second after each scroll\n )\n\n config = CrawlerRunConfig(\n virtual_scroll_config=virtual_config\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://twitter.com/search?q=AI\",\n config=config\n )\n # result.html now contains ALL tweets from the virtual scroll\n
Welcome to Crawl4AI, an open-source LLM-friendly Web Crawler & Scraper. In this tutorial, youâll:
\n\n
Run your first crawl using minimal configuration.
\n
Generate Markdown output (and learn how itâs influenced by content filters).
\n
Experiment with a simple CSS-based extraction strategy.
\n
See a glimpse of LLM-based extraction (including open-source and closed-source model options).
\n
Crawl a dynamic page that loads content via JavaScript.
\n\n\n
1. Introduction
\n
Crawl4AI provides:
\n
\n
An asynchronous crawler, AsyncWebCrawler.
\n
Configurable browser and run settings via BrowserConfig and CrawlerRunConfig.
\n
Automatic HTML-to-Markdown conversion via DefaultMarkdownGenerator (supports optional filters).
\n
Multiple extraction strategies (LLM-based or âtraditionalâ CSS/XPath-based).
\n
\n
By the end of this guide, youâll have performed a basic crawl, generated Markdown, tried out two extraction strategies, and crawled a dynamic page that uses âLoad Moreâ buttons or JavaScript updates.
\n\n
2. Your First Crawl
\n
Hereâs a minimal Python script that creates an AsyncWebCrawler, fetches a webpage, and prints the first 300 characters of its Markdown output:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasyncdefmain():\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com\")\n print(result.markdown[:300]) # Print first 300 chars\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
Whatâs happening?\n- AsyncWebCrawler launches a headless browser (Chromium by default).\n- It fetches https://example.com.\n- Crawl4AI automatically converts the HTML into Markdown.
\n
You now have a simple, working crawl!
\n\n
3. Basic Configuration (Light Introduction)
\n
Crawl4AIâs crawler can be heavily customized using two main classes:
\n
1.âBrowserConfig: Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.). \n2.âCrawlerRunConfig: Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.).
\n
Below is an example with minimal usage:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode\n\nasyncdefmain():\n browser_conf = BrowserConfig(headless=True) # or False to see the browser\n run_conf = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS\n )\n\n asyncwith AsyncWebCrawler(config=browser_conf) as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=run_conf\n )\n print(result.markdown)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n
\n
IMPORTANT: By default cache mode is set to CacheMode.BYPASS to have fresh content. Set CacheMode.ENABLED to enable caching.
\n
\n
Weâll explore more advanced config in later tutorials (like enabling proxies, PDF output, multi-tab sessions, etc.). For now, just note how you pass these objects to manage crawling.
\n\n
4. Generating Markdown Output
\n
By default, Crawl4AI automatically generates Markdown from each crawled page. However, the exact output depends on whether you specify a markdown generator or content filter.
\n
\n
result.markdown: \n The direct HTML-to-Markdown conversion.
\n
result.markdown.fit_markdown: \n The same content after applying any configured content filter (e.g., PruningContentFilter).
\n
\n
Example: Using a Filter with DefaultMarkdownGenerator
Note: If you do not specify a content filter or markdown generator, youâll typically see only the raw Markdown. PruningContentFilter may adds around 50ms in processing time. Weâll dive deeper into these strategies in a dedicated Markdown Generation tutorial.
\n\n
5. Simple Data Extraction (CSS-based)
\n
Crawl4AI can also extract structured data (JSON) using CSS or XPath selectors. Below is a minimal CSS-based example:
\n
\n
New! Crawl4AI now provides a powerful utility to automatically generate extraction schemas using LLM. This is a one-time cost that gives you a reusable schema for fast, LLM-free extractions:
\n
\n
from crawl4ai import JsonCssExtractionStrategy\nfrom crawl4ai import LLMConfig\n\n# Generate a schema (one-time cost)\nhtml = \"<div class='product'><h2>Gaming Laptop</h2><span class='price'>$999.99</span></div>\"\n\n# Using OpenAI (requires API token)\nschema = JsonCssExtractionStrategy.generate_schema(\n html,\n llm_config = LLMConfig(provider=\"openai/gpt-4o\",api_token=\"your-openai-token\") # Required for OpenAI\n)\n\n# Or using Ollama (open source, no token needed)\nschema = JsonCssExtractionStrategy.generate_schema(\n html,\n llm_config = LLMConfig(provider=\"ollama/llama3.3\", api_token=None) # Not needed for Ollama\n)\n\n# Use the schema for fast, repeated extractions\nstrategy = JsonCssExtractionStrategy(schema)\n
Why is this helpful?\n- Great for repetitive page structures (e.g., item listings, articles).\n- No AI usage or costs.\n- The crawler returns a JSON string you can parse or store.
\n
\n
Tips: You can pass raw HTML to the crawler instead of a URL. To do so, prefix the HTML with raw://.
\n
\n\n
6. Simple Data Extraction (LLM-based)
\n
For more complex or irregular pages, a language model can parse text intelligently into a structure you define. Crawl4AI supports open-source or closed-source providers:
Or any provider supported by the underlying library
\n
\n
Below is an example using open-source style (no token) and closed-source:
\n
import os\nimport json\nimport asyncio\nfrom pydantic import BaseModel, Field\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\n\nclassOpenAIModelFee(BaseModel):\n model_name: str = Field(..., description=\"Name of the OpenAI model.\")\n input_fee: str = Field(..., description=\"Fee for input token for the OpenAI model.\")\n output_fee: str = Field(\n ..., description=\"Fee for output token for the OpenAI model.\"\n )\n\nasyncdefextract_structured_data_using_llm(\n provider: str, api_token: str = None, extra_headers: Dict[str, str] = None\n):\n print(f\"\\n--- Extracting Structured Data with {provider} ---\")\n\n if api_token isNoneand provider != \"ollama\":\n print(f\"API token is required for {provider}. Skipping this example.\")\n return\n\n browser_config = BrowserConfig(headless=True)\n\n extra_args = {\"temperature\": 0, \"top_p\": 0.9, \"max_tokens\": 2000}\n if extra_headers:\n extra_args[\"extra_headers\"] = extra_headers\n\n crawler_config = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n word_count_threshold=1,\n page_timeout=80000,\n extraction_strategy=LLMExtractionStrategy(\n llm_config = LLMConfig(provider=provider,api_token=api_token),\n schema=OpenAIModelFee.model_json_schema(),\n extraction_type=\"schema\",\n instruction=\"\"\"From the crawled content, extract all mentioned model names along with their fees for input and output tokens. \n Do not miss any models in the entire content.\"\"\",\n extra_args=extra_args,\n ),\n )\n\n asyncwith AsyncWebCrawler(config=browser_config) as crawler:\n result = await crawler.arun(\n url=\"https://openai.com/api/pricing/\", config=crawler_config\n )\n print(result.extracted_content)\n\nif __name__ == \"__main__\":\n\n asyncio.run(\n extract_structured_data_using_llm(\n provider=\"openai/gpt-4o\", api_token=os.getenv(\"OPENAI_API_KEY\")\n )\n )\n
\n
Whatâs happening?\n- We define a Pydantic schema (PricingInfo) describing the fields we want.\n- The LLM extraction strategy uses that schema and your instructions to transform raw text into structured JSON.\n- Depending on the provider and api_token, you can use local models or a remote API.
\n\n
7. Adaptive Crawling (New!)
\n
Crawl4AI now includes intelligent adaptive crawling that automatically determines when sufficient information has been gathered. Here's a quick example:
What's special about adaptive crawling?\n- Automatic stopping: Stops when sufficient information is gathered\n- Intelligent link selection: Follows only relevant links\n- Confidence scoring: Know how complete your information is
If you need to crawl multiple URLs in parallel, you can use arun_many(). By default, Crawl4AI employs a MemoryAdaptiveDispatcher, automatically adjusting concurrency based on system resources. Hereâs a quick glimpse:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\n\nasyncdefquick_parallel_example():\n urls = [\n \"https://example.com/page1\",\n \"https://example.com/page2\",\n \"https://example.com/page3\"\n ]\n\n run_conf = CrawlerRunConfig(\n cache_mode=CacheMode.BYPASS,\n stream=True# Enable streaming mode\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n # Stream results as they complete\n asyncfor result inawait crawler.arun_many(urls, config=run_conf):\n if result.success:\n print(f\"[OK] {result.url}, length: {len(result.markdown.raw_markdown)}\")\n else:\n print(f\"[ERROR] {result.url} => {result.error_message}\")\n\n # Or get all results at once (default behavior)\n run_conf = run_conf.clone(stream=False)\n results = await crawler.arun_many(urls, config=run_conf)\n for res in results:\n if res.success:\n print(f\"[OK] {res.url}, length: {len(res.markdown.raw_markdown)}\")\n else:\n print(f\"[ERROR] {res.url} => {res.error_message}\")\n\nif __name__ == \"__main__\":\n asyncio.run(quick_parallel_example())\n
\n
The example above shows two ways to handle multiple URLs:\n1. Streaming mode (stream=True): Process results as they become available using async for\n2. Batch mode (stream=False): Wait for all results to complete
\n
For more advanced concurrency (e.g., a semaphore-based approach, adaptive memory usage throttling, or customized rate limiting), see Advanced Multi-URL Crawling.
\n\n
8. Dynamic Content Example
\n
Some sites require multiple âpage clicksâ or dynamic JavaScript updates. Below is an example showing how to click a âNext Pageâ button and wait for new commits to load on GitHub, using BrowserConfig and CrawlerRunConfig:
This guide covers the basics of web crawling with Crawl4AI. You'll learn how to set up a crawler, make your first request, and understand the response.
\n
Basic Usage
\n
Set up a simple crawl using BrowserConfig and CrawlerRunConfig:
The arun() method returns a CrawlResult object with several useful properties. Here's a quick overview (see CrawlResult for complete details):
\n
config = CrawlerRunConfig(\n markdown_generator=DefaultMarkdownGenerator(\n content_filter=PruningContentFilter(threshold=0.6),\n options={\"ignore_links\": True}\n )\n)\n\nresult = await crawler.arun(\n url=\"https://example.com\",\n config=config\n)\n\n# Different content formats\nprint(result.html) # Raw HTML\nprint(result.cleaned_html) # Cleaned HTML\nprint(result.markdown.raw_markdown) # Raw markdown from cleaned html\nprint(result.markdown.fit_markdown) # Most relevant content in markdown\n\n# Check success status\nprint(result.success) # True if crawl succeeded\nprint(result.status_code) # HTTP status code (e.g., 200, 404)\n\n# Access extracted media and links\nprint(result.media) # Dictionary of found media (images, videos, audio)\nprint(result.links) # Dictionary of internal and external links\n
\n
Adding Basic Options
\n
Customize your crawl using CrawlerRunConfig:
\n
run_config = CrawlerRunConfig(\n word_count_threshold=10, # Minimum words per content block\n exclude_external_links=True, # Remove external links\n remove_overlay_elements=True, # Remove popups/modals\n process_iframes=True# Process iframe content\n)\n\nresult = await crawler.arun(\n url=\"https://example.com\",\n config=run_config\n)\n
New in v0.7.3+: Table extraction now follows the Strategy Design Pattern, providing unprecedented flexibility and power for handling different table structures. Don't worry - your existing code still works! We maintain full backward compatibility while offering new capabilities.
\n
What's Changed?
\n
\n
Architecture: Table extraction now uses pluggable strategies
\n
Backward Compatible: Your existing code with table_score_threshold continues to work
\n
More Power: Choose from multiple strategies or create your own
\n
Same Default Behavior: By default, uses DefaultTableExtraction (same as before)
\n
\n
Key Points
\n
â Old code still works - No breaking changes \nâ Same default behavior - Uses the proven extraction algorithm \nâ New capabilities - Add LLM extraction or custom strategies when needed \nâ Strategy pattern - Clean, extensible architecture
\n
Quick Start
\n
The Simplest Way (Works Like Before)
\n
If you're already using Crawl4AI, nothing changes:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefextract_tables():\n asyncwith AsyncWebCrawler() as crawler:\n # This works exactly like before - uses DefaultTableExtraction internally\n result = await crawler.arun(\"https://example.com/data\")\n\n # Tables are automatically extracted and available in result.tables\n for table in result.tables:\n print(f\"Table with {len(table['rows'])} rows and {len(table['headers'])} columns\")\n print(f\"Headers: {table['headers']}\")\n print(f\"First row: {table['rows'][0] if table['rows'] else'No data'}\")\n\nasyncio.run(extract_tables())\n
\n
Using the Old Configuration (Still Supported)
\n
Your existing code with table_score_threshold continues to work:
\n
# This old approach STILL WORKS - we maintain backward compatibility\nconfig = CrawlerRunConfig(\n table_score_threshold=7 # Internally creates DefaultTableExtraction(table_score_threshold=7)\n)\nresult = await crawler.arun(url, config)\n
\n
Table Extraction Strategies
\n
Understanding the Strategy Pattern
\n
The strategy pattern allows you to choose different table extraction algorithms at runtime. Think of it as having different tools in a toolbox - you pick the right one for the job:
\n
\n
No explicit strategy? â Uses DefaultTableExtraction automatically (same as v0.7.2 and earlier)
\n
Need complex table handling? â Choose LLMTableExtraction (costs money, use sparingly)
\n
Want to disable tables? â Use NoTableExtraction
\n
Have special requirements? â Create a custom strategy
\n
\n
Available Strategies
\n
\n\n
\n
Strategy
\n
Description
\n
Use Case
\n
Cost
\n
When to Use
\n
\n\n\n
\n
DefaultTableExtraction
\n
RECOMMENDED: Same algorithm as before v0.7.3
\n
General purpose (default)
\n
Free
\n
Use this first - handles 95% of cases
\n
\n
\n
LLMTableExtraction
\n
AI-powered extraction for complex tables
\n
Tables with complex rowspan/colspan
\n
$$$ Per API call
\n
Only when DefaultTableExtraction fails
\n
\n
\n
NoTableExtraction
\n
Disables table extraction
\n
When tables aren't needed
\n
Free
\n
For text-only extraction
\n
\n
\n
Custom strategies
\n
User-defined extraction logic
\n
Specialized requirements
\n
Free
\n
Domain-specific needs
\n
\n\n
\n
\n
â ď¸ CRITICAL COST WARNING for LLMTableExtraction:
\n
DO NOT USE LLMTableExtraction UNLESS ABSOLUTELY NECESSARY!
\n
\n
Always try DefaultTableExtraction first - It's free and handles most tables perfectly
\n
LLM extraction costs money with every API call
\n
For large tables (100+ rows), LLM extraction can be very slow
\n
For large tables: If you must use LLM, choose fast providers:
\n
â Groq (fastest inference)
\n
â Cerebras (optimized for speed)
\n
â ď¸ Avoid: OpenAI, Anthropic for large tables (slower)
\n
\n
đ§ WORK IN PROGRESS: \nWe are actively developing an advanced non-LLM algorithm that will handle complex table structures (rowspan, colspan, nested tables) for FREE. This will replace the need for costly LLM extraction in most cases. Coming soon!
\n
\n
DefaultTableExtraction
\n
The default strategy uses a sophisticated scoring system to identify data tables:
â ď¸ WARNING: Only use this when DefaultTableExtraction fails with complex tables!
\n
LLMTableExtraction uses AI to understand complex table structures that traditional parsers struggle with. It automatically handles large tables through intelligent chunking and parallel processing:
\n
from crawl4ai import LLMTableExtraction, LLMConfig, CrawlerRunConfig\n\n# Configure LLM (costs money per call!)\nllm_config = LLMConfig(\n provider=\"groq/llama-3.3-70b-versatile\", # Fast provider for large tables\n api_token=\"your_api_key\",\n temperature=0.1\n)\n\n# Create LLM extraction strategy with smart chunking\ntable_strategy = LLMTableExtraction(\n llm_config=llm_config,\n max_tries=3, # Retry up to 3 times if extraction fails\n css_selector=\"table\", # Optional: focus on specific tables\n enable_chunking=True, # Automatically chunk large tables (default: True)\n chunk_token_threshold=3000, # Split tables larger than this (default: 3000 tokens)\n min_rows_per_chunk=10, # Minimum rows per chunk (default: 10)\n max_parallel_chunks=5, # Process up to 5 chunks in parallel (default: 5)\n verbose=True\n)\n\nconfig = CrawlerRunConfig(\n table_extraction=table_strategy\n)\n\nresult = await crawler.arun(url, config)\n
\n
When to Use LLMTableExtraction
\n
â Use ONLY when:\n- Tables have complex merged cells (rowspan/colspan) that break DefaultTableExtraction\n- Nested tables that need semantic understanding\n- Tables with irregular structures\n- You've tried DefaultTableExtraction and it failed
\n
â Never use when:\n- DefaultTableExtraction works (99% of cases)\n- Tables are simple or well-structured\n- You're processing many pages (costs add up!)\n- Tables have 100+ rows (very slow)
\n
How Smart Chunking Works
\n
LLMTableExtraction automatically handles large tables through intelligent chunking:
\n\n
Automatic Detection: Tables exceeding the token threshold are automatically split
\n
Smart Splitting: Chunks are created at row boundaries, preserving table structure
\n
Header Preservation: Each chunk includes the original headers for context
\n
Parallel Processing: Multiple chunks are processed simultaneously for speed
\n
Intelligent Merging: Results are merged back into a single, complete table
\n\n
Chunking Parameters:\n- enable_chunking (default: True): Automatically handle large tables\n- chunk_token_threshold (default: 3000): When to split tables\n- min_rows_per_chunk (default: 10): Ensures meaningful chunk sizes\n- max_parallel_chunks (default: 5): Concurrent processing for speed
\n
The chunking is completely transparent - you get the same output format whether the table was processed in one piece or multiple chunks.
\n
Performance Optimization for LLMTableExtraction
\n
Provider Recommendations by Table Size:
\n
\n\n
\n
Table Size
\n
Recommended Providers
\n
Why
\n
\n\n\n
\n
Small (<50 rows)
\n
Any provider
\n
Fast enough
\n
\n
\n
Medium (50-200 rows)
\n
Groq, Cerebras
\n
Optimized inference
\n
\n
\n
Large (200+ rows)
\n
Groq (best), Cerebras
\n
Fastest inference + automatic chunking
\n
\n
\n
Very Large (500+ rows)
\n
Groq with chunking
\n
Parallel processing keeps it fast
\n
\n\n
\n
NoTableExtraction
\n
Disable table extraction for better performance when tables aren't needed:
\n
from crawl4ai import NoTableExtraction, CrawlerRunConfig\n\nconfig = CrawlerRunConfig(\n table_extraction=NoTableExtraction()\n)\n\n# Tables won't be extracted, improving performance\nresult = await crawler.arun(url, config)\nassertlen(result.tables) == 0\n
\n
Extracted Table Structure
\n
Each extracted table contains:
\n
{\n \"headers\":[\"Column 1\", \"Column 2\", ...], # Column headers\n \"rows\":[# Data rows\n [\"Row 1 Col 1\", \"Row 1 Col 2\", ...],\n [\"Row 2 Col 1\", \"Row 2 Col 2\", ...],\n ],\n \"caption\":\"Table Caption\", # If present\n \"summary\":\"Table Summary\", # If present\n \"metadata\":{\n \"row_count\":10, # Number of rows\n \"column_count\":3, # Number of columns\n \"has_headers\":True, # Headers detected\n \"has_caption\":True, # Caption exists\n \"has_summary\":False, # Summary exists\n \"id\":\"data-table-1\", # Table ID if present\n \"class\":\"financial-data\"# Table class if present\n }\n}\n
\n
Configuration Options
\n
Basic Configuration
\n
config = CrawlerRunConfig(\n # Table extraction settings\n table_score_threshold=7, # Default threshold (backward compatible)\n table_extraction=strategy, # Optional: custom strategy\n\n # Filter what to process\n css_selector=\"main\", # Focus on specific area\n excluded_tags=[\"nav\", \"aside\"] # Exclude page sections\n)\n
\n
Advanced Configuration
\n
from crawl4ai import DefaultTableExtraction, CrawlerRunConfig\n\n# Fine-tuned extraction\nstrategy = DefaultTableExtraction(\n table_score_threshold=5, # Lower = more permissive\n min_rows=3, # Require at least 3 rows\n min_cols=2, # Require at least 2 columns\n verbose=True# Detailed logging\n)\n\nconfig = CrawlerRunConfig(\n table_extraction=strategy,\n css_selector=\"article.content\", # Target specific content\n exclude_domains=[\"ads.com\"], # Exclude ad domains\n cache_mode=CacheMode.BYPASS # Fresh extraction\n)\n
\n
Working with Extracted Tables
\n
Convert to Pandas DataFrame
\n
import pandas as pd\n\nasync def tables_to_dataframes(url):\n asyncwithAsyncWebCrawler() as crawler:\n result = await crawler.arun(url)\n\n dataframes = []\n for table_data in result.tables:\n # Create DataFrame\n if table_data['headers']:\n df = pd.DataFrame(\n table_data['rows'],\n columns=table_data['headers']\n )\n else:\n df = pd.DataFrame(table_data['rows'])\n\n # Add metadata as DataFrame attributes\n df.attrs['caption'] = table_data.get('caption', '')\n df.attrs['metadata'] = table_data.get('metadata', {})\n\n dataframes.append(df)\n\n return dataframes\n
\n
Filter Tables by Criteria
\n
asyncdefextract_large_tables(url):\n asyncwith AsyncWebCrawler() as crawler:\n # Configure minimum size requirements\n strategy = DefaultTableExtraction(\n min_rows=10,\n min_cols=3,\n table_score_threshold=6\n )\n\n config = CrawlerRunConfig(\n table_extraction=strategy\n )\n\n result = await crawler.arun(url, config)\n\n # Further filter results\n large_tables = [\n table for table in result.tables\n if table['metadata']['row_count'] > 10\n and table['metadata']['column_count'] > 3\n ]\n\n return large_tables\n
\n
Export Tables to Different Formats
\n
import json\nimport csv\n\nasyncdefexport_tables(url):\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(url)\n\n for i, table inenumerate(result.tables):\n # Export as JSON\n withopen(f'table_{i}.json', 'w') as f:\n json.dump(table, f, indent=2)\n\n # Export as CSV\n withopen(f'table_{i}.csv', 'w', newline='') as f:\n writer = csv.writer(f)\n if table['headers']:\n writer.writerow(table['headers'])\n writer.writerows(table['rows'])\n\n # Export as Markdown\n withopen(f'table_{i}.md', 'w') as f:\n # Write headers\n if table['headers']:\n f.write('| ' + ' | '.join(table['headers']) + ' |\\n')\n f.write('|' + '---|' * len(table['headers']) + '\\n')\n\n # Write rows\n for row in table['rows']:\n f.write('| ' + ' | '.join(str(cell) for cell in row) + ' |\\n')\n
\n
Creating Custom Strategies
\n
Extend TableExtractionStrategy to create custom extraction logic:
\n
Example: Financial Table Extractor
\n
from crawl4ai import TableExtractionStrategy\nfrom typing importList, Dict, Any\nimport re\n\nclassFinancialTableExtractor(TableExtractionStrategy):\n \"\"\"Extract tables containing financial data.\"\"\"\n\n def__init__(self, currency_symbols=None, require_numbers=True, **kwargs):\n super().__init__(**kwargs)\n self.currency_symbols = currency_symbols or ['$', 'âŹ', 'ÂŁ', 'ÂĽ']\n self.require_numbers = require_numbers\n self.number_pattern = re.compile(r'\\d+[,.]?\\d*')\n\n defextract_tables(self, element, **kwargs):\n tables_data = []\n\n for table in element.xpath(\".//table\"):\n # Check if table contains financial indicators\n table_text = ''.join(table.itertext())\n\n # Must contain currency symbols\n has_currency = any(sym in table_text for sym in self.currency_symbols)\n ifnot has_currency:\n continue\n\n # Must contain numbers if required\n if self.require_numbers:\n numbers = self.number_pattern.findall(table_text)\n iflen(numbers) < 3: # Arbitrary minimum\n continue\n\n # Extract the table data\n table_data = self._extract_financial_data(table)\n if table_data:\n tables_data.append(table_data)\n\n return tables_data\n\n def_extract_financial_data(self, table):\n \"\"\"Extract and clean financial data from table.\"\"\"\n headers = []\n rows = []\n\n # Extract headers\n for th in table.xpath(\".//thead//th | .//tr[1]//th\"):\n headers.append(th.text_content().strip())\n\n # Extract and clean rows\n for tr in table.xpath(\".//tbody//tr | .//tr[position()>1]\"):\n row = []\n for td in tr.xpath(\".//td\"):\n text = td.text_content().strip()\n # Clean currency formatting\n text = re.sub(r'[$âŹÂŁÂĽ,]', '', text)\n row.append(text)\n if row:\n rows.append(row)\n\n return {\n \"headers\": headers,\n \"rows\": rows,\n \"caption\": self._get_caption(table),\n \"summary\": table.get(\"summary\", \"\"),\n \"metadata\": {\n \"type\": \"financial\",\n \"row_count\": len(rows),\n \"column_count\": len(headers) orlen(rows[0]) if rows else0\n }\n }\n\n def_get_caption(self, table):\n caption = table.xpath(\".//caption/text()\")\n return caption[0].strip() if caption else\"\"\n\n# Usage\nstrategy = FinancialTableExtractor(\n currency_symbols=['$', 'EUR'],\n require_numbers=True\n)\n\nconfig = CrawlerRunConfig(\n table_extraction=strategy\n)\n
\n
Example: Specific Table Extractor
\n
classSpecificTableExtractor(TableExtractionStrategy):\n \"\"\"Extract only tables matching specific criteria.\"\"\"\n\n def__init__(self, \n required_headers=None, \n id_pattern=None,\n class_pattern=None,\n **kwargs):\n super().__init__(**kwargs)\n self.required_headers = required_headers or []\n self.id_pattern = id_pattern\n self.class_pattern = class_pattern\n\n defextract_tables(self, element, **kwargs):\n tables_data = []\n\n for table in element.xpath(\".//table\"):\n # Check ID pattern\n if self.id_pattern:\n table_id = table.get('id', '')\n ifnot re.match(self.id_pattern, table_id):\n continue\n\n # Check class pattern\n if self.class_pattern:\n table_class = table.get('class', '')\n ifnot re.match(self.class_pattern, table_class):\n continue\n\n # Extract headers to check requirements\n headers = self._extract_headers(table)\n\n # Check if required headers are present\n if self.required_headers:\n ifnotall(req in headers for req in self.required_headers):\n continue\n\n # Extract full table data\n table_data = self._extract_table_data(table, headers)\n tables_data.append(table_data)\n\n return tables_data\n
\n
Combining with Other Strategies
\n
Table extraction works seamlessly with other Crawl4AI strategies:
\n
from crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n DefaultTableExtraction,\n LLMExtractionStrategy,\n JsonCssExtractionStrategy\n)\n\nasyncdefcombined_extraction(url):\n asyncwith AsyncWebCrawler() as crawler:\n config = CrawlerRunConfig(\n # Table extraction\n table_extraction=DefaultTableExtraction(\n table_score_threshold=6,\n min_rows=2\n ),\n\n # CSS-based extraction for specific elements\n extraction_strategy=JsonCssExtractionStrategy({\n \"title\": \"h1\",\n \"summary\": \"p.summary\",\n \"date\": \"time\"\n }),\n\n # Focus on main content\n css_selector=\"main.content\"\n )\n\n result = await crawler.arun(url, config)\n\n # Access different extraction results\n tables = result.tables # Table data\n structured = json.loads(result.extracted_content) # CSS extraction\n\n return {\n \"tables\": tables,\n \"structured_data\": structured,\n \"markdown\": result.markdown\n }\n
\n
Performance Considerations
\n
Optimization Tips
\n\n
Disable when not needed: Use NoTableExtraction if tables aren't required
\n
Target specific areas: Use css_selector to limit processing scope
\n
Set minimum thresholds: Filter out small/irrelevant tables early
\n
Cache results: Use appropriate cache modes for repeated extractions
\n\n
# Optimized configuration for large pages\nconfig = CrawlerRunConfig(\n # Only process main content area\n css_selector=\"article.main-content\",\n\n # Exclude navigation and sidebars\n excluded_tags=[\"nav\", \"aside\", \"footer\"],\n\n # Higher threshold for stricter filtering\n table_extraction=DefaultTableExtraction(\n table_score_threshold=8,\n min_rows=5,\n min_cols=3\n ),\n\n # Enable caching for repeated access\n cache_mode=CacheMode.ENABLED\n)\n
\n
Migration Guide
\n
Important: Your Code Still Works!
\n
No changes required! The transition to the strategy pattern is fully backward compatible.
\n
How It Works Internally
\n
v0.7.2 and Earlier
\n
# Old way - directly passing table_score_threshold\nconfig = CrawlerRunConfig(\n table_score_threshold=7\n)\n# Internally: No strategy pattern, direct implementation\n
\n
v0.7.3+ (Current)
\n
# Old way STILL WORKS - we handle it internally\nconfig = CrawlerRunConfig(\n table_score_threshold=7\n)\n# Internally: Automatically creates DefaultTableExtraction(table_score_threshold=7)\n
\n
Taking Advantage of New Features
\n
While your old code works, you can now use the strategy pattern for more control:
\n
# Option 1: Keep using the old way (perfectly fine!)\nconfig = CrawlerRunConfig(\n table_score_threshold=7 # Still supported\n)\n\n# Option 2: Use the new strategy pattern (more flexibility)\nfrom crawl4ai import DefaultTableExtraction\n\nstrategy = DefaultTableExtraction(\n table_score_threshold=7,\n min_rows=2, # New capability!\n min_cols=2 # New capability!\n)\n\nconfig = CrawlerRunConfig(\n table_extraction=strategy\n)\n\n# Option 3: Use advanced strategies when needed\nfrom crawl4ai import LLMTableExtraction, LLMConfig\n\n# Only for complex tables that DefaultTableExtraction can't handle\n# Automatically handles large tables with smart chunking\nllm_strategy = LLMTableExtraction(\n llm_config=LLMConfig(\n provider=\"groq/llama-3.3-70b-versatile\",\n api_token=\"your_key\"\n ),\n max_tries=3,\n enable_chunking=True, # Automatically chunk large tables\n chunk_token_threshold=3000, # Chunk when exceeding 3000 tokens\n max_parallel_chunks=5 # Process up to 5 chunks in parallel\n)\n\nconfig = CrawlerRunConfig(\n table_extraction=llm_strategy # Advanced extraction with automatic chunking\n)\n
\n
Summary
\n
\n
â No breaking changes - Old code works as-is
\n
â Same defaults - DefaultTableExtraction is automatically used
\n
â Gradual adoption - Use new features when you need them
\n
â Full compatibility - result.tables structure unchanged
\n
\n
Best Practices
\n
1. Choose the Right Strategy (Cost-Conscious Approach)
\n
Decision Flow:\n
1. Do you need tables? \n â No: Use NoTableExtraction\n â Yes: Continue to #2\n\n2. Try DefaultTableExtraction first (FREE)\n â Works? Done! â \n â Fails? Continue to #3\n\n3.Is the table critical and complex?\n â No: Accept DefaultTableExtraction results\n â Yes: Continue to #4\n\n4. Use LLMTableExtraction (COSTS MONEY)\n â Small table (<50rows): Any LLM provider\n â Largetable (50+rows): Use Groq or Cerebras\n â Very large (500+rows): Reconsider - maybe chunk the page\n
\n
Strategy Selection Guide:\n- DefaultTableExtraction: Use for 99% of cases - it's free and effective\n- LLMTableExtraction: Only for complex tables with merged cells that break DefaultTableExtraction\n- NoTableExtraction: When you only need text/markdown content\n- Custom Strategy: For specialized requirements (financial, scientific, etc.)
\n
2. Validate Extracted Data
\n
defvalidate_table(table):\n \"\"\"Validate table data quality.\"\"\"\n # Check structure\n ifnot table.get('rows'):\n returnFalse\n\n # Check consistency\n if table.get('headers'):\n expected_cols = len(table['headers'])\n for row in table['rows']:\n iflen(row) != expected_cols:\n returnFalse\n\n # Check minimum content\n total_cells = sum(len(row) for row in table['rows'])\n non_empty = sum(1for row in table['rows'] \n for cell in row if cell.strip())\n\n if non_empty / total_cells < 0.5: # Less than 50% non-empty\n returnFalse\n\n returnTrue\n\n# Filter valid tables\nvalid_tables = [t for t in result.tables if validate_table(t)]\n
\n
3. Handle Edge Cases
\n
asyncdefrobust_table_extraction(url):\n \"\"\"Extract tables with error handling.\"\"\"\n asyncwith AsyncWebCrawler() as crawler:\n try:\n config = CrawlerRunConfig(\n table_extraction=DefaultTableExtraction(\n table_score_threshold=6,\n verbose=True\n )\n )\n\n result = await crawler.arun(url, config)\n\n ifnot result.success:\n print(f\"Crawl failed: {result.error}\")\n return []\n\n # Process tables safely\n processed_tables = []\n for table in result.tables:\n try:\n # Validate and process\n if validate_table(table):\n processed_tables.append(table)\n except Exception as e:\n print(f\"Error processing table: {e}\")\n continue\n\n return processed_tables\n\n except Exception as e:\n print(f\"Extraction error: {e}\")\n return []\n
\n
Troubleshooting
\n
Common Issues and Solutions
\n
\n\n
\n
Issue
\n
Cause
\n
Solution
\n
\n\n\n
\n
No tables extracted
\n
Score too high
\n
Lower table_score_threshold
\n
\n
\n
Layout tables included
\n
Score too low
\n
Increase table_score_threshold
\n
\n
\n
Missing tables
\n
CSS selector too specific
\n
Broaden or remove css_selector
\n
\n
\n
Incomplete data
\n
Complex table structure
\n
Create custom strategy
\n
\n
\n
Performance issues
\n
Processing entire page
\n
Use css_selector to limit scope
\n
\n\n
\n
Debug Logging
\n
Enable verbose logging to understand extraction decisions:
Web crawling comes in different flavors, each with its own strengths. Let's understand when to use URL seeding versus deep crawling.
\n
Deep Crawling: Real-Time Discovery
\n
Deep crawling is perfect when you need:\n- Fresh, real-time data - discovering pages as they're created\n- Dynamic exploration - following links based on content\n- Selective extraction - stopping when you find what you need
\n
# Deep crawling example: Explore a website dynamically\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai.deep_crawling import BFSDeepCrawlStrategy\n\nasyncdefdeep_crawl_example():\n # Configure a 2-level deep crawl\n config = CrawlerRunConfig(\n deep_crawl_strategy=BFSDeepCrawlStrategy(\n max_depth=2, # Crawl 2 levels deep\n include_external=False, # Stay within domain\n max_pages=50# Limit for efficiency\n ),\n verbose=True\n )\n\n asyncwith AsyncWebCrawler() as crawler:\n # Start crawling and follow links dynamically\n results = await crawler.arun(\"https://example.com\", config=config)\n\n print(f\"Discovered and crawled {len(results)} pages\")\n for result in results[:3]:\n print(f\"Found: {result.url} at depth {result.metadata.get('depth', 0)}\")\n\nasyncio.run(deep_crawl_example())\n
\n
URL Seeding: Bulk Discovery
\n
URL seeding shines when you want:\n- Comprehensive coverage - get thousands of URLs in seconds\n- Bulk processing - filter before crawling\n- Resource efficiency - know exactly what you'll crawl
\n
# URL seeding example: Analyze all documentation\nfrom crawl4ai import AsyncUrlSeeder, SeedingConfig\n\nseeder = AsyncUrlSeeder()\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n pattern=\"*/docs/*\"\n)\n\n# Get ALL documentation URLs instantly\nurls = await seeder.urls(\"example.com\", config)\n# 1000+ URLs discovered in seconds!\n
\n
The Trade-offs
\n
\n\n
\n
Aspect
\n
Deep Crawling
\n
URL Seeding
\n
\n\n\n
\n
Coverage
\n
Discovers pages dynamically
\n
Gets most existing URLs instantly
\n
\n
\n
Freshness
\n
Finds brand new pages
\n
May miss very recent pages
\n
\n
\n
Speed
\n
Slower, page by page
\n
Extremely fast bulk discovery
\n
\n
\n
Resource Usage
\n
Higher - crawls to discover
\n
Lower - discovers then crawls
\n
\n
\n
Control
\n
Can stop mid-process
\n
Pre-filters before crawling
\n
\n\n
\n
When to Use Each
\n
Choose Deep Crawling when:\n- You need the absolute latest content\n- You're searching for specific information\n- The site structure is unknown or dynamic\n- You want to stop as soon as you find what you need
\n
Choose URL Seeding when:\n- You need to analyze large portions of a site\n- You want to filter URLs before crawling\n- You're doing comparative analysis\n- You need to optimize resource usage
\n
The magic happens when you understand both approaches and choose the right tool for your task. Sometimes, you might even combine them - use URL seeding for bulk discovery, then deep crawl specific sections for the latest updates.
\n
Your First URL Seeding Adventure
\n
Let's see the magic in action. We'll discover blog posts about Python, filter for tutorials, and crawl only those pages.
\n
import asyncio\nfrom crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, SeedingConfig, CrawlerRunConfig\n\nasyncdefsmart_blog_crawler():\n # Step 1: Create our URL discoverer\n seeder = AsyncUrlSeeder()\n\n # Step 2: Configure discovery - let's find all blog posts\n config = SeedingConfig(\n source=\"sitemap+cc\", # Use the website's sitemap+cc\n pattern=\"*/courses/*\", # Only courses related posts\n extract_head=True, # Get page metadata\n max_urls=100# Limit for this example\n )\n\n # Step 3: Discover URLs from the Python blog\n print(\"đ Discovering course posts...\")\n urls = await seeder.urls(\"realpython.com\", config)\n print(f\"â Found {len(urls)} course posts\")\n\n # Step 4: Filter for Python tutorials (using metadata!)\n tutorials = [\n url for url in urls \n if url[\"status\"] == \"valid\"and \n any(keyword instr(url[\"head_data\"]).lower() \n for keyword in [\"tutorial\", \"guide\", \"how to\"])\n ]\n print(f\"đ Filtered to {len(tutorials)} tutorials\")\n\n # Step 5: Show what we found\n print(\"\\nđŻ Found these tutorials:\")\n for tutorial in tutorials[:5]: # First 5\n title = tutorial[\"head_data\"].get(\"title\", \"No title\")\n print(f\" - {title}\")\n print(f\" {tutorial['url']}\")\n\n # Step 6: Now crawl ONLY these relevant pages\n print(\"\\nđ Crawling tutorials...\")\n asyncwith AsyncWebCrawler() as crawler:\n config = CrawlerRunConfig(\n only_text=True,\n word_count_threshold=300, # Only substantial articles\n stream=True\n )\n\n # Extract URLs and crawl them\n tutorial_urls = [t[\"url\"] for t in tutorials[:10]]\n results = await crawler.arun_many(tutorial_urls, config=config)\n\n successful = 0\n asyncfor result in results:\n if result.success:\n successful += 1\n print(f\"â Crawled: {result.url[:60]}...\")\n\n print(f\"\\n⨠Successfully crawled {successful} tutorials!\")\n\n# Run it!\nasyncio.run(smart_blog_crawler())\n
\n
What just happened?
\n\n
We discovered all blog URLs from the sitemap+cc
\n
We filtered using metadata (no crawling needed!)
\n
We crawled only the relevant tutorials
\n
We saved tons of time and bandwidth
\n\n
This is the power of URL seeding - you see everything before you crawl anything.
\n
Understanding the URL Seeder
\n
Now that you've seen the magic, let's understand how it works.
The seeder can discover URLs from two powerful sources:
\n
1. Sitemaps (Fastest)
\n
# Discover from sitemap\nconfig = SeedingConfig(source=\"sitemap\")\nurls = await seeder.urls(\"example.com\", config)\n
\n
Sitemaps are XML files that websites create specifically to list all their URLs. It's like getting a menu at a restaurant - everything is listed upfront.
\n
Sitemap Index Support: For large websites like TechCrunch that use sitemap indexes (a sitemap of sitemaps), the seeder automatically detects and processes all sub-sitemaps in parallel:
\n
<!-- Example sitemap index -->\n<sitemapindex>\n <sitemap>\n <loc>https://techcrunch.com/sitemap-1.xml</loc>\n </sitemap>\n <sitemap>\n <loc>https://techcrunch.com/sitemap-2.xml</loc>\n </sitemap>\n <!-- ... more sitemaps ... -->\n</sitemapindex>\n
\n
The seeder handles this transparently - you'll get all URLs from all sub-sitemaps automatically!
\n
2. Common Crawl (Most Comprehensive)
\n
# Discover from Common Crawl\nconfig = SeedingConfig(source=\"cc\")\nurls = await seeder.urls(\"example.com\", config)\n
\n
Common Crawl is a massive public dataset that regularly crawls the entire web. It's like having access to a pre-built index of the internet.
\n
3. Both Sources (Maximum Coverage)
\n
# Use both sources\nconfig = SeedingConfig(source=\"sitemap+cc\")\nurls = await seeder.urls(\"example.com\", config)\n
\n
Configuration Magic: SeedingConfig
\n
The SeedingConfig object is your control panel. Here's everything you can configure:
\n
\n\n
\n
Parameter
\n
Type
\n
Default
\n
Description
\n
\n\n\n
\n
source
\n
str
\n
\"sitemap+cc\"
\n
URL source: \"cc\" (Common Crawl), \"sitemap\", or \"sitemap+cc\"
\n
\n
\n
pattern
\n
str
\n
\"*\"
\n
URL pattern filter (e.g., \"/blog/\", \"*.html\")
\n
\n
\n
extract_head
\n
bool
\n
False
\n
Extract metadata from page <head>
\n
\n
\n
live_check
\n
bool
\n
False
\n
Verify URLs are accessible
\n
\n
\n
max_urls
\n
int
\n
-1
\n
Maximum URLs to return (-1 = unlimited)
\n
\n
\n
concurrency
\n
int
\n
10
\n
Parallel workers for fetching
\n
\n
\n
hits_per_sec
\n
int
\n
5
\n
Rate limit for requests
\n
\n
\n
force
\n
bool
\n
False
\n
Bypass cache, fetch fresh data
\n
\n
\n
verbose
\n
bool
\n
False
\n
Show detailed progress
\n
\n
\n
query
\n
str
\n
None
\n
Search query for BM25 scoring
\n
\n
\n
scoring_method
\n
str
\n
None
\n
Scoring method (currently \"bm25\")
\n
\n
\n
score_threshold
\n
float
\n
None
\n
Minimum score to include URL
\n
\n
\n
filter_nonsense_urls
\n
bool
\n
True
\n
Filter out utility URLs (robots.txt, etc.)
\n
\n
\n
cache_ttl_hours
\n
int
\n
24
\n
Hours before sitemap cache expires (0 = no TTL)
\n
\n
\n
validate_sitemap_lastmod
\n
bool
\n
True
\n
Check sitemap's lastmod and refetch if newer
\n
\n\n
\n
Pattern Matching Examples
\n
# Match all blog posts\nconfig = SeedingConfig(pattern=\"*/blog/*\")\n\n# Match only HTML files\nconfig = SeedingConfig(pattern=\"*.html\")\n\n# Match product pages\nconfig = SeedingConfig(pattern=\"*/product/*\")\n\n# Match everything except admin pages\nconfig = SeedingConfig(pattern=\"*\")\n# Then filter: urls = [u for u in urls if \"/admin/\" not in u[\"url\"]]\n
\n
URL Validation: Live Checking
\n
Sometimes you need to know if URLs are actually accessible. That's where live checking comes in:
\n
config = SeedingConfig(\n source=\"sitemap\",\n live_check=True, # Verify each URL is accessible\n concurrency=20# Check 20 URLs in parallel\n)\nasyncwith AsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"example.com\", config)\n\n# Now you can filter by status\nlive_urls = [u for u in urls if u[\"status\"] == \"valid\"]\ndead_urls = [u for u in urls if u[\"status\"] == \"not_valid\"]\n\nprint(f\"Live URLs: {len(live_urls)}\")\nprint(f\"Dead URLs: {len(dead_urls)}\")\n
\n
When to use live checking:\n- Before a large crawling operation\n- When working with older sitemaps\n- When data freshness is critical
\n
When to skip it:\n- Quick explorations\n- When you trust the source\n- When speed is more important than accuracy
\n
The Power of Metadata: Head Extraction
\n
This is where URL seeding gets really powerful. Instead of crawling entire pages, you can extract just the metadata:
\n
config = SeedingConfig(\n extract_head=True# Extract metadata from <head> section\n)\nasyncwith AsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"example.com\", config)\n\n# Now each URL has rich metadata\nfor url in urls[:3]:\n print(f\"\\nURL: {url['url']}\")\n print(f\"Title: {url['head_data'].get('title')}\")\n\n meta = url['head_data'].get('meta', {})\n print(f\"Description: {meta.get('description')}\")\n print(f\"Keywords: {meta.get('keywords')}\")\n\n # Even Open Graph data!\n print(f\"OG Image: {meta.get('og:image')}\")\n
\n
What Can We Extract?
\n
The head extraction gives you a treasure trove of information:
This metadata is gold for filtering! You can find exactly what you need without crawling a single page.
\n
Smart URL-Based Filtering (No Head Extraction)
\n
When extract_head=False but you still provide a query, the seeder uses intelligent URL-based scoring:
\n
# Fast filtering based on URL structure alone\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=False, # Don't fetch page metadata\n query=\"python tutorial async\",\n scoring_method=\"bm25\",\n score_threshold=0.3\n)\nasyncwith AsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"example.com\", config)\n\n# URLs are scored based on:\n# 1. Domain parts matching (e.g., 'python' in python.example.com)\n# 2. Path segments (e.g., '/tutorials/python-async/')\n# 3. Query parameters (e.g., '?topic=python')\n# 4. Fuzzy matching using character n-grams\n\n# Example URL scoring:\n# https://example.com/tutorials/python/async-guide.html - High score\n# https://example.com/blog/javascript-tips.html - Low score\n
\n
This approach is much faster than head extraction while still providing intelligent filtering!
\n
Understanding Results
\n
Each URL in the results has this structure:
\n
{\n \"url\": \"https://example.com/blog/python-tips.html\",\n \"status\": \"valid\", # \"valid\", \"not_valid\", or \"unknown\"\n \"head_data\": { # Only if extract_head=True\n \"title\": \"Page Title\",\n \"meta\": {...},\n \"link\": {...},\n \"jsonld\": [...]\n },\n \"relevance_score\": 0.85 # Only if using BM25 scoring\n}\n
\n
Let's see a real example:
\n
config = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n live_check=True\n)\nasyncwith AsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"blog.example.com\", config)\n\n# Analyze the results\nfor url in urls[:5]:\n print(f\"\\n{'='*60}\")\n print(f\"URL: {url['url']}\")\n print(f\"Status: {url['status']}\")\n\n if url['head_data']:\n data = url['head_data']\n print(f\"Title: {data.get('title', 'No title')}\")\n\n # Check content type\n meta = data.get('meta', {})\n content_type = meta.get('og:type', 'unknown')\n print(f\"Content Type: {content_type}\")\n\n # Publication date\n pub_date = None\n for jsonld in data.get('jsonld', []):\n ifisinstance(jsonld, dict):\n pub_date = jsonld.get('datePublished')\n if pub_date:\n break\n\n if pub_date:\n print(f\"Published: {pub_date}\")\n\n # Word count (if available)\n word_count = meta.get('word_count')\n if word_count:\n print(f\"Word Count: {word_count}\")\n
\n
Smart Filtering with BM25 Scoring
\n
Now for the really cool part - intelligent filtering based on relevance!
\n
Introduction to Relevance Scoring
\n
BM25 is a ranking algorithm that scores how relevant a document is to a search query. With URL seeding, we can score URLs based on their metadata before crawling them.
\n
Think of it like this:\n- Traditional way: Read every book in the library to find ones about Python\n- Smart way: Check the titles and descriptions, score them, read only the most relevant
\n
Query-Based Discovery
\n
Here's how to use BM25 scoring:
\n
config = SeedingConfig(\n source=\"sitemap\",\n extract_head=True, # Required for scoring\n query=\"python async tutorial\", # What we're looking for\n scoring_method=\"bm25\", # Use BM25 algorithm\n score_threshold=0.3# Minimum relevance score\n)\nasyncwith AsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"realpython.com\", config)\n\n# Results are automatically sorted by relevance!\nfor url in urls[:5]:\n print(f\"Score: {url['relevance_score']:.2f} - {url['url']}\")\n print(f\" Title: {url['head_data']['title']}\")\n
\n
Real Examples
\n
Finding Documentation Pages
\n
# Find API documentation\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n query=\"API reference documentation endpoints\",\n scoring_method=\"bm25\",\n score_threshold=0.5,\n max_urls=20\n)\nasyncwithAsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"docs.example.com\", config)\n\n# The highest scoring URLs will be API docs!\n
\n
Discovering Product Pages
\n
# Find specific products\nconfig = SeedingConfig(\n source=\"sitemap+cc\", # Use both sources\n extract_head=True,\n query=\"wireless headphones noise canceling\",\n scoring_method=\"bm25\",\n score_threshold=0.4,\n pattern=\"*/product/*\"# Combine with pattern matching\n)\nasyncwithAsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"shop.example.com\", config)\n\n# Filter further by price (from metadata)\naffordable = [\n u for u in urls \n iffloat(u['head_data'].get('meta', {}).get('product:price', '0')) < 200\n]\n
\n
Filtering News Articles
\n
# Find recent news about AI\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n query=\"artificial intelligence machine learning breakthrough\",\n scoring_method=\"bm25\",\n score_threshold=0.35\n)\nasyncwithAsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"technews.com\", config)\n\n# Filter by date\nfrom datetime import datetime, timedelta\n\nrecent = []\ncutoff = datetime.now() - timedelta(days=7)\n\nfor url in urls:\n # Check JSON-LD for publication date\n for jsonld in url['head_data'].get('jsonld', []):\n if'datePublished'in jsonld:\n pub_date = datetime.fromisoformat(jsonld['datePublished'].replace('Z', '+00:00'))\n if pub_date > cutoff:\n recent.append(url)\n break\n
\n
Complex Query Patterns
\n
# Multi-concept queries\nqueries = [\n \"python async await concurrency tutorial\",\n \"data science pandas numpy visualization\",\n \"web scraping beautifulsoup selenium automation\",\n \"machine learning tensorflow keras deep learning\"\n]\n\nall_tutorials = []\n\nfor query in queries:\n config = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n query=query,\n scoring_method=\"bm25\",\n score_threshold=0.4,\n max_urls=10# Top 10 per topic\n )\n asyncwith AsyncUrlSeeder() as seeder:\n urls = await seeder.urls(\"learning-platform.com\", config)\n all_tutorials.extend(urls)\n\n# Remove duplicates while preserving order\nseen = set()\nunique_tutorials = []\nfor url in all_tutorials:\n if url['url'] notin seen:\n seen.add(url['url'])\n unique_tutorials.append(url)\n\nprint(f\"Found {len(unique_tutorials)} unique tutorials across all topics\")\n
\n
Scaling Up: Multiple Domains
\n
When you need to discover URLs across multiple websites, URL seeding really shines.
\n
The many_urls Method
\n
# Discover URLs from multiple domains in parallel\ndomains = [\"site1.com\", \"site2.com\", \"site3.com\"]\n\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n query=\"python tutorial\",\n scoring_method=\"bm25\",\n score_threshold=0.3\n)\n\n# Returns a dictionary: {domain: [urls]}\nasyncwith AsyncUrlSeeder() as seeder:\n results = await seeder.many_urls(domains, config)\n\n# Process results\nfor domain, urls in results.items():\n print(f\"\\n{domain}: Found {len(urls)} relevant URLs\")\n if urls:\n top = urls[0] # Highest scoring\n print(f\" Top result: {top['url']}\")\n print(f\" Score: {top['relevance_score']:.2f}\")\n
\n
Cross-Domain Examples
\n
Competitor Analysis
\n
# Analyze content strategies across competitors\ncompetitors = [\n \"competitor1.com\",\n \"competitor2.com\", \n \"competitor3.com\"\n]\n\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n pattern=\"*/blog/*\",\n max_urls=100\n)\nasyncwith AsyncUrlSeeder() as seeder:\n results = await seeder.many_urls(competitors, config)\n\n# Analyze content types\nfor domain, urls in results.items():\n content_types = {}\n\n for url in urls:\n # Extract content type from metadata\n og_type = url['head_data'].get('meta', {}).get('og:type', 'unknown')\n content_types[og_type] = content_types.get(og_type, 0) + 1\n\n print(f\"\\n{domain} content distribution:\")\n for ctype, count insorted(content_types.items(), key=lambda x: x[1], reverse=True):\n print(f\" {ctype}: {count}\")\n
\n
Industry Research
\n
# Research Python tutorials across educational sites\neducational_sites = [\n \"realpython.com\",\n \"pythontutorial.net\",\n \"learnpython.org\",\n \"python.org\"\n]\n\nconfig = SeedingConfig(\n source=\"sitemap\",\n extract_head=True,\n query=\"beginner python tutorial basics\",\n scoring_method=\"bm25\",\n score_threshold=0.3,\n max_urls=20# Per site\n)\nasyncwith AsyncUrlSeeder() as seeder:\n results = await seeder.many_urls(educational_sites, config)\n\n# Find the best beginner tutorials\nall_tutorials = []\nfor domain, urls in results.items():\n for url in urls:\n url['domain'] = domain # Add domain info\n all_tutorials.append(url)\n\n# Sort by relevance across all domains\nall_tutorials.sort(key=lambda x: x['relevance_score'], reverse=True)\n\nprint(\"Top 10 Python tutorials for beginners across all sites:\")\nfor i, tutorial inenumerate(all_tutorials[:10], 1):\n print(f\"{i}. [{tutorial['relevance_score']:.2f}] {tutorial['head_data']['title']}\")\n print(f\" {tutorial['url']}\")\n print(f\" From: {tutorial['domain']}\")\n
\n
Multi-Site Monitoring
\n
# Monitor news about your company across multiple sources\nnews_sites = [\n \"techcrunch.com\",\n \"theverge.com\",\n \"wired.com\",\n \"arstechnica.com\"\n]\n\ncompany_name = \"YourCompany\"\n\nconfig = SeedingConfig(\n source=\"cc\", # Common Crawl for recent content\n extract_head=True,\n query=f\"{company_name} announcement news\",\n scoring_method=\"bm25\",\n score_threshold=0.5, # High threshold for relevance\n max_urls=10\n)\nasyncwith AsyncUrlSeeder() as seeder:\n results = await seeder.many_urls(news_sites, config)\n\n# Collect all mentions\nmentions = []\nfor domain, urls in results.items():\n mentions.extend(urls)\n\nif mentions:\n print(f\"Found {len(mentions)} mentions of {company_name}:\")\n for mention in mentions:\n print(f\"\\n- {mention['head_data']['title']}\")\n print(f\" {mention['url']}\")\n print(f\" Score: {mention['relevance_score']:.2f}\")\nelse:\n print(f\"No recent mentions of {company_name} found\")\n
\n
Advanced Integration Patterns
\n
Let's put everything together in a real-world example.
\n
Building a Research Assistant
\n
Here's a complete example that discovers, scores, filters, and crawls intelligently:
\n
import asyncio\nfrom datetime import datetime\nfrom crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, SeedingConfig, CrawlerRunConfig\n\nclassResearchAssistant:\n def__init__(self):\n self.seeder = None\n\n asyncdef__aenter__(self):\n self.seeder = AsyncUrlSeeder()\n await self.seeder.__aenter__()\n return self\n\n asyncdef__aexit__(self, exc_type, exc_val, exc_tb):\n if self.seeder:\n await self.seeder.__aexit__(exc_type, exc_val, exc_tb)\n\n asyncdefresearch_topic(self, topic, domains, max_articles=20):\n \"\"\"Research a topic across multiple domains.\"\"\"\n\n print(f\"đŹ Researching '{topic}' across {len(domains)} domains...\")\n\n # Step 1: Discover relevant URLs\n config = SeedingConfig(\n source=\"sitemap+cc\", # Maximum coverage\n extract_head=True, # Get metadata\n query=topic, # Research topic\n scoring_method=\"bm25\", # Smart scoring\n score_threshold=0.4, # Quality threshold\n max_urls=10, # Per domain\n concurrency=20, # Fast discovery\n verbose=True\n )\n\n # Discover across all domains\n discoveries = await self.seeder.many_urls(domains, config)\n\n # Step 2: Collect and rank all articles\n all_articles = []\n for domain, urls in discoveries.items():\n for url in urls:\n url['domain'] = domain\n all_articles.append(url)\n\n # Sort by relevance\n all_articles.sort(key=lambda x: x['relevance_score'], reverse=True)\n\n # Take top articles\n top_articles = all_articles[:max_articles]\n\n print(f\"\\nđ Found {len(all_articles)} relevant articles\")\n print(f\"đ Selected top {len(top_articles)} for deep analysis\")\n\n # Step 3: Show what we're about to crawl\n print(\"\\nđŻ Articles to analyze:\")\n for i, article inenumerate(top_articles[:5], 1):\n print(f\"\\n{i}. {article['head_data']['title']}\")\n print(f\" Score: {article['relevance_score']:.2f}\")\n print(f\" Source: {article['domain']}\")\n print(f\" URL: {article['url'][:60]}...\")\n\n # Step 4: Crawl the selected articles\n print(f\"\\nđ Deep crawling {len(top_articles)} articles...\")\n\n asyncwith AsyncWebCrawler() as crawler:\n config = CrawlerRunConfig(\n only_text=True,\n word_count_threshold=200, # Substantial content only\n stream=True\n )\n\n # Extract URLs and crawl all articles\n article_urls = [article['url'] for article in top_articles]\n results = []\n crawl_results = await crawler.arun_many(article_urls, config=config)\n asyncfor result in crawl_results:\n if result.success:\n results.append({\n 'url': result.url,\n 'title': result.metadata.get('title', 'No title'),\n 'content': result.markdown.raw_markdown,\n 'domain': next(a['domain'] for a in top_articles if a['url'] == result.url),\n 'score': next(a['relevance_score'] for a in top_articles if a['url'] == result.url)\n })\n print(f\"â Crawled: {result.url[:60]}...\")\n\n # Step 5: Analyze and summarize\n print(f\"\\nđ Analysis complete! Crawled {len(results)} articles\")\n\n return self.create_research_summary(topic, results)\n\n defcreate_research_summary(self, topic, articles):\n \"\"\"Create a research summary from crawled articles.\"\"\"\n\n summary = {\n 'topic': topic,\n 'timestamp': datetime.now().isoformat(),\n 'total_articles': len(articles),\n 'sources': {}\n }\n\n # Group by domain\n for article in articles:\n domain = article['domain']\n if domain notin summary['sources']:\n summary['sources'][domain] = []\n\n summary['sources'][domain].append({\n 'title': article['title'],\n 'url': article['url'],\n 'score': article['score'],\n 'excerpt': article['content'][:500] + '...'iflen(article['content']) > 500else article['content']\n })\n\n return summary\n\n# Use the research assistant\nasyncdefmain():\n asyncwith ResearchAssistant() as assistant:\n # Research Python async programming across multiple sources\n topic = \"python asyncio best practices performance optimization\"\n domains = [\n \"realpython.com\",\n \"python.org\",\n \"stackoverflow.com\",\n \"medium.com\"\n ]\n\n summary = await assistant.research_topic(topic, domains, max_articles=15)\n\n # Display results\n print(\"\\n\" + \"=\"*60)\n print(\"RESEARCH SUMMARY\")\n print(\"=\"*60)\n print(f\"Topic: {summary['topic']}\")\n print(f\"Date: {summary['timestamp']}\")\n print(f\"Total Articles Analyzed: {summary['total_articles']}\")\n\n print(\"\\nKey Findings by Source:\")\n for domain, articles in summary['sources'].items():\n print(f\"\\nđ {domain} ({len(articles)} articles)\")\n for article in articles[:2]: # Top 2 per domain\n print(f\"\\n Title: {article['title']}\")\n print(f\" Relevance: {article['score']:.2f}\")\n print(f\" Preview: {article['excerpt'][:200]}...\")\n\nasyncio.run(main())\n
\n
Performance Optimization Tips
\n\n
\n
Use caching wisely\n
# First run - populate cache\nconfig = SeedingConfig(source=\"sitemap\", extract_head=True, force=True)\nurls = await seeder.urls(\"example.com\", config)\n\n# Subsequent runs - use cache (much faster)\nconfig = SeedingConfig(source=\"sitemap\", extract_head=True, force=False)\nurls = await seeder.urls(\"example.com\", config)\n
\n
\n
\n
Optimize concurrency\n
# For many small requests (like HEAD checks)\nconfig = SeedingConfig(concurrency=50, hits_per_sec=20)\n\n# For fewer large requests (like full head extraction)\nconfig = SeedingConfig(concurrency=10, hits_per_sec=5)\n
\n
\n
\n
Stream large result sets\n
# When crawling many URLs\nasyncwith AsyncWebCrawler() as crawler:\n # Assuming urls is a list of URL strings\n crawl_results = await crawler.arun_many(urls, config=config)\n\n # Process as they arrive\n asyncfor result in crawl_results:\n process_immediately(result) # Don't wait for all\n
\n
\n
\n
Memory protection for large domains
\n
\n\n
The seeder uses bounded queues to prevent memory issues when processing domains with millions of URLs:
\n
# Safe for domains with 1M+ URLs\nconfig = SeedingConfig(\n source=\"cc+sitemap\",\n concurrency=50, # Queue size adapts to concurrency\n max_urls=100000 # Process in batches if needed\n)\n\n# The seeder automatically manages memory by:\n# - Using bounded queues (prevents RAM spikes)\n# - Applying backpressure when queue is full\n# - Processing URLs as they're discovered\n
\n
Best Practices & Tips
\n
Cache Management
\n
The seeder automatically caches results to speed up repeated operations:
\n
\n
Common Crawl cache: ~/.crawl4ai/seeder_cache/[index]_[domain]_[hash].jsonl
HEAD data cache: ~/.cache/url_seeder/head/[hash].json
\n
\n
Smart TTL Cache for Sitemaps
\n
Sitemap caches now include intelligent validation:
\n
# Default: 24-hour TTL with lastmod validation\nconfig = SeedingConfig(\n source=\"sitemap\",\n cache_ttl_hours=24, # Cache expires after 24 hours\n validate_sitemap_lastmod=True# Also check if sitemap was updated\n)\n\n# Aggressive caching (1 week, no lastmod check)\nconfig = SeedingConfig(\n source=\"sitemap\",\n cache_ttl_hours=168, # 7 days\n validate_sitemap_lastmod=False# Trust TTL only\n)\n\n# Always validate (no TTL, only lastmod)\nconfig = SeedingConfig(\n source=\"sitemap\",\n cache_ttl_hours=0, # Disable TTL\n validate_sitemap_lastmod=True# Refetch if sitemap has newer lastmod\n)\n\n# Always fresh (bypass cache completely)\nconfig = SeedingConfig(\n source=\"sitemap\",\n force=True# Ignore all caching\n)\n
\n
Cache validation priority:\n1. force=True â Always refetch\n2. Cache doesn't exist â Fetch fresh\n3. validate_sitemap_lastmod=True and sitemap has newer <lastmod> â Refetch\n4. cache_ttl_hours > 0 and cache is older than TTL â Refetch\n5. Cache corrupted â Refetch (automatic recovery)\n6. Otherwise â Use cache
\n
Pattern Matching Strategies
\n
# Be specific when possible\ngood_pattern = \"*/blog/2024/*.html\"# Specific\nbad_pattern = \"*\"# Too broad\n\n# Combine patterns with metadata filtering\nconfig = SeedingConfig(\n pattern=\"*/articles/*\",\n extract_head=True\n)\nurls = await seeder.urls(\"news.com\", config)\n\n# Further filter by publish date, author, category, etc.\nrecent = [u for u in urls if is_recent(u['head_data'])]\n
\n
Rate Limiting Considerations
\n
# Be respectful of servers\nconfig = SeedingConfig(\n hits_per_sec=10, # Max 10 requests per second\n concurrency=20 # But use 20 workers\n)\n\n# For your own servers\nconfig = SeedingConfig(\n hits_per_sec=None, # No limit\n concurrency=100 # Go fast\n)\n
Check validate_sitemap_lastmod=True, or use force=True
\n
\n
\n
Incomplete URL list
\n
Delete cache file and refetch, or use force=True
\n
\n\n
\n
Performance Benchmarks
\n
Typical performance on a standard connection:
\n
\n
Sitemap discovery: 100-1,000 URLs/second
\n
Common Crawl discovery: 50-500 URLs/second
\n
HEAD checking: 10-50 URLs/second
\n
Head extraction: 5-20 URLs/second
\n
BM25 scoring: 10,000+ URLs/second
\n
\n
Conclusion
\n
URL seeding transforms web crawling from a blind expedition into a surgical strike. By discovering and analyzing URLs before crawling, you can:
\n
\n
Save hours of crawling time
\n
Reduce bandwidth usage by 90%+
\n
Find exactly what you need
\n
Scale across multiple domains effortlessly
\n
\n
Whether you're building a research tool, monitoring competitors, or creating a content aggregator, URL seeding gives you the intelligence to crawl smarter, not harder.
\n
Smart URL Filtering
\n
The seeder automatically filters out nonsense URLs that aren't useful for content crawling:
\n
# Enabled by default\nconfig = SeedingConfig(\n source=\"sitemap\",\n filter_nonsense_urls=True# Default: True\n)\n\n# URLs that get filtered:\n# - robots.txt, sitemap.xml, ads.txt\n# - API endpoints (/api/, /v1/, .json)\n# - Media files (.jpg, .mp4, .pdf)\n# - Archives (.zip, .tar.gz)\n# - Source code (.js, .css)\n# - Admin/login pages\n# - And many more...\n
\n
To disable filtering (not recommended):
\n
config = SeedingConfig(\n source=\"sitemap\",\n filter_nonsense_urls=False # Include ALL URLs\n)\n
\n
Key Features Summary
\n\n
Parallel Sitemap Index Processing: Automatically detects and processes sitemap indexes in parallel
\n
Memory Protection: Bounded queues prevent RAM issues with large domains (1M+ URLs)
\n
Context Manager Support: Automatic cleanup with async with statement
\n
URL-Based Scoring: Smart filtering even without head extraction
Smart TTL Cache: Sitemap caches with TTL expiry and lastmod validation
\n
Automatic Cache Recovery: Corrupted or incomplete caches are automatically refreshed
\n\n
Now go forth and seed intelligently!
\n
Need More Coverage?
\n
If you need to discover URLs across an entire domain â including subdomains, hidden services, and pages not listed in any sitemap â check out Domain Mapping. It combines 8 discovery sources (including Certificate Transparency, Wayback Machine, path probing, and soft-404 detection) to find everything under a domain.
Chunking strategies are critical for dividing large texts into manageable parts, enabling effective content processing and extraction. These strategies are foundational in cosine similarity-based extraction techniques, which allow users to retrieve only the most relevant chunks of content for a given query. Additionally, they facilitate direct integration into RAG (Retrieval-Augmented Generation) systems for structured and scalable workflows.
\n
Why Use Chunking?
\n
1.âCosine Similarity and Query Relevance: Prepares chunks for semantic similarity analysis.\n2.âRAG System Integration: Seamlessly processes and stores chunks for retrieval.\n3.âStructured Processing: Allows for diverse segmentation methods, such as sentence-based, topic-based, or windowed approaches.
\n
Methods of Chunking
\n
1. Regex-Based Chunking
\n
Splits text based on regular expression patterns, useful for coarse segmentation.
\n
Code Example:\n
classRegexChunking:\n def__init__(self, patterns=None):\n self.patterns = patterns or [r'\\n\\n'] # Default pattern for paragraphs\n\n defchunk(self, text):\n paragraphs = [text]\n for pattern in self.patterns:\n paragraphs = [seg for p in paragraphs for seg in re.split(pattern, p)]\n return paragraphs\n\n# Example Usage\ntext = \"\"\"This is the first paragraph.\n\nThis is the second paragraph.\"\"\"\nchunker = RegexChunking()\nprint(chunker.chunk(text))\n
\n
2. Sentence-Based Chunking
\n
Divides text into sentences using NLP tools, ideal for extracting meaningful statements.
\n
Code Example:\n
from nltk.tokenize import sent_tokenize\n\nclassNlpSentenceChunking:\n defchunk(self, text):\n sentences = sent_tokenize(text)\n return [sentence.strip() for sentence in sentences]\n\n# Example Usage\ntext = \"This is sentence one. This is sentence two.\"\nchunker = NlpSentenceChunking()\nprint(chunker.chunk(text))\n
\n
3. Topic-Based Segmentation
\n
Uses algorithms like TextTiling to create topic-coherent chunks.
\n
Code Example:\n
from nltk.tokenize import TextTilingTokenizer\n\nclassTopicSegmentationChunking:\n def__init__(self):\n self.tokenizer = TextTilingTokenizer()\n\n defchunk(self, text):\n return self.tokenizer.tokenize(text)\n\n# Example Usage\ntext = \"\"\"This is an introduction.\nThis is a detailed discussion on the topic.\"\"\"\nchunker = TopicSegmentationChunking()\nprint(chunker.chunk(text))\n
\n
4. Fixed-Length Word Chunking
\n
Segments text into chunks of a fixed word count.
\n
Code Example:\n
classFixedLengthWordChunking:\n def__init__(self, chunk_size=100):\n self.chunk_size = chunk_size\n\n defchunk(self, text):\n words = text.split()\n return [' '.join(words[i:i + self.chunk_size]) for i inrange(0, len(words), self.chunk_size)]\n\n# Example Usage\ntext = \"This is a long text with many words to be chunked into fixed sizes.\"\nchunker = FixedLengthWordChunking(chunk_size=5)\nprint(chunker.chunk(text))\n
\n
5. Sliding Window Chunking
\n
Generates overlapping chunks for better contextual coherence.
\n
Code Example:\n
classSlidingWindowChunking:\n def__init__(self, window_size=100, step=50):\n self.window_size = window_size\n self.step = step\n\n defchunk(self, text):\n words = text.split()\n chunks = []\n for i inrange(0, len(words) - self.window_size + 1, self.step):\n chunks.append(' '.join(words[i:i + self.window_size]))\n return chunks\n\n# Example Usage\ntext = \"This is a long text to demonstrate sliding window chunking.\"\nchunker = SlidingWindowChunking(window_size=5, step=2)\nprint(chunker.chunk(text))\n
\n
Combining Chunking with Cosine Similarity
\n
To enhance the relevance of extracted content, chunking strategies can be paired with cosine similarity techniques. Hereâs an example workflow:
\n
Code Example:\n
from sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity\n\nclassCosineSimilarityExtractor:\n def__init__(self, query):\n self.query = query\n self.vectorizer = TfidfVectorizer()\n\n deffind_relevant_chunks(self, chunks):\n vectors = self.vectorizer.fit_transform([self.query] + chunks)\n similarities = cosine_similarity(vectors[0:1], vectors[1:]).flatten()\n return [(chunks[i], similarities[i]) for i inrange(len(chunks))]\n\n# Example Workflow\ntext = \"\"\"This is a sample document. It has multiple sentences. \nWe are testing chunking and similarity.\"\"\"\n\nchunker = SlidingWindowChunking(window_size=5, step=3)\nchunks = chunker.chunk(text)\nquery = \"testing chunking\"\nextractor = CosineSimilarityExtractor(query)\nrelevant_chunks = extractor.find_relevant_chunks(chunks)\n\nprint(relevant_chunks)\n
The Cosine Strategy in Crawl4AI uses similarity-based clustering to identify and extract relevant content sections from web pages. This strategy is particularly useful when you need to find and extract content based on semantic similarity rather than structural patterns.
\n
How It Works
\n
The Cosine Strategy:\n1. Breaks down page content into meaningful chunks\n2. Converts text into vector representations\n3. Calculates similarity between chunks\n4. Clusters similar content together\n5. Ranks and filters content based on relevance
\n
Basic Usage
\n
from crawl4ai import CosineStrategy\n\nstrategy = CosineStrategy(\n semantic_filter=\"product reviews\", # Target content type\n word_count_threshold=10, # Minimum words per cluster\n sim_threshold=0.3# Similarity threshold\n)\n\nasyncwithAsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/reviews\",\n extraction_strategy=strategy\n )\n\n content = result.extracted_content\n
\n
Configuration Options
\n
Core Parameters
\n
CosineStrategy(\n # Content Filtering\n semantic_filter: str = None, # Keywords/topic for content filtering\n word_count_threshold: int = 10, # Minimum words per cluster\n sim_threshold: float = 0.3, # Similarity threshold (0.0 to 1.0)\n\n # Clustering Parameters\n max_dist: float = 0.2, # Maximum distance for clustering\n linkage_method: str = 'ward', # Clustering linkage method\n top_k: int = 3, # Number of top categories to extract\n\n # Model Configuration\n model_name: str = 'sentence-transformers/all-MiniLM-L6-v2', # Embedding model\n\n verbose: bool = False# Enable logging\n)\n
\n
Parameter Details
\n
1.âsemantic_filter\n - Sets the target topic or content type\n - Use keywords relevant to your desired content\n - Example: \"technical specifications\", \"user reviews\", \"pricing information\"
\n
2.âsim_threshold\n - Controls how similar content must be to be grouped together\n - Higher values (e.g., 0.8) mean stricter matching\n - Lower values (e.g., 0.3) allow more variation\n
3.âword_count_threshold\n - Filters out short content blocks\n - Helps eliminate noise and irrelevant content\n
# Only consider substantial paragraphs\nstrategy = CosineStrategy(word_count_threshold=50)\n
\n
4.âtop_k\n - Number of top content clusters to return\n - Higher values return more diverse content\n
# Get top 5 most relevant content clusters\nstrategy = CosineStrategy(top_k=5)\n
\n
Use Cases
\n
1. Article Content Extraction
\n
strategy = CosineStrategy(\n semantic_filter=\"main article content\",\n word_count_threshold=100, # Longer blocks for articles\n top_k=1 # Usually want single main content\n)\n\nresult = await crawler.arun(\n url=\"https://example.com/blog/post\",\n extraction_strategy=strategy\n)\n
\n
2. Product Review Analysis
\n
strategy = CosineStrategy(\n semantic_filter=\"customer reviews and ratings\",\n word_count_threshold=20, # Reviews can be shorter\n top_k=10, # Get multiple reviews\n sim_threshold=0.4 # Allow variety in review content\n)\n
\n
3. Technical Documentation
\n
strategy = CosineStrategy(\n semantic_filter=\"technical specifications documentation\",\n word_count_threshold=30,\n sim_threshold=0.6, # Stricter matching for technical content\n max_dist=0.3 # Allow related technical sections\n)\n
strategy = CosineStrategy(\n semantic_filter=\"pricing plans features\",\n word_count_threshold=15,\n sim_threshold=0.5,\n top_k=3\n)\n\nasyncdefextract_pricing_features(url: str):\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=url,\n extraction_strategy=strategy\n )\n\n if result.success:\n content = json.loads(result.extracted_content)\n return {\n 'pricing_features': content,\n 'clusters': len(content),\n 'similarity_scores': [item['score'] for item in content]\n }\n
\n
Best Practices
\n
1.âAdjust Thresholds Iteratively\n - Start with default values\n - Adjust based on results\n - Monitor clustering quality
\n
2.âChoose Appropriate Word Count Thresholds\n - Higher for articles (100+)\n - Lower for reviews/comments (20+)\n - Medium for product descriptions (50+)
try:\n result = await crawler.arun(\n url=\"https://example.com\",\n extraction_strategy=strategy\n )\n\n if result.success:\n content = json.loads(result.extracted_content)\n ifnot content:\n print(\"No relevant content found\")\n else:\n print(f\"Extraction failed: {result.error_message}\")\n\nexcept Exception as e:\n print(f\"Error during extraction: {str(e)}\")\n
\n
The Cosine Strategy is particularly effective when:\n- Content structure is inconsistent\n- You need semantic understanding\n- You want to find similar content blocks\n- Structure-based extraction (CSS/XPath) isn't reliable
\n
It works well with other strategies and can be used as a pre-processing step for LLM-based extraction.
In some cases, you need to extract complex or unstructured information from a webpage that a simple CSS/XPath schema cannot easily parse. Or you want AI-driven insights, classification, or summarization. For these scenarios, Crawl4AI provides an LLM-based extraction strategy that:
\n\n
Works with any large language model supported by LiteLLM (Ollama, OpenAI, Claude, and more).
\n
Automatically splits content into chunks (if desired) to handle token limits, then combines results.
\n
Lets you define a schema (like a Pydantic model) or a simpler âblockâ extraction approach.
\n\n
Important: LLM-based extraction can be slower and costlier than schema-based approaches. If your page data is highly structured, consider using JsonCssExtractionStrategy or JsonXPathExtractionStrategy first. But if you need AI to interpret or reorganize content, read on!
\n\n
1. Why Use an LLM?
\n
\n
Complex Reasoning: If the siteâs data is unstructured, scattered, or full of natural language context.
\n
Semantic Extraction: Summaries, knowledge graphs, or relational data that require comprehension.
\n
Flexible: You can pass instructions to the model to do more advanced transformations or classification.
\n
\n\n
2. Provider-Agnostic via LiteLLM
\n
You can use LLMConfig, to quickly configure multiple variations of LLMs and experiment with them to find the optimal one for your use case. You can read more about LLMConfig here.
Crawl4AI uses a âprovider stringâ (e.g., \"openai/gpt-4o\", \"ollama/llama2.0\", \"aws/titan\") to identify your LLM.âAny model that LiteLLM supports is fair game. You just provide:
\n
\n
provider: The <provider>/<model_name> identifier (e.g., \"openai/gpt-4\", \"ollama/llama2\", \"huggingface/google-flan\", etc.).
\n
api_token: If needed (for OpenAI, HuggingFace, etc.); local models or Ollama might not require it.
\n
base_url (optional): If your provider has a custom endpoint.
\n
\n
This means you arenât locked into a single LLM vendor. Switch or experiment easily.
\n\n
3. How LLM Extraction Works
\n
3.1 Flow
\n
1.âChunking (optional): The HTML or markdown is split into smaller segments if itâs very long (based on chunk_token_threshold, overlap, etc.). \n2.âPrompt Construction: For each chunk, the library forms a prompt that includes your instruction (and possibly schema or examples). \n3.âLLM Inference: Each chunk is sent to the model in parallel or sequentially (depending on your concurrency). \n4.âCombining: The results from each chunk are merged and parsed into JSON.
\n
3.2 extraction_type
\n
\n
\"schema\": The model tries to return JSON conforming to your Pydantic-based schema.
\n
\"block\": The model returns freeform text, or smaller JSON structures, which the library collects.
\n
\n
For structured data, \"schema\" is recommended. You provide schema=YourPydanticModel.model_json_schema().
\n\n
4. Key Parameters
\n
Below is an overview of important LLM extraction parameters. All are typically set inside LLMExtractionStrategy(...). You then put that strategy in your CrawlerRunConfig(..., extraction_strategy=...).
\n\n
llm_config (LLMConfig): e.g., \"openai/gpt-4\", \"ollama/llama2\".\n2.âschema (dict): A JSON schema describing the fields you want. Usually generated by YourModel.model_json_schema(). \n3.âextraction_type (str): \"schema\" or \"block\". \n4.âinstruction (str): Prompt text telling the LLM what you want extracted. E.g., âExtract these fields as a JSON array.â \n5.âchunk_token_threshold (int): Maximum tokens per chunk. If your content is huge, you can break it up for the LLM. \n6.âoverlap_rate (float): Overlap ratio between adjacent chunks. E.g., 0.1 means 10% of each chunk is repeated to preserve context continuity. \n7.âapply_chunking (bool): Set True to chunk automatically. If you want a single pass, set False. \n8.âinput_format (str): Determines which crawler result is passed to the LLM. Options include:
\n
\"markdown\": The raw markdown (default).
\n
\"fit_markdown\": The filtered âfitâ markdown if you used a content filter.
\n
\"html\": The cleaned or raw HTML. \n9.âextra_args (dict): Additional LLM parameters like temperature, max_tokens, top_p, etc. \n10.âshow_usage(): A method you can call to print out usage info (token usage per chunk, total cost if known).
\n\n
Example:
\n
extraction_strategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"openai/gpt-4\", api_token=\"YOUR_OPENAI_KEY\"),\n schema=MyModel.model_json_schema(),\n extraction_type=\"schema\",\n instruction=\"Extract a list of items from the text with 'name' and 'price' fields.\",\n chunk_token_threshold=1200,\n overlap_rate=0.1,\n apply_chunking=True,\n input_format=\"html\",\n extra_args={\"temperature\":0.1, \"max_tokens\":1000},\n verbose=True\n)\n
\n\n
5. Putting It in CrawlerRunConfig
\n
Important: In Crawl4AI, all strategy definitions should go inside the CrawlerRunConfig, not directly as a param in arun(). Hereâs a full example:
\n
import os\nimport asyncio\nimport json\nfrom pydantic import BaseModel, Field\nfrom typing importList\nfrom crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig\nfrom crawl4ai import LLMExtractionStrategy\n\nclassProduct(BaseModel):\n name: str\n price: str\n\nasyncdefmain():\n # 1. Define the LLM extraction strategy\n llm_strategy = LLMExtractionStrategy(\n llm_config = LLMConfig(provider=\"openai/gpt-4o-mini\", api_token=os.getenv('OPENAI_API_KEY')),\n schema=Product.model_json_schema(), # Or use model_json_schema()\n extraction_type=\"schema\",\n instruction=\"Extract all product objects with 'name' and 'price' from the content.\",\n chunk_token_threshold=1000,\n overlap_rate=0.0,\n apply_chunking=True,\n input_format=\"markdown\", # or \"html\", \"fit_markdown\"\n extra_args={\"temperature\": 0.0, \"max_tokens\": 800}\n )\n\n # 2. Build the crawler config\n crawl_config = CrawlerRunConfig(\n extraction_strategy=llm_strategy,\n cache_mode=CacheMode.BYPASS\n )\n\n # 3. Create a browser config if needed\n browser_cfg = BrowserConfig(headless=True)\n\n asyncwith AsyncWebCrawler(config=browser_cfg) as crawler:\n # 4. Let's say we want to crawl a single page\n result = await crawler.arun(\n url=\"https://example.com/products\",\n config=crawl_config\n )\n\n if result.success:\n # 5. The extracted content is presumably JSON\n data = json.loads(result.extracted_content)\n print(\"Extracted items:\", data)\n\n # 6. Show usage stats\n llm_strategy.show_usage() # prints token usage\n else:\n print(\"Error:\", result.error_message)\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n
\n\n
6. Chunking Details
\n
6.1 chunk_token_threshold
\n
If your page is large, you might exceed your LLMâs context window.âchunk_token_threshold sets the approximate max tokens per chunk. The library calculates wordâtoken ratio using word_token_rate (often ~0.75 by default). If chunking is enabled (apply_chunking=True), the text is split into segments.
\n
6.2 overlap_rate
\n
To keep context continuous across chunks, we can overlap them. E.g., overlap_rate=0.1 means each subsequent chunk includes 10% of the previous chunkâs text. This is helpful if your needed info might straddle chunk boundaries.
\n
6.3 Performance & Parallelism
\n
By chunking, you can potentially process multiple chunks in parallel (depending on your concurrency settings and the LLM provider). This reduces total time if the site is huge or has many sections.
\n\n
7. Input Format
\n
By default, LLMExtractionStrategy uses input_format=\"markdown\", meaning the crawlerâs final markdown is fed to the LLM. You can change to:
\n
\n
html: The cleaned HTML or raw HTML (depending on your crawler config) goes into the LLM.
\n
fit_markdown: If you used, for instance, PruningContentFilter, the âfitâ version of the markdown is used. This can drastically reduce tokens if you trust the filter.
\n
markdown: Standard markdown output from the crawlerâs markdown_generator.
\n
\n
This setting is crucial: if the LLM instructions rely on HTML tags, pick \"html\". If you prefer a text-based approach, pick \"markdown\".
\n
LLMExtractionStrategy(\n # ...\n input_format=\"html\", # Instead of \"markdown\" or \"fit_markdown\"\n)\n
\n\n
8. Token Usage & Show Usage
\n
To keep track of tokens and cost, each chunk is processed with an LLM call. We record usage in:
\n
\n
usages (list): token usage per chunk or call.
\n
total_usage: sum of all chunk calls.
\n
show_usage(): prints a usage report (if the provider returns usage data).
\n
\n
llm_strategy = LLMExtractionStrategy(...)\n# ...\nllm_strategy.show_usage()\n# e.g. âTotal usage: 1241 tokens across 2 chunk callsâ\n
\n
If your model provider doesn't return usage info, these fields might be partial or empty.
\n
\n
Tip:JsonCssExtractionStrategy.generate_schema() also supports token usage tracking via an optional usage parameter. See Token Usage Tracking in Schema Generation for details.
\n
\n\n
9. Example: Building a Knowledge Graph
\n
Below is a snippet combining LLMExtractionStrategy with a Pydantic schema for a knowledge graph. Notice how we pass an instruction telling the model what to parse.
extraction_type=\"schema\" ensures we get JSON fitting our KnowledgeGraph.
\n
input_format=\"html\" means we feed HTML to the model.
\n
instruction guides the model to output a structured knowledge graph.
\n
\n\n
10. Best Practices & Caveats
\n
1.âCost & Latency: LLM calls can be slow or expensive. Consider chunking or smaller coverage if you only need partial data. \n2.âModel Token Limits: If your page + instruction exceed the context window, chunking is essential. \n3.âInstruction Engineering: Well-crafted instructions can drastically improve output reliability. \n4.âSchema Strictness: \"schema\" extraction tries to parse the model output as JSON. If the model returns invalid JSON, partial extraction might happen, or you might get an error. \n5.âParallel vs. Serial: The library can process multiple chunks in parallel, but you must watch out for rate limits on certain providers. \n6.âCheck Output: Sometimes, an LLM might omit fields or produce extraneous text. You may want to post-validate with Pydantic or do additional cleanup.
\n\n
11. Conclusion
\n
LLM-based extraction in Crawl4AI is provider-agnostic, letting you choose from hundreds of models via LiteLLM. Itâs perfect for semantically complex tasks or generating advanced structures like knowledge graphs. However, itâs slower and potentially costlier than schema-based approaches. Keep these tips in mind:
\n
\n
Put your LLM strategy in CrawlerRunConfig.
\n
Use input_format to pick which form (markdown, HTML, fit_markdown) the LLM sees.
\n
Tweak chunk_token_threshold, overlap_rate, and apply_chunking to handle large content efficiently.
\n
Monitor token usage with show_usage().
\n
\n
If your siteâs data is consistent or repetitive, consider JsonCssExtractionStrategy first for speed and simplicity. But if you need an AI-driven approach, LLMExtractionStrategy offers a flexible, multi-provider solution for extracting structured JSON from any website.
\n
Next Steps:
\n
1.âExperiment with Different Providers \n - Try switching the provider (e.g., \"ollama/llama2\", \"openai/gpt-4o\", etc.) to see differences in speed, accuracy, or cost. \n - Pass different extra_args like temperature, top_p, and max_tokens to fine-tune your results.
\n
2.âPerformance Tuning \n - If pages are large, tweak chunk_token_threshold, overlap_rate, or apply_chunking to optimize throughput. \n - Check the usage logs with show_usage() to keep an eye on token consumption and identify potential bottlenecks.
\n
3.âValidate Outputs \n - If using extraction_type=\"schema\", parse the LLMâs JSON with a Pydantic model for a final validation step. \n - Log or handle any parse errors gracefully, especially if the model occasionally returns malformed JSON.
\n
4.âExplore Hooks & Automation \n - Integrate LLM extraction with hooks for complex pre/post-processing. \n - Use a multi-step pipeline: crawl, filter, LLM-extract, then store or index results for further analysis.
\n
Last Updated: 2025-01-01
\n\n
Thatâs it for Extracting JSON (LLM)ânow you can harness AI to parse, classify, or reorganize data on the web. Happy crawling!
One of Crawl4AI's most powerful features is extracting structured JSON from websites without relying on large language models. Crawl4AI offers several strategies for LLM-free extraction:
\n\n
Schema-based extraction with CSS or XPath selectors via JsonCssExtractionStrategy and JsonXPathExtractionStrategy
\n
Regular expression extraction with RegexExtractionStrategy for fast pattern matching
\n\n
These approaches let you extract data instantlyâeven from complex or nested HTML structuresâwithout the cost, latency, or environmental impact of an LLM.
\n
Why avoid LLM for basic extractions?
\n\n
Faster & Cheaper: No API calls or GPU overhead.
\n
Lower Carbon Footprint: LLM inference can be energy-intensive. Pattern-based extraction is practically carbon-free.
\n
Precise & Repeatable: CSS/XPath selectors and regex patterns do exactly what you specify. LLM outputs can vary or hallucinate.
\n
Scales Readily: For thousands of pages, pattern-based extraction runs quickly and in parallel.
\n\n
Below, we'll explore how to craft these schemas and use them with JsonCssExtractionStrategy (or JsonXPathExtractionStrategy if you prefer XPath). We'll also highlight advanced features like nested fields and base element attributes.
\n\n
1. Intro to Schema-Based Extraction
\n
A schema defines:
\n\n
A base selector that identifies each \"container\" element on the page (e.g., a product row, a blog post card).
\n
Fields describing which CSS/XPath selectors to use for each piece of data you want to capture (text, attribute, HTML block, etc.).
\n
Nested or list types for repeated or hierarchical structures.
\n\n
For example, if you have a list of products, each one might have a name, price, reviews, and \"related products.\" This approach is faster and more reliable than an LLM for consistent, structured pages.
\n\n
2. Simple Example: Crypto Prices
\n
Let's begin with a simple schema-based extraction using the JsonCssExtractionStrategy. Below is a snippet that extracts cryptocurrency prices from a site (similar to the legacy Coinbase example). Notice we don't call any LLM:
\n
import json\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode\nfrom crawl4ai import JsonCssExtractionStrategy\n\nasyncdefextract_crypto_prices():\n # 1. Define a simple extraction schema\n schema = {\n \"name\": \"Crypto Prices\",\n \"baseSelector\": \"div.crypto-row\", # Repeated elements\n \"fields\": [\n {\n \"name\": \"coin_name\",\n \"selector\": \"h2.coin-name\",\n \"type\": \"text\"\n },\n {\n \"name\": \"price\",\n \"selector\": \"span.coin-price\",\n \"type\": \"text\"\n }\n ]\n }\n\n # 2. Create the extraction strategy\n extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)\n\n # 3. Set up your crawler config (if needed)\n config = CrawlerRunConfig(\n # e.g., pass js_code or wait_for if the page is dynamic\n # wait_for=\"css:.crypto-row:nth-child(20)\"\n cache_mode = CacheMode.BYPASS,\n extraction_strategy=extraction_strategy,\n )\n\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n # 4. Run the crawl and extraction\n result = await crawler.arun(\n url=\"https://example.com/crypto-prices\",\n\n config=config\n )\n\n ifnot result.success:\n print(\"Crawl failed:\", result.error_message)\n return\n\n # 5. Parse the extracted JSON\n data = json.loads(result.extracted_content)\n print(f\"Extracted {len(data)} coin entries\")\n print(json.dumps(data[0], indent=2) if data else\"No data found\")\n\nasyncio.run(extract_crypto_prices())\n
\n
Highlights:
\n
\n
baseSelector: Tells us where each \"item\" (crypto row) is.
\n
fields: Two fields (coin_name, price) using simple CSS selectors.
\n
Each field defines a type (e.g., text, attribute, html, regex, etc.).
\n
Optional keys: transform, default, attribute, pattern, and source (for sibling data â see Extracting Sibling Data).
\n
\n
No LLM is needed, and the performance is near-instant for hundreds or thousands of items.
\n\n
XPath Example with raw:// HTML
\n
Below is a short example demonstrating XPath extraction plus the raw:// scheme. We'll pass a dummy HTML directly (no network request) and define the extraction strategy in CrawlerRunConfig.
\n
import json\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai import JsonXPathExtractionStrategy\n\nasyncdefextract_crypto_prices_xpath():\n # 1. Minimal dummy HTML with some repeating rows\n dummy_html = \"\"\"\n <html>\n <body>\n <div class='crypto-row'>\n <h2 class='coin-name'>Bitcoin</h2>\n <span class='coin-price'>$28,000</span>\n </div>\n <div class='crypto-row'>\n <h2 class='coin-name'>Ethereum</h2>\n <span class='coin-price'>$1,800</span>\n </div>\n </body>\n </html>\n \"\"\"\n\n # 2. Define the JSON schema (XPath version)\n schema = {\n \"name\": \"Crypto Prices via XPath\",\n \"baseSelector\": \"//div[@class='crypto-row']\",\n \"fields\": [\n {\n \"name\": \"coin_name\",\n \"selector\": \".//h2[@class='coin-name']\",\n \"type\": \"text\"\n },\n {\n \"name\": \"price\",\n \"selector\": \".//span[@class='coin-price']\",\n \"type\": \"text\"\n }\n ]\n }\n\n # 3. Place the strategy in the CrawlerRunConfig\n config = CrawlerRunConfig(\n extraction_strategy=JsonXPathExtractionStrategy(schema, verbose=True)\n )\n\n # 4. Use raw:// scheme to pass dummy_html directly\n raw_url = f\"raw://{dummy_html}\"\n\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n result = await crawler.arun(\n url=raw_url,\n config=config\n )\n\n ifnot result.success:\n print(\"Crawl failed:\", result.error_message)\n return\n\n data = json.loads(result.extracted_content)\n print(f\"Extracted {len(data)} coin rows\")\n if data:\n print(\"First item:\", data[0])\n\nasyncio.run(extract_crypto_prices_xpath())\n
\n
Key Points:
\n\n
JsonXPathExtractionStrategy is used instead of JsonCssExtractionStrategy.
\n
baseSelector and each field's \"selector\" use XPath instead of CSS.
\n
raw:// lets us pass dummy_html with no real network requestâhandy for local testing.
\n
Everything (including the extraction strategy) is in CrawlerRunConfig.
\n\n
That's how you keep the config self-contained, illustrate XPath usage, and demonstrate the raw scheme for direct HTML inputâall while avoiding the old approach of passing extraction_strategy directly to arun().
\n\n
3. Advanced Schema & Nested Structures
\n
Real sites often have nested or repeated dataâlike categories containing products, which themselves have a list of reviews or features. For that, we can define nested or list (and even nested_list) fields.
\n
Sample E-Commerce HTML
\n
We have a sample e-commerce HTML file on GitHub (example):\n
\nThis snippet includes categories, products, features, reviews, and related items. Let's see how to define a schema that fully captures that structure without LLM.\n
type: \"nested\" means a single sub-object (like details).
\n
type: \"list\" means multiple items that are simple dictionaries or single text fields.
\n
type: \"nested_list\" means repeated complex objects (like products or reviews).
\n
Base Fields: We can extract attributes from the container element via \"baseFields\". For instance, \"data_cat_id\" might be data-cat-id=\"elect123\".
\n
Transforms: We can also define a transform if we want to lower/upper case, strip whitespace, or even run a custom function.
\n
\n
Running the Extraction
\n
import json\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\nfrom crawl4ai import JsonCssExtractionStrategy\n\necommerce_schema = {\n # ... the advanced schema from above ...\n}\n\nasyncdefextract_ecommerce_data():\n strategy = JsonCssExtractionStrategy(ecommerce_schema, verbose=True)\n\n config = CrawlerRunConfig()\n\n asyncwith AsyncWebCrawler(verbose=True) as crawler:\n result = await crawler.arun(\n url=\"https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/examples/sample_ecommerce.html\",\n extraction_strategy=strategy,\n config=config\n )\n\n ifnot result.success:\n print(\"Crawl failed:\", result.error_message)\n return\n\n # Parse the JSON output\n data = json.loads(result.extracted_content)\n print(json.dumps(data, indent=2) if data else\"No data found.\")\n\nasyncio.run(extract_ecommerce_data())\n
\n
If all goes well, you get a structured JSON array with each \"category,\" containing an array of products. Each product includes details, features, reviews, etc. All of that without an LLM.
\n\n
4. RegexExtractionStrategy - Fast Pattern-Based Extraction
\n
Crawl4AI now offers a powerful new zero-LLM extraction strategy: RegexExtractionStrategy. This strategy provides lightning-fast extraction of common data types like emails, phone numbers, URLs, dates, and more using pre-compiled regular expressions.
\n
Key Features
\n
\n
Zero LLM Dependency: Extracts data without any AI model calls
\n
Blazing Fast: Uses pre-compiled regex patterns for maximum performance
\n
Built-in Patterns: Includes ready-to-use patterns for common data types
\n
Custom Patterns: Add your own regex patterns for domain-specific extraction
\n
LLM-Assisted Pattern Generation: Optionally use an LLM once to generate optimized patterns, then reuse them without further LLM calls
\n
\n
Simple Example: Extracting Common Entities
\n
The easiest way to start is by using the built-in pattern catalog:
\n
import json\nimport asyncio\nfrom crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n RegexExtractionStrategy\n)\n\nasyncdefextract_with_regex():\n # Create a strategy using built-in patterns for URLs and currencies\n strategy = RegexExtractionStrategy(\n pattern = RegexExtractionStrategy.Url | RegexExtractionStrategy.Currency\n )\n\n config = CrawlerRunConfig(extraction_strategy=strategy)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com\",\n config=config\n )\n\n if result.success:\n data = json.loads(result.extracted_content)\n for item in data[:5]: # Show first 5 matches\n print(f\"{item['label']}: {item['value']}\")\n print(f\"Total matches: {len(data)}\")\n\nasyncio.run(extract_with_regex())\n
\n
Available Built-in Patterns
\n
RegexExtractionStrategy provides these common patterns as IntFlag attributes for easy combining:
\n
# Use individual patterns\nstrategy = RegexExtractionStrategy(pattern=RegexExtractionStrategy.Email)\n\n# Combine multiple patterns\nstrategy = RegexExtractionStrategy(\n pattern = (\n RegexExtractionStrategy.Email | \n RegexExtractionStrategy.PhoneUS | \n RegexExtractionStrategy.Url\n )\n)\n\n# Use all available patterns\nstrategy = RegexExtractionStrategy(pattern=RegexExtractionStrategy.All)\n
\n
Available patterns include:\n- Email - Email addresses\n- PhoneIntl - International phone numbers\n- PhoneUS - US-format phone numbers\n- Url - HTTP/HTTPS URLs\n- IPv4 - IPv4 addresses\n- IPv6 - IPv6 addresses\n- Uuid - UUIDs\n- Currency - Currency values (USD, EUR, etc.)\n- Percentage - Percentage values\n- Number - Numeric values\n- DateIso - ISO format dates\n- DateUS - US format dates\n- Time24h - 24-hour format times\n- PostalUS - US postal codes\n- PostalUK - UK postal codes\n- HexColor - HTML hex color codes\n- TwitterHandle - Twitter handles\n- Hashtag - Hashtags\n- MacAddr - MAC addresses\n- Iban - International bank account numbers\n- CreditCard - Credit card numbers
\n
Custom Pattern Example
\n
For more targeted extraction, you can provide custom patterns:
\n
import json\nimport asyncio\nfrom crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n RegexExtractionStrategy\n)\n\nasyncdefextract_prices():\n # Define a custom pattern for US Dollar prices\n price_pattern = {\"usd_price\": r\"\\$\\s?\\d{1,3}(?:,\\d{3})*(?:\\.\\d{2})?\"}\n\n # Create strategy with custom pattern\n strategy = RegexExtractionStrategy(custom=price_pattern)\n config = CrawlerRunConfig(extraction_strategy=strategy)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://www.example.com/products\",\n config=config\n )\n\n if result.success:\n data = json.loads(result.extracted_content)\n for item in data:\n print(f\"Found price: {item['value']}\")\n\nasyncio.run(extract_prices())\n
\n
LLM-Assisted Pattern Generation
\n
For complex or site-specific patterns, you can use an LLM once to generate an optimized pattern, then save and reuse it without further LLM calls:
\n
import json\nimport asyncio\nfrom pathlib import Path\nfrom crawl4ai import (\n AsyncWebCrawler,\n CrawlerRunConfig,\n RegexExtractionStrategy,\n LLMConfig\n)\n\nasyncdefextract_with_generated_pattern():\n cache_dir = Path(\"./pattern_cache\")\n cache_dir.mkdir(exist_ok=True)\n pattern_file = cache_dir / \"price_pattern.json\"\n\n # 1. Generate or load pattern\n if pattern_file.exists():\n pattern = json.load(pattern_file.open())\n print(f\"Using cached pattern: {pattern}\")\n else:\n print(\"Generating pattern via LLM...\")\n\n # Configure LLM\n llm_config = LLMConfig(\n provider=\"openai/gpt-4o-mini\",\n api_token=\"env:OPENAI_API_KEY\",\n )\n\n # Get sample HTML for context\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\"https://example.com/products\")\n html = result.markdown.fit_html\n\n # Generate pattern (one-time LLM usage)\n pattern = RegexExtractionStrategy.generate_pattern(\n label=\"price\",\n html=html,\n query=\"Product prices in USD format\",\n llm_config=llm_config,\n )\n\n # Cache pattern for future use\n json.dump(pattern, pattern_file.open(\"w\"), indent=2)\n\n # 2. Use pattern for extraction (no LLM calls)\n strategy = RegexExtractionStrategy(custom=pattern)\n config = CrawlerRunConfig(extraction_strategy=strategy)\n\n asyncwith AsyncWebCrawler() as crawler:\n result = await crawler.arun(\n url=\"https://example.com/products\",\n config=config\n )\n\n if result.success:\n data = json.loads(result.extracted_content)\n for item in data[:10]:\n print(f\"Extracted: {item['value']}\")\n print(f\"Total matches: {len(data)}\")\n\nasyncio.run(extract_with_generated_pattern())\n
\n
This pattern allows you to:\n1. Use an LLM once to generate a highly optimized regex for your specific site\n2. Save the pattern to disk for reuse \n3. Extract data using only regex (no further LLM calls) in production
\n
Extraction Results Format
\n
The RegexExtractionStrategy returns results in a consistent format:
Each match includes:\n- url: The source URL\n- label: The pattern name that matched (e.g., \"email\", \"phone_us\")\n- value: The extracted text\n- span: The start and end positions in the source content
\n\n
5. Why \"No LLM\" Is Often Better
\n\n
Zero Hallucination: Pattern-based extraction doesn't guess text. It either finds it or not.
\n
Guaranteed Structure: The same schema or regex yields consistent JSON across many pages, so your downstream pipeline can rely on stable keys.
\n
Speed: LLM-based extraction can be 10â1000x slower for large-scale crawling.
\n
Scalable: Adding or updating a field is a matter of adjusting the schema or regex, not re-tuning a model.
\n\n
When might you consider an LLM? Possibly if the site is extremely unstructured or you want AI summarization. But always try a schema or regex approach first for repeated or consistent data patterns.
\n\n
6. Base Element Attributes & Additional Fields
\n
It's easy to extract attributes (like href, src, or data-xxx) from your base or nested elements using:
You can define them in baseFields (extracted from the main container element) or in each field's sub-lists. This is especially helpful if you need an item's link or ID stored in the parent <div>.
\n\n
7. Putting It All Together: Larger Example
\n
Consider a blog site. We have a schema that extracts the URL from each post card (via baseFields with an \"attribute\": \"href\"), plus the title, date, summary, and author:
Then run with JsonCssExtractionStrategy(schema) to get an array of blog post objects, each with \"post_url\", \"title\", \"date\", \"summary\", \"author\".
\n\n
8. Extracting Sibling Data with source
\n
Some websites split a single logical item across sibling elements rather than nesting everything inside one container. A classic example is Hacker News, where each submission spans two adjacent <tr> rows:
Normally, field selectors only search descendants of the base element â siblings are unreachable. The source field key solves this by navigating to a sibling element before running the selector.
\n
Syntax
\n
\"source\":\"+ <selector>\"\n
\n
\n
+ tr â next sibling <tr>
\n
+ div.details â next sibling <div> with class details
The score and author fields first navigate to the next sibling <tr>, then run their selectors inside that element. Fields without source work as before â searching descendants of the base element.
\n
source works with all field types (text, attribute, nested, list, etc.) and with both JsonCssExtractionStrategy and JsonXPathExtractionStrategy. If the sibling isn't found, the field returns its default value.
\n\n
9. Tips & Best Practices
\n\n
Inspect the DOM in Chrome DevTools or Firefox's Inspector to find stable selectors.
\n
Start Simple: Verify you can extract a single field. Then add complexity like nested objects or lists.
\n
Test your schema on partial HTML or a test page before a big crawl.
\n
Combine with JS Execution if the site loads content dynamically. You can pass js_code or wait_for in CrawlerRunConfig.
\n
Look at Logs when verbose=True: if your selectors are off or your schema is malformed, it'll often show warnings.
\n
Use baseFields if you need attributes from the container element (e.g., href, data-id), especially for the \"parent\" item.
\n
Performance: For large pages, make sure your selectors are as narrow as possible.
\n
Consider Using Regex First: For simple data types like emails, URLs, and dates, RegexExtractionStrategy is often the fastest approach.
\n\n\n
10. Schema Generation Utility
\n
While manually crafting schemas is powerful and precise, Crawl4AI now offers a convenient utility to automatically generate extraction schemas using LLM. This is particularly useful when:
\n\n
You're dealing with a new website structure and want a quick starting point
\n
You need to extract complex nested data structures
\n
You want to avoid the learning curve of CSS/XPath selector syntax
\n\n
Using the Schema Generator
\n
The schema generator is available as a static method on both JsonCssExtractionStrategy and JsonXPathExtractionStrategy. You can choose between OpenAI's GPT-4 or the open-source Ollama for schema generation:
\n
from crawl4ai import JsonCssExtractionStrategy, JsonXPathExtractionStrategy\nfrom crawl4ai import LLMConfig\n\n# Sample HTML with product information\nhtml = \"\"\"\n<div class=\"product-card\">\n <h2 class=\"title\">Gaming Laptop</h2>\n <div class=\"price\">$999.99</div>\n <div class=\"specs\">\n <ul>\n <li>16GB RAM</li>\n <li>1TB SSD</li>\n </ul>\n </div>\n</div>\n\"\"\"\n\n# Option 1: Using OpenAI (requires API token)\ncss_schema = JsonCssExtractionStrategy.generate_schema(\n html,\n schema_type=\"css\",\n llm_config = LLMConfig(provider=\"openai/gpt-4o\",api_token=\"your-openai-token\")\n)\n\n# Option 2: Using Ollama (open source, no token needed)\nxpath_schema = JsonXPathExtractionStrategy.generate_schema(\n html,\n schema_type=\"xpath\",\n llm_config = LLMConfig(provider=\"ollama/llama3.3\", api_token=None) # Not needed for Ollama\n)\n\n# Use the generated schema for fast, repeated extractions\nstrategy = JsonCssExtractionStrategy(css_schema)\n
\n
Schema Validation
\n
By default, generate_schemavalidates the generated schema against the HTML to ensure that it actually extracts the data you expect. If the schema doesn't produce results, it automatically refines the selectors before returning.
\n
You can control this with the validate parameter:
\n
# Default: validated (recommended)\nschema= JsonCssExtractionStrategy.generate_schema(\n url=\"https://news.ycombinator.com\",\n query=\"Extract each story: title, url, score, author\",\n)\n\n# Skip validation if you want raw LLM output\nschema= JsonCssExtractionStrategy.generate_schema(\n url=\"https://news.ycombinator.com\",\n query=\"Extract each story: title, url, score, author\",\n validate=False,\n)\n
\n
The generator also understands sibling layouts â for sites like Hacker News where data is split across sibling elements, it will automatically use the source field to reach sibling data.
\n
Token Usage Tracking
\n
generate_schema may make multiple LLM calls internally (field inference, schema generation, validation retries). To track the total token consumption across all of these calls, pass a TokenUsage accumulator:
The usage parameter is optional â omitting it changes nothing (fully backward-compatible). You can also reuse the same accumulator across multiple calls to get a grand total:
Both generate_schema (sync) and agenerate_schema (async) support the usage parameter.
\n
LLM Provider Options
\n\n
OpenAI GPT-4 (openai/gpt4o)
\n
Default provider
\n
Requires an API token
\n
Generally provides more accurate schemas
\n
\n
Set via environment variable: OPENAI_API_KEY
\n
\n
\n
Ollama (ollama/llama3.3)
\n
\n
Open source alternative
\n
No API token required
\n
Self-hosted option
\n
Good for development and testing
\n\n
Benefits of Schema Generation
\n\n
One-Time Cost: While schema generation uses LLM, it's a one-time cost. The generated schema can be reused for unlimited extractions without further LLM calls.
\n
Smart Pattern Recognition: The LLM analyzes the HTML structure and identifies common patterns, often producing more robust selectors than manual attempts.
\n
Automatic Nesting: Complex nested structures are automatically detected and properly represented in the schema.
\n
Learning Tool: The generated schemas serve as excellent examples for learning how to write your own schemas.
\n\n
Best Practices
\n\n
Review Generated Schemas: While the generator is smart, always review and test the generated schema before using it in production.
\n
Provide Representative HTML: The better your sample HTML represents the overall structure, the more accurate the generated schema will be.
\n
Consider Both CSS and XPath: Try both schema types and choose the one that works best for your specific case.
\n
Cache Generated Schemas: Since generation uses LLM, save successful schemas for reuse.
\n
API Token Security: Never hardcode API tokens. Use environment variables or secure configuration management.
\n
Choose Provider Wisely:
\n
Use OpenAI for production-quality schemas
\n
Use Ollama for development, testing, or when you need a self-hosted solution
\n\n
Multi-Sample Schema Generation
\n
When scraping multiple pages with varying DOM structures (e.g., product pages where table rows appear in different positions), single-sample schema generation may produce fragile selectors like tr:nth-child(6) that break on other pages.
The Solution: Provide multiple HTML samples so the LLM identifies stable patterns that work across all pages.
\n
from crawl4ai import JsonCssExtractionStrategy, LLMConfig\n\n# Collect HTML samples from different pages\nhtml_sample_1 = \"\"\"\n<tableclass=\"specs\">\n <tr><td>Brand</td><td>Apple</td></tr>\n <tr><td>Manufacturer</td><td><ahref=\"/m/apple\">Apple Inc</a></td></tr>\n</table>\n\"\"\"\n\nhtml_sample_2 = \"\"\"\n<tableclass=\"specs\">\n <tr><td>Manufacturer</td><td><ahref=\"/m/samsung\">Samsung</a></td></tr>\n <tr><td>Brand</td><td>Galaxy</td></tr>\n</table>\n\"\"\"\n\nhtml_sample_3 = \"\"\"\n<tableclass=\"specs\">\n <tr><td>Model</td><td>Pixel 8</td></tr>\n <tr><td>Brand</td><td>Google</td></tr>\n <tr><td>Manufacturer</td><td><ahref=\"/m/google\">Google LLC</a></td></tr>\n</table>\n\"\"\"\n\n# Combine samples with labels\ncombined_html = \"\"\"\n## HTML Sample 1 (Product A):\n```html\n\"\"\" + html_sample_1 + \"\"\"\n
\n
HTML Sample 2 (Product B):
\n
\"\"\" + html_sample_2 + \"\"\"\n
\n
HTML Sample 3 (Product C):
\n
\"\"\" + html_sample_3 + \"\"\"\n
\n\"\"\"\n
Provide instructions for stable selectors
\n
query = \"\"\"\nIMPORTANT: I'm providing 3 HTML samples from different product pages.\nThe manufacturer field appears in different row positions across pages.\nGenerate selectors using stable attributes like href patterns (e.g., a[href*='/m/'])\ninstead of fragile positional selectors like nth-child().\nExtract: manufacturer name and link.\n\"\"\"
The generated schema will use stable selectors like:
\n
a[href*=\"/m/\"] instead of tr:nth-child(6) td a
\n
print(schema)\n```
\n
Key Points for Multi-Sample Queries:
\n\n
Format samples clearly - Use markdown headers and code blocks to separate samples
\n
State the number of samples - \"I'm providing 3 HTML samples...\"
\n
Explain the variation - \"...the manufacturer field appears in different row positions\"
\n
Request stable selectors - \"Use href patterns, data attributes, or class names instead of nth-child\"
\n\n
Stable vs Fragile Selectors:
\n
\n\n
\n
Fragile (single sample)
\n
Stable (multi-sample)
\n
\n\n\n
\n
tr:nth-child(6) td a
\n
a[href*=\"/m/\"]
\n
\n
\n
div:nth-child(3) .price
\n
.price, [data-price]
\n
\n
\n
ul li:first-child
\n
li[data-featured=\"true\"]
\n
\n\n
\n
This approach lets you generate schemas once that work reliably across hundreds of similar pages with varying structures.
\n\n
11. Conclusion
\n
With Crawl4AI's LLM-free extraction strategies - JsonCssExtractionStrategy, JsonXPathExtractionStrategy, and now RegexExtractionStrategy - you can build powerful pipelines that:
\n
\n
Scrape any consistent site for structured data.
\n
Support nested objects, repeating lists, or pattern-based extraction.
\n
Scale to thousands of pages quickly and reliably.
\n
\n
Choosing the Right Strategy:
\n
\n
Use RegexExtractionStrategy for fast extraction of common data types like emails, phones, URLs, dates, etc.
\n
Use JsonCssExtractionStrategy or JsonXPathExtractionStrategy for structured data with clear HTML patterns
\n
If you need both: first extract structured data with JSON strategies, then use regex on specific fields
\n
\n
Remember: For repeated, structured data, you don't need to pay for or wait on an LLM. Well-crafted schemas and regex patterns get you the data faster, cleaner, and cheaperâthe real power of Crawl4AI.
\n
Last Updated: 2025-05-02
\n\n
That's it for Extracting JSON (No LLM)! You've seen how schema-based approaches (either CSS or XPath) and regex patterns can handle everything from simple lists to deeply nested product catalogsâinstantly, with minimal overhead. Enjoy building robust scrapers that produce consistent, structured JSON for your data pipelines!
Version 0.7.3 introduces the Table Extraction Strategy Pattern, providing a more flexible and extensible approach to table extraction while maintaining full backward compatibility.
\n
What's New
\n
Strategy Pattern Implementation
\n
Table extraction now follows the same strategy pattern used throughout Crawl4AI:
\n
\n
Consistent Architecture: Aligns with extraction, chunking, and markdown strategies
\n
Extensibility: Easy to create custom table extraction strategies
\n
Better Separation: Table logic moved from content scraping to dedicated module
\n
Full Control: Fine-grained control over table detection and extraction
\n
\n
New Classes
\n
from crawl4ai import (\n TableExtractionStrategy, # Abstract base class\n DefaultTableExtraction, # Current implementation (default)\n NoTableExtraction # Explicitly disable extraction\n)\n
\n
Backward Compatibility
\n
â All existing code continues to work without changes.
\n
No Changes Required
\n
If your code looks like this, it will continue to work:
\n
# This still works exactly the same\nconfig = CrawlerRunConfig(\n table_score_threshold=7\n)\nresult = await crawler.arun(url, config)\ntables = result.tables # Same structure, same data\n
\n
What Happens Behind the Scenes
\n
When you don't specify a table_extraction strategy:
# Still works the same\nconfig = CrawlerRunConfig(\n table_score_threshold=5\n)\n\n# Or use new explicit approach for more control\nstrategy = DefaultTableExtraction(\n table_score_threshold=5,\n min_rows=2 # Additional filtering\n)\nconfig = CrawlerRunConfig(\n table_extraction=strategy\n)\n
\n
Scenario 3: Advanced Filtering (New Feature)
\n
Before (v0.7.2):\n
# Had to filter after extraction\nconfig = CrawlerRunConfig(\n table_score_threshold=5\n)\nresult = await crawler.arun(url, config)\n\n# Manual filtering\nlarge_tables = [\n t for t in result.tables \n iflen(t['rows']) >= 5andlen(t['headers']) >= 3\n]\n
crawl4ai/\n content_scraping_strategy.py\n - LXMLWebScrapingStrategy\n # Table methods removed, uses strategy\n\n table_extraction.py (NEW)\n - TableExtractionStrategy # Base class\n - DefaultTableExtraction # Moved logic here\n - NoTableExtraction # New option\n
\n
Import Changes
\n
New imports available (optional):\n
# These are now available but not required for existing code\nfrom crawl4ai import (\n TableExtractionStrategy,\n DefaultTableExtraction,\n NoTableExtraction\n)\n
\n
Performance Implications
\n
No Performance Impact
\n
For existing code, performance remains identical:\n- Same extraction logic\n- Same scoring algorithm\n- Same processing time
\n
Performance Improvements Available
\n
New options for better performance:
\n
# Skip tables entirely (faster)\nconfig = CrawlerRunConfig(\n table_extraction=NoTableExtraction()\n)\n\n# Process only specific areas (faster)\nconfig = CrawlerRunConfig(\n css_selector=\"main.content\",\n table_extraction=DefaultTableExtraction(\n min_rows=5, # Skip small tables\n min_cols=3\n )\n)\n
\n
Testing Your Migration
\n
Verification Script
\n
Run this to verify your extraction still works:
\n
import asyncio\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig\n\nasyncdefverify_extraction():\n url = \"your_url_here\"\n\n asyncwith AsyncWebCrawler() as crawler:\n # Test 1: Old approach\n config_old = CrawlerRunConfig(\n table_score_threshold=7\n )\n result_old = await crawler.arun(url, config_old)\n\n # Test 2: New explicit approach\n from crawl4ai import DefaultTableExtraction\n config_new = CrawlerRunConfig(\n table_extraction=DefaultTableExtraction(\n table_score_threshold=7\n )\n )\n result_new = await crawler.arun(url, config_new)\n\n # Compare results\n assertlen(result_old.tables) == len(result_new.tables)\n print(f\"â Both approaches extracted {len(result_old.tables)} tables\")\n\n # Verify structure\n for old, new inzip(result_old.tables, result_new.tables):\n assert old['headers'] == new['headers']\n assert old['rows'] == new['rows']\n\n print(\"â Table content identical\")\n\nasyncio.run(verify_extraction())\n
\n
Deprecation Notes
\n
No Deprecations
\n
\n
All existing parameters continue to work
\n
table_score_threshold in CrawlerRunConfig is still supported
\n
No breaking changes
\n
\n
Internal Changes (Transparent to Users)
\n
\n
LXMLWebScrapingStrategy.is_data_table() - Moved to DefaultTableExtraction
\n
LXMLWebScrapingStrategy.extract_table_data() - Moved to DefaultTableExtraction
\n
\n
These methods were internal and not part of the public API.
\n
Benefits of Upgrading
\n
While not required, using the new pattern provides:
\n\n
Better Control: Filter tables during extraction, not after
\n
Performance Options: Skip extraction when not needed
\n
Extensibility: Create custom extractors for specific needs
\n
Consistency: Same pattern as other Crawl4AI strategies
\n
Future-Proof: Ready for upcoming advanced strategies
\n\n
Troubleshooting
\n
Issue: Different Number of Tables
\n
Cause: Threshold or filtering differences
\n
Solution: \n
# Ensure same threshold\nstrategy = DefaultTableExtraction(\n table_score_threshold=7, # Match your old setting\n min_rows=0, # No filtering (default)\n min_cols=0 # No filtering (default)\n)\n
\n
Issue: Import Errors
\n
Cause: Using new classes without importing
\n
Solution:\n
# Add imports if using new features\nfrom crawl4ai import (\n DefaultTableExtraction,\n NoTableExtraction,\n TableExtractionStrategy\n)\n
Crawl4AI has simplified its content scraping architecture. The BeautifulSoup-based WebScrapingStrategy has been deprecated in favor of the faster LXML-based implementation. However, no action is required - your existing code will continue to work.
\n
What Changed?
\n\n
WebScrapingStrategy is now an alias for LXMLWebScrapingStrategy
\n
The BeautifulSoup implementation has been removed (~1000 lines of redundant code)
\n
LXMLWebScrapingStrategy inherits directly from ContentScrapingStrategy
\n
Performance remains optimal with LXML as the sole implementation
\n\n
Backward Compatibility
\n
Your existing code continues to work without any changes:
\n
# This still works perfectly\nfrom crawl4ai import AsyncWebCrawler, CrawlerRunConfig, WebScrapingStrategy\n\nconfig = CrawlerRunConfig(\n scraping_strategy=WebScrapingStrategy() # Works as before\n)\n
\n
Migration Options
\n
You have three options:
\n
Option 1: Do Nothing (Recommended)
\n
Your code will continue to work. WebScrapingStrategy is permanently aliased to LXMLWebScrapingStrategy.
\n
Option 2: Update Imports (Optional)
\n
For clarity, you can update your imports:
\n
# Old (still works)\nfrom crawl4ai import WebScrapingStrategy\nstrategy = WebScrapingStrategy()\n\n# New (more explicit)\nfrom crawl4ai import LXMLWebScrapingStrategy\nstrategy = LXMLWebScrapingStrategy()\n
\n
Option 3: Use Default Configuration
\n
Since LXMLWebScrapingStrategy is the default, you can omit the strategy parameter:
\n
# Simplest approach - uses LXMLWebScrapingStrategy by default\nconfig = CrawlerRunConfig()\n
\n
Type Hints
\n
If you use type hints, both work:
\n
from crawl4ai import WebScrapingStrategy, LXMLWebScrapingStrategy\n\ndefprocess_with_strategy(strategy: WebScrapingStrategy) -> None:\n # Works with both WebScrapingStrategy and LXMLWebScrapingStrategy\n pass\n\n# Both are valid\nprocess_with_strategy(WebScrapingStrategy())\nprocess_with_strategy(LXMLWebScrapingStrategy())\n
\n
Subclassing
\n
If you've subclassed WebScrapingStrategy, it continues to work:
\n
classMyCustomStrategy(WebScrapingStrategy):\n def__init__(self):\n super().__init__()\n # Your custom code\n
\n
Performance Benefits
\n
By consolidating to LXML:\n- 10-20x faster HTML parsing for large documents\n- Lower memory usage\n- Consistent behavior across all use cases\n- Simplified maintenance and bug fixes
\n
Summary
\n
This change simplifies Crawl4AI's internals while maintaining 100% backward compatibility. Your existing code continues to work, and you get better performance automatically.
![\"GitHub](\"https://img.shields.io/github/stars/unclecode/crawl4ai?style=social\")
\n ![\"GitHub](\"https://img.shields.io/github/forks/unclecode/crawl4ai?style=social\")
\n ![\"PyPI](\"https://badge.fury.io/py/crawl4ai.svg\"){kind=icon}
\n ![\"Python](\"https://img.shields.io/pypi/pyversions/crawl4ai\")
\n ![\"Downloads\"](\"https://static.pepy.tech/badge/crawl4ai/month\"){kind=icon}
\n ![\"License\"](\"https://img.shields.io/github/license/unclecode/crawl4ai\")
\n ![\"Follow](\"https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white\"){kind=icon}
\n ![\"Follow](\"https://img.shields.io/badge/Follow%20on%20LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white\"){kind=icon}
\n ![\"Join](\"https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white\"){kind=icon}
\n ![\"Crawl](\"../../assets/images/dispatcher.png\")