🚀🤖 Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper

Summary: Crawl4AI is an open-source, LLM-friendly web crawler and scraper that produces clean Markdown, supports structured extraction, advanced browser control, and high-performance parallel crawling. This…

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be **drastically more cost-effective** than any of the existing solutions.

👉 **Apply [here](https://forms.gle/E9MyPaNXACnAMaqG7) for early…

Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data…

🆕 AI Assistant Skill Now Available!

🤖 Crawl4AI Skill for Claude & AI Assistants

Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:

- 📚 Complete SDK reference (23K+ words)
- 🚀 Ready-to-use extraction…

🎯 New: Adaptive Web Crawling

Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your…

Quick Start

Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    # Create an instance of AsyncWebCrawler
    async with AsyncWebCrawler() as crawler:
        # Run the crawler on a URL…

Video Tutorial

What Does Crawl4AI Do?

Crawl4AI is a feature-rich crawler and scraper that aims to:

1. **Generate Clean Markdown**: Perfect for RAG pipelines or direct ingestion into LLMs.
2. **Structured Extraction**: Parse repeated…

Documentation Structure

To help you get started, we’ve organized our docs into clear sections:

- **Setup & Installation**: Basic instructions to install Crawl4AI via pip or Docker.
- **Quick Start**: A hands-on…

How You Can Support

- **Star & Fork**: If you find Crawl4AI helpful, star the repo on GitHub or fork it to add your own features.
- **File Issues**: Encounter a bug or missing feature? Let us know by filing an issue, so…

Quick Links

- **[GitHub Repo](https://github.com/unclecode/crawl4ai)**
- **[Installation Guide](core/installation/)**
- **[Quick Start](core/quickstart/)**
- **[API Reference](api/async-webcrawler/)**
-…


Home - Crawl4AI Documentation (v0.9.x)

Summary: Landing page and overview for Crawl4AI, an open-source, LLM-friendly web crawler and scraper. It introduces the project's mission, key capabilities, quick-start example, documentation structure, and…

🚀🤖 Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 Apply here for early…

🆕 AI Assistant Skill Now Available!

🤖 Crawl4AI Skill for Claude & AI Assistants

Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:

- 📚 Complete SDK reference…

🎯 New: Adaptive Web Crawling

Quick Start

Video Tutorial

What Does Crawl4AI Do?

Crawl4AI is a feature-rich crawler and scraper that aims to:

1. Generate Clean Markdown: Perfect for RAG pipelines or direct ingestion into LLMs.
2. Structured Extraction: Parse repeated patterns…

Documentation Structure

To help you get started, we’ve organized our docs into clear sections:

- Setup & Installation — Basic instructions to install Crawl4AI via pip or Docker.
- Quick Start — A hands-on introduction…

How You Can Support

- Star & Fork: If you find Crawl4AI helpful, star the repo on GitHub or fork it to add your own features.
- File Issues: Encounter a bug or missing feature? Let us know by filing an issue, so we can…

Quick Links

- GitHub Repo
- Installation Guide
- Quick Start
- API Reference
- Changelog

Thank you for joining me on this journey. Let’s keep building an open, democratic approach to data extraction and AI…


Contributing Guide - Crawl4AI Documentation (v0.9.x)

Summary: Contribution guide for Crawl4AI explaining the branching strategy, contributor workflow, release process, and best practices for submitting pull requests.

Introduction

Welcome to the Crawl4AI project! As an open-source library for web crawling and AI integration, we value contributions from the community. This guide explains our branching strategy, how to…

Core Branches

- **main** : The stable branch containing production-ready code. It's always identical to the latest released version and is tagged for releases. Do not submit PRs directly here.

- **develop** : The…

Contributor Workflow

We encourage contributions of all kinds: bug fixes, new features, documentation improvements, tests, or even Docker enhancements. Follow these steps to contribute:

- **Fork the Repository** : Create…

git checkout develop
git checkout -b feature/your-feature-name  # Or bugfix/your-bugfix-name

Lead Maintainer's Workflow (For Reference)

- The lead maintainer (Unclecode) uses the `next` branch for isolated experimental work.

- Features from `next` are periodically merged into `develop` (via rebase and merge) to keep everything in…

Release Process (High-Level Overview)

Releases happen bi-weekly to ship improvements regularly. As a contributor, your merged changes in `develop` will be included in the next release unless specified otherwise. Here's a summary of what…

Benefits of This Approach

- **Stability** : `main` is always reliable for users.

- **Collaboration** : Fixed PR target (`develop`) makes contributing straightforward.

- **Isolation** : Experimental work in `next` doesn't…

Checklist for Contributors

Before submitting a PR:

- [ ]  Based on and targeting `develop`.

- [ ]  Tests pass (`pytest`).

- [ ]  Docs updated if needed (e.g., version refs in mkdocs.yml, Docker files).

- [ ]  No breaking…

Common Issues

- **Merge Conflicts** : Rebase your branch on latest `develop` before PR.

- **Docker Builds** : Test multi-arch (amd64/arm64) locally if changing Dockerfile.

- **Version Consistency** : Ensure any…

Communication

- Open issues for discussions or bugs.

- Join our Discord (link in README) for real-time help.

- After releases, announcements go to GitHub, Discord, and social media.

Thanks for contributing to…


Adaptive Strategies - Crawl4AI Documentation (v0.9.x)

Summary: This guide covers advanced adaptive strategies in Crawl4AI, including the three-layer scoring system, link ranking algorithm, domain-specific configurations, performance optimization, debugging,…

Overview

While the default adaptive crawling configuration works well for most use cases, understanding the underlying strategies and scoring mechanisms allows you to fine-tune the crawler for specific…

The Three-Layer Scoring System

1. Coverage Score

Coverage measures how comprehensively your knowledge base covers the query terms and related concepts.

Mathematical Foundation

Coverage(K, Q) = Σ(t ∈ Q) score(t, K) / |Q|

where score(t, K) = doc_coverage(t) × (1 + freq_boost(t))

Components

- **Document Coverage**: Percentage of documents containing the term
- **Frequency Boost**: Logarithmic bonus for term frequency
- **Query Decomposition**: Handles multi-word queries intelligently

Tuning Coverage

# For technical documentation with specific terminology
config = AdaptiveConfig(
    confidence_threshold=0.85,  # Require high coverage
    top_k_links=5              # Cast wider net
)

# For…

2. Consistency Score

Consistency evaluates whether the information across pages is coherent and non-contradictory.

How It Works

- Extracts key statements from each document
- Compares statements across documents
- Measures agreement vs. contradiction
- Returns normalized score (0-1)

Practical Impact

- **High consistency (>0.8)**: Information is reliable and coherent
- **Medium consistency (0.5-0.8)**: Some variation, but generally aligned
- **Low consistency (<0.5)**: Conflicting information,…

3. Saturation Score

Saturation detects when new pages stop providing novel information.

Detection Algorithm

# Tracks new unique terms per page
new_terms_page_1 = 50
new_terms_page_2 = 30  # 60% of first
new_terms_page_3 = 15  # 50% of second
new_terms_page_4 = 5   # 33% of third
# Saturation detected:…

Configuration

config = AdaptiveConfig(
    min_gain_threshold=0.1  # Stop if <10% new information
)

Link Ranking Algorithm

Expected Information Gain

Each uncrawled link is scored based on:

ExpectedGain(link) = Relevance × Novelty × Authority

1. Relevance Scoring

Uses BM25 algorithm on link preview text:

Factors:
- Term frequency in preview
- Inverse document frequency
- Preview length normalization

relevance = BM25(link.preview_text, query)

2. Novelty Estimation

Measures how different the link appears from already-crawled content:

Prevents crawling duplicate or highly similar pages.

novelty = 1 - max_similarity(preview, knowledge_base)

3. Authority Calculation

URL structure and domain analysis:

Factors:
- Domain reputation
- URL depth (fewer slashes = higher authority)
- Clean URL structure

authority = f(domain_rank, url_depth, url_structure)

Domain-Specific Configurations

Technical Documentation

Rationale:
- High threshold ensures comprehensive coverage
- Lower gain threshold captures edge cases
- Moderate link following for depth

tech_doc_config = AdaptiveConfig(
    confidence_threshold=0.85,
    max_pages=30,
    top_k_links=3,
    min_gain_threshold=0.05  # Keep crawling for small gains
)

News & Articles

Rationale:
- Lower threshold (articles often repeat information)
- Higher gain threshold (avoid duplicate stories)
- More links per page (explore different perspectives)

news_config = AdaptiveConfig(
    confidence_threshold=0.6,
    max_pages=10,
    top_k_links=5,
    min_gain_threshold=0.15  # Stop quickly on repetition
)

E-commerce

Rationale:
- Balanced threshold for product variations
- Focused link following (avoid infinite products)
- Standard gain threshold

ecommerce_config = AdaptiveConfig(
    confidence_threshold=0.7,
    max_pages=20,
    top_k_links=2,
    min_gain_threshold=0.1
)

Research & Academic

Rationale:
- Very high threshold for completeness
- Many pages allowed for thorough research
- Very low gain threshold to capture references

research_config = AdaptiveConfig(
    confidence_threshold=0.9,
    max_pages=50,
    top_k_links=4,
    min_gain_threshold=0.02  # Very low - capture citations
)

Performance Optimization

Memory Management

# For large crawls, use streaming
config = AdaptiveConfig(
    max_pages=100,
    save_state=True,
    state_path="large_crawl.json"
)

# Periodically clean state
if len(state.knowledge_base) >…

Parallel Processing

# Use multiple start points
start_urls = [
    "https://docs.example.com/intro",
    "https://docs.example.com/api",
    "https://docs.example.com/guides"
]

# Crawl in parallel
tasks = […

Debugging & Analysis

Enable Debugging

import logging

logging.basicConfig(level=logging.DEBUG)
adaptive = AdaptiveCrawler(crawler, config, verbose=True)

Analyze Crawl Patterns

# After crawling
state = await adaptive.digest(start_url, query)

# Analyze link selection
print("Link selection order:")
for i, url in enumerate(state.crawl_order):
    print(f"{i+1}. {url}")

#…

Export for Analysis

# Export detailed metrics
import json

metrics = {
    "query": query,
    "total_pages": len(state.crawled_urls),
    "confidence": adaptive.confidence,
    "coverage_stats":…

Custom Strategies

Implementing a Custom Strategy

from crawl4ai.adaptive_crawler import CrawlStrategy

class DomainSpecificStrategy(CrawlStrategy):
    def calculate_coverage(self, state: CrawlState) -> float:
        # Custom coverage calculation…

Combining Strategies

class HybridStrategy(CrawlStrategy):
    def __init__(self):
        self.strategies = [
            TechnicalDocStrategy(),
            SemanticSimilarityStrategy(),…

Best Practices

1. Start Conservative

Begin with default settings and adjust based on results:

# Start with defaults
result = await adaptive.digest(url, query)

# Analyze and adjust
if adaptive.confidence < 0.7:
    config.max_pages += 10
    config.confidence_threshold -= 0.1

2. Monitor Resource Usage

import psutil

# Check memory before large crawls
memory_percent = psutil.virtual_memory().percent
if memory_percent > 80:
    config.max_pages = min(config.max_pages, 20)

3. Use Domain Knowledge

# For API documentation
if "api" in start_url:
    config.top_k_links = 2  # APIs have clear structure

# For blogs
if "blog" in start_url:
    config.min_gain_threshold = 0.2  # Avoid similar posts

4. Validate Results

# Always validate the knowledge base
relevant_content = adaptive.get_relevant_content(top_k=10)

# Check coverage
query_terms = set(query.lower().split())
covered_terms = set()

for doc in…

Next Steps

Explore [Custom Strategy Implementation](../tutorials/custom-adaptive-strategies.md)
Learn about [Knowledge Base Management](../tutorials/knowledge-base-management.md)
See [Performance…


Overview of Some Important Advanced Features

Summary: A guide to advanced Crawl4AI features including proxy usage, PDF/screenshot capture, SSL certificates, custom headers, session persistence, robots.txt compliance, and anti-bot techniques.

Crawl4AI offers multiple power-user features that go beyond simple crawling. This tutorial covers:

1. **Proxy Usage**

2. **Capturing PDFs & Screenshots**

3. **Handling SSL Certificates**

4.…

1. Proxy Usage

If you need to route your crawl traffic through a proxy—whether for IP rotation, geo-testing, or privacy—Crawl4AI supports it via `BrowserConfig.proxy_config`.

**Key Points**

- **`proxy_config`**…

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    browser_cfg = BrowserConfig(
        proxy_config={
            "server":…

2. Capturing PDFs & Screenshots

Sometimes you need a visual record of a page or a PDF “printout.” Crawl4AI can do both in one pass:

**Why PDF + Screenshot?**

- Large or complex pages can be slow or error-prone with “traditional”…

import os, asyncio
from base64 import b64decode
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def main():
    run_config = CrawlerRunConfig(…

3. Handling SSL Certificates

If you need to verify or export a site’s SSL certificate—for compliance, debugging, or data analysis—Crawl4AI can fetch it during the crawl:

**Key Points**

- **`fetch_ssl_certificate=True`**…

import asyncio, os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    tmp_dir = os.path.join(os.getcwd(), "tmp")
    os.makedirs(tmp_dir, exist_ok=True)…

4. Custom Headers

Sometimes you need to set custom headers (e.g., language preferences, authentication tokens, or specialized user-agent strings). You can do this in multiple ways:

**Notes**

- Some sites may react…

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    # Option 1: Set headers at the crawler strategy level
    crawler1 = AsyncWebCrawler(
        # The underlying strategy can…

5. Session Persistence & Local Storage

Crawl4AI can preserve cookies and localStorage so you can continue where you left off—ideal for logging into sites or skipping repeated auth flows.

5.1 `storage_state`

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    storage_dict = {
        "cookies": [
            {
                "name": "session",
                "value": "abcd1234",…

5.2 Exporting & Reusing State

You can sign in once, export the browser context, and reuse it later—without re-entering credentials.

- **`await context.storage_state(path="my_storage.json")`**: Exports cookies, localStorage, etc.…

6. Robots.txt Compliance

Crawl4AI supports respecting robots.txt rules with efficient caching:

**Key Points**

- Robots.txt files are cached locally for efficiency

- Cache is stored in…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Enable robots.txt checking in config
    config = CrawlerRunConfig(
        check_robots_txt=True  #…

Putting It All Together

Here’s a snippet that combines multiple “advanced” features (proxy, PDF, screenshot, SSL, custom headers, and session reuse) into one run. Normally, you’d tailor each setting to your project’s needs.

import os, asyncio
from base64 import b64decode
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    # 1. Browser config with proxy + headless…

7. Anti-Bot Features (Stealth Mode & Undetected Browser)

Crawl4AI provides two powerful features to bypass bot detection:

7.1 Stealth Mode

Stealth mode uses playwright-stealth to modify browser fingerprints and behaviors. Enable it with a simple flag:

**When to use**: Sites with basic bot detection (checking navigator.webdriver,…

browser_config = BrowserConfig(
    enable_stealth=True,  # Activates stealth mode
    headless=False
)

7.2 Undetected Browser

For advanced bot detection, use the undetected browser adapter:

**When to use**: Sites with sophisticated bot detection (Cloudflare, DataDome, etc.)

from crawl4ai import UndetectedAdapter
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

# Create undetected adapter
adapter = UndetectedAdapter()
strategy =…

7.3 Combining Both

For maximum evasion, combine stealth mode with undetected browser:

browser_config = BrowserConfig(
    enable_stealth=True,  # Enable stealth
    headless=False
)

adapter = UndetectedAdapter()  # Use undetected browser

Choosing the Right Approach

| Detection Level | Recommended Approach |
| --- | --- |
| No protection | Regular browser |
| Basic checks | Regular + Stealth mode |
| Advanced protection | Undetected browser |
| Maximum evasion |…

Conclusion & Next Steps

You've now explored several **advanced** features:

- **Proxy Usage**

- **PDF & Screenshot** capturing for large or critical pages

- **SSL Certificate** retrieval & exporting

- **Custom Headers**…


Anti-Bot Detection & Fallback

Summary: Explains how Crawl4AI detects anti-bot blocks and describes the layered retry/fallback system using proxies, retries, and custom fetch functions to retrieve content from protected sites.

How Detection Works

After each crawl attempt, Crawl4AI inspects the HTTP status code and HTML content for known anti-bot signals:

- **HTTP 403/429** with short or empty response bodies
- **Challenge pages** —…

Configuration Options

All anti-bot retry options live on `CrawlerRunConfig`:

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `proxy_config` | `ProxyConfig`, `list[ProxyConfig]`, or `None` |…

Escalation Chain

Each retry round tries every proxy in `proxy_config` in order. If all rounds are exhausted and the page is still blocked, the fallback fetch function is called as a last resort.

Worst-case attempts…

For each round (1 + max_retries rounds):
    1. Try proxy_config[0] (or direct if proxy_config is None)
    2. If blocked → try proxy_config[1]
    3. If blocked → try proxy_config[2]
    4. ...…

Crawl Stats

Every crawl result includes a `crawl_stats` dict with detailed attempt tracking:

result.crawl_stats = {
    "attempts": 3,                    # total browser attempts made
    "retries": 1,                     # retry rounds used (0 = succeeded first round)
    "proxies_used": […

Usage Examples

Simple Retry (No Proxy)

Retry the crawl up to 3 times when blocking is detected. Useful when blocks are intermittent or IP-based.

from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
    result = await…

Single Proxy

Pass a single `ProxyConfig` — it's used on every attempt. Same behavior as always.

from crawl4ai.async_configs import ProxyConfig

config = CrawlerRunConfig(
    max_retries=2,
    proxy_config=ProxyConfig(
        server="http://proxy.example.com:8080",
        username="user",…

Direct-First, Then Proxies

Try without a proxy first, then escalate to proxies if blocked. Use `ProxyConfig.DIRECT` (or the string `"direct"`) in the list to represent a no-proxy attempt.

With this setup, each round tries…

config = CrawlerRunConfig(
    max_retries=1,
    proxy_config=[
        ProxyConfig.DIRECT,  # Try without proxy first
        ProxyConfig(…

Proxy List (Escalation)

Pass a list of proxies. They're tried in order — first one that works wins. Within each retry round, the entire list is tried again.

With this setup, each round tries the datacenter proxy first,…

config = CrawlerRunConfig(
    max_retries=1,
    proxy_config=[
        ProxyConfig(
            server="http://datacenter-proxy.example.com:8080",
            username="user",…

Fallback Fetch Function

When all browser-based attempts fail, call a custom async function as a last resort. This function receives the URL and must return raw HTML as a string. The returned HTML is processed through the…

import aiohttp

async def my_scraping_api(url: str) -> str:
    """Fetch HTML via an external scraping API."""
    async with aiohttp.ClientSession() as session:
        async with session.get(…

Full Escalation (All Features Combined)

This example combines every layer: stealth mode, a list of proxies tried in order, retries, and a final fetch function.

**What happens step by step:**

| Round | Attempt | What runs |
| --- | --- |…

import aiohttp
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, ProxyConfig

# Last-resort: fetch HTML via an external service
async def…

Tips

- Start with `max_retries=0` and a `fallback_fetch_function` if you just want a safety net without burning time on retries.
- Order proxies cheapest-first — datacenter proxies before residential,…

See Also

- [Proxy & Security](../proxy-security/) — Proxy setup, authentication, and rotation
- [Undetected Browser](../undetected-browser/) — Stealth mode and browser fingerprint evasion
- [Session…


Crawl Dispatcher

Summary: This page announces the upcoming Crawl Dispatcher module in Crawl4AI, a feature for handling thousands of crawling tasks simultaneously with efficient resource management and real-time monitoring.

Crawl Dispatcher

We’re excited to announce a  **Crawl Dispatcher**  module that can handle  **thousands**  of crawling tasks simultaneously. By efficiently managing system resources (memory, CPU, network), this…


Download Handling in Crawl4AI

Summary: This guide explains how to use Crawl4AI to handle file downloads during crawling, including enabling downloads, specifying download locations, triggering downloads, and accessing downloaded files.

Overview

This guide explains how to use Crawl4AI to handle file downloads during crawling. You'll learn how to trigger downloads, specify download locations, and access downloaded files.

Enabling Downloads

To enable downloads, set the `accept_downloads` parameter in the `BrowserConfig` object and pass it to the crawler.

from crawl4ai.async_configs import BrowserConfig, AsyncWebCrawler

async def main():
    config = BrowserConfig(accept_downloads=True)  # Enable downloads globally
    async with…

Specifying Download Location

Specify the download directory using the `downloads_path` attribute in the `BrowserConfig` object. If not provided, Crawl4AI defaults to creating a "downloads" directory inside the `.crawl4ai` folder…

from crawl4ai.async_configs import BrowserConfig
import os

downloads_path = os.path.join(os.getcwd(), "my_downloads")  # Custom download path
os.makedirs(downloads_path, exist_ok=True)

config =…

Triggering Downloads

Downloads are typically triggered by user interactions on a web page, such as clicking a download button. Use `js_code` in `CrawlerRunConfig` to simulate these actions and `wait_for` to allow…

from crawl4ai.async_configs import CrawlerRunConfig

config = CrawlerRunConfig(
    js_code="""
        const downloadLink = document.querySelector('a[href$=".exe"]');
        if (downloadLink) {…

Accessing Downloaded Files

The `downloaded_files` attribute of the `CrawlResult` object contains paths to downloaded files.

if result.downloaded_files:
    print("Downloaded files:")
    for file_path in result.downloaded_files:
        print(f"- {file_path}")
        file_size = os.path.getsize(file_path)…

Example: Downloading Multiple Files

from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
import os
from pathlib import Path

async def download_multiple_files(url: str, download_path: str):
    config =…

Important Considerations

- **Browser Context:**  Downloads are managed within the browser context. Ensure `js_code` correctly targets the download triggers on the webpage.

- **Timing:**  Use `wait_for` in `CrawlerRunConfig`…


Hooks & Auth in AsyncWebCrawler

Summary: This guide explains how to use hooks in AsyncWebCrawler to customize the crawling pipeline at specific stages, including authentication, route blocking, and pre/post-processing, with a detailed…

Introduction

##### Hooks & Auth in AsyncWebCrawler

Crawl4AI’s  **hooks**  let you customize the crawler at specific points in the pipeline:

1.  **`on_browser_created`**  – After browser creation.

2.…

Example: Using Hooks in AsyncWebCrawler

import asyncio
import json
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from playwright.async_api import Page, BrowserContext

async def main():
    print("🔗 Hooks…

Hook Lifecycle Summary

1.  **`on_browser_created`** :

   - Browser is up, but  **no**  pages or contexts yet.

   - Light setup only—don’t try to open or close pages here (that belongs in…

When to Handle Authentication

**Recommended** : Use  **`on_page_context_created`**  if you need to:

- Navigate to a login page or fill forms

- Set cookies or localStorage tokens

- Block resource routes to avoid ads

This…

Additional Considerations

- **Session Management** : If you want multiple `arun()` calls to reuse a single session, pass `session_id=` in your `CrawlerRunConfig`. Hooks remain the same.

- **Performance** : Hooks can slow…

Conclusion

Hooks provide  **fine-grained**  control over:

- **Browser**  creation (light tasks only)

- **Page**  and  **context**  creation (auth, route blocking)

- **Navigation**  phases

- **Final…


Identity Based Crawling - Crawl4AI Documentation (v0.9.x)

Summary: This guide explains how to use Crawl4AI's Managed Browsers and BrowserProfiler to preserve a user's authentic digital identity while crawling, including persistent profiles, Magic Mode fallback, and…

Preserve Your Identity with Crawl4AI

Crawl4AI empowers you to navigate and interact with the web using your **authentic digital identity**, ensuring you’re recognized as a human and not mistaken for a bot. This tutorial covers:

1.…

1. Managed Browsers: Your Digital Identity Solution

**Managed Browsers** let developers create and use **persistent browser profiles**. These profiles store local storage, cookies, and other session data, letting you browse as your **real self**…

Key Benefits

- **Authentic Browsing Experience**: Retain session data and browser fingerprints as though you’re a normal user.
- **Effortless Configuration**: Once you log in or solve CAPTCHAs in your chosen data…

Creating a User Data Directory (Command-Line Approach via Playwright)

If you installed Crawl4AI (which installs Playwright under the hood), you already have a Playwright-managed Chromium on your system. Follow these steps to launch that **Chromium** from your command…

# Linux example
~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome \
    --user-data-dir=/home/<you>/my_chrome_profile

# macOS example (Playwright’s internal binary)
~/Library/Caches/ms-playwright/chromium-1234/chrome-mac/Chromium.app/Contents/MacOS/Chromium \
    --user-data-dir=/Users/<you>/my_chrome_profile

# Windows example (PowerShell/cmd)
"C:\Users\<you>\AppData\Local\ms-playwright\chromium-1234\chrome-win\chrome.exe" ^
    --user-data-dir="C:\Users\<you>\my_chrome_profile"

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

browser_config = BrowserConfig(
    headless=True,
    use_managed_browser=True,…

Creating a Profile Using the Crawl4AI CLI (Easiest)

If you prefer a guided, interactive setup, use the built-in CLI to create and manage persistent browser profiles.

1. Launch the profile manager (`crwl profiles`).

2. Choose "Create new profile" and…

from crawl4ai import AsyncWebCrawler, BrowserConfig

profile_path = "/home/<you>/.crawl4ai/profiles/test_profile_1"

browser_config = BrowserConfig(
    headless=True,
    use_managed_browser=True,…

3. Using Managed Browsers in Crawl4AI

Once you have a data directory with your session data, pass it to **`BrowserConfig`**:

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    # 1) Reference your persistent data directory
    browser_config = BrowserConfig(…

Workflow

1. **Login** externally (via CLI or your normal Chrome with `--user-data-dir=...`).
2. **Close** that browser.
3. **Use** the same folder in `user_data_dir=` in Crawl4AI.
4. **Crawl** – The site sees…

4. Magic Mode: Simplified Automation

If you **don’t** need a persistent profile or identity-based approach, **Magic Mode** offers a quick way to simulate human-like browsing without storing long-term data.

**Magic Mode**:

- Simulates…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://example.com",
        config=CrawlerRunConfig(…

5. Comparing Managed Browsers vs. Magic Mode

| Feature | **Managed Browsers** | **Magic Mode** |
| --- | --- | --- |
| **Session Persistence** | Full localStorage/cookies retained in user_data_dir | No persistent data (fresh each run) |
|…

6. Using the BrowserProfiler Class

Crawl4AI provides a dedicated `BrowserProfiler` class for managing browser profiles, making it easy to create, list, and delete profiles for identity-based browsing.

Creating and Managing Profiles with BrowserProfiler

The `BrowserProfiler` class offers a comprehensive API for browser profile management:

**How profile creation works:**
1. A browser window opens for you to interact with
2. You log in to websites,…

import asyncio
from crawl4ai import BrowserProfiler

async def manage_profiles():
    # Create a profiler instance
    profiler = BrowserProfiler()

    # Create a profile interactively - opens a…

Interactive Profile Management

The `BrowserProfiler` also offers an interactive management console that guides you through profile creation, listing, and deletion:

import asyncio
from crawl4ai import BrowserProfiler, AsyncWebCrawler, BrowserConfig

# Define a function to use a profile for crawling
async def crawl_with_profile(profile_path, url):…

Legacy Methods

For backward compatibility, the previous methods on `ManagedBrowser` are still available, but they delegate to the new `BrowserProfiler` class:

from crawl4ai.browser_manager import ManagedBrowser

# These methods still work but use BrowserProfiler internally
profiles = ManagedBrowser.list_profiles()

Complete Example

See the full example in `docs/examples/identity_based_browsing.py` for a complete demonstration of creating and using profiles for authenticated browsing using the new `BrowserProfiler` class.

7. Locale, Timezone, and Geolocation Control

In addition to using persistent profiles, Crawl4AI supports customizing your browser's locale, timezone, and geolocation settings. These features enhance your identity-based browsing experience by…

Setting Locale and Timezone

You can set the browser's locale and timezone through `CrawlerRunConfig`:

**How it works:**
- `locale` affects language preferences, date formats, number formats, etc.
- `timezone_id` affects…

Configuring Geolocation

Control the GPS coordinates reported by the browser's geolocation API:

**Important notes:**
- When `geolocation` is specified, the browser is automatically granted permission to access location
-…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, GeolocationConfig

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://maps.google.com",  # Or any…

Combining with Managed Browsers

These settings work perfectly with managed browsers for a complete identity solution:

Combining persistent profiles with precise geolocation and region settings gives you complete control over your…

from crawl4ai import (
    AsyncWebCrawler, BrowserConfig, CrawlerRunConfig,
    GeolocationConfig
)

browser_config = BrowserConfig(
    use_managed_browser=True,…

8. Summary

- **Create** your user-data directory either:
  - By launching Chrome/Chromium externally with `--user-data-dir=/some/path`
  - Or by using the built-in `BrowserProfiler.create_profile()` method
  -…


Lazy Loading - Crawl4AI Documentation (v0.9.x)

Summary: This guide explains how to handle lazy-loaded images in Crawl4AI by using wait_for_images, scan_full_page, and scroll_delay settings, and how to combine these with media filters and domain exclusions.

Handling Lazy-Loaded Images

Many websites now load images **lazily** as you scroll. If you need to ensure they appear in your final crawl (and in `result.media`), consider:

1. **`wait_for_images=True`** – Wait for images to…

Example: Ensuring Lazy Images Appear

**Explanation**:

- **`wait_for_images=True`**

  The crawler tries to ensure images have finished loading before finalizing the HTML.

- **`scan_full_page=True`**

  Tells the crawler to attempt…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig
from crawl4ai.async_configs import CacheMode

async def main():
    config = CrawlerRunConfig(
        # Force the…

Combining with Other Link & Media Filters

You can still combine **lazy-load** logic with the usual **exclude_external_images**, **exclude_domains**, or link filtration:

This approach ensures you see **all** images from the main domain while…

config = CrawlerRunConfig(
    wait_for_images=True,
    scan_full_page=True,
    scroll_delay=0.5,

    # Filter out external images if you only want local ones
    exclude_external_images=True,…

Tips & Troubleshooting

1. **Long Pages**

   - Setting `scan_full_page=True` on extremely long or infinite-scroll pages can be resource-intensive.

   - Consider using [hooks](../../core/page-interaction/) or specialized…


Advanced Multi-URL Crawling with Dispatchers

Summary: This page covers advanced multi-URL crawling using dispatchers in Crawl4AI, including RateLimiter, CrawlerMonitor, MemoryAdaptiveDispatcher, and SemaphoreDispatcher, with usage examples and…

1. Introduction

> **Heads Up** : Crawl4AI supports advanced dispatchers for **parallel** or **throttled** crawling, providing dynamic rate limiting and memory usage checks. The built-in `arun_many()` function uses…

2. Core Components

2.1 Rate Limiter

Here’s the revised and simplified explanation of the **RateLimiter** , focusing on constructor parameters and adhering to your markdown style and mkDocs guidelines.

class RateLimiter:
    def __init__(
        # Random delay range between requests
        base_delay: Tuple[float, float] = (1.0, 3.0),

        # Maximum backoff delay
        max_delay: float =…

RateLimiter Constructor Parameters

The **RateLimiter** is a utility that helps manage the pace of requests to avoid overloading servers or getting blocked due to rate limits. It operates internally to delay requests and handle retries…

from crawl4ai import RateLimiter

# Create a RateLimiter with custom settings
rate_limiter = RateLimiter(
    base_delay=(2.0, 4.0),  # Random delay between 2-4 seconds
    max_delay=30.0,         #…

2.2 Crawler Monitor

The CrawlerMonitor provides real-time visibility into crawling operations:

 **Display Modes** :

- **DETAILED** : Shows individual task status, memory usage, and timing
- **AGGREGATED** : Displays…

from crawl4ai import CrawlerMonitor, DisplayMode
monitor = CrawlerMonitor(
    # Maximum rows in live display
    max_visible_rows=15,

    # DETAILED or AGGREGATED view…

3. Available Dispatchers

3.1 MemoryAdaptiveDispatcher (Default)

Automatically manages concurrency based on system memory usage:

 **Constructor Parameters:**

1. **`memory_threshold_percent`** (`float`, default: `90.0`)

   Specifies the memory usage threshold…

from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=90.0,  # Pause if memory exceeds this
    check_interval=1.0,…

3.2 SemaphoreDispatcher

Provides simple concurrency control with a fixed limit:

 **Constructor Parameters:**

1. **`max_session_permit`** (`int`, default: `20`)

   The maximum number of concurrent crawling tasks allowed,…

from crawl4ai.async_dispatcher import SemaphoreDispatcher

dispatcher = SemaphoreDispatcher(
    max_session_permit=20,         # Maximum concurrent tasks
    rate_limiter=RateLimiter(      #…

4. Usage Examples

4.1 Batch Processing (Default)

**Review:**

- **Purpose:** Executes a batch crawl with all URLs processed together after crawling is complete.
- **Dispatcher:** Uses `MemoryAdaptiveDispatcher` to manage concurrency and system…

async def crawl_batch():
    browser_config = BrowserConfig(headless=True, verbose=False)
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        stream=False  # Default: get…

4.2 Streaming Mode

**Review:**

- **Purpose:** Enables streaming to process results as soon as they’re available.
- **Dispatcher:** Uses `MemoryAdaptiveDispatcher` for concurrency and memory management.
- **Stream:**…

async def crawl_streaming():
    browser_config = BrowserConfig(headless=True, verbose=False)
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        stream=True  # Enable…

4.3 Semaphore-based Crawling

**Review:**

- **Purpose:** Uses `SemaphoreDispatcher` to limit concurrency with a fixed number of slots.
- **Dispatcher:** Configured with a semaphore to control parallel crawling tasks.
- **Rate…

async def crawl_with_semaphore(urls):
    browser_config = BrowserConfig(headless=True, verbose=False)
    run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

    dispatcher =…

4.4 Robots.txt Consideration

**Review:**

- **Purpose:** Ensures compliance with `robots.txt` rules for ethical and legal web crawling.
- **Configuration:** Set `check_robots_txt=True` to validate each URL against `robots.txt`…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    urls = [
        "https://example1.com",
        "https://example2.com",…

5. Dispatch Results

Each crawl result includes dispatch information:

Access via `result.dispatch_result`:

@dataclass
class DispatchResult:
    task_id: str
    memory_usage: float
    peak_memory: float
    start_time: datetime
    end_time: datetime
    error_message: str = ""
Copy

for result in results:
    if result.success:
        dr = result.dispatch_result
        print(f"URL: {result.url}")
        print(f"Memory: {dr.memory_usage:.1f}MB")
        print(f"Duration:…

6. URL-Specific Configurations

When crawling diverse content types, you often need different configurations for different URLs. For example:
- PDFs need specialized extraction
- Blog pages benefit from content filtering
- Dynamic…

6.1 Basic URL Pattern Matching

**Important** : A `CrawlerRunConfig` without `url_matcher` (or with `url_matcher=None`) matches ALL URLs. This makes it perfect as a default/fallback configuration.

The `url_matcher` parameter…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
from crawl4ai.processors.pdf import PDFContentScrapingStrategy
from crawl4ai.extraction_strategy import…

6.2 Advanced Pattern Matching

Glob Patterns (Strings)

# Simple patterns
"*.pdf"                    # Any PDF file
"*/api/*"                  # Any URL with /api/ in path
"https://*.example.com/*"  # Subdomain matching
"*://example.com/blog/*"   # Any…

Custom Functions

# Complex logic with lambdas
lambda url: url.startswith('https://') and 'secure' in url
lambda url: len(url) > 50 and url.count('/') > 5
lambda url: any(domain in url for domain in ['api.', 'data.',…

Mixed Lists with AND/OR Logic

# Combine multiple conditions
CrawlerRunConfig(
    url_matcher=[
        "https://*",                        # Must be HTTPS
        lambda url: 'internal' in url,      # Must contain 'internal'…

6.3 Practical Example: News Site Crawler

async def crawl_news_site():
    dispatcher = MemoryAdaptiveDispatcher(
        memory_threshold_percent=70.0,
        rate_limiter=RateLimiter(base_delay=(1.0, 2.0))
    )

    configs = [
        #…

6.4 Best Practices

- **Order Matters** : Configs are evaluated in order - put specific patterns before general ones
- **Default Config Behavior** :
  - A config without `url_matcher` matches ALL URLs
  - Always include…

config = CrawlerRunConfig(url_matcher="*.pdf")
print(config.is_match("https://example.com/doc.pdf"))  # True

default_config = CrawlerRunConfig()  # No…

7. Summary

1. **Two Dispatcher Types** :

   - MemoryAdaptiveDispatcher (default): Dynamic concurrency based on memory
   - SemaphoreDispatcher: Fixed concurrency limit

2. **Optional Components** :

   -…

The range for a random delay (in seconds) between consecutive requests to the same domain. A random delay is chosen between base_delay[0] and base_delay[1] for each request.

The maximum allowable delay when rate-limiting errors occur. When servers return rate-limit responses (e.g., 429 or 503), the delay increases exponentially with jitter, capped at this value.

The maximum number of retries for a request if rate-limiting errors occur. After encountering a rate-limit response, the RateLimiter retries the request up to this number of times.

A list of HTTP status codes that trigger the rate-limiting logic. These status codes indicate the server is overwhelmed or actively limiting requests.

Specifies the memory usage threshold (as a percentage). If system memory usage exceeds this value, the dispatcher pauses crawling to prevent system overload.

The interval (in seconds) at which the dispatcher checks system memory usage.

The maximum number of concurrent crawling tasks allowed. This ensures resource limits are respected while maintaining concurrency.

Optional timeout (in seconds). If memory usage exceeds memory_threshold_percent for longer than this duration, a MemoryError is raised.

Optional rate-limiting logic to avoid server-side blocking (e.g., for handling 429 or 503 errors).

monitor

Optional monitoring for real-time task tracking and performance insights.

The maximum number of concurrent crawling tasks allowed, irrespective of semaphore slots.

Optional rate-limiting logic to avoid overwhelming servers.

Optional monitoring for tracking task progress and resource usage.

Maximum rows in live display.

DETAILED or AGGREGATED view.


Network Requests & Console Message Capturing

Summary: This page explains how to capture network requests and browser console messages during a crawl using Crawl4AI, including configuration, example usage, data structures, benefits, and use cases.

Configuration

To enable network and console capturing, use these configuration options:

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

# Enable both network request capture and console message capture
config = CrawlerRunConfig(
    capture_network_requests=True,  # Capture all…

Example Usage

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Enable both network request capture and console message capture
    config =…

Captured Data Structure

The `result.network_requests` contains a list of dictionaries, each representing a network event with these common fields:

| Field | Description |
| --- | --- |
| `event_type` | Type of event:…

Network Requests

Request Event Fields

{
  "event_type": "request",
  "url": "https://example.com/api/data.json",
  "method": "GET",
  "headers": {"User-Agent": "...", "Accept": "..."},
  "post_data": "key=value&otherkey=value",…

Response Event Fields

{
  "event_type": "response",
  "url": "https://example.com/api/data.json",
  "status": 200,
  "status_text": "OK",
  "headers": {"Content-Type": "application/json", "Cache-Control": "..."},…

Failed Request Event Fields

Console Messages

The `result.console_messages` contains a list of dictionaries, each representing a console message with these common fields:

| Field | Description |
| --- | --- |
| `type` | Message type: `"log"`,…

Console Message Example

Uncaught TypeError: Cannot read property 'length' of undefined

Key Benefits

**Full Request Visibility** : Capture all network activity including:
- Requests (URLs, methods, headers, post data)
- Responses (status codes, headers, timing)
- Failed requests (with error…

Use Cases

**API Discovery** : Identify hidden endpoints and data flows in single-page applications

**Debugging** : Track down JavaScript errors affecting page functionality

**Security Auditing** : Detect…


PDF Processing Strategies

Summary: This page describes the PDF processing strategies in Crawl4AI, including PDFCrawlerStrategy and PDFContentScrapingStrategy, which enable crawling and extracting content from PDF files.

Overview

`PDFCrawlerStrategy` is an implementation of `AsyncCrawlerStrategy` designed specifically for PDF documents. Instead of interpreting the input URL as an HTML webpage, this strategy treats it as a…

When to Use

Use `PDFCrawlerStrategy` when you need to:
- Process PDF files using the `AsyncWebCrawler`.
- Handle PDFs from both web URLs (e.g., `https://example.com/document.pdf`) and local file paths (e.g.,…

Key Methods and Their Behavior

- **`__init__(self, logger: AsyncLogger = None)`** :

- Initializes the strategy.
- `logger`: An optional `AsyncLogger` instance (from `crawl4ai.async_logger`) for logging purposes.

- **`async…

Example Usage

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy

async def main():
    # Initialize the PDF…

Pros and Cons

**Pros:**
-   Enables `AsyncWebCrawler` to handle PDF sources directly using familiar `arun` calls.
-   Provides a consistent interface for specifying PDF sources (URLs or local paths).
-…

Key Configuration Attributes

When initializing `PDFContentScrapingStrategy`, you can configure its behavior using the following attributes:
-    **`extract_images: bool = False`** : If `True`, the strategy will attempt to…

logger

An optional AsyncLogger instance for logging purposes.

url

The URL pointing to a PDF file.

If True (and extract_images is also True), extracted images will be saved to disk in the image_save_dir.

If True, the strategy will attempt to extract images from the PDF.

Specifies the directory where extracted images should be saved if save_images_locally is True.

Defines how many PDF pages are processed in a single batch.

logger

An optional AsyncLogger instance for logging.

url

The path or URL to the PDF file.

html

Typically an empty string when used with PDFCrawlerStrategy, as the content is a PDF, not HTML.

url

The path or URL to the PDF file.

The URL or path to the PDF file.


Proxy & Security

Summary: This guide covers proxy configuration and security features in Crawl4AI, including SSL certificate analysis and proxy rotation strategies.

Understanding Proxy Configuration

Crawl4AI recommends configuring proxies per request through `CrawlerRunConfig.proxy_config`. This gives you precise control, enables rotation strategies, and keeps examples simple enough to copy,…

Basic Proxy Setup

Configure proxies that apply to each crawl operation:

Why request-level?

`CrawlerRunConfig.proxy_config` keeps each request self-contained, so swapping proxies or rotation strategies is just a…

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig

run_config = CrawlerRunConfig(proxy_config=ProxyConfig(server="http://proxy.example.com:8080"))
#…

Supported Proxy Formats

The `ProxyConfig.from_string()` method supports multiple formats:

from crawl4ai import ProxyConfig

# HTTP proxy with authentication
proxy1 = ProxyConfig.from_string("http://user:pass@192.168.1.1:8080")

# HTTPS proxy
proxy2 =…

Authenticated Proxies

For proxies requiring authentication:

import asyncio
from crawl4ai import AsyncWebCrawler,BrowserConfig, CrawlerRunConfig, ProxyConfig

run_config = CrawlerRunConfig(
    proxy_config=ProxyConfig(…

Environment Variable Configuration

Load proxies from environment variables for easy configuration:

import os
from crawl4ai import ProxyConfig, CrawlerRunConfig

# Set environment variable
os.environ["PROXIES"] = "ip1:port1:user1:pass1,ip2:port2:user2:pass2,ip3:port3"

# Load all proxies
proxies =…

Rotating Proxies

Crawl4AI supports automatic proxy rotation to distribute requests across multiple proxy servers. Rotation is applied per request using a rotation strategy on `CrawlerRunConfig`.

Proxy Rotation (recommended)

import asyncio
import re
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy

async def main():…

SSL Certificate Analysis

Combine proxy usage with SSL certificate inspection for enhanced security analysis. SSL certificate fetching is configured per request via `CrawlerRunConfig`.

Per-Request SSL Certificate Analysis

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

run_config = CrawlerRunConfig(
    proxy_config={
        "server": "http://proxy.example.com:8080",…

Security Best Practices

1. Proxy Rotation for Anonymity

from crawl4ai import CrawlerRunConfig, ProxyConfig
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy

# Use multiple proxies to avoid IP blocking
proxies =…

2. SSL Certificate Verification

from crawl4ai import CrawlerRunConfig

# Always verify SSL certificates when possible
# Per-request (affects specific requests)
run_config = CrawlerRunConfig(fetch_ssl_certificate=True)

3. Environment Variable Security

# Use environment variables for sensitive proxy credentials
# Avoid hardcoding usernames/passwords in code
export PROXIES="ip1:port1:user1:pass1,ip2:port2:user2:pass2"

4. SOCKS5 for Enhanced Security

from crawl4ai import CrawlerRunConfig

# Prefer SOCKS5 proxies for better protocol support
run_config = CrawlerRunConfig(proxy_config="socks5://proxy.example.com:1080")

Migration from Deprecated `proxy` Parameter

The legacy `proxy` argument on `BrowserConfig` is deprecated. Configure proxies through `CrawlerRunConfig.proxy_config` so each request fully describes its network settings.

# Old (deprecated) approach
# from crawl4ai import BrowserConfig
# browser_config = BrowserConfig(proxy_config="http://proxy.example.com:8080")

# New (preferred) approach
from crawl4ai import…

Safe Logging of Proxies

from crawl4ai import ProxyConfig

def safe_proxy_repr(proxy: ProxyConfig):
    if getattr(proxy, "username", None):
        return f"{proxy.server} (auth: ****)"
    return proxy.server

Troubleshooting

Common Issues

Proxy connection failed

- Verify the proxy server is reachable from your network.
- Double-check authentication credentials.
- Ensure the protocol matches (`http`, `https`, or `socks5`).

SSL…

See Also

[Anti-Bot Detection & Fallback](../anti-bot-and-fallback/) — Automatic retry with proxy escalation and fallback functions when anti-bot blocking is detected


Session Management - Crawl4AI Documentation (v0.9.x)

Summary: This page explains how to use session management in Crawl4AI to maintain state across multiple requests, enabling sequential crawling, dynamic content handling, and advanced techniques like custom…

Overview

Session management in Crawl4AI is a powerful feature that allows you to maintain state across multiple requests, making it particularly suitable for handling complex multi-step crawling tasks. It…

Basic Session Usage

Use `BrowserConfig` and `CrawlerRunConfig` to maintain state with a `session_id`:

from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async with AsyncWebCrawler() as crawler:
    session_id = "my_session"

    # Define configurations
    config1 =…

Dynamic Content with Sessions

Here's an example of crawling GitHub commits across multiple pages while preserving session state:

from crawl4ai.async_configs import CrawlerRunConfig
from crawl4ai import JsonCssExtractionStrategy
from crawl4ai.cache_context import CacheMode

async def crawl_dynamic_content():
    url =…

Example 1: Basic Session-Based Crawling

A simple example using session-based crawling:

This example shows:
1. Reusing the same `session_id` across multiple requests.
2. Executing JavaScript to load more content dynamically.
3. Properly…

import asyncio
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
from crawl4ai.cache_context import CacheMode

async def basic_session_crawl():
    async with AsyncWebCrawler() as…

Advanced Technique 1: Custom Execution Hooks

> Warning: You might feel confused by the end of the next few examples 😅, so make sure you are comfortable with the order of the parts before you start this.

Use custom hooks to handle complex…

async def advanced_session_crawl_with_hooks():
    first_commit = ""

    async def on_execution_started(page):
        nonlocal first_commit
        try:
            while True:…

Advanced Technique 2: Integrated JavaScript Execution and Waiting

Combine JavaScript execution and waiting logic for concise handling of dynamic content:

async def integrated_js_and_wait_crawl():
    async with AsyncWebCrawler() as crawler:
        session_id = "integrated_session"
        url = "https://github.com/example/repo/commits/main"…

Common Use Cases for Sessions

1. **Authentication Flows**: Login and interact with secured pages.

2. **Pagination Handling**: Navigate through multiple pages.

3. **Form Submissions**: Fill forms, submit, and process…


SSLCertificate Reference

Summary: Reference for the SSLCertificate class in Crawl4AI, covering how to load, inspect, and export SSL/TLS certificate data, and how to use it with fetch_ssl_certificate=True in CrawlerRunConfig.

1. Overview

The **`SSLCertificate`** class encapsulates an SSL certificate’s data and allows exporting it in various formats (PEM, DER, JSON, or text). It’s used within **Crawl4AI** whenever you set…

class SSLCertificate:
    """
    Represents an SSL certificate with methods to export in various formats.

    Main Methods:
    - from_url(url, timeout=10)
    - from_file(file_path)
    -…

Typical Use Case

- You **enable** certificate fetching in your crawl by:
- After `arun()`, if `result.ssl_certificate` is present, it’s an instance of **`SSLCertificate`**.
- You can **read** basic properties…

2. Construction & Fetching

2.1 from_url(url, timeout=10)

Manually load an SSL certificate from a given URL (port 443). Typically used internally, but you can call it directly if you want:

cert = SSLCertificate.from_url("https://example.com")
if cert:
    print("Fingerprint:", cert.fingerprint)

2.2 from_file(file_path)

Load from a file containing certificate data in ASN.1 or DER. Rarely needed unless you have local cert files:

2.3 from_binary(binary_data)

Initialize from raw binary. E.g., if you captured it from a socket or another source:

3. Common Properties

After obtaining a **`SSLCertificate`** instance (e.g. `result.ssl_certificate` from a crawl), you can read:

1. **`issuer`** *(dict)*
   - E.g. `{"CN": "My Root CA", "O": "..."}`
2. **`subject`**…

4. Export Methods

Once you have a **`SSLCertificate`** object, you can **export** or **inspect** it:

4.1 to_json(filepath=None) → Optional[str]

- Returns a JSON string containing the parsed certificate fields.
- If `filepath` is provided, saves it to disk instead, returning `None`.

**Usage**:

json_data = cert.to_json()  # returns JSON string
cert.to_json("certificate.json")  # writes file, returns None

4.2 to_pem(filepath=None) → Optional[str]

- Returns a PEM-encoded string (common for web servers).
- If `filepath` is provided, saves it to disk instead.

pem_str = cert.to_pem()              # in-memory PEM string
cert.to_pem("/path/to/cert.pem")     # saved to file

4.3 to_der(filepath=None) → Optional[bytes]

- Returns the original DER (binary ASN.1) bytes.
- If `filepath` is specified, writes the bytes there instead.

der_bytes = cert.to_der()
cert.to_der("certificate.der")

4.4 (Optional) export_as_text()

- If you see a method like `export_as_text()`, it typically returns an OpenSSL-style textual representation.
- Not always needed, but can help for debugging or manual inspection.

5. Example Usage in Crawl4AI

Below is a minimal sample showing how the crawler obtains an SSL cert from a site, then reads or exports it. The code snippet:

import asyncio
import os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    tmp_dir = "tmp"
    os.makedirs(tmp_dir, exist_ok=True)

    config =…

6. Notes & Best Practices

1. **Timeout**: `SSLCertificate.from_url` internally uses a default **10s** socket connect and wraps SSL.
2. **Binary Form**: The certificate is loaded in ASN.1 (DER) form, then re-parsed by…

Summary

- **`SSLCertificate`** is a convenience class for capturing and exporting the **TLS certificate** from your crawled site(s).
- Common usage is in the **`CrawlResult.ssl_certificate`** field,…

The URL to load the SSL certificate from (port 443).

Socket connect timeout in seconds.

Path to a file containing certificate data in ASN.1 or DER.

Raw binary certificate data.

If provided, saves the JSON output to disk instead of returning a string.

If provided, saves the PEM output to disk instead of returning a string.

If provided, writes the DER bytes to disk instead of returning bytes.


Undetected Browser Mode

Summary: This guide covers Crawl4AI's anti-bot features: Stealth Mode and Undetected Browser Mode, including how to use them, when to use each, and best practices for evading bot detection.

Overview

Crawl4AI offers two powerful anti-bot features to help you access websites with bot detection:

- **Stealth Mode** - Uses playwright-stealth to modify browser fingerprints and behaviors
-…

Anti-Bot Features Comparison

| Feature | Regular Browser | Stealth Mode | Undetected Browser |
| --- | --- | --- | --- |
| WebDriver Detection | ❌ | ✅ | ✅ |
| Navigator Properties | ❌ | ✅ | ✅ |
| Plugin Emulation | ❌ | ✅ | ✅ |
|…

When to Use Each Approach

Use Regular Browser + Stealth Mode When:

- Sites have basic bot detection (checking navigator.webdriver, plugins, etc.)
- You need good performance with basic protection
- Sites check for common automation indicators

Use Undetected Browser When:

- Sites employ sophisticated bot detection services (Cloudflare, DataDome, etc.)
- Stealth mode alone isn't sufficient
- You're willing to trade some performance for better evasion

Best Practice: Progressive Enhancement

- **Start with**: Regular browser + Stealth mode
- **If blocked**: Switch to Undetected browser
- **If still blocked**: Combine Undetected browser + Stealth mode

Stealth Mode

Stealth mode is the simpler anti-bot solution that works with both regular and undetected browsers:

from crawl4ai import AsyncWebCrawler, BrowserConfig

# Enable stealth mode with regular browser
browser_config = BrowserConfig(
    enable_stealth=True,  # Simple flag to enable
    headless=False…

What Stealth Mode Does:

- Removes `navigator.webdriver` flag
- Modifies browser fingerprints
- Emulates realistic plugin behavior
- Adjusts navigator properties
- Fixes common automation leaks

Undetected Browser Mode

For sites with sophisticated bot detection that stealth mode can't bypass, use the undetected browser adapter:

Key Features

- **Drop-in Replacement**: Uses the same API as regular browser mode
- **Enhanced Stealth**: Built-in patches to evade common detection methods
- **Browser Adapter Pattern**: Seamlessly switch…

Quick Start

import asyncio
from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CrawlerRunConfig,
    UndetectedAdapter
)
from crawl4ai.async_crawler_strategy import…

Combining Both Features

For maximum evasion, combine stealth mode with undetected browser:

from crawl4ai import AsyncWebCrawler, BrowserConfig, UndetectedAdapter
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

# Create browser config with stealth…

Examples

Example 1: Basic Stealth Mode

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def test_stealth_mode():
    # Simple stealth mode configuration
    browser_config = BrowserConfig(…

Example 2: Undetected Browser Mode

Browser Adapter Pattern

The undetected browser support is implemented using an adapter pattern, allowing seamless switching between different browser implementations:

The adapter handles:
- JavaScript execution
- Console…

# Regular browser adapter (default)
from crawl4ai import PlaywrightAdapter
regular_adapter = PlaywrightAdapter()

# Undetected browser adapter
from crawl4ai import…

Best Practices

- **Avoid Headless Mode**: Detection is easier in headless mode
- **Use Reasonable Delays**: Don't rush through pages
- **Rotate User Agents**: You can customize user agents
- **Handle Failures…

crawler_config = CrawlerRunConfig(
    wait_time=3.0,  # Wait 3 seconds after page load
    delay_before_return_html=2.0  # Additional delay
)

browser_config = BrowserConfig(
    headers={"User-Agent": "your-user-agent"}
)

if not result.success:
    print(f"Crawl failed: {result.error_message}")

Advanced Usage Tips

Progressive Detection Handling

async def crawl_with_progressive_evasion(url):
    # Step 1: Try regular browser with stealth
    browser_config = BrowserConfig(
        enable_stealth=True,
        headless=False
    )

    async…

Installation

The undetected browser dependencies are automatically installed when you run:

This command installs all necessary browser dependencies for both regular and undetected modes.

Limitations

- **Performance**: Slightly slower than regular mode due to additional patches
- **Headless Detection**: Some sites can still detect headless mode
- **Resource Usage**: May use more resources than…

Troubleshooting

Browser Not Found

Run the setup command:

Detection Still Occurring

Try combining with other features:

crawler_config = CrawlerRunConfig(
    simulate_user=True,  # Add user simulation
    magic=True,  # Enable magic mode
    wait_time=5.0,  # Longer waits
)

Performance Issues

If experiencing slow performance:

# Use selective undetected mode only for protected sites
if is_protected_site(url):
    adapter = UndetectedAdapter()
else:
    adapter = PlaywrightAdapter()  # Default adapter

Future Plans

**Note**: In future versions of Crawl4AI, we may enable stealth mode and undetected browser by default to provide better out-of-the-box success rates. For now, users should explicitly enable these…

Conclusion

Crawl4AI provides flexible anti-bot solutions:

- **Start Simple**: Use regular browser + stealth mode for most sites
- **Escalate if Needed**: Switch to undetected browser for sophisticated…

See Also

- [Advanced Features](../advanced-features/) - Overview of all advanced features
- [Proxy & Security](../proxy-security/) - Using proxies with anti-bot features
- [Session…


Virtual Scroll - Crawl4AI Documentation (v0.9.x)

Summary: This page explains Crawl4AI's Virtual Scroll feature for handling virtual scrolling websites, covering configuration, usage examples, comparison with scan_full_page, and performance tips.

Understanding Virtual Scroll

Modern websites increasingly use virtual scrolling (also called windowed rendering or viewport rendering) to handle large datasets efficiently. This technique only renders visible items in the DOM,…

Traditional Scroll:          Virtual Scroll:
┌─────────────┐             ┌─────────────┐
│ Item 1      │             │ Item 11     │  <- Items 1-10 removed
│ Item 2      │             │ Item 12     │…

Basic Usage

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig

# Configure virtual scroll
virtual_config = VirtualScrollConfig(
    container_selector="#feed",      # CSS selector for…

Configuration Parameters

###### VirtualScrollConfig

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `container_selector` | `str` | Required | CSS selector for the scrollable container |
|…

Real-World Examples

###### Twitter-like Timeline

Twitter replaces tweets as you scroll.

###### Instagram Grid

Instagram uses virtualized grid for performance.

###### Mixed Content (News Feed)

Some sites mix static and…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig, BrowserConfig

async def crawl_twitter_timeline():
    # Twitter replaces tweets as you scroll
    virtual_config =…

async def crawl_instagram_grid():
    # Instagram uses virtualized grid for performance
    virtual_config = VirtualScrollConfig(
        container_selector="article",  # Main feed container…

async def crawl_mixed_feed():
    # Featured articles stay, regular articles virtualize
    virtual_config = VirtualScrollConfig(
        container_selector=".main-feed",
        scroll_count=25,…

Virtual Scroll vs scan_full_page

Both features handle dynamic content, but serve different purposes:

| Feature | Virtual Scroll | scan_full_page |
| --- | --- | --- |
| **Purpose** | Capture content that's replaced during scroll |…

Combining with Extraction

Virtual Scroll works seamlessly with extraction strategies.

from crawl4ai import LLMExtractionStrategy, LLMConfig

# Define extraction schema
schema = {
    "type": "array",
    "items": {
        "type": "object",
        "properties": {…

Performance Tips

- **Container Selection**: Be specific with selectors. Using the correct container improves performance.
- **Scroll Count**: Start conservative and increase as needed.
- **Wait Times**: Adjust based…

# Start with fewer scrolls
virtual_config = VirtualScrollConfig(
    container_selector="#feed",
    scroll_count=10  # Test with 10, increase if needed
)

# Fast sites
wait_after_scroll=0.2

# Slower sites or heavy content
wait_after_scroll=1.5

browser_config = BrowserConfig(headless=False)
async with AsyncWebCrawler(config=browser_config) as crawler:
    # Watch the scrolling happen

How It Works Internally

- **Detection Phase**: Scrolls and compares HTML to detect behavior
- **Capture Phase**: For replaced content, stores HTML chunks at each position
- **Merge Phase**: Combines all chunks, removing…

Error Handling

Virtual Scroll handles errors gracefully. If the container isn't found, crawling continues normally without virtual scroll.

# If container not found or scrolling fails
result = await crawler.arun(url="...", config=config)

if result.success:
    # Virtual scroll worked or wasn't needed
    print(f"Captured…

Complete Example

See our comprehensive example that demonstrates:
- Twitter-like feeds
- Instagram grids
- Traditional infinite scroll
- Mixed content scenarios
- Performance comparisons

The example includes a local…

# Run the examples
cd docs/examples
python virtual_scroll_example.py


AdaptiveCrawler

Summary: The AdaptiveCrawler class implements intelligent web crawling that automatically determines when sufficient information has been gathered to answer a query. It uses a three-layer scoring system to…

Constructor

###### Parameters

- **crawler**  (`AsyncWebCrawler`): The underlying web crawler instance to use for fetching pages
- **config**  (`Optional[AdaptiveConfig]`): Configuration settings for adaptive…

AdaptiveCrawler(
    crawler: AsyncWebCrawler,
    config: Optional[AdaptiveConfig] = None
)

Primary Method

###### digest()

The main method that performs adaptive crawling starting from a URL with a specific query.

###### Parameters

- **start_url**  (`str`): The starting URL for crawling
- **query**…

async def digest(
    start_url: str,
    query: str,
    resume_from: Optional[Union[str, Path]] = None
) -> CrawlState

async with AsyncWebCrawler() as crawler:
    adaptive = AdaptiveCrawler(crawler)
    state = await adaptive.digest(
        start_url="https://docs.python.org",
        query="async context…

Properties

###### confidence

Current confidence score (0-1) indicating information sufficiency.

###### coverage_stats

Dictionary containing detailed coverage statistics.

Returns:
-  **coverage** : Query term…

@property
def coverage_stats(self) -> Dict[str, float]

@property
def is_sufficient(self) -> bool

Methods

###### get_relevant_content()

Retrieve the most relevant content from the knowledge base.

###### Parameters

- **top_k**  (`int`): Number of top relevant documents to return (default: 5)

####…

def get_relevant_content(
    self,
    top_k: int = 5
) -> List[Dict[str, Any]]

def print_stats(
    self,
    detailed: bool = False
) -> None

def export_knowledge_base(
    self,
    path: Union[str, Path]
) -> None

async def import_knowledge_base(
    self,
    path: Union[str, Path]
) -> None

Configuration

The `AdaptiveConfig` class controls the behavior of adaptive crawling:

###### Example with Custom Config

@dataclass
class AdaptiveConfig:
    confidence_threshold: float = 0.8      # Stop when confidence reaches this
    max_pages: int = 50                    # Maximum pages to crawl
    top_k_links:…

config = AdaptiveConfig(
    confidence_threshold=0.7,
    max_pages=20,
    top_k_links=3
)

adaptive = AdaptiveCrawler(crawler, config=config)

Complete Example

import asyncio
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig

async def main():
    # Configure adaptive crawling
    config = AdaptiveConfig(…

See Also

- [digest() Method Reference](../digest/)
- [Adaptive Crawling Guide](../../core/adaptive-crawling/)
- [Advanced Adaptive Strategies](../../advanced/adaptive-strategies/)

config

Configuration settings for adaptive crawling behavior. If not provided, uses default settings.

The starting URL for crawling

query

The search query that guides the crawling process

Path to a saved state file to resume from

detailed

If True, shows detailed metrics with colors. If False, shows summary table.

path

Output file path for JSONL export

Path to JSONL file to import


`arun()` Parameter Guide (New Approach)

Summary: A guide to the parameters of the `arun()` method in Crawl4AI, now organized under `CrawlerRunConfig`, covering caching, content processing, navigation, session management, media options, extraction,…

Introduction

In Crawl4AI's **latest** configuration model, nearly all parameters that once went directly to `arun()` are now part of **`CrawlerRunConfig`** . When calling `arun()`, you provide:

Below is an…

await crawler.arun(
    url="https://example.com",
    config=my_run_config
)

1. Core Usage

**Key Fields**:
- `verbose=True` logs each crawl step.
- `cache_mode` decides how to read/write the local crawl cache.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    run_config = CrawlerRunConfig(
        verbose=True,            # Detailed logging…

2. Cache Control

**`cache_mode`** (default: `CacheMode.ENABLED`)

Use a built-in enum from `CacheMode`:

- `ENABLED`: Normal caching—reads if available, writes if missing.
- `DISABLED`: No caching—always refetch…

run_config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS
)

3.1 Text Processing

run_config = CrawlerRunConfig(
    word_count_threshold=10,   # Ignore text blocks <10 words
    only_text=False,           # If True, tries to remove non-text elements
    keep_data_attributes=False…

3.2 Content Selection

run_config = CrawlerRunConfig(
    css_selector=".main-content",  # Focus on .main-content region only
    excluded_tags=["form", "nav"], # Remove entire tag blocks
    remove_forms=True,…

3.3 Link Handling

run_config = CrawlerRunConfig(
    exclude_external_links=True,         # Remove external links from final content
    exclude_social_media_links=True,     # Remove links to known social sites…

3.4 Media Filtering

run_config = CrawlerRunConfig(
    exclude_external_images=True  # Strip images from other domains
)

4.1 Basic Browser Flow

**Key Fields**:
- `wait_for`:
  - `"css:selector"` or
  - `"js:() => boolean"`
  e.g. `js:() => document.querySelectorAll('.item').length > 10`.
- `mean_delay` & `max_range`: define random delays for…

run_config = CrawlerRunConfig(
    wait_for="css:.dynamic-content", # Wait for .dynamic-content
    delay_before_return_html=2.0,    # Wait 2s before capturing final HTML
    page_timeout=60000,…

4.2 JavaScript Execution

- `js_code` can be a single string or a list of strings.
- `js_only=True` means “I’m continuing in the same session with new JS steps, no new full navigation.”

run_config = CrawlerRunConfig(
    js_code=[
        "window.scrollTo(0, document.body.scrollHeight);",
        "document.querySelector('.load-more')?.click();"
    ],
    js_only=False
)

4.3 Anti-Bot

- `magic=True` tries multiple stealth features.
- `simulate_user=True` mimics mouse movements or random delays.
- `override_navigator=True` fakes some navigator properties (like user agent checks).

run_config = CrawlerRunConfig(
    magic=True,
    simulate_user=True,
    override_navigator=True
)

5. Session Management

If re-used in subsequent `arun()` calls, the same tab/page context is continued (helpful for multi-step tasks or stateful browsing).

run_config = CrawlerRunConfig(
    session_id="my_session123"
)

6. Screenshot, PDF & Media Options

**Where they appear**:
- `result.screenshot` → Base64 screenshot string.
- `result.pdf` → Byte array with PDF data.

run_config = CrawlerRunConfig(
    screenshot=True,             # Grab a screenshot as base64
    screenshot_wait_for=1.0,     # Wait 1s before capturing
    pdf=True,                    # Also…

7. Extraction Strategy

The extracted data will appear in `result.extracted_content`.

run_config = CrawlerRunConfig(
    extraction_strategy=my_css_or_llm_strategy
)

8. Comprehensive Example

Below is a snippet combining many parameters:

**What we covered**:
1. **Crawling** the main content region, ignoring external links.
2. Running **JavaScript** to click “.show-more”.
3. **Waiting**…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def main():
    # Example schema
    schema = {
        "name":…

9. Best Practices

1. **Use `BrowserConfig` for global browser** settings (headless, user agent).
2. **Use `CrawlerRunConfig`** to handle the **specific** crawl needs: content filtering, caching, JS, screenshot,…

10. Conclusion

All parameters that used to be direct arguments to `arun()` now belong in **`CrawlerRunConfig`** . This approach:

- Makes code **clearer** and **more maintainable**.
- Minimizes confusion about…


arun_many()

Summary: Reference for the arun_many() function in Crawl4AI, which crawls multiple URLs concurrently or in batches, with support for dispatchers, streaming, and per-URL configurations.

Function Signature

async def arun_many(
    urls: Union[List[str], List[Any]],
    config: Optional[Union[CrawlerRunConfig, List[CrawlerRunConfig]]] = None,
    dispatcher: Optional[BaseDispatcher] = None,
    ...
) ->…

Differences from arun()

1. **Multiple URLs**:
- Instead of crawling a single URL, you pass a list of them (strings or tasks).
- The function returns `RunManyReturn` which contains either a **list** of `CrawlResult` or an…

Basic Example (Batch Mode)

# Minimal usage: The default dispatcher will be used
results = await crawler.arun_many(
    urls=["https://site1.com", "https://site2.com"],
    config=CrawlerRunConfig(stream=False)  # Default…

Streaming Example

config = CrawlerRunConfig(
    stream=True,  # Enable streaming mode
    cache_mode=CacheMode.BYPASS
)

# Process results as they complete
async for result in await crawler.arun_many(…

With a Custom Dispatcher

dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=70.0,
    max_session_permit=10
)
results = await crawler.arun_many(
    urls=["https://site1.com", "https://site2.com",…

URL-Specific Configurations

Instead of using one config for all URLs, provide a list of configs with `url_matcher` patterns:

**URL Matching Features**:
- **String patterns**: `"*.pdf"`, `"*/blog/*"`, `"*python.org*"`
-…

from crawl4ai import CrawlerRunConfig, MatchMode
from crawl4ai.processors.pdf import PDFContentScrapingStrategy
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
from…

Return Value

Returns a **`RunManyReturn`** object which contains either a **list** of [`CrawlResult`](../crawl-result/) objects, or an **async generator** if streaming is enabled. You can iterate to check…

Dispatcher Reference

- **`MemoryAdaptiveDispatcher`** : Dynamically manages concurrency based on system memory usage.
- **`SemaphoreDispatcher`** : Fixed concurrency limit, simpler but less adaptive.

For advanced usage…

Common Pitfalls

1. **Large Lists** : If you pass thousands of URLs, be mindful of memory or rate-limits. A dispatcher can help.
2. **Session Reuse** : If you need specialized logins or persistent contexts, ensure…

Conclusion

Use `arun_many()` when you want to **crawl multiple URLs** simultaneously or in controlled parallel tasks. If you need advanced concurrency features (like memory-based adaptive throttling or complex…

urls

A list of URLs (or tasks) to crawl.

config

Either a single CrawlerRunConfig applying to all URLs, or a list of CrawlerRunConfig objects with url_matcher patterns.

dispatcher

A concurrency controller (e.g. MemoryAdaptiveDispatcher).


AsyncWebCrawler - Crawl4AI Documentation (v0.9.x)

Summary: Extraction fallback content.

##### AsyncWebCrawler

The  **`AsyncWebCrawler`**  is the core class for asynchronous web crawling in Crawl4AI. You typically create it  **once** , optionally customize it with a  **`BrowserConfig`**  (e.g., headless, user agent), then  **run**  multiple  **`arun()`**  calls with different  **`CrawlerRunConfig`**  objects.

 **Recommended usage** :

1.  **Create**  a `BrowserConfig` for global browser settings.

2.  **Instantiate**  `AsyncWebCrawler(config=browser_config)`.

3.  **Use**  the crawler in an async context manager (`async with`) or manage start/close manually.

4.  **Call**  `arun(url, config=crawler_run_config)` for each page you want.

##### 1. Constructor Overview

 **Notes** :

- **Legacy**  parameters like `always_bypass_cache` remain for backward compatibility, but prefer to set  **caching**  in `CrawlerRunConfig`.

##### 2. Lifecycle: Start/Close or Context Manager

###### 2.1 Context Manager (Recommended)

When the `async with` block ends, the crawler cleans up (closes the browser, etc.).

###### 2.2 Manual Start & Close

Use this style if you have a  **long-running**  application or need full control of the crawler’s lifecycle.

##### 3. Primary Method: `arun()`

###### 3.1 New Approach

You pass a `CrawlerRunConfig` object that sets up everything about a crawl—content filtering, caching, session reuse, JS code, screenshots, etc.

###### 3.2 Legacy Parameters Still Accepted

For  **backward**  compatibility, `arun()` can still accept direct arguments like `css_selector=...`, `word_count_threshold=...`, etc., but we strongly advise migrating them into a  **`CrawlerRunConfig`** .

##### 4. Batch Processing: `arun_many()`

###### 4.1 Resource-Aware Crawling

The `arun_many()` method now uses an intelligent dispatcher that:

- Monitors system memory usage

- Implements adaptive rate limiting

- Provides detailed progress monitoring

- Manages concurrent crawls efficiently

###### 4.2 Example Usage

Check page [Multi-url Crawling](../../advanced/multi-url-crawling/) for a detailed example of how to use `arun_many()`.

 **Explanation** :

- We define a  **`BrowserConfig`**  with Firefox, no headless, and `verbose=True`.

- We define a  **`CrawlerRunConfig`**  that  **bypasses cache** , uses a  **CSS**  extraction schema, has a `word_count_threshold=15`, etc.

- We pass them to `AsyncWebCrawler(config=...)` and `arun(url=..., config=...)`.

##### 7. Best Practices & Migration Notes

1.  **Use**  `BrowserConfig` for  **global**  settings about the browser’s environment.
2.  **Use**  `CrawlerRunConfig` for  **per-crawl**  logic (caching, content filtering, extraction strategies, wait conditions).
3.  **Avoid**  legacy parameters like `css_selector` or `word_count_threshold` directly in `arun()`. Instead:

4.  **Context Manager**  usage is simplest unless you want a persistent crawler across many calls.

##### 8. Summary

 **AsyncWebCrawler**  is your entry point to asynchronous crawling:

- **Constructor**  accepts  **`BrowserConfig`**  (or defaults).

- **`arun(url, config=CrawlerRunConfig)`**  is the main method for single-page crawls.

- **`arun_many(urls, config=CrawlerRunConfig)`**  handles concurrency across multiple URLs.

- For advanced lifecycle control, use `start()` and `close()` explicitly.

 **Migration** :

- If you used `AsyncWebCrawler(browser_type="chromium", css_selector="...")`, move browser settings to `BrowserConfig(...)` and content/crawl logic to `CrawlerRunConfig(...)`.

This modular approach ensures your code is  **clean** ,  **scalable** , and  **easy to maintain** . For any advanced or rarely used parameters, see the [BrowserConfig docs](../parameters/).

class AsyncWebCrawler:
    def __init__(
        self,
        crawler_strategy: Optional[AsyncCrawlerStrategy] = None,
        config: Optional[BrowserConfig] = None,
        always_bypass_cache: bool = False,           # deprecated
        always_by_pass_cache: Optional[bool] = None, # also deprecated
        base_directory: str = ...,
        thread_safe: bool = False,
        **kwargs,
    ):
        """
        Create an AsyncWebCrawler instance.

        Args:
            crawler_strategy:
                (Advanced) Provide a custom crawler strategy if needed.
            config:
                A BrowserConfig object specifying how the browser is set up.
            always_bypass_cache:
                (Deprecated) Use CrawlerRunConfig.cache_mode instead.
            base_directory:
                Folder for storing caches/logs (if relevant).
            thread_safe:
                If True, attempts some concurrency safeguards. Usually False.
            **kwargs:
                Additional legacy or debugging parameters.
        """
    )

### Typical Initialization

```python
from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_cfg = BrowserConfig(
    browser_type="chromium",
    headless=True,
    verbose=True
)

crawler = AsyncWebCrawler(config=browser_cfg)

async with AsyncWebCrawler(config=browser_cfg) as crawler:
    result = await crawler.arun("https://example.com")
    # The crawler automatically starts/closes resources

crawler = AsyncWebCrawler(config=browser_cfg)
await crawler.start()

result1 = await crawler.arun("https://example.com")
result2 = await crawler.arun("https://another.com")

await crawler.close()

async def arun(
    self,
    url: str,
    config: Optional[CrawlerRunConfig] = None,
    # Legacy parameters for backward compatibility...
) -> RunManyReturn:
    ...

import asyncio
from crawl4ai import CrawlerRunConfig, CacheMode

run_cfg = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    css_selector="main.article",
    word_count_threshold=10,
    screenshot=True
)

async with AsyncWebCrawler(config=browser_cfg) as crawler:
    result = await crawler.arun("https://example.com/news", config=run_cfg)
    print("Crawled HTML length:", len(result.cleaned_html))
    if result.screenshot:
        print("Screenshot base64 length:", len(result.screenshot))

async def arun_many(
    self,
    urls: List[str],
    config: Optional[CrawlerRunConfig] = None,
    # Legacy parameters maintained for backwards compatibility...
) -> RunManyReturn:
    """
    Process multiple URLs with intelligent rate limiting and resource monitoring.
    """

### 4.3 Key Features

1. **Rate Limiting**

   - Automatic delay between requests
   - Exponential backoff on rate limit detection
   - Domain-specific rate limiting
   - Configurable retry strategy

2. **Resource Monitoring**

   - Memory usage tracking
   - Adaptive concurrency based on system load
   - Automatic pausing when resources are constrained

3. **Progress Monitoring**

   - Detailed or aggregated progress display
   - Real-time status updates
   - Memory usage statistics

4. **Error Handling**

   - Graceful handling of rate limits
   - Automatic retries with backoff
   - Detailed error reporting

---

## 5. `CrawlResult` Output

Each `arun()` returns a **`CrawlResult`** containing:

- `url`: Final URL (if redirected).
- `html`: Original HTML.
- `cleaned_html`: Sanitized HTML.
- `markdown_v2`: Removed in v0.5. Accessing it raises `AttributeError`; use `markdown`.
- `extracted_content`: If an extraction strategy was used (JSON for CSS/LLM strategies).
- `screenshot`, `pdf`: If screenshots/PDF requested.
- `media`, `links`: Information about discovered images/links.
- `success`, `error_message`: Status info.

For details, see [CrawlResult doc](./crawl-result.md).

---

## 6. Quick Example

Below is an example hooking it all together:

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy
import json

async def main():
    # 1. Browser config
    browser_cfg = BrowserConfig(
        browser_type="firefox",
        headless=False,
        verbose=True
    )

    # 2. Run config
    schema = {
        "name": "Articles",
        "baseSelector": "article.post",
        "fields": [
            {
                "name": "title",
                "selector": "h2",
                "type": "text"
            },
            {
                "name": "url",
                "selector": "a",
                "type": "attribute",
                "attribute": "href"
            }
        ]
    }

    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(schema),
        word_count_threshold=15,
        remove_overlay_elements=True,
        wait_for="css:.post"  # Wait for posts to appear
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com/blog",
            config=run_cfg
        )

        if result.success:
            print("Cleaned HTML length:", len(result.cleaned_html))
            if result.extracted_content:
                articles = json.loads(result.extracted_content)
                print("Extracted articles:", articles[:2])
        else:
            print("Error:", result.error_message)

asyncio.run(main())

run_cfg = CrawlerRunConfig(css_selector=".main-content", word_count_threshold=20)
result = await crawler.arun(url="...", config=run_cfg)