API Reference Book

Auto-generated from documentation
Source: https://docs.crawl4ai.com
Generated: 2026-08-11
v0.8.10-9-g59efd08

How to Read This Book

This reference book was generated from the publicly accessible documentation at https://docs.crawl4ai.com on 2026-08-11. It is a structured, self-contained snapshot suitable for offline reference and AI ingestion.

Where to start

  1. Table of Contents — the right pane (HTML) or page 2 (PDF) shows every chapter and sub-section. Each entry links to its anchor.
  2. Chapter introduction — each chapter opens with a one-paragraph summary explaining its scope.
  3. Code blocks — syntax-highlighted using Pygments; copy-pasteable.
  4. Search — use your reader's full-text search (Ctrl-F) for any symbol or word.

Conventions

Introduction & Overview

🚀🤖 Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper

overview
Source: https://docs.crawl4ai.com

Crawl4AI is an open-source, LLM-friendly web crawler and scraper that produces clean Markdown, supports structured extraction, advanced browser control, and high-performance parallel crawling. This…

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 **Apply here for early…

Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper

Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for large language models, AI agents, and data…

🆕 AI Assistant Skill Now Available!

🤖 Crawl4AI Skill for Claude & AI Assistants

Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:

🎯 New: Adaptive Web Crawling

Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your…

Quick Start

Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities:

python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    # Create an instance of AsyncWebCrawler
    async with AsyncWebCrawler() as crawler:
        # Run the crawler on a URL…

Video Tutorial

What Does Crawl4AI Do?

Crawl4AI is a feature-rich crawler and scraper that aims to:

  1. Generate Clean Markdown: Perfect for RAG pipelines or direct ingestion into LLMs.
  2. Structured Extraction: Parse repeated…

Documentation Structure

To help you get started, we’ve organized our docs into clear sections:

How You Can Support

Home - Crawl4AI Documentation (v0.9.x)

overview
Source: https://docs.crawl4ai.com/

Landing page and overview for Crawl4AI, an open-source, LLM-friendly web crawler and scraper. It introduces the project's mission, key capabilities, quick-start example, documentation structure, and…

🚀🤖 Crawl4AI: Open-Source LLM-Friendly Web Crawler & Scraper

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 Apply here for early…

🆕 AI Assistant Skill Now Available!

🤖 Crawl4AI Skill for Claude & AI Assistants

Supercharge your AI coding assistant with complete Crawl4AI knowledge! Download our comprehensive skill package that includes:

🎯 New: Adaptive Web Crawling

Crawl4AI now features intelligent adaptive crawling that knows when to stop! Using advanced information foraging algorithms, it determines when sufficient information has been gathered to answer your…

Quick Start

Here's a quick example to show you how easy it is to use Crawl4AI with its asynchronous capabilities:

python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    # Create an instance of AsyncWebCrawler
    async with AsyncWebCrawler() as crawler:
        # Run the crawler on a URL…

Video Tutorial

What Does Crawl4AI Do?

Crawl4AI is a feature-rich crawler and scraper that aims to:

  1. Generate Clean Markdown: Perfect for RAG pipelines or direct ingestion into LLMs.
  2. Structured Extraction: Parse repeated patterns…

Documentation Structure

To help you get started, we’ve organized our docs into clear sections:

How You Can Support

Thank you for joining me on this journey. Let’s keep building an open, democratic approach to data extraction and AI…

Overview of Some Important Advanced Features

guide
Source: https://docs.crawl4ai.com/advanced/advanced-features/

A guide to advanced Crawl4AI features including proxy usage, PDF/screenshot capture, SSL certificates, custom headers, session persistence, robots.txt compliance, and anti-bot techniques.

Overview of Some Important Advanced Features

Crawl4AI offers multiple power-user features that go beyond simple crawling. This tutorial covers:

  1. Proxy Usage

  2. Capturing PDFs & Screenshots

  3. Handling SSL Certificates

4.…

1. Proxy Usage

If you need to route your crawl traffic through a proxy—whether for IP rotation, geo-testing, or privacy—Crawl4AI supports it via BrowserConfig.proxy_config.

Key Points

python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    browser_cfg = BrowserConfig(
        proxy_config={
            "server":…

2. Capturing PDFs & Screenshots

Sometimes you need a visual record of a page or a PDF “printout.” Crawl4AI can do both in one pass:

Why PDF + Screenshot?

python
import os, asyncio
from base64 import b64decode
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def main():
    run_config = CrawlerRunConfig(…

3. Handling SSL Certificates

If you need to verify or export a site’s SSL certificate—for compliance, debugging, or data analysis—Crawl4AI can fetch it during the crawl:

Key Points

python
import asyncio, os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    tmp_dir = os.path.join(os.getcwd(), "tmp")
    os.makedirs(tmp_dir, exist_ok=True)…

4. Custom Headers

Sometimes you need to set custom headers (e.g., language preferences, authentication tokens, or specialized user-agent strings). You can do this in multiple ways:

Notes

python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    # Option 1: Set headers at the crawler strategy level
    crawler1 = AsyncWebCrawler(
        # The underlying strategy can…

5. Session Persistence & Local Storage

Crawl4AI can preserve cookies and localStorage so you can continue where you left off—ideal for logging into sites or skipping repeated auth flows.

5.1 `storage_state`

python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    storage_dict = {
        "cookies": [
            {
                "name": "session",
                "value": "abcd1234",…

5.2 Exporting & Reusing State

You can sign in once, export the browser context, and reuse it later—without re-entering credentials.

6. Robots.txt Compliance

Crawl4AI supports respecting robots.txt rules with efficient caching:

Key Points

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Enable robots.txt checking in config
    config = CrawlerRunConfig(
        check_robots_txt=True  #…

Putting It All Together

Here’s a snippet that combines multiple “advanced” features (proxy, PDF, screenshot, SSL, custom headers, and session reuse) into one run. Normally, you’d tailor each setting to your project’s needs.

python
import os, asyncio
from base64 import b64decode
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    # 1. Browser config with proxy + headless…

7. Anti-Bot Features (Stealth Mode & Undetected Browser)

Crawl4AI provides two powerful features to bypass bot detection:

7.1 Stealth Mode

Stealth mode uses playwright-stealth to modify browser fingerprints and behaviors. Enable it with a simple flag:

When to use: Sites with basic bot detection (checking navigator.webdriver,…

python
browser_config = BrowserConfig(
    enable_stealth=True,  # Activates stealth mode
    headless=False
)

7.2 Undetected Browser

For advanced bot detection, use the undetected browser adapter:

When to use: Sites with sophisticated bot detection (Cloudflare, DataDome, etc.)

python
from crawl4ai import UndetectedAdapter
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy

# Create undetected adapter
adapter = UndetectedAdapter()
strategy =…

7.3 Combining Both

For maximum evasion, combine stealth mode with undetected browser:

python
browser_config = BrowserConfig(
    enable_stealth=True,  # Enable stealth
    headless=False
)

adapter = UndetectedAdapter()  # Use undetected browser

Choosing the Right Approach

Detection Level Recommended Approach
No protection Regular browser
Basic checks Regular + Stealth mode
Advanced protection Undetected browser
Maximum evasion …

Conclusion & Next Steps

You've now explored several advanced features:

Getting Started

Installation 💻 - Crawl4AI Documentation (v0.9.x)

reference
Source: https://docs.crawl4ai.com/basic/installation/

Extraction fallback content.

Installation 💻 - Crawl4AI Documentation (v0.9.x)

Installation 💻

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package, use it with Docker, or run it as a local server.

Option 1: Python Package Installation (Recommended)

Crawl4AI is now available on PyPI, making installation easier than ever. Choose the option that best fits your needs:

Basic Installation

For basic web crawling and scraping tasks:

Installation with PyTorch

For advanced text clustering (includes CosineSimilarity cluster strategy):

Installation with Transformers

For text summarization and Hugging Face models:

Full Installation

For all features:

Development Installation

For contributors who plan to modify the source code:

💡 After installation with "torch", "transformer", or "all" options, it's recommended to run the following CLI command to load the required models:

This is optional but will boost the performance and speed of the crawler. You only need to do this once after installation.

Playwright Installation Note for Ubuntu

If you encounter issues with Playwright installation on Ubuntu, you may need to install additional dependencies:

Option 2: Using Docker (Coming Soon)

Docker support for Crawl4AI is currently in progress and will be available soon. This will allow you to run Crawl4AI in a containerized environment, ensuring consistency across different systems.

Option 3: Local Server Installation

For those who prefer to run Crawl4AI as a local server, instructions will be provided once the Docker implementation is complete.

Verifying Your Installation

After installation, you can verify that Crawl4AI is working correctly by running a simple Python script:

This script should successfully crawl the example website and print the first 500 characters of the extracted content.

Getting Help

If you encounter any issues during installation or usage, please check the documentation or raise an issue on the GitHub repository.

Happy crawling! 🕷️🤖

Page Copy Page Copy

ESC to close

bash
pip install crawl4ai
playwright install # Install Playwright dependencies
css
pip install crawl4ai[torch]
css
pip install crawl4ai[transformer]
css
pip install crawl4ai[all]
bash
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e ".[all]"
playwright install # Install Playwright dependencies
undefined
crawl4ai-download-models
csharp
sudo apt-get install -y \
    libwoff1 \
    libopus0 \
    libwebp7 \
    libwebpdemux2 \
    libenchant-2-2 \
    libgudev-1.0-0 \
    libsecret-1-0 \
    libhyphen0 \
    libgdk-pixbuf2.0-0 \
    libegl1 \
    libnotify4 \
    libxslt1.1 \
    libevent-2.1-7 \
    libgles2 \
    libxcomposite1 \
    libatk1.0-0 \
    libatk-bridge2.0-0 \
    libepoxy0 \
    libgtk-3-0 \
    libharfbuzz-icu0 \
    libgstreamer-gl1.0-0 \
    libgstreamer-plugins-bad1.0-0 \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad \
    libxt6 \
    libxaw7 \
    xvfb \
    fonts-noto-color-emoji \
    libfontconfig \
    libfreetype6 \
    xfonts-cyrillic \
    xfonts-scalable \
    fonts-liberation \
    fonts-ipafont-gothic \
    fonts-wqy-zenhei \
    fonts-tlwg-loma-otf \
    fonts-freefont-ttf
python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(url="https://www.example.com")
        print(result.markdown[:500])  # Print first 500 characters

if __name__ == "__main__":
    asyncio.run(main())

Browser, Crawler & LLM Configuration (Quick Overview)

guide
Source: https://docs.crawl4ai.com/core/browser-crawler-config/

Overview of the three core configuration classes in Crawl4AI — BrowserConfig, CrawlerRunConfig, and LLMConfig — explaining their most commonly used parameters, helper methods, and how to combine them…

Overview

Crawl4AI's flexibility stems from two key classes:

-…

1. BrowserConfig Essentials

Key Fields to Note

1.⠀ browser_type

2.⠀…

python
class BrowserConfig:
    def __init__(
        browser_type="chromium",
        headless=True,
        browser_mode="dedicated",
        use_managed_browser=False,
        cdp_url=None,…
json
{
    "server": "http://proxy.example.com:8080", 
    "username": "...", 
    "password": "..."
}
python
# Create a base browser config
base_browser = BrowserConfig(
    browser_type="chromium",
    headless=True,
    text_mode=True
)

# Create a visible browser config for debugging
debug_browser =…
python
from crawl4ai import BrowserConfig, CrawlerRunConfig

# At application startup — one time
BrowserConfig.set_defaults(
    cache_cdp_connection=True,
    cdp_close_delay=0,…
python
from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_conf = BrowserConfig(
    browser_type="firefox",
    headless=False,
    text_mode=True
)

async with…

2. CrawlerRunConfig Essentials

Key Fields to Note

1.⠀ word_count_threshold :

2.⠀…

python
class CrawlerRunConfig:
    def __init__(
        word_count_threshold=200,
        extraction_strategy=None,
        chunking_strategy=RegexChunking(),
        markdown_generator=None,…
python
# Create a base configuration
base_config = CrawlerRunConfig(
    cache_mode=CacheMode.ENABLED,
    word_count_threshold=200,
    wait_until="networkidle"
)

# Create variations for different use…

3. LLMConfig Essentials

Key fields to note

1.⠀ provider :

python
llm_config = LLMConfig(
    provider="openai/gpt-4o-mini",
    api_token=os.getenv("OPENAI_API_KEY"),
    backoff_base_delay=1, # optional
    backoff_max_attempts=5, # optional…

4. Putting It All Together

In a typical scenario, you define one BrowserConfig for your crawler session, then create one or more CrawlerRunConfig & LLMConfig depending on each call's needs:

python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig, LLMContentFilter, DefaultMarkdownGenerator
from crawl4ai import…

5. Next Steps

For a detailed list of available parameters (including advanced ones), see:

You can explore topics like:

-…

6. Conclusion

BrowserConfig , CrawlerRunConfig and LLMConfig give you straightforward ways to define:

Installation & Setup (2023 Edition)

guide
Source: https://docs.crawl4ai.com/core/installation/

Installation and setup guide for Crawl4AI, covering basic install, diagnostics, verification, optional advanced dependencies, Docker, and local server mode.

1. Basic Installation

This installs the core Crawl4AI library along with essential dependencies. No advanced features (like transformers or PyTorch) are included yet.

bash
pip install crawl4ai

2. Initial Setup & Diagnostics

2.1 Run the Setup Command

After installing, call:

What does it do?

bash
crawl4ai-setup

2.2 Diagnostics

Optionally, you can run diagnostics to confirm everything is functioning:

This command attempts to:

bash
crawl4ai-doctor

3. Verifying Installation: A Simple Crawl (Skip this step if you already run `crawl4ai-doctor`)

Below is a minimal Python script demonstrating a basic crawl. It uses our new BrowserConfig and CrawlerRunConfig for clarity, though no custom settings are passed in this…

python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(…

4. Advanced Installation (Optional)

Warning : Only install these if you truly need them . They bring in larger dependencies, including big models, which can increase disk usage and memory load significantly.

4.1 Torch, Transformers, or All

bash
pip install crawl4ai[torch]
crawl4ai-setup
bash
pip install crawl4ai[transformer]
crawl4ai-setup
bash
pip install crawl4ai[all]
crawl4ai-setup

(Optional) Pre-Fetching Models

This step caches large models locally (if needed). Only do this if your workflow requires them.

bash
crawl4ai-download-models

5. Docker (Experimental)

We provide a temporary Docker approach for testing. It’s not stable and may break with future releases. We plan a major Docker revamp in a future stable version, 2025 Q1. If you still…

bash
docker pull unclecode/crawl4ai:basic
docker run -p 11235:11235 unclecode/crawl4ai:basic

6. Local Server Mode (Legacy)

Some older docs mention running Crawl4AI as a local server. This approach has been partially replaced by the new Docker-based prototype and upcoming stable server release. You can experiment,…

Summary

  1. Install with pip install crawl4ai and run crawl4ai-setup.
  2. Diagnose with crawl4ai-doctor if you see errors.
  3. Verify by crawling example.com with minimal BrowserConfig +…

Quick Start - Crawl4AI Documentation (v0.9.x)

reference
Source: https://docs.crawl4ai.com/core/quickstart/

Extraction fallback content.

Quick Start - Crawl4AI Documentation (v0.9.x)

Getting Started with Crawl4AI

Welcome to Crawl4AI , an open-source LLM-friendly Web Crawler & Scraper. In this tutorial, you’ll:

1. Introduction

Crawl4AI provides:

By the end of this guide, you’ll have performed a basic crawl, generated Markdown, tried out two extraction strategies, and crawled a dynamic page that uses “Load More” buttons or JavaScript updates.

2. Your First Crawl

Here’s a minimal Python script that creates an AsyncWebCrawler , fetches a webpage, and prints the first 300 characters of its Markdown output:

What’s happening?

You now have a simple, working crawl!

3. Basic Configuration (Light Introduction)

Crawl4AI’s crawler can be heavily customized using two main classes:

1.  BrowserConfig : Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.).

2.  CrawlerRunConfig : Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.).

Below is an example with minimal usage:

IMPORTANT: By default cache mode is set to CacheMode.BYPASS to have fresh content. Set CacheMode.ENABLED to enable caching.

We’ll explore more advanced config in later tutorials (like enabling proxies, PDF output, multi-tab sessions, etc.). For now, just note how you pass these objects to manage crawling.

4. Generating Markdown Output

By default, Crawl4AI automatically generates Markdown from each crawled page. However, the exact output depends on whether you specify a markdown generator or content filter .

Example: Using a Filter with DefaultMarkdownGenerator

Note : If you do not specify a content filter or markdown generator, you’ll typically see only the raw Markdown. PruningContentFilter may adds around 50ms in processing time. We’ll dive deeper into these strategies in a dedicated Markdown Generation tutorial.

5. Simple Data Extraction (CSS-based)

Crawl4AI can also extract structured data (JSON) using CSS or XPath selectors. Below is a minimal CSS-based example:

New! Crawl4AI now provides a powerful utility to automatically generate extraction schemas using LLM. This is a one-time cost that gives you a reusable schema for fast, LLM-free extractions:

For a complete guide on schema generation and advanced usage, see No-LLM Extraction Strategies.

Here's a basic extraction example:

Why is this helpful?

Tips: You can pass raw HTML to the crawler instead of a URL. To do so, prefix the HTML with raw://.

6. Simple Data Extraction (LLM-based)

For more complex or irregular pages, a language model can parse text intelligently into a structure you define. Crawl4AI supports open-source or closed-source providers:

Below is an example using open-source style (no token) and closed-source:

What’s happening?

7. Adaptive Crawling (New!)

Crawl4AI now includes intelligent adaptive crawling that automatically determines when sufficient information has been gathered. Here's a quick example:

What's special about adaptive crawling?

Learn more about Adaptive Crawling →

8. Multi-URL Concurrency (Preview)

If you need to crawl multiple URLs in parallel , you can use arun_many(). By default, Crawl4AI employs a MemoryAdaptiveDispatcher , automatically adjusting concurrency based on system resources. Here’s a quick glimpse:

The example above shows two ways to handle multiple URLs:

  1. Streaming mode (stream=True): Process results as they become available using async for
  2. Batch mode (stream=False): Wait for all results to complete

For more advanced concurrency (e.g., a semaphore-based approach, adaptive memory usage throttling , or customized rate limiting), see Advanced Multi-URL Crawling.

8. Dynamic Content Example

Some sites require multiple “page clicks” or dynamic JavaScript updates. Below is an example showing how to click a “Next Page” button and wait for new commits to load on GitHub, using BrowserConfig and CrawlerRunConfig :

Key Points :

9. Next Steps

Congratulations! You have:

If you’re ready for more, check out:

Crawl4AI is a powerful, flexible tool. Enjoy building out your scrapers, data pipelines, or AI-driven extraction flows. Happy crawling!

Page Copy Page Copy

ESC to close

python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:300])  # Print first 300 chars

if __name__ == "__main__":
    asyncio.run(main())
python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_conf = BrowserConfig(headless=True)  # or False to see the browser
    run_conf = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS
    )

    async with AsyncWebCrawler(config=browser_conf) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_conf
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())
python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

md_generator = DefaultMarkdownGenerator(
    content_filter=PruningContentFilter(threshold=0.4, threshold_type="fixed")
)

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    markdown_generator=md_generator
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://news.ycombinator.com", config=config)
    print("Raw Markdown length:", len(result.markdown.raw_markdown))
    print("Fit Markdown length:", len(result.markdown.fit_markdown))
python
from crawl4ai import JsonCssExtractionStrategy
from crawl4ai import LLMConfig

# Generate a schema (one-time cost)
html = "<div class='product'><h2>Gaming Laptop</h2><span class='price'>$999.99</span></div>"

# Using OpenAI (requires API token)
schema = JsonCssExtractionStrategy.generate_schema(
    html,
    llm_config = LLMConfig(provider="openai/gpt-4o",api_token="your-openai-token")  # Required for OpenAI
)

# Or using Ollama (open source, no token needed)
schema = JsonCssExtractionStrategy.generate_schema(
    html,
    llm_config = LLMConfig(provider="ollama/llama3.3", api_token=None)  # Not needed for Ollama
)

# Use the schema for fast, repeated extractions
strategy = JsonCssExtractionStrategy(schema)
python
import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def main():
    schema = {
        "name": "Example Items",
        "baseSelector": "div.item",
        "fields": [
            {"name": "title", "selector": "h2", "type": "text"},
            {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
        ]
    }

    raw_html = "<div class='item'><h2>Item 1</h2><a href='https://example.com/item1'>Link 1</a></div>"

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="raw://" + raw_html,
            config=CrawlerRunConfig(
                cache_mode=CacheMode.BYPASS,
                extraction_strategy=JsonCssExtractionStrategy(schema)
            )
        )
        # The JSON output is stored in 'extracted_content'
        data = json.loads(result.extracted_content)
        print(data)

if __name__ == "__main__":
    asyncio.run(main())
python
import os
import json
import asyncio
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
from crawl4ai import LLMExtractionStrategy

class OpenAIModelFee(BaseModel):
    model_name: str = Field(..., description="Name of the OpenAI model.")
    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
    output_fee: str = Field(
        ..., description="Fee for output token for the OpenAI model."
    )

async def extract_structured_data_using_llm(
    provider: str, api_token: str = None, extra_headers: Dict[str, str] = None
):
    print(f"\n--- Extracting Structured Data with {provider} ---")

    if api_token is None and provider != "ollama":
        print(f"API token is required for {provider}. Skipping this example.")
        return

    browser_config = BrowserConfig(headless=True)

    extra_args = {"temperature": 0, "top_p": 0.9, "max_tokens": 2000}
    if extra_headers:
        extra_args["extra_headers"] = extra_headers

    crawler_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        word_count_threshold=1,
        page_timeout=80000,
        extraction_strategy=LLMExtractionStrategy(
            llm_config = LLMConfig(provider=provider,api_token=api_token),
            schema=OpenAIModelFee.model_json_schema(),
            extraction_type="schema",
            instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. 
            Do not miss any models in the entire content.""",
            extra_args=extra_args,
        ),
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://openai.com/api/pricing/", config=crawler_config
        )
        print(result.extracted_content)

if __name__ == "__main__":

    asyncio.run(
        extract_structured_data_using_llm(
            provider="openai/gpt-4o", api_token=os.getenv("OPENAI_API_KEY")
        )
    )
python
import asyncio
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler

async def adaptive_example():
    async with AsyncWebCrawler() as crawler:
        adaptive = AdaptiveCrawler(crawler)

        # Start adaptive crawling
        result = await adaptive.digest(
            start_url="https://docs.python.org/3/",
            query="async context managers"
        )

        # View results
        adaptive.print_stats()
        print(f"Crawled {len(result.crawled_urls)} pages")
        print(f"Achieved {adaptive.confidence:.0%} confidence")

if __name__ == "__main__":
    asyncio.run(adaptive_example())
python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def quick_parallel_example():
    urls = [
        "https://example.com/page1",
        "https://example.com/page2",
        "https://example.com/page3"
    ]

    run_conf = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        stream=True  # Enable streaming mode
    )

    async with AsyncWebCrawler() as crawler:
        # Stream results as they complete
        async for result in await crawler.arun_many(urls, config=run_conf):
            if result.success:
                print(f"[OK] {result.url}, length: {len(result.markdown.raw_markdown)}")
            else:
                print(f"[ERROR] {result.url} => {result.error_message}")

        # Or get all results at once (default behavior)
        run_conf = run_conf.clone(stream=False)
        results = await crawler.arun_many(urls, config=run_conf)
        for res in results:
            if res.success:
                print(f"[OK] {res.url}, length: {len(res.markdown.raw_markdown)}")
            else:
                print(f"[ERROR] {res.url} => {res.error_message}")

if __name__ == "__main__":
    asyncio.run(quick_parallel_example())
python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def extract_structured_data_using_css_extractor():
    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
    schema = {
        "name": "KidoCode Courses",
        "baseSelector": "section.charge-methodology .w-tab-content > div",
        "fields": [
            {
                "name": "section_title",
                "selector": "h3.heading-50",
                "type": "text",
            },
            {
                "name": "section_description",
                "selector": ".charge-content",
                "type": "text",
            },
            {
                "name": "course_name",
                "selector": ".text-block-93",
                "type": "text",
            },
            {
                "name": "course_description",
                "selector": ".course-content-text",
                "type": "text",
            },
            {
                "name": "course_icon",
                "selector": ".image-92",
                "type": "attribute",
                "attribute": "src",
            },
        ],
    }

    browser_config = BrowserConfig(headless=True, java_script_enabled=True)

    js_click_tabs = """
    (async () => {
        const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");
        for(let tab of tabs) {
            tab.scrollIntoView();
            tab.click();
            await new Promise(r => setTimeout(r, 500));
        }
    })();
    """

    crawler_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(schema),
        js_code=[js_click_tabs],
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.kidocode.com/degrees/technology", config=crawler_config
        )

        companies = json.loads(result.extracted_content)
        print(f"Successfully extracted {len(companies)} companies")
        print(json.dumps(companies[0], indent=2))

async def main():
    await extract_structured_data_using_css_extractor()

if __name__ == "__main__":
    asyncio.run(main())

Simple Crawling

guide
Source: https://docs.crawl4ai.com/core/simple-crawling/

This guide covers the basics of web crawling with Crawl4AI, including setting up a crawler, making requests, understanding responses, and handling errors.

Basic Usage

Set up a simple crawl using BrowserConfig and CrawlerRunConfig:

python
import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async def main():
    browser_config = BrowserConfig()  # Default browser…

Understanding the Response

The arun() method returns a CrawlResult object with several useful properties. Here's a quick overview (see CrawlResult for complete details):

python
config = CrawlerRunConfig(
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.6),
        options={"ignore_links": True}
    )
)

result = await…

Adding Basic Options

Customize your crawl using CrawlerRunConfig:

python
run_config = CrawlerRunConfig(
    word_count_threshold=10,        # Minimum words per content block
    exclude_external_links=True,    # Remove external links
    remove_overlay_elements=True,   #…

Handling Errors

Always check if the crawl was successful:

python
run_config = CrawlerRunConfig()
result = await crawler.arun(url="https://example.com", config=run_config)

if not result.success:
    print(f"Crawl failed: {result.error_message}")
    print(f"Status…

Logging and Debugging

Enable verbose logging in BrowserConfig:

python
browser_config = BrowserConfig(verbose=True)

async with AsyncWebCrawler(config=browser_config) as crawler:
    run_config = CrawlerRunConfig()
    result = await…

Complete Example

Here's a more comprehensive example demonstrating common usage patterns:

python
import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_config = BrowserConfig(verbose=True)…

Core Crawling Concepts

Adaptive Web Crawling

guide
Source: https://docs.crawl4ai.com/core/adaptive-crawling/

Adaptive Web Crawling is a Crawl4AI guide to intelligent crawling, using coverage, consistency, and saturation metrics to decide when enough information has been collected. It covers configuration,…

Introduction

Traditional web crawlers follow predetermined patterns, crawling pages blindly without knowing when they've gathered enough information. Adaptive Crawling changes this paradigm by introducing…

Key Concepts

The Problem It Solves

When crawling websites for specific information, you face two challenges:

  1. Under-crawling: Stopping too early and missing crucial information
  2. Over-crawling: Wasting resources by crawling…

How It Works

The AdaptiveCrawler uses three metrics to measure information sufficiency:

Quick Start

Basic Usage

from crawl4ai import AsyncWebCrawler, AdaptiveCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        # Create an adaptive crawler (config is optional)
        adaptive =…

Configuration Options

from crawl4ai import AdaptiveConfig

config = AdaptiveConfig(
    confidence_threshold=0.8,    # Stop when 80% confident (default: 0.7)
    max_pages=30,               # Maximum pages to crawl…

Crawling Strategies

Adaptive Crawling supports two distinct strategies for determining information sufficiency:

Statistical Strategy (Default)

The statistical strategy uses pure information theory and term-based analysis:

# Default configuration uses statistical strategy
config = AdaptiveConfig(
    strategy="statistical",  # This is the default
    confidence_threshold=0.8
)

Embedding Strategy

The embedding strategy uses semantic embeddings for deeper understanding:

# Configure embedding strategy with local embeddings
config = AdaptiveConfig(
    strategy="embedding",
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",  # Default…

Strategy Comparison

Feature Statistical Embedding
Speed Very fast Moderate (API calls)
Cost Free Depends on provider
Accuracy Good for exact terms Excellent…

Embedding Strategy Configuration

config = AdaptiveConfig(
    strategy="embedding",

    # Model configuration
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    embedding_llm_config=None,  # Use for API-based…

Handling Irrelevant Queries

The embedding strategy can detect when a query is completely unrelated to the content:

# This will stop quickly with low confidence
result = await adaptive.digest(
    start_url="https://docs.python.org/3/",
    query="how to cook pasta"  # Irrelevant to Python docs
)

# Check if query…

When to Use Adaptive Crawling

Perfect For:

Understanding the Output

Confidence Score

The confidence score (0-1) indicates how sufficient the gathered information is:

Statistics Display

The summary shows:

adaptive.print_stats(detailed=False)  # Summary table
adaptive.print_stats(detailed=True)   # Detailed metrics

Persistence and Resumption

Saving Progress

config = AdaptiveConfig(
    save_state=True,
    state_path="my_crawl_state.json"
)

# Crawl will auto-save progress
result = await adaptive.digest(start_url, query)

Resuming a Crawl

# Resume from saved state
result = await adaptive.digest(
    start_url,
    query,
    resume_from="my_crawl_state.json"
)

Exporting Knowledge Base

# Export collected pages to JSONL
adaptive.export_knowledge_base("knowledge_base.jsonl")

# Import into another session
new_adaptive = AdaptiveCrawler(crawler)
await…

Best Practices

1. Query Formulation

2. Threshold Tuning

3. Performance Optimization

Examples

Research Assistant

# Gather information about a programming concept
result = await adaptive.digest(
    start_url="https://realpython.com",
    query="python decorators implementation patterns"
)

# Get the most…

Knowledge Base Builder

# Build a focused knowledge base about machine learning
queries = [
    "supervised learning algorithms",
    "neural network architectures",
    "model evaluation metrics"
]

for query in queries:…

API Documentation Crawler

# Intelligently crawl API documentation
config = AdaptiveConfig(
    confidence_threshold=0.85,  # Higher threshold for completeness
    max_pages=30
)

adaptive = AdaptiveCrawler(crawler,…

Next Steps

FAQ

Q: How is this different from traditional crawling? A: Traditional crawling follows fixed patterns (BFS/DFS). Adaptive crawling makes intelligent decisions about which links to follow and when to…

C4A-Script - Crawl4AI Documentation (v0.9.x)

guide
Source: https://docs.crawl4ai.com/core/c4a-script/

A comprehensive guide to C4A-Script, a human-readable DSL for web automation, covering syntax, commands, examples, and advanced features.

What is C4A-Script?

C4A-Script is a powerful, human-readable domain-specific language (DSL) designed for web automation and interaction. Think of it as a simplified programming language that anyone can read and write,…

text
# Navigate and interact in plain English
GO https://example.com
WAIT `#search-box` 5
TYPE "Hello World"
CLICK `button[type="submit"]`
Copy

Getting Started: Your First Script

Let's create a simple script that searches for something on a website:

That's it! In just a few lines, you've automated a complete search workflow.

text
# My first C4A-Script
GO https://duckduckgo.com

# Wait for the search box to appear
WAIT `input[name="q"]` 10

# Type our search query
TYPE "Crawl4AI"

# Press Enter to search
PRESS Enter

# Wait…

Interactive Tutorial & Live Demo

Want to learn by doing? We've got you covered:

🚀 Live Demo - Try C4A-Script in your browser right now!

**📁 [Tutorial…

bash
# Clone and navigate to the tutorial
cd docs/examples/c4a_script/tutorial/

# Install dependencies
pip install -r requirements.txt

# Launch the tutorial server
python server.py

# Open…

Core Concepts

Commands and Syntax

C4A-Script uses simple, English-like commands. Each command does one specific thing:

Selectors: Finding Elements

C4A-Script uses CSS selectors to identify elements on…

text
# Comments start with #
COMMAND parameter1 parameter2

# Most commands use CSS selectors in backticks
CLICK `#submit-button`

# Text content goes in quotes
TYPE "Hello, World!"

# Numbers are used…
text
# By ID
CLICK `#login-button`

# By class
CLICK `.submit-btn`

# By attribute
CLICK `button[type="submit"]`

# By accessible attributes
CLICK `button[aria-label="Search"][title="Search"]`

# Complex…
text
# Set a variable
SETVAR username = "[email protected]"
SETVAR password = "secret123"

# Use variables (prefix with $)
TYPE $username
PRESS Tab
TYPE $password
Copy

Command Categories

🧭 Navigation Commands

Move around the web like a user would:

Command Purpose Example
GO Navigate to URL GO https://example.com
RELOAD Refresh…

Real-World Examples

Example 1: Login Flow
Example 2: E-commerce Shopping
Example 3: Form Automation with Conditions
text
# Complete login automation
GO https://myapp.com/login

# Wait for page to load
WAIT `#login-form` 5

# Fill credentials
CLICK `#email`
TYPE "[email protected]"
PRESS Tab
TYPE "mypassword"

# Submit…
text
# Shopping automation with variables
SETVAR product = "laptop"
SETVAR budget = "1000"

GO https://shop.example.com
WAIT `#search-box` 3

# Search for product
TYPE $product
PRESS Enter
WAIT…
text
# Smart form filling with error handling
GO https://forms.example.com

# Check if user is already logged in
IF (EXISTS `.user-menu`) THEN GO https://forms.example.com/new
IF (NOT EXISTS `.user-menu`)…

Visual Programming with Blockly

C4A-Script includes a powerful visual programming interface built on Google Blockly. Perfect for:

Advanced Features

Recording Mode

The tutorial interface includes a recording feature that watches your browser interactions and automatically generates C4A-Script commands:

text
# Use comments for debugging
# This will wait up to 10 seconds for the element
WAIT `#slow-loading-element` 10

# Check if element exists before clicking
IF (EXISTS `#optional-button`) THEN CLICK…
python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

# Use C4A-Script for interaction before crawling
script = """
GO https://example.com
CLICK `#load-more-content`
WAIT `.dynamic-content`…

Best Practices

1. Always Wait for Elements
2. Use Descriptive Comments
3. Handle Variable Conditions
4. Use Variables for Reusability
text
# Bad: Clicking immediately
CLICK `#button`

# Good: Wait for element to appear
WAIT `#button` 5
CLICK `#button`
Copy
text
# Login to user account
GO https://myapp.com/login
WAIT `#login-form` 5

# Enter credentials
TYPE "[email protected]"
PRESS Tab
TYPE "password123"

# Submit and wait for redirect
CLICK…
text
# Handle different page states
IF (EXISTS `.cookie-banner`) THEN CLICK `.accept-cookies`
IF (EXISTS `.popup-modal`) THEN CLICK `.close-modal`

# Proceed with main workflow
CLICK `#main-action`
Copy
text
# Define once, use everywhere
SETVAR base_url = "https://myapp.com"
SETVAR test_email = "[email protected]"

GO $base_url/login
SET `#email` $test_email
Copy

Getting Help

What's Next?

Ready to dive deeper? Check out:

Crawl4AI Cache System and Migration Guide

guide
Source: https://docs.crawl4ai.com/core/cache-modes/

This page explains the new CacheMode enum introduced in Crawl4AI v0.5.0, which replaces old boolean cache flags, and provides migration examples and a mapping table for transitioning from legacy…

Overview

Starting from version 0.5.0, Crawl4AI introduces a new caching system that replaces the old boolean flags with a more intuitive CacheMode enum. This change simplifies cache control and makes the…

Old vs New Approach

The old system used multiple boolean flags:

Migration Example

Old Code (Deprecated)

New Code (Recommended)

python
from crawl4ai import AsyncWebCrawler

async def old_code(crawler: AsyncWebCrawler):
    # Legacy `bypass_cache` / `disable_cache` / `no_cache_read` / `no_cache_write`
    # were removed in v0.5+.…
python
import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode
from crawl4ai.async_configs import CrawlerRunConfig

async def use_proxy():
    # Use CacheMode in CrawlerRunConfig
    config =…

Common Migration Patterns

Legacy Flag Replacement
bypass_cache cache_mode=CacheMode.BYPASS
disable_cache cache_mode=CacheMode.DISABLED
no_cache_read …

Deep Crawling - Crawl4AI Documentation (v0.9.x)

guide
Source: https://docs.crawl4ai.com/core/deep-crawling/

This tutorial explains how to perform configurable deep crawling with Crawl4AI, covering BFS, DFS, and BestFirst strategies, streaming vs non-streaming results, filters, scorers, crash recovery,…

Introduction

One of Crawl4AI's most powerful features is its ability to perform configurable deep crawling that can explore websites beyond a single page. With fine-tuned control over crawl depth, domain…

1. Quick Example

Here's a minimal code snippet that implements a basic deep crawl using the BFSDeepCrawlStrategy:

What's happening?

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
from crawl4ai.content_scraping_strategy import…

2. Understanding Deep Crawling Strategy Options

2.1 BFSDeepCrawlStrategy (Breadth-First Search)

The BFSDeepCrawlStrategy uses a breadth-first approach, exploring all links at one depth before moving deeper:

Key parameters: -…

python
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

# Basic configuration
strategy = BFSDeepCrawlStrategy(
    max_depth=2,               # Crawl initial page + 2 levels deep…
python
from crawl4ai.deep_crawling import DFSDeepCrawlStrategy

# Basic configuration
strategy = DFSDeepCrawlStrategy(
    max_depth=2,               # Crawl initial page + 2 levels deep…
python
from crawl4ai.deep_crawling import BestFirstCrawlingStrategy
from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer

# Create a scorer
scorer = KeywordRelevanceScorer(…

3. Streaming vs. Non-Streaming Results

Crawl4AI can return results in two modes:

3.1 Non-Streaming Mode (Default)

When to use non-streaming mode:

python
config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1),
    stream=False  # Default behavior
)

async with AsyncWebCrawler() as crawler:
    # Wait for ALL results to be…
python
config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1),
    stream=True  # Enable streaming
)

async with AsyncWebCrawler() as crawler:
    # Returns an async iterator…

4. Filtering Content with Filter Chains

Filters help you narrow down which pages to crawl. Combine multiple filters using FilterChain for powerful targeting.

4.1 Basic URL Pattern Filter
4.2 Combining Multiple Filters

###…

python
from crawl4ai.deep_crawling.filters import FilterChain, URLPatternFilter

# Only follow URLs containing "blog" or "docs"
url_filter = URLPatternFilter(patterns=["*blog*", "*docs*"])

config =…
python
from crawl4ai.deep_crawling.filters import (
    FilterChain,
    URLPatternFilter,
    DomainFilter,
    ContentTypeFilter
)

# Create a chain of filters
filter_chain = FilterChain([
    # Only…

5. Using Scorers for Prioritized Crawling

Scorers assign priority values to discovered URLs, helping the crawler focus on the most relevant content first.

5.1 KeywordRelevanceScorer

How scorers work:

python
from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer
from crawl4ai.deep_crawling import BestFirstCrawlingStrategy

# Create a keyword relevance scorer
keyword_scorer =…

6. Advanced Filtering Techniques

6.1 SEO Filter for Quality Assessment

The SEOFilter helps you identify pages with strong SEO characteristics:

6.2 Content Relevance Filter

The ContentRelevanceFilter analyzes the…

python
from crawl4ai.deep_crawling.filters import FilterChain, SEOFilter

# Create an SEO filter that looks for specific keywords in page metadata
seo_filter = SEOFilter(
    threshold=0.5,  # Minimum score…
python
from crawl4ai.deep_crawling.filters import FilterChain, ContentRelevanceFilter

# Create a content relevance filter
relevance_filter = ContentRelevanceFilter(
    query="Web crawling and data…

7. Building a Complete Advanced Crawler

This example combines multiple techniques for a sophisticated crawl:

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
from crawl4ai.deep_crawling import…

8. Limiting and Controlling Crawl Size

8.1 Using max_pages

You can limit the total number of pages crawled with the max_pages parameter:

This feature is useful for:

python
# Limit to exactly 20 pages regardless of depth
strategy = BFSDeepCrawlStrategy(
    max_depth=3,
    max_pages=20
)
python
# Only follow links with scores above 0.4
strategy = DFSDeepCrawlStrategy(
    max_depth=2,
    url_scorer=KeywordRelevanceScorer(keywords=["api", "guide", "reference"]),
    score_threshold=0.4  #…

9. Common Pitfalls & Tips

  1. Set realistic limits. Be cautious with max_depth values > 3, which can exponentially increase crawl size. Use max_pages to set hard limits.

  2. Don't neglect the scoring component.…

python
config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2),
    preserve_https_for_internal_links=True  # Keep HTTPS even if server redirects to HTTP
)

10. Crash Recovery for Long-Running Crawls

For production deployments, especially in cloud environments where instances can be terminated unexpectedly, Crawl4AI provides built-in crash recovery support for all deep crawl strategies.

10.1…
python
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
import json

# Callback to save state after each URL
async def save_state_to_redis(state: dict):
    await redis.set("crawl_state",…
json
{
    "strategy_type": "bfs",  # or "dfs", "best_first"
    "visited": ["url1", "url2", ...],  # Already crawled URLs
    "pending": [{"url": "...", "parent_url": "..."}],  # Queue/stack…
python
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

# Load saved state (e.g., from Redis, database, or file)
saved_state =…
python
import json

captured_state = None

async def capture_state(state: dict):
    global captured_state
    captured_state = state

strategy = BFSDeepCrawlStrategy(
    max_depth=2,…
python
import asyncio
import json
import redis.asyncio as redis
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

REDIS_KEY =…

11. Cancellation Support for Deep Crawls

For production environments like cloud platforms, you often need to stop a running crawl mid-execution—whether the user changed their mind, specified the wrong URL, or wants to control costs.…

python
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def check_if_cancelled():
    # Check Redis, database, or any external source
    job = await redis.get(f"job:{job_id}")
    return…
python
strategy = BFSDeepCrawlStrategy(max_depth=3, max_pages=1000)

# In another coroutine or thread:
strategy.cancel()  # Thread-safe, stops before next URL
python
async with AsyncWebCrawler() as crawler:
    results = await crawler.arun(url, config=config)

if strategy.cancelled:
    print(f"Crawl was cancelled after {len(results)} pages")
else:…
python
async def handle_state(state: dict):
    if state.get("cancelled"):
        print("Crawl was cancelled!")
        print(f"Crawled {state['pages_crawled']} pages before cancellation")
    # Save state…
python
import asyncio
import json
import redis.asyncio as redis
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def…

12. Prefetch Mode for Fast URL Discovery

When you need to quickly discover URLs without full page processing, use prefetch mode . This is ideal for two-phase crawling where you first map the site, then selectively process specific…

python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

config = CrawlerRunConfig(prefetch=True)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://example.com",…
python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def two_phase_crawl(start_url: str):
    async with AsyncWebCrawler() as crawler:
        #…

13. Summary & Next Steps

In this Deep Crawling with Crawl4AI tutorial, you learned to:

Domain Mapping: Discover Every URL Under a Domain

reference
Source: https://docs.crawl4ai.com/core/domain-mapping/

This page describes DomainMapper in Crawl4AI, which discovers every URL under a domain using 8 discovery sources, including sitemaps, Common Crawl, Wayback Machine, Certificate Transparency, path…

What Is Domain Mapping?

Domain mapping goes beyond URL seeding. Instead of checking a single sitemap or index, DomainMapper combines 8 discovery sources to find every URL under a domain — including subdomains you…

DomainMapper vs AsyncUrlSeeder

Aspect AsyncUrlSeeder DomainMapper
Scope Single host, listed URLs only Entire domain + all subdomains
Sources Sitemap + Common Crawl 8 sources…

Quick Start

Or via AsyncWebCrawler:

python
import asyncio
from crawl4ai import DomainMapper, DomainMapperConfig

async def main():
    async with DomainMapper() as mapper:
        results = await mapper.scan("example.com")

    print(f"Found…
python
from crawl4ai import AsyncWebCrawler, DomainMapperConfig

async with AsyncWebCrawler() as crawler:
    results = await crawler.amap_domain("example.com")

The 8 Discovery Sources

DomainMapper combines these sources, each catching URLs the others miss:

1. `sitemap` — Sitemap Discovery

Checks /sitemap.xml, /sitemap_index.xml, and robots.txt Sitemap: directives on every discovered host — not just the root domain.

python
config = DomainMapperConfig(source="sitemap")

2. `cc` — Common Crawl

Queries the Common Crawl CDX API for *.domain.tld/*, catching URLs and subdomains the web's largest public crawl has indexed.

python
config = DomainMapperConfig(source="cc")

3. `wayback` — Wayback Machine

Queries the Internet Archive's CDX API. Often has different coverage than Common Crawl — including historical pages that have since been removed.

python
config = DomainMapperConfig(source="wayback")

4. `crt` — Certificate Transparency

Queries crt.sh for SSL certificates issued to *.domain.tld. This is the single most effective subdomain discovery technique — it found 14 subdomains for superdesign.dev that no…

python
config = DomainMapperConfig(source="crt")

5. `probe` — Common Path Probing

Tries ~25 well-known paths on each discovered host (/docs, /api, /login, /dashboard, /openapi.json, etc.). Combined with soft-404 detection to avoid false positives.

python
config = DomainMapperConfig(source="probe")

# Add custom paths to probe
config = DomainMapperConfig(
    source="probe",
    probe_paths=["/custom-api", "/internal/status"]
)

6. `robots` — robots.txt Path Mining

Parses Disallow: and Allow: lines from robots.txt. These are confirmed real paths the site acknowledges exist — often revealing admin panels, APIs, and internal tools that aren't linked…

python
config = DomainMapperConfig(source="robots")

7. `feed` — RSS/Atom Feed Parsing

Discovers and parses RSS/Atom feeds at common paths (/feed, /rss, /atom.xml, etc.). Feeds are curated lists of content URLs maintained by the site.

python
config = DomainMapperConfig(source="feed")

Fetches each host's homepage via HTTP and extracts all internal links using quick_extract_links(). Also mines <link rel="alternate|preload|prefetch"> tags from the `` for additional URLs.…

python
config = DomainMapperConfig(source="homepage")

Combining Sources

Sources are combined with +:

python
# Default: most useful combination
config = DomainMapperConfig(source="sitemap+cc+crt+probe")

# Maximum coverage: all 8 sources
config = DomainMapperConfig(…

How It Works: The Three Phases

Phase 1: Host Discovery

DomainMapper first discovers all subdomains under your domain:

Each discovered host is validated with an HTTP HEAD request. Hosts that don't respond are dropped.

text
superdesign.dev
├── crt.sh           → docs, app, cloud, insights, staging-api, ui2web, ...
├── Wayback CDX      → api, app, docs, www, ...
├── Common Crawl     → app, www, ...
└── DNS guessing     →…

Phase 2: Per-Host Scanning

For each validated host, DomainMapper runs all enabled sources in parallel:

text
docs.superdesign.dev
├── Soft-404 fingerprint  → (404 returns proper error — no SPA issue)
├── robots.txt            → 1 sitemap URL, 1 disallow path
├── Sitemap parsing       → 19 URLs
├── Path…

Phase 3: Post-Processing

All discovered URLs go through:

Soft-404 Detection

Many modern SPAs return HTTP 200 for every URL — even pages that don't exist. DomainMapper detects this:

python
# Soft-404 detection is on by default
config = DomainMapperConfig(soft_404_detection=True)

# Disable if you want raw results
config = DomainMapperConfig(soft_404_detection=False)

Configuration Reference

DomainMapperConfig

Parameter Type Default Description
source str "sitemap+cc+crt+probe" Discovery sources joined by +
max_urls int -1 Maximum URLs to…

Output Format

Each result is a dict:

json
{
    "url": "https://docs.superdesign.dev/quickstart",
    "host": "docs.superdesign.dev",
    "source": "homepage+sitemap",     # which source(s) found it
    "status": "valid",                #…

Practical Examples

Discover and Crawl Documentation

python
import asyncio
from crawl4ai import AsyncWebCrawler, DomainMapperConfig, CrawlerRunConfig

async def crawl_all_docs():
    async with AsyncWebCrawler() as crawler:
        # Step 1: Discover all…

Security Audit: Find Exposed Services

python
async def audit_domain():
    async with DomainMapper() as mapper:
        results = await mapper.scan("company.com", DomainMapperConfig(
            source="crt+probe+robots",…

Compare Subdomains Across a Domain

python
async def map_infrastructure():
    async with DomainMapper() as mapper:
        results = await mapper.scan("company.com", DomainMapperConfig(
            source="crt+probe",…

Tips and Best Practices

See Also

Parameters

NameTypeDescriptionDefaultRequired
sourcestrDiscovery sources joined by `+`sitemap+cc+crt+probeNo
max_urlsintMaximum URLs to return (-1 = unlimited)-1No
concurrencyintMax concurrent requests across all hosts50No
hits_per_secintRate limit in requests/second10No
forceboolBypass all cachesFalseNo
extract_headboolFetch and parse `<head>` metadataTrueNo
filter_nonsense_urlsboolFilter static assets and utility URLsTrueNo
soft_404_detectionboolFingerprint and filter soft-404 pagesTrueNo
querystrBM25 relevance query (requires `extract_head=True`)NoneNo
score_thresholdfloatMinimum relevance score (0.0-1.0)NoneNo
scoring_methodstrScoring algorithmbm25No
probe_pathsList[str]Extra paths to probe on each hostNoneNo
common_subdomainsList[str]Extra subdomain prefixes to guessNoneNo
use_browser_for_homepageboolUse Playwright for JS-rendered homepagesFalseNo
verboseboolOverride logger verbose settingNoneNo
cache_ttl_hoursintHours before cached results expire24No
dns_timeoutfloatTimeout for DNS resolution (seconds)3.0No
http_timeoutfloatTimeout for HTTP requests (seconds)10.0No

Link & Media - Crawl4AI Documentation (v0.9.x)

guide
Source: https://docs.crawl4ai.com/core/link-media/

This tutorial covers how to extract and filter links (internal/external) and media (images, videos, audio) from crawled pages using Crawl4AI, including advanced link head extraction with scoring,…

When you call arun() or arun_many() on a URL, Crawl4AI automatically extracts links and stores them in the links field of CrawlResult. By default, the crawler tries to distinguish…

python
from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://www.example.com")
    if result.success:
        internal_links =…
python
result.links = {
  "internal": [
    {
      "href": "https://kidocode.com/",
      "text": "",
      "title": "",
      "base_domain": "kidocode.com"
    },
    {
      "href":…

Ever wanted to not just extract links, but also get the actual content (title, description, metadata) from those linked pages? And score them for relevance? This is exactly what Link Head Extraction…

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import LinkPreviewConfig

async def extract_link_heads_example():
    """
    Complete example showing link head…
text
✅ Successfully crawled: https://docs.python.org/3/
📄 Page title: 3.13.5 Documentation
🔗 Found 53 internal links
🌍 Found 1 external links
🧠 Links with head data extracted: 10

🏆 Top 3 Links with Full…
python
from crawl4ai import LinkPreviewConfig

link_preview_config = LinkPreviewConfig(
    # BASIC SETTINGS
    verbose=True,                    # Show detailed logs (recommended for learning)

    # LINK…
python
# High intrinsic score indicators:
# ✅ Clean URL structure (docs.python.org/api/reference)
# ✅ Meaningful link text ("API Reference Guide")
# ✅ Relevant to page context
# ✅ Not buried deep in…
python
# Example: query = "machine learning tutorial"
# High contextual score: Link to "Complete Machine Learning Guide"
# Low contextual score: Link to "Privacy Policy"
python
# When both scores available: (intrinsic * 0.3) + (contextual * 0.7)
# When only intrinsic: uses intrinsic score
# When only contextual: uses contextual score
# When neither: not calculated
python
async def research_assistant():
    config = CrawlerRunConfig(
        link_preview_config=LinkPreviewConfig(
            include_internal=True,
            include_external=True,…
python
async def api_discovery():
    config = CrawlerRunConfig(
        link_preview_config=LinkPreviewConfig(
            include_internal=True,
            include_patterns=["*/api/*", "*/reference/*"],…
python
async def quality_analysis():
    config = CrawlerRunConfig(
        link_preview_config=LinkPreviewConfig(
            include_internal=True,
            max_links=200,
            concurrency=20,…
python
# Check your configuration:
config = CrawlerRunConfig(
    link_preview_config=LinkPreviewConfig(
        verbose=True   # ← Enable to see what's happening
    )
)
python
# Make sure scoring is enabled:
config = CrawlerRunConfig(
    score_links=True,  # ← Enable intrinsic scoring
    link_preview_config=LinkPreviewConfig(
        query="your search terms"  # ← For…
python
# Optimize performance:
link_preview_config = LinkPreviewConfig(
    max_links=20,      # ← Reduce number
    concurrency=10,    # ← Increase parallelism
    timeout=3,         # ← Shorter timeout…

3. Domain Filtering

Some websites contain hundreds of third-party or affiliate links. You can filter out certain domains at crawl time by configuring the crawler. The most relevant parameters in CrawlerRunConfig…

python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    crawler_cfg = CrawlerRunConfig(
        exclude_external_links=True,          # No links…
python
crawler_cfg = CrawlerRunConfig(
    exclude_domains=["suspiciousads.com"]
)

4. Media Extraction

4.1 Accessing result.media

By default, Crawl4AI collects images, audio and video URLs it finds on the page. These are stored in result.media, a dictionary keyed by media type (e.g.,…

python
if result.success:
    # Get images
    images_info = result.media.get("images", [])
    print(f"Found {len(images_info)} images in total.")
    for i, img in enumerate(images_info[:3]):  # Inspect…
python
result.media = {
  "images": [
    {
      "src": "https://cdn.prod.website-files.com/.../Group%2089.svg",
      "alt": "coding school for kids",
      "desc": "Trial Class Degrees degrees All…
python
crawler_cfg = CrawlerRunConfig(
    exclude_external_images=True
)
python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    crawler_cfg = CrawlerRunConfig(
        capture_mhtml=True  # Enable MHTML capture
    )

    async with…

Here’s a combined example demonstrating how to filter out external links, skip certain domains, and exclude external images:

python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    # Suppose we want to keep only internal links, remove certain domains, 
    # and discard…

6. Common Pitfalls & Tips

  1. Conflicting Flags:
    • exclude_external_links=True but then also specifying exclude_social_media_links=True is typically fine, but understand that the first setting already discards all…

Prefix-Based Input Handling in Crawl4AI

guide
Source: https://docs.crawl4ai.com/core/local-files/

This guide demonstrates how to use Crawl4AI to crawl web URLs, local HTML files, and raw HTML strings using prefix-based input handling with the unified `url` parameter and `CrawlerRunConfig`.

Crawling a Web URL

To crawl a live web page, provide the URL starting with http:// or https://, using a CrawlerRunConfig object:

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def crawl_web():
    config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as…

Crawling a Local HTML File

To crawl a local HTML file, prefix the file path with file://.

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def crawl_local_file():
    local_file_path = "/path/to/apple.html"  # Replace with your file path
    file_url…

Crawling Raw HTML Content

To crawl raw HTML content, prefix the HTML string with raw:.

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode
from crawl4ai.async_configs import CrawlerRunConfig

async def crawl_raw_html():
    raw_html = "<html><body><h1>Hello,…

Complete Example

Below is a comprehensive script that:

import os
import sys
import asyncio
from pathlib import Path
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def main():
    wikipedia_url =…

Conclusion

With the unified url parameter and prefix-based handling in Crawl4AI , you can seamlessly handle web URLs, local HTML files, and raw HTML content. Use CrawlerRunConfig for flexible and…

Page Interaction - Crawl4AI Documentation (v0.9.x)

guide
Source: https://docs.crawl4ai.com/core/page-interaction/

This page explains how to interact with dynamic webpages using Crawl4AI, covering JavaScript execution, wait conditions, multi-step flows, Shadow DOM flattening, and virtual scrolling.

Page Interaction

Crawl4AI provides powerful features for interacting with dynamic webpages, handling JavaScript execution, waiting for conditions, and managing multi-step flows. By combining js_code,…

1. JavaScript Execution

Basic Execution

js_code in CrawlerRunConfig accepts either a single JS string or a list of JS snippets. It runs after wait_for and delay_before_return_html — so the page is…

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Single JS command
    config = CrawlerRunConfig(
        js_code="window.scrollTo(0,…
text
1. Page navigation (page.goto)
2. js_code_before_wait     ← triggers loading / clicks tabs
3. wait_for                ← waits for content to appear
4. delay_before_return_html ← extra safety…
python
config = CrawlerRunConfig(
    # Click a tab first
    js_code_before_wait="document.querySelector('#specs-tab')?.click();",
    # Then wait for the tab content to appear…

2. Wait Conditions

2.1 CSS-Based Waiting

Sometimes, you just want to wait for a specific element to appear. For example:

Key param:

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    config = CrawlerRunConfig(
        # Wait for at least 30 items on Hacker News…
python
wait_condition = """() => {
    const items = document.querySelectorAll('.athing');
    return items.length > 50;  // Wait for at least 51 items
}"""

config =…

3. Handling Dynamic Content

Many modern sites require multiple steps: scrolling, clicking “Load More,” or updating via JavaScript. Below are typical patterns.

3.1 Load More Example (Hacker News “More” Link)

**Key…

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Step 1: Load initial Hacker News page
    config = CrawlerRunConfig(…
python
js_form_interaction = """
document.querySelector('#your-search').value = 'TypeScript commits';
document.querySelector('form').submit();
"""

config = CrawlerRunConfig(…

4. Timing Control

  1. page_timeout (ms): Overall page load or script execution time limit.
  2. delay_before_return_html (seconds): Wait an extra moment before capturing the final HTML.
  3. mean_delay &…
python
config = CrawlerRunConfig(
    page_timeout=60000,  # 60s limit
    delay_before_return_html=2.5
)

5. Multi-Step Interaction Example

Below is a simplified script that does multiple “Load More” clicks on GitHub’s TypeScript commits page. It re-uses the same session to accumulate new commits each time. The code includes the…

python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def multi_page_commits():
    browser_cfg = BrowserConfig(
        headless=False,  # Visible…

6. Combine Interaction with Extraction

Once dynamic content is loaded, you can attach an extraction_strategy (like JsonCssExtractionStrategy or LLMExtractionStrategy). For example:

When done, check result.extracted_content…

python
from crawl4ai import JsonCssExtractionStrategy

schema = {
    "name": "Commits",
    "baseSelector": "li.Box-sc-g0xbh4-0",
    "fields": [
        {"name": "title", "selector": "h4.markdown-title",…

7. Shadow DOM Flattening

Sites built with Web Components (Stencil, Lit, Shoelace, etc.) render content inside Shadow DOM — an encapsulated sub-tree that is invisible to normal page serialization. Set…

python
config = CrawlerRunConfig(
    flatten_shadow_dom=True,
    wait_until="load",
    delay_before_return_html=3.0,  # give components time to hydrate
)

8. Relevant `CrawlerRunConfig` Parameters

Below are the key interaction-related parameters in CrawlerRunConfig. For a full list, see Configuration Parameters.

9. Conclusion

Crawl4AI's page interaction features let you:

  1. Execute JavaScript for scrolling, clicks, or form filling.
  2. Wait for CSS or custom JS conditions before capturing data.
  3. Handle…

10. Virtual Scrolling

For sites that use virtual scrolling (where content is replaced rather than appended as you scroll, like Twitter or Instagram), Crawl4AI provides a dedicated VirtualScrollConfig:

Virtual…
python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig

async def crawl_twitter_timeline():
    # Configure virtual scroll for Twitter-like feeds
    virtual_config =…

URL Seeding: The Smart Way to Crawl at Scale

guide
Source: https://docs.crawl4ai.com/core/url-seeding/

This page explains how to use URL seeding to discover and filter URLs before crawling, covering configuration, smart filtering with BM25 scoring, and scaling across multiple domains.

Why URL Seeding?

Web crawling comes in different flavors, each with its own strengths. Let's understand when to use URL seeding versus deep crawling.

Deep Crawling: Real-Time Discovery

Deep crawling is perfect when you need:

python
# Deep crawling example: Explore a website dynamically
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def…

URL Seeding: Bulk Discovery

URL seeding shines when you want:

python
# URL seeding example: Analyze all documentation
from crawl4ai import AsyncUrlSeeder, SeedingConfig

seeder = AsyncUrlSeeder()
config = SeedingConfig(
    source="sitemap",
    extract_head=True,…

The Trade-offs

Aspect Deep Crawling URL Seeding
Coverage Discovers pages dynamically Gets most existing URLs instantly
Freshness Finds brand new pages May miss very…

When to Use Each

Choose Deep Crawling when:

Your First URL Seeding Adventure

Let's see the magic in action. We'll discover blog posts about Python, filter for tutorials, and crawl only those pages.

What just happened?

python
import asyncio
from crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, SeedingConfig, CrawlerRunConfig

async def smart_blog_crawler():
    # Step 1: Create our URL discoverer
    seeder =…

Understanding the URL Seeder

Now that you've seen the magic, let's understand how it works.

Basic Usage

Creating a URL seeder is simple:

The seeder can discover URLs from two powerful sources:

python
from crawl4ai import AsyncUrlSeeder

# Method 1: Manual cleanup
seeder = AsyncUrlSeeder()
try:
    config = SeedingConfig(source="sitemap")
    urls = await seeder.urls("example.com",…

1. Sitemaps (Fastest)

Sitemaps are XML files that websites create specifically to list all their URLs. It's like getting a menu at a restaurant - everything is listed upfront.

Sitemap Index Support: For large…

python
# Discover from sitemap
config = SeedingConfig(source="sitemap")
urls = await seeder.urls("example.com", config)
xml
<!-- Example sitemap index -->
<sitemapindex>
  <sitemap>
    <loc>https://techcrunch.com/sitemap-1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://techcrunch.com/sitemap-2.xml</loc>…

2. Common Crawl (Most Comprehensive)

Common Crawl is a massive public dataset that regularly crawls the entire web. It's like having access to a pre-built index of the internet.

python
# Discover from Common Crawl
config = SeedingConfig(source="cc")
urls = await seeder.urls("example.com", config)

3. Both Sources (Maximum Coverage)

python
# Use both sources
config = SeedingConfig(source="sitemap+cc")
urls = await seeder.urls("example.com", config)

Configuration Magic: SeedingConfig

The SeedingConfig object is your control panel. Here's everything you can configure:

Pattern Matching Examples

python
# Match all blog posts
config = SeedingConfig(pattern="*/blog/*")

# Match only HTML files
config = SeedingConfig(pattern="*.html")

# Match product pages
config =…

URL Validation: Live Checking

Sometimes you need to know if URLs are actually accessible. That's where live checking comes in:

When to use live checking:

python
config = SeedingConfig(
    source="sitemap",
    live_check=True,  # Verify each URL is accessible
    concurrency=20    # Check 20 URLs in parallel
)
async with AsyncUrlSeeder() as seeder:
    urls…

The Power of Metadata: Head Extraction

This is where URL seeding gets really powerful. Instead of crawling entire pages, you can extract just the metadata:

python
config = SeedingConfig(
    extract_head=True  # Extract metadata from <head> section
)
async with AsyncUrlSeeder() as seeder:
    urls = await seeder.urls("example.com", config)

# Now each URL has…

What Can We Extract?

The head extraction gives you a treasure trove of information:

python
# Example of extracted head_data
{
    "title": "10 Python Tips for Beginners",
    "charset": "utf-8",
    "lang": "en",
    "meta": {
        "description": "Learn essential Python tips...",…

Smart URL-Based Filtering (No Head Extraction)

When extract_head=False but you still provide a query, the seeder uses intelligent URL-based scoring:

This approach is much faster than head extraction while still providing intelligent filtering!

python
# Fast filtering based on URL structure alone
config = SeedingConfig(
    source="sitemap",
    extract_head=False,  # Don't fetch page metadata
    query="python tutorial async",…

Understanding Results

Each URL in the results has this structure:

Let's see a real example:

python
{
    "url": "https://example.com/blog/python-tips.html",
    "status": "valid",        # "valid", "not_valid", or "unknown"
    "head_data": {            # Only if extract_head=True
        "title":…
python
config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    live_check=True
)
async with AsyncUrlSeeder() as seeder:
    urls = await seeder.urls("blog.example.com", config)

# Analyze…

Smart Filtering with BM25 Scoring

Now for the really cool part - intelligent filtering based on relevance!

Introduction to Relevance Scoring

BM25 is a ranking algorithm that scores how relevant a document is to a search query. With URL seeding, we can score URLs based on their metadata before crawling them.

Think of it like this: -…

Query-Based Discovery

Here's how to use BM25 scoring:

python
config = SeedingConfig(
    source="sitemap",
    extract_head=True,           # Required for scoring
    query="python async tutorial",  # What we're looking for
    scoring_method="bm25",       #…

Real Examples

Finding Documentation Pages

python
# Find API documentation
config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    query="API reference documentation endpoints",
    scoring_method="bm25",
    score_threshold=0.5,…

Discovering Product Pages

python
# Find specific products
config = SeedingConfig(
    source="sitemap+cc",  # Use both sources
    extract_head=True,
    query="wireless headphones noise canceling",
    scoring_method="bm25",…

Filtering News Articles

python
# Find recent news about AI
config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    query="artificial intelligence machine learning breakthrough",
    scoring_method="bm25",…

Complex Query Patterns

python
# Multi-concept queries
queries = [
    "python async await concurrency tutorial",
    "data science pandas numpy visualization",
    "web scraping beautifulsoup selenium automation",
    "machine…

Scaling Up: Multiple Domains

When you need to discover URLs across multiple websites, URL seeding really shines.

The `many_urls` Method

python
# Discover URLs from multiple domains in parallel
domains = ["site1.com", "site2.com", "site3.com"]

config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    query="python tutorial",…

Cross-Domain Examples

Competitor Analysis

python
# Analyze content strategies across competitors
competitors = [
    "competitor1.com",
    "competitor2.com", 
    "competitor3.com"
]

config = SeedingConfig(
    source="sitemap",…

Industry Research

python
# Research Python tutorials across educational sites
educational_sites = [
    "realpython.com",
    "pythontutorial.net",
    "learnpython.org",
    "python.org"
]

config = SeedingConfig(…

Multi-Site Monitoring

python
# Monitor news about your company across multiple sources
news_sites = [
    "techcrunch.com",
    "theverge.com",
    "wired.com",
    "arstechnica.com"
]

company_name = "YourCompany"

config =…

Advanced Integration Patterns

Let's put everything together in a real-world example.

Building a Research Assistant

Here's a complete example that discovers, scores, filters, and crawls intelligently:

python
import asyncio
from datetime import datetime
from crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, SeedingConfig, CrawlerRunConfig

class ResearchAssistant:
    def __init__(self):…

Performance Optimization Tips

python
# First run - populate cache
config = SeedingConfig(source="sitemap", extract_head=True, force=True)
urls = await seeder.urls("example.com", config)

# Subsequent runs - use cache (much…
python
# For many small requests (like HEAD checks)
config = SeedingConfig(concurrency=50, hits_per_sec=20)

# For fewer large requests (like full head extraction)
config = SeedingConfig(concurrency=10,…
python
# When crawling many URLs
async with AsyncWebCrawler() as crawler:
    # Assuming urls is a list of URL strings
    crawl_results = await crawler.arun_many(urls, config=config)

    # Process as they…
python
# Safe for domains with 1M+ URLs
config = SeedingConfig(
    source="cc+sitemap",
    concurrency=50,  # Queue size adapts to concurrency
    max_urls=100000  # Process in batches if needed
)

# The…

Best Practices & Tips

Cache Management

The seeder automatically caches results to speed up repeated operations:

text
- **Common Crawl cache** : `~/.crawl4ai/seeder_cache/[index]_[domain]_[hash].jsonl`
- **Sitemap cache** : `~/.crawl4ai/seeder_cache/sitemap_[domain]_[hash].json`
- **HEAD data cache** :…

Smart TTL Cache for Sitemaps

Sitemap caches now include intelligent validation:

Cache validation priority:

  1. force=True → Always refetch
  2. Cache doesn't exist → Fetch fresh
  3. validate_sitemap_lastmod=True and…
python
# Default: 24-hour TTL with lastmod validation
config = SeedingConfig(
    source="sitemap",
    cache_ttl_hours=24,              # Cache expires after 24 hours
    validate_sitemap_lastmod=True    #…

Pattern Matching Strategies

python
# Be specific when possible
good_pattern = "*/blog/2024/*.html"  # Specific
bad_pattern = "*"                     # Too broad

# Combine patterns with metadata filtering
config = SeedingConfig(…

Rate Limiting Considerations

python
# Be respectful of servers
config = SeedingConfig(
    hits_per_sec=10,      # Max 10 requests per second
    concurrency=20        # But use 20 workers
)

# For your own servers
config =…

Quick Reference

Common Patterns

python
# Blog post discovery
config = SeedingConfig(
    source="sitemap",
    pattern="*/blog/*",
    extract_head=True,
    query="your topic",
    scoring_method="bm25"
)

# E-commerce product…

Troubleshooting Guide

Issue Solution
No URLs found Try source="cc+sitemap", check domain spelling
Slow discovery Reduce concurrency, add hits_per_sec limit
Missing metadata Ensure…

Performance Benchmarks

Typical performance on a standard connection:

Conclusion

URL seeding transforms web crawling from a blind expedition into a surgical strike. By discovering and analyzing URLs before crawling, you can:

Smart URL Filtering

The seeder automatically filters out nonsense URLs that aren't useful for content crawling:

To disable filtering (not recommended):

python
# Enabled by default
config = SeedingConfig(
    source="sitemap",
    filter_nonsense_urls=True  # Default: True
)

# URLs that get filtered:
# - robots.txt, sitemap.xml, ads.txt
# - API endpoints…
python
config = SeedingConfig(
    source="sitemap",
    filter_nonsense_urls=False  # Include ALL URLs
)

Key Features Summary

Need More Coverage?

If you need to discover URLs across an entire domain — including subdomains, hidden services, and pages not listed in any sitemap — check out Domain Mapping. It combines 8…

Parameters

NameTypeDescriptionDefaultRequired
sourcestrURL source: "cc" (Common Crawl), "sitemap", or "sitemap+cc""sitemap+cc"No
patternstrURL pattern filter (e.g., "*/blog/*", "*.html")"*"No
extract_headboolExtract metadata from page <head>FalseNo
live_checkboolVerify URLs are accessibleFalseNo
max_urlsintMaximum URLs to return (-1 = unlimited)-1No
concurrencyintParallel workers for fetching10No
hits_per_secintRate limit for requests5No
forceboolBypass cache, fetch fresh dataFalseNo
verboseboolShow detailed progressFalseNo
querystrSearch query for BM25 scoringNoneNo
scoring_methodstrScoring method (currently "bm25")NoneNo
score_thresholdfloatMinimum score to include URLNoneNo
filter_nonsense_urlsboolFilter out utility URLs (robots.txt, etc.)TrueNo
cache_ttl_hoursintHours before sitemap cache expires (0 = no TTL)24No
validate_sitemap_lastmodboolCheck sitemap's lastmod and refetch if newerTrueNo

Content & Output

Content Selection

api
Source: https://docs.crawl4ai.com/core/content-selection/

This page explains how to select, filter, and refine content from crawls using CrawlerRunConfig parameters, including CSS selectors, content filtering, iframe handling, shadow DOM flattening, and…

1. CSS-Based Selection

Crawl4AI provides multiple ways to select, filter, and refine the content from your crawls. Whether you need to target a specific CSS region, exclude entire tags, filter out external links, or remove…

1.1 Using css_selector

A straightforward way to limit your crawl results to a certain region of the page is css_selector in CrawlerRunConfig:

Result: Only elements matching that selector remain in result.cleaned_html.

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    config = CrawlerRunConfig(
        # e.g., first 30 items from Hacker News…

1.2 Using target_elements

The target_elements parameter provides more flexibility by allowing you to target multiple elements for content extraction while preserving the entire page context for other features:

Key…

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    config = CrawlerRunConfig(
        # Target article body and sidebar, but not other content…

2. Content Filtering & Exclusions

2.1 Basic Overview

Explanation:

python
config = CrawlerRunConfig(
    # Content thresholds
    word_count_threshold=10,        # Minimum words per block

    # Tag exclusions
    excluded_tags=['form', 'header', 'footer', 'nav'],

    #…
python
[
    'facebook.com',
    'twitter.com',
    'x.com',
    'linkedin.com',
    'instagram.com',
    'pinterest.com',
    'tiktok.com',
    'snapchat.com',
    'reddit.com',
]

2.2 Example Usage

Note: If these parameters remove too much, reduce or disable them accordingly.

python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def main():
    config = CrawlerRunConfig(
        css_selector="main.content",…

3. Handling Iframes

Some sites embed content in