Crawl Result and Output

Summary: Describes the CrawlResult object returned by Crawl4AI's arun() method, including all fields, markdown generation, structured extraction, and additional outputs like links, media, tables, screenshots,…

1. The `CrawlResult` Model

Below is the core schema. Each field captures a different aspect of the crawl’s result:

```
class MarkdownGenerationResult(BaseModel):
    raw_markdown: str
    markdown_with_citations: str…

class MarkdownGenerationResult(BaseModel):
    raw_markdown: str
    markdown_with_citations: str
    references_markdown: str
    fit_markdown: Optional[str] = None
    fit_html: Optional[str] =…

2. HTML Variants

###### `html`: Raw HTML

Crawl4AI preserves the exact HTML as `result.html`. Useful for:

- Debugging page issues or checking the original content.
- Performing your own specialized parse if…

config = CrawlerRunConfig(
    excluded_tags=["form", "header", "footer"],
    keep_data_attributes=False
)
result = await crawler.arun("https://example.com",…

3. Markdown Generation

###### 3.1 `markdown`

- **`markdown`**: The current location for detailed markdown output, returning a **`MarkdownGenerationResult`** object.
- **`markdown_v2`**: Removed in v0.5. Accessing it now…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

config = CrawlerRunConfig(…

4. Structured Extraction: `extracted_content`

If you run a JSON-based extraction strategy (CSS, XPath, LLM, etc.), the structured data is **not** stored in `markdown`—it’s placed in **`result.extracted_content`** as a JSON string (or sometimes…

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def main():
    schema = {
        "name": "Example…

5. More Fields: Links, Media, Tables and More

###### 5.1 `links`

A dictionary, typically with `"internal"` and `"external"` lists. Each entry might have `href`, `text`, `title`, etc. This is automatically captured if you haven’t disabled link…

print(result.links["internal"][:3])  # Show first 3 internal links

images = result.media.get("images", [])
for img in images:
    print("Image URL:", img["src"], "Alt:", img.get("alt"))

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(…

config = CrawlerRunConfig(
    table_score_threshold=5  # Lower value = more tables detected (default: 7)
)

# Save the PDF
with open("page.pdf", "wb") as f:
    f.write(result.pdf)

# Save the MHTML
if result.mhtml:
    with open("page.mhtml", "w", encoding="utf-8") as f:
        f.write(result.mhtml)

6. Accessing These Fields

After you run:

```
result = await crawler.arun(url="https://example.com", config=some_config)
```

Check any field:

```
if result.success:
    print(result.status_code, result.response_headers)…

if result.success:
    print(result.status_code, result.response_headers)
    print("Links found:", len(result.links.get("internal", [])))
    if result.markdown:
        print("Markdown snippet:",…

7. Next Steps

- **Markdown Generation**: Dive deeper into how to configure `DefaultMarkdownGenerator` and various filters.
- **Content Filtering**: Learn how to use `BM25ContentFilter` and…

url

The final or actual URL crawled (in case of redirects).

html

Original, unmodified page HTML. Good for debugging or custom processing.

Preprocessed HTML optimized for extraction and content filtering.

success

True if the crawl completed without major errors, else False.

Sanitized HTML with scripts/styles removed; can exclude tags if configured via excluded_tags etc.

media

Extracted media info (images, audio, etc.), each with attributes like src, alt, score, etc.

links

Extracted link data, split by internal and external. Each link usually has href, text, etc.

If accept_downloads=True in BrowserConfig, this lists the filepaths of saved downloads.

Results from JavaScript execution during crawling.

screenshot

Screenshot of the page (base64-encoded) if screenshot=True.

pdf

PDF of the page if pdf=True.

mhtml

MHTML snapshot of the page if capture_mhtml=True. Contains the full page with all resources.

markdown

It holds a MarkdownGenerationResult. Over time, this will be consolidated into markdown. The generator can provide raw markdown, citations, references, and optionally fit_markdown.

The output of a structured extraction (CSS/LLM-based) stored as JSON string or other text.

metadata

Additional info about the crawl or extracted data.

If success=False, contains a short description of what went wrong.

The ID of the session used for multi-page or persistent crawling.

HTTP response headers, if captured.

HTTP status code (e.g., 200 for OK).

SSL certificate info if fetch_ssl_certificate=True.

Additional concurrency and resource usage information when crawling URLs in parallel.

The URL after any redirects (different from url which is the final URL).

HTTP status code of the final redirect destination (e.g., 200). None for non-HTTP requests (raw HTML, local files).

List of network requests, responses, and failures captured during the crawl if capture_network_requests=True.

List of browser console messages captured during the crawl if capture_console_messages=True.

tables

Table data extracted from HTML tables with structure [{headers, rows, caption, summary}].


Deep Crawling - Crawl4AI Documentation (v0.9.x)

Summary: This tutorial explains how to perform configurable deep crawling with Crawl4AI, covering BFS, DFS, and BestFirst strategies, streaming vs non-streaming results, filters, scorers, crash recovery,…

Introduction

One of Crawl4AI's most powerful features is its ability to perform **configurable deep crawling** that can explore websites beyond a single page. With fine-tuned control over crawl depth, domain…

1. Quick Example

Here's a minimal code snippet that implements a basic deep crawl using the **BFSDeepCrawlStrategy**:

**What's happening?**
- `BFSDeepCrawlStrategy(max_depth=2, include_external=False)` instructs…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
from crawl4ai.content_scraping_strategy import…

2. Understanding Deep Crawling Strategy Options

###### 2.1 BFSDeepCrawlStrategy (Breadth-First Search)

The **BFSDeepCrawlStrategy** uses a breadth-first approach, exploring all links at one depth before moving deeper:

**Key parameters:**
-…

from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

# Basic configuration
strategy = BFSDeepCrawlStrategy(
    max_depth=2,               # Crawl initial page + 2 levels deep…

from crawl4ai.deep_crawling import DFSDeepCrawlStrategy

# Basic configuration
strategy = DFSDeepCrawlStrategy(
    max_depth=2,               # Crawl initial page + 2 levels deep…

from crawl4ai.deep_crawling import BestFirstCrawlingStrategy
from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer

# Create a scorer
scorer = KeywordRelevanceScorer(…

3. Streaming vs. Non-Streaming Results

Crawl4AI can return results in two modes:

###### 3.1 Non-Streaming Mode (Default)

**When to use non-streaming mode:**
- You need the complete dataset before processing
- You're performing batch…

config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1),
    stream=False  # Default behavior
)

async with AsyncWebCrawler() as crawler:
    # Wait for ALL results to be…

config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=1),
    stream=True  # Enable streaming
)

async with AsyncWebCrawler() as crawler:
    # Returns an async iterator…

4. Filtering Content with Filter Chains

Filters help you narrow down which pages to crawl. Combine multiple filters using **FilterChain** for powerful targeting.

###### 4.1 Basic URL Pattern Filter

###### 4.2 Combining Multiple Filters

###…

from crawl4ai.deep_crawling.filters import FilterChain, URLPatternFilter

# Only follow URLs containing "blog" or "docs"
url_filter = URLPatternFilter(patterns=["*blog*", "*docs*"])

config =…

from crawl4ai.deep_crawling.filters import (
    FilterChain,
    URLPatternFilter,
    DomainFilter,
    ContentTypeFilter
)

# Create a chain of filters
filter_chain = FilterChain([
    # Only…

5. Using Scorers for Prioritized Crawling

Scorers assign priority values to discovered URLs, helping the crawler focus on the most relevant content first.

###### 5.1 KeywordRelevanceScorer

**How scorers work:**
- Evaluate each discovered URL…

from crawl4ai.deep_crawling.scorers import KeywordRelevanceScorer
from crawl4ai.deep_crawling import BestFirstCrawlingStrategy

# Create a keyword relevance scorer
keyword_scorer =…

6. Advanced Filtering Techniques

###### 6.1 SEO Filter for Quality Assessment

The **SEOFilter** helps you identify pages with strong SEO characteristics:

###### 6.2 Content Relevance Filter

The **ContentRelevanceFilter** analyzes the…

from crawl4ai.deep_crawling.filters import FilterChain, SEOFilter

# Create an SEO filter that looks for specific keywords in page metadata
seo_filter = SEOFilter(
    threshold=0.5,  # Minimum score…

from crawl4ai.deep_crawling.filters import FilterChain, ContentRelevanceFilter

# Create a content relevance filter
relevance_filter = ContentRelevanceFilter(
    query="Web crawling and data…

7. Building a Complete Advanced Crawler

This example combines multiple techniques for a sophisticated crawl:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
from crawl4ai.deep_crawling import…

8. Limiting and Controlling Crawl Size

###### 8.1 Using max_pages

You can limit the total number of pages crawled with the `max_pages` parameter:

This feature is useful for:
- Controlling API costs
- Setting predictable execution times
-…

# Limit to exactly 20 pages regardless of depth
strategy = BFSDeepCrawlStrategy(
    max_depth=3,
    max_pages=20
)

# Only follow links with scores above 0.4
strategy = DFSDeepCrawlStrategy(
    max_depth=2,
    url_scorer=KeywordRelevanceScorer(keywords=["api", "guide", "reference"]),
    score_threshold=0.4  #…

9. Common Pitfalls & Tips

1. **Set realistic limits.** Be cautious with `max_depth` values > 3, which can exponentially increase crawl size. Use `max_pages` to set hard limits.

2. **Don't neglect the scoring component.**…

config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2),
    preserve_https_for_internal_links=True  # Keep HTTPS even if server redirects to HTTP
)

10. Crash Recovery for Long-Running Crawls

For production deployments, especially in cloud environments where instances can be terminated unexpectedly, Crawl4AI provides built-in crash recovery support for all deep crawl strategies.

###### 10.1…

from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
import json

# Callback to save state after each URL
async def save_state_to_redis(state: dict):
    await redis.set("crawl_state",…

{
    "strategy_type": "bfs",  # or "dfs", "best_first"
    "visited": ["url1", "url2", ...],  # Already crawled URLs
    "pending": [{"url": "...", "parent_url": "..."}],  # Queue/stack…

import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

# Load saved state (e.g., from Redis, database, or file)
saved_state =…

import json

captured_state = None

async def capture_state(state: dict):
    global captured_state
    captured_state = state

strategy = BFSDeepCrawlStrategy(
    max_depth=2,…

import asyncio
import json
import redis.asyncio as redis
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

REDIS_KEY =…

11. Cancellation Support for Deep Crawls

For production environments like cloud platforms, you often need to stop a running crawl mid-execution—whether the user changed their mind, specified the wrong URL, or wants to control costs.…

from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def check_if_cancelled():
    # Check Redis, database, or any external source
    job = await redis.get(f"job:{job_id}")
    return…

strategy = BFSDeepCrawlStrategy(max_depth=3, max_pages=1000)

# In another coroutine or thread:
strategy.cancel()  # Thread-safe, stops before next URL

async with AsyncWebCrawler() as crawler:
    results = await crawler.arun(url, config=config)

if strategy.cancelled:
    print(f"Crawl was cancelled after {len(results)} pages")
else:…

async def handle_state(state: dict):
    if state.get("cancelled"):
        print("Crawl was cancelled!")
        print(f"Crawled {state['pages_crawled']} pages before cancellation")
    # Save state…

import asyncio
import json
import redis.asyncio as redis
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def…

12. Prefetch Mode for Fast URL Discovery

When you need to quickly discover URLs without full page processing, use **prefetch mode** . This is ideal for two-phase crawling where you first map the site, then selectively process specific…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

config = CrawlerRunConfig(prefetch=True)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://example.com",…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def two_phase_crawl(start_url: str):
    async with AsyncWebCrawler() as crawler:
        #…

13. Summary & Next Steps

In this **Deep Crawling with Crawl4AI** tutorial, you learned to:

- Configure **BFSDeepCrawlStrategy** , **DFSDeepCrawlStrategy** , and **BestFirstCrawlingStrategy**
- Process results in streaming…


Domain Mapping: Discover Every URL Under a Domain

Summary: This page describes DomainMapper in Crawl4AI, which discovers every URL under a domain using 8 discovery sources, including sitemaps, Common Crawl, Wayback Machine, Certificate Transparency, path…

What Is Domain Mapping?

Domain mapping goes beyond URL seeding. Instead of checking a single sitemap or index, `DomainMapper` combines **8 discovery sources** to find every URL under a domain — including subdomains you…

DomainMapper vs AsyncUrlSeeder

| Aspect | AsyncUrlSeeder | DomainMapper |
| --- | --- | --- |
| **Scope** | Single host, listed URLs only | Entire domain + all subdomains |
| **Sources** | Sitemap + Common Crawl | 8 sources…

Quick Start

Or via `AsyncWebCrawler`:

import asyncio
from crawl4ai import DomainMapper, DomainMapperConfig

async def main():
    async with DomainMapper() as mapper:
        results = await mapper.scan("example.com")

    print(f"Found…

from crawl4ai import AsyncWebCrawler, DomainMapperConfig

async with AsyncWebCrawler() as crawler:
    results = await crawler.amap_domain("example.com")

The 8 Discovery Sources

DomainMapper combines these sources, each catching URLs the others miss:

1. `sitemap` — Sitemap Discovery

Checks `/sitemap.xml`, `/sitemap_index.xml`, and `robots.txt` `Sitemap:` directives **on every discovered host** — not just the root domain.

2. `cc` — Common Crawl

Queries the Common Crawl CDX API for `*.domain.tld/*`, catching URLs and subdomains the web's largest public crawl has indexed.

3. `wayback` — Wayback Machine

Queries the Internet Archive's CDX API. Often has different coverage than Common Crawl — including historical pages that have since been removed.

4. `crt` — Certificate Transparency

Queries [crt.sh](https://crt.sh) for SSL certificates issued to `*.domain.tld`. This is the single most effective subdomain discovery technique — it found 14 subdomains for `superdesign.dev` that no…

5. `probe` — Common Path Probing

Tries ~25 well-known paths on each discovered host (`/docs`, `/api`, `/login`, `/dashboard`, `/openapi.json`, etc.). Combined with soft-404 detection to avoid false positives.

config = DomainMapperConfig(source="probe")

# Add custom paths to probe
config = DomainMapperConfig(
    source="probe",
    probe_paths=["/custom-api", "/internal/status"]
)

6. `robots` — robots.txt Path Mining

Parses `Disallow:` and `Allow:` lines from `robots.txt`. These are confirmed real paths the site acknowledges exist — often revealing admin panels, APIs, and internal tools that aren't linked…

7. `feed` — RSS/Atom Feed Parsing

Discovers and parses RSS/Atom feeds at common paths (`/feed`, `/rss`, `/atom.xml`, etc.). Feeds are curated lists of content URLs maintained by the site.

8. `homepage` — Homepage Link Extraction

Fetches each host's homepage via HTTP and extracts all internal links using `quick_extract_links()`. Also mines `<link rel="alternate|preload|prefetch">` tags from the `` for additional URLs.…

Combining Sources

Sources are combined with `+`:

# Default: most useful combination
config = DomainMapperConfig(source="sitemap+cc+crt+probe")

# Maximum coverage: all 8 sources
config = DomainMapperConfig(…

How It Works: The Three Phases

Phase 1: Host Discovery

DomainMapper first discovers all subdomains under your domain:

Each discovered host is validated with an HTTP HEAD request. Hosts that don't respond are dropped.

superdesign.dev
├── crt.sh           → docs, app, cloud, insights, staging-api, ui2web, ...
├── Wayback CDX      → api, app, docs, www, ...
├── Common Crawl     → app, www, ...
└── DNS guessing     →…

Phase 2: Per-Host Scanning

For each validated host, DomainMapper runs all enabled sources in parallel:

docs.superdesign.dev
├── Soft-404 fingerprint  → (404 returns proper error — no SPA issue)
├── robots.txt            → 1 sitemap URL, 1 disallow path
├── Sitemap parsing       → 19 URLs
├── Path…

Phase 3: Post-Processing

All discovered URLs go through:

- **URL normalization** — using `normalize_url()` to canonicalize
- **Deduplication** — by normalized URL, merging source attribution
- **Nonsense filtering** —…

Soft-404 Detection

Many modern SPAs return HTTP 200 for every URL — even pages that don't exist. DomainMapper detects this:

- **Fingerprinting**: Fetches a guaranteed-nonexistent URL (e.g., `/c4ai-probe-a1b2c3d4`) on…

# Soft-404 detection is on by default
config = DomainMapperConfig(soft_404_detection=True)

# Disable if you want raw results
config = DomainMapperConfig(soft_404_detection=False)

Configuration Reference

DomainMapperConfig

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `source` | str | `"sitemap+cc+crt+probe"` | Discovery sources joined by `+` |
| `max_urls` | int | `-1` | Maximum URLs to…

Output Format

Each result is a dict:

{
    "url": "https://docs.superdesign.dev/quickstart",
    "host": "docs.superdesign.dev",
    "source": "homepage+sitemap",     # which source(s) found it
    "status": "valid",                #…

Practical Examples

Discover and Crawl Documentation

import asyncio
from crawl4ai import AsyncWebCrawler, DomainMapperConfig, CrawlerRunConfig

async def crawl_all_docs():
    async with AsyncWebCrawler() as crawler:
        # Step 1: Discover all…

Security Audit: Find Exposed Services

async def audit_domain():
    async with DomainMapper() as mapper:
        results = await mapper.scan("company.com", DomainMapperConfig(
            source="crt+probe+robots",…

Compare Subdomains Across a Domain

async def map_infrastructure():
    async with DomainMapper() as mapper:
        results = await mapper.scan("company.com", DomainMapperConfig(
            source="crt+probe",…

Tips and Best Practices

- **Start with the default sources** (`sitemap+cc+crt+probe`). Add `wayback`, `robots`, `feed`, and `homepage` if you need maximum coverage.
- **Use `extract_head=False` for speed** when you just…

See Also

- [URL Seeding](../url-seeding/) — simpler, single-host URL discovery from sitemaps and Common Crawl
- [Deep Crawling](../deep-crawling/) — follow links dynamically within pages
- [Multi-URL…

source

Discovery sources joined by `+`

Maximum URLs to return (-1 = unlimited)

concurrency

Max concurrent requests across all hosts

Rate limit in requests/second

force

Bypass all caches

Fetch and parse `<head>` metadata

Filter static assets and utility URLs

Fingerprint and filter soft-404 pages

query

BM25 relevance query (requires `extract_head=True`)

Minimum relevance score (0.0-1.0)

Scoring algorithm

Extra paths to probe on each host

Extra subdomain prefixes to guess

Use Playwright for JS-rendered homepages

verbose

Override logger verbose setting

Hours before cached results expire

Timeout for DNS resolution (seconds)

Timeout for HTTP requests (seconds)


Code Examples

Summary: This page provides a comprehensive list of example scripts that demonstrate various features and capabilities of Crawl4AI, organized by category with links to code and guides.

Overview

This page provides a comprehensive list of example scripts that demonstrate various features and capabilities of Crawl4AI. Each example is designed to showcase specific functionality, making it…

Getting Started Examples

| Example | Description | Link |
| --- | --- | --- |
| Hello World | A simple introductory example demonstrating basic usage of AsyncWebCrawler with JavaScript execution and content filtering. |…

Proxies

| Example | Description | Link |
| --- | --- | --- |
| **NSTProxy** | [NSTProxy](https://www.nstproxy.com/?utm_source=crawl4ai) Seamlessly integrates with crawl4ai — no setup required. Access…

Browser & Crawling Features

| Example | Description | Link |
| --- | --- | --- |
| Built-in Browser | Demonstrates how to use the built-in browser capabilities. | [View…

Advanced Crawling & Deep Crawling

| Example | Description | Link |
| --- | --- | --- |
| Deep Crawling | An extensive tutorial on deep crawling capabilities, demonstrating BFS and BestFirst strategies, stream vs. non-stream…

Extraction Strategies

| Example | Description | Link |
| --- | --- | --- |
| Extraction Strategies | Demonstrates different extraction strategies with various input formats (markdown, HTML, fit_markdown) and JSON-based…

E-commerce & Specialized Crawling

| Example | Description | Link |
| --- | --- | --- |
| Amazon Product Extraction | Demonstrates how to extract structured product data from Amazon search results using CSS selectors. | [View…

Anti-Bot & Stealth Features

| Example | Description | Link |
| --- | --- | --- |
| Stealth Mode Quick Start | Five practical examples showing how to use stealth mode for bypassing basic bot detection. | [View…

Customization & Security

| Example | Description | Link |
| --- | --- | --- |
| Hooks | Illustrates how to use hooks at different stages of the crawling process for advanced customization. | [View…

Docker & Deployment

| Example | Description | Link |
| --- | --- | --- |
| Docker Config | Demonstrates how to create and use Docker configuration objects. | [View…

Application Examples

| Example | Description | Link |
| --- | --- | --- |
| Research Assistant | Demonstrates how to build a research assistant using Crawl4AI. | [View…

Content Generation & Markdown

| Example | Description | Link |
| --- | --- | --- |
| Content Source | Demonstrates how to work with different content sources in markdown generation. | [View…

Running the Examples

To run any of these examples, you'll need to have Crawl4AI installed:

Then, you can run an example script like this:

For examples that require additional dependencies or environment variables,…

Contributing New Examples

If you've created an interesting example that demonstrates a unique use case or feature of Crawl4AI, we encourage you to contribute it to our examples collection. Please see our [contribution…


Fit Markdown with Pruning & BM25

Summary: Explains how to use Fit Markdown with Pruning and BM25 content filters in Crawl4AI to extract concise, relevant content from web pages.

Overview

**Fit Markdown** is a specialized **filtered** version of your page’s markdown, focusing on the most relevant content. By default, Crawl4AI converts the entire HTML into a broad **raw_markdown**.…

1. How “Fit Markdown” Works

1.1 The `content_filter`

In **`CrawlerRunConfig`**, you can specify a **`content_filter`** to shape how content is pruned or ranked before final markdown generation. A filter’s logic is applied **before** or **during** the…

1.2 Common Filters

1. **PruningContentFilter** – Scores each node by text density, link density, and tag importance, discarding those below a threshold.
2. **BM25ContentFilter** – Focuses on textual relevance using…

2. PruningContentFilter

**Pruning** discards less relevant nodes based on **text density, link density, and tag importance**. It’s a heuristic-based approach—if certain sections appear too “thin” or too “spammy,” they’re…

2.1 Usage Example

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import…

2.2 Key Parameters

- **`min_word_threshold`** (int): If a block has fewer words than this, it’s pruned.
- **`threshold_type`** (str):
  - `"fixed"` → each node must exceed `threshold` (0–1).
  - `"dynamic"` → node…

3. BM25ContentFilter

**BM25** is a classical text ranking algorithm often used in search engines. If you have a **user query** or rely on page metadata to derive a query, BM25 can identify which text chunks best match…

3.1 Usage Example

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import BM25ContentFilter
from crawl4ai.markdown_generation_strategy import…

3.2 Parameters

- **`user_query`** (str, optional): E.g. `"machine learning"`. If blank, the filter tries to glean a query from page metadata.
- **`bm25_threshold`** (float, default 1.0):
  - Higher → fewer chunks…

4. Accessing the “Fit” Output

After the crawl, your “fit” content is found in **`result.markdown.fit_markdown`**.

If the content filter is **BM25**, you might see additional logic or references in `fit_markdown` that highlight…

fit_md = result.markdown.fit_markdown
fit_html = result.markdown.fit_html

5. Code Patterns Recap

5.1 Pruning

prune_filter = PruningContentFilter(
    threshold=0.5,
    threshold_type="fixed",
    min_word_threshold=10
)
md_generator = DefaultMarkdownGenerator(content_filter=prune_filter)
config =…

5.2 BM25

bm25_filter = BM25ContentFilter(
    user_query="health benefits fruit",
    bm25_threshold=1.2
)
md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)
config =…

6. Combining with “word_count_threshold” & Exclusions

Remember you can also specify:

Thus, **multi-level** filtering occurs:

- The crawler’s `excluded_tags` are removed from the HTML first.
- The content filter (Pruning, BM25, or custom) prunes or…

config = CrawlerRunConfig(
    word_count_threshold=10,
    excluded_tags=["nav", "footer", "header"],
    exclude_external_links=True,
    markdown_generator=DefaultMarkdownGenerator(…

7. Custom Filters

If you need a different approach (like a specialized ML model or site-specific heuristics), you can create a new class inheriting from `RelevantContentFilter` and implement `filter_content(html)`.…

from crawl4ai.content_filter_strategy import RelevantContentFilter

class MyCustomFilter(RelevantContentFilter):
    def filter_content(self, html, min_word_threshold=None):
        # parse HTML,…

8. Final Thoughts

**Fit Markdown** is a crucial feature for:

- **Summaries**: Quickly get the important text from a cluttered page.
- **Search**: Combine with **BM25** to produce content relevant to a query.
- **AI…


Installation & Setup (2023 Edition)

Summary: Installation and setup guide for Crawl4AI, covering basic install, diagnostics, verification, optional advanced dependencies, Docker, and local server mode.

1. Basic Installation

This installs the  **core**  Crawl4AI library along with essential dependencies.  **No**  advanced features (like transformers or PyTorch) are included yet.

2. Initial Setup & Diagnostics

2.1 Run the Setup Command

After installing, call:

 **What does it do?**
- Installs or updates required browser dependencies for both regular and undetected modes
- Performs OS-level checks (e.g., missing libs on Linux)
-…

2.2 Diagnostics

Optionally, you can run  **diagnostics**  to confirm everything is functioning:

This command attempts to:
- Check Python version compatibility
- Verify Playwright installation
- Inspect environment…

3. Verifying Installation: A Simple Crawl (Skip this step if you already run `crawl4ai-doctor`)

Below is a minimal Python script demonstrating a  **basic**  crawl. It uses our new  **`BrowserConfig`**  and  **`CrawlerRunConfig`**  for clarity, though no custom settings are passed in this…

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(…

4. Advanced Installation (Optional)

**Warning** : Only install these  **if you truly need them** . They bring in larger dependencies, including big models, which can increase disk usage and memory load significantly.

4.1 Torch, Transformers, or All

- **Text Clustering (Torch)**

  Installs PyTorch-based features (e.g., cosine similarity or advanced semantic chunking).

- **Transformers**

  Adds Hugging Face-based summarization or generation…

pip install crawl4ai[torch]
crawl4ai-setup

pip install crawl4ai[transformer]
crawl4ai-setup

pip install crawl4ai[all]
crawl4ai-setup

(Optional) Pre-Fetching Models

This step caches large models locally (if needed).  **Only do this**  if your workflow requires them.

5. Docker (Experimental)

We provide a  **temporary**  Docker approach for testing.  **It’s not stable and may break**  with future releases. We plan a major Docker revamp in a future stable version, 2025 Q1. If you still…

docker pull unclecode/crawl4ai:basic
docker run -p 11235:11235 unclecode/crawl4ai:basic

6. Local Server Mode (Legacy)

Some older docs mention running Crawl4AI as a local server. This approach has been  **partially replaced**  by the new Docker-based prototype and upcoming stable server release. You can experiment,…

Summary

1. **Install** with `pip install crawl4ai` and run `crawl4ai-setup`.
2. **Diagnose** with `crawl4ai-doctor` if you see errors.
3. **Verify** by crawling `example.com` with minimal `BrowserConfig` +…


Link & Media - Crawl4AI Documentation (v0.9.x)

Summary: This tutorial covers how to extract and filter links (internal/external) and media (images, videos, audio) from crawled pages using Crawl4AI, including advanced link head extraction with scoring,…

1. Link Extraction

When you call `arun()` or `arun_many()` on a URL, Crawl4AI automatically extracts links and stores them in the `links` field of `CrawlResult`. By default, the crawler tries to distinguish…

from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://www.example.com")
    if result.success:
        internal_links =…

result.links = {
  "internal": [
    {
      "href": "https://kidocode.com/",
      "text": "",
      "title": "",
      "base_domain": "kidocode.com"
    },
    {
      "href":…

2. Advanced Link Head Extraction & Scoring

Ever wanted to not just extract links, but also get the actual content (title, description, metadata) from those linked pages? And score them for relevance? This is exactly what Link Head Extraction…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import LinkPreviewConfig

async def extract_link_heads_example():
    """
    Complete example showing link head…

✅ Successfully crawled: https://docs.python.org/3/
📄 Page title: 3.13.5 Documentation
🔗 Found 53 internal links
🌍 Found 1 external links
🧠 Links with head data extracted: 10

🏆 Top 3 Links with Full…

from crawl4ai import LinkPreviewConfig

link_preview_config = LinkPreviewConfig(
    # BASIC SETTINGS
    verbose=True,                    # Show detailed logs (recommended for learning)

    # LINK…

# High intrinsic score indicators:
# ✅ Clean URL structure (docs.python.org/api/reference)
# ✅ Meaningful link text ("API Reference Guide")
# ✅ Relevant to page context
# ✅ Not buried deep in…

# Example: query = "machine learning tutorial"
# High contextual score: Link to "Complete Machine Learning Guide"
# Low contextual score: Link to "Privacy Policy"

# When both scores available: (intrinsic * 0.3) + (contextual * 0.7)
# When only intrinsic: uses intrinsic score
# When only contextual: uses contextual score
# When neither: not calculated

async def research_assistant():
    config = CrawlerRunConfig(
        link_preview_config=LinkPreviewConfig(
            include_internal=True,
            include_external=True,…

async def api_discovery():
    config = CrawlerRunConfig(
        link_preview_config=LinkPreviewConfig(
            include_internal=True,
            include_patterns=["*/api/*", "*/reference/*"],…

async def quality_analysis():
    config = CrawlerRunConfig(
        link_preview_config=LinkPreviewConfig(
            include_internal=True,
            max_links=200,
            concurrency=20,…

# Check your configuration:
config = CrawlerRunConfig(
    link_preview_config=LinkPreviewConfig(
        verbose=True   # ← Enable to see what's happening
    )
)

# Make sure scoring is enabled:
config = CrawlerRunConfig(
    score_links=True,  # ← Enable intrinsic scoring
    link_preview_config=LinkPreviewConfig(
        query="your search terms"  # ← For…

# Optimize performance:
link_preview_config = LinkPreviewConfig(
    max_links=20,      # ← Reduce number
    concurrency=10,    # ← Increase parallelism
    timeout=3,         # ← Shorter timeout…

3. Domain Filtering

Some websites contain hundreds of third-party or affiliate links. You can filter out certain domains at **crawl time** by configuring the crawler. The most relevant parameters in `CrawlerRunConfig`…

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    crawler_cfg = CrawlerRunConfig(
        exclude_external_links=True,          # No links…

crawler_cfg = CrawlerRunConfig(
    exclude_domains=["suspiciousads.com"]
)

4. Media Extraction

###### 4.1 Accessing `result.media`

By default, Crawl4AI collects images, audio and video URLs it finds on the page. These are stored in `result.media`, a dictionary keyed by media type (e.g.,…

if result.success:
    # Get images
    images_info = result.media.get("images", [])
    print(f"Found {len(images_info)} images in total.")
    for i, img in enumerate(images_info[:3]):  # Inspect…

result.media = {
  "images": [
    {
      "src": "https://cdn.prod.website-files.com/.../Group%2089.svg",
      "alt": "coding school for kids",
      "desc": "Trial Class Degrees degrees All…

crawler_cfg = CrawlerRunConfig(
    exclude_external_images=True
)

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    crawler_cfg = CrawlerRunConfig(
        capture_mhtml=True  # Enable MHTML capture
    )

    async with…

5. Putting It All Together: Link & Media Filtering

Here’s a combined example demonstrating how to filter out external links, skip certain domains, and exclude external images:

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig

async def main():
    # Suppose we want to keep only internal links, remove certain domains,
    # and discard…

6. Common Pitfalls & Tips

1. **Conflicting Flags**:
   - `exclude_external_links=True` but then also specifying `exclude_social_media_links=True` is typically fine, but understand that the first setting already discards *all*…


Llmtxt - Crawl4AI Documentation (v0.9.x)

Summary: This page documents the Llmtxt feature in Crawl4AI, which provides LLM-friendly text for crawled content. The provided raw content contains only navigation and UI elements, so the actual…

Llmtxt

The provided raw content for this page contains only navigation and UI elements. No actual documentation text was found in the provided content.


Prefix-Based Input Handling in Crawl4AI

Summary: This guide demonstrates how to use Crawl4AI to crawl web URLs, local HTML files, and raw HTML strings using prefix-based input handling with the unified `url` parameter and `CrawlerRunConfig`.

Crawling a Web URL

To crawl a live web page, provide the URL starting with `http://` or `https://`, using a `CrawlerRunConfig` object:

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def crawl_web():
    config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler() as…

Crawling a Local HTML File

To crawl a local HTML file, prefix the file path with `file://`.

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def crawl_local_file():
    local_file_path = "/path/to/apple.html"  # Replace with your file path
    file_url…

Crawling Raw HTML Content

To crawl raw HTML content, prefix the HTML string with `raw:`.

import asyncio
from crawl4ai import AsyncWebCrawler, CacheMode
from crawl4ai.async_configs import CrawlerRunConfig

async def crawl_raw_html():
    raw_html = "<html><body><h1>Hello,…

Complete Example

Below is a comprehensive script that:

- Crawls the Wikipedia page for "Apple."
- Saves the HTML content to a local file (`apple.html`).
- Crawls the local HTML file and verifies the markdown length…

import os
import sys
import asyncio
from pathlib import Path
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig

async def main():
    wikipedia_url =…

Conclusion

With the unified `url` parameter and prefix-based handling in **Crawl4AI** , you can seamlessly handle web URLs, local HTML files, and raw HTML content. Use `CrawlerRunConfig` for flexible and…


Markdown Generation Basics

Summary: This tutorial explains how to generate clean, structured markdown from web pages using Crawl4AI's DefaultMarkdownGenerator, including configuration options, content filters (BM25, Pruning, LLM), and…

Prerequisites

> **Prerequisites**
>
> - You’ve completed or read [AsyncWebCrawler Basics](../simple-crawling/) to understand how to run a simple crawl.
>
> - You know how to configure `CrawlerRunConfig`.

1. Quick Example

Here’s a minimal code snippet that uses the **DefaultMarkdownGenerator** with no additional filtering:

**What’s happening?**

- `CrawlerRunConfig( markdown_generator = DefaultMarkdownGenerator() )`…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    config = CrawlerRunConfig(…

2. How Markdown Generation Works

Under the hood, **DefaultMarkdownGenerator** uses a specialized HTML-to-text approach that:

- Preserves headings, code blocks, bullet points, etc.
- Removes extraneous tags (scripts, styles) that…

2.1 HTML-to-Text Conversion (Forked & Modified)

2.2 Link Citations & References

By default, the generator can convert `<a href="...">` elements into `[text][1]` citations, then place the actual links at the bottom of the document. This is handy for research workflows that demand…

2.3 Optional Content Filters

Before or after the HTML-to-Markdown step, you can apply a **content filter** (like BM25 or Pruning) to reduce noise and produce a “fit_markdown”—a heavily pruned version focusing on the page’s main…

3. Configuring the Default Markdown Generator

You can tweak the output by passing an `options` dict to `DefaultMarkdownGenerator`. For example:

Some commonly used `options`:

- **`ignore_links`** (bool): Whether to remove all hyperlinks in the…

from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Example: ignore all links, don't escape…

4. Selecting the HTML Source for Markdown Generation

The `content_source` parameter allows you to control which HTML content is used as input for markdown generation. This gives you flexibility in how the HTML is processed before conversion to markdown.

from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Option 1: Use the raw HTML directly from the…

HTML Source Options

- **`"cleaned_html"`** (default): Uses the HTML after it has been processed by the scraping strategy. This HTML is typically cleaner and more focused on content, with some boilerplate removed.
-…

When to Use Each Option

- Use **`"cleaned_html"`** (default) for most cases where you want a balance of content preservation and noise removal.
- Use **`"raw_html"`** when you need to preserve all original content, or when…

5. Content Filters

**Content filters** selectively remove or rank sections of text before turning them into Markdown. This is especially helpful if your page has ads, nav bars, or other clutter you don’t want.

5.1 BM25ContentFilter

If you have a **search query**, BM25 is a good choice:

- **`user_query`**: The term you want to focus on. BM25 tries to keep only content blocks relevant to that query.
- **`bm25_threshold`**: Raise…

from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import BM25ContentFilter
from crawl4ai import CrawlerRunConfig

bm25_filter =…

5.2 PruningContentFilter

If you **don’t** have a specific query, or if you just want a robust “junk remover,” use `PruningContentFilter`. It analyzes text density, link density, HTML structure, and known patterns (like…

from crawl4ai.content_filter_strategy import PruningContentFilter

prune_filter = PruningContentFilter(
    threshold=0.5,
    threshold_type="fixed",  # or "dynamic"
    min_word_threshold=50
)

5.3 LLMContentFilter

For intelligent content filtering and high-quality markdown generation, you can use the **LLMContentFilter**. This filter leverages LLMs to generate relevant markdown while preserving the original…

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, LLMConfig, DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import LLMContentFilter

async def main():
    #…

filter = LLMContentFilter(
    instruction="""
    Extract the main educational content while preserving its original wording and substance completely.
    1. Maintain the exact language and…

filter = LLMContentFilter(
    instruction="""
    Focus on extracting specific types of content:
    - Technical documentation
    - Code examples
    - API references
    Reformat the content into…

6. Using Fit Markdown

When a content filter is active, the library produces two forms of markdown inside `result.markdown`:

1. **`raw_markdown`**: The full unfiltered markdown.
2. **`fit_markdown`**: A “fit” version…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import…

7. The `MarkdownGenerationResult` Object

If your library stores detailed markdown output in an object like `MarkdownGenerationResult`, you’ll see fields such as:

- **`raw_markdown`**: The direct HTML-to-markdown transformation (no…

md_obj = result.markdown  # your library’s naming may vary
print("RAW:\n", md_obj.raw_markdown)
print("CITED:\n", md_obj.markdown_with_citations)
print("REFERENCES:\n",…

8. Combining Filters (BM25 + Pruning) in Two Passes

You might want to **prune out** noisy boilerplate first (with `PruningContentFilter`), and then **rank what’s left** against a user query (with `BM25ContentFilter`). You don’t have to crawl the page…

Two-Pass Example

###### What’s Happening?

1. **Raw HTML**: We crawl once and store the raw HTML in `result.html`.
2. **PruningContentFilter**: Takes HTML + optional parameters. It extracts blocks of text or partial…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter, BM25ContentFilter
from bs4 import BeautifulSoup

async def…

Tips & Variations

- **Plain Text vs. HTML**: If your pruned output is mostly text, BM25 can still handle it; just keep in mind it expects a valid string input. If you supply partial HTML (like `"<p>some text</p>"`),…

One-Pass Combination?

If your codebase or pipeline design allows applying multiple filters in one pass, you could do so. But often it’s simpler—and more transparent—to run them sequentially, analyzing each step’s…

9. Common Pitfalls & Tips

1. **No Markdown Output?**
   - Make sure the crawler actually retrieved HTML. If the site is heavily JS-based, you may need to enable dynamic rendering or wait for elements.
   - Check if your…

10. Summary & Next Steps

In this **Markdown Generation Basics** tutorial, you learned to:

- Configure the **DefaultMarkdownGenerator** with HTML-to-text options.
- Select different HTML sources using the `content_source`…


Page Interaction - Crawl4AI Documentation (v0.9.x)

Summary: This page explains how to interact with dynamic webpages using Crawl4AI, covering JavaScript execution, wait conditions, multi-step flows, Shadow DOM flattening, and virtual scrolling.

Page Interaction

Crawl4AI provides powerful features for interacting with **dynamic** webpages, handling JavaScript execution, waiting for conditions, and managing multi-step flows. By combining **js_code**,…

1. JavaScript Execution

###### Basic Execution

**`js_code`** in **`CrawlerRunConfig`** accepts either a single JS string or a list of JS snippets. It runs **after** `wait_for` and `delay_before_return_html` — so the page is…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Single JS command
    config = CrawlerRunConfig(
        js_code="window.scrollTo(0,…

1. Page navigation (page.goto)
2. js_code_before_wait     ← triggers loading / clicks tabs
3. wait_for                ← waits for content to appear
4. delay_before_return_html ← extra safety…

config = CrawlerRunConfig(
    # Click a tab first
    js_code_before_wait="document.querySelector('#specs-tab')?.click();",
    # Then wait for the tab content to appear…

2. Wait Conditions

###### 2.1 CSS-Based Waiting

Sometimes, you just want to wait for a specific element to appear. For example:

**Key param**:
- **`wait_for="css:..."`**: Tells the crawler to wait until that CSS…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    config = CrawlerRunConfig(
        # Wait for at least 30 items on Hacker News…

wait_condition = """() => {
    const items = document.querySelectorAll('.athing');
    return items.length > 50;  // Wait for at least 51 items
}"""

config =…

3. Handling Dynamic Content

Many modern sites require **multiple steps**: scrolling, clicking “Load More,” or updating via JavaScript. Below are typical patterns.

###### 3.1 Load More Example (Hacker News “More” Link)

**Key…

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    # Step 1: Load initial Hacker News page
    config = CrawlerRunConfig(…

js_form_interaction = """
document.querySelector('#your-search').value = 'TypeScript commits';
document.querySelector('form').submit();
"""

config = CrawlerRunConfig(…

4. Timing Control

1. **`page_timeout`** (ms): Overall page load or script execution time limit.
2. **`delay_before_return_html`** (seconds): Wait an extra moment before capturing the final HTML.
3. **`mean_delay`** &…

config = CrawlerRunConfig(
    page_timeout=60000,  # 60s limit
    delay_before_return_html=2.5
)

5. Multi-Step Interaction Example

Below is a simplified script that does multiple “Load More” clicks on GitHub’s TypeScript commits page. It **re-uses** the same session to accumulate new commits each time. The code includes the…

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def multi_page_commits():
    browser_cfg = BrowserConfig(
        headless=False,  # Visible…

6. Combine Interaction with Extraction

Once dynamic content is loaded, you can attach an **`extraction_strategy`** (like `JsonCssExtractionStrategy` or `LLMExtractionStrategy`). For example:

When done, check `result.extracted_content`…

from crawl4ai import JsonCssExtractionStrategy

schema = {
    "name": "Commits",
    "baseSelector": "li.Box-sc-g0xbh4-0",
    "fields": [
        {"name": "title", "selector": "h4.markdown-title",…

7. Shadow DOM Flattening

Sites built with **Web Components** (Stencil, Lit, Shoelace, etc.) render content inside Shadow DOM — an encapsulated sub-tree that is invisible to normal page serialization. Set…

config = CrawlerRunConfig(
    flatten_shadow_dom=True,
    wait_until="load",
    delay_before_return_html=3.0,  # give components time to hydrate
)

8. Relevant `CrawlerRunConfig` Parameters

Below are the key interaction-related parameters in `CrawlerRunConfig`. For a full list, see [Configuration Parameters](../../api/parameters/).

- **`js_code`**: JavaScript to run after `wait_for` +…

9. Conclusion

Crawl4AI's **page interaction** features let you:

1. **Execute JavaScript** for scrolling, clicks, or form filling.
2. **Wait** for CSS or custom JS conditions before capturing data.
3. **Handle**…

10. Virtual Scrolling

For sites that use **virtual scrolling** (where content is replaced rather than appended as you scroll, like Twitter or Instagram), Crawl4AI provides a dedicated `VirtualScrollConfig`:

###### Virtual…

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig

async def crawl_twitter_timeline():
    # Configure virtual scroll for Twitter-like feeds
    virtual_config =…


Quick Start - Crawl4AI Documentation (v0.9.x)

Summary: Extraction fallback content.

##### Getting Started with Crawl4AI

Welcome to  **Crawl4AI** , an open-source LLM-friendly Web Crawler & Scraper. In this tutorial, you’ll:

- Run your  **first crawl**  using minimal configuration.

- Generate  **Markdown**  output (and learn how it’s influenced by content filters).

- Experiment with a simple  **CSS-based extraction**  strategy.

- See a glimpse of  **LLM-based extraction**  (including open-source and closed-source model options).

- Crawl a  **dynamic**  page that loads content via JavaScript.

##### 1. Introduction

Crawl4AI provides:

- An asynchronous crawler,  **`AsyncWebCrawler`** .

- Configurable browser and run settings via  **`BrowserConfig`**  and  **`CrawlerRunConfig`** .

- Automatic HTML-to-Markdown conversion via  **`DefaultMarkdownGenerator`**  (supports optional filters).

- Multiple extraction strategies (LLM-based or “traditional” CSS/XPath-based).

By the end of this guide, you’ll have performed a basic crawl, generated Markdown, tried out two extraction strategies, and crawled a dynamic page that uses “Load More” buttons or JavaScript updates.

##### 2. Your First Crawl

Here’s a minimal Python script that creates an  **`AsyncWebCrawler`** , fetches a webpage, and prints the first 300 characters of its Markdown output:

 **What’s happening?**
-  **`AsyncWebCrawler`**  launches a headless browser (Chromium by default).
- It fetches `https://example.com`.
- Crawl4AI automatically converts the HTML into Markdown.

You now have a simple, working crawl!

##### 3. Basic Configuration (Light Introduction)

Crawl4AI’s crawler can be heavily customized using two main classes:

1.  **`BrowserConfig`** : Controls browser behavior (headless or full UI, user agent, JavaScript toggles, etc.).

2.  **`CrawlerRunConfig`** : Controls how each crawl runs (caching, extraction, timeouts, hooking, etc.).

Below is an example with minimal usage:

> IMPORTANT: By default cache mode is set to `CacheMode.BYPASS` to have fresh content. Set `CacheMode.ENABLED` to enable caching.

We’ll explore more advanced config in later tutorials (like enabling proxies, PDF output, multi-tab sessions, etc.). For now, just note how you pass these objects to manage crawling.

##### 4. Generating Markdown Output

By default, Crawl4AI automatically generates Markdown from each crawled page. However, the exact output depends on whether you specify a  **markdown generator**  or  **content filter** .

- **`result.markdown`** :

  The direct HTML-to-Markdown conversion.

- **`result.markdown.fit_markdown`** :

  The same content after applying any configured  **content filter**  (e.g., `PruningContentFilter`).

###### Example: Using a Filter with `DefaultMarkdownGenerator`

 **Note** : If you do  **not**  specify a content filter or markdown generator, you’ll typically see only the raw Markdown. `PruningContentFilter` may adds around `50ms` in processing time. We’ll dive deeper into these strategies in a dedicated  **Markdown Generation**  tutorial.

##### 5. Simple Data Extraction (CSS-based)

Crawl4AI can also extract structured data (JSON) using CSS or XPath selectors. Below is a minimal CSS-based example:

> **New!**  Crawl4AI now provides a powerful utility to automatically generate extraction schemas using LLM. This is a one-time cost that gives you a reusable schema for fast, LLM-free extractions:

For a complete guide on schema generation and advanced usage, see [No-LLM Extraction Strategies](../../extraction/no-llm-strategies/).

Here's a basic extraction example:

 **Why is this helpful?**
- Great for repetitive page structures (e.g., item listings, articles).
- No AI usage or costs.
- The crawler returns a JSON string you can parse or store.

> Tips: You can pass raw HTML to the crawler instead of a URL. To do so, prefix the HTML with `raw://`.

##### 6. Simple Data Extraction (LLM-based)

For more complex or irregular pages, a language model can parse text intelligently into a structure you define. Crawl4AI supports  **open-source**  or  **closed-source**  providers:

- **Open-Source Models**  (e.g., `ollama/llama3.3`, `no_token`)

- **OpenAI Models**  (e.g., `openai/gpt-4`, requires `api_token`)

- Or any provider supported by the underlying library

Below is an example using  **open-source**  style (no token) and closed-source:

 **What’s happening?**
- We define a Pydantic schema (`PricingInfo`) describing the fields we want.
- The LLM extraction strategy uses that schema and your instructions to transform raw text into structured JSON.
- Depending on the  **provider**  and  **api_token** , you can use local models or a remote API.

##### 7. Adaptive Crawling (New!)

Crawl4AI now includes intelligent adaptive crawling that automatically determines when sufficient information has been gathered. Here's a quick example:

 **What's special about adaptive crawling?**
-  **Automatic stopping** : Stops when sufficient information is gathered
-  **Intelligent link selection** : Follows only relevant links
-  **Confidence scoring** : Know how complete your information is

[Learn more about Adaptive Crawling →](../adaptive-crawling/)

##### 8. Multi-URL Concurrency (Preview)

If you need to crawl multiple URLs in  **parallel** , you can use `arun_many()`. By default, Crawl4AI employs a  **MemoryAdaptiveDispatcher** , automatically adjusting concurrency based on system resources. Here’s a quick glimpse:

The example above shows two ways to handle multiple URLs:
1.  **Streaming mode**  (`stream=True`): Process results as they become available using `async for`
2.  **Batch mode**  (`stream=False`): Wait for all results to complete

For more advanced concurrency (e.g., a  **semaphore-based**  approach,  **adaptive memory usage throttling** , or customized rate limiting), see [Advanced Multi-URL Crawling](../../advanced/multi-url-crawling/).

##### 8. Dynamic Content Example

Some sites require multiple “page clicks” or dynamic JavaScript updates. Below is an example showing how to  **click**  a “Next Page” button and wait for new commits to load on GitHub, using  **`BrowserConfig`**  and  **`CrawlerRunConfig`** :

 **Key Points** :

- **`BrowserConfig(headless=False)`** : We want to watch it click “Next Page.”

- **`CrawlerRunConfig(...)`** : We specify the extraction strategy, pass `session_id` to reuse the same page.

- **`js_code`**  and  **`wait_for`**  are used for subsequent pages (`page > 0`) to click the “Next” button and wait for new commits to load.

- **`js_only=True`**  indicates we’re not re-navigating but continuing the existing session.

- Finally, we call `kill_session()` to clean up the page and browser session.

##### 9. Next Steps

Congratulations! You have:

- Performed a basic crawl and printed Markdown.

- Used  **content filters**  with a markdown generator.

- Extracted JSON via  **CSS**  or  **LLM**  strategies.

- Handled  **dynamic**  pages with JavaScript triggers.

If you’re ready for more, check out:

- **Installation** : A deeper dive into advanced installs, Docker usage (experimental), or optional dependencies.

- **Hooks & Auth** : Learn how to run custom JavaScript or handle logins with cookies, local storage, etc.

- **Deployment** : Explore ephemeral testing in Docker or plan for the upcoming stable Docker release.

- **Browser Management** : Delve into user simulation, stealth modes, and concurrency best practices.

Crawl4AI is a powerful, flexible tool. Enjoy building out your scrapers, data pipelines, or AI-driven extraction flows. Happy crawling!

Page Copy
Page Copy

- [Copy as Markdown
Copy page for LLMs](#)

- [View as Markdown
Open raw source](#)

- [Open in ChatGPT
Ask questions about this page](#)

ESC to close

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:300])  # Print first 300 chars

if __name__ == "__main__":
    asyncio.run(main())

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_conf = BrowserConfig(headless=True)  # or False to see the browser
    run_conf = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS
    )

    async with AsyncWebCrawler(config=browser_conf) as crawler:
        result = await crawler.arun(
            url="https://example.com",
            config=run_conf
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

md_generator = DefaultMarkdownGenerator(
    content_filter=PruningContentFilter(threshold=0.4, threshold_type="fixed")
)

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    markdown_generator=md_generator
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://news.ycombinator.com", config=config)
    print("Raw Markdown length:", len(result.markdown.raw_markdown))
    print("Fit Markdown length:", len(result.markdown.fit_markdown))

from crawl4ai import JsonCssExtractionStrategy
from crawl4ai import LLMConfig

# Generate a schema (one-time cost)
html = "

Gaming Laptop
$999.99
"

# Using OpenAI (requires API token)
schema = JsonCssExtractionStrategy.generate_schema(
    html,
    llm_config = LLMConfig(provider="openai/gpt-4o",api_token="your-openai-token")  # Required for OpenAI
)

# Or using Ollama (open source, no token needed)
schema = JsonCssExtractionStrategy.generate_schema(
    html,
    llm_config = LLMConfig(provider="ollama/llama3.3", api_token=None)  # Not needed for Ollama
)

# Use the schema for fast, repeated extractions
strategy = JsonCssExtractionStrategy(schema)

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def main():
    schema = {
        "name": "Example Items",
        "baseSelector": "div.item",
        "fields": [
            {"name": "title", "selector": "h2", "type": "text"},
            {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
        ]
    }

    raw_html = "

Item 1
Link 1
"

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="raw://" + raw_html,
            config=CrawlerRunConfig(
                cache_mode=CacheMode.BYPASS,
                extraction_strategy=JsonCssExtractionStrategy(schema)
            )
        )
        # The JSON output is stored in 'extracted_content'
        data = json.loads(result.extracted_content)
        print(data)

if __name__ == "__main__":
    asyncio.run(main())

import os
import json
import asyncio
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
from crawl4ai import LLMExtractionStrategy

class OpenAIModelFee(BaseModel):
    model_name: str = Field(..., description="Name of the OpenAI model.")
    input_fee: str = Field(..., description="Fee for input token for the OpenAI model.")
    output_fee: str = Field(
        ..., description="Fee for output token for the OpenAI model."
    )

async def extract_structured_data_using_llm(
    provider: str, api_token: str = None, extra_headers: Dict[str, str] = None
):
    print(f"\n--- Extracting Structured Data with {provider} ---")

    if api_token is None and provider != "ollama":
        print(f"API token is required for {provider}. Skipping this example.")
        return

    browser_config = BrowserConfig(headless=True)

    extra_args = {"temperature": 0, "top_p": 0.9, "max_tokens": 2000}
    if extra_headers:
        extra_args["extra_headers"] = extra_headers

    crawler_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        word_count_threshold=1,
        page_timeout=80000,
        extraction_strategy=LLMExtractionStrategy(
            llm_config = LLMConfig(provider=provider,api_token=api_token),
            schema=OpenAIModelFee.model_json_schema(),
            extraction_type="schema",
            instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens.
            Do not miss any models in the entire content.""",
            extra_args=extra_args,
        ),
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://openai.com/api/pricing/", config=crawler_config
        )
        print(result.extracted_content)

if __name__ == "__main__":

    asyncio.run(
        extract_structured_data_using_llm(
            provider="openai/gpt-4o", api_token=os.getenv("OPENAI_API_KEY")
        )
    )

import asyncio
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler

async def adaptive_example():
    async with AsyncWebCrawler() as crawler:
        adaptive = AdaptiveCrawler(crawler)

        # Start adaptive crawling
        result = await adaptive.digest(
            start_url="https://docs.python.org/3/",
            query="async context managers"
        )

        # View results
        adaptive.print_stats()
        print(f"Crawled {len(result.crawled_urls)} pages")
        print(f"Achieved {adaptive.confidence:.0%} confidence")

if __name__ == "__main__":
    asyncio.run(adaptive_example())

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

async def quick_parallel_example():
    urls = [
        "https://example.com/page1",
        "https://example.com/page2",
        "https://example.com/page3"
    ]

    run_conf = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        stream=True  # Enable streaming mode
    )

    async with AsyncWebCrawler() as crawler:
        # Stream results as they complete
        async for result in await crawler.arun_many(urls, config=run_conf):
            if result.success:
                print(f"[OK] {result.url}, length: {len(result.markdown.raw_markdown)}")
            else:
                print(f"[ERROR] {result.url} => {result.error_message}")

        # Or get all results at once (default behavior)
        run_conf = run_conf.clone(stream=False)
        results = await crawler.arun_many(urls, config=run_conf)
        for res in results:
            if res.success:
                print(f"[OK] {res.url}, length: {len(res.markdown.raw_markdown)}")
            else:
                print(f"[ERROR] {res.url} => {res.error_message}")

if __name__ == "__main__":
    asyncio.run(quick_parallel_example())

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def extract_structured_data_using_css_extractor():
    print("\n--- Using JsonCssExtractionStrategy for Fast Structured Output ---")
    schema = {
        "name": "KidoCode Courses",
        "baseSelector": "section.charge-methodology .w-tab-content > div",
        "fields": [
            {
                "name": "section_title",
                "selector": "h3.heading-50",
                "type": "text",
            },
            {
                "name": "section_description",
                "selector": ".charge-content",
                "type": "text",
            },
            {
                "name": "course_name",
                "selector": ".text-block-93",
                "type": "text",
            },
            {
                "name": "course_description",
                "selector": ".course-content-text",
                "type": "text",
            },
            {
                "name": "course_icon",
                "selector": ".image-92",
                "type": "attribute",
                "attribute": "src",
            },
        ],
    }

    browser_config = BrowserConfig(headless=True, java_script_enabled=True)

    js_click_tabs = """
    (async () => {
        const tabs = document.querySelectorAll("section.charge-methodology .tabs-menu-3 > div");
        for(let tab of tabs) {
            tab.scrollIntoView();
            tab.click();
            await new Promise(r => setTimeout(r, 500));
        }
    })();
    """

    crawler_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(schema),
        js_code=[js_click_tabs],
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.kidocode.com/degrees/technology", config=crawler_config
        )

        companies = json.loads(result.extracted_content)
        print(f"Successfully extracted {len(companies)} companies")
        print(json.dumps(companies[0], indent=2))

async def main():
    await extract_structured_data_using_css_extractor()

if __name__ == "__main__":
    asyncio.run(main())


Simple Crawling

Summary: This guide covers the basics of web crawling with Crawl4AI, including setting up a crawler, making requests, understanding responses, and handling errors.

Basic Usage

Set up a simple crawl using `BrowserConfig` and `CrawlerRunConfig`:

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

async def main():
    browser_config = BrowserConfig()  # Default browser…

Understanding the Response

The `arun()` method returns a `CrawlResult` object with several useful properties. Here's a quick overview (see [CrawlResult](../../api/crawl-result/) for complete details):

config = CrawlerRunConfig(
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.6),
        options={"ignore_links": True}
    )
)

result = await…

Adding Basic Options

Customize your crawl using `CrawlerRunConfig`:

run_config = CrawlerRunConfig(
    word_count_threshold=10,        # Minimum words per content block
    exclude_external_links=True,    # Remove external links
    remove_overlay_elements=True,   #…

Handling Errors

Always check if the crawl was successful:

run_config = CrawlerRunConfig()
result = await crawler.arun(url="https://example.com", config=run_config)

if not result.success:
    print(f"Crawl failed: {result.error_message}")
    print(f"Status…

Logging and Debugging

Enable verbose logging in `BrowserConfig`:

browser_config = BrowserConfig(verbose=True)

async with AsyncWebCrawler(config=browser_config) as crawler:
    run_config = CrawlerRunConfig()
    result = await…

Complete Example

Here's a more comprehensive example demonstrating common usage patterns:

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_config = BrowserConfig(verbose=True)…


Table Extraction Strategies - Crawl4AI Documentation (v0.9.x)

Summary: This page covers Crawl4AI's table extraction strategies, including the default algorithm, LLM-based extraction, and custom strategies. It explains the strategy design pattern, configuration options,…

Overview

**New in v0.7.3+** : Table extraction now follows the **Strategy Design Pattern** , providing unprecedented flexibility and power for handling different table structures. Don't worry - **your…

What's Changed?

- **Architecture** : Table extraction now uses pluggable strategies
- **Backward Compatible** : Your existing code with `table_score_threshold` continues to work
- **More Power** : Choose from…

Key Points

✅ **Old code still works** - No breaking changes
✅ **Same default behavior** - Uses the proven extraction algorithm
✅ **New capabilities** - Add LLM extraction or custom strategies when needed
✅…

Quick Start

The Simplest Way (Works Like Before)

If you're already using Crawl4AI, nothing changes:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def extract_tables():
    async with AsyncWebCrawler() as crawler:
        # This works exactly like before - uses…

Using the Old Configuration (Still Supported)

Your existing code with `table_score_threshold` continues to work:

# This old approach STILL WORKS - we maintain backward compatibility
config = CrawlerRunConfig(
    table_score_threshold=7  # Internally creates…

Table Extraction Strategies

Understanding the Strategy Pattern

The strategy pattern allows you to choose different table extraction algorithms at runtime. Think of it as having different tools in a toolbox - you pick the right one for the job:

- **No explicit…

Available Strategies

| Strategy | Description | Use Case | Cost | When to Use |
| --- | --- | --- | --- | --- |
| `DefaultTableExtraction` | **RECOMMENDED** : Same algorithm as before v0.7.3 | General purpose (default) |…

DefaultTableExtraction

The default strategy uses a sophisticated scoring system to identify data tables:

from crawl4ai import DefaultTableExtraction, CrawlerRunConfig

# Customize the default extraction
table_strategy = DefaultTableExtraction(
    table_score_threshold=7,  # Scoring threshold (default:…

Scoring System

The scoring system evaluates multiple factors:

| Factor | Score Impact | Description |
| --- | --- | --- |
| Has `<thead>` | +2 | Semantic table structure |
| Has `<tbody>` | +1 | Organized table…

LLMTableExtraction (Use Sparingly!)

**⚠ WARNING** : Only use this when `DefaultTableExtraction` fails with complex tables!

LLMTableExtraction uses AI to understand complex table structures that traditional parsers struggle with. It…

from crawl4ai import LLMTableExtraction, LLMConfig, CrawlerRunConfig

# Configure LLM (costs money per call!)
llm_config = LLMConfig(
    provider="groq/llama-3.3-70b-versatile",  # Fast provider for…

When to Use LLMTableExtraction

✅ **Use ONLY when** :
- Tables have complex merged cells (rowspan/colspan) that break DefaultTableExtraction
- Nested tables that need semantic understanding
- Tables with irregular structures
-…

How Smart Chunking Works

LLMTableExtraction automatically handles large tables through intelligent chunking:

- **Automatic Detection** : Tables exceeding the token threshold are automatically split
- **Smart Splitting** :…

Performance Optimization for LLMTableExtraction

**Provider Recommendations by Table Size** :

| Table Size | Recommended Providers | Why |
| --- | --- | --- |
| Small (<50 rows) | Any provider | Fast enough |
| Medium (50-200 rows) | Groq,…

NoTableExtraction

Disable table extraction for better performance when tables aren't needed:

from crawl4ai import NoTableExtraction, CrawlerRunConfig

config = CrawlerRunConfig(
    table_extraction=NoTableExtraction()
)

# Tables won't be extracted, improving performance
result = await…

Extracted Table Structure

Each extracted table contains:

{
    "headers": ["Column 1", "Column 2", ...],  # Column headers
    "rows": [                                   # Data rows
        ["Row 1 Col 1", "Row 1 Col 2", ...],
        ["Row 2 Col 1", "Row…

Configuration Options

Basic Configuration

config = CrawlerRunConfig(
    # Table extraction settings
    table_score_threshold=7,      # Default threshold (backward compatible)
    table_extraction=strategy,     # Optional: custom strategy…

Advanced Configuration

from crawl4ai import DefaultTableExtraction, CrawlerRunConfig

# Fine-tuned extraction
strategy = DefaultTableExtraction(
    table_score_threshold=5,      # Lower = more permissive
    min_rows=3,…

Working with Extracted Tables

Convert to Pandas DataFrame

import pandas as pd

async def tables_to_dataframes(url):
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url)

        dataframes = []
        for table_data in…

Filter Tables by Criteria

async def extract_large_tables(url):
    async with AsyncWebCrawler() as crawler:
        # Configure minimum size requirements
        strategy = DefaultTableExtraction(
            min_rows=10,…

Export Tables to Different Formats

import json
import csv

async def export_tables(url):
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url)

        for i, table in enumerate(result.tables):…

Creating Custom Strategies

Extend `TableExtractionStrategy` to create custom extraction logic:

Example: Financial Table Extractor

from crawl4ai import TableExtractionStrategy
from typing import List, Dict, Any
import re

class FinancialTableExtractor(TableExtractionStrategy):
    """Extract tables containing financial…

Example: Specific Table Extractor

class SpecificTableExtractor(TableExtractionStrategy):
    """Extract only tables matching specific criteria."""

    def __init__(self,
                 required_headers=None,…

Combining with Other Strategies

Table extraction works seamlessly with other Crawl4AI strategies:

from crawl4ai import (
    AsyncWebCrawler,
    CrawlerRunConfig,
    DefaultTableExtraction,
    LLMExtractionStrategy,
    JsonCssExtractionStrategy
)

async def combined_extraction(url):
    async…

Performance Considerations

Optimization Tips

- **Disable when not needed** : Use `NoTableExtraction` if tables aren't required
- **Target specific areas** : Use `css_selector` to limit processing scope
- **Set minimum thresholds** : Filter out…

# Optimized configuration for large pages
config = CrawlerRunConfig(
    # Only process main content area
    css_selector="article.main-content",

    # Exclude navigation and sidebars…

Migration Guide

Important: Your Code Still Works!

**No changes required!** The transition to the strategy pattern is **fully backward compatible** .

How It Works Internally

v0.7.2 and Earlier

# Old way - directly passing table_score_threshold
config = CrawlerRunConfig(
    table_score_threshold=7
)
# Internally: No strategy pattern, direct implementation

v0.7.3+ (Current)

# Old way STILL WORKS - we handle it internally
config = CrawlerRunConfig(
    table_score_threshold=7
)
# Internally: Automatically creates DefaultTableExtraction(table_score_threshold=7)

Taking Advantage of New Features

While your old code works, you can now use the strategy pattern for more control:

# Option 1: Keep using the old way (perfectly fine!)
config = CrawlerRunConfig(
    table_score_threshold=7  # Still supported
)

# Option 2: Use the new strategy pattern (more flexibility)
from…

Summary

- ✅ **No breaking changes** - Old code works as-is
- ✅ **Same defaults** - DefaultTableExtraction is automatically used
- ✅ **Gradual adoption** - Use new features when you need them
- ✅ **Full…

Best Practices

1. Choose the Right Strategy (Cost-Conscious Approach)

**Decision Flow** :

**Strategy Selection Guide** :
- **DefaultTableExtraction** : Use for 99% of cases - it's free and effective
- **LLMTableExtraction** : Only for complex tables with merged cells…

1. Do you need tables?
   → No: Use NoTableExtraction
   → Yes: Continue to #2

2. Try DefaultTableExtraction first (FREE)
   → Works? Done! ✅
   → Fails? Continue to #3

3. Is the table critical…

2. Validate Extracted Data

def validate_table(table):
    """Validate table data quality."""
    # Check structure
    if not table.get('rows'):
        return False

    # Check consistency
    if table.get('headers'):…

3. Handle Edge Cases

async def robust_table_extraction(url):
    """Extract tables with error handling."""
    async with AsyncWebCrawler() as crawler:
        try:
            config = CrawlerRunConfig(…

Troubleshooting

Common Issues and Solutions

| Issue | Cause | Solution |
| --- | --- | --- |
| No tables extracted | Score too high | Lower `table_score_threshold` |
| Layout tables included | Score too low | Increase `table_score_threshold`…

Debug Logging

Enable verbose logging to understand extraction decisions:

import logging

# Configure logging
logging.basicConfig(level=logging.DEBUG)

# Enable verbose mode in strategy
strategy = DefaultTableExtraction(
    table_score_threshold=7,
    verbose=True  #…

See Also

- [Extraction Strategies](extraction-strategies.md) - Overview of all extraction strategies
- [Content Selection](../content-selection/) - Using CSS selectors and filters
- [Performance…


URL Seeding: The Smart Way to Crawl at Scale

Summary: This page explains how to use URL seeding to discover and filter URLs before crawling, covering configuration, smart filtering with BM25 scoring, and scaling across multiple domains.

Why URL Seeding?

Web crawling comes in different flavors, each with its own strengths. Let's understand when to use URL seeding versus deep crawling.

Deep Crawling: Real-Time Discovery

Deep crawling is perfect when you need:
- **Fresh, real-time data** - discovering pages as they're created
- **Dynamic exploration** - following links based on content
- **Selective extraction** -…

# Deep crawling example: Explore a website dynamically
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

async def…

URL Seeding: Bulk Discovery

URL seeding shines when you want:
- **Comprehensive coverage** - get thousands of URLs in seconds
- **Bulk processing** - filter before crawling
- **Resource efficiency** - know exactly what you'll…

# URL seeding example: Analyze all documentation
from crawl4ai import AsyncUrlSeeder, SeedingConfig

seeder = AsyncUrlSeeder()
config = SeedingConfig(
    source="sitemap",
    extract_head=True,…

The Trade-offs

| Aspect | Deep Crawling | URL Seeding |
| --- | --- | --- |
| **Coverage** | Discovers pages dynamically | Gets most existing URLs instantly |
| **Freshness** | Finds brand new pages | May miss very…

When to Use Each

**Choose Deep Crawling when:**
- You need the absolute latest content
- You're searching for specific information
- The site structure is unknown or dynamic
- You want to stop as soon as you find…

Your First URL Seeding Adventure

Let's see the magic in action. We'll discover blog posts about Python, filter for tutorials, and crawl only those pages.

**What just happened?**
- We discovered all blog URLs from the sitemap+cc
-…

import asyncio
from crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, SeedingConfig, CrawlerRunConfig

async def smart_blog_crawler():
    # Step 1: Create our URL discoverer
    seeder =…

Understanding the URL Seeder

Now that you've seen the magic, let's understand how it works.

Basic Usage

Creating a URL seeder is simple:

The seeder can discover URLs from two powerful sources:

from crawl4ai import AsyncUrlSeeder

# Method 1: Manual cleanup
seeder = AsyncUrlSeeder()
try:
    config = SeedingConfig(source="sitemap")
    urls = await seeder.urls("example.com",…

1. Sitemaps (Fastest)

Sitemaps are XML files that websites create specifically to list all their URLs. It's like getting a menu at a restaurant - everything is listed upfront.

**Sitemap Index Support**: For large…

# Discover from sitemap
config = SeedingConfig(source="sitemap")
urls = await seeder.urls("example.com", config)

<!-- Example sitemap index -->
<sitemapindex>
  <sitemap>
    <loc>https://techcrunch.com/sitemap-1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://techcrunch.com/sitemap-2.xml</loc>…

2. Common Crawl (Most Comprehensive)

Common Crawl is a massive public dataset that regularly crawls the entire web. It's like having access to a pre-built index of the internet.

# Discover from Common Crawl
config = SeedingConfig(source="cc")
urls = await seeder.urls("example.com", config)

3. Both Sources (Maximum Coverage)

# Use both sources
config = SeedingConfig(source="sitemap+cc")
urls = await seeder.urls("example.com", config)

Configuration Magic: SeedingConfig

The `SeedingConfig` object is your control panel. Here's everything you can configure:

Pattern Matching Examples

# Match all blog posts
config = SeedingConfig(pattern="*/blog/*")

# Match only HTML files
config = SeedingConfig(pattern="*.html")

# Match product pages
config =…

URL Validation: Live Checking

Sometimes you need to know if URLs are actually accessible. That's where live checking comes in:

**When to use live checking:**
- Before a large crawling operation
- When working with older…

config = SeedingConfig(
    source="sitemap",
    live_check=True,  # Verify each URL is accessible
    concurrency=20    # Check 20 URLs in parallel
)
async with AsyncUrlSeeder() as seeder:
    urls…

The Power of Metadata: Head Extraction

This is where URL seeding gets really powerful. Instead of crawling entire pages, you can extract just the metadata:

config = SeedingConfig(
    extract_head=True  # Extract metadata from <head> section
)
async with AsyncUrlSeeder() as seeder:
    urls = await seeder.urls("example.com", config)

# Now each URL has…

What Can We Extract?

The head extraction gives you a treasure trove of information:

# Example of extracted head_data
{
    "title": "10 Python Tips for Beginners",
    "charset": "utf-8",
    "lang": "en",
    "meta": {
        "description": "Learn essential Python tips...",…

Smart URL-Based Filtering (No Head Extraction)

When `extract_head=False` but you still provide a query, the seeder uses intelligent URL-based scoring:

This approach is much faster than head extraction while still providing intelligent filtering!

# Fast filtering based on URL structure alone
config = SeedingConfig(
    source="sitemap",
    extract_head=False,  # Don't fetch page metadata
    query="python tutorial async",…

Understanding Results

Each URL in the results has this structure:

Let's see a real example:

{
    "url": "https://example.com/blog/python-tips.html",
    "status": "valid",        # "valid", "not_valid", or "unknown"
    "head_data": {            # Only if extract_head=True
        "title":…

config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    live_check=True
)
async with AsyncUrlSeeder() as seeder:
    urls = await seeder.urls("blog.example.com", config)

# Analyze…

Smart Filtering with BM25 Scoring

Now for the really cool part - intelligent filtering based on relevance!

Introduction to Relevance Scoring

BM25 is a ranking algorithm that scores how relevant a document is to a search query. With URL seeding, we can score URLs based on their metadata *before* crawling them.

Think of it like this:
-…

Query-Based Discovery

Here's how to use BM25 scoring:

config = SeedingConfig(
    source="sitemap",
    extract_head=True,           # Required for scoring
    query="python async tutorial",  # What we're looking for
    scoring_method="bm25",       #…

Real Examples

Finding Documentation Pages

# Find API documentation
config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    query="API reference documentation endpoints",
    scoring_method="bm25",
    score_threshold=0.5,…

Discovering Product Pages

# Find specific products
config = SeedingConfig(
    source="sitemap+cc",  # Use both sources
    extract_head=True,
    query="wireless headphones noise canceling",
    scoring_method="bm25",…

Filtering News Articles

# Find recent news about AI
config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    query="artificial intelligence machine learning breakthrough",
    scoring_method="bm25",…

Complex Query Patterns

# Multi-concept queries
queries = [
    "python async await concurrency tutorial",
    "data science pandas numpy visualization",
    "web scraping beautifulsoup selenium automation",
    "machine…

Scaling Up: Multiple Domains

When you need to discover URLs across multiple websites, URL seeding really shines.

The `many_urls` Method

# Discover URLs from multiple domains in parallel
domains = ["site1.com", "site2.com", "site3.com"]

config = SeedingConfig(
    source="sitemap",
    extract_head=True,
    query="python tutorial",…

Cross-Domain Examples

Competitor Analysis

# Analyze content strategies across competitors
competitors = [
    "competitor1.com",
    "competitor2.com",
    "competitor3.com"
]

config = SeedingConfig(
    source="sitemap",…

Industry Research

# Research Python tutorials across educational sites
educational_sites = [
    "realpython.com",
    "pythontutorial.net",
    "learnpython.org",
    "python.org"
]

config = SeedingConfig(…

Multi-Site Monitoring

# Monitor news about your company across multiple sources
news_sites = [
    "techcrunch.com",
    "theverge.com",
    "wired.com",
    "arstechnica.com"
]

company_name = "YourCompany"

config =…

Advanced Integration Patterns

Let's put everything together in a real-world example.

Building a Research Assistant

Here's a complete example that discovers, scores, filters, and crawls intelligently:

import asyncio
from datetime import datetime
from crawl4ai import AsyncUrlSeeder, AsyncWebCrawler, SeedingConfig, CrawlerRunConfig

class ResearchAssistant:
    def __init__(self):…

Performance Optimization Tips

# First run - populate cache
config = SeedingConfig(source="sitemap", extract_head=True, force=True)
urls = await seeder.urls("example.com", config)

# Subsequent runs - use cache (much…

# For many small requests (like HEAD checks)
config = SeedingConfig(concurrency=50, hits_per_sec=20)

# For fewer large requests (like full head extraction)
config = SeedingConfig(concurrency=10,…

# When crawling many URLs
async with AsyncWebCrawler() as crawler:
    # Assuming urls is a list of URL strings
    crawl_results = await crawler.arun_many(urls, config=config)

    # Process as they…

# Safe for domains with 1M+ URLs
config = SeedingConfig(
    source="cc+sitemap",
    concurrency=50,  # Queue size adapts to concurrency
    max_urls=100000  # Process in batches if needed
)

# The…

Best Practices & Tips

Cache Management

The seeder automatically caches results to speed up repeated operations:

- **Common Crawl cache** : `~/.crawl4ai/seeder_cache/[index]_[domain]_[hash].jsonl`
- **Sitemap cache** : `~/.crawl4ai/seeder_cache/sitemap_[domain]_[hash].json`
- **HEAD data cache** :…

Smart TTL Cache for Sitemaps

Sitemap caches now include intelligent validation:

**Cache validation priority:**
1. `force=True` → Always refetch
2. Cache doesn't exist → Fetch fresh
3. `validate_sitemap_lastmod=True` and…

# Default: 24-hour TTL with lastmod validation
config = SeedingConfig(
    source="sitemap",
    cache_ttl_hours=24,              # Cache expires after 24 hours
    validate_sitemap_lastmod=True    #…

Pattern Matching Strategies

# Be specific when possible
good_pattern = "*/blog/2024/*.html"  # Specific
bad_pattern = "*"                     # Too broad

# Combine patterns with metadata filtering
config = SeedingConfig(…

Rate Limiting Considerations

# Be respectful of servers
config = SeedingConfig(
    hits_per_sec=10,      # Max 10 requests per second
    concurrency=20        # But use 20 workers
)

# For your own servers
config =…

Quick Reference

Common Patterns

# Blog post discovery
config = SeedingConfig(
    source="sitemap",
    pattern="*/blog/*",
    extract_head=True,
    query="your topic",
    scoring_method="bm25"
)

# E-commerce product…

Troubleshooting Guide

| Issue | Solution |
| --- | --- |
| No URLs found | Try `source="cc+sitemap"`, check domain spelling |
| Slow discovery | Reduce `concurrency`, add `hits_per_sec` limit |
| Missing metadata | Ensure…

Performance Benchmarks

Typical performance on a standard connection:
- **Sitemap discovery** : 100-1,000 URLs/second
- **Common Crawl discovery** : 50-500 URLs/second
- **HEAD checking** : 10-50 URLs/second
- **Head…

Conclusion

URL seeding transforms web crawling from a blind expedition into a surgical strike. By discovering and analyzing URLs before crawling, you can:
- Save hours of crawling time
- Reduce bandwidth usage…

Smart URL Filtering

The seeder automatically filters out nonsense URLs that aren't useful for content crawling:

To disable filtering (not recommended):

# Enabled by default
config = SeedingConfig(
    source="sitemap",
    filter_nonsense_urls=True  # Default: True
)

# URLs that get filtered:
# - robots.txt, sitemap.xml, ads.txt
# - API endpoints…

config = SeedingConfig(
    source="sitemap",
    filter_nonsense_urls=False  # Include ALL URLs
)

Key Features Summary

- **Parallel Sitemap Index Processing** : Automatically detects and processes sitemap indexes in parallel
- **Memory Protection** : Bounded queues prevent RAM issues with large domains (1M+ URLs)
-…

Need More Coverage?

If you need to discover URLs across an entire domain — including subdomains, hidden services, and pages not listed in any sitemap — check out [Domain Mapping](../domain-mapping/). It combines 8…

source

URL source: "cc" (Common Crawl), "sitemap", or "sitemap+cc"

pattern

URL pattern filter (e.g., "*/blog/*", "*.html")

Extract metadata from page <head>

Verify URLs are accessible

Maximum URLs to return (-1 = unlimited)

concurrency

Parallel workers for fetching

Rate limit for requests

force

Bypass cache, fetch fresh data

verbose

Show detailed progress

query

Search query for BM25 scoring

Scoring method (currently "bm25")

Minimum score to include URL

Filter out utility URLs (robots.txt, etc.)

Hours before sitemap cache expires (0 = no TTL)

Check sitemap's lastmod and refetch if newer


Chunking Strategies

Summary: This page explains various chunking strategies for dividing large texts into manageable parts, including regex-based, sentence-based, topic-based, fixed-length word, and sliding window chunking, as…

Chunking Strategies

Chunking strategies are critical for dividing large texts into manageable parts, enabling effective content processing and extraction. These strategies are foundational in cosine similarity-based…

Why Use Chunking?

1. **Cosine Similarity and Query Relevance**: Prepares chunks for semantic similarity analysis.
2. **RAG System Integration**: Seamlessly processes and stores chunks for retrieval.
3. **Structured…

Methods of Chunking

1. Regex-Based Chunking

Splits text based on regular expression patterns, useful for coarse segmentation.

**Code Example**:

class RegexChunking:
    def __init__(self, patterns=None):
        self.patterns = patterns or [r'\n\n']  # Default pattern for paragraphs

    def chunk(self, text):
        paragraphs = [text]…

2. Sentence-Based Chunking

Divides text into sentences using NLP tools, ideal for extracting meaningful statements.

**Code Example**:

from nltk.tokenize import sent_tokenize

class NlpSentenceChunking:
    def chunk(self, text):
        sentences = sent_tokenize(text)
        return [sentence.strip() for sentence in sentences]

#…

3. Topic-Based Segmentation

Uses algorithms like TextTiling to create topic-coherent chunks.

**Code Example**:

from nltk.tokenize import TextTilingTokenizer

class TopicSegmentationChunking:
    def __init__(self):
        self.tokenizer = TextTilingTokenizer()

    def chunk(self, text):
        return…

4. Fixed-Length Word Chunking

Segments text into chunks of a fixed word count.

**Code Example**:

class FixedLengthWordChunking:
    def __init__(self, chunk_size=100):
        self.chunk_size = chunk_size

    def chunk(self, text):
        words = text.split()
        return [' '.join(words[i:i…

5. Sliding Window Chunking

Generates overlapping chunks for better contextual coherence.

**Code Example**:

class SlidingWindowChunking:
    def __init__(self, window_size=100, step=50):
        self.window_size = window_size
        self.step = step

    def chunk(self, text):
        words =…

Combining Chunking with Cosine Similarity

To enhance the relevance of extracted content, chunking strategies can be paired with cosine similarity techniques. Here’s an example workflow:

**Code Example**:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

class CosineSimilarityExtractor:
    def __init__(self, query):
        self.query…


Clustering Strategies - Crawl4AI Documentation (v0.9.x)

Summary: This page provides a comprehensive guide to using the Cosine Strategy in Crawl4AI for semantic content extraction, covering configuration options, usage examples, best practices, and error handling.

Cosine Strategy

The Cosine Strategy in Crawl4AI uses similarity-based clustering to identify and extract relevant content sections from web pages. This strategy is particularly useful when you need to find and…

How It Works

The Cosine Strategy:
1. Breaks down page content into meaningful chunks
2. Converts text into vector representations
3. Calculates similarity between chunks
4. Clusters similar content together
5.…

Basic Usage

from crawl4ai import CosineStrategy

strategy = CosineStrategy(
    semantic_filter="product reviews",    # Target content type
    word_count_threshold=10,             # Minimum words per cluster…

Configuration Options

Core Parameters

CosineStrategy(
    # Content Filtering
    semantic_filter: str = None,       # Keywords/topic for content filtering
    word_count_threshold: int = 10,    # Minimum words per cluster…

Parameter Details

1. **semantic_filter**
   - Sets the target topic or content type
   - Use keywords relevant to your desired content
   - Example: "technical specifications", "user reviews", "pricing…

# Strict matching
strategy = CosineStrategy(sim_threshold=0.8)

# Loose matching
strategy = CosineStrategy(sim_threshold=0.3)

# Only consider substantial paragraphs
strategy = CosineStrategy(word_count_threshold=50)

# Get top 5 most relevant content clusters
strategy = CosineStrategy(top_k=5)

Use Cases

1. Article Content Extraction

strategy = CosineStrategy(
    semantic_filter="main article content",
    word_count_threshold=100,  # Longer blocks for articles
    top_k=1                   # Usually want single main…

2. Product Review Analysis

strategy = CosineStrategy(
    semantic_filter="customer reviews and ratings",
    word_count_threshold=20,   # Reviews can be shorter
    top_k=10,                 # Get multiple reviews…

3. Technical Documentation

strategy = CosineStrategy(
    semantic_filter="technical specifications documentation",
    word_count_threshold=30,
    sim_threshold=0.6,        # Stricter matching for technical content…

Advanced Features

Custom Clustering

strategy = CosineStrategy(
    linkage_method='complete',  # Alternative clustering method
    max_dist=0.4,              # Larger clusters…

Content Filtering Pipeline

strategy = CosineStrategy(
    semantic_filter="pricing plans features",
    word_count_threshold=15,
    sim_threshold=0.5,
    top_k=3
)

async def extract_pricing_features(url: str):
    async…

Best Practices

1. **Adjust Thresholds Iteratively**
   - Start with default values
   - Adjust based on results
   - Monitor clustering quality

2. **Choose Appropriate Word Count Thresholds**
   - Higher for…

strategy = CosineStrategy(
    word_count_threshold=10,  # Filter early
    top_k=5,                 # Limit results
    verbose=True             # Monitor performance
)

# For mixed content pages
strategy = CosineStrategy(
    semantic_filter="product features",
    sim_threshold=0.4,      # More flexible matching
    max_dist=0.3,          # Larger clusters…

Error Handling

try:
    result = await crawler.arun(
        url="https://example.com",
        extraction_strategy=strategy
    )

    if result.success:
        content = json.loads(result.extracted_content)…

Conclusion

The Cosine Strategy is particularly effective when:
- Content structure is inconsistent
- You need semantic understanding
- You want to find similar content blocks
- Structure-based extraction…