Extracting JSON (LLM)

Summary: A guide to using Crawl4AI's LLM-based extraction strategy to extract structured JSON from web pages using any LLM via LiteLLM, including schema definition, chunking, input formats, and practical…

Overview

In some cases, you need to extract **complex or unstructured** information from a webpage that a simple CSS/XPath schema cannot easily parse. Or you want **AI**-driven insights, classification, or…

1. Why Use an LLM?

- **Complex Reasoning**: If the site’s data is unstructured, scattered, or full of natural language context.
- **Semantic Extraction**: Summaries, knowledge graphs, or relational data that require…

2. Provider-Agnostic via LiteLLM

You can use LLMConfig, to quickly configure multiple variations of LLMs and experiment with them to find the optimal one for your use case. You can read more about LLMConfig…

llm_config = LLMConfig(provider="openai/gpt-4o-mini", api_token=os.getenv("OPENAI_API_KEY"))

3. How LLM Extraction Works

3.1 Flow

1. **Chunking** (optional): The HTML or markdown is split into smaller segments if it’s very long (based on `chunk_token_threshold`, overlap, etc.).
2. **Prompt Construction**: For each chunk, the…

3.2 `extraction_type`

- **`"schema"`**: The model tries to return JSON conforming to your Pydantic-based schema.
- **`"block"`**: The model returns freeform text, or smaller JSON structures, which the library…

4. Key Parameters

Below is an overview of important LLM extraction parameters. All are typically set inside `LLMExtractionStrategy(...)`. You then put that strategy in your `CrawlerRunConfig(...,…

extraction_strategy = LLMExtractionStrategy(
    llm_config = LLMConfig(provider="openai/gpt-4", api_token="YOUR_OPENAI_KEY"),
    schema=MyModel.model_json_schema(),
    extraction_type="schema",…

5. Putting It in `CrawlerRunConfig`

**Important**: In Crawl4AI, all strategy definitions should go inside the `CrawlerRunConfig`, not directly as a param in `arun()`. Here’s a full example:

import os
import asyncio
import json
from pydantic import BaseModel, Field
from typing import List
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from…

6. Chunking Details

6.1 `chunk_token_threshold`

If your page is large, you might exceed your LLM’s context window. **`chunk_token_threshold`** sets the approximate max tokens per chunk. The library calculates word→token ratio using…

6.2 `overlap_rate`

To keep context continuous across chunks, we can overlap them. E.g., `overlap_rate=0.1` means each subsequent chunk includes 10% of the previous chunk’s text. This is helpful if your needed info…

6.3 Performance & Parallelism

By chunking, you can potentially process multiple chunks in parallel (depending on your concurrency settings and the LLM provider). This reduces total time if the site is huge or has many sections.

7. Input Format

By default, **LLMExtractionStrategy** uses `input_format="markdown"`, meaning the **crawler’s final markdown** is fed to the LLM. You can change to:

- **`html`**: The cleaned HTML or raw HTML…

LLMExtractionStrategy(
    # ...
    input_format="html",  # Instead of "markdown" or "fit_markdown"
)

8. Token Usage & Show Usage

To keep track of tokens and cost, each chunk is processed with an LLM call. We record usage in:

- **`usages`** (list): token usage per chunk or call.
- **`total_usage`**: sum of all chunk calls.
-…

llm_strategy = LLMExtractionStrategy(...)
# ...
llm_strategy.show_usage()
# e.g. “Total usage: 1241 tokens across 2 chunk calls”

9. Example: Building a Knowledge Graph

Below is a snippet combining **`LLMExtractionStrategy`** with a Pydantic schema for a knowledge graph. Notice how we pass an **`instruction`** telling the model what to parse.

**Key…

import os
import json
import asyncio
from typing import List
from pydantic import BaseModel, Field
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig
from…

10. Best Practices & Caveats

1. **Cost & Latency**: LLM calls can be slow or expensive. Consider chunking or smaller coverage if you only need partial data.
2. **Model Token Limits**: If your page + instruction exceed the…

11. Conclusion

**LLM-based extraction** in Crawl4AI is **provider-agnostic**, letting you choose from hundreds of models via LiteLLM. It’s perfect for **semantically complex** tasks or generating advanced…


Extracting JSON (No LLM)

Summary: This page covers Crawl4AI's LLM-free extraction strategies, including schema-based extraction with CSS/XPath selectors (JsonCssExtractionStrategy, JsonXPathExtractionStrategy) and regex-based…

1. Intro to Schema-Based Extraction

A schema defines:

- A **base selector** that identifies each "container" element on the page (e.g., a product row, a blog post card).
- **Fields** describing which CSS/XPath selectors to use for…

2. Simple Example: Crypto Prices

Let's begin with a **simple** schema-based extraction using the `JsonCssExtractionStrategy`. Below is a snippet that extracts cryptocurrency prices from a site (similar to the legacy Coinbase…

import json
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai import JsonCssExtractionStrategy

async def extract_crypto_prices():
    # 1. Define a…

import json
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import JsonXPathExtractionStrategy

async def extract_crypto_prices_xpath():
    # 1. Minimal dummy…

3. Advanced Schema & Nested Structures

Real sites often have **nested** or repeated data—like categories containing products, which themselves have a list of reviews or features. For that, we can define **nested** or **list** (and even…

schema = {
    "name": "E-commerce Product Catalog",
    "baseSelector": "div.category",
    # (1) We can define optional baseFields if we want to extract attributes
    # from the category…

import json
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai import JsonCssExtractionStrategy

ecommerce_schema = {
    # ... the advanced schema from above…

4. RegexExtractionStrategy - Fast Pattern-Based Extraction

Crawl4AI now offers a powerful new zero-LLM extraction strategy: `RegexExtractionStrategy`. This strategy provides lightning-fast extraction of common data types like emails, phone numbers, URLs,…

import json
import asyncio
from crawl4ai import (
    AsyncWebCrawler,
    CrawlerRunConfig,
    RegexExtractionStrategy
)

async def extract_with_regex():
    # Create a strategy using built-in…

# Use individual patterns
strategy = RegexExtractionStrategy(pattern=RegexExtractionStrategy.Email)

# Combine multiple patterns
strategy = RegexExtractionStrategy(
    pattern = (…

import json
import asyncio
from crawl4ai import (
    AsyncWebCrawler,
    CrawlerRunConfig,
    RegexExtractionStrategy
)

async def extract_prices():
    # Define a custom pattern for US Dollar…

import json
import asyncio
from pathlib import Path
from crawl4ai import (
    AsyncWebCrawler,
    CrawlerRunConfig,
    RegexExtractionStrategy,
    LLMConfig
)

async def…

[
  {
    "url": "https://example.com",
    "label": "email",
    "value": "contact@example.com",
    "span": [145, 163]
  },
  {
    "url": "https://example.com",
    "label": "url",
    "value":…

5. Why "No LLM" Is Often Better

- **Zero Hallucination**: Pattern-based extraction doesn't guess text. It either finds it or not.
- **Guaranteed Structure**: The same schema or regex yields consistent JSON across many pages, so…

6. Base Element Attributes & Additional Fields

It's easy to **extract attributes** (like `href`, `src`, or `data-xxx`) from your base or nested elements using:

```json
{
  "name": "href",
  "type": "attribute",
  "attribute": "href",…

7. Putting It All Together: Larger Example

Consider a blog site. We have a schema that extracts the **URL** from each post card (via `baseFields` with an `"attribute": "href"`), plus the title, date, summary, and author:

```json
schema = {…

schema = {
  "name": "Blog Posts",
  "baseSelector": "a.blog-post-card",
  "baseFields": [
    {"name": "post_url", "type": "attribute", "attribute": "href"}
  ],
  "fields": [
    {"name": "title",…

8. Extracting Sibling Data with `source`

Some websites split a single logical item across **sibling elements** rather than nesting everything inside one container. A classic example is Hacker News, where each submission spans two adjacent…

schema = {
    "name": "HN Submissions",
    "baseSelector": "tr.athing.submission",
    "fields": [
        {"name": "rank", "selector": "span.rank", "type": "text"},
        {"name": "title",…

9. Tips & Best Practices

- **Inspect the DOM** in Chrome DevTools or Firefox's Inspector to find stable selectors.
- **Start Simple**: Verify you can extract a single field. Then add complexity like nested objects or…

10. Schema Generation Utility

While manually crafting schemas is powerful and precise, Crawl4AI now offers a convenient utility to **automatically generate** extraction schemas using LLM. This is particularly useful when:

-…

from crawl4ai import JsonCssExtractionStrategy, JsonXPathExtractionStrategy
from crawl4ai import LLMConfig

# Sample HTML with product information
html = """
<div class="product-card">
    <h2…

# Default: validated (recommended)
schema = JsonCssExtractionStrategy.generate_schema(
    url="https://news.ycombinator.com",
    query="Extract each story: title, url, score, author",
)

# Skip…

from crawl4ai import JsonCssExtractionStrategy
from crawl4ai.models import TokenUsage

usage = TokenUsage()

schema = JsonCssExtractionStrategy.generate_schema(…

usage = TokenUsage()
schema1 = JsonCssExtractionStrategy.generate_schema(url=url1, query=q1, usage=usage)
schema2 = JsonCssExtractionStrategy.generate_schema(url=url2, query=q2,…

from crawl4ai import JsonCssExtractionStrategy, LLMConfig

# Collect HTML samples from different pages
html_sample_1 = """

Brand	Apple
…

11. Conclusion

With Crawl4AI's LLM-free extraction strategies - `JsonCssExtractionStrategy`, `JsonXPathExtractionStrategy`, and now `RegexExtractionStrategy` - you can build powerful pipelines that:

- Scrape any…


Migration Guide: Table Extraction v0.7.3

Summary: A migration guide for Crawl4AI v0.7.3 introducing the Table Extraction Strategy Pattern, highlighting new classes and options, full backward compatibility, migration scenarios, code organization…

Overview

Version 0.7.3 introduces the **Table Extraction Strategy Pattern** , providing a more flexible and extensible approach to table extraction while maintaining full backward compatibility.

What's New

Strategy Pattern Implementation

Table extraction now follows the same strategy pattern used throughout Crawl4AI:

- **Consistent Architecture** : Aligns with extraction, chunking, and markdown strategies
- **Extensibility** : Easy…

New Classes

from crawl4ai import (
    TableExtractionStrategy,    # Abstract base class
    DefaultTableExtraction,      # Current implementation (default)
    NoTableExtraction           # Explicitly disable…

Backward Compatibility

**✅ All existing code continues to work without changes.**

No Changes Required

If your code looks like this, it will continue to work:

# This still works exactly the same
config = CrawlerRunConfig(
    table_score_threshold=7
)
result = await crawler.arun(url, config)
tables = result.tables  # Same structure, same data

What Happens Behind the Scenes

When you don't specify a `table_extraction` strategy:

- `CrawlerRunConfig` automatically creates `DefaultTableExtraction`
- It uses your `table_score_threshold` parameter
- Tables are extracted…

New Capabilities

1. Explicit Strategy Configuration

You can now explicitly configure table extraction:

# New: Explicit control
strategy = DefaultTableExtraction(
    table_score_threshold=7,
    min_rows=2,              # New: minimum row filter
    min_cols=2,              # New: minimum column…

2. Disable Table Extraction

Improve performance when tables aren't needed:

# New: Skip table extraction entirely
config = CrawlerRunConfig(
    table_extraction=NoTableExtraction()
)
# No CPU cycles spent on table detection/extraction

3. Custom Extraction Strategies

Create specialized extractors:

class MyTableExtractor(TableExtractionStrategy):
    def extract_tables(self, element, **kwargs):
        # Custom extraction logic
        return custom_tables

config = CrawlerRunConfig(…

Migration Scenarios

Scenario 1: Basic Usage (No Changes Needed)

**Before (v0.7.2):**

**After (v0.7.3):**

# Exactly the same - no changes required
config = CrawlerRunConfig()
result = await crawler.arun(url, config)
for table in result.tables:
    print(table['headers'])

Scenario 2: Custom Threshold (No Changes Needed)

# Still works the same
config = CrawlerRunConfig(
    table_score_threshold=5
)

# Or use new explicit approach for more control
strategy = DefaultTableExtraction(
    table_score_threshold=5,…

Scenario 3: Advanced Filtering (New Feature)

# Had to filter after extraction
config = CrawlerRunConfig(
    table_score_threshold=5
)
result = await crawler.arun(url, config)

# Manual filtering
large_tables = [
    t for t in result.tables…

# Filter during extraction (more efficient)
strategy = DefaultTableExtraction(
    table_score_threshold=5,
    min_rows=5,
    min_cols=3
)
config = CrawlerRunConfig(…

Code Organization Changes

Module Structure

crawl4ai/
  content_scraping_strategy.py
    - LXMLWebScrapingStrategy
      - is_data_table()      # Table detection
      - extract_table_data() # Table extraction

crawl4ai/
  content_scraping_strategy.py
    - LXMLWebScrapingStrategy
      # Table methods removed, uses strategy

  table_extraction.py (NEW)
    - TableExtractionStrategy    # Base class
    -…

Import Changes

**New imports available (optional):**

# These are now available but not required for existing code
from crawl4ai import (
    TableExtractionStrategy,
    DefaultTableExtraction,
    NoTableExtraction
)

Performance Implications

No Performance Impact

For existing code, performance remains identical:
- Same extraction logic
- Same scoring algorithm
- Same processing time

Performance Improvements Available

New options for better performance:

# Skip tables entirely (faster)
config = CrawlerRunConfig(
    table_extraction=NoTableExtraction()
)

# Process only specific areas (faster)
config = CrawlerRunConfig(…

Testing Your Migration

Verification Script

Run this to verify your extraction still works:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def verify_extraction():
    url = "your_url_here"

    async with AsyncWebCrawler() as crawler:
        # Test 1: Old…

Deprecation Notes

No Deprecations

- All existing parameters continue to work
- `table_score_threshold` in `CrawlerRunConfig` is still supported
- No breaking changes

Internal Changes (Transparent to Users)

- `LXMLWebScrapingStrategy.is_data_table()` - Moved to `DefaultTableExtraction`
- `LXMLWebScrapingStrategy.extract_table_data()` - Moved to `DefaultTableExtraction`

These methods were internal and…

Benefits of Upgrading

While not required, using the new pattern provides:

- **Better Control** : Filter tables during extraction, not after
- **Performance Options** : Skip extraction when not needed
- **Extensibility**…

Troubleshooting

Issue: Different Number of Tables

**Cause** : Threshold or filtering differences

**Solution** :

# Ensure same threshold
strategy = DefaultTableExtraction(
    table_score_threshold=7,  # Match your old setting
    min_rows=0,               # No filtering (default)
    min_cols=0…

Issue: Import Errors

**Cause** : Using new classes without importing

**Solution** :

# Add imports if using new features
from crawl4ai import (
    DefaultTableExtraction,
    NoTableExtraction,
    TableExtractionStrategy
)

Issue: Custom Strategy Not Working

**Cause** : Incorrect method signature

**Solution** :

class CustomExtractor(TableExtractionStrategy):
    def extract_tables(self, element, **kwargs):  # Correct signature
        # Not: extract_tables(self, html)
        # Not: extract(self, element)…

Getting Help

If you encounter issues:

- Check your `table_score_threshold` matches previous settings
- Verify imports if using new classes
- Enable verbose logging: `DefaultTableExtraction(verbose=True)`
-…

Summary

- ✅  **Full backward compatibility**  - No code changes required
- ✅  **Same results**  - Identical extraction behavior by default
- ✅  **New options**  - Additional control when needed
- ✅  **Better…


WebScrapingStrategy Migration Guide

Summary: This guide explains the deprecation of BeautifulSoup-based WebScrapingStrategy in favor of LXMLWebScrapingStrategy, and confirms backward compatibility with no required changes.

Overview

Crawl4AI has simplified its content scraping architecture. The BeautifulSoup-based `WebScrapingStrategy` has been deprecated in favor of the faster LXML-based implementation. However, **no action is…

What Changed?

- **`WebScrapingStrategy` is now an alias** for `LXMLWebScrapingStrategy`
- **The BeautifulSoup implementation has been removed** (~1000 lines of redundant code)
- **`LXMLWebScrapingStrategy`…

Backward Compatibility

**Your existing code continues to work without any changes:**

# This still works perfectly
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, WebScrapingStrategy

config = CrawlerRunConfig(
    scraping_strategy=WebScrapingStrategy()  # Works as before
)

Migration Options

You have three options:

Option 1: Do Nothing (Recommended)

Your code will continue to work. `WebScrapingStrategy` is permanently aliased to `LXMLWebScrapingStrategy`.

Option 2: Update Imports (Optional)

For clarity, you can update your imports:

# Old (still works)
from crawl4ai import WebScrapingStrategy
strategy = WebScrapingStrategy()

# New (more explicit)
from crawl4ai import LXMLWebScrapingStrategy
strategy = LXMLWebScrapingStrategy()

Option 3: Use Default Configuration

Since `LXMLWebScrapingStrategy` is the default, you can omit the strategy parameter:

# Simplest approach - uses LXMLWebScrapingStrategy by default
config = CrawlerRunConfig()

Type Hints

If you use type hints, both work:

from crawl4ai import WebScrapingStrategy, LXMLWebScrapingStrategy

def process_with_strategy(strategy: WebScrapingStrategy) -> None:
    # Works with both WebScrapingStrategy and…

Subclassing

If you've subclassed `WebScrapingStrategy`, it continues to work:

class MyCustomStrategy(WebScrapingStrategy):
    def __init__(self):
        super().__init__()
        # Your custom code

Performance Benefits

- **10-20x faster** HTML parsing for large documents
- **Lower memory usage**
- **Consistent behavior** across all use cases
- **Simplified maintenance** and bug fixes

Summary

This change simplifies Crawl4AI's internals while maintaining 100% backward compatibility. Your existing code continues to work, and you get better performance automatically.