R10-040

crawl4ai v0.9.2 Feature Equivalence

Every row is enforced by tests/test_feature_equivalence.py. E = implemented equivalently, N = deliberately rejected, M = no project equivalent needed.

port 11235health / ready / metrics / playgroundweekly buildsamd64 + arm64

Equivalent (E)

Implemented equivalently or superseded

Rejected (N)

Deliberately rejected — architectural rationale

No equivalent (M)

crawl4ai feature not needed

Equivalent features

E1–E22 — implemented equivalently or superseded by a project-native mechanism

BrowserProfiler — 3-tier pool (permanent / hot / cold)Equivalent

Project: src/crawler.py:CrawlerPool + src/browser_pool.py:BrowserProfiler (R10-037)

3-tier pool opt-in via CRAWLER_POOL_TIER=2|3; per-host concurrency via src/dispatcher.py.

CrawlResult / MarkdownGenerationResultEquivalent

Project: src/llm_extractor.py:extract_page_content → standard-page-record-schema

PageContent dataclass; raw_markdown / fit_markdown map to raw_markdown / llm_markdown.

prompt_registry — 50+ prebuilt promptsEquivalent

Project: src/prompts.py:EXTRACT_PAGE_PROMPT (+ TABLE/TOC prompts)

Single prompt for single extractor; crawl4ai has 50+ prebuilt prompts.

markdown_generation_strategy / page.pdf()Equivalent

Project: src/pdf_builder.py:_assemble_book_html — Playwright page.pdf() + Pygments

Server-side syntax highlighting embedded in stylesheet; HTML shares page.evaluate() path.

AdaptiveRateLimiter + MemoryAdaptiveDispatcherEquivalent

Project: src/llm_extractor.py:AdaptiveRateLimiter + src/dispatcher.py:MemoryAdaptiveDispatcher

Token bucket with multiplicative decrease on 429; per-host concurrency via CRAWL_PER_HOST_CONCURRENCY.

AsyncDatabaseCache (SQLite-backed)Equivalent

Project: File-based resume cache — raw_pages.json + extracted_pages.jsonl

.jsonl wins on resume; both files are NOT redundant.

BFSDeepCrawlStrategy / BestFirstCrawlingStrategyEquivalent

Project: src/sitemap.py:discover_urls_via_sitemap (BFS) + discover_urls_bestfirst (BestFirst)

BFS is default; BestFirst opt-in via CRAWL_STRATEGY=bestfirst with keyword-overlap scoring.

content_filter_strategy (LXMLWebScrapingStrategy, PruningContentFilter)Equivalent

Project: src/llm_extractor.py:pre_clean_html — BeautifulSoup + lxml selector walker

Strips <script>/<style>/<meta>/<noscript> before LLM call; equivalent to crawl4ai's pruning/BM25 filters.

DFSDeepCrawlStrategy / BFSDeepCrawlStrategyEquivalent

Project: src/sitemap.py:_discover_bfs_async (BFS via asyncio queue)

BFS is the project's default; DFS is rejected (N9).

E10

extract_links / CrawlerRunConfig link fieldsEquivalent

Project: src/crawler.py:categorize_links_by_host:298 (CATEGORIZE_LINKS=true)

Splits links into internal/external; opt-in only.

E11

exclude_selectors (CSS list)Equivalent

Project: CrawlerConfig.exclude_selectors → src/crawler.py:page.route:751 + JS hook

Asset blocking via route interception is mandatory for text-only crawls.

E12

screenshot / screenshot_wait_forEquivalent

Project: CrawlerConfig.screenshot → src/crawler.py:_take_failure_screenshot:1424

Base64 screenshot on CrawlResult.screenshot equivalent.

E13

CacheMode (ENABLED/BYPASS/…)Equivalent

Project: CrawlerConfig.cache_mode → src/pipeline.py:_cache_path:113 / extracted_pages.jsonl

ON=use cache, OFF=bypass, READ_ONLY=no-write.

E14

exclude_external_imagesEquivalent

Project: CrawlerConfig.exclude_external_images → src/crawler.py:_filter_image_alts:651

Blocks <img> src loads to external hosts.

E15

magic_mode / js_codeEquivalent

Project: CrawlerConfig.magic_mode → src/crawler.py:_MAGIC_INIT_JS:786 init-script hoist

--disable-blink-features=AutomationControlled; crawl4ai uses stealth.js.

E16

JsonCssExtractionStrategyEquivalent

Project: CrawlerConfig.css_schema → src/css_extractor.py:load_css_schema:89 + extract_page_content

CSS selectors as an alternative to pure LLM extraction.

E17

crwl CLIEquivalent

Project: scripts/crwl.py:main:168 → crawl_and_build.py (thin flag mapper)

Same flag surface as crawl4ai's CLI.

E18

CrawlerRunConfig URL validationEquivalent

Project: src/config.py:validate_url() — rejects private IPs, unexpected schemes

CRAWL_ALLOW_PRIVATE_HOSTS opt-in.

E19

BM25ContentFilterEquivalent

Project: CrawlerConfig.bm25_content_filter → src/llm_extractor.py:_apply_bm25:2241

User-query-driven BM25 relevance scoring before LLM call.

E20

BrowserConfig.proxy_config / CrawlerRunConfig.proxy_configEquivalent

Project: CrawlerConfig.proxy_url (CRAWL_PROXY) → src/crawler.py:890

Single outbound proxy for Chromium; upstream also supports rotation (N14).

E21

BrowserConfig.user_agent / CrawlerRunConfig.user_agentEquivalent

Project: CrawlerConfig.user_agent (CRAWL_USER_AGENT) → src/pipeline.py:_resolve_user_agent:4011

Custom UA for stealth posture; default = crawl4ai-docker/X.Y.

E22

html2text / CustomHTML2Text / DefaultMarkdownGeneratorEquivalent

Project: src/llm_extractor.py:_html_to_markdown_node (lines 1252+)

Bespoke iterative HTML→MD converter; crawl4ai's html2text wraps the same algorithm.

Rejected features

N1–N16 — deliberately rejected with architectural rationale

FastAPI Docker server (--serve)Rejected

Project: Next.js UI + NDJSON stream bridge (no FastAPI/uvicorn)

The Python subprocess is orchestrated by crawl_and_build.py.

Cloud / Hub (CrawlerHub, distributed crawl)Rejected

Project: Single-site, single-node reference-book builder

No Hub account or vendor lock-in.

AsyncDatabaseCache (SQLite-backed) / AsyncUrlSeederRejected

Project: File-based resume cache (raw_pages.json / extracted_pages.jsonl)

File cache is load-bearing for resumability and portability.

Torch / embedding extras (pip install crawl4ai[torch])Rejected

Project: MiniMax/Mistral/OpenAI-compatible LLMs only

Local embedding models (BERT, spaCy, NLTK) not needed; torch ~700 MB avoided.

AsyncLogger / LogEnabledResponse (cloud-only)Rejected

Project: src/log_config.py — stdlib logging + NDJSON stream to UI

Besoke logging; not dependent on Cloud Hub.

Legacy compatibility (_v0_4_compat migration paths)Rejected

Project: Schema 1.0.0 — no v0.3/v0.4 data to migrate

Compat shim would be dead code.

security_hardening release notesRejected

Project: AGENTS.md §Security — bespoke SSRF/XXE/iframe-bypass hardening

Project hardening is stricter and documented separately.

PDF_CHUNK_CHAPTERS / PDF_PARALLEL_RENDERERSRejected

Project: Pipeline-internal performance knobs (CrawlerConfig src/config.py:647-657)

Build-phase parallel-renderer contract; not an SDK feature.

Deep-crawl scorers (CompositeScorer, DomainAuthorityScorer, …)Rejected

Project: BestFirst with simple keyword-relevance scorer (crawl_query + BM25)

Composite scorer adds complexity without improving reference-book crawl quality.

N10

XPath / Lxml / Element JSON extraction strategiesRejected

Project: CSS selectors (E16) + LLM extraction

XPath not needed; LLM fallback handles anything CSS misses.

N11

Chunking strategies (Regex, FixedLength, NlpSentence, SlidingWindow, …)Rejected

Project: src/rag_exporter.py — section-based + window overlap + sentence splitting

Project chunking is tuned for TOC hierarchy + changelog diffing.

N12

Additional content filters (RelevantContentFilter, SEOFilter, LLMContentFilter, CosineStrategy)Rejected

Project: BM25ContentFilter (E19) + LLM extractor

BM25 + LLM covers the same use cases; CosineStrategy requires local embeddings (N4).

N13

NoExtractionStrategyRejected

Project: N/A — extraction is mandatory for a reference-book factory

A crawl-without-extracting strategy has no use case in this pipeline.

N14

Proxy rotation (RoundRobinProxyStrategy)Rejected

Project: Single outbound proxy (CRAWL_PROXY, E20)

Proxy rotation is for large-scale multi-host crawling; project crawls one docs site at a time.

N15

LLMTableExtractionRejected

Project: src/specialized.py — project-native table extractors (P2-022..P2-025)

Tuned for reference-book use case: table caption extraction, header normalization.

N16

SemanticChunkingRejected

Project: Section-based chunking (rag_exporter.py) — deterministic, fast, TOC-aligned

Requires local embedding models (N4); produces chapter breaks that don't match the book's logical structure.

No project equivalent

M1–M14 — crawl4ai has the feature but the project has no counterpart

LinkPreview / LinkPreviewConfigNo equivalent

OG/Twitter meta tags are irrelevant for reference books.

SSLCertificateNo equivalent

Cert validation is handled by the browser engine.

Crawl4aiDockerClient (cloud module)No equivalent

Pipeline runs locally; no distributed crawl architecture.

AsyncUrlSeederNo equivalent

Uses synchronous sitemap parsing + BFS/BestFirst nav discovery.

OnlineUAGeneratorNo equivalent

Static configurable UA (CRAWL_USER_AGENT, E21) is sufficient.

HeadPeekr (antibibot_detector)No equivalent

magic_mode (E15) handles anti-detection; HEAD fingerprinting is premature.

extract_blocks / extract_blocks_batchNo equivalent

LLM extractor (E2) already extracts structured page content.

get_text_embeddingsNo equivalent

No embedding index; RAG export is chapter-based, not embedding-based.

Model loaders (BERT, HF, spacy, nltk)No equivalent

Remote LLM APIs only; local models would bloat the venv.

M10

Text classifiers (load_text_classifier, load_text_multilabel_classifier)No equivalent

BM25-based filtering (E19); zero-shot adds latency for no gain.

M11

advanced_split / merge_chunksNo equivalent

RAG chunking (rag_exporter.py) handles assembly at chapter level.

M12

UserAgentGenerator (local variant)No equivalent

Same rationale as M5 — static UA is sufficient.

M13

html2text standalone functionNo equivalent

Project has its own _html_to_markdown_node (E22).

M14

download_all_modelsNo equivalent

No local models used; would download ~1 GB of unused artifacts.

Final assessment

Should the pipeline be built around crawl4ai?

NoThe project should continue using Playwright + httpx + BS4 directly.

The project is a reference-book factorywith fundamentally different goals than crawl4ai's generic extraction platform:

Different output contract. crawl4ai emits CrawlResult objects. The project emits structured reference books (PDF, HTML, Markdown) with TOC hierarchy, chapter assembly, changelog diffing, and RAG-ready chunking.
Different extraction model. crawl4ai's strategies are designed for single-page structured extraction. The project's llm_extractor.py extracts page-level semantics (section, subsection, category) and feeds them into a book assembly pipeline.
Feature coverage is already equivalent. 22 crawl4ai-equivalent features (E1–E22) are implemented natively.
Switching cost is high. crawl4ai is not a dependency — it is not in requirements.txt or pyproject.toml. Migrating would require rewriting CrawlerPool, llm_extractor.py, pdf_builder.py, rag_exporter.py, mhtml.py, and adaptive.py.
The project is already well-audited. 98 implementation gaps, 90 performance findings, and 86 future improvements resolved. 210+ tests pass.

Read the full equivalence document →