API reference

These pages are generated from the docstrings and type annotations in the source. Each page documents the modules that define the names; import them from the package named at the top of the page, not from the defining module, since module layout inside a package may change between releases.

Page	Package	Covers
Pipelines	sci_etl_core	AsyncETLPipeline, ETLPipeline, memory ingestion, sync adapters, ShutdownSignal, progress events and RunMetrics
Configuration	sci_etl_core	BaseAppConfig and its sections, load_config, load_config_async
Models and exceptions	sci_etl_core	RawRecord, TokenUsage, the SciEtlError hierarchy, the discovery read-model
Extractors	sci_etl_core.extractors	AsyncExtractor, the arXiv, PubMed, Semantic Scholar, and OpenAlex extractors
Parsers	sci_etl_core.parsers	PDF, LaTeX, HTML, DOCX, and JATS XML parsers, reference trimming
LLM	sci_etl_core.llm	LLM clients, response caching, relevance filters, entity extractors
Embeddings	sci_etl_core.embeddings	embedders, chunking, vector stores, similarity search
Search	sci_etl_core.search	query language, text stores, fusion, hybrid search, discovery graphs
Exporters	sci_etl_core.exporters	CSV, SQL, and Plotly exporters
Processors	sci_etl_core.processors	DataFrame steps, key normalizers, record validators
State	sci_etl_core.state	file and SQLite state managers
Utilities	defining module	HTTP clients, rate limiters, including per-host limits, logging

For how the pieces fit together, see Architecture.