Skip to content

sci-etl-core

A reusable, domain-agnostic Python library for scientific text mining and ETL. sci-etl-core gives you composable building blocks — extractors, parsers, LLM clients, embedding memory, processors, exporters, and state managers — behind abstract base classes, so you can assemble a pipeline for any corpus without inheriting constants tied to a specific field of science.

The library is async-first. Every component is an async implementation, orchestrated by AsyncETLPipeline. For scripts that don't want to manage an event loop, ETLPipeline is a single blocking entrypoint that runs the same pipeline on a background loop.

Prefer configuration to code? The sci-etl command-line tool runs these pipelines from a single YAML file.

Features

  • Pluggable async interfaces for every stage: AsyncExtractor, Parser, AsyncLLMClient, AsyncRelevanceFilter, AsyncEntityExtractor, AsyncExporter, AsyncStateManager, plus AsyncEmbedder, TextChunker, and AsyncEmbeddingStore for semantic memory. Existing blocking implementations plug in through Sync*Adapter wrappers.
  • Built-in orchestration — AsyncETLPipeline processes records with bounded concurrency; ETLPipeline wraps it for blocking code.
  • Explicit failure signaling — a transport fault or malformed listing aborts the run with PipelineAborted (carrying the partial count) instead of looking like the end of the data. A single failing record is logged and left for the next run, and records that keep failing while nothing is processed stop the run instead of burning through the rest of the listing.
  • Resumable, crash-safe state — plain-file or SQLite backends record processed ids and the listing offset; CSV and metadata writes use atomic renames. Newest-first listings pick up new papers without rescanning.
  • Graceful shutdown — Ctrl+C or SIGTERM lets in-flight records finish, flushes state, and raises PipelineInterrupted.
  • Polite retries and rate limits — every bundled extractor and the OpenAI-compatible chat and embedding clients wait as long as a throttled response's Retry-After header asks, up to a configurable cap, and take rate limiters that can be shared and set per host.
  • Progress events, metrics, and token usage — typed per-record events, run metrics with counts, durations, and failures, and the tokens each run used.
  • LLM response caching — an in-memory or SQLite cache answers repeated prompts without another API call.
  • Semantic memory (optional) — chunk and embed full texts into an in-memory or SQLite vector store, search for similar articles, or gate relevance by embedding similarity instead of an LLM call.
  • Local search and discovery — Boolean queries over a SQLite FTS5 text index that needs only the standard library, hybrid search that fuses BM25 with embedding similarity, metadata facets, and graphs of related papers.
  • Dependency injection everywhere — HTTP clients, parsers, models, prompts, and destinations are constructor arguments.
  • Concrete implementations included — arXiv, PubMed, Semantic Scholar, and OpenAlex extractors; OpenAI-compatible chat and embedding clients; local sentence-transformers embedder; PDF / LaTeX / HTML / DOCX / JATS XML parsers; CSV upsert, SQL table, and 3D Plotly exporters; dataframe processors and record validators.
  • Typed configuration from YAML + .env with Pydantic validation and SecretStr API keys.
  • Offline test suite — pytest with mocks, Hypothesis property tests, and ABC conformance tests.
  • PEP 561 typed (py.typed) for downstream type checking.

Getting help