Models and exceptions¶
Records, pipeline metadata, token usage, the exception hierarchy, and the discovery read-model. All of them are importable from sci_etl_core.
dataclass
¶
RawRecord(
*,
record_id: str,
title: str,
abstract: str,
source_url: str | None = None,
metadata: dict[str, Any] = dict(),
)
One article as a listing describes it, before its full text is fetched.
record_id is the id state managers track; the pipeline skips a record
whose id is blank. metadata holds whatever else the extractor knows, as
JSON-friendly values: :class:~sci_etl_core.extractors.arxiv_async.AsyncArxivExtractor
fills categories, authors, published, and year.
:class:~sci_etl_core.search.index_async.AsyncSearchIndexer copies it into
the text index, where the string and integer values under a store's facet
keys become filterable tags. The chunks
:class:~sci_etl_core.embeddings.ingest_async.AsyncChunkIngestor stores do
not carry it.
dataclass
¶
Tokens an API reported across a client's completed requests.
dataclass
¶
ListingPage(
*,
records: tuple[RawRecord, ...],
entries: int,
next_cursor: str | None,
truncated: bool = False,
)
One page of a listing, parsed.
records are the page's entries that could be read, in listing order.
entries counts every entry on the page, including records the pipeline
will skip as processed and entries without an id. next_cursor is
None when the listing has no further page. truncated is True
when the source stopped because of its own result cap rather than because
the listing ended; next_cursor is then None as well.
dataclass
¶
PipelineMetadata(
*,
last_run_at: str | None = None,
cursor: str | None = None,
truncated: bool = False,
head_ids: list[str] = list(),
head_offset: int = 0,
tail_ids: list[str] = list(),
)
What a state manager keeps about the listing between runs.
cursor is the listing cursor the next run resumes from, or None
to start from the first page, and last_run_at is the time
:meth:touch last stamped, or None. truncated is True while
the most recent run that reached the end of its listing stopped at the
source's result cap.
head_ids, head_offset, and tail_ids serve runs with
newest_first=True, whose cursors are decimal offsets, and describe the
same listing snapshot as cursor. head_ids are ids seen near the
top of the listing, in listing order, the first at position
head_offset; a run finds them again to learn how far new submissions
have pushed the backlog down. tail_ids are the ids just before the
offset in cursor, which a run checks for before skipping to the
backlog. Other runs leave all three unchanged.
Bases: SciEtlError
Raised when a source extractor fails to retrieve or parse data.
Bases: ExtractionError
Raised when a remote source is unreachable or answers with a server error.
Distinguishes a transport-level failure from a well-formed response that simply contains no further results.
Bases: ExtractionError
Raised when a listing payload cannot be interpreted as a valid feed.
Distinguishes an unparseable payload from a valid but empty listing, which legitimately signals the end of available data.
Bases: ExtractionError
Raised when a source no longer accepts a listing cursor it handed out earlier.
:class:~sci_etl_core.pipeline_async.AsyncETLPipeline restarts the
listing from its first page once per run when it sees this error.
Bases: SciEtlError
Raised when a document parser fails to extract content.
Bases: SciEtlError
Raised when an LLM client call fails irrecoverably.
Bases: SciEtlError
Raised when an LLM response cache cannot be read or written.
Kept distinct from :class:LLMError so a cache fault is never mistaken for
a failed completion; a caching client logs it and calls the LLM instead.
Bases: SciEtlError
Raised when an embedding backend fails to vectorize text.
Kept distinct from :class:LLMError so a semantic-similarity failure is
never silently confused with a chat-completion failure.
Bases: SciEtlError
Raised when the vector memory cannot be written to or read from.
Kept distinct from :class:EmbeddingError so a storage fault is never
mistaken for a failure to produce the embedding itself.
Bases: SciEtlError
Raised when a text search operation fails.
Bases: SearchError
Raised when a Boolean query cannot be parsed or cannot be run as asked.
Kept distinct from :class:SearchStoreError so a typo in a query is never
mistaken for a corrupt index. position is the character offset into the
query at fault and token is the text found there, so a UI can underline
it. position is None and token is empty when the error concerns
the request rather than a character of the query.
Bases: SearchError
Raised when the text index cannot be read or written.
Kept distinct from :class:SearchQueryError so a storage fault is never
mistaken for a malformed query.
Bases: SciEtlError
Raised when a state file or database cannot be used as run state.
A bundled state manager raises it for a file written by a newer sci-etl-core, whose schema version it does not know.
Bases: SciEtlError
Raised when an exporter cannot read or write its destination.
Raised from open, it aborts the run before any listing request; from
write, it fails the record, which is retried on the next run.
Bases: SciEtlError
Raised when a claim operation fails.
Bases: ClaimError
Raised when a claim store or rejection store cannot be read or written.
It is not a memory fault: raised from an exporter's write, it fails
the record, which is retried on the next run.
Bases: SciEtlError
Raised when configuration loading or validation fails.
Bases: SciEtlError
Raised when a pipeline run terminates early because of an upstream fault.
Carries the number of records successfully processed before the abort so callers can distinguish partial progress from a run that never started.
Bases: PipelineAborted
Raised when a pipeline run stops early because a shutdown was requested.
A subclass of :class:PipelineAborted, so code that already handles an
abort keeps working. The records in flight when the request arrived were
finished, the rest of their page is left for the next run, and state was
flushed before this was raised.
The read-model a presentation layer renders: plain, immutable dataclasses.
Importing this module loads no store, no event-loop machinery, and no optional dependency, so a thin UI can depend on it alone. The graph, hit, and chip types it refers to are annotations only.
dataclass
¶
Matching-document counts for the values of one metadata key.
counts holds (value, count) pairs sorted by count descending, then
value, as AsyncTextSearchStore.facet_counts returns them: each count
ignores filters on this key, so it tells what selecting the value would give.
dataclass
¶
DiscoveryResult(
query_text: str,
chips: tuple[QueryChip, ...],
hits: tuple[FusedHit, ...],
graph: DiscoveryGraph | None,
facets: tuple[Facet, ...],
total_matched: int,
elapsed_ms: float,
degraded: tuple[str, ...] = (),
skipped: tuple[str, ...] = (),
)
Everything a UI needs to render one search, in render-agnostic form.
chips describe the parsed query, and hits are the fused results,
best first. graph is the discovery graph around a selected record, or
None; whether its communities converged is on the graph itself.
total_matched counts the records satisfying the query and filters, not
only the hits shown. degraded names retrieval legs that failed, such as
("semantic",) when the embedding service was unreachable, and
skipped names legs that had nothing to run, such as a semantic leg for a
query of prefix terms only. A UI must tell the user about both rather than
silently showing weaker results.