Models and exceptions¶
Records, pipeline metadata, token usage, the exception hierarchy, and the discovery read-model. All of them are importable from sci_etl_core.
dataclass
¶
RawRecord(
record_id: str,
title: str,
abstract: str,
source_url: str | None = None,
metadata: dict[str, Any] = dict(),
)
One article as a listing describes it, before its full text is fetched.
record_id is the id state managers track; the pipeline skips a record
whose id is blank. metadata holds whatever else the extractor knows, as
JSON-friendly values: :class:~sci_etl_core.extractors.arxiv_async.AsyncArxivExtractor
fills categories, authors, published, and year.
:class:~sci_etl_core.search.index_async.AsyncSearchIndexer copies it into
the text index, where the string and integer values under a store's facet
keys become filterable tags. The chunks
:class:~sci_etl_core.embeddings.ingest_async.AsyncChunkIngestor stores do
not carry it.
dataclass
¶
Tokens an API reported across a client's completed requests.
dataclass
¶
PipelineMetadata(
last_run_at: str | None = None,
last_start_index: int = 0,
head_ids: list[str] = list(),
head_offset: int = 0,
tail_ids: list[str] = list(),
)
What a state manager keeps about the listing between runs.
last_start_index is the listing offset the next run resumes from, and
last_run_at is the time :meth:touch last stamped, or None.
head_ids, head_offset, and tail_ids serve runs with
newest_first=True, and describe the same listing snapshot as
last_start_index. head_ids are ids seen near the top of the
listing, in listing order, the first at position head_offset; a run
finds them again to learn how far new submissions have pushed the backlog
down. tail_ids are the ids just before last_start_index, which a
run checks for before skipping to the backlog. Other runs leave all three
unchanged.
Bases: SciEtlError
Raised when a source extractor fails to retrieve or parse data.
Bases: ExtractionError
Raised when a remote source is unreachable or answers with a server error.
Distinguishes a transport-level failure from a well-formed response that simply contains no further results.
Bases: ExtractionError
Raised when a listing payload cannot be interpreted as a valid feed.
Distinguishes an unparseable payload from a valid but empty listing, which legitimately signals the end of available data.
Bases: SciEtlError
Raised when a document parser fails to extract content.
Bases: SciEtlError
Raised when an LLM client call fails irrecoverably.
Bases: SciEtlError
Raised when an LLM response cache cannot be read or written.
Kept distinct from :class:LLMError so a cache fault is never mistaken for
a failed completion; a caching client logs it and calls the LLM instead.
Bases: SciEtlError
Raised when an embedding backend fails to vectorize text.
Kept distinct from :class:LLMError so a semantic-similarity failure is
never silently confused with a chat-completion failure.
Bases: SciEtlError
Raised when the vector memory cannot be written to or read from.
Kept distinct from :class:EmbeddingError so a storage fault is never
mistaken for a failure to produce the embedding itself.
Bases: SciEtlError
Raised when a text search operation fails.
Bases: SearchError
Raised when a Boolean query cannot be parsed or cannot be run as asked.
Kept distinct from :class:SearchStoreError so a typo in a query is never
mistaken for a corrupt index. position is the character offset into the
query at fault and token is the text found there, so a UI can underline
it. position is None and token is empty when the error concerns
the request rather than a character of the query.
Bases: SearchError
Raised when the text index cannot be read or written.
Kept distinct from :class:SearchQueryError so a storage fault is never
mistaken for a malformed query.
Bases: SciEtlError
Raised when configuration loading or validation fails.
Bases: SciEtlError
Raised when a pipeline run terminates early because of an upstream fault.
Carries the number of records successfully processed before the abort so callers can distinguish partial progress from a run that never started.
Bases: PipelineAborted
Raised when a pipeline run stops early because a shutdown was requested.
A subclass of :class:PipelineAborted, so code that already handles an
abort keeps working. The records in flight when the request arrived were
finished, the rest of their page is left for the next run, and state was
flushed before this was raised.
The read-model a presentation layer renders: plain, immutable dataclasses.
Importing this module loads no store, no event-loop machinery, and no optional dependency, so a thin UI can depend on it alone. The graph, hit, and chip types it refers to are annotations only.
dataclass
¶
Matching-document counts for the values of one metadata key.
counts holds (value, count) pairs sorted by count descending, then
value, as AsyncTextSearchStore.facet_counts returns them: each count
ignores filters on this key, so it tells what selecting the value would give.
dataclass
¶
DiscoveryResult(
query_text: str,
chips: tuple[QueryChip, ...],
hits: tuple[FusedHit, ...],
graph: DiscoveryGraph | None,
facets: tuple[Facet, ...],
total_matched: int,
elapsed_ms: float,
degraded: tuple[str, ...] = (),
skipped: tuple[str, ...] = (),
)
Everything a UI needs to render one search, in render-agnostic form.
chips describe the parsed query, and hits are the fused results,
best first. graph is the discovery graph around a selected record, or
None; whether its communities converged is on the graph itself.
total_matched counts the records satisfying the query and filters, not
only the hits shown. degraded names retrieval legs that failed, such as
("semantic",) when the embedding service was unreachable, and
skipped names legs that had nothing to run, such as a semantic leg for a
query of prefix terms only. A UI must tell the user about both rather than
silently showing weaker results.