Search¶
Every name on this page is importable from sci_etl_core.search.
module-attribute
¶
The text fields of a document, in index column order; a term or phrase can be scoped to any of them.
dataclass
¶
One tokenizer-normalized word, matched exactly or, with prefix, as a prefix.
fields limits the match to those fields; empty means every field.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
dataclass
¶
Tokenizer-normalized words that must appear adjacent and in this order.
fields limits the match to those fields; empty means every field.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
dataclass
¶
Near(
operands: tuple[Term | Phrase, ...],
distance: int = NEAR_DISTANCE,
fields: tuple[str, ...] = (),
)
Terms and phrases that must all occur in one field, close to each other, in any order.
With the occurrences chosen so that the one starting last starts at token
p, every other operand must end at most distance tokens before
p: NEAR(a b, 2) matches a x x b and b x a but not
a x x x b. The operands carry no field scope of their own; fields
limits the whole group, and empty means any one field.
Raises:
| Type | Description |
|---|---|
ValueError
|
There are fewer than two operands, an operand has fields
of its own, |
dataclass
¶
dataclass
¶
dataclass
¶
QueryChip(
text: str,
fields: tuple[str, ...] = (),
operator: str = "",
negated: bool = False,
prefix: bool = False,
phrase: bool = False,
depth: int = 0,
near: int | None = None,
)
One word or phrase of a query, labelled for display.
text is the term, or the phrase's words joined by single spaces.
fields is empty when every field is searched. operator is "AND"
or "OR", the operator of the innermost group holding the chip, and is
empty when the whole query is this one word or phrase. negated is true
when the chip must not match, and depth counts the groups around it, so
a UI can bracket a (b OR c) without re-implementing the grammar.
near is the distance of the NEAR group the chip stands for, whose
text is then its terms and phrases, phrases in double quotes, joined by
single spaces; it is None for any other chip.
Return the canonical form of node, which normalize leaves unchanged.
The rules apply bottom-up until nothing changes:
Not(Not(x))becomesx.- Nested
AndandOrflatten, and a single-operandAndorOrbecomes its operand. - Inside an
And, two or moreNotoperands merge by De Morgan's law into one, placed where the first stood:And((a, Not(b), Not(c)))becomesAnd((a, Not(Or((b, c))))). - Operand order is preserved, never sorted.
The result matches exactly the documents node matches.
Return the words and phrases of node as chips, in the order they were written.
node is normalized first, so NOT NOT a is described as a.
Return the words an embedder should see for node: its meaning, not its syntax.
node is normalized first. The text of every term and phrase that is not
negated is joined with single spaces, in the order written. Operators,
field scopes, and every negated subtree are dropped, so quasar -dwarf
becomes quasar rather than pulling results towards dwarfs.
Prefix terms are left out entirely: photometr is a matching feature, not
a word an embedder understands, so photometr* dwarf gives dwarf and
photometr* gives the empty string.
OR loses its meaning here. A single vector cannot represent a
disjunction, so quasar OR blazar gives quasar blazar, the same text
as quasar AND blazar.
module-attribute
¶
How deeply :func:parse_query lets parentheses nest, which bounds the parser's recursion.
Parse a Boolean query into a normalized AST.
Syntax, with NOT binding tightest, then AND, then OR:
galaxyis a term, folded by :class:Unicode61Tokenizer. A word the tokenizer splits, such asH-alpha, is a phrase of its parts."dwarf galaxy"is a phrase; a quoted single word is a term.photometr*is a prefix term. The*must directly follow one word.title:quasarandtitle,abstract:"dwarf galaxy"scope a term or a phrase to fields, named case-insensitively from :data:FIELDS.NEAR(dwarf "dark matter" halo*, 5)matches when its terms and phrases all occur in one field within 5 tokens of each other, in any order (:class:~sci_etl_core.search.query.Near). The distance defaults to 10, as inNEAR(dwarf halo). A group holding one quoted phrase, such asNEAR("dwarf halo", 3), searches for its words near each other. The group can be scoped as a whole, as intitle:NEAR(dwarf halo), but nothing inside it can, and it holds no operators such as-orOR.a AND b,a && b, anda bare conjunctions;a OR banda || bare disjunctions;NOT aand-aare negations.- Parentheses group, nesting at most :data:
MAX_GROUP_DEPTHdeep.
Operators are upper case; and is an ordinary word. A term or phrase
without a scope gets default_fields, and empty means every field.
Parsing is pure: it runs nothing and touches no I/O.
Raises:
| Type | Description |
|---|---|
SearchQueryError
|
The query is empty or malformed. |
Parse a query for a ranked search, which needs a term that is not negated.
This is :func:parse_query followed by
:func:~sci_etl_core.search.compile_fts5.require_rankable, with the
rankability error located in text.
Raises:
| Type | Description |
|---|---|
SearchQueryError
|
The query is malformed, as for :func: |
Parse a query for a semantic search, returning the AST and the text to embed.
The text is :func:~sci_etl_core.search.query.semantic_text of the AST.
Raises:
| Type | Description |
|---|---|
SearchQueryError
|
The query is malformed or cannot be ranked, as for
:func: |
module-attribute
¶
SURROGATE_CODE_POINTS = range(55296, 57344)
Code points :class:Unicode61Tokenizer treats as separators; FTS5 cannot receive them, so parity is unchecked.
module-attribute
¶
The UTF-8 length at which FTS5 truncates an indexed term; :class:Unicode61Tokenizer never truncates.
dataclass
¶
Reproduce FTS5's unicode61 remove_diacritics 2 tokenizer in pure Python.
A token is a maximal run of letters, numbers, and private-use characters as
Unicode 6.1 classifies them; every other character separates tokens, so
H-alpha gives h and alpha. Each character is case-folded one to
one, never with :meth:str.casefold, and loses its diacritic: Müller
gives muller while Straße stays straße, exactly as FTS5 indexes
them. A run whose characters all fold away, such as a lone combining accent,
gives no token.
The character tables come from FTS5 itself and are checked against it for every code point. Two differences are known:
- Surrogate code points (:data:
SURROGATE_CODE_POINTS) are separators here. SQLite cannot receive them, so parity cannot be checked. - FTS5 truncates an indexed term to :data:
FTS5_MAX_TOKEN_BYTESbytes of UTF-8. This tokenizer never truncates.
dataclass
¶
TokenizedDocument(
title: tuple[str, ...] = (),
abstract: tuple[str, ...] = (),
body: tuple[str, ...] = (),
)
A document's fields as token texts, tokenized once to be matched many times.
classmethod
¶
from_document(
document: SearchDocument,
tokenizer: Tokenizer | None = None,
) -> TokenizedDocument
Tokenize every field of document, by default with :class:Unicode61Tokenizer.
matches(
node: Node,
document: SearchDocument | TokenizedDocument,
*,
tokenizer: Tokenizer | None = None,
) -> bool
Report whether document satisfies the Boolean query node.
The semantics are those FTS5 gives the expression that
:func:~sci_etl_core.search.compile_fts5.to_match_expression builds:
- The text of a :class:
Termand the words of a :class:~sci_etl_core.search.query.Phraseare tokenized again, as FTS5 tokenizes a quoted string. Text that splits into several tokens matches as a phrase, and text with no token matches nothing. - A phrase matches consecutive tokens within one field. With
prefix, a term's last token matches every token that starts with it. - A leaf without
fieldssearches every field. - A :class:
~sci_etl_core.search.query.Neargroup matches as :func:trim_neardescribes. An operand whose text has no token is dropped from the group, and a group left with one operand matches as that operand.
Any query can be evaluated, including a pure negation such as NOT b.
node is interpreted exactly as given, without
:func:~sci_etl_core.search.query.normalize, so evaluation stays an
independent check on normalization.
tokenizer defaults to :class:Unicode61Tokenizer. It splits the query's
words, and document unless it is already a :class:TokenizedDocument,
which must then come from the same tokenizer.
near_occurrences(
near: Near,
document: TokenizedDocument,
*,
tokenizer: Tokenizer | None = None,
) -> list[Hits]
Return where each operand of near occurs as part of a match, one mapping per operand.
The mappings are those of :func:occurrences, trimmed by :func:trim_near
to the occurrences FTS5 counts, and are all empty when the group does not
match. An operand dropped for having no token gets an empty mapping.
Apply NEAR to each operand's occurrences, given how many tokens each operand spans.
Operands spanning no token are dropped, as FTS5 drops them, and one left
alone keeps all its occurrences. Otherwise the result is :func:trim_near.
Keep the occurrences that FTS5 counts as part of a NEAR match, following its algorithm step for step.
Occurrences in different fields are never near each other. A set of
occurrences, one per operand, matches when every operand ends at most
distance tokens before the start of the occurrence that starts last.
FTS5 walks every operand's start positions together, and each time the
current positions match it keeps them and advances the operand whose next
position is smallest. The occurrences kept are exactly those it walks past
in a match, which is what it scores and highlights, and every mapping is
empty when no set matches.
occurrences(
leaf: Term | Phrase,
document: TokenizedDocument,
*,
tokenizer: Tokenizer | None = None,
) -> dict[str, list[tuple[int, int]]]
Return where leaf occurs in document, as [start, stop) token ranges per field.
Only the fields leaf searches are considered, with the same semantics as
:func:matches, and a field without an occurrence is omitted. Overlapping
occurrences are all reported: the phrase a a occurs twice in a a a.
module-attribute
¶
The SQL condition each maximal rankable subtree becomes in :func:to_filter_expression.
Its ? takes that subtree's MATCH expression, and it reads d.doc_id of the documents table.
Report whether node compiles to a single FTS5 MATCH expression, and so can be ranked.
FTS5 has no unary NOT; its NOT subtracts one match set from another.
After :func:~sci_etl_core.search.query.normalize:
- a :class:
Term, :class:Phrase, or :class:Nearis rankable; - an :class:
Oris rankable when every operand is; - an :class:
Andis rankable when every operand that is not a :class:Notis rankable, and so is the operand of its :class:Not, if it has one; - a :class:
Noton its own is not rankable.
Normalization leaves every And with at least one operand that is not a
Not, and at most one that is.
require_rankable(node: Node) -> None
Raise unless node can be ranked, as :func:is_rankable defines it.
Raises:
| Type | Description |
|---|---|
SearchQueryError
|
|
Compile the normalized node into one FTS5 MATCH expression.
Every word reaches FTS5 inside a double-quoted string literal with each
" doubled, so no query text can act as an FTS5 operator. Bind the result
as a ? parameter; never format it into SQL. SQLite cannot take a NUL or
a surrogate inside a literal, so each becomes a space; the tokenizer treats
both as separators, so what matches is unchanged.
Every AND, OR, and nested NOT is parenthesized, so FTS5 operator
precedence never decides the meaning.
Raises:
| Type | Description |
|---|---|
SearchQueryError
|
|
Compile the normalized node into a SQL boolean expression, with its parameters.
Any query compiles, including a pure negation, because SQL's AND,
OR, and NOT are total where FTS5's are not. Each maximal rankable
subtree becomes one :data:FILTER_LEAF, whose ? takes that subtree's
:func:to_match_expression; the parameters are returned in the order their
placeholders appear. The expression reads d.doc_id, so it runs as the
WHERE clause of a query over documents d.
The SQL text comes only from the fixed grammar. Every word reaches SQLite as a bound parameter, inside a quoted FTS5 literal.
For example, NOT b OR c compiles to
((NOT <leaf>) OR <leaf>) with parameters ('"b"', '"c"').
module-attribute
¶
Marks where a highlighted span starts in a raw snippet; :func:sanitize_text keeps it out of stored text.
module-attribute
¶
Marks where a highlighted span ends in a raw snippet; :func:sanitize_text keeps it out of stored text.
module-attribute
¶
Stands for the text a snippet leaves out before or after it.
module-attribute
¶
SearchFilter = MetadataFilter | RangeFilter
A filter a text search store accepts beside a query.
dataclass
¶
Keep the records tagged with any of values under key, or with negated, drop them.
A record's tags under key are the values :func:tag_rows derives from
its metadata, so 2024 stored as an integer matches the value "2024".
values may be any collection of strings; it is stored as a
:class:frozenset.
Raises:
| Type | Description |
|---|---|
TypeError
|
|
ValueError
|
|
dataclass
¶
RangeFilter(
key: str,
low: int | str | None = None,
high: int | str | None = None,
negated: bool = False,
)
Keep the records with a tag under key between low and high, or with negated, drop them.
Both bounds are inclusive, and a bound left as None is open, so
RangeFilter("year", low="2020") keeps 2020 and later. A record passes
when any of its tags under key is in range, as a record passes a
:class:MetadataFilter when any of its tags is among the values.
The type of the bounds decides how tags compare:
- Integer bounds compare tags numerically. Only a tag written as Python
writes an integer, such as
2024,0, or-3, can be in range; a tag such as"2024-05-01","07", or"+3"never is. - Text bounds compare tags as text, character by character.
highis compared with as many leading characters of the tag as it has, sohigh="2024-06"keeps"2024-06-30T23:59:59Z". ISO 8601 dates and years of four digits order correctly this way.
Raises:
| Type | Description |
|---|---|
TypeError
|
A bound is neither an integer nor text, such as a |
ValueError
|
Both bounds are |
Report whether the tag value lies between low and high, as :class:RangeFilter defines it.
The bounds are those of a valid :class:RangeFilter. The SQLite store
registers this function with its connection, so both stores compare tags
with the same code.
validate_filters(
filters: Iterable[SearchFilter],
allowed_keys: Collection[str] | None = None,
) -> None
Raise unless filters holds at most one filter per key, each on an allowed key.
Several values of one key belong in a single filter's values, whatever
the filters' negated flags, and a :class:MetadataFilter and a
:class:RangeFilter on one key are two filters too. allowed_keys is
None when any key is allowed.
Raises:
| Type | Description |
|---|---|
ValueError
|
Two filters share a key, or a key is not in |
validate_facet_keys(
keys: Iterable[str], allowed_keys: Collection[str]
) -> None
Raise unless every key in keys is one of allowed_keys.
Raises:
| Type | Description |
|---|---|
ValueError
|
A key is not in |
Report whether a record with metadata passes every filter in filters.
Return the distinct (key, value) tags of metadata under facet_keys.
A string, or an integer that is not a bool, gives one tag, and a list or
tuple of them gives one tag per element. Empty strings and every other type
are skipped. Keys are in sorted order, and the values of each key are sorted
and never repeat, so the rows can never collide with each other.
Encode metadata as the JSON text a text search store keeps.
Non-ASCII text stays readable UTF-8. A value JSON cannot hold never fails the
encoding: a date, datetime, or time becomes its ISO 8601 text, a
bytes or bytearray its repr, and anything else its str. The
encoding is one-way, so a datetime reads back as a string. A lone
surrogate is written as a JSON escape, so the text can always be stored.
Replace control characters other than tab and line breaks, and lone surrogates, with spaces.
The FTS5 tokenizer already treats every replaced character as a separator,
so what matches is unchanged. The replacement guarantees that the snippet
markers :data:SNIPPET_OPEN and :data:SNIPPET_CLOSE never occur in stored
text, and that SQLite can always receive it.
Strip snippet markers from raw, returning the plain text and the highlighted spans.
Each span is a half-open [start, end) character range of the plain text.
:data:SNIPPET_OPEN opens a span and :data:SNIPPET_CLOSE closes it. The
function never raises: a second opening marker inside a span and a closing
marker outside one are ignored, a span still open at the end closes there,
and an empty span is dropped.
dataclass
¶
SearchDocument(
record_id: str,
title: str = "",
abstract: str = "",
body: str = "",
metadata: dict[str, Any] = dict(),
)
One article's searchable text, as a single indexable unit.
The lexical index holds one document per record, while the vector memory holds one row per chunk. BM25 saturates on term frequency and normalizes by document length, so chunking a document would fragment its statistics and inflate short chunks.
metadata is stored as JSON and reads back as JSON: a datetime put in
comes back as its ISO 8601 string. Field types are not enforced.
:class:~sci_etl_core.search.filters.MetadataFilter matches exact values,
and :class:~sci_etl_core.search.filters.RangeFilter ranges of integers or
of text such as ISO 8601 dates.
dataclass
¶
A passage of one field, with the matched words marked.
field is one of :data:~sci_etl_core.search.query.FIELDS. text is
plain text of at most :data:~sci_etl_core.search.filters.SNIPPET_TOKENS
tokens, starting or ending with
:data:~sci_etl_core.search.filters.SNIPPET_ELLIPSIS where the field goes
on, and highlights are half-open [start, end) character offsets into
text.
dataclass
¶
TextHit(
record_id: str,
score: float,
snippet: str = "",
highlights: tuple[tuple[int, int], ...] = (),
metadata: dict[str, Any] = dict(),
title: str = "",
snippets: tuple[Snippet, ...] = (),
)
A document returned from a Boolean query, with its lexical score.
score is higher for a better match, and is never the raw negative value
SQLite's bm25() returns. Its scale depends on the corpus, so compare
scores only within one result list. snippet is plain text from the
field matching best, and highlights are half-open [start, end)
character offsets into it covering the matched words, so a UI applies its
own markup. snippets holds one :class:Snippet for every field with a
highlighted match, in :data:~sci_etl_core.search.query.FIELDS order, so a
UI can show a match in the title and one in the body together. title
and metadata are the document's, as the store reads them back.
class-attribute
instance-attribute
¶
dataclass
¶
How much a match in each field counts towards a document's BM25 score.
Raises:
| Type | Description |
|---|---|
ValueError
|
A weight is negative or not finite. |
Bases: ABC
An accumulating lexical index of articles, searchable by Boolean query.
The index is per record, while the vector memory is per chunk (see
:class:SearchDocument), so lexical and semantic results are fused at the
record level.
Queries arrive as parsed, normalized ASTs, never as text: a store never
parses, so no backend can invent its own dialect. Metadata filters apply to
the keys in :attr:facet_keys only, and a filter or facet on any other key
raises :class:ValueError before any I/O.
abstractmethod
property
¶
The metadata keys this store tags documents by, for filters and facets.
abstractmethod
async
¶
index(documents: Sequence[SearchDocument]) -> None
Store documents, replacing any stored document with the same record_id.
Control characters other than tab and line breaks are stored as spaces.
When documents repeats a record_id, the last one is kept.
abstractmethod
async
¶
delete_record(record_id: str) -> None
Remove the document of record_id, if one is stored.
async
¶
replace_record(document: SearchDocument) -> None
Make document the stored document of its record.
The default deletes and then indexes; a backend that can upsert atomically should override it.
abstractmethod
async
¶
search(
query: Node,
limit: int = 20,
exclude_record_id: str | None = None,
filters: Sequence[SearchFilter] = (),
) -> list[TextHit]
Rank the documents matching query and filters by BM25, best first.
Equal scores are ordered by record_id. filters apply before
limit, so a filtered search still returns up to limit hits.
A limit below 1 returns no hits.
Raises:
| Type | Description |
|---|---|
ValueError
|
A filter's key is not in :attr: |
SearchQueryError
|
|
SearchStoreError
|
The index cannot be read, or a filter's key has no
built tags yet (see |
abstractmethod
async
¶
filter_ids(
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> frozenset[str]
Return every record satisfying query and filters, in no order.
Any query is accepted, including a pure negation, which is why the
result carries no order. query=None matches every document.
Raises:
| Type | Description |
|---|---|
ValueError
|
A filter's key is not in :attr: |
SearchStoreError
|
The index cannot be read, or a filter's key has no
built tags yet (see |
abstractmethod
async
¶
get_documents(
record_ids: Collection[str],
) -> dict[str, SearchDocument]
Return the stored documents for record_ids; unknown ids are omitted.
abstractmethod
async
¶
facet_counts(
keys: Sequence[str],
*,
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> dict[str, tuple[tuple[str, int], ...]]
Count matching documents per value of each facet key.
For each key, documents are counted when they satisfy query and
every filter whose key is not key; filters on key itself are
ignored for that key's counts. Each count answers "how many results
would this value give if it were selected instead". When filters
has no filter on key, the exclusion is a no-op, and that key's
counts are over the full match set of query and every other key's
filter. Values are sorted by count descending, then value ascending.
Values with a count of 0 are omitted from the returned tuple, and a key
with no matching value maps to an empty tuple.
Raises:
| Type | Description |
|---|---|
ValueError
|
A key in |
SearchStoreError
|
The index cannot be read, or a key in |
async
¶
range_counts(
ranges: Sequence[RangeFilter],
*,
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> tuple[int, ...]
Count matching documents inside each of ranges, in the order given.
Each count is of the documents satisfying query, the range, and every
filter whose key is not the range's key, so a count answers "how many
results would this range give if it were selected instead", as
:meth:facet_counts does for exact values. Ranges may share a key and
overlap, as buckets of a histogram or presets such as "last 5 years" do,
and a negated range counts the documents outside it.
The default runs :meth:filter_ids once per range; a backend that can
count in one read should override it.
Raises:
| Type | Description |
|---|---|
ValueError
|
A key in |
SearchStoreError
|
The index cannot be read, or a key has no built
tags yet (see |
InMemoryTextSearchStore(
*,
facet_keys: Iterable[str] = (),
weights: BM25Weights | None = None,
tokenizer: Tokenizer | None = None,
)
Bases: AsyncTextSearchStore
Non-persistent lexical index backed by a dict, ranked as SQLite FTS5 ranks.
Documents are tokenized once, when indexed, by tokenizer (by default
:class:~sci_etl_core.search.tokenize.Unicode61Tokenizer, which reproduces
FTS5's own tokenizer). Matching follows
:func:~sci_etl_core.search.evaluate.matches, so with the default tokenizer
the store matches, filters, and counts exactly the records
:class:~sci_etl_core.search.store_sqlite_fts5.AsyncSqliteFts5Store does,
which makes it both an offline test double and a check on that store.
Scores follow FTS5's bm25() formula: k1 of 1.2, b of 0.75, an IDF
floored at 1e-6, and the field weights. A word counts towards the
score, and is highlighted, only where the part of the query holding it
matches the document: an OR alternative or an AND group that does not
match, and anything negated, adds nothing. FTS5 applies the same rule in
most cases but not all: whether it counts a word inside a part of the query
that fails to match depends on the state of its internal iterators, so for
such queries the two stores can score, and so order, the same records
differently. Known divergence: FTS5 may count a term inside a failing
OR branch, so raw scores can differ on mixed-negation queries; matched
record sets are identical. The property-based parity test compares sets,
not scores.
The snippet is taken from the field with the most counted matches, with
ties going to title, then abstract, then body, and snippets
holds one for every field with a counted match.
Tags are derived from metadata on every write, so a filter or facet on any
of facet_keys works at once: the store never needs a rebuild_tags
and never raises :class:~sci_etl_core.exceptions.SearchStoreError.
async
¶
search(
query: Node,
limit: int = 20,
exclude_record_id: str | None = None,
filters: Sequence[SearchFilter] = (),
) -> list[TextHit]
async
¶
filter_ids(
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> frozenset[str]
AsyncSqliteFts5Store(
path: str | Path,
*,
facet_keys: Iterable[str] = (),
weights: BM25Weights | None = None,
now: Callable[[], datetime] | None = None,
)
Bases: AsyncTextSearchStore
Durable lexical index on SQLite FTS5, keeping each article's text once.
Backed by the standard-library sqlite3 driver run off the event loop, so
it carries no third-party dependency. The text lives in a documents
table and FTS5 holds only the inverted index, kept consistent by triggers.
Every write is one transaction, so re-indexing a record replaces its text,
metadata, and tags atomically. The single connection is used by one worker
thread at a time, even when an awaiting task is cancelled, and every SQLite
failure, including a file that is not a database, surfaces as
:class:~sci_etl_core.exceptions.SearchStoreError.
Ranking is FTS5's bm25() with the field weights, reported as a
score where higher is better. snippet comes from the one field FTS5
chooses, and snippets holds FTS5's snippet of every field it
highlights a match in. On the queries where FTS5 counts a word inside a
part of the query that fails to match, as
:class:~sci_etl_core.search.store_memory.InMemoryTextSearchStore
describes, the two stores can highlight different words and so list
different fields. Each document's indexed_at is the UTC time from
now, as an ISO 8601 string with an offset.
facet_keys names the metadata keys tagged for filters and facets. It is
fixed for the life of the instance and recorded in the file. To change it,
construct a store with the new keys and call :meth:rebuild_tags; until
then a filter or facet on a key whose tags are not built raises
:class:~sci_etl_core.exceptions.SearchStoreError.
The file is opened, and created or migrated to the current schema, on the
first operation rather than at construction. A file that is not a database,
or whose schema is newer than this library supports, raises
:class:~sci_etl_core.exceptions.SearchStoreError then.
:meth:aclose is not terminal: a later call reopens the connection. Give
each instance exactly one owner, which awaits :meth:aclose, and use it
from one event loop. For a one-shot script, list the store in the
pipeline's closeables.
Raises:
| Type | Description |
|---|---|
SearchStoreError
|
This interpreter's SQLite was built without FTS5. |
async
¶
replace_record(document: SearchDocument) -> None
Replace the record's text, metadata, and tags in one transaction.
async
¶
search(
query: Node,
limit: int = 20,
exclude_record_id: str | None = None,
filters: Sequence[SearchFilter] = (),
) -> list[TextHit]
async
¶
filter_ids(
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> frozenset[str]
async
¶
facet_counts(
keys: Sequence[str],
*,
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> dict[str, tuple[tuple[str, int], ...]]
async
¶
range_counts(
ranges: Sequence[RangeFilter],
*,
query: Node | None = None,
filters: Sequence[SearchFilter] = (),
) -> tuple[int, ...]
Count matching documents inside each of ranges in one consistent read.
The counts are those :meth:AsyncTextSearchStore.range_counts describes.
async
¶
Merge the FTS5 index's segments, which speeds up later queries. Reads no text.
async
¶
Re-tokenize every stored document into a fresh FTS5 index.
The text already stored in the file is used; nothing is fetched again.
This repairs an index that :meth:integrity_check reports inconsistent.
async
¶
Re-derive every document's tags from its stored metadata, for this store's facet keys.
One transaction replaces every tag and records :attr:facet_keys as
built, so filters and facets on those keys work afterwards, and tags of
keys no longer in :attr:facet_keys are gone. No text is read.
Raises:
| Type | Description |
|---|---|
SearchStoreError
|
The tags cannot be written, or a stored document's metadata is not a JSON object. |
async
¶
integrity_check() -> bool
Report whether the FTS5 index agrees with the stored documents.
False means the index is inconsistent, for example after the
documents table was written without its triggers;
:meth:rebuild_index repairs it.
Raises:
| Type | Description |
|---|---|
SearchStoreError
|
The check could not run. |
fts5_available(
connect: Callable[[], Connection] | None = None,
) -> bool
Report whether this interpreter's SQLite was built with FTS5.
connect opens the connection to probe, by default an in-memory database
opened through :func:sqlite3.connect as it is at call time. A probe that
cannot connect, or cannot create an FTS5 table, reports False.
snippet_window(
text: str,
tokens: Sequence[Token],
ranges: Sequence[tuple[int, int]],
) -> tuple[str, tuple[tuple[int, int], ...]]
Cut text to a window of at most :data:SNIPPET_TOKENS tokens around its first highlighted range.
tokens are the tokens of text, and ranges are [start, stop)
token ranges to highlight, in any order and possibly overlapping. The window
starts at the first range, or earlier when that range is near the end, and
:data:SNIPPET_ELLIPSIS marks text left out before or after it. Returns the
window's text and its highlights as [start, end) character offsets into
that text, with overlapping ranges merged into one span.
passage_snippet(
query: Node,
text: str,
field: str = "body",
*,
tokenizer: Tokenizer | None = None,
) -> Snippet
Return a snippet of text, read as field, highlighting every word of query it contains.
Unlike a store's snippet, which highlights only the parts of the query a
document satisfies, every term and phrase that is not negated is
highlighted wherever it occurs, whether or not text matches the whole
query. That suits text found another way, such as the passage a semantic
search ranked best. A term or phrase scoped to other fields is not
highlighted, and the terms of a NEAR group are highlighted wherever they
occur. Without a highlight, the snippet is the start of text.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
AsyncSearchIndexer(store: AsyncTextSearchStore)
Index an article's title, abstract, and full text for Boolean search.
The lexical counterpart of
:class:~sci_etl_core.embeddings.ingest_async.AsyncChunkIngestor: the same
ingest method and the same replace-not-append semantics, with one
document per record instead of one row per chunk. It satisfies
:class:~sci_etl_core.ingest_protocol.MemoryIngestor, so a search-only
deployment passes it straight to the pipeline's memory_ingestor.
The indexer never swallows a failure and never parses a query; whether a
store fault is fatal is decided by its caller. It borrows store and
never closes it.
async
¶
Replace the record's indexed document with its title, abstract, and text.
A record whose title, abstract, and text are all blank (empty or
whitespace only) is removed from the index, as AsyncChunkIngestor
clears a record whose text yields no passages. A record with any one of
them non-blank is stored, with the blank fields stored as empty strings.
record.metadata is copied into the document.
Returns:
| Type | Description |
|---|---|
int
|
1 when a document was stored, 0 when the record was cleared. |
Raises:
| Type | Description |
|---|---|
SearchStoreError
|
The index could not be written. |
dataclass
¶
What :func:backfill_text_index did with each record of the vector memory.
indexed records were written to the text index. skipped_existing
were already in it and left alone, and skipped_empty had no document to
write, such as a record whose passages and title are all blank.
Join overlapping passages back into one text, keeping each overlapping word once.
Passages are split into words at whitespace, as
:class:~sci_etl_core.embeddings.chunking.SlidingWindowChunker splits
text, and the result joins the words with single spaces. Each passage after
the first drops its first overlap_words words when they repeat the last
words so far and it has more words than that, so the text of a
sliding-window chunker comes back with every word counted once. Any other
passage is kept whole, so passages from another chunker lose nothing.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
stored_record_document(
record: StoredRecord, body: str
) -> SearchDocument | None
Build the text-index document for a record read back from the vector memory.
The title is the title of the record's chunk metadata, the body is
body, and the abstract is empty, because the vector memory does not
hold it. The rest of the chunk metadata, such as source_url, becomes the
document's metadata; the record's original metadata, such as
categories or year, is not in the vector memory. A record with a
blank title and a blank body gives None.
async
¶
backfill_text_index(
vector_store: AsyncEmbeddingStore,
text_store: AsyncTextSearchStore,
*,
overlap_words: int,
replace_existing: bool = False,
build_document: Callable[
[StoredRecord, str], SearchDocument | None
] = stored_record_document,
batch_size: int = 100,
) -> BackfillReport
Index the text already in the vector memory, so a deployment need not fetch full texts again.
Every record of vector_store is read with
:meth:~sci_etl_core.embeddings.store_base.AsyncEmbeddingStore.iter_records,
its passages are joined by :func:merge_passages with overlap_words,
the overlap of the chunker that wrote them (for example
SlidingWindowChunker().overlap_words), and build_document turns the
record and the joined text into the document to index. Joining without
removing the overlap would count the words at every window boundary twice,
and BM25 would over-weigh them.
The default :func:stored_record_document has no abstract and none of the
record's original metadata, so filters and facets on keys such as year
do not match a backfilled document. Pass a build_document that adds
them from a source of your own, or return None to leave a record out.
Records already in text_store are left alone unless
replace_existing is true, because a document the pipeline indexed holds
the abstract and metadata a backfilled one lacks. Documents are written
batch_size at a time. Neither store is closed.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
NotImplementedError
|
|
EmbeddingStoreError
|
The vector memory cannot be read. |
SearchStoreError
|
The text index cannot be read or written. |
dataclass
¶
FusedHit(
record_id: str,
score: float,
lexical_rank: int | None = None,
semantic_rank: int | None = None,
snippet: str = "",
highlights: tuple[tuple[int, int], ...] = (),
metadata: dict[str, Any] = dict(),
title: str = "",
snippets: tuple[Snippet, ...] = (),
)
One record of a fused result list, with where each retrieval leg ranked it.
lexical_rank and semantic_rank are 1-based, and None when that
leg did not return the record. snippet, highlights, and
snippets are the lexical hit's when the lexical leg returned the
record. A record found only by the semantic leg gets a snippet of the
passage that ranked it, with the query's words highlighted where they occur
in it, as its only entry in snippets, whose field is "body". score
comes from the fusion strategy and is comparable only within one result
list; render the rank, never the score as a percentage.
class-attribute
instance-attribute
¶
dataclass
¶
How ranked lists are fused.
k damps reciprocal rank fusion: a larger k flattens the difference
between neighbouring ranks. weights holds one weight per fused list, in
list order, and None weighs every list equally. A list of weights is
stored as a tuple.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Bases: Protocol
Fuses ranked lists of (record_id, score) pairs into one ranking.
The result holds each record of the input lists once, as a
(record_id, fused_score) pair, best first.
reciprocal_rank_fusion(
ranked_lists: Sequence[ScoredList],
params: FusionParams | None = None,
) -> list[tuple[str, float]]
Fuse ranked lists by 1 / (k + rank), the rank-only fusion of Cormack et al.
Only the order of each list is read; its scores are ignored. That makes the fusion scale-free. BM25 scores are unbounded and depend on the corpus, while cosine similarity lies in [-1, 1], so any weighted sum of the two would make the effective blend drift as the corpus grows; RRF needs no calibration.
A record's rank in a list is its 1-based position among the list's distinct
records, so a repeated record counts once, at its first position. Its fused
score is the sum over lists of weight / (k + rank), rounded exactly, so
the result does not depend on the order of the lists. The result is best
first, and equal fused scores are ordered by record_id.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
normalized_score_fusion(
ranked_lists: Sequence[ScoredList],
params: FusionParams | None = None,
) -> list[tuple[str, float]]
Fuse ranked lists by min-max normalizing each list's scores, then adding them with weights.
Unlike :func:reciprocal_rank_fusion, score mass matters: a record far
ahead of the next one keeps that lead. The price is calibration. Raw BM25
scores depend on the corpus, so how much the lexical list counts drifts as
the corpus changes. params.k is not used.
Within a list, the best score becomes 1 and the worst 0, and a list whose
scores are all equal gives every record 1. A repeated record keeps the
score of its first occurrence. The result is best first, and equal fused
scores are ordered by record_id.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
module-attribute
¶
SearchMode = Literal['lexical', 'semantic', 'hybrid']
Which retrieval legs :meth:AsyncHybridSearcher.search runs.
module-attribute
¶
Every :data:SearchMode, for checking a mode that arrives as plain text.
dataclass
¶
How many candidates each retrieval leg fetches before fusion.
Each leg fetches candidate_pool records, and never fewer than the
top_k asked for, so a record ranked #40 lexically and #3 semantically
can still reach the top 20.
The semantic leg asks the vector memory for candidate_pool ×
chunk_pool_factor chunks and keeps each record's best chunk. It yields
candidate_pool distinct records whenever those chunks span at least that
many records, that is, when records contribute on average no more than
chunk_pool_factor chunks to that prefix of the chunk ranking. When a few
long records dominate the top chunks, fewer records are returned. The
searcher does not retry, because a retry would call the embedder again for
the same text. Raise the factor for corpora of long documents.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
dataclass
¶
SearchOutcome(
hits: list[FusedHit] = list(),
degraded: tuple[str, ...] = (),
skipped: tuple[str, ...] = (),
)
The fused hits of one search, and which retrieval legs did not contribute.
degraded names the legs that were attempted and failed, such as
"semantic" when the embedding service was unreachable. skipped names
the legs that had nothing to run, such as a semantic leg without a finder or
for a query made only of prefix terms. A UI words the two differently.
AsyncHybridSearcher(
text_store: AsyncTextSearchStore,
finder: AsyncSimilarArticleFinder | None = None,
*,
strategy: FusionStrategy = reciprocal_rank_fusion,
params: HybridParams | None = None,
fusion: FusionParams | None = None,
logger: Callable[[str], None] | None = None,
)
Search the corpus by Boolean query, by meaning, or by both at once.
mode="lexical" ranks documents by BM25 in the text index.
mode="semantic" ranks whole articles by their best-matching chunk in the
vector memory. mode="hybrid" runs both at once and fuses the two
rankings with strategy, by default reciprocal rank fusion.
The query is parsed once, before any I/O. The semantic leg embeds
:func:~sci_etl_core.search.query.semantic_text of the query, so operators,
field scopes, negated terms, and prefix terms never reach the embedder.
Metadata filters apply to both legs: the lexical leg filters inside the
index, and the semantic leg's candidates are narrowed to the records passing
the filters before fusion, so a filtered-out record never takes a slot.
A hybrid search whose semantic leg raises
:class:~sci_etl_core.exceptions.EmbeddingError falls back to the lexical
results, logs the failure, and reports it in :attr:SearchOutcome.degraded.
A lexical failure is never swallowed, because it means the local index is
broken.
The searcher never closes; the caller that constructed the stores closes them.
Configure the searcher; finder may be None to search the text index alone.
Without a finder, a hybrid search runs the lexical leg alone and reports
the semantic leg in :attr:SearchOutcome.skipped, not in degraded,
and a semantic search raises
:class:~sci_etl_core.exceptions.SearchQueryError.
fusion.weights, when given, weighs the lexical and the semantic list,
in that order. logger receives the line logged when a hybrid search
falls back to lexical results.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
async
¶
search(
query: str,
top_k: int = 20,
*,
mode: SearchMode = "hybrid",
filters: Sequence[SearchFilter] = (),
) -> SearchOutcome
Return the top_k best records for query, fused across the legs mode runs.
Every error below except the last two is raised before any I/O. A
top_k below 1 returns no hits.
A hit the lexical leg found carries its snippets and highlights. A hit
only the semantic leg found takes its title and metadata from the text
index, or from its best chunk's metadata when the index does not hold
the record, and its snippet from that chunk, with the query's words
highlighted where they occur in it
(:func:~sci_etl_core.search.snippets.passage_snippet).
Raises:
| Type | Description |
|---|---|
ValueError
|
|
SearchQueryError
|
The query is malformed or cannot be ranked, such
as a pure negation; |
EmbeddingError
|
The semantic leg failed in |
SearchStoreError
|
The text index cannot be read. |
dataclass
¶
GraphNode(
record_id: str,
title: str = "",
degree: int = 0,
community: int = 0,
metadata: dict[str, Any] = dict(),
)
A record in a discovery graph.
degree counts the distinct records the node shares an edge with.
community is the canonical id :func:label_communities assigned.
title and metadata come from the text index, and are empty for a
record the index does not hold.
dataclass
¶
An undirected weighted edge, with source sorting before target.
kind names the edge source that produced it, such as "semantic",
"metadata", or "citation".
dataclass
¶
DiscoveryGraph(
nodes: tuple[GraphNode, ...],
edges: tuple[GraphEdge, ...],
seed_record_id: str | None,
communities_converged: bool,
)
The neighbourhood of a seed record, as topology only.
It holds no coordinates, colours, or layout: layout is a rendering concern,
so a UI runs force-directed layout over this data. communities_converged
is False when label propagation stopped at its pass limit, so a UI can
say the communities are approximate.
dataclass
¶
GraphParams(
depth: int = 2,
fanout: int = 8,
min_weight: float = 0.35,
max_nodes: int = 200,
mutual_only: bool = True,
max_iterations: int = 20,
)
How far and how densely a discovery graph grows around its seed.
The graph grows depth breadth-first levels. Each record expands to the
up to fanout neighbours per edge source whose weight is at least
min_weight. max_nodes is a hard cap, checked before each level is
expanded. With mutual_only, an edge is kept only when each record is
among the other's fanout nearest, which keeps a hub record from
connecting to everything. max_iterations bounds label propagation.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
select_edges(
neighbour_lists: Sequence[
tuple[
str, Mapping[str, Sequence[tuple[str, float]]]
]
],
nodes: Collection[str],
*,
min_weight: float,
mutual_only: bool,
) -> list[GraphEdge]
Turn each edge source's neighbour lists into undirected edges between nodes.
neighbour_lists pairs each source's edge kind with its lists, which map a
record to its nearest records, already cut to the fanout. A record gets an
edge of that kind to a neighbour in its list when both are in nodes, they
differ, and the weight is at least min_weight. With mutual_only, the
record must also be in the neighbour's list. Edges of one kind between one
pair merge, keeping the heaviest weight. Edges are sorted by source, target,
and kind.
label_communities(
nodes: Sequence[str],
edges: Sequence[GraphEdge],
*,
max_iterations: int = 20,
) -> tuple[dict[str, int], bool]
Assign each node a community by label propagation.
Returns the communities at convergence and True, or, if a pass limit of
max_iterations is reached first, the communities after the last pass and
False. The result does not depend on the order of nodes or edges:
- A pair joined by several edges gets the heaviest weight, in both
directions. Self-loops and edges touching a record outside
nodesare ignored. - Each node starts with its index in sorted
record_idorder as its label. - A pass visits nodes in sorted order and updates labels in place. A node sums edge weights per neighbouring label, visiting neighbours in sorted order, and switches only when the heaviest label's total is strictly greater than its current label's; among labels tied for heaviest, it takes the lowest.
- A pass that changes no label ends the run as converged.
max_iterationscounts every pass, including that last one. - Communities are numbered 0, 1, 2, … in order of their smallest member.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
filter_graph(
graph: DiscoveryGraph,
*,
matched_ids: Collection[str] | None = None,
filters: Sequence[SearchFilter] = (),
) -> DiscoveryGraph
Keep the nodes whose record is in matched_ids and whose metadata passes filters.
The seed is always kept. Edges losing an endpoint are dropped, and degrees
count the remaining edges; communities are kept as they were, so a UI's
colours stay stable while filtering. matched_ids usually comes from the
text store's filter_ids, which also answers a pure negation, and
filters are applied in memory with the same semantics as the stores, so
a facet toggle needs no I/O.
Raises:
| Type | Description |
|---|---|
ValueError
|
Two filters share a key. |
async
¶
build_discovery_graph(
seed_record_id: str,
sources: Sequence[AsyncEdgeSource],
text_store: AsyncTextSearchStore,
*,
params: GraphParams | None = None,
) -> DiscoveryGraph
Grow the discovery graph around seed_record_id.
Each breadth-first level issues one batched neighbours call per source,
with the sources running concurrently. A record's neighbour lists are fetched
at most once per build, and reused for the mutual check. When sources fail,
every source is awaited first, and then the first failure in source order is
raised. Each source's lists are cleaned before use: a record never neighbours
itself, a repeated neighbour keeps its heaviest weight, weights that are not
finite are dropped, and each list is cut to the fanout.
Edges are chosen by :func:select_edges, and only records connected to the
seed through them are kept; the seed always is. Communities come from
:func:label_communities, and titles and metadata from one
get_documents call on text_store.
A neighbour query costs what the source costs. With an exact-scan vector
store, a build issues up to one query per node, each scanning every stored
chunk, so keep max_nodes small for large memories or use an approximate
nearest-neighbour store behind the same interface.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Bases: ABC
Finds each record's nearest records by one notion of relatedness, for a discovery graph.
A source borrows the stores it reads and never closes them.
abstractmethod
property
¶
kind: str
The kind of the edges this source produces, such as "semantic".
abstractmethod
async
¶
Return up to limit weighted neighbours for each record, best first.
A neighbour is a (record_id, weight) pair, and a higher weight means
more related. The ids arrive as one batch, one breadth-first level of a
graph build, so a source can share work across them. Every requested
record is a key of the result, mapping to an empty list when it has no
neighbour.
MetadataEdgeSource(
text_store: AsyncTextSearchStore,
keys: Sequence[str] = ("categories", "authors"),
)
Bases: AsyncEdgeSource
Relates records that share tag values, such as an arXiv category or an author.
The weight is the Jaccard index of two records' tags under keys: the
(key, value) tags they share divided by all their distinct tags. It needs
no embeddings, only tags built for keys in the text store (see
:func:~sci_etl_core.search.filters.tag_rows).
A batch costs one get_documents call for its records, one filter_ids
call per distinct (key, values) among them, and one get_documents
call for the candidates, so a tag shared by much of the corpus makes the
candidate set large.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
EmbeddingEdgeSource(
embedder: AsyncEmbedder,
store: AsyncEmbeddingStore,
text_store: AsyncTextSearchStore,
*,
chunk_pool_factor: int = 5,
)
Bases: AsyncEdgeSource
Relates records whose text means similar things, by cosine similarity in the vector memory.
For each batch, the records' title and abstract are read from text_store
and embedded in one embed call. Each record then runs one vector-memory
query for limit × chunk_pool_factor chunks, excluding its own chunks,
and keeps each other record's best chunk, as
:meth:~sci_etl_core.embeddings.finder_async.AsyncSimilarArticleFinder.find_similar_articles
does. chunk_pool_factor has the meaning of
:attr:~sci_etl_core.search.hybrid_async.HybridParams.chunk_pool_factor.
A record absent from the text index, or whose title and abstract are both
blank, has no neighbours. Only similarities of at least 0 count.
The embedder and stores are used through embed and query only, so
importing this module never loads NumPy. With an exact-scan vector store,
every query scans every stored chunk.
Raises:
| Type | Description |
|---|---|
ValueError
|
|