Processors¶
Every name on this page is importable from sci_etl_core.processors.
Bases: KeyNormalizer
Domain-agnostic key: case, width, spacing, and punctuation differences are ignored.
NormalizationStep(
key_column: str,
normalizer: KeyNormalizer,
output_column: str = "_norm_key",
)
DeduplicationStep(
norm_key_column: str,
matcher: NeighborMatcher | None = None,
match_threshold: float = 0.0,
mergeable_columns: list[str] | None = None,
*,
source_column: str | None = None,
sources_column: str = "sources",
)
Bases: Processor
Collapse rows sharing a normalized key, then merge matched neighbors.
Rows with the same key become one row holding the first non-missing value of each column, ordered by key. Rows whose key is missing or empty carry no identity to match on, so each passes through as its own row, after the keyed rows, instead of being merged with every other keyless row.
Matched pairs only fill the kept row's gaps. When a pair names a row that was already merged away as the one to keep, its values flow into the row that absorbed it, so a chain of matches never discards data.
With source_column, such as the record_id column an exporter
writes, each output row gains sources_column: the distinct non-empty
source_column values of every input row merged into it, sorted and
joined with "; ". The column says which papers a merged row came from,
though each value in the row is still the first one found.
ClusteringStep(
feature_extractor: FeatureExtractor,
output_column: str = "cluster_id",
eps: float = 1.0,
min_samples: int = 2,
)
QualityFlagStep(
completeness_column: str = "completeness_pct",
output_column: str = "quality_flag",
review_threshold: float = 50.0,
)
Bases: Processor
Label each row from its completeness, as computed by :class:CompletenessStep.
dataclass
¶
One reason a validator gives for rejecting, or warning about, an entity.
code is a short machine-readable rule name such as "out-of-range",
field is the entity key it concerns, or None when it concerns the
entity as a whole, and message is a sentence a reviewer can read.
dataclass
¶
Bases: ABC
Contract for a domain rule that decides whether an extracted entity is kept.
:meth:is_valid answers yes or no, and :meth:validate also says why.
:class:~sci_etl_core.llm.extraction_async.AsyncLLMEntityExtractor calls
:meth:validate. A validator that overrides only :meth:is_valid keeps
working, because the default :meth:validate turns its rejection into one
violation.
abstractmethod
¶
Return whether a raw extracted record should be kept.
validate(record: dict[str, Any]) -> ValidationResult
Return every reason record is rejected.
The default wraps :meth:is_valid: a rejection becomes one
:class:Violation with code "rejected", no field, and the
validator's class name in the message. Override it to name the field
and the rule. An override must reject exactly the records
:meth:is_valid rejects.
Bases: RecordValidator
Reject an entity whose key_field is null-like or contains a forbidden keyword as whole words.
Return False for a missing, empty, or null-like key, or one containing a forbidden phrase.
Null-like values are null, none, unknown, n/a, and
nan, in any case. Keys and keywords are split into runs of letters
and runs of numbers after NFKC case folding, and a keyword matches only
a contiguous sequence of whole runs, so "star" never matches
"starburst".
validate(record: dict[str, Any]) -> ValidationResult
Name the rule a rejected key broke: "null-key" or "forbidden-keyword".
Bases: RecordValidator
Reject an entity whose value for a field lies outside that field's inclusive range.
CompositeValidator(validators: list[RecordValidator])
Bases: RecordValidator
Keep an entity only when every validator keeps it.
Bases: Processor
Clamp numeric columns into closed ranges during post-processing.
Each column named in bounds is converted to numbers, with values that
are not numeric becoming NaN, and then clipped to (low, high). This
is the post-processing counterpart of the CSV exporter's numeric_clip.
Columns missing from the frame are skipped, and the input frame is never
modified.
Store the bounds to apply.
Raises:
| Type | Description |
|---|---|
ValueError
|
A bound is |
TableLayoutStep(
sort_by: Sequence[tuple[str, bool]] = (),
leading_columns: Sequence[str] = (),
hidden_prefixes: Iterable[str] = (),
reset_index: bool = False,
)
Bases: Processor
Sort rows and arrange columns for a published table.
Rows are sorted by the sort_by columns that exist in the frame, each
ascending or descending as paired, with missing values last and ties kept
in their input order. Columns whose name starts with any of
hidden_prefixes are dropped, then the leading_columns that exist
come first in the given order, followed by the rest in their input order.
The input frame is never modified.
Store the layout.
sort_by pairs each column with True for ascending order.
reset_index renumbers the rows from 0 after sorting.
Raises:
| Type | Description |
|---|---|
ValueError
|
A column appears twice in |
The table sinks need the processors extra, and SqlTableSink and Plotly3DSink also need sql and
viz when constructed. ScatterPlotConfig and render_scatter_3d are imported from
sci_etl_core.processors.sinks.
Destinations for a post-processed table.
Importing this module needs the processors extra (pandas). Constructing
:class:SqlTableSink also needs the sql extra, and :class:Plotly3DSink
the viz extra; each imports its library when constructed, so importing
this module never fails for a missing optional library.
Bases: TableSink
Write a table to a SQL database through SQLAlchemy. Needs the sql extra.
url is a synchronous SQLAlchemy database URL, such as
"sqlite:///catalogue.db", and if_exists is passed to
DataFrame.to_sql. Each :meth:write creates an engine, writes the
table without its index in one transaction, and disposes of the engine.
Raises:
| Type | Description |
|---|---|
ImportError
|
SQLAlchemy is not installed. |
dataclass
¶
ScatterPlotConfig(
x_column: str,
y_column: str,
z_column: str,
color_column: str,
size_column: str | None = None,
hover_name_column: str | None = None,
title: str = "3D Scatter",
template: str = "plotly_dark",
hover_data_columns: Sequence[str] = (),
hover_template: str | None = None,
color_continuous_scale: str
| Sequence[str]
| None = None,
color_range: tuple[float, float] | None = None,
color_label: str | None = None,
marker: Mapping[str, Any] = dict(),
layout: Mapping[str, Any] = dict(),
)
What a 3D scatter plot shows and how it looks.
hover_data_columns are passed to the figure as custom data, so
hover_template can show them as %{customdata[0]}, %{customdata[1]},
and so on, in the order listed; the hover_name_column value is
%{hovertext}. A numeric color_column is drawn with
color_continuous_scale, such as "Viridis", over color_range
when one is given, so colors mean the same in every export regardless of
the values present. color_label names the color bar or legend.
marker is applied to every trace's marker, such as
{"sizemode": "diameter", "sizemin": 3}, and layout is applied to the
figure layout last, overriding template and the default margins.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
class-attribute
instance-attribute
¶
Plotly3DSink(config: ScatterPlotConfig, path: str | Path)
Bases: TableSink
Write a table as an interactive 3D scatter plot in a standalone HTML file. Needs the viz extra.
Rows missing an x, y, or z value are left out, and nothing is written when no row remains. The file is replaced atomically.
Raises:
| Type | Description |
|---|---|
ImportError
|
Plotly is not installed. |
render_scatter_3d(
frame: DataFrame, config: ScatterPlotConfig
) -> str | None
Render frame as a standalone HTML 3D scatter plot, or return None when no row has x, y, and z.
Needs the viz extra.