Processors¶
Every name on this page is importable from sci_etl_core.processors.
Bases: KeyNormalizer
Domain-agnostic key: case, width, spacing, and punctuation differences are ignored.
NormalizationStep(
key_column: str,
normalizer: KeyNormalizer,
output_column: str = "_norm_key",
)
DeduplicationStep(
norm_key_column: str,
matcher: NeighborMatcher | None = None,
match_threshold: float = 0.0,
mergeable_columns: list[str] | None = None,
)
Bases: Processor
Collapse rows sharing a normalized key, then merge matched neighbors.
Rows with the same key become one row holding the first non-missing value of each column, ordered by key. Rows whose key is missing or empty carry no identity to match on, so each passes through as its own row, after the keyed rows, instead of being merged with every other keyless row.
Matched pairs only fill the kept row's gaps. When a pair names a row that was already merged away as the one to keep, its values flow into the row that absorbed it, so a chain of matches never discards data.
ClusteringStep(
feature_extractor: FeatureExtractor,
output_column: str = "cluster_id",
eps: float = 1.0,
min_samples: int = 2,
)
QualityFlagStep(
completeness_column: str = "completeness_pct",
output_column: str = "quality_flag",
review_threshold: float = 50.0,
)
Bases: Processor
Label each row from its completeness, as computed by :class:CompletenessStep.
Bases: RecordValidator
Reject an entity whose key_field is null-like or contains a forbidden keyword as whole words.
Return False for a missing, empty, or null-like key, or one containing a forbidden phrase.
Null-like values are null, none, unknown, n/a, and
nan, in any case. Keys and keywords are split into runs of letters
and runs of numbers after NFKC case folding, and a keyword matches only
a contiguous sequence of whole runs, so "star" never matches
"starburst".
Bases: RecordValidator
Reject an entity whose value for a field lies outside that field's inclusive range.
CompositeValidator(validators: list[RecordValidator])
Bases: Processor
Clamp numeric columns into closed ranges during post-processing.
Each column named in bounds is converted to numbers, with values that
are not numeric becoming NaN, and then clipped to (low, high). This
is the post-processing counterpart of the CSV exporter's numeric_clip.
Columns missing from the frame are skipped, and the input frame is never
modified.
Store the bounds to apply.
Raises:
| Type | Description |
|---|---|
ValueError
|
A bound is |
TableLayoutStep(
sort_by: Sequence[tuple[str, bool]] = (),
leading_columns: Sequence[str] = (),
hidden_prefixes: Iterable[str] = (),
reset_index: bool = False,
)
Bases: Processor
Sort rows and arrange columns for a published table.
Rows are sorted by the sort_by columns that exist in the frame, each
ascending or descending as paired, with missing values last and ties kept
in their input order. Columns whose name starts with any of
hidden_prefixes are dropped, then the leading_columns that exist
come first in the given order, followed by the rest in their input order.
The input frame is never modified.
Store the layout.
sort_by pairs each column with True for ascending order.
reset_index renumbers the rows from 0 after sorting.
Raises:
| Type | Description |
|---|---|
ValueError
|
A column appears twice in |