Exporters¶
Every name on this page is importable from sci_etl_core.exporters.
Bases: ABC
Contract for a sink that persists extracted data.
:class:~sci_etl_core.pipeline_async.AsyncETLPipeline calls :meth:export
once per relevant record with that record's entities as a
list[dict[str, Any]], possibly from several records concurrently. An
implementation used in a pipeline must accept that shape and serialize its
own writes.
AsyncCsvUpsertExporter(
key_column: str,
value_columns: list[str],
normalizer: KeyNormalizer,
numeric_clip: dict[str, tuple[float, float]]
| None = None,
escape_formulas: bool = True,
)
Bases: AsyncExporter
Concurrency-safe, crash-safe CSV upsert exporter.
All read-modify-write cycles are serialized through a single
:class:asyncio.Lock, so concurrent export calls can never overwrite
one another. Each call renders the full merged snapshot and publishes it
with an atomic rename, and the in-memory buffer is only advanced once the
rename succeeds, keeping memory and disk consistent after a failure.
The existing file is read before the first write to each destination, and the snapshot is adopted only once that read succeeds. A file that cannot be read (a foreign encoding, a malformed row, a lock held by another program) fails the export instead of letting a later call overwrite the file with only the new rows.
Keys come from LLM output, so by default any key a spreadsheet would treat as a formula is written with a leading apostrophe and restored on reload. Value columns are numeric and need no escaping.
Bases: AsyncExporter
Write a dataframe to a SQL table through an async SQLAlchemy engine. Needs the sql extra.
It takes a DataFrame, not the pipeline's list[dict] of entities, so
it serves post-processing output rather than
:class:~sci_etl_core.pipeline_async.AsyncETLPipeline directly.
Target table_name; if_exists is passed to DataFrame.to_sql.
dataclass
¶
ScatterPlotConfig(
x_column: str,
y_column: str,
z_column: str,
color_column: str,
size_column: str | None = None,
hover_name_column: str | None = None,
title: str = "3D Scatter",
template: str = "plotly_dark",
hover_data_columns: Sequence[str] = (),
hover_template: str | None = None,
color_continuous_scale: str
| Sequence[str]
| None = None,
color_range: tuple[float, float] | None = None,
color_label: str | None = None,
marker: Mapping[str, Any] = dict(),
layout: Mapping[str, Any] = dict(),
)
What an :class:AsyncPlotly3DExporter plots and how it looks.
hover_data_columns are passed to the figure as custom data, so
hover_template can show them as %{customdata[0]}, %{customdata[1]},
and so on, in the order listed; the hover_name_column value is
%{hovertext}. A numeric color_column is drawn with
color_continuous_scale, such as "Viridis", over color_range
when one is given, so colors mean the same in every export regardless of
the values present. color_label names the color bar or legend.
marker is applied to every trace's marker, such as
{"sizemode": "diameter", "sizemin": 3}, and layout is applied to the
figure layout last, overriding template and the default margins.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
class-attribute
instance-attribute
¶
AsyncPlotly3DExporter(config: ScatterPlotConfig)
Bases: AsyncExporter
Write a DataFrame as an interactive 3D scatter plot in a standalone HTML file.
Rows missing an x, y, or z value are left out, and nothing is written when no row remains.