Commands¶

Command	What it does	Network	Writes
`sci-etl init [DIRECTORY]`	Creates a starter project. Refuses to overwrite existing files unless given `--force`.	none	project files
`sci-etl validate CONFIG`	Checks the config, prompts, API key, and packages. `--online` adds one arXiv and one LLM request once the offline checks pass.	only with `--online`	nothing
`sci-etl search CONFIG`	Shows one page of arXiv results, each marked `new` or `processed`.	arXiv	nothing
`sci-etl run CONFIG`	Runs the pipeline, resuming from saved state, and logs token usage at the end.	arXiv and the LLM	export CSV, state, log
`sci-etl status CONFIG`	Shows processed records, the saved offset, the last run time, and export rows.	none	nothing
`sci-etl parse FILE`	Prints the text a bundled parser extracts from a PDF, arXiv e-print, or HTML file.	none	nothing

Every command accepts --help. Logs and errors go to stderr; search --json, status --json, and parse print their results to stdout, so they can be piped.

Running pipelines¶

Stopping a run. Press Ctrl+C once: records in flight are cancelled and left unmarked, state is saved, and the command exits with 130. The next run retries those records. A second Ctrl+C exits immediately.

Resuming. A run starts at the listing offset saved by the previous one. arXiv lists the newest submissions first, so new papers push older ones to higher offsets; --rescan starts again at offset 0 and skips processed papers by id, which costs listing requests but no LLM calls.

Throttling. When arXiv or the LLM provider answers 429 with a Retry-After header, the request waits as long as it asks, up to 60 seconds, and each retry is logged.

Token usage. When a run finishes, aborts, or is interrupted, run logs how many LLM requests it made and the prompt and completion tokens they used. Set llm.input_cost_per_million and llm.output_cost_per_million to your provider's prices to add an estimated cost, in the currency of those prices:

LLM usage: 42 requests, 1,204,388 prompt and 9,112 completion tokens, estimated cost 0.1861

Reference¶

This reference is generated from the command definitions, so it always matches sci-etl --help for this release.

sci-etl¶

Run sci-etl-core extraction pipelines from a YAML config file.

Usage:

sci-etl [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--version`	boolean	Show the CLI and library versions and exit.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`

Subcommands

init: Create a starter project.
parse: Print the text extracted from a PDF, e-print or HTML file.
run: Run the pipeline for CONFIG, resuming from its saved state.
search: Preview arXiv results without calling the LLM.
status: Show processed records, saved offset and export rows.
validate: Check CONFIG and its prompts without running the pipeline.

sci-etl init¶

Create a starter project in DIRECTORY (default: the current folder).

Writes config.yaml, prompts/relevance.txt, prompts/extraction.txt and .env.example. The example collects hot Jupiter measurements from arXiv; change the query, prompts and export columns for your own subject.

Usage:

sci-etl init [OPTIONS] [DIRECTORY]

Options:

Name	Type	Description	Default
`--force`	boolean	Overwrite files that already exist.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`

sci-etl parse¶

Print the text a bundled parser extracts from FILE.

Accepts a PDF, an arXiv e-print (a tarball or gzipped TeX file) or an HTML page. Use it to check full-text quality without running a pipeline.

Usage:

sci-etl parse [OPTIONS] FILE

Options:

Name	Type	Description	Default
`--format`	choice (`auto` \| `pdf` \| `latex` \| `html`)	How to read FILE. auto recognizes PDFs, arXiv e-prints and HTML.	`auto`
`--trim-references`	boolean	Cut the text at the references or acknowledgments.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`

sci-etl run¶

Run the pipeline for CONFIG, resuming from its saved state.

Searches arXiv, asks the LLM which papers are relevant, extracts entities from their full text and upserts them into the export CSV. Press Ctrl+C once to stop after saving state; the next run retries unfinished records.

Usage:

sci-etl run [OPTIONS] CONFIG

Options:

Name	Type	Description	Default
`--limit`	integer range (`0` and above)	Stop after this many relevant records. Overrides pipeline.total_limit.	`Sentinel.UNSET`
`--page-size`	integer range (`1` and above)	Listing entries per request. Overrides pipeline.page_size.	`Sentinel.UNSET`
`--workers`	integer range (`1` and above)	Records processed at once. Overrides pipeline.max_concurrency.	`Sentinel.UNSET`
`--rescan`	boolean	Start at offset 0 to pick up new submissions; processed records are skipped.	`Sentinel.UNSET`
`--start-index`	integer range (`0` and above)	Start at this listing offset instead of the saved one.	`Sentinel.UNSET`
`--log-file`	file	Write the log to this file. Overrides logging.file.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`

sci-etl search¶

Show one page of arXiv results for CONFIG without calling the LLM.

Each entry is marked new or processed according to the saved state, so a query can be tuned before spending tokens. Nothing is written.

Usage:

sci-etl search [OPTIONS] CONFIG

Options:

Name	Type	Description	Default
`--query`	text	Search this instead of pipeline.search_query.	`Sentinel.UNSET`
`--limit`	integer range (between `1` and `2000`)	Number of entries to request.	`10`
`--start-index`	integer range (`0` and above)	Listing offset to start from.	`0`
`--json`	boolean	Print JSON instead of a table.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`

sci-etl status¶

Show CONFIG's progress: processed records, saved offset, last run and export rows.

Reads state without creating it, so this is safe before the first run.

Usage:

sci-etl status [OPTIONS] CONFIG

Options:

Name	Type	Description	Default
`--json`	boolean	Print JSON instead of a table.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`

sci-etl validate¶

Check CONFIG and its prompts without running the pipeline.

Exits 3 when a configuration check fails, 4 when a required package is missing, and 1 when an online check fails. Online checks run only after every offline check passes.

Usage:

sci-etl validate [OPTIONS] CONFIG

Options:

Name	Type	Description	Default
`--online`	boolean	Also send one arXiv listing request and one LLM request.	`Sentinel.UNSET`
`-h`, `--help`	boolean	Show this message and exit.	`Sentinel.UNSET`