Commands¶
| Command | What it does | Network | Writes |
|---|---|---|---|
sci-etl init [DIRECTORY] |
Creates a starter project. Refuses to overwrite existing files unless given --force. |
none | project files |
sci-etl validate CONFIG |
Checks the config, prompts, API key, and packages. --online adds one arXiv and one LLM request once the offline checks pass. |
only with --online |
nothing |
sci-etl search CONFIG |
Shows one page of arXiv results, each marked new or processed. |
arXiv | nothing |
sci-etl run CONFIG |
Runs the pipeline, resuming from saved state, and logs token usage at the end. | arXiv and the LLM | export CSV, state, log |
sci-etl status CONFIG |
Shows processed records, the saved offset, the last run time, and export rows. | none | nothing |
sci-etl parse FILE |
Prints the text a bundled parser extracts from a PDF, arXiv e-print, or HTML file. | none | nothing |
Every command accepts --help. Logs and errors go to stderr; search --json,
status --json, and parse print their results to stdout, so they can be
piped.
Running pipelines¶
Stopping a run. Press Ctrl+C once: no new paper starts, the papers in flight finish and are saved, state is saved, and the command exits with 130. The next run continues with the rest. A second Ctrl+C exits immediately.
Resuming. A run starts at the listing offset saved by the previous one.
arXiv lists the newest submissions first, so new papers push older ones to
higher offsets. Set pipeline.newest_first: true to have every run look for
new submissions at the head of the listing first and then continue from the
saved offset. Or pass --rescan to start once at offset 0; processed papers
are skipped by id, which costs listing requests but no LLM calls. --rescan
and --start-index cannot be combined with newest_first.
Papers that keep failing. A paper that failed in pipeline.max_attempts
runs (3 by default) is skipped and logged once per run. Set it to null to
retry such papers forever.
Caching. LLM answers are kept in llm.cache (state/llm_cache.db by
default), so a paper retried after a crash or a rescan costs no tokens for the
questions already answered. Answers the library rejects are not kept. Set
llm.cache: null to turn the cache off.
Throttling. When arXiv or the LLM provider answers 429 with a
Retry-After header, the request waits as long as it asks, up to 60 seconds,
and each retry is logged.
Token usage. When a run finishes, aborts, or is interrupted, run logs how
many LLM requests it made and the prompt and completion tokens they used. Set
llm.input_cost_per_million and llm.output_cost_per_million to your
provider's prices to add an estimated cost, in the currency of those prices:
Reference¶
This reference is generated from the command definitions, so it always matches
sci-etl --help for this release.
sci-etl¶
Run sci-etl-core extraction pipelines from a YAML config file.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--version |
boolean | Show the CLI and library versions and exit. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |
Subcommands
- init: Create a starter project.
- parse: Print the text extracted from a PDF, e-print or HTML file.
- run: Run the pipeline for CONFIG, resuming from its saved state.
- search: Preview arXiv results without calling the LLM.
- status: Show processed records, saved offset and export rows.
- validate: Check CONFIG and its prompts without running the pipeline.
sci-etl init¶
Create a starter project in DIRECTORY (default: the current folder).
Writes config.yaml, prompts/relevance.txt, prompts/extraction.txt and .env.example. The example collects hot Jupiter measurements from arXiv; change the query, prompts and export columns for your own subject.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--force |
boolean | Overwrite files that already exist. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |
sci-etl parse¶
Print the text a bundled parser extracts from FILE.
Accepts a PDF, an arXiv e-print (a tarball or gzipped TeX file) or an HTML page. Use it to check full-text quality without running a pipeline.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--format |
choice (auto | pdf | latex | html) |
How to read FILE. auto recognizes PDFs, arXiv e-prints and HTML. | auto |
--trim-references |
boolean | Cut the text at the references or acknowledgments. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |
sci-etl run¶
Run the pipeline for CONFIG, resuming from its saved state.
Searches arXiv, asks the LLM which papers are relevant, extracts entities from their full text, and writes them to the export CSV, one row per entity tagged with its paper's arXiv id. Press Ctrl+C once to let the papers in progress finish and save state; the next run continues with the rest. Press it again to stop at once.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--limit |
integer range (0 and above) |
Stop after this many relevant records. Overrides pipeline.total_limit. | Sentinel.UNSET |
--page-size |
integer range (1 and above) |
Listing entries per request. Overrides pipeline.page_size. | Sentinel.UNSET |
--workers |
integer range (1 and above) |
Records processed at once. Overrides pipeline.max_concurrency. | Sentinel.UNSET |
--rescan |
boolean | Start at offset 0 to pick up new submissions; processed records are skipped. | Sentinel.UNSET |
--start-index |
integer range (0 and above) |
Start at this listing offset instead of the saved one. | Sentinel.UNSET |
--log-file |
file | Write the log to this file. Overrides logging.file. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |
sci-etl search¶
Show one page of arXiv results for CONFIG without calling the LLM.
Each entry is marked new or processed according to the saved state, so a query can be tuned before spending tokens. Nothing is written.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--query |
text | Search this instead of pipeline.search_query. | Sentinel.UNSET |
--limit |
integer range (between 1 and 2000) |
Number of entries to request. | 10 |
--start-index |
integer range (0 and above) |
Listing offset to start from. | 0 |
--json |
boolean | Print JSON instead of a table. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |
sci-etl status¶
Show CONFIG's progress: processed records, saved offset, last run and export rows.
Reads state without creating it, so this is safe before the first run.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--json |
boolean | Print JSON instead of a table. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |
sci-etl validate¶
Check CONFIG and its prompts without running the pipeline.
Exits 3 when a configuration check fails, 4 when a required package is missing, and 1 when an online check fails. Online checks run only after every offline check passes.
Usage:
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--online |
boolean | Also send one arXiv listing request and one LLM request. | Sentinel.UNSET |
-h, --help |
boolean | Show this message and exit. | Sentinel.UNSET |