Configuration¶
Typed settings loaded from YAML and .env. The models and loaders are importable from sci_etl_core.
Bases: BaseModel
Chat-completion endpoint settings.
:meth:~sci_etl_core.llm.openai_compatible_async.AsyncOpenAICompatibleClient.from_config
builds a client from them. api_key is a :class:~pydantic.SecretStr, so it never appears in a
repr or a log line. :func:load_config takes it from the environment
variable it names, and falls back to the YAML value when that is unset.
timeout is in seconds.
Bases: BaseModel
HTTP settings for the clients and extractors that call a remote source.
user_agent defaults to sci-etl-core/<installed version>.
class-attribute
instance-attribute
¶
backoff_factor: float = Field(default=2.0, ge=0)
Build an httpx.AsyncClient with this timeout and user_agent.
max_retries and backoff_factor are applied by the extractor, as
:meth:~sci_etl_core.extractors.arxiv_async.AsyncArxivExtractor.from_config
does, not by the client. Needs the async extra.
Bases: BaseModel
A concurrency cap, or a token bucket when max_rate is set.
build_limiter() -> AsyncRateLimiter
Build the limiter these settings describe, as :func:~sci_etl_core.rate_limiter.build_rate_limiter does.
Bases: BaseModel
Settings for :class:~sci_etl_core.pipeline_async.AsyncETLPipeline and its runs.
max_concurrency configures the pipeline (see from_config), and
:meth:run_arguments returns the arguments for run().
search_delay is the arXiv extractor's pause before each listing
request.
.. deprecated:: 0.4.0
The keys max_records and max_workers are read as
total_limit and max_concurrency, with a
:class:DeprecationWarning. They will stop being accepted in 0.5.0.
Bases: BaseModel
Per-field BM25 weights, as :class:~sci_etl_core.search.store_base.BM25Weights.
class-attribute
instance-attribute
¶
title: float = Field(
default=10.0, ge=0, allow_inf_nan=False
)
class-attribute
instance-attribute
¶
abstract: float = Field(
default=4.0, ge=0, allow_inf_nan=False
)
Bases: BaseModel
Rank fusion settings, as :class:~sci_etl_core.search.fusion.FusionParams.
to_params() -> FusionParams
Build the fusion parameters.
Raises:
| Type | Description |
|---|---|
ValueError
|
A weight is negative or not finite. |
Bases: BaseModel
Candidate pool sizes, as :class:~sci_etl_core.search.hybrid_async.HybridParams.
Bases: BaseModel
Settings for local search and discovery graphs.
class-attribute
instance-attribute
¶
bm25: BM25WeightsConfig = Field(
default_factory=BM25WeightsConfig
)
class-attribute
instance-attribute
¶
fusion: FusionConfig = Field(default_factory=FusionConfig)
class-attribute
instance-attribute
¶
hybrid: HybridConfig = Field(default_factory=HybridConfig)
Bases: BaseModel
Root of an application config: the library's sections, plus any keys a subclass adds.
Unknown top-level keys are kept rather than rejected, so an application can
read its own sections from the same YAML file. Subclass it to type those
sections, and set extra="forbid" in the subclass to reject typos.
class-attribute
instance-attribute
¶
full_text: RateLimitConfig = Field(
default_factory=RateLimitConfig
)
class-attribute
instance-attribute
¶
pipeline: PipelineConfig = Field(
default_factory=PipelineConfig
)
class-attribute
instance-attribute
¶
search: SearchConfig = Field(default_factory=SearchConfig)
Read a YAML config file.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
The file is missing, is not valid YAML, or does not hold a mapping at the top level. |
Parse YAML text that must hold a mapping; an empty document is {}.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
The text is not valid YAML or its top level is not a mapping. |
Resolve the LLM API key, letting the environment override the YAML file.
A key supplied at runtime through api_key_env_var always wins, so a
stale or leaked key written into a config file can never silently replace
it. The YAML value is only a fallback for when the variable is unset or
empty. A non-mapping llm section is left for validation to reject.
load_config(
config_cls: type[T],
yaml_path: Path,
env_path: Path | None = None,
api_key_env_var: str = "LLM_API_KEY",
) -> T
Load and validate a config from YAML plus a .env file.
Without env_path, .env is looked up from the current working
directory upward, so the caller's project is searched rather than the
location the library is installed in.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
The YAML file is missing or unreadable, or the configuration fails validation. |
Validate settings read from source against config_cls.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
Validation failed. The message names each failing key and the reason but never the value that was read, so a secret such as an API key taken from the environment cannot reach a log or a terminal. The validation error is not chained for the same reason. |
async
¶
Read a YAML config file without blocking the event loop.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
The file is missing, is not valid YAML, or does not hold a mapping at the top level. |