Skip to content

Utilities

HTTP clients, rate limiters, and logging. Import these names from the module that defines them.

sci_etl_core.http_async

build_async_client 

build_async_client(
    total_retries: int = 5,
    timeout: float = 25.0,
    user_agent: str = DEFAULT_USER_AGENT,
) -> AsyncClient

Build an httpx.AsyncClient for the bundled extractors.

total_retries applies to connection failures only, at the transport level; the extractors retry HTTP status failures themselves. Redirects are followed, and timeout is in seconds. user_agent defaults to sci-etl-core/<installed version>. The caller owns the client and closes it with aclose.

sci_etl_core.http

build_retrying_session 

build_retrying_session(
    total_retries: int = 5,
    backoff_factor: float = 2.0,
    status_forcelist: tuple[int, ...] = (
        429,
        500,
        502,
        503,
        504,
    ),
) -> Session

Build a synchronous requests session that retries transient faults.

.. deprecated:: 0.4.0 No component uses this helper. It will be removed in 0.5.0, together with requests in the full extra. Use :func:sci_etl_core.http_async.build_async_client with the async components instead.

sci_etl_core.rate_limiter

RateLimiting module-attribute 

RateLimiting: TypeAlias = AsyncRateLimiter | HostRateLimiter

AsyncRateLimiter

Bases: ABC

Async context manager that gates concurrent access to a resource.

NullRateLimiter

Bases: AsyncRateLimiter

No-op limiter that imposes no concurrency or rate constraint.

SemaphoreRateLimiter 

SemaphoreRateLimiter(max_concurrency: int = 4)

Bases: AsyncRateLimiter

Bound concurrency to a fixed number of simultaneous slots.

The slot is released from a finally block so bookkeeping failures or a cancellation delivered during exit can never permanently consume a slot.

held property 

held: int

Number of slots currently checked out.

AioLimiterRateLimiter 

AioLimiterRateLimiter(
    max_rate: float, time_period: float = 1.0
)

Bases: AsyncRateLimiter

Token-bucket limiter backed by the optional aiolimiter package.

HostRateLimiter 

HostRateLimiter(
    limiters: Mapping[str, AsyncRateLimiter],
    default: AsyncRateLimiter | None = None,
)

Pick a limiter by the host a request goes to.

A request's host is matched against the configured hosts from the most specific name down, so "arxiv.org" also covers export.arxiv.org unless that host has a limiter of its own. Host names match without regard to case. A request to a host with no match uses default, which imposes no limit when omitted. Pass one instance to several components to have them share each host's budget.

Store the limiter for each host.

Raises:

Type Description
ValueError

A host is blank, or two hosts differ only in case or surrounding dots.

for_url 

for_url(url: str) -> AsyncRateLimiter

Return the limiter governing requests to url.

build_rate_limiter 

build_rate_limiter(
    max_concurrency: int = 4,
    max_rate: float | None = None,
    time_period: float = 1.0,
) -> AsyncRateLimiter

Build a limiter, preferring a semaphore unless a rate cap is supplied.

limiter_for 

limiter_for(
    rate_limiter: RateLimiting | None, url: str
) -> AsyncRateLimiter

Resolve the limiter a component applies to a request for url.

sci_etl_core.log_utils

configure_logging 

configure_logging(
    name: str, log_file: str | Path, level: int = INFO
) -> Logger

Return a logger that writes to log_file and to stdout.

A repeated call with the same file and level returns the logger unchanged. A call with a different file or level replaces the handlers this function installed earlier, closing the old file, so the logger always reflects the latest request. The log file's folder is created when it does not exist.