Utilities¶
HTTP clients and rate limiters. Import these names from the module that defines them.
build_async_client(
total_retries: int = 5,
timeout: float = 25.0,
user_agent: str = DEFAULT_USER_AGENT,
) -> AsyncClient
Build an httpx.AsyncClient for the bundled extractors.
total_retries applies to connection failures only, at the transport
level; the extractors retry HTTP status failures themselves. Redirects are
followed, and timeout is in seconds. user_agent defaults to
sci-etl-core/<installed version>. The caller owns the client and
closes it with aclose.
SemaphoreRateLimiter(max_concurrency: int = 4)
Bases: AsyncRateLimiter
Bound concurrency to a fixed number of simultaneous slots.
The slot is released from a finally block so bookkeeping failures or a
cancellation delivered during exit can never permanently consume a slot.
HostRateLimiter(
limiters: Mapping[str, AsyncRateLimiter],
default: AsyncRateLimiter | None = None,
)
Pick a limiter by the host a request goes to.
A request's host is matched against the configured hosts from the most
specific name down, so "arxiv.org" also covers export.arxiv.org
unless that host has a limiter of its own. Host names match without regard
to case. A request to a host with no match uses default, which imposes
no limit when omitted. Pass one instance to several components to have
them share each host's budget.
Store the limiter for each host.
Raises:
| Type | Description |
|---|---|
ValueError
|
A host is blank, or two hosts differ only in case or surrounding dots. |
build_rate_limiter(
max_concurrency: int = 4,
max_rate: float | None = None,
time_period: float = 1.0,
) -> AsyncRateLimiter
Build a limiter, preferring a semaphore unless a rate cap is supplied.
limiter_for(
rate_limiter: RateLimiting | None, url: str
) -> AsyncRateLimiter
Resolve the limiter a component applies to a request for url.