Extractors¶
Every name on this page is importable from sci_etl_core.extractors.
Bases: ABC
Contract for a paged source of scientific records.
:class:~sci_etl_core.pipeline_async.AsyncETLPipeline calls :meth:search
once per page and never concurrently, :meth:parse_listing on the payload
it returned, and :meth:fetch_full_text concurrently for the relevant
records of that page.
abstractmethod
async
¶
Fetch a raw listing page from the source.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
The source could not be reached or answered with a server-side failure. Implementations must not collapse this into a falsy return value. |
ExtractionError
|
The source rejected the request permanently. The
pipeline aborts on any |
abstractmethod
¶
Parse a raw listing into records, skipping already-seen ids.
Returns an empty result only when the payload is a valid listing that genuinely contains no entries.
Raises:
| Type | Description |
|---|---|
MalformedResponseError
|
The payload could not be interpreted. |
AsyncArxivExtractor(
client: AsyncClient,
pdf_parser: Parser,
latex_parser: Parser,
max_retries: int = 3,
backoff_factor: float = 2.0,
sleep_before_search: float = 3.0,
logger: Callable[[str], None] | None = None,
sleep: Any = sleep,
max_retry_after: float = 60.0,
rate_limiter: RateLimiting | None = None,
)
Bases: AsyncExtractor
Page through the arXiv Atom API, newest submissions first, and fetch each article's full text.
Full text comes from the e-print LaTeX source or the PDF, with the abstract
as a last resort (:meth:fetch_full_text). Each record's metadata holds
the entry's categories, authors, and publication date
(:meth:parse_listing), which a text search store can filter and facet on.
The injected client is borrowed and never closed.
Configure the extractor.
Between attempts the extractor waits backoff_factor ** attempt
seconds, or longer when arXiv's Retry-After header asks for it, up
to max_retry_after seconds. Each retry is logged with its wait.
Every HTTP request, including each retry, first enters rate_limiter:
an :class:~sci_etl_core.rate_limiter.AsyncRateLimiter for all
requests, or a :class:~sci_etl_core.rate_limiter.HostRateLimiter to
budget the API host (export.arxiv.org) and the full-text host
(arxiv.org) apart. The slot is released once the response arrives,
so no slot is held while waiting to retry.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
classmethod
¶
from_config(
http: HttpConfig,
pipeline: PipelineConfig | None = None,
*,
client: AsyncClient,
pdf_parser: Parser,
latex_parser: Parser,
**options: Any,
) -> AsyncArxivExtractor
Build an extractor from config sections.
http supplies max_retries and backoff_factor, and
pipeline supplies search_delay as sleep_before_search.
options pass any other constructor argument, such as logger or
rate_limiter, and override a value taken from the config.
async
¶
Fetch one listing page, retrying transient faults.
Redirects are followed, so a moved endpoint is not mistaken for a failure regardless of how the injected client was configured.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
Every attempt failed. A transport fault is never reported as an empty result. |
ExtractionError
|
arXiv rejected the request with a status that
retrying cannot fix, such as |
Parse an Atom listing, treating an unreadable payload as an error.
An entry without an <id> cannot be tracked as processed, so it is
skipped; it still counts toward the page total so paging advances.
Each record's metadata holds the entry's categories and
authors as lists, and its published date and year as strings
(see :meth:_listing_metadata).
Raises:
| Type | Description |
|---|---|
MalformedResponseError
|
The payload is empty or lacks a feed root. |
async
¶
Return the best-available full text, falling back to the abstract.
The LaTeX source is tried first, then the PDF. A source that is
permanently unavailable (such as a 404) or whose payload its parser
cannot read (such as a PDF-only submission served as the e-print) is
passed over for the next one. The abstract is used only when no source
yields text and none failed transiently. A transport failure is raised
instead, so the record stays unmarked and is retried on the next run.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
At least one source failed transiently and no source yielded usable text. |
AsyncPubMedExtractor(
client: AsyncClient,
*,
api_key: str | None = None,
tool: str | None = None,
email: str | None = None,
sort: str | None = "pub_date",
full_text_parser: Parser | None = None,
max_retries: int = 3,
backoff_factor: float = 2.0,
max_retry_after: float = 60.0,
sleep: Any = sleep,
logger: Callable[[str], None] | None = None,
rate_limiter: RateLimiting | None = None,
)
Bases: AsyncExtractor
Page through PubMed search results through NCBI's E-utilities, newest first by default.
query is a PubMed search term, with the same syntax as the PubMed
website. Each listing page costs two requests: esearch for the ids and
efetch for their records. NCBI allows 3 requests per second without an
api_key and 10 with one; pass a rate_limiter that stays under that,
and identify your project with tool and email.
Each record's record_id is its PMID, and its metadata holds
authors, categories (MeSH headings), published, year,
journal, and doi and pmcid when known.
Full text is read from PubMed Central for a record with a PMC id, parsed
as JATS, and falls back to the abstract when PMC has no body for it. The
injected client is borrowed and never closed.
Configure the extractor.
full_text_parser reads PubMed Central's full text and defaults to
:class:~sci_etl_core.parsers.jats.JatsXmlParser. E-utilities only
pages through the first 10,000 results of a search, so search
returns an empty listing beyond them.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
async
¶
Fetch the records at offsets start_index to start_index + max_results.
The page is returned as a <pubmed-listing> element whose
entries attribute is the number of ids the search returned, wrapped
around the efetch response, so a record PubMed no longer serves
still counts toward paging.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
Every attempt failed transiently. |
ExtractionError
|
NCBI rejected the request. |
MalformedResponseError
|
The |
Parse a page returned by :meth:search.
Raises:
| Type | Description |
|---|---|
MalformedResponseError
|
The payload is not a |
async
¶
Return the PubMed Central full text, or the abstract when there is none.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
PubMed Central could not be reached; the record is retried on the next run. |
AsyncSemanticScholarExtractor(
client: AsyncClient,
pdf_parser: Parser | None = None,
*,
api_key: str | None = None,
year: str | None = None,
fields_of_study: str | None = None,
max_retries: int = 3,
backoff_factor: float = 2.0,
max_retry_after: float = 60.0,
sleep: Any = sleep,
logger: Callable[[str], None] | None = None,
rate_limiter: RateLimiting | None = None,
)
Bases: AsyncExtractor
Page through Semantic Scholar's relevance search for papers.
Each record's record_id is the Semantic Scholar paper id, and its
metadata holds authors, categories (fields of study),
published, year, venue, doi, arxiv_id and pmid when
known, and pdf_url for an open-access PDF.
Without an api_key, requests share Semantic Scholar's public rate
limit, so pass a rate_limiter, for example one request per second.
Full text comes from the open-access PDF when a pdf_parser is given,
and the abstract otherwise. The injected client is borrowed and never
closed.
Configure the extractor.
year and fields_of_study are passed as Semantic Scholar's
filters of the same names, such as "2020-" and "Physics". The
relevance search returns at most the first 1,000 results, 100 at a
time, so search returns an empty listing beyond them.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
class-attribute
instance-attribute
¶
async
¶
Fetch the papers at offsets start_index to start_index + max_results.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
Every attempt failed transiently. |
ExtractionError
|
Semantic Scholar rejected the request. |
Parse a listing page, skipping papers already seen or without an id.
A response without data, which Semantic Scholar sends when nothing
matches, is an empty listing.
Raises:
| Type | Description |
|---|---|
MalformedResponseError
|
The payload is not a JSON object, or its
|
async
¶
Return the open-access PDF's text, or the abstract when there is no usable PDF.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
The PDF download failed transiently. |
AsyncOpenAlexExtractor(
client: AsyncClient,
pdf_parser: Parser | None = None,
*,
filter: str | None = None,
sort: str | None = "publication_date:desc",
mailto: str | None = None,
api_key: str | None = None,
max_retries: int = 3,
backoff_factor: float = 2.0,
max_retry_after: float = 60.0,
sleep: Any = sleep,
logger: Callable[[str], None] | None = None,
rate_limiter: RateLimiting | None = None,
)
Bases: AsyncExtractor
Page through OpenAlex works matching a search, newest first by default.
query is OpenAlex's full-text search; filter narrows it with
OpenAlex filter syntax, such as "type:article,from_publication_date:2020-01-01".
Pass mailto to join OpenAlex's polite pool, and api_key when you
have one.
Each record's record_id is the OpenAlex work id, such as
"W2741809807", and its metadata holds authors, categories
(the work's topics), published, year, doi, venue, and
references, the ids of the works it cites, when OpenAlex has them.
Full text comes from the PDF of the best open-access location when a
pdf_parser is given, and the abstract otherwise. The injected
client is borrowed and never closed.
Configure the extractor.
OpenAlex pages at most 200 works and only the first 10,000 results of
a search through page numbers, so search asks for at most 200 at a
time and returns an empty listing beyond 10,000.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
async
¶
Fetch the works at offsets start_index to start_index + max_results.
OpenAlex pages by page number, so the page holding start_index is
requested and the works before start_index on it are dropped.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
Every attempt failed transiently. |
ExtractionError
|
OpenAlex rejected the request, for example for an invalid filter. |
Parse a listing page, skipping works already seen or without an id.
Raises:
| Type | Description |
|---|---|
MalformedResponseError
|
The payload is not JSON with a |
async
¶
Return the open-access PDF's text, or the abstract when there is no usable PDF.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
The PDF download failed transiently; the record is retried on the next run rather than settled on its abstract. |
Bases: ABC
Blocking counterpart of :class:~sci_etl_core.extractors.async_base.AsyncExtractor.
Wrap an implementation in
:class:~sci_etl_core._adapters.SyncExtractorAdapter to use it in a
pipeline.
abstractmethod
¶
Fetch a raw listing page from the source.
Raises:
| Type | Description |
|---|---|
UpstreamError
|
The source could not be reached or answered with a server-side failure. |
abstractmethod
¶
Parse a raw listing into records, skipping already-seen ids.
Raises:
| Type | Description |
|---|---|
MalformedResponseError
|
The payload could not be interpreted. |