Extractors¶

Every name on this page is importable from sci_etl_core.extractors.

sci_etl_core.extractors.async_base ¶

AsyncExtractor ¶

Bases: ABC

Contract for a paged source of scientific records.

:class:~sci_etl_core.pipeline_async.AsyncETLPipeline calls :meth:fetch_page once per page and never concurrently, and :meth:fetch_full_text concurrently for the relevant records of that page. The pipeline skips records it has already processed, so an extractor returns every entry it can read.

fetch_page `abstractmethod` `async` ¶

fetch_page(
    query: str, cursor: str | None, page_size: int
) -> ListingPage

Fetch and parse one page; cursor=None is the first page.

cursor is a next_cursor an earlier page returned, possibly in an earlier run. page_size is how many entries to ask for; a source may return fewer.

Raises:

Type	Description
`UpstreamError`	The source could not be reached or answered with a server-side failure. Implementations must not collapse this into an empty page.
`MalformedResponseError`	The payload could not be interpreted.
`StaleCursorError`	The source no longer accepts `cursor`.
`ExtractionError`	The source rejected the request permanently.

fetch_full_text `abstractmethod` `async` ¶

fetch_full_text(record: RawRecord) -> str

Retrieve the best-available full text for a record.

OffsetListing ¶

Bases: Protocol

An extractor whose cursors are decimal offsets into the listing.

newest_first runs and run(start_index=) need it. The cursor for offset 0 is the first page, the same as None.

cursor_for_offset ¶

cursor_for_offset(offset: int) -> str

Return the cursor of the listing page that starts at offset.

sci_etl_core.extractors.arxiv_async ¶

AsyncArxivExtractor ¶

AsyncArxivExtractor(
    client: AsyncClient,
    pdf_parser: Parser,
    latex_parser: Parser,
    max_retries: int = 3,
    backoff_factor: float = 2.0,
    sleep_before_search: float = 3.0,
    sleep: Any = sleep,
    max_retry_after: float = 60.0,
    rate_limiter: RateLimiting | None = None,
    max_download_bytes: int | None = None,
)

Bases: AsyncExtractor

Page through the arXiv Atom API, newest submissions first, and fetch each article's full text.

Its cursors are decimal offsets, so it is an :class:~sci_etl_core.extractors.async_base.OffsetListing and supports newest_first runs. Full text comes from the e-print LaTeX source or the PDF, with the abstract as a last resort (:meth:fetch_full_text). Each record's metadata holds the entry's categories, authors, and publication date (:meth:fetch_page), which a text search store can filter and facet on. The injected client is borrowed and never closed.

Configure the extractor.

Transport faults, 408, 429, and server errors are retried, as by every bundled extractor, and so is 406, which arXiv's gateway returns intermittently for valid requests. Between attempts the extractor waits backoff_factor ** attempt seconds, or longer when arXiv's Retry-After header asks for it, up to max_retry_after seconds. Each retry is logged with its wait.

Every HTTP request, including each retry, first enters rate_limiter: an :class:~sci_etl_core.rate_limiter.AsyncRateLimiter for all requests, or a :class:~sci_etl_core.rate_limiter.HostRateLimiter to budget the API host (export.arxiv.org) and the full-text host (arxiv.org) apart. The slot is released once the response arrives, so no slot is held while waiting to retry.

With max_download_bytes, a response body is read only up to that many bytes, so a huge listing or e-print cannot exhaust memory. A larger listing page raises :class:~sci_etl_core.exceptions.ExtractionError; a larger e-print or PDF is logged and passed over like an unavailable one.

Raises:

Type	Description
`ValueError`	`max_retries` is less than 1, which would fail every request without making a single attempt, `max_retry_after` is negative, or `max_download_bytes` is less than 1.

API_URL `class-attribute` `instance-attribute` ¶

API_URL = 'https://export.arxiv.org/api/query'

from_config `classmethod` ¶

from_config(
    http: HttpConfig,
    pipeline: PipelineConfig | None = None,
    *,
    client: AsyncClient,
    pdf_parser: Parser,
    latex_parser: Parser,
    full_text: RateLimitConfig | None = None,
    **options: Any,
) -> AsyncArxivExtractor

Build an extractor from config sections.

http supplies max_retries and backoff_factor, pipeline supplies search_delay as sleep_before_search, and full_text supplies the rate_limiter every request enters, from :meth:~sci_etl_core.config.RateLimitConfig.build_limiter. Without full_text the extractor has no rate limiter, and a pipeline with max_concurrency 6 downloads six papers from arxiv.org at once. options pass any other constructor argument, such as rate_limiter, and override a value taken from the config.

cursor_for_offset ¶

cursor_for_offset(offset: int) -> str

Return the decimal cursor of the listing page at offset.

fetch_page `async` ¶

fetch_page(
    query: str, cursor: str | None, page_size: int
) -> ListingPage

Fetch and parse the listing page at the cursor's offset, retrying transient faults.

Redirects are followed, so a moved endpoint is not mistaken for a failure regardless of how the injected client was configured. The listing continues until a page has no entries.

An entry without an <id> cannot be tracked as processed, so it is skipped; it still counts toward the page's entries so paging advances. A later version of an entry already on the page is skipped too. Each record's metadata holds the entry's categories and authors as lists, and its published date and year as strings (see :meth:_listing_metadata).

Raises:

Type	Description
`UpstreamError`	Every attempt failed. A transport fault is never reported as an empty page.
`ExtractionError`	arXiv rejected the request with a status that retrying cannot fix, such as `400` for a malformed query.
`MalformedResponseError`	The payload is empty or lacks a feed root.
`StaleCursorError`	`cursor` is not a decimal offset.

fetch_full_text `async` ¶

fetch_full_text(record: RawRecord) -> str

Return the best-available full text, falling back to the abstract.

The LaTeX source is tried first, then the PDF. A source that is permanently unavailable (such as a 404) or whose payload its parser cannot read (such as a PDF-only submission served as the e-print) is passed over for the next one. The abstract is used only when no source yields text and none failed transiently. A transport failure is raised instead, so the record stays unmarked and is retried on the next run.

Raises:

Type	Description
`UpstreamError`	At least one source failed transiently and no source yielded usable text.

sci_etl_core.extractors.pubmed_async ¶

AsyncPubMedExtractor ¶

AsyncPubMedExtractor(
    client: AsyncClient,
    *,
    api_key: str | None = None,
    tool: str | None = None,
    email: str | None = None,
    sort: str | None = "pub_date",
    full_text_parser: Parser | None = None,
    max_retries: int = 3,
    backoff_factor: float = 2.0,
    max_retry_after: float = 60.0,
    sleep: Any = sleep,
    rate_limiter: RateLimiting | None = None,
    max_download_bytes: int | None = None,
)

Bases: AsyncExtractor

Page through PubMed search results through NCBI's E-utilities, newest first by default.

query is a PubMed search term, with the same syntax as the PubMed website. Its cursors are decimal offsets, so it is an :class:~sci_etl_core.extractors.async_base.OffsetListing. Each listing page costs two requests: esearch for the ids and efetch for their records. NCBI allows 3 requests per second without an api_key and 10 with one; pass a rate_limiter that stays under that, and identify your project with tool and email.

Each record's record_id is its PMID, and its metadata holds authors, categories (MeSH headings), published, year, journal, and doi and pmcid when known.

Full text is read from PubMed Central for a record with a PMC id, parsed as JATS, and falls back to the abstract when PMC has no body for it. The injected client is borrowed and never closed.

Configure the extractor.

full_text_parser reads PubMed Central's full text and defaults to :class:~sci_etl_core.parsers.jats.JatsXmlParser. E-utilities only pages through the first 9,999 results of a search, so the listing is truncated there.

With max_download_bytes, a response body is read only up to that many bytes, so a huge response cannot exhaust memory. A larger listing page raises :class:~sci_etl_core.exceptions.ExtractionError; a larger full-text download is logged and passed over like an unavailable one.

Raises:

Type	Description
`ValueError`	`max_retries` is less than 1, `max_retry_after` is negative, or `max_download_bytes` is less than 1.

cursor_for_offset ¶

cursor_for_offset(offset: int) -> str

Return the decimal cursor of the listing page at offset.

fetch_page `async` ¶

fetch_page(
    query: str, cursor: str | None, page_size: int
) -> ListingPage

Fetch the records at the cursor's offset: esearch for their ids, then efetch for the records.

The page's entries is the number of ids the search returned, so a record PubMed no longer serves still counts toward paging. The listing ends on the page that reaches the search's result count. E-utilities only serves the first 9,999 results of a search, so a page that reaches that cap, or a cursor past it, is truncated and ends the listing.

Raises:

Type	Description
`UpstreamError`	Every attempt failed transiently.
`ExtractionError`	NCBI rejected the request.
`MalformedResponseError`	The `esearch` or `efetch` response could not be read.
`StaleCursorError`	`cursor` is not a decimal offset.

fetch_full_text `async` ¶

fetch_full_text(record: RawRecord) -> str

Return the PubMed Central full text, or the abstract when there is none.

Raises:

Type	Description
`UpstreamError`	PubMed Central could not be reached; the record is retried on the next run.

sci_etl_core.extractors.semantic_scholar_async ¶

AsyncSemanticScholarExtractor ¶

AsyncSemanticScholarExtractor(
    client: AsyncClient,
    pdf_parser: Parser | None = None,
    *,
    api_key: str | None = None,
    year: str | None = None,
    fields_of_study: str | None = None,
    max_retries: int = 3,
    backoff_factor: float = 2.0,
    max_retry_after: float = 60.0,
    sleep: Any = sleep,
    rate_limiter: RateLimiting | None = None,
    max_download_bytes: int | None = None,
)

Bases: AsyncExtractor

Page through Semantic Scholar's relevance search for papers.

Its cursors are decimal offsets, so it is an :class:~sci_etl_core.extractors.async_base.OffsetListing.

Each record's record_id is the Semantic Scholar paper id, and its metadata holds authors, categories (fields of study), published, year, venue, doi, arxiv_id and pmid when known, and pdf_url for an open-access PDF.

Without an api_key, requests share Semantic Scholar's public rate limit, so pass a rate_limiter, for example one request per second. Full text comes from the open-access PDF when a pdf_parser is given, and the abstract otherwise. The injected client is borrowed and never closed.

Configure the extractor.

year and fields_of_study are passed as Semantic Scholar's filters of the same names, such as "2020-" and "Physics". The relevance search returns at most the first 1,000 results, 100 at a time, so the listing is truncated there.

With max_download_bytes, a response body is read only up to that many bytes, so a huge response cannot exhaust memory. A larger listing page raises :class:~sci_etl_core.exceptions.ExtractionError; a larger full-text download is logged and passed over like an unavailable one.

Raises:

Type	Description
`ValueError`	`max_retries` is less than 1, `max_retry_after` is negative, or `max_download_bytes` is less than 1.

API_URL `class-attribute` `instance-attribute` ¶

API_URL = (
    "https://api.semanticscholar.org/graph/v1/paper/search"
)

cursor_for_offset ¶

cursor_for_offset(offset: int) -> str

Return the decimal cursor of the listing page at offset.

fetch_page `async` ¶

fetch_page(
    query: str, cursor: str | None, page_size: int
) -> ListingPage

Fetch the papers at the cursor's offset, skipping papers without an id.

The listing ends on the page that reaches the search's total, or on a page with no papers. The relevance search serves only the first 1,000 results, so a page that reaches them, or a cursor past them, is truncated and ends the listing. A response without data, which Semantic Scholar sends when nothing matches, is an empty page.

Raises:

Type	Description
`UpstreamError`	Every attempt failed transiently.
`ExtractionError`	Semantic Scholar rejected the request.
`MalformedResponseError`	The payload is not a JSON object, or its `data` is not a list.
`StaleCursorError`	`cursor` is not a decimal offset.

fetch_full_text `async` ¶

fetch_full_text(record: RawRecord) -> str

Return the open-access PDF's text, or the abstract when there is no usable PDF.

Raises:

Type	Description
`UpstreamError`	The PDF download failed transiently.

sci_etl_core.extractors.openalex_async ¶

AsyncOpenAlexExtractor ¶

AsyncOpenAlexExtractor(
    client: AsyncClient,
    pdf_parser: Parser | None = None,
    *,
    filter: str | None = None,
    sort: str | None = "publication_date:desc",
    mailto: str | None = None,
    api_key: str | None = None,
    max_retries: int = 3,
    backoff_factor: float = 2.0,
    max_retry_after: float = 60.0,
    sleep: Any = sleep,
    rate_limiter: RateLimiting | None = None,
    max_download_bytes: int | None = None,
)

Bases: AsyncExtractor

Page through OpenAlex works matching a search, newest first by default.

query is OpenAlex's full-text search; filter narrows it with OpenAlex filter syntax, such as "type:article,from_publication_date:2020-01-01". Pass mailto to join OpenAlex's polite pool, and api_key when you have one. Paging uses OpenAlex's opaque cursors, so the extractor is not an :class:~sci_etl_core.extractors.async_base.OffsetListing and does not support newest_first runs; a finished listing starts again from its first page on the next run, skipping processed works by id.

Each record's record_id is the OpenAlex work id, such as "W2741809807", and its metadata holds authors, categories (the work's topics), published, year, doi, venue, and references, the ids of the works it cites, when OpenAlex has them.

Full text comes from the PDF of the best open-access location when a pdf_parser is given, and the abstract otherwise. The injected client is borrowed and never closed.

Configure the extractor.

OpenAlex serves at most 200 works per page, so a page asks for at most 200.

With max_download_bytes, a response body is read only up to that many bytes, so a huge response cannot exhaust memory. A larger listing page raises :class:~sci_etl_core.exceptions.ExtractionError; a larger full-text download is logged and passed over like an unavailable one.

Raises:

Type	Description
`ValueError`	`max_retries` is less than 1, `max_retry_after` is negative, or `max_download_bytes` is less than 1.

API_URL `class-attribute` `instance-attribute` ¶

API_URL = 'https://api.openalex.org/works'

fetch_page `async` ¶

fetch_page(
    query: str, cursor: str | None, page_size: int
) -> ListingPage

Fetch one page of works through OpenAlex's cursor paging, skipping works without an id.

cursor=None asks for the first page (cursor=*), and the page's next_cursor is the meta.next_cursor OpenAlex returned, so paging is not limited to the first 10,000 results. A page asks for at most 200 works. The listing ends on a page without works or without a next cursor.

Raises:

Type	Description
`UpstreamError`	Every attempt failed transiently.
`StaleCursorError`	OpenAlex rejected `cursor` as invalid, or `cursor` is a decimal listing offset, as sci-etl-core 0.4 saved.
`ExtractionError`	OpenAlex rejected the request, for example for an invalid filter.
`MalformedResponseError`	The payload is not JSON with a `results` list.

fetch_full_text `async` ¶

fetch_full_text(record: RawRecord) -> str

Return the open-access PDF's text, or the abstract when there is no usable PDF.

Raises:

Type	Description
`UpstreamError`	The PDF download failed transiently; the record is retried on the next run rather than settled on its abstract.

reconstruct_abstract ¶

reconstruct_abstract(inverted_index: Any) -> str

Rebuild an abstract from OpenAlex's abstract_inverted_index, which maps each word to its positions.

Extractors¶

sci_etl_core.extractors.async_base ¶

AsyncExtractor ¶

fetch_page abstractmethod async ¶

fetch_full_text abstractmethod async ¶

OffsetListing ¶

cursor_for_offset ¶

sci_etl_core.extractors.arxiv_async ¶

AsyncArxivExtractor ¶

API_URL class-attribute instance-attribute ¶

from_config classmethod ¶

cursor_for_offset ¶

fetch_page async ¶

fetch_full_text async ¶

sci_etl_core.extractors.pubmed_async ¶

AsyncPubMedExtractor ¶

cursor_for_offset ¶

fetch_page async ¶

fetch_full_text async ¶

sci_etl_core.extractors.semantic_scholar_async ¶

AsyncSemanticScholarExtractor ¶

API_URL class-attribute instance-attribute ¶

cursor_for_offset ¶

fetch_page async ¶

fetch_full_text async ¶

sci_etl_core.extractors.openalex_async ¶

AsyncOpenAlexExtractor ¶

API_URL class-attribute instance-attribute ¶

fetch_page async ¶

fetch_full_text async ¶

reconstruct_abstract ¶

fetch_page `abstractmethod` `async` ¶

fetch_full_text `abstractmethod` `async` ¶

API_URL `class-attribute` `instance-attribute` ¶

from_config `classmethod` ¶

fetch_page `async` ¶

fetch_full_text `async` ¶

fetch_page `async` ¶

fetch_full_text `async` ¶

API_URL `class-attribute` `instance-attribute` ¶

fetch_page `async` ¶

fetch_full_text `async` ¶

API_URL `class-attribute` `instance-attribute` ¶

fetch_page `async` ¶

fetch_full_text `async` ¶