Parsers¶
Every name on this page is importable from sci_etl_core.parsers.
Bases: ABC
Contract for turning one document format into plain text.
Parsers are synchronous and CPU-bound; async callers run them in a worker thread.
abstractmethod
¶
Extract plain text from raw document bytes.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The bytes are not a document this parser can read.
Callers such as :class: |
Bases: Parser, TableParser
Read PDF text and tables with pdfplumber. Needs the pdf extra.
Return the text of every page, followed by any tables found.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The payload is not a readable PDF. |
AsyncPdfPlumberParser(
sync_parser: PdfPlumberParser | None = None,
)
Run a :class:~sci_etl_core.parsers.pdf.PdfPlumberParser in a worker thread, off the event loop.
Wrap sync_parser, or a new PdfPlumberParser when it is None.
async
¶
Return the PDF's text, as :meth:PdfPlumberParser.extract_text does.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The payload is not a readable PDF. |
Bases: Parser
Extract TeX source from an arXiv e-print.
arXiv serves a multi-file submission as a tarball and a single-file
submission as one gzipped .tex file; both are accepted. Line comments
are removed from each file before anything else, so commented-out
\input commands are ignored.
In a tarball, each root document (a file declaring \documentclass) is
expanded in place by following \input, \include and \subfile,
so the body reads in document order and a \bibliography command at the
end of the root cannot land before the sections it includes. .tex files
that no root includes are appended afterwards in name order.
Return the submission's TeX source as text.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The payload is neither a tarball nor gzipped TeX, for example a PDF-only submission. |
Bases: Parser
Extract text from a Word .docx document using only the standard library and lxml.
The body is read in document order: each paragraph on its own line, and
each table row as its cells joined by tabs. Tabs and line breaks inside a
paragraph are kept. Deleted revisions and field codes are left out, while
inserted revisions and hyperlink text are kept. Headers and footers are
not read. With include_notes, footnotes and endnotes follow the body.
The XML is parsed without resolving entities or loading DTDs, and a part
that decompresses to more than max_part_bytes is refused, so a hostile
file cannot exhaust memory.
Configure the parser.
Raises:
| Type | Description |
|---|---|
ValueError
|
|
Return the document's text.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The bytes are not a zip archive, the archive has no
|
dataclass
¶
dataclass
¶
JatsArticle(
title: str,
abstract: str,
sections: tuple[JatsSection, ...] = (),
authors: tuple[str, ...] = (),
keywords: tuple[str, ...] = (),
journal: str = "",
published: str | None = None,
identifiers: dict[str, str] = dict(),
references: tuple[str, ...] = (),
)
The parts of a JATS article that text mining uses.
published is an ISO 8601 date of the earliest <pub-date> given, as
precise as the source: "2024", "2024-03", or "2024-03-07".
identifiers maps each <article-id> type, such as doi,
pmid, or pmcid, to its value. references holds the text of each
reference in the reference list, in order.
class-attribute
instance-attribute
¶
JatsXmlParser(include_tables: bool = True)
Bases: Parser
Parse JATS XML, the format of PubMed Central and many publishers' full texts.
:meth:parse_article returns a :class:JatsArticle. :meth:extract_text
returns the title, abstract, and body with section headings, leaving out
the reference list, footnotes, and graphics, so no reference trimming is
needed. Table cells are kept, one row per line with cells joined by tabs,
unless include_tables is false; captions are always kept.
An <article> may be the document root or wrapped, as in a PubMed
Central <pmc-articleset>; the first one is read. Namespaces are
ignored. The XML is parsed without resolving entities, loading DTDs, or
using the network.
Return the article's title, abstract, and body as text.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The bytes are not well-formed XML or hold no |
parse_article(content: bytes) -> JatsArticle
Return the first article in content.
Raises:
| Type | Description |
|---|---|
ParsingError
|
The bytes are not well-formed XML or hold no |
module-attribute
¶
DEFAULT_TRIM_PATTERNS: tuple[str, ...] = (
"\\\\begin\\{thebibliography\\}",
"\\\\bibliography\\{",
"\\\\printbibliography",
"\\n\\s*references\\s*\\n",
"\\n\\s*bibliography\\s*\\n",
"\\n\\s*acknowledgments?\\s*\\n",
"\\n\\s*literature cited\\s*\\n",
)
trim_after_references(
text: str | None,
patterns: tuple[str, ...] = DEFAULT_TRIM_PATTERNS,
) -> str | None
Cut text at the earliest match of any pattern, such as a references or acknowledgments heading.
Patterns are regular expressions matched without regard to case. The kept
text has trailing whitespace removed. text is returned unchanged when
no pattern matches, and None or "" is returned as given.