Skip to content

Entity Resolution (fullmap)

The fullmap module resolves free-text strings to standardized biological CURIEs against the embedded redb database — call resolve() for low-level, LazyFrame-based entity resolution inside a pipeline.

resolve()

Primary entity resolution function, querying the embedded fullmap redb database.

Function Signature

def resolve(
  lf: pl.LazyFrame,
  col: str,
  db: Path,
  taxon: Optional[str] = None,
  prioritize: Optional[list[Categories]] = None,
  avoid: Optional[list[Categories]] = None,
  log: bool = True,
  section_hash: Optional[str] = None,
  config_file: Optional[str] = None,
  column_context: bool = True,
  tag: str = "_two",
  threads: Optional[int] = None,
) -> pl.LazyFrame

Parameters

lf: pl.LazyFrame

Input LazyFrame containing the data to process. Internally collected at explicit collection points for redb lookups and joins.

col: str

Column name containing text strings to resolve.

db: Path

Path to the fullmap redb file (already resolved — see fullmap_db_path() and Fullmap).

taxon: Optional[str]

Optional NCBI Taxon ID for filtering results.

Example: "9606" filters to human-specific entities.

prioritize: Optional[list[Categories]]

Optional list of Biolink categories to prefer when multiple matches exist.

Example: [Categories.GENE, Categories.PROTEIN] prefers gene/protein mappings.

avoid: Optional[list[Categories]]

Optional list of Biolink categories to exclude from results.

Example: [Categories.GENE] prevents gene mappings.

log: bool (default: True)

Controls unmatched-value logging. When enabled, unresolved terms are logged with section/config/column context.

section_hash: Optional[str] / config_file: Optional[str]

Optional context fields used for operational logging when unmatched values are encountered.

column_context: bool (default: True)

Controls category-frequency tie-breaking when multiple matches exist for a term. When True, the query result adds a category frequency score and prefers more frequent category hits.

tag: str (default: "_two")

Suffix appended to col to locate the level_two output column.

resolve() expects the LazyFrame to already have two NLP columns applied upstream: - col — the level_one output (whitespace stripped, lowercased) - col + tag — the level_two output (non-word characters removed via \W+)

The default "_two" matches level_two's default tag.

threads: Optional[int] (default: None)

Optional worker-thread count passed through to the Rust lookup for parallel term batching.

Return Value

Returns a Polars LazyFrame with these columns added:

Column Description Example
{col} CURIE identifier "HGNC:11998"
{col}_name Preferred entity name "TP53"
{col}_category Biolink category "biolink:Gene"
{col}_taxon NCBI Taxon ID "NCBITaxon:9606"
{col}_source Source database "HGNC"
{col}_source_version Database version "2025-01"
{col}_nlp_level NLP processing level 1 or 2

Lookup Pipeline

The function:

  1. Builds an in-memory term table by collecting terms from both NLP levels and deduplicating by keeping first occurrences for deterministic ordering, then looks them up against the redb records table via the Rust lookup_fullmap_terms() extension function.

  2. Ranks matches by:

  3. Category priority (if prioritize specified)
  4. Preferred-name exactness (case-insensitive exact match of normalized term to preferred name)
  5. NLP level (exact case match preferred over normalized)

  6. Category frequency (if column_context=True)

  7. Filters by:

  8. Taxon ID (if specified)

  9. Category avoidance (if specified)

  10. Deduplicates to one CURIE per input string

Example Usage

from pathlib import Path
from tablassert.fullmap import resolve
from tablassert.biolink import Categories
import polars as pl

# Path to the fullmap redb file
db = Path("/path/to/fullmap/data/fullmap.redb")

# LazyFrame with data to resolve
lf = pl.scan_parquet("data.parquet")

# Resolve gene symbols to CURIEs
result = resolve(
  lf=lf,
  col="gene_symbol",
  db=db,
  taxon="9606",  # Human only
  prioritize=[Categories.GENE],
  avoid=[Categories.PROTEIN],
  log=True,
  section_hash="tutorial-section",
  config_file="tutorial-table.yaml",
  column_context=True,
)

# Result LazyFrame includes:
# - gene_symbol: "HGNC:11998"
# - gene_symbol_name: "TP53"
# - gene_symbol_category: "biolink:Gene"
# - etc.

Mapping a Python List

Resolve a plain Python list by building a LazyFrame and applying the NLP levels before resolve():

import polars as pl
from pathlib import Path
from tablassert.fullmap import resolve
from tablassert.nlp import level_one, level_two
from tablassert.biolink import Categories

db = Path("/path/to/fullmap/data/fullmap.redb")
lf = pl.LazyFrame({"gene": ["TP53", "BRCA1", "EGFR", "KRAS"]})
lf = level_one(lf, "gene")   # lowercase + strip
lf = level_two(lf, "gene")   # remove non-word chars → "gene_two" column

result = resolve(lf=lf, col="gene", db=db, taxon="9606",
                 prioritize=[Categories.GENE], log=False).collect()
print(result.select(["gene", "gene_name", "gene_category"]))

NLP Processing Levels

resolve() requires that level_one and level_two have been applied to the LazyFrame before calling it:

level_one output (column: col): - Whitespace stripped, lowercased - Queried first; preferred for acronyms and gene symbols

level_two output (column: col + "_two"): - All non-word characters removed (\W+ → "") from the level_one result - Used as fallback when level_one produces no match - Preferred for disease names and free text

Rows without a valid CURIE are filtered from the returned frame.

Provenance Tracking

Every resolved entity carries its source database, source version (snapshot date), and the matched synonym that triggered the match — enabling auditing and quality control. Case is handled by the NLP levels above: level_one matches any case variant, level_two further strips punctuation for hyphenated or slash-delimited names.

Integration with QC

Entity resolution output is validated by fullmap_audit() from the qc module before being included in the knowledge graph.

See Quality Control for details.

Next Steps