Comparison of biomedical relationship extraction methods and models for knowledge graph creation | Summary

This article explains the key points of Comparison of biomedical relationship extraction methods and models for knowledge graph creation.

• 2022-01-05 (arXiv), Preprint submitted to Web Semantics
• Milosevic, Nikola, Thielemann, Wolfgang.
• University of Manchester, Faculty of Science and Engineering, Bayer Pharmaceuticals R&D, Berlin, Bayer Pharmaceuticals R&D, Wuppertal
• Paper, version 4 (revised 2022-08-07; this review uses the revised version)

Read this article in Korean

Summary

• The paper compares methods for identifying typed relationships between biomedical entities in literature for a knowledge graph, a network of linked concepts. On a balanced gene-disease dataset after five epochs, PubMedBERT, a biomedical language model, achieved a weighted F1-score of 0.92, compared with 0.91 for DistilBERT and 0.88 for T5.
• The authors use manually corrected and rule-labeled sentences to train traditional classifiers and fine-tuned transformer models. Balancing improved the reported overall scores, but the classifier data came from sentences selected by the rule-based extractor, limiting evidence about performance on unfiltered literature.
• A rule-based scan of approximately 35 million Medline abstracts produced 4,784,985 typed relationships. In a manual evaluation of 100 abstracts containing at least one relationship, the extractor achieved 0.88 precision, 0.74 recall, and 0.80 F1-score.

1. Introduction

The authors address the difficulty of finding and validating biomedical claims across a growing literature. They propose normalized relationships among drugs, genes, and diseases, with links to the sentences supporting each graph edge.

2. Background

The paper distinguishes detecting whether a relationship exists, extracting an open-ended predicate, and mapping a relationship to a predefined type. It notes that granular, manually annotated biomedical relationship datasets are scarce and that distant learning, which assigns labels from known relationships to entity mentions, can introduce noise.

3. Method

The comparison covers pattern-based extraction, Naive Bayes and Random Forest classifiers, and fine-tuned DistilBERT, PubMedBERT, T5, and SciFive models. All approaches use the same internally modified Linnaeus tool to identify and normalize entities.

3.1. Named entity recognition and normalization (modification of Linnaeus)

Linnaeus identifies entity mentions and maps synonyms to preferred identifiers using internally developed dictionaries for human genes, diseases, and approved drugs. Modifications handle whitespace, case, pluralization, Greek characters, diacritics, regular expressions, and longest-match selection.

3.2. Relationship data model

Eighteen participants contributed to a relationship model covering Drug-Gene, Drug-Disease, and Gene-Disease pairs. Drug-Gene types adapt chemical-protein classes from the BioCreative shared task; Drug-Disease types distinguish treatment, adverse events, changes in disease, unspecified effects, and biomarkers, which indicate a disease or pathology.

• Gene-Disease types include Plays a role, Target, and Biomarker, with target subtypes for causation and disease modulation.
• Mode of action describes the action on a gene, such as activation, inhibition, or mutation; the model also records negation and speculation.

The BioCreative chemical-protein taxonomy supplies the starting point for the Drug-Gene model; the authors merge or exclude some categories when adapting it.

3.3. Relationship extraction using rule-based method

The rule-based extractor matches entity pairs, trigger phrases, and cues for mode of action, negation, and speculation against grammar patterns with permitted word gaps. Its confidence score divides the number of words matching named phrases by the total words in the matched pattern; co-occurrences receive a fixed score of 0.0001, and records retain the evidence sentence and Medline ID.

The example shows how a trigger phrase and mode-of-action cue can be matched with subject and predicate entities under word-distance rules.

3.4. Machine learning

The unbalanced gene-disease dataset combines 2,000 human-reviewed sentences with 10,000 rule-extracted sentences assigned confidence 1, and uses a 90% training and 10% test split. A more balanced dataset supplements the reviewed sentences with selected rule-extracted examples; No Explicit Relationship remains at 446 examples and Biomarker at 1,243.

• Naive Bayes and Random Forest use stemmed word sequences of one to four words as features.
• Transformer inputs replace the gene and disease mentions with SUBJECT and PREDICATE. Models were fine-tuned for five epochs at a learning rate of 0.00002; T5 and its biomedical variant SciFive use task prefixes, with the Adafactor optimizer specified for T5.
• For mode-of-action classification, the authors also constructed a balanced dataset of 300 examples per class, allowing duplicate examples for underrepresented classes.

The unbalanced set contains 7,393 Plays a role examples but only 198 Biomarker examples. The balanced set reduces this disparity while leaving No Explicit Relationship at 446 examples.

4. Results

The evaluation reports extraction scale and a manual check for the rule-based system, followed by sentence-level classification results for traditional and transformer models. The rule-based and classifier scores come from different evaluation sets and are not a direct head-to-head comparison.

4.1. Rule-based relationship extraction

Processing Medline data through January 2021 took about seven days on one machine and produced 4,784,985 typed relationships, or 35,900,521 including co-occurrences. On 100 manually evaluated abstracts containing at least one relationship, the rule-based method reached 0.88 precision, 0.74 recall, and 0.80 F1-score.

For an autosomal dominant polycystic kidney disease case study, the authors retained graph edges mentioned at least five times in PubMed. Kidney disease experts recognized all displayed entities from the literature; this assessment does not by itself validate every extracted edge.

The disease-centered graph includes PKD1 and PKD2 and drugs such as Tolvaptan. The case study retains relationships mentioned at least five times.

4.2. Naive Bayes and Random Forests-based Relationship extraction

On the unbalanced dataset, overall F1-scores were 0.48 for Naive Bayes and 0.66 for Random Forests; on the balanced dataset they rose to 0.74 and 0.78. The No Explicit Relationship class remained difficult: its balanced-set F1-score was 0 for Naive Bayes and 0.24 for Random Forests.

Balancing raises overall F1 for both traditional classifiers while the No Explicit Relationship class remains poorly recognized.

4.3. Transformer-based relationship extraction

After five epochs, weighted F1-scores on the balanced dataset were 0.92 for PubMedBERT, 0.91 for DistilBERT, 0.89 for SciFive, and 0.88 for T5. On the unbalanced dataset the corresponding scores were 0.90, 0.89, 0.88, and 0.86; the abstract’s 0.89 DistilBERT score matches the unbalanced result, whereas the balanced result is 0.91.

• For T5, No Explicit Relationship recall rose from 0.26 on the unbalanced dataset to 0.72 on the balanced dataset, with precision at 0.88 in both cases.
• T5 mode-of-action classification reported weighted F1-scores of 0.94 on the unbalanced dataset and 0.97 on the balanced dataset with 300 examples per class; some examples were duplicated.

The curves track four transformer models across five epochs on both data distributions; balanced PubMedBERT ends highest.

DistilBERT's balanced weighted F1 is 0.91 versus T5's 0.88; the per-class rows show much lower T5 recall for No Explicit Relationship on unbalanced data.

PubMedBERT achieves weighted F1 of 0.92 on balanced data and 0.90 on unbalanced data; SciFive achieves 0.89 and 0.88 respectively.

The balanced mode-of-action dataset yields a weighted F1 of 0.97; the authors allowed duplication when a class lacked enough examples.

5. Discussion

The authors report that domain-specific models improved overall F1 by about one percentage point over their general-domain counterparts in these comparisons. Comparisons with published systems use different datasets and relationship definitions, so they do not isolate model superiority on a shared benchmark.

6. Future work

Proposed work includes training models for additional entity pairs, combining structured and unstructured sources, building an interface for inspecting relationships and evidence, and exploring temporal graph learning for candidate prediction.

7. Conclusions

The authors conclude that transformer models were less sensitive to class imbalance than the traditional classifiers and that DistilBERT offers a size and speed trade-off against PubMedBERT. They report that approximately 7,000 commissioned sentence annotations cost 16,200 euros, in addition to infrastructure and engineering costs.

Appendix

• The paper states that access to the generated datasets can be requested at https://zenodo.org/record/6466316#.Ylw3T-dS9Ea.

Brief Thoughts

The class-level results clarify a change hidden by weighted averages: T5’s recall for No Explicit Relationship increased from 0.26 to 0.72. An independently annotated test set drawn from unfiltered literature would help assess performance beyond the patterns selected by the rule-based extractor.