Repository navigation
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms - #1167
Closed
krickert wants to merge 32 commits into
Closed
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167krickert wants to merge 32 commits into
krickert wants to merge 32 commits into
Conversation
krickert
changed the title
krickert
changed the title
krickert
force-pushed
the
wordnet-expansion
branch
6 times, most recently
from
July 17, 2026 20:45
5db6a8a to
917e4be
Compare
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 19, 2026 11:41
1b90059 to
f4cc90c
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 24, 2026 19:27
ff066b0 to
b1aef66
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
The clean-room CFSA2/FSA5 readers and the PoliMorf lemmatizer had no node because until today they had no branch: they were written on the OPENNLP-1887 line under formats: and lemmatizer: titles and were only visible as a sentence in that pull request's notes. They now sit on their own branch off current main and get a node and a row like every other staged feature. The note on apache#1167 says what is actually true: the commits were copied out, and dropping them from that branch is a separate rebase that has not happened.
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 31, 2026 10:38
89a5a27 to
840d4fd
Compare
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
August 9, 2026 13:09
af19962 to
be24524
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Aug 9, 2026
Adds the symbol-joiner and dehyphenation rows, marks the apache#1167 restack and the morfologik reconciliation, and moves the state line to fc9824a.
krickert
force-pushed
the
wordnet-expansion
branch
from
August 10, 2026 10:24
be24524 to
3dddfed
Compare
krickert
added a commit
that referenced
this pull request
Aug 15, 2026
# Conflicts: # opennlp-api/src/main/java/opennlp/tools/util/StringUtil.java # opennlp-tools/src/test/java/opennlp/tools/util/StringUtilTest.java
krickert
added a commit
that referenced
this pull request
Aug 16, 2026
krickert
added a commit
that referenced
this pull request
Aug 30, 2026
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 4, 2026
Adds the symbol-joiner and dehyphenation rows, marks the apache#1167 restack and the morfologik reconciliation, and moves the state line to fc9824a.
krickert
force-pushed
the
wordnet-expansion
branch
from
September 4, 2026 21:58
a68762f to
1d13232
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 5, 2026
# Conflicts: # opennlp-api/src/main/java/opennlp/tools/wordnet/LexicalKnowledgeBase.java # opennlp-api/src/main/java/opennlp/tools/wordnet/Synset.java # opennlp-api/src/main/java/opennlp/tools/wordnet/WordNetPOS.java # opennlp-api/src/main/java/opennlp/tools/wordnet/WordNetRelation.java # opennlp-api/src/test/java/opennlp/tools/wordnet/LexicalKnowledgeBaseTest.java # opennlp-distr/pom.xml # opennlp-distr/src/main/assembly/bin.xml # opennlp-docs/src/docbkx/wordnet.xml # opennlp-extensions/opennlp-wordnet/pom.xml # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/HypernymTyper.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/InMemoryWordNetLexicon.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/LemmaFolding.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/LexicalExpander.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyExceptions.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyLemmatizer.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/SynsetSimilarity.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfDependency.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfLexicon.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfResource.java # opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/HypernymTyperTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/InMemoryWordNetLexiconTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LemmaFoldingTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexicalExpanderLexiconTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexicalExpanderTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexicalExpansionUsageExampleTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/LexiconConcurrencyTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/MorphyLemmatizerTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/ReaderEquivalenceTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/SynsetSimilarityTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WnLmfReaderTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WndbReaderTest.java # opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WordNetUsageExampleTest.java # opennlp-extensions/opennlp-wordnet/src/test/resources/opennlp/wordnet/mini-wndb/.gitattributes # pom.xml # rat-excludes
…s and a Morphy lemmatizer Adds the LexicalKnowledgeBase contract in opennlp.tools.wordnet and the opennlp-wordnet module implementing it twice: WnLmfReader for WN-LMF XML and WndbReader for the legacy WNDB database files, with reader-equivalence coverage over miniature fixtures of both formats. The WN-LMF reader skips DOCTYPE declarations unresolved with DTD support off, so Open English WordNet releases parse unmodified while entity expansion stays closed; the WNDB fixtures are pinned to LF so their embedded byte offsets survive Windows checkout. The Morphy lemmatizer resolves inflected forms through suffix rules and the format's exception lists. Null arguments fail loudly with IllegalArgumentException, malformed data raises the checked InvalidFormatException, and the public seam carries no brand name: WordNet stays in the names of the classes that actually read WordNet formats.
…-lexicon resources Extends WordNetRelation with every typed synset and sense relation in the WN-LMF 1.4 DTD and rejects relations placed on the wrong element kind. Adds WnLmfReader.readResource with WnLmfResource, WnLmfLexicon, and WnLmfDependency so multi-lexicon documents keep per-language lookup indexes and Requires metadata; the single-lexicon read methods reject that shape instead of merging, and LexiconExtension is rejected clearly. Unlexicalized synsets used by real OMW releases are retained as relation targets with empty lemma lists. Includes relation-conformance and resource tests, an optional pinned OMW 2.0 integration check behind dev/test-omw-wordnets.sh, and manual coverage with mirror tests.
The canonical lemma fold used String.toLowerCase(Locale.ROOT), which applies the SpecialCasing one-to-many mappings. Fold with StringUtil.toLowerCase instead so the sense-index keys, the Morphy exception keys, and the query path all share the locale-independent one-to-one mapping.
…plainly Repeated exception message literals become constants, and the seam, reader, and lemmatizer Javadoc and the manual chapter say what the types do in plainer terms.
…nsion over the lexical knowledge base Expands a term into the synonyms sharing its synsets, the lemmas of its hypernym ancestors up to a configured depth (following both the direct and the instance relation), and optionally its direct hyponyms. Each expansion carries a deterministic heuristic weight: sense rank and every relation step multiply configurable decays, so consumers can discount looser expansions instead of treating them as the original term. Results exclude the input, deduplicate case-insensitively keeping the highest weight, and order stably by weight, kind, and term. Inflected input works through an optional Lemmatizer invoked with the WordNetPos name as the tag, which the Morphy lemmatizer understands: dogs expands through dog, mice through the exception list to mouse. Hypernym walks are visited-checked so malformed cyclic data terminates. Tests cover the graph behavior on a hand-built lexicon (sense ranking, decay arithmetic, dedup, cycle termination, validation) and run the whole stack over the miniature WN-LMF and WNDB fixtures, asserting both readers produce identical expansions.
…yping SynsetSimilarity scores noun synset pairs with the path, Wu-Palmer, and Leacock-Chodorow measures over the knowledge base seam. HypernymTyper labels a word by walking its senses' hypernym and instance-hypernym chains to the nearest caller-registered anchor concept, so a knowledge base can type names as person, organization, or location without a model. Blank checks follow the toolkit whitespace definition.
…examples Add the lexical expansion section to docbkx/wordnet.xml and LexicalExpansionUsageExampleTest asserting the expansion values the chapter prints; carry the WordNet usage example alongside it.
…ment with the review conventions
…and default behavior Show the expansion and similarity taxonomies in the chapter's listing comments so every synset id and word in the listings exists in the shown lexicon, add a hypernym-anchored typing section, and extend the mirror test to build exactly those taxonomies and assert every printed value. Pin the formula-driven numeric regimes (Wu-Palmer zero at the root pair, negative Leacock-Chodorow under an understated depth budget) and the expander builder's accepting boundaries and defaults (senseDecay 1.0, hypernymDepth 0, maxSenses 3, maxExpansions 20) against fixture lemmas that exceed the defaults.
…m no ancestry city (n8) and company (n10) share only the taxonomy root, yet wuPalmer returns 0.0 for them, the same score as the disconnected chemist/abstract pair. The documented contract reserves 0 for pairs with no shared ancestor. Red: testWuPalmerDistinguishesRootOnlyAncestryFromNoAncestry fails with 'city and company share entity; expected positive Wu-Palmer, got 0.0'.
…ores positive The edge-counted formula gave the taxonomy root depth zero, so pairs whose only shared ancestor is the root collapsed to 0.0 and were indistinguishable from pairs sharing no ancestor, violating the documented contract that 0 is reserved for no shared ancestor. Wu and Palmer's node counting places the root at depth one; the formula 2 * depth(lcs) / (depth(a) + depth(b)) keeps its shape, the zero denominator guard becomes unreachable, and (root, root) self-similarity rises from the pinned 0.0 to a full 1.0. Under the corrected convention wuPalmer(n5, n6) moves from 8/9 to 10/11; the pinned tests, the usage example mirror test, and the manual's similarity listing and prose are updated to match.
Reword the expander, similarity, and typer Javadoc and the manual chapter in plainer terms, state the Leacock-Chodorow example in the manual with a matching assertion, and name the tests after the behavior they pin.
krickert
force-pushed
the
wordnet-expansion
branch
from
September 16, 2026 06:57
4f91b17 to
ccf3a4f
Compare
Contributor
|
This shall be a PR / feature for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on main. This branch stacks on #1155 (the
LexicalKnowledgeBaseinterface and readers) and includes those 18 commits until that PR merges; the expansion work is the commits after them. #1182 (StringUtil.isBlank) is merged.Adds
LexicalExpanderto theopennlp-wordnetmodule: given a term, it produces weighted related terms from aLexicalKnowledgeBase, the synonyms sharing its synsets, the lemmas of hypernym ancestors up to a configured depth (following both the direct and the instance relation, visited-checked so cyclic data terminates), and optionally direct hyponyms. Each expansion has a deterministic heuristic weight (sense-rank decay times depth decay, both configurable), results are deduplicated case-insensitively keeping the highest weight, the input term is excluded, and ordering is stable. An optionalLemmatizerfallback expands inflected input; the Morphy lemmatizer of this module plugs in directly, sodogsexpands viadogandmiceviamouse.SynsetSimilarityscores synset pairs on the hypernym graph with path, Wu-Palmer (depths counted in nodes, so a shared ancestor scores above zero), and Leacock-Chodorow measures, andHypernymTypertypes a noun by the nearest caller-registered anchor in its hypernym graph.Tests cover the behavior over a hand-built graph with a controlled shape (sense ranking, depth decay, dedupe, cycle termination, hyponym opt-in, configuration validation, taxonomy depth, missing parents) plus integration over the miniature WN-LMF and WNDB fixtures asserting both readers expand identically. The manual chapter's examples are mirrored by
LexicalExpansionUsageExampleTest.Review round of 2026-09-04
Javadoc on every method and constructor with
@throwson the validating paths, validation at the public boundary, the Wu-Palmer depth fix with a failing test ahead of it, and the expander, similarity, and typer Javadoc reworded in plainer terms.