Class UnigramSegmenter
java.lang.Object
opennlp.tools.tokenize.lattice.UnigramSegmenter
- All Implemented Interfaces:
opennlp.tools.tokenize.Tokenizer
Frequency-driven segmentation for Chinese and similar scripts: a Viterbi search that
maximizes the summed log-probability of the words in a user-supplied frequency
lexicon, with unlisted characters falling back to single-character words. This is the
unigram model behind common Chinese segmenters. It omits the connection costs used
by
LatticeTokenizer, making it suitable for lexicons containing words and
counts.
The lexicon format is one entry per line: the word, its count, and optionally a tag, separated by whitespace. The lexicon file is user-supplied; no lexicon data is bundled. Every reported span is in original text coordinates.
Instances are immutable and safe to share between threads.
- Since:
- 3.0.0
-
Method Summary
Modifier and TypeMethodDescriptionstatic UnigramSegmenterload(InputStream lexiconStream, Charset charset) Loads a frequency lexicon from a stream.static UnigramSegmenterLoads a frequency lexicon encoded in UTF-8.static UnigramSegmenterLoads a frequency lexicon.String[]opennlp.tools.util.Span[]tokenizePos(String text)
-
Method Details
-
load
Loads a frequency lexicon encoded in UTF-8.- Parameters:
lexicon- The lexicon file. Must not benull.- Returns:
- The segmenter. Not
null. - Throws:
opennlp.tools.util.InvalidFormatException- Thrown if the lexicon is empty or malformed.IOException- Thrown if reading fails or the lexicon exceeds a size limit.IllegalArgumentException- Thrown iflexiconisnull.
-
load
Loads a frequency lexicon.- Parameters:
lexicon- The lexicon file: one word, its count, and an optional tag per line. Must not benull.charset- The lexicon encoding. Must not benull.- Returns:
- The segmenter. Not
null. - Throws:
opennlp.tools.util.InvalidFormatException- Thrown if the lexicon is empty or malformed.IOException- Thrown if reading fails or the lexicon exceeds a size limit.IllegalArgumentException- Thrown if a parameter isnull.
-
load
Loads a frequency lexicon from a stream.- Parameters:
lexiconStream- The lexicon content. Must not benull. Not closed.charset- The lexicon encoding. Must not benull.- Returns:
- The segmenter. Not
null. - Throws:
opennlp.tools.util.InvalidFormatException- Thrown if the lexicon is empty or malformed.IOException- Thrown if reading fails or the lexicon exceeds a size limit.IllegalArgumentException- Thrown if a parameter isnull.
-
tokenize
Reports the segmented surfaces, whitespace omitted.
- Specified by:
tokenizein interfaceopennlp.tools.tokenize.Tokenizer- Throws:
IllegalArgumentException- Thrown iftextisnull.
-
tokenizePos
Reports the segmented spans in original text coordinates, whitespace omitted.
- Specified by:
tokenizePosin interfaceopennlp.tools.tokenize.Tokenizer- Throws:
IllegalArgumentException- Thrown iftextisnull.
-