Class UnigramSegmenter

java.lang.Object
opennlp.tools.tokenize.lattice.UnigramSegmenter
All Implemented Interfaces:
opennlp.tools.tokenize.Tokenizer

public final class UnigramSegmenter extends Object implements opennlp.tools.tokenize.Tokenizer
Frequency-driven segmentation for Chinese and similar scripts: a Viterbi search that maximizes the summed log-probability of the words in a user-supplied frequency lexicon, with unlisted characters falling back to single-character words. This is the unigram model behind common Chinese segmenters. It omits the connection costs used by LatticeTokenizer, making it suitable for lexicons containing words and counts.

The lexicon format is one entry per line: the word, its count, and optionally a tag, separated by whitespace. The lexicon file is user-supplied; no lexicon data is bundled. Every reported span is in original text coordinates.

Instances are immutable and safe to share between threads.

Since:
3.0.0
  • Method Details

    • load

      public static UnigramSegmenter load(Path lexicon) throws IOException
      Loads a frequency lexicon encoded in UTF-8.
      Parameters:
      lexicon - The lexicon file. Must not be null.
      Returns:
      The segmenter. Not null.
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if the lexicon is empty or malformed.
      IOException - Thrown if reading fails or the lexicon exceeds a size limit.
      IllegalArgumentException - Thrown if lexicon is null.
    • load

      public static UnigramSegmenter load(Path lexicon, Charset charset) throws IOException
      Loads a frequency lexicon.
      Parameters:
      lexicon - The lexicon file: one word, its count, and an optional tag per line. Must not be null.
      charset - The lexicon encoding. Must not be null.
      Returns:
      The segmenter. Not null.
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if the lexicon is empty or malformed.
      IOException - Thrown if reading fails or the lexicon exceeds a size limit.
      IllegalArgumentException - Thrown if a parameter is null.
    • load

      public static UnigramSegmenter load(InputStream lexiconStream, Charset charset) throws IOException
      Loads a frequency lexicon from a stream.
      Parameters:
      lexiconStream - The lexicon content. Must not be null. Not closed.
      charset - The lexicon encoding. Must not be null.
      Returns:
      The segmenter. Not null.
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if the lexicon is empty or malformed.
      IOException - Thrown if reading fails or the lexicon exceeds a size limit.
      IllegalArgumentException - Thrown if a parameter is null.
    • tokenize

      public String[] tokenize(String text)

      Reports the segmented surfaces, whitespace omitted.

      Specified by:
      tokenize in interface opennlp.tools.tokenize.Tokenizer
      Throws:
      IllegalArgumentException - Thrown if text is null.
    • tokenizePos

      public opennlp.tools.util.Span[] tokenizePos(String text)

      Reports the segmented spans in original text coordinates, whitespace omitted.

      Specified by:
      tokenizePos in interface opennlp.tools.tokenize.Tokenizer
      Throws:
      IllegalArgumentException - Thrown if text is null.