Class TokenSampleStream

java.lang.Object
opennlp.tools.tokenize.lang.en.TokenSampleStream
All Implemented Interfaces:
Iterator<opennlp.tools.tokenize.TokenSample>

public class TokenSampleStream extends Object implements Iterator<opennlp.tools.tokenize.TokenSample>
Class which produces an Iterator<TokenSample> from a file of whitespace delimited tokens. This class uses a number of English-specific heuristics to un-separate tokens which are typically found together in text.

Tokens are separated by runs of Unicode White_Space, see StringUtil.splitOnUnicodeWhitespace(CharSequence), independent of the whitespace mode; leading and trailing whitespace adds no token. A line without a token resets the quote state, and a token holding a letter or digit of any script is a word rather than punctuation.

  • Constructor Details

  • Method Details

    • hasNext

      public boolean hasNext()
      Specified by:
      hasNext in interface Iterator<opennlp.tools.tokenize.TokenSample>
    • next

      public opennlp.tools.tokenize.TokenSample next()
      Specified by:
      next in interface Iterator<opennlp.tools.tokenize.TokenSample>
    • remove

      public void remove()
      Specified by:
      remove in interface Iterator<opennlp.tools.tokenize.TokenSample>