Class WordpieceEncoder

java.lang.Object
opennlp.tools.tokenize.WordpieceEncoder
All Implemented Interfaces:
SubwordTokenizer

@ThreadSafe public final class WordpieceEncoder extends Object implements SubwordTokenizer
A SubwordTokenizer implementing the BERT tokenization stages: basic tokenization (control removal, whitespace normalization, CJK isolation, optional lower casing with accent stripping, punctuation isolation) followed by greedy longest-match wordpiece segmentation.

Each result includes a vocabulary id and range in the original text. The range refers to the input before normalization. encode(CharSequence) always starts with the classification entry, with an empty range at offset 0, and ends with the separator entry, with an empty range at the end of the text.

Segmentation is greedy longest-match-first with the ## continuation marker, as in the BERT reference implementation.

Ids follow the line-number convention of BERT vocab.txt files. List constructors use the list index, while the map constructor uses the supplied ids. The classification, separator, and unknown tokens must all be present in the vocabulary, because each emitted piece must have an id. Vocabulary entries starting with ## are continuation pieces and can match only after the first piece of a word.

Lower casing applies the Unicode simple case mapping of Character.toLowerCase(int), one code point to one code point, plus the Final_Sigma context, so a word-final Greek capital sigma becomes U+03C2 as in the reference implementation. Other SpecialCasing mappings are not applied; for example U+0130 becomes i without a combining dot.

A word exceeding the configured maximum number of normalized Unicode code points becomes the unknown piece. The default is 100, the value used by the Hugging Face transformers BERT tokenizer; the original google-research/bert code uses 200. Both count code points, and a constructor parameter selects another limit.

Since:
3.0.0
See Also:
  • Constructor Details

    • WordpieceEncoder

      public WordpieceEncoder(List<String> vocabulary)
      Instantiates an encoder for an uncased BERT model with the BERT special tokens.
      Parameters:
      vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.
      Throws:
      IllegalArgumentException - Thrown if the vocabulary is null, contains a null, empty, or duplicate entry, or is missing a BERT special token.
    • WordpieceEncoder

      public WordpieceEncoder(List<String> vocabulary, boolean lowerCase)
      Instantiates an encoder with the BERT special tokens.
      Parameters:
      vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.
      lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.
      Throws:
      IllegalArgumentException - Thrown if the vocabulary is null, contains a null, empty, or duplicate entry, or is missing a BERT special token.
    • WordpieceEncoder

      public WordpieceEncoder(List<String> vocabulary, boolean lowerCase, int maxWordCodePoints)
      Instantiates an encoder with the BERT special tokens and a custom word-length limit.
      Parameters:
      vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.
      lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.
      maxWordCodePoints - The non-negative maximum number of normalized Unicode code points in one word.
      Throws:
      IllegalArgumentException - Thrown if an argument is invalid or a BERT special token is missing.
    • WordpieceEncoder

      public WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken)
      Instantiates an encoder with custom special tokens.
      Parameters:
      vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.
      lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.
      classificationToken - The CLS token; must not be null or empty and must be in the vocabulary.
      separatorToken - The SEP token; must not be null or empty and must be in the vocabulary.
      unknownToken - The UNK token; must not be null or empty and must be in the vocabulary.
      Throws:
      IllegalArgumentException - Thrown if any argument is null, the vocabulary contains a null, empty, or duplicate entry, or a special token is empty or missing.
    • WordpieceEncoder

      public WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints)
      Instantiates an encoder with custom special tokens and a custom word-length limit.
      Parameters:
      vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.
      lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.
      classificationToken - The CLS token; must not be null or empty and must be in the vocabulary.
      separatorToken - The SEP token; must not be null or empty and must be in the vocabulary.
      unknownToken - The UNK token; must not be null or empty and must be in the vocabulary.
      maxWordCodePoints - The non-negative maximum number of normalized Unicode code points in one word.
      Throws:
      IllegalArgumentException - Thrown if an argument is invalid or a special token is missing.
    • WordpieceEncoder

      public WordpieceEncoder(Map<String,Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken)
      Instantiates an encoder from an explicit piece-to-id mapping for vocabularies with noncontiguous ids.
      Parameters:
      vocabularyIds - The piece-to-id mapping. Must not be null or contain null or empty keys, null values, or negative ids.
      lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.
      classificationToken - The CLS token; must not be null or empty and must be in the vocabulary.
      separatorToken - The SEP token; must not be null or empty and must be in the vocabulary.
      unknownToken - The UNK token; must not be null or empty and must be in the vocabulary.
      Throws:
      IllegalArgumentException - Thrown if any argument is null, the mapping contains a null or empty key, null value, or negative id, or a special token is empty or missing.
    • WordpieceEncoder

      public WordpieceEncoder(Map<String,Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints)
      Instantiates an encoder from a piece-to-id mapping with a custom word-length limit.
      Parameters:
      vocabularyIds - The piece-to-id mapping. Must not be null or contain null or empty keys, null values, or negative ids.
      lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.
      classificationToken - The CLS token; must not be null or empty and must be in the vocabulary.
      separatorToken - The SEP token; must not be null or empty and must be in the vocabulary.
      unknownToken - The UNK token; must not be null or empty and must be in the vocabulary.
      maxWordCodePoints - The non-negative maximum number of normalized Unicode code points in one word.
      Throws:
      IllegalArgumentException - Thrown if an argument is invalid or a special token is missing.
  • Method Details

    • encode

      public List<SubwordPiece> encode(CharSequence text)
      Encodes text into subword pieces.
      Specified by:
      encode in interface SubwordTokenizer
      Parameters:
      text - The text to encode; must not be null.
      Returns:
      The pieces in model order; may be empty.