Class TokenizerFactory

java.lang.Object
opennlp.tools.util.BaseToolFactory
opennlp.tools.tokenize.TokenizerFactory

public class TokenizerFactory extends BaseToolFactory
The factory that provides Tokenizer default implementation and resources. Users can extend this class if their application requires overriding the TokenContextGenerator, Dictionary etc.
  • Constructor Details

    • TokenizerFactory

      public TokenizerFactory()
      Instantiates a TokenizerFactory that provides the default implementation of the resources.
    • TokenizerFactory

      public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization)
      Instantiates a TokenizerFactory with the TokenizerCharacterPolicy of its language. Use this constructor to programmatically create a factory.
      Parameters:
      languageCode - The ISO language code to be used for this factory.
      abbreviationDictionary - The Dictionary which holds abbreviations.
      useAlphaNumericOptimization - Whether alphanumerics are skipped, or not.
    • TokenizerFactory

      public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy)
      Instantiates a TokenizerFactory. Use this constructor to programmatically create a factory.
      Parameters:
      languageCode - The ISO language code to be used for this factory.
      abbreviationDictionary - The Dictionary which holds abbreviations.
      useAlphaNumericOptimization - Whether alphanumerics are skipped, or not.
      characterPolicy - null or a custom TokenizerCharacterPolicy (default is the policy for languageCode, provided by Factory.getCharacterPolicy(String)).
    • TokenizerFactory

      @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Instantiates a TokenizerFactory whose alphanumeric optimization checks candidate tokens with a regular expression.
      Parameters:
      languageCode - The ISO language code to be used for this factory.
      abbreviationDictionary - The Dictionary which holds abbreviations.
      useAlphaNumericOptimization - Whether alphanumerics are skipped, or not.
      alphaNumericPattern - null for the policy of languageCode, or a custom alphanumeric Pattern.
  • Method Details

    • validateArtifactMap

      public void validateArtifactMap() throws opennlp.tools.util.InvalidFormatException
      Description copied from class: BaseToolFactory
      Validates the parsed artifacts.

      Note: Subclasses should generally invoke super.validateArtifactMap at the beginning of this method.

      Specified by:
      validateArtifactMap in class BaseToolFactory
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if validation found invalid states.
    • createArtifactMap

      public Map<String,Object> createArtifactMap()
      Description copied from class: BaseToolFactory
      A model's implementation should call this constructor that creates a model programmatically.

      The base implementation will return a HashMap that should be populated by subclasses.

      Overrides:
      createArtifactMap in class BaseToolFactory
      Returns:
      Retrieves a Map with pairs of keys and objects.
    • createManifestEntries

      public Map<String,String> createManifestEntries()
      Overrides:
      createManifestEntries in class BaseToolFactory
      Returns:
      Retrieves the manifest entries to be added to the model manifest.
    • create

      public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) throws opennlp.tools.util.InvalidFormatException
      Factory method the framework uses to instantiate a new TokenizerFactory.
      Parameters:
      subclassName - The name of the class implementing the TokenizerFactory.
      languageCode - The ISO language code the Tokenizer should use.
      abbreviationDictionary - An optional Dictionary containing abbreviations, or null if not present.
      useAlphaNumericOptimization - Whether the alphanumeric optimization is enabled or not.
      characterPolicy - The TokenizerCharacterPolicy the alphanumeric optimization should use, if enabled, or null for the language default.
      Returns:
      A valid TokenizerFactory instance.
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if one of the input parameters doesn't comply the expected format.
    • create

      public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization) throws opennlp.tools.util.InvalidFormatException
      Factory method the framework uses to instantiate a new TokenizerFactory with the TokenizerCharacterPolicy of its language.
      Parameters:
      subclassName - The name of the class implementing the TokenizerFactory.
      languageCode - The ISO language code the Tokenizer should use.
      abbreviationDictionary - An optional Dictionary containing abbreviations, or null if not present.
      useAlphaNumericOptimization - Whether the alphanumeric optimization is enabled or not.
      Returns:
      A valid TokenizerFactory instance.
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if one of the input parameters doesn't comply the expected format.
    • create

      @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) throws opennlp.tools.util.InvalidFormatException
      Deprecated, for removal: This API element is subject to removal in a future version.
      Factory method the framework uses to instantiate a new TokenizerFactory whose alphanumeric optimization checks candidate tokens with a regular expression.
      Parameters:
      subclassName - The name of the class implementing the TokenizerFactory.
      languageCode - The ISO language code the Tokenizer should use.
      abbreviationDictionary - An optional Dictionary containing abbreviations, or null if not present.
      useAlphaNumericOptimization - Whether the alphanumeric optimization is enabled or not.
      alphaNumericPattern - null for the policy of languageCode, or a custom alphanumeric Pattern.
      Returns:
      A valid TokenizerFactory instance.
      Throws:
      opennlp.tools.util.InvalidFormatException - Thrown if one of the input parameters doesn't comply the expected format.
    • getCharacterPolicy

      public TokenizerCharacterPolicy getCharacterPolicy()
      Returns:
      Retrieves the (user-)specified TokenizerCharacterPolicy, the one read from the model manifest, or the default for the language in use; null if this factory checks candidate tokens with a custom alphanumeric Pattern.
      Throws:
      IllegalArgumentException - Thrown if the manifest has a character policy or an alphanumeric pattern that cannot be read.
    • getAlphaNumericPattern

      @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public Pattern getAlphaNumericPattern()
      Deprecated, for removal: This API element is subject to removal in a future version.
      Use getCharacterPolicy(). This method will be removed in 4.0.
      Returns:
      Retrieves the custom alphanumeric Pattern of this factory, or else the pattern of earlier releases that accepts the same characters as its TokenizerCharacterPolicy; null for a custom policy that no such pattern matches.
      Throws:
      IllegalArgumentException - Thrown if the manifest has a character policy or an alphanumeric pattern that cannot be read.
    • isUseAlphaNumericOptimization

      public boolean isUseAlphaNumericOptimization()
      Returns:
      true if the alphanumeric optimization is enabled, otherwise false.
    • getAbbreviationDictionary

      public Dictionary getAbbreviationDictionary()
      Returns:
      The abbreviation Dictionary or null if none is active.
    • getLanguageCode

      public String getLanguageCode()
      Returns:
      Retrieves the ISO language code in use.
    • getContextGenerator

      public opennlp.tools.tokenize.TokenContextGenerator getContextGenerator()
      Returns:
      Retrieves a TokenContextGenerator instance.