Class TokenizerFactory
java.lang.Object
opennlp.tools.util.BaseToolFactory
opennlp.tools.tokenize.TokenizerFactory
The factory that provides
Tokenizer default implementation and
resources. Users can extend this class if their application requires
overriding the TokenContextGenerator, Dictionary etc.-
Constructor Summary
ConstructorsConstructorDescriptionInstantiates aTokenizerFactorythat provides the default implementation of the resources.TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization) Instantiates aTokenizerFactorywith theTokenizerCharacterPolicyof its language.TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) Deprecated, for removal: This API element is subject to removal in a future version.TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) Instantiates aTokenizerFactory. -
Method Summary
Modifier and TypeMethodDescriptionstatic TokenizerFactorycreate(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization) Factory method the framework uses to instantiate a newTokenizerFactorywith theTokenizerCharacterPolicyof its language.static TokenizerFactorycreate(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) Deprecated, for removal: This API element is subject to removal in a future version.static TokenizerFactorycreate(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) Factory method the framework uses to instantiate a newTokenizerFactory.A model's implementation should call this constructor that creates a model programmatically.Deprecated, for removal: This API element is subject to removal in a future version.UsegetCharacterPolicy().opennlp.tools.tokenize.TokenContextGeneratorbooleanvoidValidates the parsed artifacts.Methods inherited from class BaseToolFactory
create, create, createArtifactSerializersMap
-
Constructor Details
-
TokenizerFactory
public TokenizerFactory()Instantiates aTokenizerFactorythat provides the default implementation of the resources. -
TokenizerFactory
public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization) Instantiates aTokenizerFactorywith theTokenizerCharacterPolicyof its language. Use this constructor to programmatically create a factory.- Parameters:
languageCode- The ISO language code to be used for this factory.abbreviationDictionary- TheDictionarywhich holds abbreviations.useAlphaNumericOptimization- Whether alphanumerics are skipped, or not.
-
TokenizerFactory
public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) Instantiates aTokenizerFactory. Use this constructor to programmatically create a factory.- Parameters:
languageCode- The ISO language code to be used for this factory.abbreviationDictionary- TheDictionarywhich holds abbreviations.useAlphaNumericOptimization- Whether alphanumerics are skipped, or not.characterPolicy-nullor a customTokenizerCharacterPolicy(default is the policy forlanguageCode, provided byFactory.getCharacterPolicy(String)).
-
TokenizerFactory
@Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public TokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) Deprecated, for removal: This API element is subject to removal in a future version.UseTokenizerFactory(String, Dictionary, boolean, TokenizerCharacterPolicy). This constructor will be removed in 4.0.Instantiates aTokenizerFactorywhose alphanumeric optimization checks candidate tokens with a regular expression.- Parameters:
languageCode- The ISO language code to be used for this factory.abbreviationDictionary- TheDictionarywhich holds abbreviations.useAlphaNumericOptimization- Whether alphanumerics are skipped, or not.alphaNumericPattern-nullfor the policy oflanguageCode, or a custom alphanumericPattern.
-
-
Method Details
-
validateArtifactMap
public void validateArtifactMap() throws opennlp.tools.util.InvalidFormatExceptionDescription copied from class:BaseToolFactoryValidates the parsed artifacts.Note: Subclasses should generally invoke
super.validateArtifactMapat the beginning of this method.- Specified by:
validateArtifactMapin classBaseToolFactory- Throws:
opennlp.tools.util.InvalidFormatException- Thrown if validation found invalid states.
-
createArtifactMap
Description copied from class:BaseToolFactoryA model's implementation should call this constructor that creates a model programmatically.The base implementation will return a
HashMapthat should be populated by subclasses.- Overrides:
createArtifactMapin classBaseToolFactory- Returns:
- Retrieves a
Mapwith pairs of keys and objects.
-
createManifestEntries
- Overrides:
createManifestEntriesin classBaseToolFactory- Returns:
- Retrieves the manifest entries to be added to the model manifest.
-
create
public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) throws opennlp.tools.util.InvalidFormatException Factory method the framework uses to instantiate a newTokenizerFactory.- Parameters:
subclassName- The name of the class implementing theTokenizerFactory.languageCode- The ISO language code theTokenizershould use.abbreviationDictionary- An optionalDictionarycontaining abbreviations, ornullif not present.useAlphaNumericOptimization- Whether the alphanumeric optimization is enabled or not.characterPolicy- TheTokenizerCharacterPolicythe alphanumeric optimization should use, if enabled, ornullfor the language default.- Returns:
- A valid
TokenizerFactoryinstance. - Throws:
opennlp.tools.util.InvalidFormatException- Thrown if one of the input parameters doesn't comply the expected format.
-
create
public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization) throws opennlp.tools.util.InvalidFormatException Factory method the framework uses to instantiate a newTokenizerFactorywith theTokenizerCharacterPolicyof its language.- Parameters:
subclassName- The name of the class implementing theTokenizerFactory.languageCode- The ISO language code theTokenizershould use.abbreviationDictionary- An optionalDictionarycontaining abbreviations, ornullif not present.useAlphaNumericOptimization- Whether the alphanumeric optimization is enabled or not.- Returns:
- A valid
TokenizerFactoryinstance. - Throws:
opennlp.tools.util.InvalidFormatException- Thrown if one of the input parameters doesn't comply the expected format.
-
create
@Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public static TokenizerFactory create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, Pattern alphaNumericPattern) throws opennlp.tools.util.InvalidFormatException Deprecated, for removal: This API element is subject to removal in a future version.Usecreate(String, String, Dictionary, boolean, TokenizerCharacterPolicy). This method will be removed in 4.0.Factory method the framework uses to instantiate a newTokenizerFactorywhose alphanumeric optimization checks candidate tokens with a regular expression.- Parameters:
subclassName- The name of the class implementing theTokenizerFactory.languageCode- The ISO language code theTokenizershould use.abbreviationDictionary- An optionalDictionarycontaining abbreviations, ornullif not present.useAlphaNumericOptimization- Whether the alphanumeric optimization is enabled or not.alphaNumericPattern-nullfor the policy oflanguageCode, or a custom alphanumericPattern.- Returns:
- A valid
TokenizerFactoryinstance. - Throws:
opennlp.tools.util.InvalidFormatException- Thrown if one of the input parameters doesn't comply the expected format.
-
getCharacterPolicy
- Returns:
- Retrieves the (user-)specified
TokenizerCharacterPolicy, the one read from the model manifest, or the default for the language in use;nullif this factory checks candidate tokens with a custom alphanumericPattern. - Throws:
IllegalArgumentException- Thrown if the manifest has a character policy or an alphanumeric pattern that cannot be read.
-
getAlphaNumericPattern
@Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public Pattern getAlphaNumericPattern()Deprecated, for removal: This API element is subject to removal in a future version.UsegetCharacterPolicy(). This method will be removed in 4.0.- Returns:
- Retrieves the custom alphanumeric
Patternof this factory, or else the pattern of earlier releases that accepts the same characters as itsTokenizerCharacterPolicy;nullfor a custom policy that no such pattern matches. - Throws:
IllegalArgumentException- Thrown if the manifest has a character policy or an alphanumeric pattern that cannot be read.
-
isUseAlphaNumericOptimization
public boolean isUseAlphaNumericOptimization()- Returns:
trueif the alphanumeric optimization is enabled, otherwisefalse.
-
getAbbreviationDictionary
- Returns:
- The abbreviation
Dictionaryornullif none is active.
-
getLanguageCode
- Returns:
- Retrieves the ISO language code in use.
-
getContextGenerator
public opennlp.tools.tokenize.TokenContextGenerator getContextGenerator()- Returns:
- Retrieves a
TokenContextGeneratorinstance.
-
TokenizerFactory(String, Dictionary, boolean, TokenizerCharacterPolicy).