Uses of Class
opennlp.tools.tokenize.TokenizerCharacterPolicy
Packages that use TokenizerCharacterPolicy
-
Uses of TokenizerCharacterPolicy in opennlp.tools.tokenize
Methods in opennlp.tools.tokenize that return TokenizerCharacterPolicyModifier and TypeMethodDescriptionstatic TokenizerCharacterPolicyTokenizerCharacterPolicy.ascii()Returns a policy for ASCII letters and digits, with no marks.TokenizerFactory.getCharacterPolicy()static TokenizerCharacterPolicyTokenizerCharacterPolicy.of(opennlp.tools.util.normalizer.CodePointSet letters, opennlp.tools.util.normalizer.CodePointSet digits, opennlp.tools.util.normalizer.CodePointSet marks) Creates a policy from explicit, pairwise-disjoint code point sets.static TokenizerCharacterPolicyParses the text form written byTokenizerCharacterPolicy.toString(): the three fieldsletters=,;digits=and;marks=in that order, each followed by zero or more comma separated hex code points (0041) or inclusive ranges (0041-005A).Methods in opennlp.tools.tokenize with parameters of type TokenizerCharacterPolicyModifier and TypeMethodDescriptionstatic TokenizerFactoryTokenizerFactory.create(String subclassName, String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) Factory method the framework uses to instantiate a newTokenizerFactory.protected voidTokenizerFactory.init(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) Constructors in opennlp.tools.tokenize with parameters of type TokenizerCharacterPolicyModifierConstructorDescriptionTokenizerFactory(String languageCode, Dictionary abbreviationDictionary, boolean useAlphaNumericOptimization, TokenizerCharacterPolicy characterPolicy) Instantiates aTokenizerFactory.TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenizerCharacterPolicy characterPolicy, opennlp.tools.tokenize.TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGenerator. -
Uses of TokenizerCharacterPolicy in opennlp.tools.tokenize.lang
Fields in opennlp.tools.tokenize.lang declared as TokenizerCharacterPolicyModifier and TypeFieldDescriptionstatic final TokenizerCharacterPolicyFactory.DEFAULT_CHARACTER_POLICYThe policy for languages without a dedicated one: ASCII letters and digits, no marks.Methods in opennlp.tools.tokenize.lang that return TokenizerCharacterPolicyModifier and TypeMethodDescriptionFactory.getCharacterPolicy(String languageCode) Gets the character policy for a language.static TokenizerCharacterPolicyFactory.policyOfLegacyExpression(String expression) Deprecated, for removal: This API element is subject to removal in a future version.Methods in opennlp.tools.tokenize.lang with parameters of type TokenizerCharacterPolicyModifier and TypeMethodDescriptionstatic StringFactory.legacyExpressionOf(TokenizerCharacterPolicy policy) Deprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.static PatternFactory.legacyPatternOf(TokenizerCharacterPolicy policy) Deprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.
PatternAPI and will be removed with it in 4.0.