Class WhitespaceTokenizer
java.lang.Object
opennlp.tools.tokenize.WhitespaceTokenizer
- All Implemented Interfaces:
Tokenizer
A basic
Tokenizer implementation which performs tokenization
using white spaces.
To obtain an instance of this tokenizer use the static final
INSTANCE field. Use newInstance(boolean) for a tokenizer with a
non-default setKeepNewLines(boolean) setting.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final WhitespaceTokenizerUse this static reference to retrieve an instance of theWhitespaceTokenizer. -
Method Summary
Modifier and TypeMethodDescriptionstatic WhitespaceTokenizernewInstance(boolean keepNewLines) Creates a tokenizer that is not shared withINSTANCE, for a caller that setssetKeepNewLines(boolean)without touching other users of the shared instance.voidsetKeepNewLines(boolean keepNewLines) Switches whether to keep new lines or not.String[]Splits a string into its atomic parts.Span[]Finds the boundaries of atomic parts in a string.
-
Field Details
-
INSTANCE
Use this static reference to retrieve an instance of theWhitespaceTokenizer.This instance is shared by every caller in the process, so it must not be reconfigured: calling
setKeepNewLines(boolean)on it changes the output of all other users and is not safely published to other threads. UsenewInstance(boolean)to obtain a tokenizer with a different setting.
-
-
Method Details
-
newInstance
Creates a tokenizer that is not shared withINSTANCE, for a caller that setssetKeepNewLines(boolean)without touching other users of the shared instance.- Parameters:
keepNewLines-trueif line separators are returned as tokens.- Returns:
- A new
WhitespaceTokenizerwith that setting.
-
tokenizePos
Description copied from interface:TokenizerFinds the boundaries of atomic parts in a string.- Parameters:
d- The string to be tokenized.- Returns:
- The
spans (offsets intofor each token as the individuals array elements.s)
-
tokenize
-
setKeepNewLines
public void setKeepNewLines(boolean keepNewLines) Switches whether to keep new lines or not.- Parameters:
keepNewLines-Trueif new lines are kept,falseotherwise.
-