Class StringUtil
java.lang.Object
opennlp.tools.util.StringUtil
-
Field Summary
FieldsModifier and TypeFieldDescriptionThe immutable list of the ten ASCII digit strings"0"to"9", indexed by digit value. -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionstatic voidcomputeShortestEditScript(String wordForm, String lemma, int[][] distance, StringBuffer permutations) Computes the Shortest Edit Script (SES) to convert a word into its lemma.static booleancontainsAsciiDigit(CharSequence input) Tests whetherinputcontains an ASCII digit,0to9.static booleanTests whetherinputcontains an ASCII capital letter,AtoZ.static StringdecodeShortestEditScript(String wordForm, String permutations) Reads the predicted Shortest Edit Script (SES) by a lemmatizer model and applies the permutations to obtain the lemma from thewordForm.static intendOfAsciiDigits(CharSequence text, int from) Finds the end of the run of ASCII digits that starts atfrom.static StringgetShortestEditScript(String wordForm, String lemma) static intindexOfLineTerminator(CharSequence text, int from) Finds the first line terminator at or afterfrom, as defined byisLineTerminator(int).static booleanisAsciiDigit(int codePoint) Tests for an ASCII digit,0to9.static booleanisAsciiLetter(int codePoint) Tests for an ASCII letter,atozorAtoZ.static booleanisAsciiLowerCase(int codePoint) Tests for an ASCII lowercase letter,atoz.static booleanisAsciiWhitespace(int codePoint) Tests for an ASCII whitespace character: space, tab, line feed, vertical tab, form feed or carriage return.static booleanisBlank(CharSequence theString) Determines whether aCharSequenceis blank: empty, or made up entirely of code points thatisWhitespace(int)accepts.static booleanisEmpty(CharSequence theString) Determines whether aCharSequenceis empty.static booleanisLineTerminator(int codePoint) Tests for a line terminator: line feedU+000A, carriage returnU+000D, next lineU+0085, line separatorU+2028, or paragraph separatorU+2029.static booleanisUnicodeBlank(CharSequence input) truewheninputisnull, empty, or consists only of UnicodeWhite_Spacecode points.static booleanisUnicodeWhitespace(char charCode) Determines if the specifiedCharacteris a whitespace under the UnicodeWhite_Spaceproperty; delegates toUnicodeWhitespace.isWhitespace(int).static booleanisUnicodeWhitespace(int charCode) Determines if the specified code point is a whitespace under the UnicodeWhite_Spaceproperty; delegates toUnicodeWhitespace.isWhitespace(int).static booleanisUnpairedSurrogate(int codePoint) Checks whether a code point taken from aCharSequenceis a lone surrogate,U+D800toU+DFFF, rather than a character.static booleanisWhitespace(char charCode) Determines if the specifiedCharacteris a whitespace under the activeWhitespaceMode: the UnicodeWhite_Spaceproperty by default, or, when the "opennlp.whitespace.mode" system property selectsWhitespaceMode.LEGACY, the union ofCharacter.isWhitespace(int)and the UnicodeZscategory (Character.SPACE_SEPARATOR) that OpenNLP 1.x and 2.x used.static booleanisWhitespace(int charCode) Determines if the specified code point is a whitespace under the activeWhitespaceMode; seeisWhitespace(char)for details.static int[][]levenshteinDistance(String wordForm, String lemma) Computes the Levenshtein distance of two strings in a matrix.static String[]split(CharSequence input, char separator) Splitsinputaround literal occurrences ofseparator, with the same result asString.split(String)called with the single-character patternCharacter.toString(separator)and a limit of0: leading and middle empty fields are kept, trailing empty fields are removed, and empty input yields an array containing one empty string.static String[]split(CharSequence input, char separator, int limit) Splitsinputaround literal occurrences ofseparator, with the same limit contract asString.split(String, int)applied to the single-character patternCharacter.toString(separator): a positivelimitapplies at mostlimit - 1splits and keeps trailing empty fields;0applies unlimited splits and removes trailing empty fields, exactly likesplit(CharSequence, char); a negativelimitapplies unlimited splits and keeps trailing empty fields.static String[]splitNonEmpty(CharSequence input, char... separators) Splitsinputaround literal occurrences of any of theseparatorsand drops the empty fields that leading, trailing or adjacent separators would produce, so a run of separators acts as one and input made only of separators yields an empty array.static String[]Splitsinputon runs of UnicodeWhite_Space.static String[]splitOnWhitespace(CharSequence input) static booleanTests whetherinputstarts with a byte order mark, U+FEFF, which a UTF-8 or UTF-16 text file may carry as its first character.static StringstripByteOrderMark(String input) Removes one leading byte order mark, U+FEFF, frominput.static StringtoLowerCase(CharSequence string) Converts aCharSequenceto lower case, independent of the currentLocaleviaCharacter.toLowerCase(int)which uses mapping information from the UnicodeData file.static CharBuffertoLowerCaseCharBuffer(CharSequence sequence) static StringtoUpperCase(CharSequence string) Converts aCharSequenceto upper case, independent of the currentLocaleviaCharacter.toUpperCase(int)which uses mapping information from the UnicodeData file.static StringTrims leading and trailing runs of UnicodeWhite_Space, the same setsplitOnUnicodeWhitespace(CharSequence)breaks terms on.
-
Field Details
-
ASCII_DIGIT_STRINGS
-
-
Constructor Details
-
StringUtil
public StringUtil()
-
-
Method Details
-
isWhitespace
public static boolean isWhitespace(char charCode) Determines if the specifiedCharacteris a whitespace under the activeWhitespaceMode: the UnicodeWhite_Spaceproperty by default, or, when the "opennlp.whitespace.mode" system property selectsWhitespaceMode.LEGACY, the union ofCharacter.isWhitespace(int)and the UnicodeZscategory (Character.SPACE_SEPARATOR) that OpenNLP 1.x and 2.x used.Tokenization, corpus format parsing, and feature generation all resolve whitespace through this method, so they share one
WhitespaceModefor the life of the process; useisUnicodeWhitespace(char)directly where the Unicode definition should apply unconditionally, such as user-text normalization with no trained-model dependency.- Parameters:
charCode- The character to check.- Returns:
trueifcharCoderepresents a white space under the activeWhitespaceMode,falseotherwise.
-
isWhitespace
public static boolean isWhitespace(int charCode) Determines if the specified code point is a whitespace under the activeWhitespaceMode; seeisWhitespace(char)for details.- Parameters:
charCode- An int representation of a character to check.- Returns:
trueifcharCoderepresents a white space under the activeWhitespaceMode,falseotherwise.
-
isUnicodeWhitespace
public static boolean isUnicodeWhitespace(char charCode) Determines if the specifiedCharacteris a whitespace under the UnicodeWhite_Spaceproperty; delegates toUnicodeWhitespace.isWhitespace(int). This is also whatisWhitespace(char)resolves to under the defaultWhitespaceMode.- Parameters:
charCode- The character to check.- Returns:
trueifcharCodehas theWhite_Spaceproperty,falseotherwise.
-
isUnicodeWhitespace
public static boolean isUnicodeWhitespace(int charCode) Determines if the specified code point is a whitespace under the UnicodeWhite_Spaceproperty; delegates toUnicodeWhitespace.isWhitespace(int). This is also whatisWhitespace(int)resolves to under the defaultWhitespaceMode.- Parameters:
charCode- An int representation of a character to check.- Returns:
trueifcharCodehas theWhite_Spaceproperty,falseotherwise.
-
splitOnUnicodeWhitespace
Splitsinputon runs of UnicodeWhite_Space. Leading and trailing runs are ignored, so whitespace-only input yields an empty array.- Parameters:
input- The text to split. Must not benull.- Returns:
- The non-whitespace terms in order.
- Throws:
IllegalArgumentException- Ifinputisnull.
-
splitOnWhitespace
Splitsinputon runs of whitespace under the activeWhitespaceMode, seeisWhitespace(char). Leading and trailing runs are ignored, so whitespace-only input yields an empty array. UsesplitOnUnicodeWhitespace(CharSequence)where the Unicode definition should apply regardless of the mode.- Parameters:
input- The text to split. Must not benull.- Returns:
- The non-whitespace fields in order.
- Throws:
IllegalArgumentException- Ifinputisnull.
-
split
Splitsinputaround literal occurrences ofseparator, with the same result asString.split(String)called with the single-character patternCharacter.toString(separator)and a limit of0: leading and middle empty fields are kept, trailing empty fields are removed, and empty input yields an array containing one empty string. The scan compares characters, soseparatoris always treated as a literal, including when it is a regular-expression metacharacter such as'.','$'or'|', and supplementary code points in the input pass through untouched.- Parameters:
input- The text to split. Must not benull.separator- The literal field separator. Must not be a surrogate.- Returns:
- The fields in order.
- Throws:
IllegalArgumentException- Ifinputisnullorseparatoris a surrogate.
-
split
Splitsinputaround literal occurrences ofseparator, with the same limit contract asString.split(String, int)applied to the single-character patternCharacter.toString(separator): a positivelimitapplies at mostlimit - 1splits and keeps trailing empty fields;0applies unlimited splits and removes trailing empty fields, exactly likesplit(CharSequence, char); a negativelimitapplies unlimited splits and keeps trailing empty fields. Empty input yields an array containing one empty string at every limit. The scan compares characters, soseparatoris always treated as a literal, including when it is a regular-expression metacharacter, and supplementary code points in the input pass through untouched.- Parameters:
input- The text to split. Must not benull.separator- The literal field separator. Must not be a surrogate, so that a supplementary code point ininputis never split in half.limit- The maximum number of fields, or0for the default behavior, or negative for no limit.- Returns:
- The fields in order.
- Throws:
IllegalArgumentException- Ifinputisnullorseparatoris a surrogate.
-
startsWithByteOrderMark
Tests whetherinputstarts with a byte order mark, U+FEFF, which a UTF-8 or UTF-16 text file may carry as its first character.- Parameters:
input- The text to test. Must not benull.- Returns:
trueif the first character ofinputis U+FEFF.- Throws:
IllegalArgumentException- Ifinputisnull.
-
stripByteOrderMark
Removes one leading byte order mark, U+FEFF, frominput. A U+FEFF anywhere else is left in place, since there it is a zero width no-break space and part of the text.- Parameters:
input- The text to strip. Must not benull.- Returns:
inputwithout its leading byte order mark, orinputitself if it has none.- Throws:
IllegalArgumentException- Ifinputisnull.
-
splitNonEmpty
Splitsinputaround literal occurrences of any of theseparatorsand drops the empty fields that leading, trailing or adjacent separators would produce, so a run of separators acts as one and input made only of separators yields an empty array. The scan compares characters, so every separator is a literal, including a regular-expression metacharacter, and supplementary code points in the input pass through untouched.- Parameters:
input- The text to split. Must not benull.separators- The literal field separators. Must not benullor empty, and must not contain a surrogate.- Returns:
- The non-empty fields in order.
- Throws:
IllegalArgumentException- Ifinputorseparatorsisnull,separatorsis empty, or a separator is a surrogate.
-
containsAsciiUpperCase
Tests whetherinputcontains an ASCII capital letter,AtoZ. Capital letters outside ASCII do not count.- Parameters:
input- The text to check. Must not benull.- Returns:
trueif at least one character is an ASCII capital letter.- Throws:
IllegalArgumentException- Ifinputisnull.
-
containsAsciiDigit
Tests whetherinputcontains an ASCII digit,0to9. Digits outside ASCII do not count.- Parameters:
input- The text to check. Must not benull.- Returns:
trueif at least one character is an ASCII digit.- Throws:
IllegalArgumentException- Ifinputisnull.
-
isAsciiLetter
public static boolean isAsciiLetter(int codePoint) Tests for an ASCII letter,atozorAtoZ.- Parameters:
codePoint- The code point to test.- Returns:
trueifcodePointis an ASCII letter.
-
isAsciiLowerCase
public static boolean isAsciiLowerCase(int codePoint) Tests for an ASCII lowercase letter,atoz.- Parameters:
codePoint- The code point to test.- Returns:
trueifcodePointis an ASCII lowercase letter.
-
isAsciiDigit
public static boolean isAsciiDigit(int codePoint) Tests for an ASCII digit,0to9.- Parameters:
codePoint- The code point to test.- Returns:
trueifcodePointis an ASCII digit.
-
isAsciiWhitespace
public static boolean isAsciiWhitespace(int codePoint) Tests for an ASCII whitespace character: space, tab, line feed, vertical tab, form feed or carriage return. This is the set matched by\sin aPatterncompiled withoutUNICODE_CHARACTER_CLASS.- Parameters:
codePoint- The code point to test.- Returns:
trueifcodePointis an ASCII whitespace character.
-
endOfAsciiDigits
Finds the end of the run of ASCII digits that starts atfrom.- Parameters:
text- The text to scan. Must not benull.from- The offset the run starts at, between0andtext.length().- Returns:
- The offset after the last digit of the run, or
fromif no digit is there. - Throws:
IllegalArgumentException- Iftextisnullorfromis out of range.
-
isLineTerminator
public static boolean isLineTerminator(int codePoint) Tests for a line terminator: line feedU+000A, carriage returnU+000D, next lineU+0085, line separatorU+2028, or paragraph separatorU+2029. This is the setPatterntreats as line terminators outsideUNIX_LINESmode. It is narrower thanUnicodeWhitespace.lineBreakCodePointSet(), which also holds vertical tabU+000Band form feedU+000C.- Parameters:
codePoint- The code point to test.- Returns:
trueifcodePointends a line.
-
isUnpairedSurrogate
public static boolean isUnpairedSurrogate(int codePoint) Checks whether a code point taken from aCharSequenceis a lone surrogate,U+D800toU+DFFF, rather than a character.String.codePoints()andCharacter.codePointAt(CharSequence, int)return a surrogate only when it is not part of a valid high and low surrogate pair.- Parameters:
codePoint- The code point to test.- Returns:
trueifcodePointis in the surrogate range.
-
indexOfLineTerminator
Finds the first line terminator at or afterfrom, as defined byisLineTerminator(int).- Parameters:
text- The text to scan. Must not benull.from- The offset to start at, between0andtext.length().- Returns:
- The offset of the first line terminator, or
-1if there is none. - Throws:
IllegalArgumentException- Iftextisnullorfromis out of range.
-
trimUnicodeWhitespace
Trims leading and trailing runs of UnicodeWhite_Space, the same setsplitOnUnicodeWhitespace(CharSequence)breaks terms on.- Parameters:
input- The text to trim. Must not benull.- Returns:
- The trimmed string; may be empty when
inputis whitespace-only. - Throws:
IllegalArgumentException- Ifinputisnull.
-
isUnicodeBlank
truewheninputisnull, empty, or consists only of UnicodeWhite_Spacecode points.- Parameters:
input- The text to test;nullis treated as blank.- Returns:
trueif there is no non-whitespace code point.
-
toLowerCase
Converts aCharSequenceto lower case, independent of the currentLocaleviaCharacter.toLowerCase(int)which uses mapping information from the UnicodeData file.- Parameters:
string- TheCharSequenceto transform.- Returns:
- The lower-cased String.
-
toLowerCaseCharBuffer
-
toUpperCase
Converts aCharSequenceto upper case, independent of the currentLocaleviaCharacter.toUpperCase(int)which uses mapping information from the UnicodeData file.- Parameters:
string- TheCharSequenceto transform.- Returns:
- The upper-cased String
-
isEmpty
Determines whether aCharSequenceis empty.- Parameters:
theString- TheCharSequenceto examine. Must not benull.- Returns:
trueifCharSequence.length()is0, otherwisefalse.- Throws:
IllegalArgumentException- Thrown iftheStringisnull.- Since:
- 1.5.1
-
isBlank
Determines whether aCharSequenceis blank: empty, or made up entirely of code points thatisWhitespace(int)accepts. UnlikeString.isBlank(), this follows the toolkit's whitespace definition, which includes the no-break spaces the JDK predicate leaves out, so a value spelled entirely from them cannot pass a blank check as content. UnlikeisUnicodeBlank(CharSequence), it resolves through the activeWhitespaceModeand does not treatnullas blank.- Parameters:
theString- TheCharSequenceto examine. Must not benull.- Returns:
trueiftheStringis empty or all whitespace.- Throws:
IllegalArgumentException- Thrown iftheStringisnull.
-
levenshteinDistance
Computes the Levenshtein distance of two strings in a matrix.Based on this pseudo-code which in turn is based on the paper Wagner, Robert A.; Fischer, Michael J. (1974), "The String-to-String Correction Problem", Journal of the ACM 21 (1): 168-173
- Parameters:
wordForm- The form as input.lemma- The target lemma.- Returns:
- A 2-dimensional Levenshtein distance matrix.
-
computeShortestEditScript
public static void computeShortestEditScript(String wordForm, String lemma, int[][] distance, StringBuffer permutations) Computes the Shortest Edit Script (SES) to convert a word into its lemma. This is based on Chrupala's PhD thesis (2008).- Parameters:
wordForm- The token.lemma- The target lemma.distance- A 2-dimensional Levenshtein distance matrix.permutations- The number of permutations.
-
decodeShortestEditScript
Reads the predicted Shortest Edit Script (SES) by a lemmatizer model and applies the permutations to obtain the lemma from thewordForm.- Parameters:
wordForm- The wordForm as input.permutations- The permutations predicted by the lemmatizer model.- Returns:
- The decoded lemma.
-
getShortestEditScript
-