Class EmojiCharSequenceNormalizer

java.lang.Object
opennlp.tools.util.normalizer.EmojiCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer

public class EmojiCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.CharSequenceNormalizer
A CharSequenceNormalizer implementation that replaces each run of complete emoji with one space.

An emoji is a fully-qualified or minimally-qualified sequence of Unicode Emoji 18.0 in the sense of UTS #51: a single emoji, an emoji with U+FE0F, a flag, a keycap, a skin tone modifier sequence, a ZWJ sequence or a tag sequence, as listed in emoji-test.txt. A minimally-qualified ZWJ sequence lacks U+FE0F after a later element only, such as a woman raising hand written without U+FE0F after U+2640. Adjacent emoji form one run and become one space; one redundant U+FE0F right after an emoji is removed with it. Everything else is kept: a symbol with text presentation such as U+2764 without U+FE0F, an unqualified sequence starting with such a symbol, letters and ideographs from other planes, private use characters, unpaired surrogates, and an emoji with a stray joiner, modifier, selector, flag letter or tag next to it, which is kept as a whole instead of leaving a fragment behind.

Since 3.0.0 only complete emoji sequences are removed. Under CompatibilityMode.LEGACY the earlier output is produced for language detector models trained with it: each maximal run of ASCII hyphens, characters from U+E000 to U+FFFF (private use characters, CJK compatibility ideographs, Arabic and Hebrew presentation forms, fullwidth and halfwidth forms, variation selectors), supplementary characters up to U+10FC00 whether they are emoji or not, and unpaired surrogates from U+D83C on becomes one space, while emoji in the Basic Multilingual Plane such as U+231A are kept.

See Also:
  • Constructor Details

    • EmojiCharSequenceNormalizer

      public EmojiCharSequenceNormalizer()
  • Method Details

    • getInstance

      public static EmojiCharSequenceNormalizer getInstance()
      Returns the shared, stateless instance.
      Returns:
      the shared, stateless instance
    • normalize

      public CharSequence normalize(CharSequence text)
      Specified by:
      normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer