Class TermVectorAnnotator
- All Implemented Interfaces:
opennlp.tools.document.DocumentAnnotator
TermVector per distinct term, carrying the term string, its occurrence count,
and (in full mode) the occurrence offsets.
Term identity comes from the configured normalizer: without one, the term is the
token's covered text; with a CharSequenceNormalizer, each token is normalized
on its own; with an OffsetAwareNormalizer, the document is normalized once and
each token span is mapped into the normalized text. Occurrence spans always point into
the original text. A token whose term is empty is omitted.
The layer is document-scoped: each TermVector
is a whole-document statistic, so the annotations carry no span of their own and the
occurrence offsets live inside the payload. The layer preserves first-occurrence
order: the first token of a term fixes its position in the layer.
The annotator holds no per-call state; it is as thread-safe as the normalizer it was built with.
- Since:
- 3.0.0
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic enumHow much eachTermVectorrecords. -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final opennlp.tools.document.LayerKey<opennlp.tools.termvector.TermVector> The key of the term vector layer this annotator provides: a document-scoped layer ofTermVectorvalues, one per distinct term. -
Constructor Summary
ConstructorsConstructorDescriptionInitializes afull modeannotator that groups tokens by their covered text as-is.Initializes an annotator that groups tokens by their covered text as-is.TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer) Initializes afull modeannotator that groups tokens by their per-token normalized form.TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer, TermVectorAnnotator.Mode mode) Initializes an annotator that groups tokens by their per-token normalized form.TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer) Initializes afull modeannotator that groups tokens by their normalized form through a whole-document alignment.TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer, TermVectorAnnotator.Mode mode) Initializes an annotator that groups tokens by their normalized form through a whole-document alignment. -
Method Summary
Modifier and TypeMethodDescriptionopennlp.tools.document.Documentannotate(opennlp.tools.document.Document document) Aggregates the token layer into theTERM_VECTORSlayer.Set<opennlp.tools.document.LayerKey<?>> provides()Set<opennlp.tools.document.LayerKey<?>> requires()toString()Returns the adapter's simple class name, which names it in pipeline validation messages.
-
Field Details
-
TERM_VECTORS
public static final opennlp.tools.document.LayerKey<opennlp.tools.termvector.TermVector> TERM_VECTORSThe key of the term vector layer this annotator provides: a document-scoped layer ofTermVectorvalues, one per distinct term.
-
-
Constructor Details
-
TermVectorAnnotator
public TermVectorAnnotator()Initializes afull modeannotator that groups tokens by their covered text as-is. -
TermVectorAnnotator
Initializes an annotator that groups tokens by their covered text as-is.- Parameters:
mode- How much eachTermVectorrecords. Must not benull.- Throws:
IllegalArgumentException- Thrown ifmodeisnull.
-
TermVectorAnnotator
public TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer) Initializes afull modeannotator that groups tokens by their per-token normalized form. This is the general path: the normalizer is applied to each token's covered text on its own, so anyCharSequenceNormalizerworks, including the folds that cannot report an alignment (case folding, NFC, accent folding, stemmer-backed normalizers). The occurrence spans stay the tokens' own spans in the original text.- Parameters:
normalizer- The normalizer that defines term identity, applied to each token's covered text. Must not benull.- Throws:
IllegalArgumentException- Thrown ifnormalizerisnull.
-
TermVectorAnnotator
public TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer, TermVectorAnnotator.Mode mode) Initializes an annotator that groups tokens by their per-token normalized form. This is the general path: the normalizer is applied to each token's covered text on its own, so anyCharSequenceNormalizerworks, including the folds that cannot report an alignment (case folding, NFC, accent folding, stemmer-backed normalizers). The occurrence spans stay the tokens' own spans in the original text.- Parameters:
normalizer- The normalizer that defines term identity, applied to each token's covered text. Must not benull.mode- How much eachTermVectorrecords. Must not benull.- Throws:
IllegalArgumentException- Thrown ifnormalizerormodeisnull.
-
TermVectorAnnotator
public TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer) Initializes afull modeannotator that groups tokens by their normalized form through a whole-document alignment. Prefer theplain-normalizer constructoras the general path; this one only accepts alignment-reporting normalizers but can see across token boundaries, for example a whitespace collapse spanning two tokens.- Parameters:
normalizer- The normalizer that defines term identity, applied to the whole document text so token spans can be mapped into the normalized form through its alignment. Must not benull.- Throws:
IllegalArgumentException- Thrown ifnormalizerisnull.
-
TermVectorAnnotator
public TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer, TermVectorAnnotator.Mode mode) Initializes an annotator that groups tokens by their normalized form through a whole-document alignment. Prefer theplain-normalizer constructoras the general path; this one only accepts alignment-reporting normalizers but can see across token boundaries, for example a whitespace collapse spanning two tokens.- Parameters:
normalizer- The normalizer that defines term identity, applied to the whole document text so token spans can be mapped into the normalized form through its alignment. Must not benull.mode- How much eachTermVectorrecords. Must not benull.- Throws:
IllegalArgumentException- Thrown ifnormalizerormodeisnull.
-
-
Method Details
-
annotate
public opennlp.tools.document.Document annotate(opennlp.tools.document.Document document) Aggregates the token layer into theTERM_VECTORSlayer. A present-but-empty token layer yields a present-but-empty term vector layer.- Specified by:
annotatein interfaceopennlp.tools.document.DocumentAnnotator- Parameters:
document- The document to annotate. Must not benulland must contain theLayers.TOKENSlayer.- Returns:
- A new
Documentwith theTERM_VECTORSlayer added. Nevernull. - Throws:
IllegalArgumentException- Thrown ifdocumentisnull, lacks theLayers.TOKENSlayer, or already carries theTERM_VECTORSlayer.
-
requires
- Specified by:
requiresin interfaceopennlp.tools.document.DocumentAnnotator
-
provides
- Specified by:
providesin interfaceopennlp.tools.document.DocumentAnnotator
-
toString
-