Class WordDelimiterGraphFilter
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.TokenFilter
-
- org.apache.lucene.analysis.miscellaneous.WordDelimiterGraphFilter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class WordDelimiterGraphFilter extends TokenFilter
Splits words into subwords and performs optional transformations on subword groups, producing a correct token graph so that e.g.PhraseQuery
can work correctly when this filter is used in the search-time analyzer. Unlike the deprecatedWordDelimiterFilter
, this token filter produces a correct token graph as output. However, it cannot consume an input token graph correctly. Processing is suppressed byKeywordAttribute.isKeyword()
=true.Words are split into subwords with the following rules:
- split on intra-word delimiters (by default, all non alpha-numeric
characters):
"Wi-Fi"
→"Wi", "Fi"
- split on case transitions:
"PowerShot"
→"Power", "Shot"
- split on letter-number transitions:
"SD500"
→"SD", "500"
- leading and trailing intra-word delimiters on each subword are ignored:
"//hello---there, 'dude'"
→"hello", "there", "dude"
- trailing "'s" are removed for each subword:
"O'Neil's"
→"O", "Neil"
- Note: this step isn't performed in a separate filter because of possible subword combinations.
- If no CATENATE option is set, then no subword combinations are generated:
"PowerShot"
→0:"Power", 1:"Shot"
(0 and 1 are the token positions) - CATENATE_WORDS means that in addition to the subwords, maximum runs of
non-numeric subwords are catenated and produced at the same position of the
last subword in the run:
"PowerShot"
→0:"Power", 1:"Shot" 1:"PowerShot"
"A's+B's&C's"
>0:"A", 1:"B", 2:"C", 2:"ABC"
"Super-Duper-XL500-42-AutoCoder!"
→0:"Super", 1:"Duper", 2:"XL", 2:"SuperDuperXL", 3:"500" 4:"42", 5:"Auto", 6:"Coder", 6:"AutoCoder"
- CATENATE_NUMBERS works like CATENATE_WORDS, but for adjacent digit sequences.
- CATENATE_ALL smushes together all the token parts without distinguishing numbers and words.
WordDelimiterGraphFilter
is to help match words with different subword delimiters. For example, if the source text contained "wi-fi" one may want "wifi" "WiFi" "wi-fi" "wi+fi" queries to all match. One way of doing so is to specify CATENATE options in the analyzer used for indexing, and not in the analyzer used for querying. Given that the currentStandardTokenizer
immediately removes many intra-word delimiters, it is recommended that this filter be used after a tokenizer that does not do this (such asWhitespaceTokenizer
).
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private class
WordDelimiterGraphFilter.PositionSorter
(package private) class
WordDelimiterGraphFilter.WordDelimiterConcatenation
A WDF concatenated 'run'-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description private int
accumPosInc
private boolean
adjustingOffsets
private boolean
adjustInternalOffsets
private int
bufferedLen
private int[]
bufferedParts
private int
bufferedPos
private char[][]
bufferedTermParts
static int
CATENATE_ALL
Causes all subword parts to be catenated:static int
CATENATE_NUMBERS
Causes maximum runs of number parts to be catenated:static int
CATENATE_WORDS
Causes maximum runs of word parts to be catenated:private WordDelimiterGraphFilter.WordDelimiterConcatenation
concat
private WordDelimiterGraphFilter.WordDelimiterConcatenation
concatAll
private int
flags
static int
GENERATE_NUMBER_PARTS
Causes number subwords to be generated:static int
GENERATE_WORD_PARTS
Causes parts of words to be generated:static int
IGNORE_KEYWORDS
Suppresses processing terms withKeywordAttribute.isKeyword()
=true.private WordDelimiterIterator
iterator
private KeywordAttribute
keywordAttribute
private int
lastConcatCount
private int
lastStartOffset
private OffsetAttribute
offsetAttribute
private PositionIncrementAttribute
posIncAttribute
private PositionLengthAttribute
posLenAttribute
static int
PRESERVE_ORIGINAL
Causes original words are preserved and added to the subword list (Defaults to false)(package private) CharArraySet
protWords
If not null is the set of tokens to protect from being delimitedprivate int
savedEndOffset
private int
savedStartOffset
private AttributeSource.State
savedState
private char[]
savedTermBuffer
private int
savedTermLength
(package private) WordDelimiterGraphFilter.PositionSorter
sorter
static int
SPLIT_ON_CASE_CHANGE
Causes lowercase -> uppercase transition to start a new subword.static int
SPLIT_ON_NUMERICS
If not set, causes numeric changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens).static int
STEM_ENGLISH_POSSESSIVE
Causes trailing "'s" to be removed for each subwordprivate CharTermAttribute
termAttribute
private int
wordPos
-
Fields inherited from class org.apache.lucene.analysis.TokenFilter
input
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description WordDelimiterGraphFilter(TokenStream in, boolean adjustInternalOffsets, byte[] charTypeTable, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterGraphFilterWordDelimiterGraphFilter(TokenStream in, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterGraphFilter usingWordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE
as its charTypeTable
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) void
buffer(char[] termPart, int startPos, int endPos, int startPart, int endPart)
a null termPart means it's a simple slice of the original term(package private) void
buffer(int startPos, int endPos, int startPart, int endPart)
startPos, endPos -> graph start/end position startPart, endPart -> slice of the original term for this partprivate void
bufferWordParts()
Iterates all words parts and concatenations, buffering up the term parts we should return.private void
concatenate(WordDelimiterGraphFilter.WordDelimiterConcatenation concatenation)
Concatenates the saved buffer to the given WordDelimiterConcatenationstatic java.lang.String
flagsToString(int flags)
Returns string representation of configuration flagsprivate void
flushConcatenation(WordDelimiterGraphFilter.WordDelimiterConcatenation concat)
Flushes the given WordDelimiterConcatenation by either writing its concat and then clearing, or just clearing.private boolean
has(int flag)
Determines whether the given flag is setboolean
incrementToken()
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.void
reset()
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.private void
saveState()
Saves the existing attribute statesprivate boolean
shouldConcatenate(int wordType)
Determines whether to concatenate a word or number if the current word is the given typeprivate boolean
shouldGenerateParts(int wordType)
Determines whether a word/number part should be generated for a word of the given typejava.lang.String
toString()
Returns a string consisting of the class's simple name, the hex representation of the identity hash code, and the current reflection of all attributes.-
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState
-
-
-
-
Field Detail
-
GENERATE_WORD_PARTS
public static final int GENERATE_WORD_PARTS
Causes parts of words to be generated:"PowerShot" => "Power" "Shot"
- See Also:
- Constant Field Values
-
GENERATE_NUMBER_PARTS
public static final int GENERATE_NUMBER_PARTS
Causes number subwords to be generated:"500-42" => "500" "42"
- See Also:
- Constant Field Values
-
CATENATE_WORDS
public static final int CATENATE_WORDS
Causes maximum runs of word parts to be catenated:"wi-fi" => "wifi"
- See Also:
- Constant Field Values
-
CATENATE_NUMBERS
public static final int CATENATE_NUMBERS
Causes maximum runs of number parts to be catenated:"500-42" => "50042"
- See Also:
- Constant Field Values
-
CATENATE_ALL
public static final int CATENATE_ALL
Causes all subword parts to be catenated:"wi-fi-4000" => "wifi4000"
- See Also:
- Constant Field Values
-
PRESERVE_ORIGINAL
public static final int PRESERVE_ORIGINAL
Causes original words are preserved and added to the subword list (Defaults to false)"500-42" => "500" "42" "500-42"
- See Also:
- Constant Field Values
-
SPLIT_ON_CASE_CHANGE
public static final int SPLIT_ON_CASE_CHANGE
Causes lowercase -> uppercase transition to start a new subword.- See Also:
- Constant Field Values
-
SPLIT_ON_NUMERICS
public static final int SPLIT_ON_NUMERICS
If not set, causes numeric changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens).- See Also:
- Constant Field Values
-
STEM_ENGLISH_POSSESSIVE
public static final int STEM_ENGLISH_POSSESSIVE
Causes trailing "'s" to be removed for each subword"O'Neil's" => "O", "Neil"
- See Also:
- Constant Field Values
-
IGNORE_KEYWORDS
public static final int IGNORE_KEYWORDS
Suppresses processing terms withKeywordAttribute.isKeyword()
=true.- See Also:
- Constant Field Values
-
protWords
final CharArraySet protWords
If not null is the set of tokens to protect from being delimited
-
flags
private final int flags
-
bufferedParts
private int[] bufferedParts
-
bufferedLen
private int bufferedLen
-
bufferedPos
private int bufferedPos
-
bufferedTermParts
private char[][] bufferedTermParts
-
termAttribute
private final CharTermAttribute termAttribute
-
keywordAttribute
private final KeywordAttribute keywordAttribute
-
offsetAttribute
private final OffsetAttribute offsetAttribute
-
posIncAttribute
private final PositionIncrementAttribute posIncAttribute
-
posLenAttribute
private final PositionLengthAttribute posLenAttribute
-
iterator
private final WordDelimiterIterator iterator
-
concat
private final WordDelimiterGraphFilter.WordDelimiterConcatenation concat
-
adjustInternalOffsets
private final boolean adjustInternalOffsets
-
lastConcatCount
private int lastConcatCount
-
concatAll
private final WordDelimiterGraphFilter.WordDelimiterConcatenation concatAll
-
accumPosInc
private int accumPosInc
-
savedTermBuffer
private char[] savedTermBuffer
-
savedTermLength
private int savedTermLength
-
savedStartOffset
private int savedStartOffset
-
savedEndOffset
private int savedEndOffset
-
savedState
private AttributeSource.State savedState
-
lastStartOffset
private int lastStartOffset
-
adjustingOffsets
private boolean adjustingOffsets
-
wordPos
private int wordPos
-
sorter
final WordDelimiterGraphFilter.PositionSorter sorter
-
-
Constructor Detail
-
WordDelimiterGraphFilter
public WordDelimiterGraphFilter(TokenStream in, boolean adjustInternalOffsets, byte[] charTypeTable, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterGraphFilter- Parameters:
in
- TokenStream to be filteredadjustInternalOffsets
- if the offsets of partial terms should be adjustedcharTypeTable
- table containing character typesconfigurationFlags
- Flags configuring the filterprotWords
- If not null is the set of tokens to protect from being delimited
-
WordDelimiterGraphFilter
public WordDelimiterGraphFilter(TokenStream in, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterGraphFilter usingWordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE
as its charTypeTable- Parameters:
in
- TokenStream to be filteredconfigurationFlags
- Flags configuring the filterprotWords
- If not null is the set of tokens to protect from being delimited
-
-
Method Detail
-
bufferWordParts
private void bufferWordParts() throws java.io.IOException
Iterates all words parts and concatenations, buffering up the term parts we should return.- Throws:
java.io.IOException
-
incrementToken
public boolean incrementToken() throws java.io.IOException
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
java.io.IOException
-
reset
public void reset() throws java.io.IOException
Description copied from class:TokenFilter
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call
super.reset()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on further usage).NOTE: The default implementation chains the call to the input TokenStream, so be sure to call
super.reset()
when overriding this method.- Overrides:
reset
in classTokenFilter
- Throws:
java.io.IOException
-
buffer
void buffer(int startPos, int endPos, int startPart, int endPart)
startPos, endPos -> graph start/end position startPart, endPart -> slice of the original term for this part
-
buffer
void buffer(char[] termPart, int startPos, int endPos, int startPart, int endPart)
a null termPart means it's a simple slice of the original term
-
saveState
private void saveState()
Saves the existing attribute states
-
flushConcatenation
private void flushConcatenation(WordDelimiterGraphFilter.WordDelimiterConcatenation concat)
Flushes the given WordDelimiterConcatenation by either writing its concat and then clearing, or just clearing.- Parameters:
concat
- WordDelimiterConcatenation that will be flushed
-
shouldConcatenate
private boolean shouldConcatenate(int wordType)
Determines whether to concatenate a word or number if the current word is the given type- Parameters:
wordType
- Type of the current word used to determine if it should be concatenated- Returns:
true
if concatenation should occur,false
otherwise
-
shouldGenerateParts
private boolean shouldGenerateParts(int wordType)
Determines whether a word/number part should be generated for a word of the given type- Parameters:
wordType
- Type of the word used to determine if a word/number part should be generated- Returns:
true
if a word/number part should be generated,false
otherwise
-
concatenate
private void concatenate(WordDelimiterGraphFilter.WordDelimiterConcatenation concatenation)
Concatenates the saved buffer to the given WordDelimiterConcatenation- Parameters:
concatenation
- WordDelimiterConcatenation to concatenate the buffer to
-
has
private boolean has(int flag)
Determines whether the given flag is set- Parameters:
flag
- Flag to see if set- Returns:
true
if flag is set
-
flagsToString
public static java.lang.String flagsToString(int flags)
Returns string representation of configuration flags
-
toString
public java.lang.String toString()
Description copied from class:AttributeSource
Returns a string consisting of the class's simple name, the hex representation of the identity hash code, and the current reflection of all attributes.- Overrides:
toString
in classAttributeSource
- See Also:
AttributeSource.reflectAsString(boolean)
-
-