Class WikipediaTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.wikipedia.WikipediaTokenizer
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class WikipediaTokenizer extends Tokenizer
Extension of StandardTokenizer that is aware of Wikipedia syntax. It is based off of the Wikipedia tutorial available at http://en.wikipedia.org/wiki/Wikipedia:Tutorial, but it may not be complete.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description static int
ACRONYM_ID
static int
ALPHANUM_ID
static int
APOSTROPHE_ID
static java.lang.String
BOLD
static int
BOLD_ID
static java.lang.String
BOLD_ITALICS
static int
BOLD_ITALICS_ID
static int
BOTH
Output the both the untokenized token and the splitsstatic java.lang.String
CATEGORY
static int
CATEGORY_ID
static java.lang.String
CITATION
static int
CITATION_ID
static int
CJ_ID
static int
COMPANY_ID
static int
EMAIL_ID
static java.lang.String
EXTERNAL_LINK
static int
EXTERNAL_LINK_ID
static java.lang.String
EXTERNAL_LINK_URL
static int
EXTERNAL_LINK_URL_ID
private boolean
first
private FlagsAttribute
flagsAtt
static java.lang.String
HEADING
static int
HEADING_ID
static int
HOST_ID
static java.lang.String
INTERNAL_LINK
static int
INTERNAL_LINK_ID
static java.lang.String
ITALICS
static int
ITALICS_ID
static int
NUM_ID
private OffsetAttribute
offsetAtt
private PositionIncrementAttribute
posIncrAtt
private WikipediaTokenizerImpl
scanner
A private instance of the JFlex-constructed scannerstatic java.lang.String
SUB_HEADING
static int
SUB_HEADING_ID
private CharTermAttribute
termAtt
static java.lang.String[]
TOKEN_TYPES
String token types that correspond to token type int constantsprivate int
tokenOutput
private java.util.Iterator<AttributeSource.State>
tokens
static int
TOKENS_ONLY
Only output tokensprivate TypeAttribute
typeAtt
static int
UNTOKENIZED_ONLY
Only output untokenized tokens, which are tokens that would normally be split into several tokensstatic int
UNTOKENIZED_TOKEN_FLAG
This flag is used to indicate that the produced "Token" would, ifTOKENS_ONLY
was used, produce multiple tokens.private java.util.Set<java.lang.String>
untokenizedTypes
-
Fields inherited from class org.apache.lucene.analysis.TokenStream
DEFAULT_TOKEN_ATTRIBUTE_FACTORY
-
-
Constructor Summary
Constructors Constructor Description WikipediaTokenizer()
Creates a new instance of theWikipediaTokenizer
.WikipediaTokenizer(int tokenOutput, java.util.Set<java.lang.String> untokenizedTypes)
Creates a new instance of theWikipediaTokenizer
.WikipediaTokenizer(AttributeFactory factory, int tokenOutput, java.util.Set<java.lang.String> untokenizedTypes)
Creates a new instance of theWikipediaTokenizer
.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
Releases resources associated with this stream.private void
collapseAndSaveTokens(int tokenType, java.lang.String type)
private void
collapseTokens(int tokenType)
void
end()
This method is called by the consumer after the last token has been consumed, afterTokenStream.incrementToken()
returnedfalse
(using the newTokenStream
API).boolean
incrementToken()
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.private void
init(int tokenOutput, java.util.Set<java.lang.String> untokenizedTypes)
void
reset()
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.private void
setupSavedToken(int positionInc, java.lang.String type)
private void
setupToken()
-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
correctOffset, setReader
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString
-
-
-
-
Field Detail
-
INTERNAL_LINK
public static final java.lang.String INTERNAL_LINK
- See Also:
- Constant Field Values
-
EXTERNAL_LINK
public static final java.lang.String EXTERNAL_LINK
- See Also:
- Constant Field Values
-
EXTERNAL_LINK_URL
public static final java.lang.String EXTERNAL_LINK_URL
- See Also:
- Constant Field Values
-
CITATION
public static final java.lang.String CITATION
- See Also:
- Constant Field Values
-
CATEGORY
public static final java.lang.String CATEGORY
- See Also:
- Constant Field Values
-
BOLD
public static final java.lang.String BOLD
- See Also:
- Constant Field Values
-
ITALICS
public static final java.lang.String ITALICS
- See Also:
- Constant Field Values
-
BOLD_ITALICS
public static final java.lang.String BOLD_ITALICS
- See Also:
- Constant Field Values
-
HEADING
public static final java.lang.String HEADING
- See Also:
- Constant Field Values
-
SUB_HEADING
public static final java.lang.String SUB_HEADING
- See Also:
- Constant Field Values
-
ALPHANUM_ID
public static final int ALPHANUM_ID
- See Also:
- Constant Field Values
-
APOSTROPHE_ID
public static final int APOSTROPHE_ID
- See Also:
- Constant Field Values
-
ACRONYM_ID
public static final int ACRONYM_ID
- See Also:
- Constant Field Values
-
COMPANY_ID
public static final int COMPANY_ID
- See Also:
- Constant Field Values
-
EMAIL_ID
public static final int EMAIL_ID
- See Also:
- Constant Field Values
-
HOST_ID
public static final int HOST_ID
- See Also:
- Constant Field Values
-
NUM_ID
public static final int NUM_ID
- See Also:
- Constant Field Values
-
CJ_ID
public static final int CJ_ID
- See Also:
- Constant Field Values
-
INTERNAL_LINK_ID
public static final int INTERNAL_LINK_ID
- See Also:
- Constant Field Values
-
EXTERNAL_LINK_ID
public static final int EXTERNAL_LINK_ID
- See Also:
- Constant Field Values
-
CITATION_ID
public static final int CITATION_ID
- See Also:
- Constant Field Values
-
CATEGORY_ID
public static final int CATEGORY_ID
- See Also:
- Constant Field Values
-
BOLD_ID
public static final int BOLD_ID
- See Also:
- Constant Field Values
-
ITALICS_ID
public static final int ITALICS_ID
- See Also:
- Constant Field Values
-
BOLD_ITALICS_ID
public static final int BOLD_ITALICS_ID
- See Also:
- Constant Field Values
-
HEADING_ID
public static final int HEADING_ID
- See Also:
- Constant Field Values
-
SUB_HEADING_ID
public static final int SUB_HEADING_ID
- See Also:
- Constant Field Values
-
EXTERNAL_LINK_URL_ID
public static final int EXTERNAL_LINK_URL_ID
- See Also:
- Constant Field Values
-
TOKEN_TYPES
public static final java.lang.String[] TOKEN_TYPES
String token types that correspond to token type int constants
-
TOKENS_ONLY
public static final int TOKENS_ONLY
Only output tokens- See Also:
- Constant Field Values
-
UNTOKENIZED_ONLY
public static final int UNTOKENIZED_ONLY
Only output untokenized tokens, which are tokens that would normally be split into several tokens- See Also:
- Constant Field Values
-
BOTH
public static final int BOTH
Output the both the untokenized token and the splits- See Also:
- Constant Field Values
-
UNTOKENIZED_TOKEN_FLAG
public static final int UNTOKENIZED_TOKEN_FLAG
This flag is used to indicate that the produced "Token" would, ifTOKENS_ONLY
was used, produce multiple tokens.- See Also:
- Constant Field Values
-
scanner
private final WikipediaTokenizerImpl scanner
A private instance of the JFlex-constructed scanner
-
tokenOutput
private int tokenOutput
-
untokenizedTypes
private java.util.Set<java.lang.String> untokenizedTypes
-
tokens
private java.util.Iterator<AttributeSource.State> tokens
-
offsetAtt
private final OffsetAttribute offsetAtt
-
typeAtt
private final TypeAttribute typeAtt
-
posIncrAtt
private final PositionIncrementAttribute posIncrAtt
-
termAtt
private final CharTermAttribute termAtt
-
flagsAtt
private final FlagsAttribute flagsAtt
-
first
private boolean first
-
-
Constructor Detail
-
WikipediaTokenizer
public WikipediaTokenizer()
Creates a new instance of theWikipediaTokenizer
. Attaches theinput
to a newly created JFlex scanner.
-
WikipediaTokenizer
public WikipediaTokenizer(int tokenOutput, java.util.Set<java.lang.String> untokenizedTypes)
Creates a new instance of theWikipediaTokenizer
. Attaches theinput
to the newly created JFlex scanner.- Parameters:
tokenOutput
- One ofTOKENS_ONLY
,UNTOKENIZED_ONLY
,BOTH
-
WikipediaTokenizer
public WikipediaTokenizer(AttributeFactory factory, int tokenOutput, java.util.Set<java.lang.String> untokenizedTypes)
Creates a new instance of theWikipediaTokenizer
. Attaches theinput
to the newly created JFlex scanner. Uses the givenAttributeFactory
.- Parameters:
tokenOutput
- One ofTOKENS_ONLY
,UNTOKENIZED_ONLY
,BOTH
-
-
Method Detail
-
init
private void init(int tokenOutput, java.util.Set<java.lang.String> untokenizedTypes)
-
incrementToken
public final boolean incrementToken() throws java.io.IOException
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
java.io.IOException
-
collapseAndSaveTokens
private void collapseAndSaveTokens(int tokenType, java.lang.String type) throws java.io.IOException
- Throws:
java.io.IOException
-
setupSavedToken
private void setupSavedToken(int positionInc, java.lang.String type)
-
collapseTokens
private void collapseTokens(int tokenType) throws java.io.IOException
- Throws:
java.io.IOException
-
setupToken
private void setupToken()
-
close
public void close() throws java.io.IOException
Description copied from class:Tokenizer
Releases resources associated with this stream.If you override this method, always call
super.close()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on reuse).NOTE: The default implementation closes the input Reader, so be sure to call
super.close()
when overriding this method.
-
reset
public void reset() throws java.io.IOException
Description copied from class:TokenStream
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call
super.reset()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on further usage).
-
end
public void end() throws java.io.IOException
Description copied from class:TokenStream
This method is called by the consumer after the last token has been consumed, afterTokenStream.incrementToken()
returnedfalse
(using the newTokenStream
API). Streams implementing the old API should upgrade to use this feature.This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.
Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.
If you override this method, always call
super.end()
.- Overrides:
end
in classTokenStream
- Throws:
java.io.IOException
- If an I/O error occurs
-
-