org.apache.lucene.analysis.standard
Class StandardTokenizer

java.lang.Object
  extended by org.apache.lucene.analysis.TokenStream
      extended by org.apache.lucene.analysis.Tokenizer
          extended by org.apache.lucene.analysis.standard.StandardTokenizer

public class StandardTokenizer
extends Tokenizer

A grammar-based tokenizer constructed with JFlex

This should be a good tokenizer for most European-language documents:

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.


Field Summary
 
Fields inherited from class org.apache.lucene.analysis.Tokenizer
input
 
Constructor Summary
StandardTokenizer(java.io.Reader input)
          Creates a new instance of the StandardTokenizer.
StandardTokenizer(java.io.Reader input, boolean replaceInvalidAcronym)
          Creates a new instance of the StandardTokenizer.
 
Method Summary
 int getMaxTokenLength()
           
 boolean isReplaceInvalidAcronym()
          Deprecated. Remove in 3.X and make true the only valid value
 Token next(Token result)
          Returns the next token in the stream, or null at EOS.
 void reset()
          Resets this stream to the beginning.
 void reset(java.io.Reader reader)
          Expert: Reset the tokenizer to a new reader.
 void setMaxTokenLength(int length)
          Set the max allowed token length.
 void setReplaceInvalidAcronym(boolean replaceInvalidAcronym)
          Deprecated. Remove in 3.X and make true the only valid value See https://issues.apache.org/jira/browse/LUCENE-1068
 
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close
 
Methods inherited from class org.apache.lucene.analysis.TokenStream
next
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

StandardTokenizer

public StandardTokenizer(java.io.Reader input)
Creates a new instance of the StandardTokenizer. Attaches the input to a newly created JFlex scanner.


StandardTokenizer

public StandardTokenizer(java.io.Reader input,
                         boolean replaceInvalidAcronym)
Creates a new instance of the StandardTokenizer. Attaches the input to the newly created JFlex scanner.

Parameters:
input - The input reader
replaceInvalidAcronym - Set to true to replace mischaracterized acronyms with HOST. See http://issues.apache.org/jira/browse/LUCENE-1068
Method Detail

setMaxTokenLength

public void setMaxTokenLength(int length)
Set the max allowed token length. Any token longer than this is skipped.


getMaxTokenLength

public int getMaxTokenLength()
See Also:
setMaxTokenLength(int)

next

public Token next(Token result)
           throws java.io.IOException
Description copied from class: TokenStream
Returns the next token in the stream, or null at EOS. When possible, the input Token should be used as the returned Token (this gives fastest tokenization performance), but this is not required and a new Token may be returned. Callers may re-use a single Token instance for successive calls to this method.

This implicitly defines a "contract" between consumers (callers of this method) and producers (implementations of this method that are the source for tokens):

Note that a TokenFilter is considered a consumer.

Overrides:
next in class TokenStream
Parameters:
result - a Token that may or may not be used to return
Returns:
next token in the stream or null if end-of-stream was hit
Throws:
java.io.IOException

reset

public void reset()
           throws java.io.IOException
Description copied from class: TokenStream
Resets this stream to the beginning. This is an optional operation, so subclasses may or may not implement this method. Reset() is not needed for the standard indexing process. However, if the Tokens of a TokenStream are intended to be consumed more than once, it is necessary to implement reset().

Overrides:
reset in class TokenStream
Throws:
java.io.IOException

reset

public void reset(java.io.Reader reader)
           throws java.io.IOException
Description copied from class: Tokenizer
Expert: Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.

Overrides:
reset in class Tokenizer
Throws:
java.io.IOException

isReplaceInvalidAcronym

public boolean isReplaceInvalidAcronym()
Deprecated. Remove in 3.X and make true the only valid value

Prior to https://issues.apache.org/jira/browse/LUCENE-1068, StandardTokenizer mischaracterized as acronyms tokens like www.abc.com when they should have been labeled as hosts instead.

Returns:
true if StandardTokenizer now returns these tokens as Hosts, otherwise false

setReplaceInvalidAcronym

public void setReplaceInvalidAcronym(boolean replaceInvalidAcronym)
Deprecated. Remove in 3.X and make true the only valid value See https://issues.apache.org/jira/browse/LUCENE-1068

Parameters:
replaceInvalidAcronym - Set to true to replace mischaracterized acronyms as HOST.


Copyright © 2000-2009 Apache Software Foundation. All Rights Reserved.