org.apache.lucene.index.memory
Class PatternAnalyzer
public class PatternAnalyzer
Efficient Lucene analyzer/tokenizer that preferably operates on a String rather than a
Reader
, that can flexibly separate text into terms via a regular expression
Pattern
(with behaviour identical to
String.split(String)
),
and that combines the functionality of
LetterTokenizer
,
LowerCaseTokenizer
,
WhitespaceTokenizer
,
StopFilter
into a single efficient
multi-purpose class.
If you are unsure how exactly a regular expression should look like, consider
prototyping by simply trying various expressions on some test texts via
String.split(String)
. Once you are satisfied, give that regex to
PatternAnalyzer. Also see
Java Regular Expression Tutorial.
This class can be considerably faster than the "normal" Lucene tokenizers.
It can also serve as a building block in a compound Lucene
TokenFilter
chain. For example as in this
stemming example:
PatternAnalyzer pat = ...
TokenStream tokenStream = new SnowballFilter(
pat.tokenStream("content", "James is running round in the woods"),
"English"));
static PatternAnalyzer | DEFAULT_ANALYZER - A lower-casing word analyzer with English stop words (can be shared
freely across threads without harm); global per class loader.
|
static PatternAnalyzer | EXTENDED_ANALYZER - A lower-casing word analyzer with extended English stop words
(can be shared freely across threads without harm); global per class
loader.
|
static Pattern | NON_WORD_PATTERN "\\W+" ; Divides text at non-letters (Character.isLetter(c))
|
static Pattern | WHITESPACE_PATTERN "\\s+" ; Divides text at whitespaces (Character.isWhitespace(c))
|
PatternAnalyzer(Pattern pattern, boolean toLowerCase, Set stopWords) - Constructs a new instance with the given parameters.
|
boolean | equals(Object other) - Indicates whether some other object is "equal to" this one.
|
int | hashCode() - Returns a hash code value for the object.
|
TokenStream | tokenStream(String fieldName, Reader reader) - Creates a token stream that tokenizes all the text in the given Reader;
This implementation forwards to
tokenStream(String, String) and is
less efficient than tokenStream(String, String) .
|
TokenStream | tokenStream(String fieldName, String text) - Creates a token stream that tokenizes the given string into token terms
(aka words).
|
DEFAULT_ANALYZER
public static final PatternAnalyzer DEFAULT_ANALYZER
A lower-casing word analyzer with English stop words (can be shared
freely across threads without harm); global per class loader.
EXTENDED_ANALYZER
public static final PatternAnalyzer EXTENDED_ANALYZER
A lower-casing word analyzer with extended English stop words
(can be shared freely across threads without harm); global per class
loader. The stop words are borrowed from
http://thomas.loc.gov/home/stopwords.html, see
http://thomas.loc.gov/home/all.about.inquery.html
NON_WORD_PATTERN
public static final Pattern NON_WORD_PATTERN
"\\W+"
; Divides text at non-letters (Character.isLetter(c))
WHITESPACE_PATTERN
public static final Pattern WHITESPACE_PATTERN
"\\s+"
; Divides text at whitespaces (Character.isWhitespace(c))
PatternAnalyzer
public PatternAnalyzer(Pattern pattern,
boolean toLowerCase,
Set stopWords)
Constructs a new instance with the given parameters.
pattern
- a regular expression delimiting tokenstoLowerCase
- if true
returns tokens after applying
String.toLowerCase()stopWords
- if non-null, ignores all tokens that are contained in the
given stop set (after previously having applied toLowerCase()
if applicable). For example, created via
StopFilter.makeStopSet(String[])
and/or
WordlistLoader
as in
WordlistLoader.getWordSet(new File("samples/fulltext/stopwords.txt")
or other stop words
lists .
equals
public boolean equals(Object other)
Indicates whether some other object is "equal to" this one.
other
- the reference object with which to compare.
- true if equal, false otherwise
hashCode
public int hashCode()
Returns a hash code value for the object.
tokenStream
public TokenStream tokenStream(String fieldName,
Reader reader)
Creates a token stream that tokenizes all the text in the given Reader;
This implementation forwards to tokenStream(String, String)
and is
less efficient than tokenStream(String, String)
.
- tokenStream in interface Analyzer
fieldName
- the name of the field to tokenize (currently ignored).reader
- the reader delivering the text
tokenStream
public TokenStream tokenStream(String fieldName,
String text)
Creates a token stream that tokenizes the given string into token terms
(aka words).
fieldName
- the name of the field to tokenize (currently ignored).text
- the string to tokenize
Copyright © 2000-2006 Apache Software Foundation. All Rights Reserved.