org.apache.lucene.analysis

Class Tokenizer

Known Direct Subclasses:
CharTokenizer, ChineseTokenizer, CJKTokenizer, KeywordTokenizer, StandardTokenizer

public abstract class Tokenizer
extends TokenStream

A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class.

Field Summary

protected Reader
input
The text source for this Tokenizer.

Constructor Summary

Tokenizer()
Construct a tokenizer with null input.
Tokenizer(Reader input)
Construct a token stream processing the given input.

Method Summary

void
close()
By default, closes the input Reader.

Methods inherited from class org.apache.lucene.analysis.TokenStream

close, next

Field Details

input

protected Reader input
The text source for this Tokenizer.

Constructor Details

Tokenizer

protected Tokenizer()
Construct a tokenizer with null input.

Tokenizer

protected Tokenizer(Reader input)
Construct a token stream processing the given input.

Method Details

close

public void close()
            throws IOException
By default, closes the input Reader.
Overrides:
close in interface TokenStream

Copyright © 2000-2007 Apache Software Foundation. All Rights Reserved.