Public Member Functions
ibis::text::tokenizer Struct Reference

A tokenizer class to turn a string buffer into tokens. More...

#include <category.h>

Inheritance diagram for ibis::text::tokenizer:
ibis::keywords::tokenizer

List of all members.

Public Member Functions

virtual int operator() (std::vector< const char * > &tkns, char *buf)=0
 A tokenizer must implement a two-argument operator().
virtual ~tokenizer ()
 Destructor.

Detailed Description

A tokenizer class to turn a string buffer into tokens.

Used by ibis::keywords to build a term-document index.


Member Function Documentation

virtual int ibis::text::tokenizer::operator() ( std::vector< const char * > &  tkns,
char *  buf 
) [pure virtual]

A tokenizer must implement a two-argument operator().

It takes an input string in buf to produce a list of tokens in tkns. The input buffer may be modified in this function. The return value shall be zero (0) to indicate success, and a positive value to carray a warning message and a negative value to indicate fatal error.

Implemented in ibis::keywords::tokenizer.


The documentation for this struct was generated from the following file:

Make It A Bit Faster
Contact us
Disclaimers
FastBit source code
FastBit mailing list archive