Public Member Functions
ibis::keywords::tokenizer Class Reference

A simple tokenizer used to parse the keywords. More...

#include <ikeywords.h>

Inheritance diagram for ibis::keywords::tokenizer:
ibis::text::tokenizer

List of all members.

Public Member Functions

virtual int operator() (std::vector< const char * > &tkns, char *buf)
 Tokenizer.
 tokenizer (const char *d=ibis::util::delimiters)
 Constructor.
virtual ~tokenizer ()
 Destructor.

Detailed Description

A simple tokenizer used to parse the keywords.


Constructor & Destructor Documentation

ibis::keywords::tokenizer::tokenizer ( const char *  d = ibis::util::delimiters) [inline]

Constructor.

It takes a list of delimiters. Any character in the list of delimiters will terminate a token. If no delimiter is given, anything other than alphanumerical characters will terminate a token. By default, the delimiters defined in ibis::util::delimiters are used.


Member Function Documentation

int ibis::keywords::tokenizer::operator() ( std::vector< const char * > &  tkns,
char *  buf 
) [virtual]

Tokenizer.

Turn the buffer buf into a list of tokens based on the following rules.

  • If no delimiter is specified, it turns all non-alphanumeric characters into the null character and returns the starting positions of groups of alphanumeric characters as tokens.
  • If a list of delimiters are provided, any of the delimiters will terminate a token. Blank spaces surrounding the delimiters will be turned into null characters along with the delimiters.

This function returns 0 in normal cases.

Implements ibis::text::tokenizer.


The documentation for this class was generated from the following files:

Make It A Bit Faster
Contact us
Disclaimers
FastBit source code
FastBit mailing list archive