it.unimi.dsi.mg4j.index
Class MultiTermIndexIterator

java.lang.Object
  extended by it.unimi.dsi.fastutil.ints.AbstractIntIterator
      extended by it.unimi.dsi.mg4j.search.AbstractDocumentIterator
          extended by it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator
              extended by it.unimi.dsi.mg4j.search.AbstractUnionDocumentIterator
                  extended by it.unimi.dsi.mg4j.index.MultiTermIndexIterator
All Implemented Interfaces:
IntIterator, IndexIterator, DocumentIterator, Iterable<Interval>, Iterator<Integer>

public class MultiTermIndexIterator
extends AbstractUnionDocumentIterator
implements IndexIterator

A virtual index iterator that merges several component index iterators.

This class adds to AbstractUnionDocumentIterator an interval interator generating the OR of the intervals returned for each of the documents involved. The main difference with an OrDocumentIterator built on the same array of component iterators is that this class implements IndexIterator and hence provides a count() (the sum of counts of those component iterators positioned on the current document) and a frequency(). The latter is by default the maximum frequency of a component iterator, but it can be set at construction time.

The main raison d'être of this class is support for query expansion: a blind application of OrDocumentIterator to an array of index iterators would mislead scorers such as BM25Scorer because low-frequency terms (e.g., hapax legomena) would be responsible for most of the score.


Nested Class Summary
 
Nested classes/interfaces inherited from class it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator
AbstractCompositeDocumentIterator.AbstractCompositeIndexIntervalIterator, AbstractCompositeDocumentIterator.AbstractCompositeIntervalIterator
 
Nested classes/interfaces inherited from class it.unimi.dsi.mg4j.search.AbstractDocumentIterator
AbstractDocumentIterator.AbstractIntervalIterator
 
Field Summary
protected  int id
          The id of this iterator.
protected  String term
          The term of this iterator.
 
Fields inherited from class it.unimi.dsi.mg4j.search.AbstractUnionDocumentIterator
curr, front, frontSize, queue
 
Fields inherited from class it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator
documentIterator, indexIterator, indices, n, soleIndex
 
Fields inherited from class it.unimi.dsi.mg4j.search.AbstractDocumentIterator
last, next
 
Constructor Summary
protected MultiTermIndexIterator(int defaultFrequency, IndexIterator... indexIterator)
          Creates a new document iterator that merges the given array of iterators.
 
Method Summary
 boolean accept(DocumentIteratorVisitor visitor)
          Accepts a visitor.
 boolean acceptOnTruePaths(DocumentIteratorVisitor visitor)
          Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.
 int count()
          The count is the sum of counts of those component iterators positioned on the current document.
 int frequency()
          The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.
protected  IntervalIterator getComposedIntervalIterator(Index index)
           
static IndexIterator getInstance(Index index, IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
static IndexIterator getInstance(IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
static IndexIterator getInstance(int defaultFrequency, Index index, IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
static IndexIterator getInstance(int defaultFrequency, IndexIterator... indexIterator)
          Returns an index iterator that merges the given array of iterators.
 int id()
          Returns the id of this index iterator.
 void id(int id)
          Sets the id of this index iterator.
 Index index()
          Returns the index over which this iterator is built.
 int nextDocument()
          Returns the next document provided by this document iterator, or -1 if no more documents are available.
 Payload payload()
          This method is not implemented by this class.
 int[] positionArray()
          Returns the positions at which the term appears in the current document in an array.
 IntIterator positions()
          Returns the positions at which the term appears in the current document.
 int positions(int[] position)
          Stores the positions at which the term appears in the current document in a given array.
 int skipTo(int n)
          Skips all documents smaller than n.
 String term()
          Returns the term whose inverted list is returned by this index iterator.
 void term(CharSequence term)
          Sets the term whose inverted list is returned by this index iterator.
 int termNumber()
          Returns the number of the term whose inverted list is returned by this index iterator.
 
Methods inherited from class it.unimi.dsi.mg4j.search.AbstractUnionDocumentIterator
computeFront, intervalIterator, intervalIterators
 
Methods inherited from class it.unimi.dsi.mg4j.search.AbstractCompositeDocumentIterator
dispose, indices, intervalIterator, toString
 
Methods inherited from class it.unimi.dsi.mg4j.search.AbstractDocumentIterator
document, hasNext, iterator, nextInt
 
Methods inherited from class it.unimi.dsi.fastutil.ints.AbstractIntIterator
next, remove, skip
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface it.unimi.dsi.mg4j.search.DocumentIterator
dispose, document, indices, intervalIterator, intervalIterator, intervalIterators, iterator, nextInt
 
Methods inherited from interface it.unimi.dsi.fastutil.ints.IntIterator
skip
 
Methods inherited from interface java.util.Iterator
hasNext, next, remove
 

Field Detail

term

protected String term
The term of this iterator.


id

protected int id
The id of this iterator.

Constructor Detail

MultiTermIndexIterator

protected MultiTermIndexIterator(int defaultFrequency,
                                 IndexIterator... indexIterator)
                          throws IOException
Creates a new document iterator that merges the given array of iterators.

Parameters:
defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).
indexIterator - the iterators to be joined.
Throws:
IOException
Method Detail

getInstance

public static IndexIterator getInstance(IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided. The frequency is computed as a max, and index() will return the result of the same method on the first iterator.

Parameters:
indexIterator - the iterators to be joined (at least one).
Returns:
a merged index iterator.
Throws:
IllegalArgumentException - if no iterators were provided.
IOException

getInstance

public static IndexIterator getInstance(Index index,
                                        IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently. The frequency is computed as a max, and index() will return index.

Parameters:
index - the index that wil be returned by index().
indexIterator - the iterators to be joined.
Returns:
a merged index iterator.
Throws:
IOException

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators. This method requires that at least one iterator is provided.

Parameters:
defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).
indexIterator - the iterators to be joined (at least one).
Returns:
a merged index iterator.
Throws:
IllegalArgumentException - if no iterators were provided, or they run on different indices.
IOException

getInstance

public static IndexIterator getInstance(int defaultFrequency,
                                        Index index,
                                        IndexIterator... indexIterator)
                                 throws IOException
Returns an index iterator that merges the given array of iterators.

Note that the special case of the empty and of the singleton arrays are handled efficiently.

Parameters:
defaultFrequency - the default term frequency (or Integer.MIN_VALUE for the max).
index - the index that wil be returned by index().
indexIterator - the iterators to be joined.
Returns:
a merged index iterator.
Throws:
IllegalArgumentException - if there is some iterator on an index different from index.
IOException

getComposedIntervalIterator

protected IntervalIterator getComposedIntervalIterator(Index index)
Specified by:
getComposedIntervalIterator in class AbstractUnionDocumentIterator

skipTo

public int skipTo(int n)
           throws IOException
Description copied from interface: DocumentIterator
Skips all documents smaller than n.

Define the current document k associated with this document iterator as follows:

If k is larger than or equal to n, then this method does nothing and returns k. Otherwise, a call to this method is equivalent to

 while( ( k = nextDocument() ) < n && k != -1 );
 return k == -1 ? Integer.MAX_VALUE : k;
 

Thus, when a result kInteger.MAX_VALUE is returned, the state of this iterator will be exactly the same as after a call to DocumentIterator.nextDocument() that returned k. In particular, the first document larger than or equal to n (when returned by this method) will not be returned by the next call to DocumentIterator.nextDocument().

Specified by:
skipTo in interface DocumentIterator
Overrides:
skipTo in class AbstractUnionDocumentIterator
Parameters:
n - a document pointer.
Returns:
a document pointer larger than or equal to n if available, Integer.MAX_VALUE otherwise.
Throws:
IOException

nextDocument

public int nextDocument()
                 throws IOException
Description copied from interface: DocumentIterator
Returns the next document provided by this document iterator, or -1 if no more documents are available.

Warning: the specification of this method has significantly changed as of MG4J 1.2. The special return value -1 is used to mark the end of iteration (a NoSuchElementException would have been thrown before in that case, so ho harm should be caused by this change). The reason for this change is providing fully lazy iteration over documents. Fully lazy iteration does not provide an hasNext() method—you have to actually ask for the next element and check the return value. Fully lazy iteration is much lighter on method calls (half) and in most (if not all) MG4J classes leads to a much simpler logic. Moreover, DocumentIterator.nextDocument() can be specified as throwing an IOException, which avoids the pernicious proliferation of try/catch blocks in very short, low-level methods (it was having a detectable impact on performance).

Specified by:
nextDocument in interface DocumentIterator
Overrides:
nextDocument in class AbstractUnionDocumentIterator
Returns:
the next document, or -1 if no more documents are available.
Throws:
IOException

count

public int count()
          throws IOException
The count is the sum of counts of those component iterators positioned on the current document.

Specified by:
count in interface IndexIterator
Returns:
the sum of counts.
Throws:
IOException

frequency

public int frequency()
              throws IOException
The frequency is either the default frequency set at construction time, or the maximum frequency of the component iterators.

Specified by:
frequency in interface IndexIterator
Returns:
the frequency.
Throws:
IOException

term

public void term(CharSequence term)
Description copied from interface: IndexIterator
Sets the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by Index.documents(CharSequence) or by IndexReader.documents(CharSequence), but you can use this method to ensure that IndexIterator.term() doesn't throw an exception.

Instances of Index.EmptyIndexIterator are allowed to ignore silently calls to this method.

Specified by:
term in interface IndexIterator
Parameters:
term - a character sequence (that will be defensively copied) that will be assumed to be the term whose inverted list is returned by this index iterator.

term

public String term()
Description copied from interface: IndexIterator
Returns the term whose inverted list is returned by this index iterator.

Usually, the term is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(int), but you can supply your own term with IndexIterator.term(CharSequence).

Instances of Index.EmptyIndexIterator can return null.

Specified by:
term in interface IndexIterator
Returns:
the term over which this iterator is built, as a compact mutable string.
See Also:
IndexIterator.termNumber()

termNumber

public int termNumber()
Description copied from interface: IndexIterator
Returns the number of the term whose inverted list is returned by this index iterator.

Usually, the term number is automatically set by IndexReader.documents(CharSequence) or IndexReader.documents(int). Instances of Index.EmptyIndexIterator can return -1.

Specified by:
termNumber in interface IndexIterator
Returns:
the number of the term over which this iterator is built, or -1 for an Index.EmptyIndexIterator.
See Also:
IndexIterator.term()

id

public void id(int id)
Description copied from interface: IndexIterator
Sets the id of this index iterator.

The id is an integer associated to each index iterator. It has no specific semantics, and can be used differently in different contexts. A typical usage pattern, for instance, is using it to assign a unique number to the index iterators contained in a composited document iterator (say, numbering consecutively the leaves of the composite).

Instances of Index.EmptyIndexIterator are allowed to ignore silently calls to this method.

Specified by:
id in interface IndexIterator
Parameters:
id - the new id for this index iterator.

id

public int id()
Description copied from interface: IndexIterator
Returns the id of this index iterator.

Instances of Index.EmptyIndexIterator are allowed to return -1.

Specified by:
id in interface IndexIterator
Returns:
the id of this index iterator.
See Also:
IndexIterator.id(int)

index

public Index index()
Description copied from interface: IndexIterator
Returns the index over which this iterator is built.

Specified by:
index in interface IndexIterator
Returns:
the index over which this iterator is built.

payload

public Payload payload()
This method is not implemented by this class.

Specified by:
payload in interface IndexIterator
Returns:
the payload associated with the current document.

positionArray

public int[] positionArray()
                    throws IOException
Description copied from interface: IndexIterator
Returns the positions at which the term appears in the current document in an array.

Implementations are allowed to return the same array across different calls to this method.

Specified by:
positionArray in interface IndexIterator
Returns:
an array whose first IndexIterator.count() elements contain the document positions.
Throws:
IOException

positions

public IntIterator positions()
                      throws IOException
Description copied from interface: IndexIterator
Returns the positions at which the term appears in the current document.

Specified by:
positions in interface IndexIterator
Returns:
the positions of the current document in which the current term appears.
Throws:
IOException

positions

public int positions(int[] position)
              throws IOException
Description copied from interface: IndexIterator
Stores the positions at which the term appears in the current document in a given array.

If the array is not large enough (i.e., it does not contain IndexIterator.count() elements), this method will return a negative number (the opposite of the count).

Specified by:
positions in interface IndexIterator
Parameters:
position - an array that will be used to store positions.
Returns:
the count; it will have the sign changed if positions cannot hold all positions.
Throws:
IOException

accept

public boolean accept(DocumentIteratorVisitor visitor)
               throws IOException
Description copied from interface: DocumentIterator
Accepts a visitor.

A document iterator is usually structured as composite, with operators as internal nodes and IndexIterators as leaves. This method implements the visitor pattern.

Specified by:
accept in interface DocumentIterator
Overrides:
accept in class AbstractCompositeDocumentIterator
Parameters:
visitor - the visitor.
Returns:
true if the visit should continue.
Throws:
IOException

acceptOnTruePaths

public boolean acceptOnTruePaths(DocumentIteratorVisitor visitor)
                          throws IOException
Description copied from class: AbstractUnionDocumentIterator
Invokes AbstractUnionDocumentIterator.acceptOnTruePaths(DocumentIteratorVisitor) only on component iterators positioned on the current document.

Specified by:
acceptOnTruePaths in interface DocumentIterator
Overrides:
acceptOnTruePaths in class AbstractUnionDocumentIterator
Parameters:
visitor - a visitor.
Returns:
true if the visit should continue.
Throws:
IOException
See Also:
DocumentIterator.accept(DocumentIteratorVisitor), CounterCollectionVisitor