Classes | Public Types | Public Member Functions | Static Public Member Functions | Protected Member Functions | Static Protected Attributes | Friends
ibis::fileManager Class Reference

This fileManager is intended to allow different objects to share the same open file. More...

#include <fileManager.h>

List of all members.

Classes

class  buffer
 A buffer is intended to be a temporary workspace in memory. More...
class  cleaner
 A function object to be used to register external cleaners. More...
class  readLock
 A read lock on the file manager. More...
class  roFile
 This class manages content of a whole (read-only) file. More...
class  softWriteLock
 A soft write lock for controlling access to the two internal lists. More...
class  storage
 The storage class treats all memory as char*. More...
class  writeLock
 A write lock for controlling access to the two internal lists. More...

Public Types

enum  ACCESS_PREFERENCE { MMAP_LARGE_FILES, PREFER_READ, PREFER_MMAP }
 Hint passed to the function getFile. More...

Public Member Functions

void addCleaner (const cleaner *cl)
void clear ()
 Close all files and remove all records of them.
void flushDir (const char *name)
 Close all files in the named directory, but not subdirectories.
void flushFile (const char *name)
 Close the file, remove the record about it from the file manager.
void gainReadAccess (const char *mesg) const
 Obtain a read lock on the file manager.
template<typename T >
int getFile (const char *name, array_t< T > &arr, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Given a file name, place the content in an array_t<T>.
int getFile (const char *name, storage **st, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Retrieve the file content as a storage object.
const double & pageCount () const
 Returns the number of pages accessed by function read from stdlib.h.
void printStatus (std::ostream &out) const
 Prints status information about the file manager.
void recordPages (off_t start, off_t stop)
 Given the starting and ending addresses, this function computes the number of pages involved.
void releaseAccess (const char *mesg) const
 Release a read lock on the file manager.
void removeCleaner (const cleaner *cl)
void signalMemoryAvailable () const
 Signal to the file manager that some memory have been freed.
template<typename T >
int tryGetFile (const char *name, array_t< T > &arr, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Given a file name, place the content in an array_t<T>.
int tryGetFile (const char *name, storage **st, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Try to retrieve the content of the named file.

Static Public Member Functions

static int adjustCacheSize (uint64_t)
 Change the size of memory cache allocated to the file manager.
static uint64_t bytesFree ()
 Return the number of bytes free.
static uint64_t bytesInUse ()
 Returns the number of bytes currently on records.
static uint64_t currentCacheSize ()
 Return the current cache size in bytes.
static void decreaseUse (size_t dec, const char *evt)
static storagegetFileSegment (const char *name, const int fdes, const off_t b, const off_t e)
 Retrieve a portion of a file content.
static time_t iBeat ()
 Returns the value of a simple counter. It is not thread-safe!
static void increaseUse (size_t inc, const char *evt)
static fileManagerinstance ()
 Returns a pointer to the one and only file manager.
static uint32_t pageSize ()
 Returns the page size (in bytes) used by the file system.

Protected Member Functions

 fileManager ()
 The protected constructor of the ibis::fileManager class.
 fileManager (const fileManager &rhs)
fileManageroperator= (const fileManager &rhs)
void recordFile (roFile *)
 Record a newly allocated storage in the two lists.
 ~fileManager ()
 Destructor.

Static Protected Attributes

static uint64_t maxBytes = 0
 The maximum number of bytes allowed.
static unsigned int maxOpenFiles = 0
 The Maximum number of files that can be kept open at the same time.
static ibis::util::sharedInt64 totalBytes
 The total number of bytes of all managed objects.

Friends

class roFile
class softWriteLock
class storage
class writeLock

Detailed Description

This fileManager is intended to allow different objects to share the same open file.

It does not manage writing of files.


Member Enumeration Documentation

Hint passed to the function getFile.

The main choice is whether to use memory map or use the read function to access the content of a file.


Constructor & Destructor Documentation

ibis::fileManager::fileManager ( ) [protected]

The protected constructor of the ibis::fileManager class.

There are three parameters that can be specified in a configuration file to control this object, fileManager.maxBytes, fileManager.maxOpenFiles, and fileManager.minMapSize. If you are unsure of what to do, then don't specify anything -- the default values are typically acceptable.

  • filemanager.maxBytes The maximum number of bytes of all objects under control of the file manager, e.g.,
    /// fileManager.maxBytes = 500MB
    /// 
    One may specify a number followed by KB, MB, or GB (without space in between). If not specified, this constructor attempts to determine the size of the physical memory available and will use half of the memory for caching FastBit objects.
  • fileManager.maxOpenFiles This file manager will keep the number of open files below this specified maximum. Note that FastBit usually invokes the lower level function open, which typical can use more file handles than the higher level ones such as fopen. If not specified, it will use three quarters of maximum file halder defined by _SC_OPEN_MAX.
  • fileManager.minMapSize The minimal size of a file before FastBit will attempt to use memory map on it. For smaller files, it is more efficient to read the whole content into memory rather than keeping a file open. The default value is defined by the macro FASTBIT_MIN_MAP_SIZE.

References ibis::resource::getNumber(), ibis::gParameters(), ibis::gVerbose, maxBytes, and maxOpenFiles.


Member Function Documentation

int ibis::fileManager::adjustCacheSize ( uint64_t  newsize) [static]

Change the size of memory cache allocated to the file manager.

Change the class variable maxBytes to the newsize in bytes.

Return 0 if successful, a negative number otherwise.

This function simply changes the maximum bytes allowed, without enforcing this limit. Future operations that require more memory will be subject to the new cache size limit.

Reducing the cache size while there are on-going operations can have very undesirable effects, therefore this function will not accept a new size if it is less than the current number of bytes in memory. It might be helpful to call ibis::fileManager::clear to reduce the memory usage before changing the cache size.

References ibis::util::envLock, and ibis::gVerbose.

void ibis::fileManager::clear ( )

Close all files and remove all records of them.

The function cleans the memory cache used by FastBit.

It destroys the two lists of files it holds and therefore make the file not accessible any new objects. However, the actual underlying memory may still be present if they are being actively used. This function is effective on if all other operations have ceased! To force an individual file to be unloaded use ibis::fileManager::flushFile. To force all files in a directory to be unloaded used ibis::fileManager::flushDir.

Note:
This function will not do anything if it is not able to acquire a write lock in the file manager object.

References ibis::gVerbose, and ibis::fileManager::softWriteLock::isLocked().

Referenced by ibis::part::rollback().

int ibis::fileManager::getFile ( const char *  name,
storage **  st,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Retrieve the file content as a storage object.

The object *st returned from this function is owned by the fileManager. The caller should NOT delete *st! This function will wait for the fileManager to unload some in-memory objects if there isn't enough memory for the file.

Upon successful completion of the task, it returns zero; otherwise, it returns a non-zero value to indicate an error and it does not modify the content of storage object.

References ibis::fileManager::storage::begin(), ibis::horometer::CPUTime(), ibis::fileManager::roFile::doRead(), ibis::gVerbose, ibis::fileManager::roFile::isFileMap(), ibis::fileManager::roFile::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), and ibis::horometer::stop().

template<typename T >
int ibis::fileManager::getFile ( const char *  name,
array_t< T > &  arr,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Given a file name, place the content in an array_t<T>.

This function waits for memory to become available if there is enough memory to read the file content into memory. The compiler macro FASTBIT_MAX_WAIT_TIME defines the maximum amount of time (in seconds) it may wait.

The return value is zero (0) if the function is successful, otherwise returns a non-zero value.

References ibis::array_t< T >::clear(), ibis::gVerbose, ibis::array_t< T >::size(), and ibis::array_t< T >::swap().

Referenced by ibis::column::actualMinMax(), ibis::zone::append(), ibis::pale::append(), ibis::pack::append(), ibis::ambit::append(), ibis::relic::append(), ibis::range::append(), ibis::mesa::append(), ibis::direkte::append(), ibis::bin::append(), ibis::category::append(), ibis::bin::binning(), ibis::bin::binningT(), ibis::bin::binOrderT(), ibis::bundle1::bundle1(), ibis::bundles::bundles(), ibis::column::computeMax(), ibis::column::computeMin(), ibis::column::computeSum(), ibis::ambit::construct(), ibis::range::construct(), ibis::egale::construct(), ibis::bin::construct(), ibis::blob::countRawBytes(), ibis::part::doScan(), ibis::blob::getBlob(), ibis::column::getDoubleArray(), ibis::column::getFloatArray(), ibis::column::getIntArray(), ibis::column::getNullMask(), ibis::column::getRawData(), ibis::roster::icSearch(), ibis::index::mapValues(), ibis::bak2::mapValues(), ibis::bak::mapValues(), ibis::part::negativeScan(), ibis::bitvector64::read(), ibis::bitvector::read(), ibis::array_t< T >::read(), ibis::query::readRIDs(), ibis::bundle::readRIDs(), ibis::part::reorder(), ibis::part::rollback(), ibis::column::searchSorted(), ibis::column::selectBytes(), ibis::column::selectDoubles(), ibis::column::selectFloats(), ibis::column::selectInts(), ibis::column::selectLongs(), ibis::text::selectLongs(), ibis::blob::selectRawBytes(), ibis::column::selectShorts(), ibis::column::selectUBytes(), ibis::column::selectUInts(), ibis::column::selectULongs(), ibis::column::selectUShorts(), ibis::column::selectValuesT(), ibis::column::truncateData(), and ibis::roster::writeSorted().

ibis::fileManager::storage * ibis::fileManager::getFileSegment ( const char *  name,
const int  fdes,
const off_t  b,
const off_t  e 
) [static]

Retrieve a portion of a file content.

Both the file name and the file descriptor are passed in to this function so that it can make a decision on whether to use a file map or directly read the content into memory. It prefers the read option more because the caller is more like to touch every bytes in an explicitly specified portion of a file. More specifically, it uses file map only if the file of the file segement is 4 * FASTBIT_MIN_MAP_SIZE and the number of mapped files is less than half of the maxOpenFiles.

References ibis::horometer::CPUTime(), ibis::gVerbose, instance(), ibis::fileManager::storage::isFileMap(), ibis::fileManager::storage::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), and ibis::horometer::stop().

ibis::fileManager & ibis::fileManager::instance ( ) [static]

Returns a pointer to the one and only file manager.

The instance function of the fileManager singleton.

Referenced by ibis::column::actualMinMax(), ibis::part::append(), ibis::zone::append(), ibis::pale::append(), ibis::pack::append(), ibis::ambit::append(), ibis::relic::append(), ibis::range::append(), ibis::mesa::append(), ibis::direkte::append(), ibis::bin::append(), ibis::column::append(), ibis::category::append(), ibis::part::append1(), ibis::part::append2(), ibis::part::appendToBackup(), ibis::bord::backup(), ibis::fileManager::roFile::beginUse(), ibis::bin::binning(), ibis::bin::binningT(), ibis::bin::binOrderT(), ibis::bundle1::bundle1(), ibis::bundles::bundles(), bytesFree(), ibis::query::clear(), ibis::part::commit(), ibis::column::computeMax(), ibis::column::computeMin(), ibis::column::computeSum(), ibis::ambit::construct(), ibis::relic::construct(), ibis::range::construct(), ibis::egale::construct(), ibis::bin::construct(), ibis::blob::countRawBytes(), ibis::index::create(), ibis::part::deactivate(), ibis::part::doCompare(), ibis::fileManager::roFile::doRead(), ibis::part::doScan(), ibis::part::emptyCache(), ibis::fileManager::roFile::endUse(), ibis::query::estimate(), ibis::query::evaluate(), fastbit_cleanup(), ibis::column::findLower(), ibis::text::findString(), ibis::column::findUpper(), ibis::part::get1DBins_(), ibis::blob::getBlob(), ibis::column::getDoubleArray(), getFileSegment(), ibis::column::getFloatArray(), ibis::column::getIntArray(), ibis::column::getNullMask(), ibis::column::getRawData(), ibis::roster::icSearch(), ibis::init(), ibis::util::intersect(), ibis::index::mapValues(), ibis::bak2::mapValues(), ibis::bak::mapValues(), ibis::part::negativeCompare(), ibis::part::negativeScan(), ibis::part::vault::open(), ibis::part::barrel::open(), ibis::query::orderPairs(), ibis::part::part(), ibis::part::purgeInactive(), ibis::column::purgeIndexFile(), ibis::part::reactivate(), ibis::zone::read(), ibis::zona::read(), ibis::pale::read(), ibis::pack::read(), ibis::fuzz::read(), ibis::fuge::read(), ibis::bylt::read(), ibis::ambit::read(), ibis::slice::read(), ibis::relic::read(), ibis::range::read(), ibis::keywords::read(), ibis::fade::read(), ibis::direkte::read(), ibis::egale::read(), ibis::bin::read(), ibis::fileManager::storage::read(), ibis::bitvector64::read(), ibis::bitvector::read(), ibis::array_t< T >::read(), ibis::query::readRIDs(), ibis::part::readRIDs(), ibis::bundle::readRIDs(), ibis::text::readString(), ibis::query::removeFiles(), ibis::part::reorder(), ibis::part::rollback(), ibis::column::saveSelected(), ibis::text::saveSelected(), ibis::column::searchSorted(), ibis::column::searchSortedOOCC(), ibis::column::searchSortedOOCD(), ibis::table::select(), ibis::filter::select(), ibis::column::selectBytes(), ibis::column::selectDoubles(), ibis::column::selectFloats(), ibis::column::selectInts(), ibis::column::selectLongs(), ibis::text::selectLongs(), ibis::blob::selectRawBytes(), ibis::column::selectShorts(), ibis::column::selectUBytes(), ibis::column::selectUInts(), ibis::column::selectULongs(), ibis::column::selectUShorts(), ibis::column::selectValuesT(), signalMemoryAvailable(), ibis::fileManager::softWriteLock::softWriteLock(), ibis::fileManager::storage::storage(), ibis::column::string2int(), ibis::text::stringSearch(), ibis::column::truncateData(), ibis::tafel::write(), ibis::zone::write(), ibis::zona::write(), ibis::pale::write(), ibis::pack::write(), ibis::fuzz::write(), ibis::fuge::write(), ibis::bylt::write(), ibis::ambit::write(), ibis::slice::write(), ibis::sbiad::write(), ibis::sapid::write(), ibis::roster::write(), ibis::relic::write(), ibis::range::write(), ibis::mesa::write(), ibis::keywords::write(), ibis::fade::write(), ibis::direkte::write(), ibis::moins::write(), ibis::egale::write(), ibis::bin::write(), ibis::column::writeData(), ibis::tafel::writeMetaData(), ibis::roster::writeSorted(), ibis::part::~part(), and ibis::fileManager::softWriteLock::~softWriteLock().

void ibis::fileManager::recordFile ( roFile st) [protected]

Record a newly allocated storage in the two lists.

The caller needs to hold a mutex lock on the file manager to ensure correct operations.

References ibis::fileManager::storage::begin(), ibis::fileManager::storage::end(), ibis::fileManager::storage::filename(), ibis::gVerbose, and ibis::fileManager::storage::size().

void ibis::fileManager::recordPages ( off_t  start,
off_t  stop 
) [inline]
void ibis::fileManager::signalMemoryAvailable ( ) const

Signal to the file manager that some memory have been freed.

To be used by clients that are aware of the memory usages of in-memory objects since the in-memory objects based on ibis::fileManager::storage does not produce signals when they are freed.

References ibis::gVerbose, and instance().

Referenced by ibis::relic::construct().

template<typename T >
int ibis::fileManager::tryGetFile ( const char *  name,
array_t< T > &  arr,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Given a file name, place the content in an array_t<T>.

This function will fail if there isn't enough memory to read the content of the file immediately.

The return value is zero (0) if the function is successful, otherwise returns a non-zero value.

References ibis::array_t< T >::clear(), ibis::gVerbose, ibis::array_t< T >::size(), and ibis::array_t< T >::swap().

Referenced by ibis::index::create(), and ibis::column::selectValuesT().

int ibis::fileManager::tryGetFile ( const char *  name,
storage **  st,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Try to retrieve the content of the named file.

The storage object *st returned from this function is owned by the fileManager. The caller is NOT to delete *st. This function will not wait for the fileManager to free any memory if there isn't enough free space available.

It returns 0 to indicate success and a negative value to indicate error. In particular, it returns -102 if there is not enough space to read the whole file into memory.

References ibis::fileManager::storage::begin(), ibis::horometer::CPUTime(), ibis::fileManager::roFile::doRead(), ibis::gVerbose, ibis::fileManager::roFile::isFileMap(), ibis::fileManager::roFile::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), and ibis::horometer::stop().


The documentation for this class was generated from the following files:

Make It A Bit Faster
Contact us
Disclaimers
FastBit source code
FastBit mailing list archive