com.sleepycat.je.cleaner
Class Cleaner

java.lang.Object
  extended by com.sleepycat.je.cleaner.Cleaner
All Implemented Interfaces:
EnvConfigObserver, DaemonRunner

public class Cleaner
extends Object
implements DaemonRunner, EnvConfigObserver

The Cleaner is responsible for effectively garbage collecting the JE log. It looks through log files and locates log records (IN's and LN's of all flavors) that are superceded by later versions. Those that are "current" are propagated to a newer log file so that older log files can be deleted.


Field Summary
(package private) static String CLEAN_IN
           
(package private) static String CLEAN_LN
           
(package private) static String CLEAN_MIGRATE_LN
           
(package private) static String CLEAN_PENDING_LN
           
(package private)  long cleanerBytesInterval
           
(package private)  boolean clusterAll
           
(package private)  boolean clusterResident
           
(package private)  Level detailedTraceLevel
           
(package private) static boolean DO_CRITICAL_EVICTION
          Whether the cleaner should participate in critical eviction.
(package private)  boolean expunge
           
(package private)  boolean fetchObsoleteSize
           
(package private)  long lockTimeout
           
(package private)  int lookAheadCacheSize
           
(package private)  Set lowUtilizationFiles
          All files that are below the minUtilization threshold.
(package private)  int maxBatchFiles
           
(package private)  Set mustBeCleanedFiles
          All files that are to-be-cleaning or being-cleaned.
(package private)  int nBacklogFiles
           
(package private)  int nCleanerDeletions
           
(package private)  int nCleanerRuns
           
(package private)  int nClusterLNsProcessed
           
(package private)  int nDeadlockRetries
           
(package private)  int nEntriesRead
           
(package private)  int nINsCleaned
           
(package private)  int nINsDead
           
(package private)  int nINsMigrated
           
(package private)  int nINsObsolete
           
(package private)  int nLNQueueHits
           
(package private)  int nLNsCleaned
           
(package private)  int nLNsDead
           
(package private)  int nLNsLocked
           
(package private)  int nLNsMarked
           
(package private)  int nLNsMigrated
           
(package private)  int nLNsObsolete
           
(package private)  int nMarkedLNsProcessed
           
(package private)  int nPendingLNsLocked
           
(package private)  int nPendingLNsProcessed
           
(package private)  long nRepeatIteratorReads
           
(package private)  int nToBeCleanedLNsProcessed
           
(package private) static boolean PROACTIVE_MIGRATION
          Whether to fetch LNs for files in the to-be-cleaned set during lazy migration.
(package private)  int readBufferSize
           
(package private)  boolean trackDetail
           
(package private) static boolean UPDATE_GENERATION
          Whether to update the IN generation count during searches.
 
Constructor Summary
Cleaner(EnvironmentImpl env, String name)
           
 
Method Summary
(package private)  void addPendingDB(DatabaseImpl db)
          Adds the DB ID to the pending DB set if it is being deleted but deletion is not yet complete.
 void clearDeleteProhibited()
           
(package private)  void deleteSafeToDeleteFiles()
          Deletes all files that are safe-to-delete, if there are no read/only processes and concurrent backups.
 int doClean(boolean cleanMultipleFiles, boolean forceCleaning)
          Cleans selected files and returns the number of files cleaned.
 void envConfigUpdate(DbConfigManager cm)
          Process notifications of mutable property changes.
 boolean getFetchObsoleteSize()
           
 FileSelector.CheckpointStartCleanerState getFilesAtCheckpointStart()
          Returns a copy of the cleaned and processed files at the time a checkpoint starts.
 int getNWakeupRequests()
           
 UtilizationProfile getUtilizationProfile()
           
 UtilizationTracker getUtilizationTracker()
           
(package private)  void injectFileForCleaning(Long fileNum)
          For unit testing.
 boolean isEvictable(BIN bin, int index)
          Returns whether the given BIN entry may be stripped by the evictor.
 void lazyMigrateDupCountLN(DIN din, ChildReference dclRef, boolean proactiveMigration)
          This method should be called just before logging a root DIN.
 void lazyMigrateLNs(BIN bin, boolean proactiveMigration, boolean backgroundIO)
          This method should be called just before logging a BIN.
 void loadStats(StatsConfig config, EnvironmentStats stat)
          Load stats.
(package private)  void processPending()
          If any LNs are pending, process them.
 void requestShutdown()
           
 void runOrPause(boolean run)
           
 void setDeleteProhibited()
           
 void shutdown()
           
(package private)  void trace(Level level, String action, Node node, long logLsn, boolean completed, boolean obsolete, boolean dirtiedMigrated)
          Send trace messages to the java.util.logger.
 void updateFilesAtCheckpointEnd(FileSelector.CheckpointStartCleanerState info)
          When a checkpoint is complete, update the files that were returned at the beginning of the checkpoint.
 void updateReadOnlyFileCollections()
          Update the lowUtilizationFiles and mustBeCleanedFiles fields with new read-only collections, and update the backlog file count.
 void wakeup()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

CLEAN_IN

static final String CLEAN_IN
See Also:
Constant Field Values

CLEAN_LN

static final String CLEAN_LN
See Also:
Constant Field Values

CLEAN_MIGRATE_LN

static final String CLEAN_MIGRATE_LN
See Also:
Constant Field Values

CLEAN_PENDING_LN

static final String CLEAN_PENDING_LN
See Also:
Constant Field Values

PROACTIVE_MIGRATION

static final boolean PROACTIVE_MIGRATION
Whether to fetch LNs for files in the to-be-cleaned set during lazy migration. This is currently enabled because we do not support the dynamic addition of cleaner threads; that way, if the configured cleaner threads cannot keep up, we use proactive migration to keep up.

See Also:
Constant Field Values

UPDATE_GENERATION

static final boolean UPDATE_GENERATION
Whether to update the IN generation count during searches. This is currently disabled because 1) we update the generation of the BIN when we set a MIGRATE flag and 2) if the BIN is not evicted its parents will not be, so not updating the generation during the search has no benefit. By not updating the generation during searches for which we do NOT set the MIGRATE flag, we avoid holding INs in the cache that are not needed for lazy migration. However, we do very few searches for obsolete LNs because the obsolete tracking info prevents this, so the benefit of not updating the generation during searches is questionable. In other words, changing this setting will have little effect.

See Also:
Constant Field Values

DO_CRITICAL_EVICTION

static final boolean DO_CRITICAL_EVICTION
Whether the cleaner should participate in critical eviction. Ideally the cleaner would not participate in eviction, since that would reduce the cost of cleaning. However, the cleaner can add large numbers of nodes to the cache. By not participating in eviction, other threads could be kept in a constant state of eviction and would effectively starve. Therefore, this setting is currently enabled.

See Also:
Constant Field Values

nBacklogFiles

int nBacklogFiles

nCleanerRuns

int nCleanerRuns

nCleanerDeletions

int nCleanerDeletions

nINsObsolete

int nINsObsolete

nINsCleaned

int nINsCleaned

nINsDead

int nINsDead

nINsMigrated

int nINsMigrated

nLNsObsolete

int nLNsObsolete

nLNsCleaned

int nLNsCleaned

nLNsDead

int nLNsDead

nLNsLocked

int nLNsLocked

nLNsMigrated

int nLNsMigrated

nLNsMarked

int nLNsMarked

nLNQueueHits

int nLNQueueHits

nPendingLNsProcessed

int nPendingLNsProcessed

nMarkedLNsProcessed

int nMarkedLNsProcessed

nToBeCleanedLNsProcessed

int nToBeCleanedLNsProcessed

nClusterLNsProcessed

int nClusterLNsProcessed

nPendingLNsLocked

int nPendingLNsLocked

nEntriesRead

int nEntriesRead

nRepeatIteratorReads

long nRepeatIteratorReads

lockTimeout

long lockTimeout

readBufferSize

int readBufferSize

lookAheadCacheSize

int lookAheadCacheSize

nDeadlockRetries

int nDeadlockRetries

expunge

boolean expunge

clusterResident

boolean clusterResident

clusterAll

boolean clusterAll

maxBatchFiles

int maxBatchFiles

detailedTraceLevel

Level detailedTraceLevel

cleanerBytesInterval

long cleanerBytesInterval

trackDetail

boolean trackDetail

fetchObsoleteSize

boolean fetchObsoleteSize

mustBeCleanedFiles

Set mustBeCleanedFiles
All files that are to-be-cleaning or being-cleaned. Used to perform proactive migration. Is read-only after assignment, so no synchronization is needed.


lowUtilizationFiles

Set lowUtilizationFiles
All files that are below the minUtilization threshold. Used to perform clustering migration. Is read-only after assignment, so no synchronization is needed.

Constructor Detail

Cleaner

public Cleaner(EnvironmentImpl env,
               String name)
        throws DatabaseException
Throws:
DatabaseException
Method Detail

envConfigUpdate

public void envConfigUpdate(DbConfigManager cm)
                     throws DatabaseException
Process notifications of mutable property changes.

Specified by:
envConfigUpdate in interface EnvConfigObserver
Throws:
DatabaseException

getUtilizationTracker

public UtilizationTracker getUtilizationTracker()

getUtilizationProfile

public UtilizationProfile getUtilizationProfile()

getFetchObsoleteSize

public boolean getFetchObsoleteSize()

runOrPause

public void runOrPause(boolean run)
Specified by:
runOrPause in interface DaemonRunner

wakeup

public void wakeup()

requestShutdown

public void requestShutdown()
Specified by:
requestShutdown in interface DaemonRunner

shutdown

public void shutdown()
Specified by:
shutdown in interface DaemonRunner

getNWakeupRequests

public int getNWakeupRequests()
Specified by:
getNWakeupRequests in interface DaemonRunner

doClean

public int doClean(boolean cleanMultipleFiles,
                   boolean forceCleaning)
            throws DatabaseException
Cleans selected files and returns the number of files cleaned. This method is not invoked by a deamon thread, it is programatically.

Parameters:
cleanMultipleFiles - is true to clean until we're under budget, or false to clean at most one file.
forceCleaning - is true to clean even if we're not under the utilization threshold.
Returns:
the number of files cleaned, not including files cleaned unsuccessfully.
Throws:
DatabaseException

loadStats

public void loadStats(StatsConfig config,
                      EnvironmentStats stat)
               throws DatabaseException
Load stats.

Throws:
DatabaseException

injectFileForCleaning

void injectFileForCleaning(Long fileNum)
For unit testing.


deleteSafeToDeleteFiles

void deleteSafeToDeleteFiles()
                       throws DatabaseException
Deletes all files that are safe-to-delete, if there are no read/only processes and concurrent backups. Deletion is coordinated by the synchronization variable deleteFileLock AND by the deleteProhibited state variable. The reason that two different mechanisms are use is because file deletion must be prevented both inter and intra-process. File locks must be used for inter-process, and the state bit for intra-process. To guard against read/only processes, the would-be deleter tries to get an exclusive lock on the environment. This will not be possible if a read/only process exists. To guard against backup mode, the would-be deleter checks the deleteProhibited state. Backup and file deletion can only be carried out by a read/write process, so both activities are working in the same process. Note that file locks are not supported intra-process. The deleteProhibited state is used rather than a simple synchronization on deleteFileLock because the start/endBackup is controlled by the application, and the copying of log files can take an arbitrarily long time. Using synchronization on deleteFileLock would make it possible to lock out a cleaner thread for an unacceptable amount of time.

Throws:
DatabaseException

setDeleteProhibited

public void setDeleteProhibited()

clearDeleteProhibited

public void clearDeleteProhibited()

getFilesAtCheckpointStart

public FileSelector.CheckpointStartCleanerState getFilesAtCheckpointStart()
                                                                   throws DatabaseException
Returns a copy of the cleaned and processed files at the time a checkpoint starts.

If non-null is returned, the checkpoint should flush an extra level, and addCheckpointedFiles() should be called when the checkpoint is complete.

Throws:
DatabaseException

updateFilesAtCheckpointEnd

public void updateFilesAtCheckpointEnd(FileSelector.CheckpointStartCleanerState info)
                                throws DatabaseException
When a checkpoint is complete, update the files that were returned at the beginning of the checkpoint.

Throws:
DatabaseException

updateReadOnlyFileCollections

public void updateReadOnlyFileCollections()
Update the lowUtilizationFiles and mustBeCleanedFiles fields with new read-only collections, and update the backlog file count.


processPending

void processPending()
              throws DatabaseException
If any LNs are pending, process them. This method should be called often enough to prevent the pending LN set from growing too large.

Throws:
DatabaseException

isEvictable

public boolean isEvictable(BIN bin,
                           int index)
Returns whether the given BIN entry may be stripped by the evictor. True is always returned if the BIN is not dirty. False is returned if the BIN is dirty and the entry will be migrated soon.


lazyMigrateLNs

public void lazyMigrateLNs(BIN bin,
                           boolean proactiveMigration,
                           boolean backgroundIO)
                    throws DatabaseException
This method should be called just before logging a BIN. LNs will be migrated if the MIGRATE flag is set, or if they are in a file to be cleaned, or if the LNs qualify according to the rules for cluster and clusterAll.

On return this method guarantees that no MIGRATE flag will be set on any child entry. If this method is *not* called before logging a BIN, then the addPendingLNs method must be called.

Parameters:
bin - is the latched BIN. The latch will not be released by this method.
proactiveMigration - perform proactive migration if needed; this is false during a split, to reduce the delay in the user operation.
Throws:
DatabaseException

lazyMigrateDupCountLN

public void lazyMigrateDupCountLN(DIN din,
                                  ChildReference dclRef,
                                  boolean proactiveMigration)
                           throws DatabaseException
This method should be called just before logging a root DIN. The DupCountLN will be migrated if the MIGRATE flag is set, or if it is in a file to be cleaned, or if the LN qualifies according to the rules for cluster and clusterAll.

On return this method guarantees that the MIGRATE flag will not be set on the child entry. If this method is *not* called before logging a root DIN, then the addPendingDupCountLN method must be called.

Parameters:
din - is the latched DIN. The latch will not be released by this method.
dclRef - is the reference to the DupCountLN.
proactiveMigration - perform proactive migration if needed; this is false during a split, to reduce the delay in the user operation.
Throws:
DatabaseException

addPendingDB

void addPendingDB(DatabaseImpl db)
Adds the DB ID to the pending DB set if it is being deleted but deletion is not yet complete.


trace

void trace(Level level,
           String action,
           Node node,
           long logLsn,
           boolean completed,
           boolean obsolete,
           boolean dirtiedMigrated)
Send trace messages to the java.util.logger. Don't rely on the logger alone to conditionalize whether we send this message, we don't even want to construct the message if the level is not enabled.



Copyright 2004,2008 Oracle. All rights reserved.