com.sleepycat.je.cleaner
Class Cleaner

java.lang.Object
  extended by com.sleepycat.je.cleaner.Cleaner
All Implemented Interfaces:
EnvConfigObserver, DaemonRunner, ExceptionListenerUser

public class Cleaner
extends Object
implements DaemonRunner, EnvConfigObserver, ExceptionListenerUser

The Cleaner is responsible for effectively garbage collecting the JE log. It looks through log files and locates log records (IN's and LN's of all flavors) that are superceded by later versions. Those that are "current" are propagated to a newer log file so that older log files can be deleted.


Field Summary
(package private) static String CLEAN_IN
           
(package private) static String CLEAN_LN
           
(package private) static String CLEAN_MIGRATE_LN
           
(package private) static String CLEAN_PENDING_LN
           
(package private)  long cleanerBytesInterval
           
(package private)  boolean clusterAll
           
(package private)  boolean clusterResident
           
(package private) static boolean DO_CRITICAL_EVICTION
          Whether the cleaner should participate in critical eviction.
(package private)  boolean expunge
           
(package private)  boolean fetchObsoleteSize
           
(package private)  boolean lazyMigration
           
(package private)  long lockTimeout
           
(package private)  int lookAheadCacheSize
           
(package private)  int maxBatchFiles
           
(package private)  LongStat nCleanerDeletions
           
(package private)  LongStat nCleanerRuns
           
(package private)  LongStat nClusterLNsProcessed
           
(package private)  long nDeadlockRetries
           
(package private)  LongStat nEntriesRead
           
(package private)  LongStat nINsCleaned
           
(package private)  LongStat nINsDead
           
(package private)  LongStat nINsMigrated
           
(package private)  LongStat nINsObsolete
           
(package private)  LongStat nLNQueueHits
           
(package private)  LongStat nLNsCleaned
           
(package private)  LongStat nLNsDead
           
(package private)  LongStat nLNsLocked
           
(package private)  LongStat nLNsMarked
           
(package private)  LongStat nLNsMigrated
           
(package private)  LongStat nLNsObsolete
           
(package private)  LongStat nMarkedLNsProcessed
           
(package private)  LongStat nPendingLNsLocked
           
(package private)  LongStat nPendingLNsProcessed
           
(package private)  LongStat nRepeatIteratorReads
           
(package private)  LongStat nToBeCleanedLNsProcessed
           
(package private)  int readBufferSize
           
(package private)  StatGroup stats
           
(package private)  LongStat totalLogSize
           
(package private)  boolean trackDetail
           
(package private) static CacheMode UPDATE_GENERATION
          Whether to update the IN generation count during searches.
 
Constructor Summary
Cleaner(EnvironmentImpl env, String name)
           
 
Method Summary
(package private)  void addPendingDB(DatabaseImpl db)
          Adds the DB ID to the pending DB set if it is being deleted but deletion is not yet complete.
 void addProtectedFileRange(long firstProtectedFile)
          Adds a range of log files to be protected from deletion during a backup or similar procedures where log files must not be deleted.
 void close()
          Release resources and update memory budget.
(package private)  void deleteSafeToDeleteFiles()
          Deletes all files that are safe-to-delete and which are not protected by a DbBackup or replication.
 int doClean(boolean cleanMultipleFiles, boolean forceCleaning)
          Cleans selected files and returns the number of files cleaned.
 void envConfigUpdate(DbConfigManager cm, EnvironmentMutableConfig ignore)
          Process notifications of mutable property changes.
 boolean getFetchObsoleteSize()
           
 FileSelector.CheckpointStartCleanerState getFilesAtCheckpointStart()
          Returns a copy of the cleaned and processed files at the time a checkpoint starts.
(package private)  FileSelector getFileSelector()
          For unit testing.
(package private)  byte[] getLNMainKey(BIN bin, int index)
          Returns the main key for a given BIN entry.
 int getNWakeupRequests()
           
 UtilizationProfile getUtilizationProfile()
           
 UtilizationTracker getUtilizationTracker()
           
 boolean isEvictable(BIN bin, int index)
          Returns whether the given BIN entry may be stripped by the evictor.
 void lazyMigrateDupCountLN(DIN din, ChildReference dclRef, boolean backgroundIO)
          This method should be called just before logging a root DIN.
 void lazyMigrateLNs(BIN bin, boolean backgroundIO)
          This method should be called just before logging a BIN.
 StatGroup loadStats(StatsConfig config)
          Load stats.
(package private)  void logFine(String action, Node node, long logLsn, boolean completed, boolean obsolete, boolean dirtiedMigrated)
          Send trace messages to the java.util.logger.
(package private)  void processPending()
          If any LNs are pending, process them.
 void removeProtectedFileRange(long firstProtectedFile)
          Removes a range of log files to be protected after calling addProtectedFileRange(long).
 void requestShutdown()
           
 void runOrPause(boolean run)
           
 void setExceptionListener(ExceptionListener exceptionListener)
           
 void shutdown()
           
 void updateFilesAtCheckpointEnd(FileSelector.CheckpointStartCleanerState info)
          When a checkpoint is complete, update the files that were returned at the beginning of the checkpoint.
 void updateReadOnlyFileCollections()
          Update the lowUtilizationFiles and toBeCleanedFiles fields with new read-only collections.
 void wakeup()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

CLEAN_IN

static final String CLEAN_IN
See Also:
Constant Field Values

CLEAN_LN

static final String CLEAN_LN
See Also:
Constant Field Values

CLEAN_MIGRATE_LN

static final String CLEAN_MIGRATE_LN
See Also:
Constant Field Values

CLEAN_PENDING_LN

static final String CLEAN_PENDING_LN
See Also:
Constant Field Values

UPDATE_GENERATION

static final CacheMode UPDATE_GENERATION
Whether to update the IN generation count during searches. This is currently disabled because 1) we update the generation of the BIN when we set a MIGRATE flag and 2) if the BIN is not evicted its parents will not be, so not updating the generation during the search has no benefit. By not updating the generation during searches for which we do NOT set the MIGRATE flag, we avoid holding INs in the cache that are not needed for lazy migration. However, we do very few searches for obsolete LNs because the obsolete tracking info prevents this, so the benefit of not updating the generation during searches is questionable. In other words, changing this setting will have little effect.


DO_CRITICAL_EVICTION

static final boolean DO_CRITICAL_EVICTION
Whether the cleaner should participate in critical eviction. Ideally the cleaner would not participate in eviction, since that would reduce the cost of cleaning. However, the cleaner can add large numbers of nodes to the cache. By not participating in eviction, other threads could be kept in a constant state of eviction and would effectively starve. Therefore, this setting is currently enabled.

See Also:
Constant Field Values

stats

StatGroup stats

nCleanerRuns

LongStat nCleanerRuns

nCleanerDeletions

LongStat nCleanerDeletions

nINsObsolete

LongStat nINsObsolete

nINsCleaned

LongStat nINsCleaned

nINsDead

LongStat nINsDead

nINsMigrated

LongStat nINsMigrated

nLNsObsolete

LongStat nLNsObsolete

nLNsCleaned

LongStat nLNsCleaned

nLNsDead

LongStat nLNsDead

nLNsLocked

LongStat nLNsLocked

nLNsMigrated

LongStat nLNsMigrated

nLNsMarked

LongStat nLNsMarked

nLNQueueHits

LongStat nLNQueueHits

nPendingLNsProcessed

LongStat nPendingLNsProcessed

nMarkedLNsProcessed

LongStat nMarkedLNsProcessed

nToBeCleanedLNsProcessed

LongStat nToBeCleanedLNsProcessed

nClusterLNsProcessed

LongStat nClusterLNsProcessed

nPendingLNsLocked

LongStat nPendingLNsLocked

nEntriesRead

LongStat nEntriesRead

nRepeatIteratorReads

LongStat nRepeatIteratorReads

totalLogSize

LongStat totalLogSize

lockTimeout

long lockTimeout

readBufferSize

int readBufferSize

lookAheadCacheSize

int lookAheadCacheSize

nDeadlockRetries

long nDeadlockRetries

expunge

boolean expunge

clusterResident

boolean clusterResident

clusterAll

boolean clusterAll

maxBatchFiles

int maxBatchFiles

cleanerBytesInterval

long cleanerBytesInterval

trackDetail

boolean trackDetail

fetchObsoleteSize

boolean fetchObsoleteSize

lazyMigration

boolean lazyMigration
Constructor Detail

Cleaner

public Cleaner(EnvironmentImpl env,
               String name)
        throws DatabaseException
Throws:
DatabaseException
Method Detail

envConfigUpdate

public void envConfigUpdate(DbConfigManager cm,
                            EnvironmentMutableConfig ignore)
                     throws DatabaseException
Process notifications of mutable property changes.

Specified by:
envConfigUpdate in interface EnvConfigObserver
Throws:
IllegalArgumentException - via Environment ctor and setMutableConfig.
DatabaseException

getUtilizationTracker

public UtilizationTracker getUtilizationTracker()

getUtilizationProfile

public UtilizationProfile getUtilizationProfile()

getFileSelector

FileSelector getFileSelector()
For unit testing.


getFetchObsoleteSize

public boolean getFetchObsoleteSize()

runOrPause

public void runOrPause(boolean run)
Specified by:
runOrPause in interface DaemonRunner

wakeup

public void wakeup()

requestShutdown

public void requestShutdown()
Specified by:
requestShutdown in interface DaemonRunner

shutdown

public void shutdown()
Specified by:
shutdown in interface DaemonRunner

getNWakeupRequests

public int getNWakeupRequests()
Specified by:
getNWakeupRequests in interface DaemonRunner

setExceptionListener

public void setExceptionListener(ExceptionListener exceptionListener)
Specified by:
setExceptionListener in interface DaemonRunner
Specified by:
setExceptionListener in interface ExceptionListenerUser
See Also:
ExceptionListenerUser.setExceptionListener(ExceptionListener)

doClean

public int doClean(boolean cleanMultipleFiles,
                   boolean forceCleaning)
            throws DatabaseException
Cleans selected files and returns the number of files cleaned. This method is not invoked by a deamon thread, it is programatically.

Parameters:
cleanMultipleFiles - is true to clean until we're under budget, or false to clean at most one file.
forceCleaning - is true to clean even if we're not under the utilization threshold.
Returns:
the number of files cleaned, not including files cleaned unsuccessfully.
Throws:
DatabaseException

loadStats

public StatGroup loadStats(StatsConfig config)
Load stats.


deleteSafeToDeleteFiles

void deleteSafeToDeleteFiles()
                       throws DatabaseException
Deletes all files that are safe-to-delete and which are not protected by a DbBackup or replication. Files are deleted only if there are no read-only processes. Log file deletion is coordinated by the use of three mechanisms: 1) To guard against read/only processes, the would-be deleter tries to get an exclusive lock on the environment. This will not be possible if a read/only process exists. File locks must be used for inter-process coordination. But note that file locks are not supported intra-process. 2) Synchronization on the protectedFileRanges field. Elements are added to and removed from the protectedFileRanges collection by DbBackup. More than one backup may be occuring at once, hence a collection of protectedFileRanges is maintained, and the files protected are the range starting with the minimum value returned by the objects in that collection. 3) In a replicated environment, files are protected from deletion by the CBVLSN (CleanerBarrier VLSN). No file greater or equal to the CBVLSN file may be deleted. For case (2) and (3), all coordinated activities -- replication, backup and file deletion -- can only be carried out by a read-write process, so we know that all activities are occurring in the same process because there can only be one JE read-write process per environment. This method is synchronized to prevent multiple threads from requesting the environment lock or deleting the same files.

Throws:
DatabaseException

addProtectedFileRange

public void addProtectedFileRange(long firstProtectedFile)
Adds a range of log files to be protected from deletion during a backup or similar procedures where log files must not be deleted.

This method is called automatically by the DbBackup utility and is provided here as a separate API for advanced applications that may implement a custom backup procedure.

WARNING: After calling this method, deletion of log files in the file range by the JE log cleaner will be disabled until removeProtectedFileRange(long) is called. To prevent unbounded growth of disk usage, be sure to call removeProtectedFileRange(long) to re-enable log file deletion.

Parameters:
firstProtectedFile - the number of the first file to be protected. The protected range is from this file number to the last (highest numbered) file in the log.
Since:
4.0

removeProtectedFileRange

public void removeProtectedFileRange(long firstProtectedFile)
Removes a range of log files to be protected after calling addProtectedFileRange(long).

Parameters:
firstProtectedFile - the value previously passed to addProtectedFileRange(long).
Throws:
EnvironmentFailureException - if firstProtectedFile is not currently the start of a protected range.
Since:
4.0

getFilesAtCheckpointStart

public FileSelector.CheckpointStartCleanerState getFilesAtCheckpointStart()
                                                                   throws DatabaseException
Returns a copy of the cleaned and processed files at the time a checkpoint starts.

If non-null is returned, the checkpoint should flush an extra level, and addCheckpointedFiles() should be called when the checkpoint is complete.

Throws:
DatabaseException

updateFilesAtCheckpointEnd

public void updateFilesAtCheckpointEnd(FileSelector.CheckpointStartCleanerState info)
                                throws DatabaseException
When a checkpoint is complete, update the files that were returned at the beginning of the checkpoint.

Throws:
DatabaseException

updateReadOnlyFileCollections

public void updateReadOnlyFileCollections()
Update the lowUtilizationFiles and toBeCleanedFiles fields with new read-only collections.


processPending

void processPending()
              throws DatabaseException
If any LNs are pending, process them. This method should be called often enough to prevent the pending LN set from growing too large.

Throws:
DatabaseException

isEvictable

public boolean isEvictable(BIN bin,
                           int index)
Returns whether the given BIN entry may be stripped by the evictor. True is always returned if the BIN is not dirty. False is returned if the BIN is dirty and the entry will be migrated soon. Note that the BIN may or may not be latched when this method is called. Returning the wrong answer is OK in that case (it will be called again later when latched), but an exception should not occur.


lazyMigrateLNs

public void lazyMigrateLNs(BIN bin,
                           boolean backgroundIO)
                    throws DatabaseException
This method should be called just before logging a BIN. LNs will be migrated if the MIGRATE flag is set, or if they are in a file to be cleaned, or if the LNs qualify according to the rules for cluster and clusterAll.

On return this method guarantees that no MIGRATE flag will be set on any child entry. If this method is *not* called before logging a BIN, then the addPendingLNs method must be called.

Parameters:
bin - is the latched BIN. The latch will not be released by this method.
Throws:
DatabaseException

lazyMigrateDupCountLN

public void lazyMigrateDupCountLN(DIN din,
                                  ChildReference dclRef,
                                  boolean backgroundIO)
                           throws DatabaseException
This method should be called just before logging a root DIN. The DupCountLN will be migrated if the MIGRATE flag is set, or if it is in a file to be cleaned, or if the LN qualifies according to the rules for cluster and clusterAll.

On return this method guarantees that the MIGRATE flag will not be set on the child entry. If this method is *not* called before logging a root DIN, then the addPendingDupCountLN method must be called.

Parameters:
din - is the latched DIN. The latch will not be released by this method.
dclRef - is the reference to the DupCountLN.
Throws:
DatabaseException

getLNMainKey

byte[] getLNMainKey(BIN bin,
                    int index)
Returns the main key for a given BIN entry.


addPendingDB

void addPendingDB(DatabaseImpl db)
Adds the DB ID to the pending DB set if it is being deleted but deletion is not yet complete.


logFine

void logFine(String action,
             Node node,
             long logLsn,
             boolean completed,
             boolean obsolete,
             boolean dirtiedMigrated)
Send trace messages to the java.util.logger. Don't rely on the logger alone to conditionalize whether we send this message, we don't even want to construct the message if the level is not enabled.


close

public void close()
Release resources and update memory budget. Should only be called when this environment is closed and will never be accessed again.



Copyright (c) 2004-2010 Oracle. All rights reserved.