Class FixedSizeDataStore
- java.lang.Object
-
- com.mckoi.database.FixedSizeDataStore
-
public final class FixedSizeDataStore extends java.lang.Object
A file format that allows for the very quick retreival of data that is stored within it. To allow for such quick reference of information in the file, we must make stipulations about how the data is stored.- Each data element in the table must be a fixed length. This could be thought of like a 'sector' of a disk drive. Or in a table, each row may be of the fixed size.
- We keep track of deleted rows via a linked list of sectors.
0 4 (int) : MAGIC - used to identify file type. 4 4 (int) : version - version of the file store. 8 4 (int) : sector_size - the size of each sector in the store. 12 8 (long) : delete_head - the head sector of the delete list. 20 8 (long) : sectors_used - number of sectors being used (not deleted). 28 1 (byte) : open - set to 1 when file opened, 0 when closed. 29 4 (int) : sector_start - offset where sector information starts. 33 ... 63 - reserved 64 ... 191 - reserved buffer for misc. state data. 192 ... sector_start - reserved
Each sector contains a 5 byte header. This header includes a byte that contains either USED or DELETED, and a int pointer to the next chained sector. The int pointer is used to either represent a pointer to the next sector in the chain of USED sectors, with -1 indicating the end. Or, if the sector is DELETED, it points to the next deleted sector in the chain.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private class
FixedSizeDataStore.SectorInputStream
An input stream that reads information across a sector chain starting at the given head sector.private class
FixedSizeDataStore.SectorOutputStream
A buffered OutputStream object that writes all data written to the stream out to the store.
-
Field Summary
Fields Modifier and Type Field Description private int
buffered_sector
The sector that we currently have loaded into the buffer.private java.io.File
data_file
The File that keeps the data.private java.io.RandomAccessFile
data_store
The RandomAccessFile object for the data file.private long
data_store_size
The file size of the data store.private DebugLogger
debug
A DebugLogger object we can use to write debug messages to.private int
delete_head
The head of the deleted sectors.private static byte
DELETED
The mark that indicates whether a sector is deleted (available) or being used.private static int
EXTRA_SECTOR_SIZE
The number of bytes that are stored in a sector in addition to the user data in a sector.private int
lock_count
The number of locks that have been put on this store.private static int
MAGIC
The Magic number used to help identify that the file we are reading is formatted as a fixed size data store.private boolean
read_only
Set to true if we opened the store in read only mode, otherwise false.private byte[]
sector_buffer
The sector buffer.private Cache
sector_cache
A cache of sectors read from the store.private static int
SECTOR_DATA_OFFSET
The offset in the file where the sector data starts.private int
sector_offset
The offset where the header information finishes and the sector data starts.private FixedSizeDataStore.SectorOutputStream
sector_output_stream
The last sector output stream that was created.private int
sector_size
The size of each 'sector'private static boolean
SECTORS_CACHED
If true then sectors are cached in the 'sector_cache'.private byte[]
sync_buffer
private static byte
USED
The mark that indicates whether a sector is deleted (available) or being used.private int
used_sector_count
The number of used sectors in the store.
-
Constructor Summary
Constructors Constructor Description FixedSizeDataStore(java.io.File data_file, int sector_size, boolean cache_access, DebugLogger logger)
Constructs the data store.FixedSizeDataStore(java.io.File data_file, int sector_size, DebugLogger logger)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description int
addSector(byte[] buf)
Adds a new sector into the store.int
addSector(byte[] buf, int offset, int length)
Adds a new sector into the store.int
calculateSectorSpan(int length)
Calculates the number of sectors the given length of bytes will span.boolean
clearDeletedSectors()
Cleans up so all deleted sectors are completely removed from the store.void
close()
Closes the data store.void
copyTo(java.io.File path)
Copies the entire contents of this store to a destination directory.void
delete()
Deletes the data store from the file system.void
deleteAcross(int sector_head)
Deletes a set of sectors that have been chained together.void
deleteAllSectors()
Deletes all sectors in the entire store.void
deleteSector(int sector)
Deletes a sector from the store.boolean
exists()
Returns true if the file for this store exists.private int
findFreeSector()
Finds the first free available sector that we can use.private int
findFreeSectorPastNext()
Finds the first free available sector past the next one.private int[]
findFreeSectors(int count)
Finds the first 'n' available sectors that we can use.void
fix(UserTerminal terminal)
Attempts to repair this data store to a correct state.int
getLengthOfLastOutputStream()
Returns the number of bytes that were written out by the last closed output stream returned by 'getSectorOutputStream'.byte[]
getSector(int sector)
Gets the contents of the sector at the given index.byte[]
getSector(int sector, byte[] buf)
Gets the contents of the sector at the given index.byte[]
getSector(int sector, byte[] buf, int offset, int length)
Gets the contents of the sector at the given index.int[]
getSectorAsIntArray(int sector, int[] buf)
Gets the contents of the sector at the given index as an int[] array.int[]
getSectorChain(int sector_head)
Traverses a sector chain and returns an array of all sectors that are part of the chain.int[]
getSectorChain(int sector_head, int length)
Traverses a sector chain and returns an array of all sectors that are part of the chain.java.io.InputStream
getSectorInputStream(int sector_head)
Returns an InputStream implementation that is used to read a stream of information from the store.int
getSectorOfLastOutputStream()
Returns the first sector the OutputStream returned by 'getSectorOutputStream' wrote to.java.io.OutputStream
getSectorOutputStream()
Returns an OutputStream implementation that is used to write a stream of information into this data store.int
getSectorSize()
Returns the number of bytes that the user may store in a sector.int
getSectorUseCount()
Returns the number of sectors in the store that are being used (as opposed to being deleted).void
hardSynch()
Performs a hard synchronization of this store.boolean
isClosed()
Returns true if the store is closed.boolean
isReadOnly()
Returns true if the store has been opened in read only mode.boolean
isSectorDeleted(int sector)
Returns true if the sector number is flagged as deleted.void
lock()
Locks the store by some process so that we may not reclaim deleted sectors.(package private) boolean
locked()
Returns true if the store is locked from reclaiming deleted rows.boolean
open(boolean read_only)
Opens the data store.int
overwriteSector(int sector, byte[] buf)
Writes the contents of a sector into the store overwritting any other information that may be stored there.int
overwriteSector(int sector, byte[] buf, int offset, int length)
Writes the contents of a sector into the store overwritting any other information that may be stored there.int
rawSectorCount()
Returns the total number of sectors that are currently available (includes used and deleted sectors).int
readAcross(int sector_head, byte[] buf, int offset, int length)
Reads information across a chain of sectors and fills the byte[] array buffer.void
readReservedBuffer(byte[] info, int offset, int length)
Reads from the buffer reserve into the given byte array.private void
readSector(int sector)
Read the 'nth' sector from the store and fills the internal 'sector_buffer' with the contents.private int
reclaimTopFree()
Reclaims the first sector from the free sector list.void
repair()
Repairs the consistancy of the store.private int
sectorCount()
Returns the total number of sectors in the file.private long
seekSector(int sector)
Seeks to the 'nth' sector in the store.private void
setDataStoreSize(long new_size)
Sets the length of the data store to the given size.private void
setSectorHeader(byte status, int next_sector)
Sets up the sector header information in 'sector_buffer'.java.lang.String
statusString()
Returns a string that contains diagnostic information.void
synch()
Synchronizes the memory store with the file header.long
totalSize()
Returns the size of the data store file.void
unlock()
Unlocks the store.void
wipeLastOutputStream()
Wipes the SectorOutputStream from this object.int
writeAcross(byte[] buf, int offset, int length)
Writes a byte[] array of data across as many sectors as it takes to store the data.private int
writeBufToSector(int sector, int next_sector, byte[] buf, int offset, int length)
Writes the contents of the byte[] array to the sector, setting the USED flag to true, and the 'next' int in the sector header.void
writeReservedBuffer(byte[] info, int offset, int length)
void
writeReservedBuffer(byte[] info, int offset, int length, int res_offset)
Every data store has a 128 byte buffer that can be used to store state information.private void
writeSector(int sector)
Writes the sector data in 'sector_buffer' to the given sector offset in the store.private void
writeSector(int sector, int length)
Writes the sector data in 'sector_buffer' to the given sector offset in the store.
-
-
-
Field Detail
-
MAGIC
private static final int MAGIC
The Magic number used to help identify that the file we are reading is formatted as a fixed size data store.- See Also:
- Constant Field Values
-
SECTOR_DATA_OFFSET
private static final int SECTOR_DATA_OFFSET
The offset in the file where the sector data starts.- See Also:
- Constant Field Values
-
EXTRA_SECTOR_SIZE
private static final int EXTRA_SECTOR_SIZE
The number of bytes that are stored in a sector in addition to the user data in a sector.- See Also:
- Constant Field Values
-
USED
private static final byte USED
The mark that indicates whether a sector is deleted (available) or being used.- See Also:
- Constant Field Values
-
DELETED
private static final byte DELETED
The mark that indicates whether a sector is deleted (available) or being used.- See Also:
- Constant Field Values
-
SECTORS_CACHED
private static final boolean SECTORS_CACHED
If true then sectors are cached in the 'sector_cache'.- See Also:
- Constant Field Values
-
debug
private DebugLogger debug
A DebugLogger object we can use to write debug messages to.
-
sector_size
private int sector_size
The size of each 'sector'
-
data_file
private java.io.File data_file
The File that keeps the data.
-
data_store
private java.io.RandomAccessFile data_store
The RandomAccessFile object for the data file.
-
read_only
private boolean read_only
Set to true if we opened the store in read only mode, otherwise false.
-
data_store_size
private long data_store_size
The file size of the data store.
-
sector_offset
private int sector_offset
The offset where the header information finishes and the sector data starts.
-
sector_buffer
private byte[] sector_buffer
The sector buffer. This is filled with the information in some given sector.
-
buffered_sector
private int buffered_sector
The sector that we currently have loaded into the buffer.
-
delete_head
private int delete_head
The head of the deleted sectors.
-
used_sector_count
private int used_sector_count
The number of used sectors in the store.
-
lock_count
private int lock_count
The number of locks that have been put on this store. If this number is > 0 then we may not reclaim deleted sector because another thread may be reading the data.
-
sector_cache
private Cache sector_cache
A cache of sectors read from the store.
-
sync_buffer
private byte[] sync_buffer
-
sector_output_stream
private FixedSizeDataStore.SectorOutputStream sector_output_stream
The last sector output stream that was created.
-
-
Constructor Detail
-
FixedSizeDataStore
public FixedSizeDataStore(java.io.File data_file, int sector_size, boolean cache_access, DebugLogger logger)
Constructs the data store. If 'sector_size' <= 0 then we determine sector size when the file is opened. If cached_access is true then all access to the store is through a cache which will greatly improve the performance of read dominated access.
-
FixedSizeDataStore
public FixedSizeDataStore(java.io.File data_file, int sector_size, DebugLogger logger)
-
-
Method Detail
-
locked
boolean locked()
Returns true if the store is locked from reclaiming deleted rows.
-
sectorCount
private int sectorCount() throws java.io.IOException
Returns the total number of sectors in the file.- Throws:
java.io.IOException
-
seekSector
private long seekSector(int sector) throws java.io.IOException
Seeks to the 'nth' sector in the store.- Throws:
java.io.IOException
-
readSector
private void readSector(int sector) throws java.io.IOException
Read the 'nth' sector from the store and fills the internal 'sector_buffer' with the contents.- Throws:
java.io.IOException
-
setDataStoreSize
private void setDataStoreSize(long new_size) throws java.io.IOException
Sets the length of the data store to the given size. This has a side- effect of setting the file pointer to the end of the file.- Throws:
java.io.IOException
-
writeSector
private void writeSector(int sector, int length) throws java.io.IOException
Writes the sector data in 'sector_buffer' to the given sector offset in the store.- Throws:
java.io.IOException
-
writeSector
private void writeSector(int sector) throws java.io.IOException
Writes the sector data in 'sector_buffer' to the given sector offset in the store.- Throws:
java.io.IOException
-
setSectorHeader
private void setSectorHeader(byte status, int next_sector) throws java.io.IOException
Sets up the sector header information in 'sector_buffer'.- Throws:
java.io.IOException
-
writeBufToSector
private int writeBufToSector(int sector, int next_sector, byte[] buf, int offset, int length) throws java.io.IOException
Writes the contents of the byte[] array to the sector, setting the USED flag to true, and the 'next' int in the sector header.NOTE: Assumes length is less than user space size of sector.
- Throws:
java.io.IOException
-
reclaimTopFree
private int reclaimTopFree() throws java.io.IOException
Reclaims the first sector from the free sector list.- Throws:
java.io.IOException
-
findFreeSector
private int findFreeSector() throws java.io.IOException
Finds the first free available sector that we can use. If we are reclaiming from the deleted list, the deleted row is taken from the linked list immediately.NOTE: This method may alter 'delete_head' changing the list of deleted sectors.
- Throws:
java.io.IOException
-
findFreeSectorPastNext
private int findFreeSectorPastNext() throws java.io.IOException
Finds the first free available sector past the next one. This means, if locked or delete_head == -1 then we return the sectorCount() + 1, otherwise we reclaim the next available of the delete queue.- Throws:
java.io.IOException
-
findFreeSectors
private int[] findFreeSectors(int count) throws java.io.IOException
Finds the first 'n' available sectors that we can use. If we are reclaiming from the deleted list, the deleted row(s) are taken from the linked list immediately.NOTE: This method may alter 'delete_head' changing the list of deleted sectors.
- Throws:
java.io.IOException
-
totalSize
public long totalSize()
Returns the size of the data store file. This is the total number of bytes stored in the data store.
-
writeReservedBuffer
public void writeReservedBuffer(byte[] info, int offset, int length, int res_offset) throws java.io.IOException
Every data store has a 128 byte buffer that can be used to store state information. The buffer starts at offset 64 of the file until offset 192. This method writes data to that offset.- Throws:
java.io.IOException
-
writeReservedBuffer
public void writeReservedBuffer(byte[] info, int offset, int length) throws java.io.IOException
- Throws:
java.io.IOException
-
readReservedBuffer
public void readReservedBuffer(byte[] info, int offset, int length) throws java.io.IOException
Reads from the buffer reserve into the given byte array.- Throws:
java.io.IOException
-
synch
public void synch() throws java.io.IOException
Synchronizes the memory store with the file header. This writes information into the header. This should be called periodically. Synch does nothing for a read only store.- Throws:
java.io.IOException
-
hardSynch
public void hardSynch() throws java.io.IOException
Performs a hard synchronization of this store. This will force the OS to synchronize the contents of the data store. hardSynch does nothing for a read only store.- Throws:
java.io.IOException
-
isReadOnly
public boolean isReadOnly()
Returns true if the store has been opened in read only mode.
-
open
public boolean open(boolean read_only) throws java.io.IOException
Opens the data store. The data store can be opened in 'read only' mode. Returns 'true' if the open procedure should repair itself (dirty open) or false if the file was cleanly closed down.It is not possible to open a damaged store in read only mode.
- Parameters:
read_only
- if true, then the database is opened in read only mode, otherwise it is opened in read/write mode.- Throws:
java.io.IOException
-
close
public void close() throws java.io.IOException
Closes the data store.- Throws:
java.io.IOException
-
isClosed
public boolean isClosed()
Returns true if the store is closed.
-
delete
public void delete()
Deletes the data store from the file system.
-
exists
public boolean exists() throws java.io.IOException
Returns true if the file for this store exists.- Throws:
java.io.IOException
-
getSectorSize
public int getSectorSize()
Returns the number of bytes that the user may store in a sector. The actual sector space in the file may be slightly larger.
-
getSectorUseCount
public int getSectorUseCount()
Returns the number of sectors in the store that are being used (as opposed to being deleted).
-
rawSectorCount
public int rawSectorCount() throws java.io.IOException
Returns the total number of sectors that are currently available (includes used and deleted sectors).- Throws:
java.io.IOException
-
lock
public void lock()
Locks the store by some process so that we may not reclaim deleted sectors. The purpose of this is in the situation where we have a slow thread accessing information from the store, and a seperate thread is still able to modifying (delete and add) to the store.
-
unlock
public void unlock()
Unlocks the store.
-
isSectorDeleted
public boolean isSectorDeleted(int sector) throws java.io.IOException
Returns true if the sector number is flagged as deleted. If returns false then the sector is being used.- Throws:
java.io.IOException
-
getSector
public byte[] getSector(int sector, byte[] buf, int offset, int length) throws java.io.IOException
Gets the contents of the sector at the given index.- Throws:
java.io.IOException
-
getSector
public byte[] getSector(int sector, byte[] buf) throws java.io.IOException
Gets the contents of the sector at the given index.- Throws:
java.io.IOException
-
getSector
public byte[] getSector(int sector) throws java.io.IOException
Gets the contents of the sector at the given index.- Throws:
java.io.IOException
-
getSectorAsIntArray
public int[] getSectorAsIntArray(int sector, int[] buf) throws java.io.IOException
Gets the contents of the sector at the given index as an int[] array. The array size is /4 of the sector size. If the sector size is not divisible by 4 then the last 1-3 bytes are truncated.- Throws:
java.io.IOException
-
readAcross
public int readAcross(int sector_head, byte[] buf, int offset, int length) throws java.io.IOException
Reads information across a chain of sectors and fills the byte[] array buffer. Returns the number of bytes that were read (should always be equal to 'length').- Throws:
java.io.IOException
-
getSectorChain
public int[] getSectorChain(int sector_head, int length) throws java.io.IOException
Traverses a sector chain and returns an array of all sectors that are part of the chain. Useful for diagnostic, repair and statistical operations.- Throws:
java.io.IOException
-
getSectorChain
public int[] getSectorChain(int sector_head) throws java.io.IOException
Traverses a sector chain and returns an array of all sectors that are part of the chain. Useful for diagnostic, repair and statistical operations.- Throws:
java.io.IOException
-
deleteSector
public void deleteSector(int sector) throws java.io.IOException
Deletes a sector from the store. The sector is only marked as deleted, however, and the contents may still be accessed via the 'getSector' methods. If the store is add locked, then it is guarenteed that no deleted sectors will be overwritten until the add lock is taken from the table.Throws an IO error if the sector is marked as deleted.
- Throws:
java.io.IOException
-
deleteAcross
public void deleteAcross(int sector_head) throws java.io.IOException
Deletes a set of sectors that have been chained together. This should be used to delete data added via the 'write' method. However, it can be used to delete data added via the 'addSector'- Throws:
java.io.IOException
-
deleteAllSectors
public void deleteAllSectors() throws java.io.IOException
Deletes all sectors in the entire store. Use with care.- Throws:
java.io.IOException
-
overwriteSector
public int overwriteSector(int sector, byte[] buf, int offset, int length) throws java.io.IOException
Writes the contents of a sector into the store overwritting any other information that may be stored there. This is used as a rough data editting command.- Throws:
java.io.IOException
-
overwriteSector
public int overwriteSector(int sector, byte[] buf) throws java.io.IOException
Writes the contents of a sector into the store overwritting any other information that may be stored there. This is used as a rough data editting command.- Throws:
java.io.IOException
-
addSector
public int addSector(byte[] buf, int offset, int length) throws java.io.IOException
Adds a new sector into the store. It finds a suitable sector to store the information and returns the sector number. If lock_count > 0 then we do not reclaim deleted sectors, otherwise we do.- Throws:
java.io.IOException
-
addSector
public int addSector(byte[] buf) throws java.io.IOException
Adds a new sector into the store. It finds a suitable sector to store the information and returns the sector number. If lock_count > 0 then we do not reclaim deleted sectors, otherwise we do.- Throws:
java.io.IOException
-
calculateSectorSpan
public int calculateSectorSpan(int length)
Calculates the number of sectors the given length of bytes will span.
-
writeAcross
public int writeAcross(byte[] buf, int offset, int length) throws java.io.IOException
Writes a byte[] array of data across as many sectors as it takes to store the data. Returns the index to the first sector that contains the start of the data.- Throws:
java.io.IOException
-
getSectorOutputStream
public java.io.OutputStream getSectorOutputStream() throws java.io.IOException
Returns an OutputStream implementation that is used to write a stream of information into this data store. As data is written into the stream, the data is flushed into this store at the next available sector. When the stream is closed, the entire contents of the stream will be contained within the store. A call to 'getSectorOfLastOutputStream' can be used to return an index that is used to reference this stream of information in the store.NOTE: While an output stream returned by this method is not closed, it is unsafe to use any methods in the FixedSizeDataStore object.
- Throws:
java.io.IOException
-
getSectorOfLastOutputStream
public int getSectorOfLastOutputStream()
Returns the first sector the OutputStream returned by 'getSectorOutputStream' wrote to. This is the start of the chain.
-
getLengthOfLastOutputStream
public int getLengthOfLastOutputStream()
Returns the number of bytes that were written out by the last closed output stream returned by 'getSectorOutputStream'.
-
wipeLastOutputStream
public void wipeLastOutputStream()
Wipes the SectorOutputStream from this object. This should be closed after the stream is closed.
-
getSectorInputStream
public java.io.InputStream getSectorInputStream(int sector_head) throws java.io.IOException
Returns an InputStream implementation that is used to read a stream of information from the store. This input stream will iterate through the sector chain given.NOTE: Using this InputStream, an end of stream identifier is never produced. When the last sector in the chain is reached, the input stream will first read padding whitespace, then it will either loop to the start of the last sector, or move to another undefined sector. You must not rely on this stream reaching an EOF.
- Throws:
java.io.IOException
-
copyTo
public void copyTo(java.io.File path) throws java.io.IOException
Copies the entire contents of this store to a destination directory. This can only be called when the data store is open. It makes an exact copy of the file.The purpose of this method is so we can make a copy of the data in this store while the store is open and 'live'.
We assume synchronization on this object.
- Parameters:
path
- the directory to copy this file to.- Throws:
java.io.IOException
-
fix
public void fix(UserTerminal terminal) throws java.io.IOException
Attempts to repair this data store to a correct state. The UserTerminal object can be used to ask the user questions and to output information on the progress of the repair.The store must have been opened before this method is called.
- Throws:
java.io.IOException
-
clearDeletedSectors
public boolean clearDeletedSectors() throws java.io.IOException
Cleans up so all deleted sectors are completely removed from the store. This has the effect of reducing the size of the file by the size of every deleted sector.It is extremely important that nothing can be read/written from the file while this is happening. And certainly, we can not have any locks on this store.
Returns true if the layout of the sectors changed (so we can fix indices that point to sectors).
- Throws:
java.io.IOException
-
repair
public void repair() throws java.io.IOException
Repairs the consistancy of the store. This is an expensive operation that runs through every sector and determines if it's deleted or used. If it's deleted it is added into the deleted linked list.Repair assumes we can at least get past the 'open' method. This method does not change the order of the sectors in the store. However it may change the order in which deleted sectors are reclaimed.
In a perfect world, this should never need to be called. However, it's a good idea to call this every so often because we are assured that the delete linked list and 'used_sector_count' variables will be correct when the method returns.
It is not possible to repair a store that's been opened in read only mode.
- Throws:
java.io.IOException
-
statusString
public java.lang.String statusString() throws java.io.IOException
Returns a string that contains diagnostic information.- Throws:
java.io.IOException
-
-