Safe Haskell | None |
---|---|
Language | Haskell2010 |
Ouroboros.Consensus.Storage.VolatileDB.Impl
Contents
Description
Volatile on-disk database of blocks
Logic
The VolatileDB is a key-value store of blocks indexed by their hashes. It is
parameterised by the block type
blk
.
The "volatile" in the name refers to the fact that the blocks stored in it
make up the
volatile
part of the chain, i.e., the last
k
blocks of the
chain, which can still be rolled back. Not only the last
k
blocks of the
current chain are stored in this database, but also blocks of forks which we
have switched from or will switch to.
The VolatileDB appends new blocks sequentially to a file. When
volMaxBlocksPerFile
are stored in the current file, a new file is started.
The VolatileDB provides four main operations:
-
Adding blocks with
putBlock
-
Get blocks or information about them with
getBlockComponent
-
Accessing the in-memory indices using
getBlockInfo
andfilterByPredecessor
-
Garbage collecting blocks older than a given slot using
garbageCollect
Garbage collection will only delete a file from the VolatileDB when all
blocks in it have a slot older than the one passed to
garbageCollect
.
Errors
When an exception occurs while modifying the VolatileDB, we close the database as a safety measure, e.g., in case a file could not be written to disk, as we can no longer make sure the in-memory indices match what's stored on the file system. When reopening, we validate the blocks stored in the file system and reconstruct the in-memory indices.
NOTE: this means that when a thread modifying the VolatileDB is killed, the database will be closed. This is an intentional choice to simplify things.
The in-memory indices can always be reconstructed from the file system. This is important, as we must be resilient against unexpected shutdowns, power losses, etc.
We achieve this by only performing basic operations on the VolatileDB:
*
putBlock
only appends a new block to a file. Losing an update means we
only lose a block, which is not a problem, it can be redownloaded.
*
garbageCollect
only deletes entire files.
* There is no operation that modifies a file in-place. This means we do not
have to keep any rollback journals to make sure we are safe in case of
unexpected shutdowns.
We only throw
VolatileDBError
. File-system errors are caught, wrapped in a
VolatileDBError
, and rethrown. We make sure that all calls to
HasFS
functions are properly wrapped. This wrapping is automatically done when
inside the scope of
modifyOpenState
and
withOpenState
. Otherwise, we use
wrapFsError
.
Concurrency
A single folder should only be used by a single VolatileDB. Naturally, a VolatileDB can be accessed concurrently by multiple threads.
File-system layout:
The on-disk representation is as follows:
dbFolder/ blocks-0.dat blocks-1.dat ...
Files not fitting the naming scheme are ignored. The numbering of these files does not correlate to the blocks stored in them.
Each file stores a fixed number of blocks, specified by
volMaxBlocksPerFile
. When opening the VolatileDB, it will start appending
to the file with the highest number that is not yet full. If all are full or
none exist, a new file will be created.
There is an implicit ordering of block files, which is NOT alpharithmetic.
For example,
blocks-20.dat
<
blocks-100.dat
.
Recovery
The VolatileDB will always try to recover to a consistent state even if this means deleting all of its contents. In order to achieve this, it truncates the files containing blocks if some blocks fail to parse, are invalid, or are duplicated.
Synopsis
-
data
VolatileDbArgs
f m blk =
VolatileDbArgs
{
- volCheckIntegrity :: HKD f (blk -> Bool )
- volCodecConfig :: HKD f ( CodecConfig blk)
- volHasFS :: SomeHasFS m
- volMaxBlocksPerFile :: BlocksPerFile
- volTracer :: Tracer m ( TraceEvent blk)
- volValidationPolicy :: BlockValidationPolicy
- type VolatileDbSerialiseConstraints blk = ( EncodeDisk blk blk, DecodeDisk blk ( ByteString -> blk), DecodeDiskDep ( NestedCtxt Header ) blk, HasNestedContent Header blk, HasBinaryBlockInfo blk)
- defaultArgs :: Applicative m => SomeHasFS m -> VolatileDbArgs Defaults m blk
- openDB :: forall m blk ans. ( HasCallStack , IOLike m, HasHeader blk, GetPrevHash blk, VolatileDbSerialiseConstraints blk) => VolatileDbArgs Identity m blk -> ( forall st. WithTempRegistry st m ( VolatileDB m blk, st) -> ans) -> ans
- data BlockValidationPolicy
- data BlocksPerFile
-
data
ParseError
blk
- = BlockReadErr ReadIncrementalErr
- | BlockCorruptedErr ( HeaderHash blk)
- | DuplicatedBlock ( HeaderHash blk) FsPath FsPath
-
data
TraceEvent
blk
- = DBAlreadyClosed
- | BlockAlreadyHere ( HeaderHash blk)
- | Truncate ( ParseError blk) FsPath BlockOffset
- | InvalidFileNames [ FsPath ]
- extractBlockInfo :: ( GetPrevHash blk, HasBinaryBlockInfo blk) => blk -> BlockInfo blk
- mkBlocksPerFile :: Word32 -> BlocksPerFile
Opening the database
data VolatileDbArgs f m blk Source #
Constructors
VolatileDbArgs | |
Fields
|
type VolatileDbSerialiseConstraints blk = ( EncodeDisk blk blk, DecodeDisk blk ( ByteString -> blk), DecodeDiskDep ( NestedCtxt Header ) blk, HasNestedContent Header blk, HasBinaryBlockInfo blk) Source #
EncodeDisk
and
DecodeDisk
constraints needed for the VolatileDB.
defaultArgs :: Applicative m => SomeHasFS m -> VolatileDbArgs Defaults m blk Source #
Default arguments
openDB :: forall m blk ans. ( HasCallStack , IOLike m, HasHeader blk, GetPrevHash blk, VolatileDbSerialiseConstraints blk) => VolatileDbArgs Identity m blk -> ( forall st. WithTempRegistry st m ( VolatileDB m blk, st) -> ans) -> ans Source #
Re-exported
data BlockValidationPolicy Source #
When block validation is enabled, the parser checks for each block a number of properties and stops parsing if it finds any invalid blocks.
Constructors
NoValidation | |
ValidateAll |
Instances
Eq BlockValidationPolicy Source # | |
Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types Methods (==) :: BlockValidationPolicy -> BlockValidationPolicy -> Bool Source # (/=) :: BlockValidationPolicy -> BlockValidationPolicy -> Bool Source # |
data BlocksPerFile Source #
The maximum number of blocks to store per file.
Instances
Show BlocksPerFile Source # | |
Generic BlocksPerFile Source # | |
Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types Methods from :: BlocksPerFile -> Rep BlocksPerFile x Source # to :: Rep BlocksPerFile x -> BlocksPerFile Source # |
|
type Rep BlocksPerFile Source # | |
Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types
type
Rep
BlocksPerFile
=
D1
('
MetaData
"BlocksPerFile" "Ouroboros.Consensus.Storage.VolatileDB.Impl.Types" "ouroboros-consensus-0.1.0.1-DT4Cvwf63DZKctsEvaJqCU" '
True
) (
C1
('
MetaCons
"BlocksPerFile" '
PrefixI
'
True
) (
S1
('
MetaSel
('
Just
"unBlocksPerFile") '
NoSourceUnpackedness
'
NoSourceStrictness
'
DecidedLazy
) (
Rec0
Word32
)))
|
data ParseError blk Source #
Note that we recover from the error, and thus never throw it as an
Exception
.
Defined here instead of in the
Parser
module because
TraceEvent
depends
on it.
Constructors
BlockReadErr ReadIncrementalErr |
A block could not be parsed. |
BlockCorruptedErr ( HeaderHash blk) |
A block was corrupted, e.g., checking its signature and/or hash failed. |
DuplicatedBlock ( HeaderHash blk) FsPath FsPath |
A block with the same hash occurred twice in the VolatileDB files. We include the file in which it occurred first and the file in which it occured the second time. The two files can be the same. |
Instances
StandardHash blk => Eq ( ParseError blk) Source # | |
Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types Methods (==) :: ParseError blk -> ParseError blk -> Bool Source # (/=) :: ParseError blk -> ParseError blk -> Bool Source # |
|
StandardHash blk => Show ( ParseError blk) Source # | |
data TraceEvent blk Source #
Constructors
DBAlreadyClosed | |
BlockAlreadyHere ( HeaderHash blk) | |
Truncate ( ParseError blk) FsPath BlockOffset | |
InvalidFileNames [ FsPath ] |
Instances
extractBlockInfo :: ( GetPrevHash blk, HasBinaryBlockInfo blk) => blk -> BlockInfo blk Source #
mkBlocksPerFile :: Word32 -> BlocksPerFile Source #
Create a
BlocksPerFile
.
PRECONDITION: the given number must be greater than 0, if not, this
function will throw an
error
.