| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
Ouroboros.Consensus.Storage.ImmutableDB.Impl
Description
Immutable on-disk database of binary blobs
Internal format
The API of the ImmutableDB uses
SlotNo
to indicate a location in the
chain/immutable database. To distinguish EBBs from regular blocks, the hash
is used (together they form a
RealPoint
). The contents of the database are
not stored in one big file that is appended to in eternity, but a separate
file is created for each
ChunkNo
.
Within each
ChunkNo
, the entries are numbered by
RelativeSlot
s. Each
SlotNo
can be converted to a combination of an
ChunkNo
and a
RelativeSlot
(=
ChunkSlot
) and vice versa. This conversion depends on the
size of the chunks:
ChunkSize
. This size may not be the same for each
chunk. When opening the database, the user must give a
ChunkInfo
that will
be used to find out the size of each chunk.
For example:
Chunks: <──────── 0 ────────> <────── 1 ──────>
chunk size: 4 3
┌───┬───┬───┬───┬───┐ ┌───┬───┬───┬───┐
│ │ │ │ │ │ │ │ │ │ │
└───┴───┴───┴───┴───┘ └───┴───┴───┴───┘
'RelativeSlot': 0 1 2 3 4 0 1 2 3
'SlotNo': EBB 0 1 2 3 EBB 4 5 6
Not all chunks can contain EBBs; see
ChunkInfo
for details.
Errors
Whenever an
UnexpectedFailure
is thrown during an operation, e.g.,
appendBlock
, the database will be automatically closed because we can not
guarantee a consistent state in the face of file system errors.
Opening the database
The database can be closed and opened again. In case the database was closed because of an unexpected error. When the database is opened again, invalid data will be truncated from the database until a valid prefix is recovered.
Concurrency
The same database should not be opened multiple times concurrently. This is ensured by the file lock of the ChainDB.
The database can have multiple readers, but should only have one writer.
Layout on disk
The database is structured on disk as follows:
/ 00000.chunk 00000.primary 00000.secondary .. 00008.chunk 00008.primary 00008.secondary
For each chunk, there are three files on disk:
- A "chunk file" that stores the actual blocks. But nothing more, so nothing is stored for empty slots.
- A "secondary index file" that stores information about each block: its hash, the slot number or epoch number in case of an EBB, a checksum of the block, the offset of the block in the chunk file, and more. This index is sparse to save space.
- A "primary index file" that maps slots to offsets in the secondary index file.
Synopsis
-
data
ImmutableDbArgs
f m blk =
ImmutableDbArgs
{
- immCacheConfig :: CacheConfig
- immCheckIntegrity :: HKD f (blk -> Bool )
- immChunkInfo :: HKD f ChunkInfo
- immCodecConfig :: HKD f ( CodecConfig blk)
- immHasFS :: SomeHasFS m
- immRegistry :: HKD f ( ResourceRegistry m)
- immTracer :: Tracer m ( TraceEvent blk)
- immValidationPolicy :: ValidationPolicy
- type ImmutableDbSerialiseConstraints blk = ( EncodeDisk blk blk, DecodeDisk blk ( ByteString -> blk), DecodeDiskDep ( NestedCtxt Header ) blk, ReconstructNestedCtxt Header blk, HasBinaryBlockInfo blk)
- defaultArgs :: Applicative m => SomeHasFS m -> ImmutableDbArgs Defaults m blk
- openDB :: forall m blk ans. ( IOLike m, GetPrevHash blk, ConvertRawHash blk, ImmutableDbSerialiseConstraints blk, HasCallStack ) => ImmutableDbArgs Identity m blk -> ( forall st. WithTempRegistry st m ( ImmutableDB m blk, st) -> ans) -> ans
-
data
ChunkFileError
blk
- = ChunkErrRead ReadIncrementalErr
- | ChunkErrHashMismatch ( HeaderHash blk) ( ChainHash blk)
- | ChunkErrCorrupt ( Point blk)
- data CacheConfig = CacheConfig { }
-
data
TraceChunkValidation
blk validateTo
- = StartedValidatingChunk ChunkNo validateTo
- | ValidatedChunk ChunkNo validateTo
- | MissingChunkFile ChunkNo
- | InvalidChunkFile ChunkNo ( ChunkFileError blk)
- | MissingPrimaryIndex ChunkNo
- | MissingSecondaryIndex ChunkNo
- | InvalidPrimaryIndex ChunkNo
- | InvalidSecondaryIndex ChunkNo
- | RewritePrimaryIndex ChunkNo
- | RewriteSecondaryIndex ChunkNo
-
data
TraceEvent
blk
- = NoValidLastLocation
- | ValidatedLastLocation ChunkNo ( Tip blk)
- | ChunkValidationEvent ( TraceChunkValidation blk ChunkNo )
- | ChunkFileDoesntFit ( ChainHash blk) ( ChainHash blk)
- | Migrating Text
- | DeletingAfter ( WithOrigin ( Tip blk))
- | DBAlreadyClosed
- | DBClosed
- | TraceCacheEvent ! TraceCacheEvent
- data ValidationPolicy
-
data
Internal
m blk =
Internal
{
- deleteAfter_ :: HasCallStack => WithOrigin ( Tip blk) -> m ()
- deleteAfter :: HasCallStack => Internal m blk -> WithOrigin ( Tip blk) -> m ()
- openDBInternal :: forall m blk ans. ( IOLike m, GetPrevHash blk, ConvertRawHash blk, ImmutableDbSerialiseConstraints blk, HasCallStack ) => ImmutableDbArgs Identity m blk -> ( forall h. WithTempRegistry ( OpenState m blk h) m (( ImmutableDB m blk, Internal m blk), OpenState m blk h) -> ans) -> ans
Opening the databse
data ImmutableDbArgs f m blk Source #
Constructors
| ImmutableDbArgs | |
|
Fields
|
|
type ImmutableDbSerialiseConstraints blk = ( EncodeDisk blk blk, DecodeDisk blk ( ByteString -> blk), DecodeDiskDep ( NestedCtxt Header ) blk, ReconstructNestedCtxt Header blk, HasBinaryBlockInfo blk) Source #
EncodeDisk
and
DecodeDisk
constraints needed for the ImmutableDB.
defaultArgs :: Applicative m => SomeHasFS m -> ImmutableDbArgs Defaults m blk Source #
Default arguments
openDB :: forall m blk ans. ( IOLike m, GetPrevHash blk, ConvertRawHash blk, ImmutableDbSerialiseConstraints blk, HasCallStack ) => ImmutableDbArgs Identity m blk -> ( forall st. WithTempRegistry st m ( ImmutableDB m blk, st) -> ans) -> ans Source #
Re-exported
data ChunkFileError blk Source #
Defined here instead of in the
Parser
module because
TraceEvent
depends on it.
Constructors
| ChunkErrRead ReadIncrementalErr |
A block could not be decoded |
| ChunkErrHashMismatch |
The previous hash of a block did not match the hash of the previous block. |
|
Fields
|
|
| ChunkErrCorrupt ( Point blk) |
The integrity verification of the block with the given point returned
|
Instances
| StandardHash blk => Eq ( ChunkFileError blk) Source # | |
|
Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types Methods (==) :: ChunkFileError blk -> ChunkFileError blk -> Bool Source # (/=) :: ChunkFileError blk -> ChunkFileError blk -> Bool Source # |
|
| StandardHash blk => Show ( ChunkFileError blk) Source # | |
data CacheConfig Source #
Constructors
| CacheConfig | |
|
Fields
|
|
Instances
| Eq CacheConfig Source # | |
|
Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Index.Cache Methods (==) :: CacheConfig -> CacheConfig -> Bool Source # (/=) :: CacheConfig -> CacheConfig -> Bool Source # |
|
| Show CacheConfig Source # | |
data TraceChunkValidation blk validateTo Source #
Constructors
Instances
data TraceEvent blk Source #
Constructors
| NoValidLastLocation | |
| ValidatedLastLocation ChunkNo ( Tip blk) | |
| ChunkValidationEvent ( TraceChunkValidation blk ChunkNo ) | |
| ChunkFileDoesntFit ( ChainHash blk) ( ChainHash blk) |
The hash of the last block in the previous epoch doesn't match the previous hash of the first block in the current epoch |
| Migrating Text |
Performing a migration of the on-disk files |
| DeletingAfter ( WithOrigin ( Tip blk)) | |
| DBAlreadyClosed | |
| DBClosed | |
| TraceCacheEvent ! TraceCacheEvent |
Instances
data ValidationPolicy Source #
The validation policy used when opening an
ImmutableDB
.
The validation policy is used by
openDB
: the initial opening of
the database, either an empty database or a database that was previously
closed.
The recovery policy dictates which on-disk files should be validated.
Constructors
| ValidateMostRecentChunk |
The chunk and index files of the most recent chunk stored on disk will be validated. Prior chunk and index files are ignored, even their presence will not be checked.
A
Because not all files are validated, subsequent operations on the database after opening may result in unexpected errors. |
| ValidateAllChunks |
The chunk and index files of all chunks starting from the first one up to the last chunk stored on disk will be validated.
A
|
Instances
| Eq ValidationPolicy Source # | |
|
Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types Methods (==) :: ValidationPolicy -> ValidationPolicy -> Bool Source # (/=) :: ValidationPolicy -> ValidationPolicy -> Bool Source # |
|
| Show ValidationPolicy Source # | |
| Generic ValidationPolicy Source # | |
|
Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types Methods from :: ValidationPolicy -> Rep ValidationPolicy x Source # to :: Rep ValidationPolicy x -> ValidationPolicy Source # |
|
| type Rep ValidationPolicy Source # | |
|
Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types
type
Rep
ValidationPolicy
=
D1
('
MetaData
"ValidationPolicy" "Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types" "ouroboros-consensus-0.1.0.1-DT4Cvwf63DZKctsEvaJqCU" '
False
) (
C1
('
MetaCons
"ValidateMostRecentChunk" '
PrefixI
'
False
) (
U1
::
Type
->
Type
)
:+:
C1
('
MetaCons
"ValidateAllChunks" '
PrefixI
'
False
) (
U1
::
Type
->
Type
))
|
|
Internals for testing purposes
Constructors
| Internal | |
|
Fields
|
|
deleteAfter :: HasCallStack => Internal m blk -> WithOrigin ( Tip blk) -> m () Source #
Wrapper around
deleteAfter_
to ensure
HasCallStack
constraint
See documentation of
deleteAfter_
.
openDBInternal :: forall m blk ans. ( IOLike m, GetPrevHash blk, ConvertRawHash blk, ImmutableDbSerialiseConstraints blk, HasCallStack ) => ImmutableDbArgs Identity m blk -> ( forall h. WithTempRegistry ( OpenState m blk h) m (( ImmutableDB m blk, Internal m blk), OpenState m blk h) -> ans) -> ans Source #
For testing purposes: exposes internals via
Internal