blockchain: skip prevout fetch from disk on cache miss until checkpoint#1
blockchain: skip prevout fetch from disk on cache miss until checkpoint#1allocz wants to merge 2 commits into
Conversation
|
This is a first draft just to validate the concept, the added metrics package will be removed later in a rebase. |
|
|
||
| // Return early and don't attempt access the database if we don't have any | ||
| // missing outpoints. | ||
| // Return early and don't attempt access the database if we don't have |
There was a problem hiding this comment.
You've got a trailing whitespace here. Worth setting up something in your editor to automatically remove it
|
Bit of background: For btcd, we'd ideally want to follow want bitcoin core is doing with the assumevalid. Right now we're using checkpoints because when things are not going wrong, it's essentially doing the same checks. So right now for btcd, we're also checking the inputs/outputs and that'd make it easier to change over to assumevalid. This change is going in the wrong direction. FWIW, I think for database optimizations, implementing swiftsync is a better way going forward. |
|
Thanks for the review. Do you see any new attack vectors enabled by this approach? |
allocz
force-pushed
the
checkpoint_no_prevout_disk_lookup
branch
5 times, most recently
from
fd262d0 to
3fc47a5
Compare
misses, and use hardcoded utxoset hash to assert correct utxoset state. Doing this removes de previous main bottleneck when executing IBD in machines constrained in RAM. The assumevalid checkpoints gives us the proof that the old blocks are consensus compliant and the utxoset hash confirms that there's no software or hardware failures causing a bad utxoset index. The trust assumption is the code and the full validation can be performed by executing btcd with `--nocheckpoints`
allocz
force-pushed
the
checkpoint_no_prevout_disk_lookup
branch
from
3fc47a5 to
6cccc58
Compare
2 participants
Motivation: In devices where the utxoset does not fit in ram, there's a lot of cache misses when performing IBD, this adds latency. This PR skips the fetch of prevouts until the last checkpoint block, significantly reducing IBD elapsed time by avoiding the disk lookups during IBD.
Trust assumptions: is assumed that all blocks until the checkpoint are valid, with this premise being truth, there's no risk of double spends or other consensus violations.
The full verification can still be enabled by passing the
--nocheckpointsflag.As we can see,
skip_prevoutpatch improved block processing speed in 68x,with 1.22% of total disk reads and 17.18% of total disk writes when compared to
btcd_master.We also can see that
btcd_mastertakes 23% more time to process each blockwhen compared with
bitcoin_core_v31, andbtcd_skip_prevouttakes 1.78% ofthe time that
bitcoin_core_v31takes per block.In summary, the benchmark started with btcd being 23% slower than Bitcoin Core,
and now, btcd is 5495% faster than Bitcoin Core.
The results suggest that in resource constrained computers, skipping the prevout
check while processing assumed valid blocks does improve total IBD speed.
UTXOSet hash was added to provide sanity check that in the end we have the expected utxoset.