Skip to content

Use hash writer for aristo computeKey #3572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

Use hash writer for aristo computeKey #3572

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7b4e814
incomplete
chirag-parmar Aug 12, 2025
b383533
commented code
chirag-parmar Aug 12, 2025
2253512
use similar structure to ordered trie
chirag-parmar Aug 20, 2025
60f5dad
comment
chirag-parmar Aug 20, 2025
c92907b
reintroduce the line
chirag-parmar Aug 20, 2025
1e9f6ed
Remove hack.
bhartnett Sep 9, 2025
13bc9b7
Merge branch 'master' into hash-writer-aristo
bhartnett Sep 9, 2025
3dc24fa
Remove unneeded usages of untyped. Update encodeExt to a func to prev…
bhartnett Sep 9, 2025
157f1a4
review
chirag-parmar Sep 9, 2025
126c0a8
Remove usages of auto in encodeLeaf.
bhartnett Sep 12, 2025
184a373
fix imports
chirag-parmar Sep 12, 2025
2246c31
Make encodeLeaf a func and store hashKeys in encodeBranch.
bhartnett Sep 13, 2025
0419fde
Update execution_chain/db/aristo/aristo_compute.nim
bhartnett Sep 14, 2025
9f71e5e
Merge branch 'master' into hash-writer-aristo
bhartnett Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 111 additions & 45 deletions execution_chain/db/aristo/aristo_compute.nim
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,19 @@
import
std/strformat,
chronicles,
eth/rlp,
eth/common/[accounts_rlp, base_rlp, hashes_rlp],
results,
"."/[aristo_desc, aristo_get, aristo_layers],
./aristo_desc/desc_backend

type WriteBatch = tuple[writer: PutHdlRef, count: int, depth: int, prefix: uint64]
type
WriteBatch = tuple[writer: PutHdlRef, count: int, depth: int, prefix: uint64]

AristoTrieWriter = object
tracker: DynamicRlpLengthTracker
twoPassWriter: RlpTwoPassWriter
hashWriter: RlpHashWriter

# Keep write batch size _around_ 1mb, give or take some overhead - this is a
# tradeoff between efficiency and memory usage with diminishing returns the
Expand Down Expand Up @@ -89,24 +96,86 @@ proc putKeyAtLevel(

ok()

template encodeLeaf(w: var RlpWriter, pfx: NibblesBuf, leafData: untyped): HashKey =
func init(T: type AristoTrieWriter): T =
var trk = DynamicRlpLengthTracker()
trk.initLengthTracker()

AristoTrieWriter(
tracker: trk,
twoPassWriter: initTwoPassWriter(trk),
hashWriter: initHashWriter(trk)
)

func clear(self: var AristoTrieWriter) =
self.tracker.clear()
self.twoPassWriter.clear()
self.hashWriter.clear()

template appendLeaf(w: var RlpWriter, pfx: NibblesBuf, leafData: Account | UInt256) =
w.startList(2)
w.append(pfx.toHexPrefix(isLeaf = true).data())
w.wrapEncoding(1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't this call to wrapEnconding depend on leafData being of a particular type? if yes, why use auto?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about wrapEncoding but the reason I changed this line was that we are passing a type account or slot and don't need to pass in a block of statements. Perhaps it would be better to make it even more strict and use Account | UInt256 as the type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok - it feels odd that it should be needed cc @chirag-parmar - ie append should be able to figure this out ideally - but it seems to work, so ..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the late reply. the call to wrapEncoding doesn't depend on the leafData being of a particular type. Only leaf nodes have nested serializations i.e. Serializing an object/list/tuple that already has a serialized rlp value in it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

append should be able to figure this out ideally

I didn't understand what it must be able to figure out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what I would have expected maybe is a call like appendEncoded(leafData) which does the extra round of encoding of the passed-in item - wrapEncoding looks like an internal low-level operation, specially with that number being passed in, because presumably it must be followed up with actual data - that gives the caller an unnecessary opportunity to use it the wrong way

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense actually. status-im/nim-eth#813

w.append(leafData)
w.finish().digestTo(HashKey)

template encodeBranch(w: var RlpWriter, vtx: VertexRef, subKeyForN: untyped): HashKey =
func encodeLeaf(w: var AristoTrieWriter, pfx: NibblesBuf, leafData: Account | UInt256): HashKey =
w.clear()
w.tracker.appendLeaf(pfx, leafData)

if w.tracker.totalLength < 32:
w.twoPassWriter.reInit(w.tracker)
w.twoPassWriter.appendLeaf(pfx, leafData)
let buf = HashKey.fromBytes(w.twoPassWriter.finish)
buf.value
else:
w.hashWriter.reInit(w.tracker)
w.hashWriter.appendLeaf(pfx, leafData)
let buf = w.hashWriter.finish()
buf.to(HashKey)

template appendBranch(w: var RlpWriter, vtx: VertexRef, hashKeys: array[16, HashKey]) =
w.startList(17)
for (n {.inject.}, subvid {.inject.}) in vtx.allPairs():
w.append(subKeyForN)
for key in hashKeys:
w.append(key)
w.append EmptyBlob
w.finish().digestTo(HashKey)

template encodeExt(w: var RlpWriter, pfx: NibblesBuf, branchKey: HashKey): HashKey =
template encodeBranch(w: var AristoTrieWriter, rvtx: VertexRef, subKeyForN: untyped): HashKey =
var hashKeys {.noinit.}: array[16, HashKey]
for (n {.inject.}, subvid {.inject.}) in vtx.allPairs():
hashKeys[n] = subKeyForN

w.clear()
w.tracker.appendBranch(vtx, hashKeys)

if w.tracker.totalLength < 32:
w.twoPassWriter.reInit(w.tracker)
w.twoPassWriter.appendBranch(vtx, hashKeys)
let buf = HashKey.fromBytes(w.twoPassWriter.finish)
buf.value
else:
w.hashWriter.reInit(w.tracker)
w.hashWriter.appendBranch(vtx, hashKeys)
let buf = w.hashWriter.finish()
buf.to(HashKey)

template appendExt(w: var RlpWriter, pfx: NibblesBuf, branchKey: HashKey) =
w.startList(2)
w.append(pfx.toHexPrefix(isLeaf = false).data())
w.append(branchKey)
w.finish().digestTo(HashKey)

func encodeExt(w: var AristoTrieWriter, pfx: NibblesBuf, branchKey: HashKey): HashKey =
w.clear()
w.tracker.appendExt(pfx, branchKey)

if w.tracker.totalLength < 32:
w.twoPassWriter.reInit(w.tracker)
w.twoPassWriter.appendExt(pfx, branchKey)
let buf = HashKey.fromBytes(w.twoPassWriter.finish)
buf.value
else:
w.hashWriter.reInit(w.tracker)
w.hashWriter.appendExt(pfx, branchKey)
let buf = w.hashWriter.finish()
buf.to(HashKey)

proc getKey(
db: AristoTxRef, rvid: RootedVertexID, skipLayers: static bool
Expand Down Expand Up @@ -144,49 +213,47 @@ proc computeKeyImpl(
# empty state

# Top-most level of all the verticies this hash computation depends on
var level = level

# TODO this is the same code as when serializing NodeRef, without the NodeRef
var writer = initRlpWriter()
var
level = level
writer = AristoTrieWriter.init()

let key =
case vtx.vType
of AccLeaf:
let vtx = AccLeafRef(vtx)
writer.encodeLeaf(vtx.pfx):
let
stoID = vtx.stoID
skey =
if stoID.isValid:
let
keyvtxl = ?db.getKey((stoID.vid, stoID.vid), skipLayers)
(skey, sl) =
if keyvtxl[0][0].isValid:
(keyvtxl[0][0], keyvtxl[1])
else:
?db.computeKeyImpl(
(stoID.vid, stoID.vid),
batch,
keyvtxl[0][1],
keyvtxl[1],
skipLayers = skipLayers,
)
level = max(level, sl)
skey
else:
VOID_HASH_KEY

rlp.encode Account(
let
vtx = AccLeafRef(vtx)
stoID = vtx.stoID
skey =
if stoID.isValid:
let
keyvtxl = ?db.getKey((stoID.vid, stoID.vid), skipLayers)
(skey, sl) =
if keyvtxl[0][0].isValid:
(keyvtxl[0][0], keyvtxl[1])
else:
?db.computeKeyImpl(
(stoID.vid, stoID.vid),
batch,
keyvtxl[0][1],
keyvtxl[1],
skipLayers = skipLayers,
)
level = max(level, sl)
skey
else:
VOID_HASH_KEY

writer.encodeLeaf(vtx.pfx,
Account(
nonce: vtx.account.nonce,
balance: vtx.account.balance,
storageRoot: skey.to(Hash32),
codeHash: vtx.account.codeHash,
)
)
of StoLeaf:
let vtx = StoLeafRef(vtx)
writer.encodeLeaf(vtx.pfx):
# TODO avoid memory allocation when encoding storage data
rlp.encode(vtx.stoData)
writer.encodeLeaf(vtx.pfx, vtx.stoData)
of Branches:
# For branches, we need to load the vertices before recursing into them
# to exploit their on-disk order
Expand Down Expand Up @@ -245,7 +312,7 @@ proc computeKeyImpl(
)
batch.leave(n)

template writeBranch(w: var RlpWriter, vtx: BranchRef): HashKey =
template writeBranch(w: var AristoTrieWriter, vtx: BranchRef): HashKey =
w.encodeBranch(vtx):
if subvid.isValid:
level = max(level, keyvtxs[n][1])
Expand All @@ -255,9 +322,8 @@ proc computeKeyImpl(

if vtx.vType == ExtBranch:
let vtx = ExtBranchRef(vtx)
writer.encodeExt(vtx.pfx):
var bwriter = initRlpWriter()
bwriter.writeBranch(vtx)
var bwriter = AristoTrieWriter.init()
writer.encodeExt(vtx.pfx, bwriter.writeBranch(vtx))
else:
writer.writeBranch(vtx)

Expand Down
Loading