WIP: First part of the DHT blog post #390
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
||
| For that reason, mature systems such as the mainline DHT restrict value size to 1000 bytes, and we are going to limit value size to 1024 bytes or 1KiB. | ||
|
|
||
| You could write a DHT to store arbitrary values, but in almost all cases the value should have some relationship with the key. E.g. for mainline, the value in most cases is a set of socket addresses where you can download the data for the SHA1 hash of the key. So in principle you could validate the key by checking if you can actually download the data from the socket addresses contained in the data. In some mainline extensions, like bep_0044, the key is the SHA1 hash of an ED25519 public key, and the value contains the actual public key, a signature computed from the corresponding private key, and some user data. Again, it is possible to validate the value based on the key - if the SHA1 hash of the public key contained in the value does not match the lookup key, the value is invalid for the key. |
There was a problem hiding this comment.
You sort of seem to suggest that enforcing only storing values that can be validated is a good idea. But you don't actually go as far as suggesting that. What is the aim of this paragraph? It may need to be a bit stronger worded in it's suggestion/recommendation?
There was a problem hiding this comment.
I think it is a good idea, and mainline kinda sorta does it. All the extensions have validatable values (bep_0044 and the one for immutable data).
For the main use case, you can not store arbitrary socket addrs for a SHA1 hash, but only the socket addrs of your own node as seen from the callee. The only free parameter is the port number.
I think I want to do this as well, but for blake3 providers you either let the DHT node do BLAKE3 probes (costly), or you store a signed record or something. Not quite worked out yet.
There was a problem hiding this comment.
So, possibility 1: values for content discovery are just node ids. Tracker can lazily do BLAKE3 probes to make sure the data is there, and then purge values if not.
Possibility 2: values are a signed promise by the announcer that the data is there. Tracker can check the signature, but that does not really tell us if the promise is upheld. If you want to know if the data is there, you have to check.
Co-authored-by: Floris Bruynooghe <[email protected]>
Co-authored-by: Floris Bruynooghe <[email protected]>
Co-authored-by: Floris Bruynooghe <[email protected]>
Co-authored-by: Floris Bruynooghe <[email protected]>
|
|
||
| And that's it. That is the entire rpc protocol. Many DHT implementations also add a `Ping` call, but since querying the routing table is so cheap, if you want to know if a node is alive you might as well ask it for the closest nodes to some random key and get some extra information for free. | ||
|
|
||
| ## RPC client |
There was a problem hiding this comment.
You could skip this entire section for a blog post, it would also shorten it. This is more tutorial-material. To me the protocol is the important bit, this is just boilerplate.
There was a problem hiding this comment.
Maybe publish the crate (under a better name) and just point to the docs / impl?
There was a problem hiding this comment.
Yeah, that's also a good option if you can point to the code in a repo somewhere. Though maybe @b5 is interested in a tutorial as well to turn into another video?
|
|
||
| So let's define the routing table. First of all we need some simple integer arithmetic like xor and leading_zeros for 256 bit numbers. There are various crates that provide this, but since we don't need anything fancy like multiplication or division, we just quickly implemented it inline. | ||
|
|
||
| The routing table itself is just a 2d array of node ids. Each row (k-bucket) has a small fixed upper size, so we are going to use the [ArrayVec] crate to prevent allocations. For each node id we just keep a tiny bit of extra information - a timestamp when we have last seen any evidence that the node actually exists and responds, to decide which nodes to check for liveness. |
There was a problem hiding this comment.
I haven't really been paying much attention to this till now, so there might be other occurances. But perhaps we should stick to "node ID" in text when referring to the iroh NodeId. It's mostly a stylistic question though.
|
Okay, this is great. Note, I did not read for grammar or correctness, just read it while thinking about structure. My suggestion here would be to immediately lean into the fact that this is going to be a series and already split this into 3 separate posts. The first post would be about what a DHT is and what properties we want in a DHT. We can link to code as a preview for what is to come, and add a conclusion that describes the next few blog posts. The second post would be the implementation post, ending with something to the effect of "so now we have an implementation, how do we know it works? Next blog post we will describe how to test our DHT implementation and tips for testing iroh networks on the order of 100k locally by using irpc in our protocols." The third post would be illustrating the way you tested the DHT using irpc. The conclusion can describe some topics for future posts on DHTs. Each post should have a table of contents and the top with links to each blog post in the series as they get published, as well as a link to the next post in the series at the bottom of each page. @rklaehn what are your thoughts on splitting it this way? Your post is already basically structured in this fashion, it would just be splitting it explicitly. |
Yeah, makes sense. I wonder if the first part is interesting enough, but we can follow it up qucikly with at least the second part. |
|
@rklaehn I do think it's interesting enough, it's just not to you because you understand it all already. Most people in our space know what a DHT is, but they haven't spent the time actually thinking practically about they should want in a DHT, which is what your blog post especially highlights. |
Co-authored-by: Floris Bruynooghe <[email protected]>
Co-authored-by: Floris Bruynooghe <[email protected]>
|
@ramfox I split it now. Hope it is roughly where you intended the split. We can review the first part so that it is ready to go and then split off the rest into another PR. For at least the second part I would love to publish the code, so I can refer to the code on docs.rs, but that requires the conn pool to be published, which requires a blobs release, which requires an iroh release. But for the first part, which is really just an appetizer, we could do without I think. |
No description provided.