Shard the message processor so a channel's messages stay in order (#5240) * Rename `message_processor::put` to `enqueue` `put` said nothing about what happens to the message. `enqueue` says it is queued for the processing threads, next to `process`, which handles a message on the calling thread. The two declarations name their parameters. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Hand the message processor's routing to a `message_dispatcher` The message processor both queued messages and routed them, with a visitor over every node component a message can go to, so it could not be built without a node and nothing could stand in for the routing. The routing is now `nano::message_dispatcher`, an interface with one method, and `nano::node_message_dispatcher` is the node's implementation holding the visitor. The processor takes a dispatcher, the stats and the logger at construction, and `process` counts and traces a message before handing it over. The node owns a dispatcher ahead of the processor and passes it in. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Benchmark the message processor under storm, steady and hot-peer loads `nano/benchmarks/message_processing.cpp` drives a `nano::message_processor` of its own with four producer threads handing keepalives over on 256 channels, a few per channel in a row as a peer sends them: a storm that fills every channel's queue before the processor starts, steady rates of 200k, 1M and 4M messages per second, and a peer sending half of the traffic, at one, two and four processing threads. The processor dispatches to the benchmark's own dispatcher, so no handler runs: a keepalive, the cheapest message to build, carries its channel and its position in its first peer, and the dispatcher records when it arrived. Every run reports the messages dispatched per second, the wait in the queue as p50, p99 and max, the share refused by a full channel queue, the share dispatched after a later message of their channel, and the cost of a hand-over to the delivering thread or the rate the producers reached. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Shard the message processor so a channel's messages stay in order The processor drained one fair queue with up to two threads, each taking a batch of 4096 messages round-robin over channels, so two threads held consecutive messages of one channel and raced. A protocol that pairs messages on a channel, like the vote relay's acks and the empty ack that terminates them, saw them arrive out of order, and the relay client keeps completed requests around to catch the late ones. The processor now holds one shard per configured thread, each with its own fair queue, mutex, condition variable and thread, and a channel's messages always go to the shard its address hashes to. One thread drains a shard in arrival order, so a channel's messages are processed one after another in the order they arrived. The batch size, the per-channel cap and the fairness between the channels of a shard stay as they were, and a shard wakes its one thread instead of all of them. Measured with the benchmark of the previous commit, the sharded processor keeps up with a steady load under which the pool's one lock, shared between the io threads and its workers, stalled the hand-overs more with every thread added; a peer sending half of the traffic loses a small fraction of its messages instead of a quarter; and a storm of full queues drains somewhat slower, since the shards the channels hash into are not equally full. `message_processor.channel_order` fills a stopped processor with a thousand numbered keepalives on each of 64 channels, starts four threads and checks that no message was dispatched after a later one of its channel; on the pool thousands are. `message_processor.channel_overfill` checks that a channel's queue takes exactly `max_queue` messages, drops and counts the next one, and leaves another channel of the same thread unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Run two to six message processing threads by default The processor ran at most two threads, one per four cores, since every thread added contended on its one lock and made it slower under load. Shards do not contend, so the default is now one thread per two cores, two at least and six at most. Two shards keep one peer's slow handler, a telemetry signature check or a bootstrap reply, from stalling every other peer; beyond six the shards thin out over a few hundred peers and the fullest one holds a storm's drain back for no gain at steady load. The config description of `threads` goes back to its one sentence; which thread serves which peer is the processor's business. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Extract the address hash into `nano::pointer_hash` and test it The processor picked a channel's shard by mixing the channel's address with the splitmix64 finalizer inline, with nothing pinning the function or the property it was there for. `nano/lib/hashing.hpp` now has the mix as `nano::mix64` and the address hash as `nano::pointer_hash`, both reachable through the utility header, and the processor calls it. The tests pin the mix by reference values, show it is a bijection over the small values and the single bits, and check that 256 addresses laid out 16, 64 or 4096 bytes apart spread over two to six buckets within half and one and a half times their share, where the raw address never reaches every bucket of an even count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Tidy the node's message dispatcher The visitor behind `node_message_dispatcher` met a `frontier_req` with `debug_assert (false)`, and the TCP server forwarded that type while the deserializer still builds it, so one message from any peer aborted a debug build and vanished without a stat in a release one. Three more handlers asserted on types the server never forwards, the handshake handler was dead code that would have counted its message twice, and every message copied the channel's shared_ptr into the visitor. The server no longer forwards `frontier_req`, which the ignored-message stat now counts like the other legacy bootstrap types. The visitor keeps a handler for every type the node routes and sends everything else, telemetry requests included, through `default_handler`, which counts a drop by message type; a request a non-voting node cannot answer is counted the same way. It holds the channel by reference, the dispatcher's argument outliving it, says in a line why each drop happens, tests the self-reported address with `is_unspecified`, and is named for what it does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
__________
Type: commit
Hash: 306b97c1cff7e0050cdbb1536f6ab4afa376ed19
Created: 20 day(s) ago
__________
gr0v1ty/nano-node:306b97c1cff7e0050cdbb1536f6ab4afa376ed19
| Test Case | Status | Duration | Median Deviation | Log | Graph |
|---|---|---|---|---|---|
| 5n4pr_conf_10k_bintree | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_conf_10k_change | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_conf_change_dependant | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_conf_change_independant | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_conf_send_dependant | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_conf_send_independant | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_rocks_10k_bintree | FAIL | N/A | N/A | GitHub job logs | - |
| 5n4pr_rocks_10k_change | FAIL | N/A | N/A | GitHub job logs | - |