Test Details Overview

Shard the message processor so a channel's messages stay in order (#5240)

* Rename `message_processor::put` to `enqueue`

`put` said nothing about what happens to the message. `enqueue` says it
is queued for the processing threads, next to `process`, which handles
a message on the calling thread. The two declarations name their
parameters.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Hand the message processor's routing to a `message_dispatcher`

The message processor both queued messages and routed them, with a
visitor over every node component a message can go to, so it could not
be built without a node and nothing could stand in for the routing.

The routing is now `nano::message_dispatcher`, an interface with one
method, and `nano::node_message_dispatcher` is the node's
implementation holding the visitor. The processor takes a dispatcher,
the stats and the logger at construction, and `process` counts and
traces a message before handing it over. The node owns a dispatcher
ahead of the processor and passes it in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Benchmark the message processor under storm, steady and hot-peer loads

`nano/benchmarks/message_processing.cpp` drives a `nano::message_processor`
of its own with four producer threads handing keepalives over on 256
channels, a few per channel in a row as a peer sends them: a storm that
fills every channel's queue before the processor starts, steady rates
of 200k, 1M and 4M messages per second, and a peer sending half of the
traffic, at one, two and four processing threads.

The processor dispatches to the benchmark's own dispatcher, so no
handler runs: a keepalive, the cheapest message to build, carries its
channel and its position in its first peer, and the dispatcher records
when it arrived. Every run reports the messages dispatched per second,
the wait in the queue as p50, p99 and max, the share refused by a full
channel queue, the share dispatched after a later message of their
channel, and the cost of a hand-over to the delivering thread or the
rate the producers reached.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Shard the message processor so a channel's messages stay in order

The processor drained one fair queue with up to two threads, each taking
a batch of 4096 messages round-robin over channels, so two threads held
consecutive messages of one channel and raced. A protocol that pairs
messages on a channel, like the vote relay's acks and the empty ack that
terminates them, saw them arrive out of order, and the relay client
keeps completed requests around to catch the late ones.

The processor now holds one shard per configured thread, each with its
own fair queue, mutex, condition variable and thread, and a channel's
messages always go to the shard its address hashes to. One thread drains
a shard in arrival order, so a channel's messages are processed one
after another in the order they arrived. The batch size, the per-channel
cap and the fairness between the channels of a shard stay as they were,
and a shard wakes its one thread instead of all of them.

Measured with the benchmark of the previous commit, the sharded
processor keeps up with a steady load under which the pool's one lock,
shared between the io threads and its workers, stalled the hand-overs
more with every thread added; a peer sending half of the traffic loses
a small fraction of its messages instead of a quarter; and a storm of
full queues drains somewhat slower, since the shards the channels hash
into are not equally full.

`message_processor.channel_order` fills a stopped processor with a
thousand numbered keepalives on each of 64 channels, starts four threads
and checks that no message was dispatched after a later one of its
channel; on the pool thousands are. `message_processor.channel_overfill`
checks that a channel's queue takes exactly `max_queue` messages, drops
and counts the next one, and leaves another channel of the same thread
unaffected.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Run two to six message processing threads by default

The processor ran at most two threads, one per four cores, since every
thread added contended on its one lock and made it slower under load.
Shards do not contend, so the default is now one thread per two cores,
two at least and six at most. Two shards keep one peer's slow handler,
a telemetry signature check or a bootstrap reply, from stalling every
other peer; beyond six the shards thin out over a few hundred peers and
the fullest one holds a storm's drain back for no gain at steady load.

The config description of `threads` goes back to its one sentence;
which thread serves which peer is the processor's business.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Extract the address hash into `nano::pointer_hash` and test it

The processor picked a channel's shard by mixing the channel's address
with the splitmix64 finalizer inline, with nothing pinning the function
or the property it was there for. `nano/lib/hashing.hpp` now has the
mix as `nano::mix64` and the address hash as `nano::pointer_hash`, both
reachable through the utility header, and the processor calls it.

The tests pin the mix by reference values, show it is a bijection over
the small values and the single bits, and check that 256 addresses laid
out 16, 64 or 4096 bytes apart spread over two to six buckets within
half and one and a half times their share, where the raw address never
reaches every bucket of an even count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Tidy the node's message dispatcher

The visitor behind `node_message_dispatcher` met a `frontier_req` with
`debug_assert (false)`, and the TCP server forwarded that type while
the deserializer still builds it, so one message from any peer aborted
a debug build and vanished without a stat in a release one. Three more
handlers asserted on types the server never forwards, the handshake
handler was dead code that would have counted its message twice, and
every message copied the channel's shared_ptr into the visitor.

The server no longer forwards `frontier_req`, which the ignored-message
stat now counts like the other legacy bootstrap types. The visitor keeps
a handler for every type the node routes and sends everything else,
telemetry requests included, through `default_handler`, which counts a
drop by message type; a request a non-voting node cannot answer is
counted the same way. It holds the channel by reference, the
dispatcher's argument outliving it, says in a line why each drop
happens, tests the self-reported address with `is_unspecified`, and is
named for what it does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

__________

Type: commit

Hash: 306b97c1cff7e0050cdbb1536f6ab4afa376ed19

Created: 20 day(s) ago

__________

gr0v1ty/nano-node:306b97c1cff7e0050cdbb1536f6ab4afa376ed19

Test Results

Test Case Status Duration Median Deviation Log Graph
5n4pr_conf_10k_bintree FAIL N/A N/A GitHub job logs -
5n4pr_conf_10k_change FAIL N/A N/A GitHub job logs -
5n4pr_conf_change_dependant FAIL N/A N/A GitHub job logs -
5n4pr_conf_change_independant FAIL N/A N/A GitHub job logs -
5n4pr_conf_send_dependant FAIL N/A N/A GitHub job logs -
5n4pr_conf_send_independant FAIL N/A N/A GitHub job logs -
5n4pr_rocks_10k_bintree FAIL N/A N/A GitHub job logs -
5n4pr_rocks_10k_change FAIL N/A N/A GitHub job logs -