Test Details Overview

Fix wallet lock ordering and raise the LMDB reader slot limit (#5216)

* Open wallet read transactions under `wallets::mutex`

`refresh_rep_keys_cache`, `exists` and `exists_any` opened their wallet read transaction before
waiting for the mutex, so every caller queued behind it held one of LMDB's reader slots, 126 by
default, and a refresh that lost the race for the mutex could overwrite a newer cache with its
older snapshot.

They now take the mutex first and open the transaction under it: queued callers hold nothing, and
refreshes install their snapshots in mutex order. `exists_impl` expects the caller to hold the
mutex.

`wallet.password_race_corrupt_seed` kept its thread count below the slot limit for this reason, its
comment now gives the reason that remains.

* Take the wallet write transaction before `wallets::mutex`

`reload`, `create`, `create_from_json` and `destroy` waited for the wallet write transaction while
holding the mutex, so a writer busy with a rekey's key derivation or a commit's fsync stalled every
wallet lookup behind it: `open`, `all_wallets`, `wallet_ids`, `exists` and both representative
refreshes. On the dev network `reload` runs every second, which is what piled the test threads up.

The transaction now comes first in all four and in the constructor, so the only thing a mutex
holder waits for is the mutex. Where a wallet table is created the transaction is committed before
the mutex is released, because LMDB keeps a new table private to its transaction until the commit
and a lookup reaching the wallet earlier would fail on its table handle.

* Give LMDB environments 1024 reader slots

Every open read transaction holds a slot in the lock file's reader table and the node aborts when
the table is full. The environment flags comment claimed MDB_NOTLS avoids this: it only ties slots
to transactions instead of threads, the table still caps concurrent read transactions at LMDB's
default of 126, below the thread counts the node may run with.

Both the ledger and the wallet environment now ask for 1024 slots, 64 KB of lock file. LMDB grows
an existing lock file to the requested size when the node opens it exclusively, so upgraded nodes
get the larger table too.

* Register new representatives after their insert commits

`deterministic_insert_impl` registered a qualifying account with `check_rep` and the wallet's
representative set while its write transaction was still open. A reps scan whose read transaction
opened in that window rebuilt both sets from a snapshot without the account, and the key cache
refresh after the commit then left it out of the cache until the next scan, 15 minutes on the live
network. `insert_adhoc` already committed before registering.

The single inserts now register the account after the commit through one helper shared with
`insert_adhoc`. `change_seed` and `deterministic_restore` insert their accounts inside one
transaction and run a full representative refresh once it has committed, instead of only the key
cache refresh.

* Commit the wallet load transaction before `wallets::mutex` is released

The constructor declares its lock guard after its write transaction, so the guard is destroyed
first: the mutex was released and the transaction committed by its destructor afterwards, the
reverse of the order `create` and `reload` keep. Nothing else can reach the wallet map during
construction, so this only read as an exception to the rule.

The constructor now commits explicitly before the lock guard goes out of scope, like the other paths
that open wallet tables under the mutex.

* Fill the wallet reader table in the queued-callers test

`wallets.queued_callers_hold_no_reader_slot` queued 200 threads behind the mutex and relied on the
reader table being smaller than that. That held at LMDB's default of 126 slots and stopped holding
once the environments asked for 1024: with the old ordering restored, the waiters hold 200 slots out
of 1024 and the test passes anyway.

The test now derives its numbers from `max_readers`. It opens read transactions until fewer slots
are free than the threads of either call path, so a slot per waiter overflows the table while the
fixed code, which opens one transaction at a time under the mutex, stays under it. Reverting only
`exists` to the old order makes it abort with MDB_READERS_FULL.

* Set the wallets stopped flag under `wallets::mutex`

`stop` set `stopped` under `action_mutex` and then notified the reps and receivable scan threads,
but those wait on `reps_condition` and `receivable_condition` with `wallets::mutex`, which `stop`
never took. A notify landing between a scan thread's predicate check and its wait was lost, and the
thread slept its full interval before `stop` could join it: 15 minutes for the reps scan on the live
network.

`stop` now takes `wallets::mutex` before `action_mutex`, the order `destroy` uses, so the flag is
set under the mutex the waiters check it with.

* Read the wallet action queue size under `action_mutex`

`container_info` read `actions.size ()` under `wallets::mutex`, but the queue is guarded by
`action_mutex` and the action thread erases from it concurrently. The size now comes from its own
`action_mutex` scope, and the wallet map count keeps its own mutex scope.

__________

Type: commit

Hash: 14d9b7434b4aca7ea9b1948562d8a6e21810995b

Created: yesterday

__________

gr0v1ty/nano-node:14d9b7434b4aca7ea9b1948562d8a6e21810995b