MobileGL/docs/archive/notes/p7/x2-caps-flake.md

X2 — the InitialCapsStartup unit-test flake under load

Wave 2, package X2. Base 9ae1e6c0. Branch p7/x2-caps-flake.

1. What actually failed

InitialCapsStartup.ANullInitialSnapshotFailsInsteadOfStartingWithAPlaceholder failed in 3 of the last 4 integration gates (ctest -L '^unit$' -j 8 on a host where other agents build) and passed 3/3 alone. Reproduced on the same host with 20 busy loops plus a repeating -j 20 clean rebuild in a second build directory (load average 40-60), 8 test processes at a time:

[ RUN      ] InitialCapsStartup.ANullInitialSnapshotFailsInsteadOfStartingWithAPlaceholder
MGPipe: Fatal{ProtocolCorruption, "StreamLink"} invalid data frame header
### rc=134 (SIGABRT) wall=262ms

It is not an EXPECT and it is not the caps path: the abort is StreamLink.cpp:229, the DATA link's reader thread failing FrameReader::Feed on the first 8 bytes it ever reads. The same line kills DelayedTcpSnapshotArmsRunAheadAndLaterSnapshotsCanOnlyDemote (the case project memory recorded flaking before), and AHealthyPeerWithoutInitialCapsFailsStartupWithoutDeviceLoss never fails only because its peer sends nothing.

2. The mechanism

The fixture rendezvoused over loopback TCP with SocketTransport::ConnectTo (two back-to-back connects: control, then aux) against AcceptPair on a thread (two accepts: "first connection is control, second is aux. The order IS the protocol"). Under load the kernel does not deliver two loopback connects to the listener's accept queue in connect order: the final ACK of each handshake is processed by the loopback softirq, and once the connecting thread migrates or ksoftirqd is starved the second connection's ACK can land first. Measured with a raw-socket probe that does exactly ConnectTcp's dance (~/w7/x2bin/swapprobe.c): 0 of 2000 on an idle host, 5 of 3000 (0.17 %) with 20 CPU hogs.

When the pair is swapped the peer's control transport is the client's data socket. The test's first control frame (CapsSnapshot, or the LogLine of the delayed case) then arrives on the client's StreamLink reader, whose framing magic is MGLD, and the reader aborts by name. No timing assumption in the tests, no 250 ms window, no fixed sleep: an ordering the fixture assumed and TCP never promised.

Verdict: TEST. The product does not pair by arrival order on TCP. PH-7 (4) / ID-P7-3 took that out (SocketTransport.h: "never paired by arrival order"; ClientSession.cpp:780: ConnectControl then ConnectDataConnection with Welcome.dataNonce; ServerMain.cpp: AcceptOne + ReceiveOneFrame + DecodeDataBind). ConnectTo/AcceptPair survive for AF_UNIX only, where unix_stream_connect enqueues on the listener synchronously and the order is real. Window 1b's device notes show the same defect in the pre-F product over the LAN ("only 1 of 2 connections within 250 ms", connections reordered under adb forward) - the fixture was simply left behind on the old rendezvous. CapsMirror::ServerConsumes, AwaitFirstCapsSnapshot and FinishStartup step 7 were read end to end and are not implicated; no product bytes change.

3. The fix

MG_Test/Wire/InitialCapsStartupTest.cpp (9f7e8013): the fixture does what ServerMain::ListenerSource does. ConnectControl is accepted with AcceptOne and wrapped as SocketTransport(fd, -1, Server) BEFORE the data connection is opened; the data connection is ConnectDataConnection carrying EncodeDataBind(MintNonce()), accepted with AcceptOne, read with ReceiveOneFrame, decoded with DecodeDataBind and matched with ConstantTimeNonceMatch. No accept thread: a loopback connect completes against the backlog without a concurrent accept. The three cases are untouched.

MG_Test/Wire/SocketTransportTest.cpp (d0284455): TcpCarriesControlAndIndependentDataWithKeepalive had the same ConnectTo/AcceptPair shape and, swapped, hung in recv(serverData, MSG_WAITALL) until the 60 s kill (rc=124). Same rendezvous now; it keeps the NODELAY / keepalive / ShareFd checks on both control transports, the independent-data check on the bound data sockets, and asserts a control-only TCP transport owns no data descriptor (TakeDataFd() == -1).

4. Red-once, under the same load (20 hogs + -j 20 rebuild loop, 8 processes at a time)

case	before	after
InitialCapsStartup.ANullInitialSnapshotFailsInsteadOfStartingWithAPlaceholder	2 / 200, then 2 / 500 (SIGABRT, text above)	0 / 500
InitialCapsStartup.DelayedTcpSnapshotArmsRunAheadAndLaterSnapshotsCanOnlyDemote	5 / 500 (same abort)	0 / 500
SocketTransportTest.TcpCarriesControlAndIndependentDataWithKeepalive	3 / 500 (60 s hang)	0 / 500

Pass wall time did not move (median ~250 ms in both columns).

5. Gates

• ctest -L '^unit$' -j 8: 2435 / 2435, rc 0.
• scripts/ci/fatal_census.py: rc 0, 79 abort sites / 20 files, 44 family words, 3 refusal words, 0 unmarked (unchanged).
• scripts/link_ratchet.py --build-dir build-split --baseline scripts/data/link_ratchet_baseline.txt --assert-monotone: 173, unchanged.
• G1 pull build (build-linux, the p5_g1base flags): .text 0xa52203, nm --defined-only against ~/w7/p7-before/pull-syms.txt +0 / −0 (30570 lines).
• integration-split not re-run: no product file changed.

6. Open

1. AcceptPair over a tcp:// listener is still callable and still pairs by order. The product never calls it that way, but nothing refuses it by name; a MOBILEGL_ERR_UNSUPPORTED for TCP listeners in AcceptPair (and ConnectTo for tcp://) would close the door the fixture walked through. Not in X2 - it is a product change and needs its own red-once.
2. The ::alarm(10) in the fixture is the only remaining time bound; AHealthyPeer... spends 5 s of it in kHandshakeTimeoutMs by design. Not observed to fire.