mirror of
https://github.com/MobileGL-Dev/MobileGL
synced 2026-11-16 22:43:24 +09:00
[MG_Remote, MG_Impl] (P6/dl): the device-lost latch, taken from a descriptor and never from a deadline
Before this, CONTRACT-P6 5.3's own sentence held: "a dead server leaves the
client silently DECLINE-ing every verb forever with glGetError clean".
FatalCode::DeviceLost had zero producers and zero consumers and
glGetGraphicsResetStatus returned GL_NO_ERROR unconditionally, so an
application had no way to learn that the process holding its GPU objects was
gone. Measured cost of that: a server killed on its first Clear left the
client waiting out the full 120 s barrier and reporting Fatal{BarrierTimeout}
- the wrong diagnosis, two minutes late. It now notices in 0.01 s.
THE MACHINERY WAS ALREADY THERE. Doorbell::Wait breaks out on Dead(),
SessionProducer::Park turns that into SessionWait::ShutDown, and ClientSession
already had the named DECLINED path. What did not exist was the death FACT:
D5c's finding is that the client parks on clientBell[0] and rings ITSELF
through clientBell[1], holding both ends, so EOF can never arrive on the one
descriptor it watches. A bell that can wake itself cannot hear a death.
SetDeathWitness(fd) fixes that with no new descriptor: the control socket's
far end is held only by the server. The subtlety is that the bell must watch a
socket it must NEVER read - the transport's reassembler owns those bytes and a
control reply is not a death. So the witness enters the poll set with
`events = POLLRDHUP` and nothing else: POLLHUP/POLLERR/POLLNVAL arrive in
revents unrequested, so the hangup is seen while POLLIN never is. Nothing is
recv'd from it and its ownership stays with SocketTransport.
Dead() AND PeerHungUp() ARE TWO FACTS. Park sets m_dead for POLLERR, POLLNVAL
and unrecognised revents too - each a fault in THIS process's descriptor - and
CondVarDoorbell::Kill() sets it on every orderly Stop(). A latch taken from
Dead() would report a lost GPU for a local fd bug and a context reset for
every clean exit, so only PeerHungUp() arms it.
GL_UNKNOWN_CONTEXT_RESET, not GUILTY or INNOCENT: those assign blame, and the
server died for a reason that never crossed the wire.
D5b, in the control-reply wait, which had the right comment and no code: EOF
on the control stream is device loss; a TIMEOUT with the socket still open is
"ALIVE BUT SILENT" and says so by name. 5.4 forbids calling a frozen or merely
slow peer dead, and under P5e run-ahead a server one frame behind is the
INTENDED steady state - a latch armed by a deadline would fire on a healthy
session under load.
MOBILEGL_IPC_RESPAWN is now a NAMED REFUSAL rather than a variable nothing
reads. The latch is one-way on purpose: every handle this client minted names
an object inside the server process, so recovery means re-pushing the world.
THREE TEST NAMES ADDED, none renamed (G14), and R-16 run both ways:
* remove the witness -> AKilledServerLatchesDeviceLostFromTheHangupAndNot-
ADeadline goes red, takes 2 s instead of 0.01 s, and names D5c.
* collapse PeerHungUp() into Dead() -> ADeadBellIsNotAlwaysAHungUpPeer goes
red naming the local-fd-bug consequence.
* AnOrderlyStopIsNotADeviceLoss stays GREEN under that collapse, and its
comment now says so. The first draft claimed it was the control for it;
running the falsification proved otherwise - under spawn the self bell is a
SocketDoorbell and Stop() never kills one. The comment was corrected rather
than the claim quietly kept.
G1 RE-MEASURED, NOT ASSUMED: 0 symbol diff against the c6 baseline and .text
unchanged at 0xa52203. That measurement earned its keep - it caught a 16-byte
drift that had nothing to do with dl: P6's O_APPEND log-sink fix was compiling
into the PULL build, where a second process can never exist. Now guarded, and
.text is back to the baseline exactly.
Full unit suite: 2350 tests, the same two pre-existing environmental failures
and no others. All six OpenRA arms (2 backends x monolith/inproc/spawn) green.
This commit is contained in:
@@ -22,6 +22,22 @@
|
||||
|
||||
// Same fallback as FdPassing.cpp: on macOS / BSD the protection is SO_NOSIGPIPE on the
|
||||
// socket, set in SocketDoorbell's constructor, not a per-send flag.
|
||||
// THE WITNESS'S `events`, and the whole reason it can watch a socket it must not read.
|
||||
//
|
||||
// POLLRDHUP is what Linux and bionic report when the PEER shuts down its writing end - the exact
|
||||
// fact the death witness wants - and asking for it alone means poll never reports POLLIN for that
|
||||
// descriptor, so unread control replies queued on it can neither wake this bell nor be consumed
|
||||
// from under SocketTransport's reassembler. POLLHUP / POLLERR / POLLNVAL arrive in revents whether
|
||||
// requested or not, so an 0 here would still catch a full close; POLLRDHUP only makes the
|
||||
// half-close case - the one a dying peer actually produces first - visible too.
|
||||
#if !defined(_WIN32)
|
||||
#if defined(POLLRDHUP)
|
||||
#define MOBILEGL_POLL_PEER_HANGUP POLLRDHUP
|
||||
#else
|
||||
#define MOBILEGL_POLL_PEER_HANGUP 0
|
||||
#endif
|
||||
#endif
|
||||
|
||||
#if !defined(_WIN32) && !defined(MSG_NOSIGNAL)
|
||||
#define MSG_NOSIGNAL 0
|
||||
#endif
|
||||
@@ -275,7 +291,7 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
if (written < 0 && (errno == EPIPE || errno == ECONNRESET)) {
|
||||
// The peer is gone: it can never ring back either, so latch it
|
||||
// here too rather than waiting for a Park to discover it.
|
||||
m_dead = true;
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return;
|
||||
}
|
||||
MGLOG_D("MG_Remote doorbell: send failed (errno=%d)", errno);
|
||||
@@ -289,7 +305,7 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
if (m_fd < 0) {
|
||||
return false;
|
||||
}
|
||||
if (m_fd < 0 || m_dead) {
|
||||
if (m_fd < 0 || m_dead.load(std::memory_order_acquire)) {
|
||||
return false;
|
||||
}
|
||||
const auto start = std::chrono::steady_clock::now();
|
||||
@@ -302,10 +318,22 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
const long long remaining = static_cast<long long>(timeoutMs) - elapsed;
|
||||
pollTimeout = remaining <= 0 ? 0 : static_cast<int>(remaining);
|
||||
}
|
||||
struct pollfd pfd{};
|
||||
pfd.fd = m_fd;
|
||||
pfd.events = POLLIN;
|
||||
const int ready = ::poll(&pfd, 1, pollTimeout);
|
||||
// THE WITNESS ASKS FOR HANGUP AND NOTHING ELSE. POLLHUP/POLLERR/POLLNVAL are
|
||||
// reported in revents whether or not they were requested, so an `events` of
|
||||
// POLLRDHUP alone gets the peer's shutdown and never POLLIN - which is the whole
|
||||
// point: the control socket legitimately carries unread REPLY bytes, and asking for
|
||||
// readability there would make every queued reply a wakeup, spin this loop, and race
|
||||
// SocketTransport's reassembler for the same bytes. Nothing is ever recv'd from it.
|
||||
struct pollfd pfds[2]{};
|
||||
pfds[0].fd = m_fd;
|
||||
pfds[0].events = POLLIN;
|
||||
const bool watching = m_witnessFd >= 0;
|
||||
if (watching) {
|
||||
pfds[1].fd = m_witnessFd;
|
||||
pfds[1].events = MOBILEGL_POLL_PEER_HANGUP;
|
||||
}
|
||||
const int ready = ::poll(pfds, watching ? 2 : 1, pollTimeout);
|
||||
struct pollfd& pfd = pfds[0];
|
||||
if (ready < 0) {
|
||||
if (errno == EINTR) {
|
||||
continue; // a signal is not a wakeup; keep the deadline
|
||||
@@ -316,6 +344,20 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
if (ready == 0) {
|
||||
return false; // timed out
|
||||
}
|
||||
// THE WITNESS FIRST, because a peer that is gone makes every other answer stale. It
|
||||
// reports only hangup shapes by construction, so any revent on it IS the death.
|
||||
if (watching && pfds[1].revents != 0) {
|
||||
MGLOG_I("MG_Remote doorbell: the peer hung up (witness fd %d, revents=0x%X) - "
|
||||
"this is the DEATH FACT, taken from a descriptor and not from a "
|
||||
"deadline: a server one frame behind is the intended steady state and "
|
||||
"must never be mistaken for one that is gone (CONTRACT-P6 5.4)",
|
||||
m_witnessFd, static_cast<unsigned>(pfds[1].revents));
|
||||
// THE CAUSE BEFORE THE FACT, so a reader that sees Dead() can never find
|
||||
// PeerHungUp() still false and conclude "orderly teardown".
|
||||
m_peerHungUp.store(true, std::memory_order_release);
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return false;
|
||||
}
|
||||
// revents has to be inspected, not just `ready > 0`. Once the peer
|
||||
// closes its end the descriptor is permanently poll-ready with
|
||||
// nothing to read (measured on Linux: revents=POLLIN|POLLHUP,
|
||||
@@ -326,14 +368,14 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
if ((pfd.revents & (POLLERR | POLLNVAL)) != 0) {
|
||||
MGLOG_D("MG_Remote doorbell: fd %d unusable (revents=0x%X)", m_fd,
|
||||
static_cast<unsigned>(pfd.revents));
|
||||
m_dead = true;
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return false;
|
||||
}
|
||||
if ((pfd.revents & POLLIN) != 0) {
|
||||
if (Drain() != 0) {
|
||||
return true; // a real wakeup byte
|
||||
}
|
||||
if (m_dead) {
|
||||
if (m_dead.load(std::memory_order_acquire)) {
|
||||
return false; // EOF, not an event
|
||||
}
|
||||
// Ready but empty and still alive: someone else drained it.
|
||||
@@ -341,7 +383,7 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
return true;
|
||||
}
|
||||
if ((pfd.revents & POLLHUP) != 0) {
|
||||
m_dead = true;
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return false;
|
||||
}
|
||||
// Readiness with no bit we requested or recognise: there is
|
||||
@@ -349,7 +391,7 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
// poll this descriptor again.
|
||||
MGLOG_D("MG_Remote doorbell: fd %d ready with revents=0x%X", m_fd,
|
||||
static_cast<unsigned>(pfd.revents));
|
||||
m_dead = true;
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -368,7 +410,7 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
if (got == 0) {
|
||||
// Orderly shutdown on a stream socket: the peer is gone and
|
||||
// will never ring again.
|
||||
m_dead = true;
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return consumed;
|
||||
}
|
||||
if (errno == EINTR) {
|
||||
@@ -378,13 +420,13 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
return consumed; // drained
|
||||
}
|
||||
MGLOG_D("MG_Remote doorbell: recv failed (errno=%d)", errno);
|
||||
m_dead = true;
|
||||
m_dead.store(true, std::memory_order_release);
|
||||
return consumed;
|
||||
}
|
||||
}
|
||||
|
||||
void SocketDoorbell::Reset() {
|
||||
if (m_fd < 0 || m_dead) {
|
||||
if (m_fd < 0 || m_dead.load(std::memory_order_acquire)) {
|
||||
return;
|
||||
}
|
||||
(void)Drain();
|
||||
|
||||
@@ -171,6 +171,17 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
// bell that cannot die.
|
||||
virtual bool Dead() const { return false; }
|
||||
|
||||
// P6 `dl`: DEAD AND DEAD-BECAUSE-THE-PEER-DIED ARE NOT THE SAME FACT, and only the
|
||||
// second one is a device loss. Dead() is also true after an ORDERLY teardown -
|
||||
// CondVarDoorbell::Kill() is how Stop() wakes a parked applier, and the barrier reports
|
||||
// SessionWait::ShutDown for that too. Latching device-lost from Dead() would therefore
|
||||
// arm it on every clean exit.
|
||||
//
|
||||
// This says the peer HUNG UP: a descriptor whose far end only the peer held reported
|
||||
// hangup. Nothing this side did can produce it. False for every doorbell that has no
|
||||
// peer process, which is what makes the whole latch a no-op under inproc and monolith.
|
||||
virtual bool PeerHungUp() const { return false; }
|
||||
|
||||
// How many Wait() calls on this bell exhausted their spin budget and
|
||||
// really blocked. THE WAIT LEDGER'S RAW READING: the pair (waits, parks)
|
||||
// is what says whether the spin budget is sized for the workload, and it
|
||||
@@ -419,10 +430,30 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
void Notify() override;
|
||||
bool Park(std::uint32_t timeoutMs) override;
|
||||
void Reset() override;
|
||||
bool Dead() const override { return m_dead; }
|
||||
bool Dead() const override { return m_dead.load(std::memory_order_acquire); }
|
||||
bool PeerHungUp() const override { return m_peerHungUp.load(std::memory_order_acquire); }
|
||||
|
||||
int Fd() const { return m_fd; }
|
||||
|
||||
// P6 `dl` (CONTRACT-P6 D5c). THE DESCRIPTOR THAT ANSWERS "IS THE PEER STILL THERE",
|
||||
// which for this object can never be the one it parks on.
|
||||
//
|
||||
// The client parks on clientBell[0] and rings ITSELF through clientBell[1], so it holds
|
||||
// BOTH ends of that socketpair. EOF arrives only when every writer closes and the client
|
||||
// is one of them, so that descriptor cannot hang up no matter what happens to the server.
|
||||
// A bell that can wake itself is a bell that cannot hear a death. Measured: a server that
|
||||
// died on its first Clear left the client waiting out the full 120 s barrier and
|
||||
// reporting Fatal{BarrierTimeout} - the wrong diagnosis, two minutes late.
|
||||
//
|
||||
// The witness is a descriptor whose FAR END ONLY THE PEER HOLDS - in practice the control
|
||||
// socket. It is NOT OWNED here and is NEVER READ FROM: Park adds it to the poll set with
|
||||
// `events` asking for hangup ALONE, so a control reply sitting unread in its queue cannot
|
||||
// wake the bell and cannot be consumed from under SocketTransport's reassembler. What it
|
||||
// contributes is exactly one fact, and only once: the peer is gone.
|
||||
//
|
||||
// -1 disables it, which is what every bell that has no peer socket uses.
|
||||
void SetDeathWitness(int fd) { m_witnessFd = fd; }
|
||||
|
||||
private:
|
||||
// Consumes every queued wakeup byte and returns how many. Latches
|
||||
// m_dead on EOF: recv returning 0 on a stream socket is the peer's
|
||||
@@ -432,9 +463,16 @@ namespace MobileGL::MG_Remote::Transport {
|
||||
|
||||
int m_fd; // polled
|
||||
int m_notifyFd; // written; equals m_fd in the single-fd (cross-process) form
|
||||
int m_witnessFd = -1; // polled for hangup only; NOT owned, NEVER read
|
||||
// Set ONLY by the witness branch of Park, never by Kill or by our own EOF. See
|
||||
// Doorbell::PeerHungUp for why the distinction is the whole design.
|
||||
std::atomic<bool> m_peerHungUp{false};
|
||||
std::uint8_t m_code;
|
||||
bool m_ownsFd;
|
||||
bool m_dead = false;
|
||||
// ATOMIC BECAUSE THE LATCH READS IT ACROSS THREADS. Park() runs on whichever thread is
|
||||
// waiting; the device-lost latch is consulted from the GL thread and from
|
||||
// glGetGraphicsResetStatus. It was a plain bool while Dead() had no cross-thread reader.
|
||||
std::atomic<bool> m_dead{false};
|
||||
};
|
||||
#endif
|
||||
|
||||
|
||||
Reference in New Issue
Block a user