Compare commits

...
22 Commits
Author SHA1 Message Date
swung0x48 3a1058dbab [CI] (.github/workflows, CONTRIBUTING.md, scripts): compare new branches with the default branch 2026-11-16 02:46:57 -04:00
swung0x48 c9003c89a1 [Test] (MG_Test/Util): hold the drain tests' queue until StopAndDrain takes it 2026-11-16 02:46:56 -04:00
swung0x48 9ce8caa1d0 [Fix, CI] (.github/actions, .github/workflows, scripts/ci): stop creating the swap file over a zram or partition device 2026-11-16 02:46:55 -04:00
swung0x48 8d3d3d13a7 [CI, Tools] (android-plugin): log each shard case's settings, record and transport lines
- The monolith lane fails when the library log shows a split transport
2026-11-15 18:38:03 -04:00
swung0x48 d41563fb8d [CI, Tools] (.github/workflows, scripts/ci, tools/trace_replay): pack emulator shards by measured replay seconds
- Report free disk space around the swap, fixture and AVD steps
- Print the configure command line of the Linux builds
2026-11-15 16:48:24 -04:00
swung0x48 25c984927d [CI] (.github/workflows, docs/ci, scripts/ci): trim setup time in the build and split lane jobs 2026-11-15 11:47:46 -04:00
swung0x48 8165793e0d [CI, Tools] (...): replay only smoke cases on Android monolith and spawn
- Android inproc and all three Linux transports still replay every case
- A manual run with android_full_matrix replays the full emulator matrix
2026-11-15 11:47:38 -04:00
swung0x48 697bcc6d45 [CI, Tools] (...): replay trace cases in shards from one fixture cache 2026-11-15 11:47:18 -04:00
swung0x48 222f6209a0 [CI] (.github/actions, .github/workflows): create swap even when the runner image has none 2026-11-15 05:45:04 -04:00
swung0x48 b2248b240f [CI, Docs] (.github/workflows, userguide): build the dev APK split so it defaults to inproc 2026-11-14 22:07:43 -04:00
swung0x48 8db74d69ae [Chore] (android-plugin/app, MobileGL, userguide): bump the version to 26.11 2026-11-08 20:20:39 -04:00
swung0x48 04c8773d77 [Fix, Test] (MG_Impl/GLImpl, MG_Test/Texture, MG_Util/Converters): report proxy texture level sizes
- Record the size a proxy glTexImage* asks for and answer it from
  glGetTexLevelParameter, or zeros when it passes the texture size limits
- Map GL_PROXY_TEXTURE_1D and GL_PROXY_TEXTURE_3D to their upload targets
2026-11-08 19:43:21 -04:00
swung0x48 8497eee197 [CI] (.github/workflows): run dev in the split CI shape and pin the verify build to monolith 2026-11-07 08:08:50 -04:00
swung0x48 13e9d0df5c [Fix] (MG_Backend/DirectVulkan): take a deferred window clear's sRGB decision at the glClear 2026-11-07 08:08:48 -04:00
swung0x48 4256a66d09 [Fix] (MG_Impl/Pipe): zero the pixel pack record's padding before it is sent 2026-11-07 07:39:14 -04:00
swung0x48 cefb70d1a7 [Fix] (MG_Util/ShaderTranspiler, MG_Util/SelfTest, MG_Test): split clip distance code to fit the size and format checks 2026-11-07 07:27:58 -04:00
swung0x48 cdf43a4a59 [Fix] (MG_Backend/DirectVulkan, MG_Util/ShaderTranspiler): apply GL_CLIP_DISTANCEi in the shader where Magma clips natively 2026-11-07 02:47:30 -04:00
swung0x48 e8dc4e2320 [Test] (MG_Test): pin the emulated eight clip distances on both backends without native support 2026-11-07 02:28:22 -04:00
swung0x48 6182704c03 [Feat] (MG_Util/ShaderTranspiler, MG_Backend, MG_Util/SelfTest): emulate gl_ClipDistance where the backend cannot clip natively and report it in POST 2026-11-07 02:06:25 -04:00
BZLZHH d3ec9b8179 [Docs] (docs/Performance): add a skill for reading Mali GPU counters 2026-11-01 19:51:20 +08:00
BZLZHH 05dfff9100 [Perf] (MG_Backend/DirectVulkan, MG_IntegrationTest/Scenarios): defer window clears while a window pass is open
- A whole-window clear with a window pass open ends that pass and folds
  into the next one, instead of running a load-clear-store pass of its own
- A present drops the depth/stencil half of a pending window clear
2026-11-01 19:20:48 +08:00
BZLZHH 6c59b0391c [Perf] (MG_Backend/DirectVulkan): defer window clears into the next window pass
- A whole-window glClear with no pass open on the window waits for the next
  window pass, which loads it with LOAD_OP_CLEAR; other uses run it first
- The first window pass after a present loads undefined content as DONT_CARE
2026-11-01 13:42:44 +08:00
92 changed files with 9189 additions and 1233 deletions
+15
View File
@@ -0,0 +1,15 @@
name: Set swap space
description: Replace the runner's swap with a swap file of the given size
inputs:
swap-size-gb:
description: Size of the swap file in GiB
required: true
runs:
using: composite
steps:
# scripts/ci/set_swap_space.sh says which swap it replaces, where the file goes and when it
# is sized down; it prints the swap state and free space before and after.
- shell: bash
env:
SWAP_SIZE_GB: ${{ inputs.swap-size-gb }}
run: bash "${GITHUB_ACTION_PATH}/../../../scripts/ci/set_swap_space.sh"
+304
View File
@@ -0,0 +1,304 @@
#!/usr/bin/env bash
# One cache entry that holds every Git LFS trace fixture of the CI cases.
#
# key derive the actions/cache key, restore prefix and path of the bundle
# ensure after a cache miss: complete the bundle directory, fetching only the
# cases whose files are missing from it or do not match their pointers
# wait <key> [secs] wait until the cache entry <key> is listed for this ref (needs GH_TOKEN)
# install <case>... move the fixtures of these cases from the restored bundle directory into
# the fixture directory, fetch the cases it does not hold, and check every
# file against its pointer
#
# The key is content-addressed on the pointer oids tracked at HEAD, which a plain checkout
# reads without smudging. The bundle directory is the same path for every revision, so
# `ensure` can start from an older bundle. Fixtures stored in Git itself rather than in Git
# LFS (OpenRA) are not in the bundle; `install` checks that they are real files.
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=trace-fixture-lib.sh
. "${script_dir}/trace-fixture-lib.sh"
# Bump when the key derivation or the bundle layout changes in a way that must invalidate
# old entries; the content digest alone would not notice a format change.
key_schema="v1"
bundle_dir="trace-fixture-bundle"
fixture_dir="tools/trace_replay/fixtures"
python_bin="${PYTHON:-python3}"
if ! command -v "${python_bin}" >/dev/null 2>&1 && command -v python >/dev/null 2>&1; then
python_bin=python
fi
usage() {
echo "usage: $0 <key|ensure|wait <key> [seconds]|install <trace-case>...>" >&2
exit 2
}
# Writes "name=value" to $GITHUB_OUTPUT when running under Actions, and to stdout as well so
# the script stays runnable off-CI.
emit_output() {
local name="$1"
local value="$2"
if [ -n "${GITHUB_OUTPUT:-}" ]; then
printf '%s=%s\n' "${name}" "${value}" >> "${GITHUB_OUTPUT}"
fi
printf '%s=%s\n' "${name}" "${value}"
}
# "<case><TAB><file>" for every fixture file of every CI case.
ci_fixture_manifest() {
"${python_bin}" tools/trace_replay/trace_cases.py --ci --format fixture-manifest \
--fixture-root "${fixture_dir}" | tr -d '\r'
}
is_lfs_fixture() {
get_lfs_metadata "$1" >/dev/null 2>&1
}
# A fixture that Git stores directly must be a real file, not a pointer left by a wrong
# attribute.
check_plain_fixture() {
local file="$1"
if [ ! -s "${file}" ]; then
echo "fixture file is missing or empty: ${file}" >&2
return 1
fi
if head -n 1 "${file}" | grep -q "version https://git-lfs.github.com/spec/v1"; then
echo "fixture should not be stored as an LFS pointer: ${file}" >&2
return 1
fi
}
# Puts the tracked pointer files back. The smudge filter is skipped so that a clone with Git
# LFS set up does not download the objects here.
restore_pointers() {
GIT_LFS_SKIP_SMUDGE=1 git checkout -- "$@"
}
verify_lfs_fixture() {
local on_disk="$1"
local tracked="$2"
local metadata expected_oid expected_size
metadata="$(get_lfs_metadata "${tracked}")" || return 1
read -r expected_oid expected_size <<< "${metadata}"
verify_fixture_file "${on_disk}" "${tracked}" "${expected_oid}" "${expected_size}"
}
# Moves the given Git LFS fixtures of one case from the bundle directory into the fixture
# directory. Succeeds when every one of them is then in place and matches its pointer.
take_from_bundle() {
local file bundled
local incomplete=0
for file in "$@"; do
bundled="${bundle_dir}/$(basename "${file}")"
# Two cases may share a file; the second finds it already in place.
if [ -f "${bundled}" ]; then
mv -f "${bundled}" "${file}"
fi
if ! verify_lfs_fixture "${file}" "${file}" 2>/dev/null; then
incomplete=1
fi
done
return "${incomplete}"
}
# Downloads the given Git LFS fixtures of one case over their pointers: the mirrors first,
# Git LFS for what no mirror serves. The fetch script verifies what it downloaded.
fetch_case_fixtures() {
local case_name="$1"
shift
local file
for file in "$@"; do
rm -f "${file}" "${file}.tmp"
done
restore_pointers "$@" || return 1
bash "${script_dir}/fetch-trace-fixture-lfs.sh" "${case_name}" "${fixture_dir}"
}
[ "$#" -ge 1 ] || usage
command_name="$1"
shift
case "${command_name}" in
key)
[ "$#" -eq 0 ] || usage
# One "<basename> <oid> <size>" line per bundled file, sorted, each file once.
manifest=""
count=0
while IFS= read -r file; do
[ -n "${file}" ] || continue
if ! metadata="$(get_lfs_metadata "${file}" 2>/dev/null)"; then
continue
fi
read -r expected_oid expected_size <<< "${metadata}"
manifest+="$(basename "${file}") ${expected_oid} ${expected_size}"$'\n'
count=$((count + 1))
done < <(ci_fixture_manifest | cut -f2 | LC_ALL=C sort -u)
if [ "${count}" -eq 0 ]; then
echo "no Git LFS fixture found for the CI trace cases; the bundle would be empty" >&2
exit 1
fi
duplicates="$(printf '%s' "${manifest}" | cut -d' ' -f1 | LC_ALL=C sort | uniq -d)"
if [ -n "${duplicates}" ]; then
echo "two fixture files share a name, so the bundle directory cannot hold both: ${duplicates}" >&2
exit 1
fi
digest="$(printf '%s' "${manifest}" | sha256sum | awk '{ print substr($1, 1, 16) }')"
emit_output "key" "trace-fixture-bundle-${key_schema}-${digest}"
emit_output "restore-prefix" "trace-fixture-bundle-${key_schema}-"
emit_output "path" "${bundle_dir}"
emit_output "files" "${count}"
;;
ensure)
[ "$#" -eq 0 ] || usage
mkdir -p "${bundle_dir}" "${fixture_dir}"
mapfile -t manifest_lines < <(ci_fixture_manifest)
declare -A case_seen=()
case_order=()
for line in "${manifest_lines[@]}"; do
case_name="${line%%$'\t'*}"
if [ -z "${case_seen[${case_name}]:-}" ]; then
case_seen["${case_name}"]=1
case_order+=("${case_name}")
fi
done
fetched=0
reused=0
bundled_files=()
for case_name in "${case_order[@]}"; do
lfs_files=()
for line in "${manifest_lines[@]}"; do
[ "${line%%$'\t'*}" = "${case_name}" ] || continue
file="${line#*$'\t'}"
if is_lfs_fixture "${file}"; then
lfs_files+=("${file}")
else
check_plain_fixture "${file}"
fi
done
[ "${#lfs_files[@]}" -gt 0 ] || continue
# Take what an older bundle already holds, and fetch the case if that is not all of it.
if take_from_bundle "${lfs_files[@]}"; then
reused=$((reused + 1))
else
echo "Fetching trace fixtures of ${case_name}: the bundle does not hold them"
fetch_case_fixtures "${case_name}" "${lfs_files[@]}"
fetched=$((fetched + 1))
fi
bundled_files+=("${lfs_files[@]}")
done
# The bundle is rebuilt from the verified files, so nothing an older bundle carried and
# no case names any more is saved again.
rm -rf "${bundle_dir}"
mkdir -p "${bundle_dir}"
declare -A file_seen=()
saved=0
for file in "${bundled_files[@]}"; do
[ -z "${file_seen[${file}]:-}" ] || continue
file_seen["${file}"]=1
verify_lfs_fixture "${file}" "${file}"
mv -f "${file}" "${bundle_dir}/$(basename "${file}")"
saved=$((saved + 1))
done
if [ "${saved}" -eq 0 ]; then
echo "the trace fixture bundle came out empty" >&2
exit 1
fi
# Leave the working tree as a checkout has it: pointers in the fixture directory.
restore_pointers "${!file_seen[@]}"
echo "Trace fixture bundle: ${saved} file(s), ${reused} case(s) from the restored bundle, ${fetched} case(s) fetched."
emit_output "fetched" "${fetched}"
;;
wait)
[ "$#" -ge 1 ] && [ "$#" -le 2 ] || usage
key="$1"
limit="${2:-300}"
# Another workflow of the same push may be the one saving this key. The replay jobs
# restore the entry, so it has to be listed before this job ends. The caller checks the
# entry itself afterwards; this only waits, and never fails.
#
# The listing is narrowed to the ref of this run. The key is content-addressed, so another
# branch may already hold an entry under it, and that entry cannot be restored from here:
# counting it would end the wait while the entry of this ref is still being uploaded.
query=(-f "key=${key}" -f per_page=100)
if [ -n "${GITHUB_REF:-}" ]; then
query+=(-f "ref=${GITHUB_REF}")
fi
waited=0
refused=0
while :; do
if listed="$(gh api -X GET "repos/${GITHUB_REPOSITORY}/actions/caches" "${query[@]}" \
--jq '[.actions_caches[] | select(.key == "'"${key}"'")] | length' 2>/dev/null)"; then
if [ "${listed:-0}" -ge 1 ]; then
echo "cache entry ${key} is listed after ${waited} s"
break
fi
else
refused=$((refused + 1))
if [ "${refused}" -ge 3 ]; then
echo "the cache listing cannot be read (no token, or no actions: read permission); not waiting"
break
fi
fi
if [ "${waited}" -ge "${limit}" ]; then
echo "cache entry ${key} is not listed after ${waited} s"
break
fi
sleep 10
waited=$((waited + 10))
done
;;
install)
[ "$#" -ge 1 ] || usage
installed=0
fetched=0
for case_name in "$@"; do
mapfile -t files < <(trace_fixture_files "${case_name}" "${fixture_dir}" "${python_bin}")
if [ "${#files[@]}" -eq 0 ]; then
echo "no fixture files declared for trace case: ${case_name}" >&2
exit 1
fi
lfs_files=()
for file in "${files[@]}"; do
if is_lfs_fixture "${file}"; then
lfs_files+=("${file}")
else
check_plain_fixture "${file}"
fi
done
[ "${#lfs_files[@]}" -gt 0 ] || continue
# A bundle that was not restored (the cache entry was evicted, or its download failed)
# or that lacks a file costs this job a download of the case, not the replay.
if ! take_from_bundle "${lfs_files[@]}"; then
echo "::warning::the restored fixture bundle does not hold the trace fixtures of ${case_name}; fetching them"
if ! fetch_case_fixtures "${case_name}" "${lfs_files[@]}"; then
echo "::error::the trace fixtures of ${case_name} are not in the restored bundle and could not be fetched"
exit 1
fi
fetched=$((fetched + 1))
for file in "${lfs_files[@]}"; do
if ! verify_lfs_fixture "${file}" "${file}"; then
echo "::error::trace fixture ${file} of ${case_name} does not match its Git LFS pointer after the fetch"
exit 1
fi
done
fi
installed=$((installed + ${#lfs_files[@]}))
done
# The rest of the bundle belongs to other shards.
rm -rf "${bundle_dir}"
echo "Installed and verified ${installed} Git LFS fixture file(s) for $# trace case(s), ${fetched} case(s) fetched."
;;
*)
usage
;;
esac
-117
View File
@@ -1,117 +0,0 @@
#!/usr/bin/env bash
# Cache-side helper for trace fixtures.
#
# key <case> [fixture-dir] derive the actions/cache key and path list
# verify <case> [fixture-dir] check restored fixtures against their pointers
# reset <case> [fixture-dir] drop restored fixtures, leaving the pointers
#
# The cache key is content-addressed on the Git LFS pointer oids tracked at
# HEAD, which are readable from a plain checkout without smudging. Fixture
# content therefore maps 1:1 onto a key: unchanged content hits, changed
# content is a new key and thus a miss, and the download path handles it. The
# key deliberately carries no restore-keys prefix in the workflow - a fixture
# that does not match the pointer exactly must never be restored.
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=trace-fixture-lib.sh
. "${script_dir}/trace-fixture-lib.sh"
# Bump when the key derivation changes in a way that must invalidate old
# entries; the content digest alone would not notice a format change.
key_schema="v1"
if [ "$#" -lt 2 ] || [ "$#" -gt 3 ]; then
echo "usage: $0 <key|verify|reset> <trace-case> [fixture-dir]" >&2
exit 2
fi
command_name="$1"
case_name="$2"
fixture_dir="${3:-tools/trace_replay/fixtures}"
python_bin="${PYTHON:-python3}"
if ! command -v "${python_bin}" >/dev/null 2>&1 && command -v python >/dev/null 2>&1; then
python_bin=python
fi
mapfile -t files < <(trace_fixture_files "${case_name}" "${fixture_dir}" "${python_bin}")
if [ "${#files[@]}" -eq 0 ]; then
echo "no fixture files declared for trace case: ${case_name}" >&2
exit 1
fi
# Writes "name=value" to $GITHUB_OUTPUT when running under Actions, and to
# stdout otherwise so the script stays runnable (and testable) off-CI.
emit_output() {
local name="$1"
local value="$2"
if [ -n "${GITHUB_OUTPUT:-}" ]; then
if [[ "${value}" == *$'\n'* ]]; then
local delimiter="ghadelim_$(date +%s%N)_$$"
{
printf '%s<<%s\n' "${name}" "${delimiter}"
printf '%s\n' "${value}"
printf '%s\n' "${delimiter}"
} >> "${GITHUB_OUTPUT}"
else
printf '%s=%s\n' "${name}" "${value}" >> "${GITHUB_OUTPUT}"
fi
fi
printf '%s=%s\n' "${name}" "${value}"
}
sanitize_case() {
printf '%s' "$1" | sed 's/[^A-Za-z0-9._-]/_/g'
}
case "${command_name}" in
key)
manifest=""
for file in "${files[@]}"; do
# A case whose fixtures are committed directly rather than through Git LFS
# (OpenRA) has no pointer oid to key on, and nothing to download either.
# Report it as uncacheable so the workflow skips the cache entirely.
if ! metadata="$(get_lfs_metadata "${file}" 2>/dev/null)"; then
echo "trace case ${case_name} is not stored in Git LFS; skipping fixture cache" >&2
emit_output "cacheable" "false"
emit_output "key" ""
exit 0
fi
read -r expected_oid expected_size <<< "${metadata}"
manifest+="$(basename "${file}") ${expected_oid} ${expected_size}"$'\n'
done
digest="$(printf '%s' "${manifest}" | sha256sum | awk '{ print substr($1, 1, 16) }')"
safe_case="$(sanitize_case "${case_name}")"
emit_output "cacheable" "true"
emit_output "key" "trace-fixture-${key_schema}-${safe_case}-${digest}"
emit_output "paths" "$(printf '%s\n' "${files[@]}")"
;;
verify)
for file in "${files[@]}"; do
metadata="$(get_lfs_metadata "${file}")"
read -r expected_oid expected_size <<< "${metadata}"
verify_fixture_file "${file}" "${file}" "${expected_oid}" "${expected_size}"
done
echo "Verified ${#files[@]} fixture file(s) for ${case_name} against the tracked Git LFS pointers."
;;
reset)
# Put the working tree back to the pointer files a fresh checkout would
# have, so that a rejected cache entry falls through to exactly the same
# download path a cache miss takes.
for file in "${files[@]}"; do
rm -f "${file}" "${file}.tmp"
done
git checkout -- "${files[@]}"
echo "Reset ${#files[@]} fixture file(s) for ${case_name} to their tracked Git LFS pointers."
;;
*)
echo "unknown command: ${command_name}" >&2
exit 2
;;
esac
+1 -1
View File
@@ -2,7 +2,7 @@
# Shared helpers for trace-fixture handling: reading the in-tree Git LFS pointer # Shared helpers for trace-fixture handling: reading the in-tree Git LFS pointer
# metadata and verifying a fixture file against it. Sourced by # metadata and verifying a fixture file against it. Sourced by
# fetch-trace-fixture-lfs.sh (verify after download) and by # fetch-trace-fixture-lfs.sh (verify after download) and by
# trace-fixture-cache.sh (cache key derivation and verify after cache restore), # trace-fixture-bundle.sh (cache key derivation and verify after cache restore),
# so both paths agree on what a valid fixture is. # so both paths agree on what a valid fixture is.
# Reads the Git LFS pointer tracked at HEAD for a fixture path and prints # Reads the Git LFS pointer tracked at HEAD for a fixture path and prints
+342 -306
View File
@@ -12,16 +12,44 @@ on:
- feat/disaggregated - feat/disaggregated
- codex/fix-github-ci - codex/fix-github-ci
- cleanup - cleanup
# A docs-only push builds nothing this workflow tests (231 emulator legs per run). The Test # A push that only touches these paths changes nothing this workflow builds or replays. The
# workflow keeps running for docs: it hosts the doc citation lint. # Test workflow keeps running for docs: it hosts the doc citation lint. The Android build
# compiles none of the unit, integration or benchmark trees, with one exception that keeps
# MobileGL/MG_Benchmark from being listed whole: MG_Util/SelfTest/DriverBenchJni.cpp
# includes MG_Benchmark/Driver/DriverBenchCases.inc.
paths-ignore: paths-ignore:
- 'docs/**' - 'docs/**'
- '**/*.md' - '**/*.md'
- 'MobileGL/MG_Test/**'
- 'MobileGL/MG_IntegrationTest/**'
- 'MobileGL/MG_Benchmark/CMakeLists.txt'
- 'MobileGL/MG_Benchmark/SanityBench.cpp'
- 'MobileGL/MG_Benchmark/Buffer/**'
- 'MobileGL/MG_Benchmark/Container/**'
- 'MobileGL/MG_Benchmark/Program/**'
- 'MobileGL/MG_Benchmark/ShaderCache/**'
- 'MobileGL/MG_Benchmark/Transpile/**'
- 'tools/cts/**'
- 'tools/device_bench/**'
- '.github/workflows/test.yml'
- '.github/workflows/hygiene.yml'
- '.github/actions/setup-test-runner/**'
workflow_dispatch: workflow_dispatch:
inputs:
android_full_matrix:
description: Replay every trace case on the monolith and spawn emulator lanes too
type: boolean
default: false
env: env:
# Software-rendered CI submissions may exceed the interactive GPU hang budget. # Software-rendered CI submissions may exceed the interactive GPU hang budget.
MOBILEGL_GPU_HANG_BUDGET_MS: "0" MOBILEGL_GPU_HANG_BUDGET_MS: "0"
# The emulator replays every trace case on inproc, the transport the launcher ships. On
# monolith and spawn it replays the android_smoke cases of trace_cases.json, because
# test.yml replays every case on all three transports on Linux. A manual run with
# android_full_matrix replays every case on all three emulator lanes. A push has no inputs,
# so it gets the reduced lanes.
MOBILEGL_ANDROID_LANES: ${{ inputs.android_full_matrix && 'full' || 'inproc-full' }}
# ONE LIVE RUN PER BRANCH, as in test.yml: a superseded push is canceled rather than left queued # ONE LIVE RUN PER BRANCH, as in test.yml: a superseded push is canceled rather than left queued
# behind its successor; a manual dispatch gets a group of its own; dev queues rather than cancels. # behind its successor; a manual dispatch gets a group of its own; dev queues rather than cancels.
@@ -39,16 +67,26 @@ jobs:
CCACHE_BASEDIR: ${{ github.workspace }} CCACHE_BASEDIR: ${{ github.workspace }}
CCACHE_COMPRESS: "true" CCACHE_COMPRESS: "true"
CCACHE_DIR: ${{ github.workspace }}/.ccache CCACHE_DIR: ${{ github.workspace }}/.ccache
CCACHE_MAXSIZE: 4G # One build's objects (three native configurations) take about 1 GB. The limit leaves
# half of that again for the objects a push replaces, and is what the cleanup before
# the save trims the entry to.
CCACHE_MAXSIZE: 1500M
CCACHE_NOHASHDIR: "true" CCACHE_NOHASHDIR: "true"
MOBILEGL_CMAKE_COMPILER_LAUNCHER: ccache MOBILEGL_CMAKE_COMPILER_LAUNCHER: ccache
MOBILEGL_CI_DISAGGREGATED: ${{ (github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && 'ON' || 'OFF' }} MOBILEGL_CI_DISAGGREGATED: ${{ (github.ref_name == 'dev' || github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && 'ON' || 'OFF' }}
steps: steps:
- name: Checkout repo - name: Checkout repo
uses: actions/checkout@v6 uses: actions/checkout@v6
with:
submodules: recursive # Only the submodules the two APKs are built from, shallow and in parallel, in place of a
# recursive checkout of every submodule in the tree: the ones the root CMakeLists.txt
# configures, and apitrace with the directories its thirdparty/CMakeLists.txt requires,
# for the trace APK. The step fails when a file the build reads from one of them did
# not arrive (scripts/ci/checkout_submodules.sh).
- name: Check out build submodules
timeout-minutes: 15
run: bash scripts/ci/checkout_submodules.sh retrace
- name: Set artifact metadata - name: Set artifact metadata
run: | run: |
@@ -82,6 +120,9 @@ jobs:
sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=30 update sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=30 update
sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=30 install -y ccache sudo apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=30 install -y ccache
ccache --version ccache --version
# The restored entry carries the counters of every earlier build; zero them so the
# statistics printed after the build are this run's.
ccache --zero-stats
- name: Setup Android SDK - name: Setup Android SDK
id: netsetup84 id: netsetup84
@@ -183,11 +224,20 @@ jobs:
if: always() if: always()
run: ccache --show-stats run: ccache --show-stats
# Rewrite one rolling entry per job and branch. The upload stays cumulative - it # A build served from the cache stores nothing, so nothing would ever evict the objects
# carries every object restored at the top of this run plus the few TUs that # of older source states. Trim to CCACHE_MAXSIZE, least recently used first, before the
# changed - but Actions cache keys are immutable, so this branch's superseded # entry is saved; the objects this build hit were just touched and stay.
# blob is released first; --ref keeps the delete off other branches' entries. - name: Trim ccache to its size limit
# Running after the build means a failed build leaves the entry untouched. run: |
ccache --cleanup
ccache --show-stats | grep -i 'cache size' || true
du -sh "${CCACHE_DIR}"
# Rewrite one rolling entry per job and branch. The upload carries the objects this
# build used plus what fits under the size limit, but Actions cache keys are
# immutable, so this branch's superseded blob is released first; --ref keeps the
# delete off other branches' entries. Running after the build means a failed build
# leaves the entry untouched.
- name: Release superseded ccache entry - name: Release superseded ccache entry
env: env:
GH_TOKEN: ${{ github.token }} GH_TOKEN: ${{ github.token }}
@@ -251,10 +301,18 @@ jobs:
name: trace case matrix name: trace case matrix
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
env: env:
MOBILEGL_CI_DISAGGREGATED: ${{ (github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && 'ON' || 'OFF' }} MOBILEGL_CI_DISAGGREGATED: ${{ (github.ref_name == 'dev' || github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && 'ON' || 'OFF' }}
# How much replay one emulator job aims for, in seconds as the duration hints count them.
# A (lane, transport, backend) cell is cut into as many shards as that takes, and its
# cases are spread so the shards of a cell come out even (tools/trace_replay/trace_cases.py,
# shard_matrix). A smaller value gives more and shorter jobs, each of which pays the
# emulator boot again; every case is replayed once per cell whatever the value.
RETRACE_SHARD_SECONDS: "580"
# The most a shard runner may spend on its cases. A shard gets the time in which every
# one of its cases could run to the end of its own time budget, up to this cap.
RETRACE_SHARD_DEADLINE_CAP_SECONDS: "7200"
outputs: outputs:
android: ${{ steps.trace-cases.outputs.android }} android: ${{ steps.trace-cases.outputs.android }}
names: ${{ steps.trace-cases.outputs.names }}
steps: steps:
- name: Checkout repo - name: Checkout repo
uses: actions/checkout@v6 uses: actions/checkout@v6
@@ -263,77 +321,109 @@ jobs:
id: trace-cases id: trace-cases
run: | run: |
python3 tools/trace_replay/trace_cases.py --self-test-transport-matrices python3 tools/trace_replay/trace_cases.py --self-test-transport-matrices
matrix_format=github-apk-matrix # The retrace jobs replay shards: several cases of one (lane, transport, backend) cell
# per emulator boot. The self-test proves every shard matrix replays exactly the rows
# of the per-case matrix it is built from, and trace_cases.py refuses to print one
# that does not.
python3 tools/trace_replay/trace_cases.py --self-test-shard-matrices
matrix_format=github-apk-shard-matrix
if [ "${MOBILEGL_CI_DISAGGREGATED}" = "ON" ]; then if [ "${MOBILEGL_CI_DISAGGREGATED}" = "ON" ]; then
matrix_format=github-apk-transport-matrix matrix_format=github-apk-transport-shard-matrix
fi fi
echo "android=$(python3 tools/trace_replay/trace_cases.py --ci --format "${matrix_format}")" >> "$GITHUB_OUTPUT" android="$(python3 tools/trace_replay/trace_cases.py --ci --format "${matrix_format}" \
echo "names=$(python3 tools/trace_replay/trace_cases.py --ci --format names)" >> "$GITHUB_OUTPUT" --shard-seconds "${RETRACE_SHARD_SECONDS}" --android-lanes "${MOBILEGL_ANDROID_LANES}")"
# An empty `include` is not an error to GitHub: it skips the retrace job with no red.
shards="$(printf '%s' "${android}" | python3 -c 'import json, sys; print(len(json.load(sys.stdin)["include"]))')"
if [ "${shards}" -lt 1 ]; then
echo "::error::the Android retrace matrix is empty, so the retrace job would be skipped with no red"
exit 1
fi
# The cases of a shard share one job, so a case that hangs spends time the cases after
# it need. retrace_shard_budget.py gives every shard its own deadline_seconds, the
# time in which each of its cases could run to the end of its time budget, capped so
# that a shard whose replays all hang cannot hold a runner for hours, and the
# timeout_minutes of its job. The retrace job reads both from the matrix.
android="$(printf '%s' "${android}" | python3 android-plugin/retrace_shard_budget.py \
--cap-seconds "${RETRACE_SHARD_DEADLINE_CAP_SECONDS}")"
echo "android=${android}" >> "$GITHUB_OUTPUT"
lanes=""
if [ "${matrix_format}" = "github-apk-transport-shard-matrix" ]; then
lanes=" (${MOBILEGL_ANDROID_LANES} lanes)"
fi
echo "Android retrace: ${shards} shard(s) from ${matrix_format}${lanes} at ${RETRACE_SHARD_SECONDS} s per shard"
# One cache entry holds every Git LFS fixture of the CI cases, and each retrace job restores
# that entry itself. This job only makes sure the entry exists. On a hit it downloads
# nothing. On a miss it starts from the newest older bundle, fetches from the mirror the
# cases whose pointers changed, saves the entry under the new key and confirms it is there,
# so that no retrace job has to download fixtures itself. The key, the path and the steps
# are .github/scripts/trace-fixture-bundle.sh's, and test.yml has the same job for the same
# entry: the two workflows of one push look the key up at the same time, and on a miss both
# fetch and both save. The save that loses is not an error, since the entry it wanted to
# write is the one the confirming step then finds. The timeout bounds a fixture download
# that stalls, which would otherwise hold every retrace job behind it.
trace-fixtures: trace-fixtures:
name: trace fixture (${{ matrix.case }}) name: trace fixtures
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
needs: trace-cases timeout-minutes: 30
strategy: permissions:
fail-fast: false actions: read
max-parallel: 4 contents: read
matrix: outputs:
case: ${{ fromJSON(needs.trace-cases.outputs.names) }} key: ${{ steps.bundle.outputs.key }}
path: ${{ steps.bundle.outputs.path }}
steps: steps:
- name: Checkout repo - name: Checkout repo
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: Derive trace fixture cache key - name: Derive the fixture bundle key
id: fixture-key id: bundle
run: bash .github/scripts/trace-fixture-cache.sh key '${{ matrix.case }}' run: bash .github/scripts/trace-fixture-bundle.sh key
- name: Restore trace fixture cache - name: Look up the fixture bundle
id: fixture-cache id: lookup
if: steps.fixture-key.outputs.cacheable == 'true'
uses: actions/cache/restore@v5 uses: actions/cache/restore@v5
with: with:
path: ${{ steps.fixture-key.outputs.paths }} path: ${{ steps.bundle.outputs.path }}
key: ${{ steps.fixture-key.outputs.key }} key: ${{ steps.bundle.outputs.key }}
lookup-only: true
- name: Verify restored trace fixture - name: Restore the newest older bundle
id: fixture-verify if: steps.lookup.outputs.cache-hit != 'true'
if: steps.fixture-cache.outputs.cache-hit == 'true' uses: actions/cache/restore@v5
run: | with:
if bash .github/scripts/trace-fixture-cache.sh verify '${{ matrix.case }}'; then path: ${{ steps.bundle.outputs.path }}
echo "ok=true" >> "$GITHUB_OUTPUT" key: ${{ steps.bundle.outputs.key }}
else restore-keys: ${{ steps.bundle.outputs.restore-prefix }}
echo "ok=false" >> "$GITHUB_OUTPUT"
echo "::warning::Cached fixture for ${{ matrix.case }} failed verification; falling back to the download path"
bash .github/scripts/trace-fixture-cache.sh reset '${{ matrix.case }}'
fi
- name: Fetch trace fixture - name: Complete the fixture bundle
if: steps.fixture-verify.outputs.ok != 'true' if: steps.lookup.outputs.cache-hit != 'true'
run: bash .github/scripts/fetch-trace-fixture-lfs.sh '${{ matrix.case }}' run: bash .github/scripts/trace-fixture-bundle.sh ensure
- name: Save trace fixture cache - name: Save the fixture bundle
if: steps.fixture-key.outputs.cacheable == 'true' && steps.fixture-cache.outputs.cache-hit != 'true' if: steps.lookup.outputs.cache-hit != 'true'
continue-on-error: true
uses: actions/cache/save@v5 uses: actions/cache/save@v5
with: with:
path: ${{ steps.fixture-key.outputs.paths }} path: ${{ steps.bundle.outputs.path }}
key: ${{ steps.fixture-key.outputs.key }} key: ${{ steps.bundle.outputs.key }}
- name: Stage trace fixture # The Test workflow of the same push may be the one whose save won. Its entry is as good,
run: | # once it is listed.
safe_case="$(printf '%s' '${{ matrix.case }}' | sed 's/[^A-Za-z0-9._-]/_/g')" - name: Wait for the fixture bundle entry
stage_dir="trace-fixtures/${safe_case}" if: steps.lookup.outputs.cache-hit != 'true'
mkdir -p "${stage_dir}" env:
python3 tools/trace_replay/trace_cases.py --format fixture-files --case '${{ matrix.case }}' | GH_TOKEN: ${{ github.token }}
while IFS= read -r file; do run: bash .github/scripts/trace-fixture-bundle.sh wait '${{ steps.bundle.outputs.key }}'
cp "${file}" "${stage_dir}/"
done
- name: Upload trace fixture - name: Confirm the fixture bundle entry
uses: actions/upload-artifact@v7 if: steps.lookup.outputs.cache-hit != 'true'
uses: actions/cache/restore@v5
with: with:
name: MobileGL-trace-fixture-${{ matrix.case }} path: ${{ steps.bundle.outputs.path }}
path: trace-fixtures/** key: ${{ steps.bundle.outputs.key }}
if-no-files-found: error lookup-only: true
fail-on-cache-miss: true
android-avd: android-avd:
name: android avd image name: android avd image
@@ -347,8 +437,33 @@ jobs:
- name: Checkout repo - name: Checkout repo
uses: actions/checkout@v6 uses: actions/checkout@v6
# The emulator shard loop and the artifact cleanup, driven by fakes: what each case is
# run with, a verdict for every case, and what stays after a failure. It runs in this
# job because the retrace jobs wait for it and the APK build does not.
- name: Emulator shard runner regression
run: |
python3 android-plugin/test_trace_replay_shard.py
python3 android-plugin/test_retrace_artifact_cleanup.py
# This job only makes sure the cache entry exists; the retrace jobs restore it. A hit
# downloads nothing and skips every step below; a miss builds the AVD, and the post step
# of this action saves it.
- name: Look up Android AVD cache
id: android-avd-cache
uses: actions/cache@v5
with:
lookup-only: true
path: |
${{ env.ANDROID_AVD_HOME }}
${{ env.ANDROID_SDK_ROOT }}/emulator
${{ env.ANDROID_SDK_ROOT }}/platform-tools
${{ env.ANDROID_SDK_ROOT }}/platforms/android-35
${{ env.ANDROID_SDK_ROOT }}/system-images/android-35/google_apis/x86_64
key: ${{ runner.os }}-mobilegl-avd-api35-google_apis-x86_64-pixel_6-v4-${{ hashFiles('android-plugin/run-avd-ci.sh') }}
- name: Setup Android SDK - name: Setup Android SDK
id: netsetup336 id: netsetup336
if: steps.android-avd-cache.outputs.cache-hit != 'true'
continue-on-error: true continue-on-error: true
timeout-minutes: 15 timeout-minutes: 15
uses: android-actions/setup-android@v4 uses: android-actions/setup-android@v4
@@ -367,20 +482,9 @@ jobs:
cmdline-tools-version: 13114758 cmdline-tools-version: 13114758
- name: Accept Android SDK licenses - name: Accept Android SDK licenses
if: steps.android-avd-cache.outputs.cache-hit != 'true'
run: yes | sdkmanager --licenses >/dev/null run: yes | sdkmanager --licenses >/dev/null
- name: Restore Android AVD cache
id: android-avd-cache
uses: actions/cache@v5
with:
path: |
${{ env.ANDROID_AVD_HOME }}
${{ env.ANDROID_SDK_ROOT }}/emulator
${{ env.ANDROID_SDK_ROOT }}/platform-tools
${{ env.ANDROID_SDK_ROOT }}/platforms/android-35
${{ env.ANDROID_SDK_ROOT }}/system-images/android-35/google_apis/x86_64
key: ${{ runner.os }}-mobilegl-avd-api35-google_apis-x86_64-pixel_6-v4-${{ hashFiles('android-plugin/run-avd-ci.sh') }}
- name: Create AVD - name: Create AVD
if: steps.android-avd-cache.outputs.cache-hit != 'true' if: steps.android-avd-cache.outputs.cache-hit != 'true'
run: | run: |
@@ -418,16 +522,27 @@ jobs:
fi fi
du -sh "${ANDROID_AVD_HOME}" du -sh "${ANDROID_AVD_HOME}"
# One job replays a shard: some of the cases of one lane, transport and backend, one after
# another on one emulator boot (android-plugin/trace_replay_shard.py). The shard's step
# summary has a row per case, and a case that did not pass has an error annotation of its own.
retrace: retrace:
name: retrace (${{ matrix.backend.name }} / ${{ matrix.transport || 'monolith' }}, ${{ matrix.case.name }}) name: retrace (${{ matrix.backend.name }} / ${{ matrix.transport || 'monolith' }}, shard ${{ matrix.shard }}/${{ matrix.shards }})
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
needs: needs:
- build - build
- android-avd - android-avd
- trace-cases - trace-cases
- trace-fixtures - trace-fixtures
if: ${{ !cancelled() && needs.build.result == 'success' && needs.android-avd.result == 'success' && needs.trace-cases.result == 'success' }} if: ${{ !cancelled() && needs.build.result == 'success' && needs.android-avd.result == 'success' && needs.trace-cases.result == 'success' && needs.trace-fixtures.result == 'success' }}
timeout-minutes: 75 # Paired with --deadline-seconds below, and far above what a shard takes when its replays
# end by themselves. Both come from the matrix. The deadline is the smaller of the time in
# which every case of the shard could run to the end of its own time budget
# (trace_replay_shard.py, needed_seconds) and RETRACE_SHARD_DEADLINE_CAP_SECONDS. With
# budgets of up to 1800 s a case the cap is what a shard of several cases gets: about
# three replays can hang to the end of their budgets before a later case is recorded as
# not-run. The minutes between the deadline and this limit are the setup before the
# replays and the uploads after them.
timeout-minutes: ${{ matrix.timeout_minutes }}
env: env:
MOBILEGL_CI_TRANSPORT: ${{ matrix.transport || 'monolith' }} MOBILEGL_CI_TRANSPORT: ${{ matrix.transport || 'monolith' }}
MOBILEGL_CI_LANE: ${{ matrix.lane || 'monolith-control' }} MOBILEGL_CI_LANE: ${{ matrix.lane || 'monolith-control' }}
@@ -435,39 +550,60 @@ jobs:
ANDROID_AVD_HOME: ${{ github.workspace }}/.android/avd ANDROID_AVD_HOME: ${{ github.workspace }}/.android/avd
ANDROID_HOME: ${{ github.workspace }}/.android/sdk ANDROID_HOME: ${{ github.workspace }}/.android/sdk
ANDROID_SDK_ROOT: ${{ github.workspace }}/.android/sdk ANDROID_SDK_ROOT: ${{ github.workspace }}/.android/sdk
# The shard's cases in run order, as the JSON array of case objects the shard runner
# takes: every field of the case in trace_cases.json, with the fixture paths in place.
MOBILEGL_SHARD_CASES: ${{ toJSON(matrix.cases) }}
# Set the repository variable to 1 to cold boot the emulator before every case of a
# shard, should one boot not survive a whole shard of software-rendered replays.
MOBILEGL_SHARD_RESTART_EMULATOR_EACH_CASE: ${{ vars.MOBILEGL_SHARD_RESTART_EMULATOR_EACH_CASE || '0' }}
# The emulator renders on the host with a software rasterizer that can keep compiled
# shaders on disk in the runner's home directory, where neither clearing the app's data
# nor restarting the emulator reaches. Off, so every case of a shard compiles its
# shaders as a job with a fresh machine does.
MESA_SHADER_CACHE_DISABLE: "true"
strategy: strategy:
fail-fast: false fail-fast: false
# The account runs at most 20 jobs at once, shared with Test. These ~230 emulator jobs are # No max-parallel: the account's limit of 20 concurrent jobs, shared with Test, already
# the longer of the two runs, so they may take the whole pool: what is minimized is when the # bounds this matrix, and a cap below the number of shards only makes the round longer.
# later of APK and Test finishes, not Test alone.
max-parallel: 20
matrix: ${{ fromJSON(needs.trace-cases.outputs.android) }} matrix: ${{ fromJSON(needs.trace-cases.outputs.android) }}
steps: steps:
- name: Set swap space
uses: pierotofy/set-swap-space@v1.0
with:
swap-size-gb: 8
- name: Checkout repo - name: Checkout repo
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: Download trace fixture # First, so the steps that run whatever happened (diagnostics, stopping the emulator, the
uses: actions/download-artifact@v8 # result upload) have these even when a step before the emulator launch failed.
with:
name: MobileGL-trace-fixture-${{ matrix.case.name }}
path: trace-fixture-download
- name: Install trace fixture
run: |
mkdir -p tools/trace_replay/fixtures
find trace-fixture-download -type f -exec cp {} tools/trace_replay/fixtures/ \;
- name: Set artifact metadata - name: Set artifact metadata
run: | run: |
echo "date_today=$(date +'%Y-%m-%d')" >> "$GITHUB_ENV" echo "date_today=$(date +'%Y-%m-%d')" >> "$GITHUB_ENV"
echo "EMULATOR_LOG=${RUNNER_TEMP}/mobilegl-emulator.log" >> "$GITHUB_ENV" echo "EMULATOR_LOG=${RUNNER_TEMP}/mobilegl-emulator.log" >> "$GITHUB_ENV"
echo "EMULATOR_PID_FILE=${RUNNER_TEMP}/mobilegl-emulator.pid" >> "$GITHUB_ENV" echo "EMULATOR_PID_FILE=${RUNNER_TEMP}/mobilegl-emulator.pid" >> "$GITHUB_ENV"
# The swap file, the fixture bundle and the AVD image share the runner's disk, and a
# runner that starts short of space fails the whole shard. The two reports say how much
# was free before and after them.
- name: Report free disk space
run: df -h / "${GITHUB_WORKSPACE}"
- name: Set swap space
uses: ./.github/actions/set-swap-space
with:
swap-size-gb: 8
# A miss does not fail the job: the entry may have been evicted since trace-fixtures
# confirmed it (a re-run of failed jobs on an older run does not run that job again).
- name: Restore trace fixtures
uses: actions/cache/restore@v5
with:
path: ${{ needs.trace-fixtures.outputs.path }}
key: ${{ needs.trace-fixtures.outputs.key }}
# Moves this shard's fixtures into place and checks them against their Git LFS pointers.
# A case the restored bundle does not hold is downloaded here, with a warning annotation;
# the timeout bounds a download that stalls.
- name: Install trace fixtures
timeout-minutes: 15
run: bash .github/scripts/trace-fixture-bundle.sh install ${{ join(matrix.names, ' ') }}
- name: Restore Android AVD cache - name: Restore Android AVD cache
id: android-avd-cache id: android-avd-cache
uses: actions/cache/restore@v5 uses: actions/cache/restore@v5
@@ -480,6 +616,9 @@ jobs:
${{ env.ANDROID_SDK_ROOT }}/system-images/android-35/google_apis/x86_64 ${{ env.ANDROID_SDK_ROOT }}/system-images/android-35/google_apis/x86_64
key: ${{ runner.os }}-mobilegl-avd-api35-google_apis-x86_64-pixel_6-v4-${{ hashFiles('android-plugin/run-avd-ci.sh') }} key: ${{ runner.os }}-mobilegl-avd-api35-google_apis-x86_64-pixel_6-v4-${{ hashFiles('android-plugin/run-avd-ci.sh') }}
- name: Report free disk space after the restores
run: df -h / "${GITHUB_WORKSPACE}"
# The cache holds everything a retrace job runs (emulator, adb, image, AVD); the SDK manager # The cache holds everything a retrace job runs (emulator, adb, image, AVD); the SDK manager
# is only needed to create the AVD when the cache missed. # is only needed to create the AVD when the cache missed.
- name: Use the cached SDK tools - name: Use the cached SDK tools
@@ -543,135 +682,44 @@ jobs:
--pid-file "${EMULATOR_PID_FILE}" \ --pid-file "${EMULATOR_PID_FILE}" \
--boot-timeout 300 --boot-timeout 300
# One emulator boot replays every case of the shard in order, each in a fresh app process
# with cleared app data and with the settings its case, backend and transport need. The
# runner keeps going after a failing case, on a cold boot so that case cannot touch the
# next one's verdict, retries a case once on a restarted emulator when the emulator
# itself failed, and exits non-zero when a case failed or a verdict is missing. The
# deadline stops it while the steps below can still upload what it has: the job timeout
# less the setup above and the uploads below. A case it stops or never starts is
# recorded as not-run, which is red.
- name: Retrace and validate - name: Retrace and validate
env: env:
MOBILEGL_ESPRYT_USE_ANGLE: ${{ matrix.backend.name == 'DirectGLES' && '1' || '0' }} # Lets the artifact cleanup tell which job a shard's verdicts came from.
MOBILEGL_TRACE_ANGLE_VARIANT: ${{ matrix.case.name == 'minecraft-1.21.4-fabric-iris-bliss-in-world' && '90a62123d794' || 'ec889e6ea831' }} MOBILEGL_SHARD_JOB_ID: ${{ job.check_run_id }}
MOBILEGL_MAGMA_R11G11B10F_FALLBACK: ${{ matrix.backend.name == 'DirectVulkan' && '1' || '0' }}
MOBILEGL_ITRP_FIX: ${{ matrix.case.name == 'minecraft-1.21.4-fabric-iris-iterationrp-in-world' && 'ForceOn' || 'Auto' }}
run: | run: |
apk_file="android-retrace-apks/MobileGL-plugin-trace-release-${GITHUB_SHA}.apk" sh android-plugin/trace-replay-shard.sh \
test -f "${apk_file}" --apk-file "android-retrace-apks/MobileGL-plugin-trace-release-${GITHUB_SHA}.apk" \
extra_retrace_args=() --backend "${{ matrix.backend.name }}" \
if [ "${MOBILEGL_CI_TRANSPORT}" = "inproc" ]; then --gpu "${{ matrix.backend.gpu }}" \
extra_retrace_args+=( --transport "${MOBILEGL_CI_TRANSPORT}" \
--env "MOBILEGL_TRANSPORT=inproc;MOBILEGL_IPC_ROLE_SPLIT_STATE=1;MOBILEGL_IPC_STRICT_ERRORS=1;MOBILEGL_IPC_RUN_AHEAD=1;MOBILEGL_GPU_HANG_BUDGET_MS=${MOBILEGL_GPU_HANG_BUDGET_MS}" --lane "${MOBILEGL_CI_LANE}" \
--require-inproc --cases-env MOBILEGL_SHARD_CASES \
) --deadline-seconds "${{ matrix.deadline_seconds }}"
elif [ "${MOBILEGL_CI_TRANSPORT}" = "spawn" ]; then
# THE SAME CLIENT KNOBS AS inproc, so the two acceptance lanes differ in exactly one
# thing: which PROCESS the server role runs in. They are scrubbed out of the CHILD's
# environment by the launcher and read by the client, which is where they belong.
#
# MOBILEGL_IPC_SERVER_PATH IS DELIBERATELY ABSENT. nativeLibraryDir carries an
# install-time hash, so nothing off the device can spell it; TraceReplayActivity
# resolves it from getApplicationInfo() when the env asks for spawn, and an explicit
# value would override that. --use-pbuffer is NOT optional here: a spawned
# server cannot be handed the client's window, because an ANativeWindow* is a pointer
# into the CLIENT's process and SetWindowHandle is refused on the way across with
# Fatal{UnmigratedSurface, "AndroidNativeWindow:use-MOBILEGL_IPC_SURFACE=server"} (Rule H, permanent;
# a windowed client uses MOBILEGL_IPC_SURFACE=server instead).
extra_retrace_args+=(
--env "MOBILEGL_TRANSPORT=spawn;MOBILEGL_IPC_ROLE_SPLIT_STATE=1;MOBILEGL_IPC_STRICT_ERRORS=1;MOBILEGL_IPC_RUN_AHEAD=1;MOBILEGL_GPU_HANG_BUDGET_MS=${MOBILEGL_GPU_HANG_BUDGET_MS}"
--use-pbuffer
--require-spawn
)
else
extra_retrace_args+=(--env "MOBILEGL_TRANSPORT=monolith;MOBILEGL_GPU_HANG_BUDGET_MS=${MOBILEGL_GPU_HANG_BUDGET_MS}")
fi
# Bliss needs the newer signed ANGLE variant plus sampler mipmap
# min-filter downgrading on ANGLE llvmpipe.
if [ "${{ matrix.backend.name }}" = "DirectGLES" ] && [ "${{ matrix.case.name }}" = "minecraft-1.21.4-fabric-iris-bliss-in-world" ]; then
extra_retrace_args+=(--avoid-angle-llvmpipe-sampler-mipmap-min-filter)
fi
if [ "${{ matrix.backend.name }}" = "DirectGLES" ] && [ "${{ matrix.case.avoid_angle_llvmpipe_explicit_lod_bias || false }}" = "true" ]; then
extra_retrace_args+=(--avoid-angle-llvmpipe-explicit-lod-bias)
fi
if [ "${{ matrix.case.coherent_as_flush || false }}" = "true" ]; then
extra_retrace_args+=(--coherent-as-flush)
fi
run_retrace() {
timeout "$(( ${{ matrix.case.timeout_seconds }} + 300 ))" sh android-plugin/trace-replay-ci.sh \
--apk-file "${apk_file}" \
--package top.mobilegl.plugin.trace \
--backend "${{ matrix.backend.name }}" \
--result-root android-retrace-result \
--fixture-root android-retrace-fixture \
--case "${{ matrix.case.name }}" \
--trace-archive "${{ matrix.case.trace_archive }}" \
--trace-file "${{ matrix.case.trace_file }}" \
--golden "${{ matrix.case.golden }}" \
--alternate-golden "${{ matrix.case.alternate_golden || '' }}" \
--target-call "${{ matrix.case.target_call }}" \
--width "${{ matrix.case.width }}" \
--height "${{ matrix.case.height }}" \
--ssim-threshold "${{ matrix.case.ssim_threshold || '0.99' }}" \
--crop-x "${{ matrix.case.crop_x }}" \
--crop-y "${{ matrix.case.crop_y }}" \
--crop-width "${{ matrix.case.crop_width }}" \
--crop-height "${{ matrix.case.crop_height }}" \
--timeout-seconds "${{ matrix.case.timeout_seconds }}" \
"${extra_retrace_args[@]}"
}
retrace_status=0
run_retrace || retrace_status=$?
if [ "${retrace_status}" -eq 75 ]; then
echo "::warning::Android emulator infrastructure failed; restarting it and retrying this retrace once."
# Surface-lost is retried rather than failed, so it would otherwise
# be invisible. Report it per job - a healthy run prints nothing and
# a rate spike shows up as a row per affected case.
reason_file="android-retrace-result/infrastructure-failure-reason.txt"
surface_lost_retries=0
if [ -f "${reason_file}" ]; then
surface_lost_retries="$(grep -c 'angle-surface-lost' "${reason_file}" || true)"
fi
if [ "${surface_lost_retries}" -gt 0 ]; then
echo "surface-lost retries: ${surface_lost_retries} (${{ matrix.backend.name }}, ${{ matrix.case.name }})" \
>> "${GITHUB_STEP_SUMMARY}"
fi
# The restart truncates EMULATOR_LOG, and the attempt that lost the
# emulator is the one worth reading - the retry usually only shows
# the wreckage. Keep the first attempt's log before it is clobbered.
if [ -f "${EMULATOR_LOG}" ]; then
cp "${EMULATOR_LOG}" "${EMULATOR_LOG}.first-attempt" || true
fi
sh android-plugin/run-avd-ci.sh stop \
--avd-name "${AVD_NAME}" \
--emulator-log "${EMULATOR_LOG}" \
--pid-file "${EMULATOR_PID_FILE}"
adb kill-server || true
sleep 2
sh android-plugin/run-avd-ci.sh start \
--avd-name "${AVD_NAME}" \
--gpu "${{ matrix.backend.gpu }}" \
--emulator-log "${EMULATOR_LOG}" \
--pid-file "${EMULATOR_PID_FILE}" \
--boot-timeout 300
run_retrace
elif [ "${retrace_status}" -ne 0 ]; then
exit "${retrace_status}"
fi
# Runs whatever happened above, without the device: every case of the shard gets its
# lane.json and golden copies for the summary, and a not-run verdict when the runner was
# killed or never started.
- name: Collect retrace summary inputs - name: Collect retrace summary inputs
if: always() if: always()
env:
MOBILEGL_SHARD_JOB_ID: ${{ job.check_run_id }}
run: | run: |
safe_case="$(printf '%s' '${{ matrix.case.name }}' | sed 's/[^A-Za-z0-9._-]/_/g')" sh android-plugin/trace-replay-shard.sh \
result_dir="android-retrace-result/${safe_case}-${{ matrix.backend.name }}" --apk-file "android-retrace-apks/MobileGL-plugin-trace-release-${GITHUB_SHA}.apk" \
mkdir -p "${result_dir}" --backend "${{ matrix.backend.name }}" \
python3 - "${result_dir}/lane.json" <<'PY' --gpu "${{ matrix.backend.gpu }}" \
import json, os, sys --transport "${MOBILEGL_CI_TRANSPORT}" \
from pathlib import Path --lane "${MOBILEGL_CI_LANE}" \
Path(sys.argv[1]).write_text(json.dumps({"lane": os.environ["MOBILEGL_CI_LANE"], --cases-env MOBILEGL_SHARD_CASES \
"transport": os.environ["MOBILEGL_CI_TRANSPORT"]}, indent=2)) --finalize
PY
if [ -s "${{ matrix.case.golden }}" ]; then
cp "${{ matrix.case.golden }}" "${result_dir}/${safe_case}-${{ matrix.backend.name }}-golden.png"
fi
if [ -n "${{ matrix.case.alternate_golden || '' }}" ] && [ -s "${{ matrix.case.alternate_golden || '' }}" ]; then
cp "${{ matrix.case.alternate_golden || '' }}" "${result_dir}/${safe_case}-${{ matrix.backend.name }}-alternate-golden.png"
fi
- name: Collect emulator diagnostics - name: Collect emulator diagnostics
if: always() if: always()
@@ -679,12 +727,18 @@ jobs:
mkdir -p android-retrace-result/diagnostics mkdir -p android-retrace-result/diagnostics
adb devices -l > android-retrace-result/diagnostics/adb-devices.txt || true adb devices -l > android-retrace-result/diagnostics/adb-devices.txt || true
timeout 30 adb logcat -d -t 1000 > android-retrace-result/diagnostics/logcat.txt || true timeout 30 adb logcat -d -t 1000 > android-retrace-result/diagnostics/logcat.txt || true
# What the guest was given; replay time inside the emulator depends on it.
timeout 30 adb shell 'nproc; grep -E "MemTotal|SwapTotal" /proc/meminfo' \
> android-retrace-result/diagnostics/guest-resources.txt 2>&1 || true
cp "${ANDROID_AVD_HOME}/${AVD_NAME}.avd/config.ini" android-retrace-result/diagnostics/avd-config.ini 2>/dev/null || true
# Expected to list nothing: the host rasterizer's on-disk shader cache is disabled.
du -sh "${XDG_CACHE_HOME:-${HOME}/.cache}"/mesa_shader_cache* \
> android-retrace-result/diagnostics/host-shader-cache.txt 2>&1 || true
# The log of the last boot. The shard runner keeps the log of every boot it ended
# as diagnostics/emulator-boot-<n>-*.log before a restart truncates it.
if [ -f "${EMULATOR_LOG}" ]; then if [ -f "${EMULATOR_LOG}" ]; then
cp "${EMULATOR_LOG}" android-retrace-result/diagnostics/emulator.log cp "${EMULATOR_LOG}" android-retrace-result/diagnostics/emulator.log
fi fi
if [ -f "${EMULATOR_LOG}.first-attempt" ]; then
cp "${EMULATOR_LOG}.first-attempt" android-retrace-result/diagnostics/emulator-first-attempt.log
fi
# A vanished emulator looks identical whether the host OOM killer took # A vanished emulator looks identical whether the host OOM killer took
# qemu or the renderer faulted. These two say which. # qemu or the renderer faulted. These two say which.
free -h > android-retrace-result/diagnostics/host-memory.txt 2>&1 || true free -h > android-retrace-result/diagnostics/host-memory.txt 2>&1 || true
@@ -704,9 +758,14 @@ jobs:
continue-on-error: true continue-on-error: true
uses: actions/upload-artifact@v7 uses: actions/upload-artifact@v7
with: with:
name: MobileGL-android-retrace-result-${{ matrix.lane || 'monolith-control' }}-${{ matrix.transport || 'monolith' }}-${{ env.date_today }}-${{ github.sha }}-${{ matrix.backend.name }}-${{ matrix.case.name }} # The summary job downloads by the lane and transport at the front of this name, and
# its renderer reads the backend and the case from the paths below it, so the name
# ends in the shard number and holds no case name.
name: MobileGL-android-retrace-result-${{ matrix.lane || 'monolith-control' }}-${{ matrix.transport || 'monolith' }}-${{ env.date_today }}-${{ github.sha }}-${{ matrix.backend.name }}-shard${{ matrix.shard }}
path: android-retrace-result/** path: android-retrace-result/**
if-no-files-found: warn if-no-files-found: warn
# A re-run of a failed job replaces the artifact its first attempt kept.
overwrite: true
# The artifact service sometimes times out (ETIMEDOUT on CreateArtifact, one job in 231 on # The artifact service sometimes times out (ETIMEDOUT on CreateArtifact, one job in 231 on
# 2026-10-18), which turned a passing case red; one delayed retry absorbs that. # 2026-10-18), which turned a passing case red; one delayed retry absorbs that.
@@ -718,7 +777,7 @@ jobs:
if: always() && steps.upload-retrace-result.outcome == 'failure' if: always() && steps.upload-retrace-result.outcome == 'failure'
uses: actions/upload-artifact@v7 uses: actions/upload-artifact@v7
with: with:
name: MobileGL-android-retrace-result-${{ matrix.lane || 'monolith-control' }}-${{ matrix.transport || 'monolith' }}-${{ env.date_today }}-${{ github.sha }}-${{ matrix.backend.name }}-${{ matrix.case.name }} name: MobileGL-android-retrace-result-${{ matrix.lane || 'monolith-control' }}-${{ matrix.transport || 'monolith' }}-${{ env.date_today }}-${{ github.sha }}-${{ matrix.backend.name }}-shard${{ matrix.shard }}
path: android-retrace-result/** path: android-retrace-result/**
if-no-files-found: warn if-no-files-found: warn
overwrite: true overwrite: true
@@ -727,46 +786,67 @@ jobs:
name: retrace summary name: retrace summary
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
needs: retrace needs: retrace
if: ${{ !cancelled() }} # Also runs for a canceled run, where it only downloads the results and deletes the
# intermediate artifacts; the rendering steps are skipped then. The checkout and the
# downloads say so themselves, because a step with no condition is skipped once its job is
# canceled.
if: always()
permissions:
actions: write
contents: read
env: env:
MOBILEGL_CI_DISAGGREGATED: ${{ (github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && 'ON' || 'OFF' }} MOBILEGL_CI_DISAGGREGATED: ${{ (github.ref_name == 'dev' || github.ref_name == 'feat/disaggregated' || github.ref_name == 'codex/fix-github-ci' || github.ref_name == 'cleanup') && 'ON' || 'OFF' }}
steps: steps:
- name: Checkout repo - name: Checkout repo
if: ${{ success() || cancelled() }}
uses: actions/checkout@v6 uses: actions/checkout@v6
- name: Set artifact metadata
run: |
echo "date_today=$(date +'%Y-%m-%d')" >> "$GITHUB_ENV"
- name: Set up Node.js - name: Set up Node.js
if: ${{ !cancelled() }}
uses: actions/setup-node@v7 uses: actions/setup-node@v7
with: with:
node-version: '22' node-version: '22'
- name: Download Android monolith control results - name: Download Android monolith control results
if: ${{ success() || cancelled() }}
uses: actions/download-artifact@v8 uses: actions/download-artifact@v8
with: with:
pattern: MobileGL-android-retrace-result-monolith-control-monolith-* pattern: MobileGL-android-retrace-result-monolith-control-monolith-*
path: retrace-artifacts/monolith-control path: retrace-artifacts/monolith-control
- name: Download Android inproc acceptance results - name: Download Android inproc acceptance results
if: env.MOBILEGL_CI_DISAGGREGATED == 'ON' if: ${{ (success() || cancelled()) && env.MOBILEGL_CI_DISAGGREGATED == 'ON' }}
uses: actions/download-artifact@v8 uses: actions/download-artifact@v8
with: with:
pattern: MobileGL-android-retrace-result-inproc-acceptance-inproc-* pattern: MobileGL-android-retrace-result-inproc-acceptance-inproc-*
path: retrace-artifacts/inproc-acceptance path: retrace-artifacts/inproc-acceptance
- name: Download Android spawn acceptance results - name: Download Android spawn acceptance results
if: env.MOBILEGL_CI_DISAGGREGATED == 'ON' if: ${{ (success() || cancelled()) && env.MOBILEGL_CI_DISAGGREGATED == 'ON' }}
uses: actions/download-artifact@v8 uses: actions/download-artifact@v8
with: with:
pattern: MobileGL-android-retrace-result-spawn-acceptance-spawn-* pattern: MobileGL-android-retrace-result-spawn-acceptance-spawn-*
path: retrace-artifacts/spawn-acceptance path: retrace-artifacts/spawn-acceptance
# The reduced monolith and spawn lanes replay the android_smoke cases only, so their
# summaries list those cases and no others. A branch outside the split shape replays
# every case on monolith, its only lane.
- name: Select the cases of the reduced lanes
if: ${{ !cancelled() }}
run: |
reduced_lane_cases=""
if [ "${MOBILEGL_CI_DISAGGREGATED}" = "ON" ] && [ "${MOBILEGL_ANDROID_LANES}" = "inproc-full" ]; then
reduced_lane_cases="$(python3 tools/trace_replay/trace_cases.py --format android-smoke-names)"
test -n "${reduced_lane_cases}"
fi
echo "REDUCED_LANE_CASES=${reduced_lane_cases}" >> "$GITHUB_ENV"
- name: Render monolith control summary - name: Render monolith control summary
if: ${{ !cancelled() }}
run: | run: |
node tools/trace_replay/render_retrace_summary.mjs \ node tools/trace_replay/render_retrace_summary.mjs \
--cases "${REDUCED_LANE_CASES}" \
--input retrace-artifacts/monolith-control \ --input retrace-artifacts/monolith-control \
--output-dir android-retrace-summary/monolith-control \ --output-dir android-retrace-summary/monolith-control \
--title "MobileGL Android monolith control" \ --title "MobileGL Android monolith control" \
@@ -774,7 +854,7 @@ jobs:
--html mobilegl-android-monolith-control.html --html mobilegl-android-monolith-control.html
- name: Render inproc acceptance summary - name: Render inproc acceptance summary
if: env.MOBILEGL_CI_DISAGGREGATED == 'ON' if: ${{ !cancelled() && env.MOBILEGL_CI_DISAGGREGATED == 'ON' }}
run: | run: |
node tools/trace_replay/render_retrace_summary.mjs \ node tools/trace_replay/render_retrace_summary.mjs \
--input retrace-artifacts/inproc-acceptance \ --input retrace-artifacts/inproc-acceptance \
@@ -784,11 +864,18 @@ jobs:
--html mobilegl-android-inproc-acceptance.html --html mobilegl-android-inproc-acceptance.html
- name: Render spawn acceptance summary - name: Render spawn acceptance summary
if: env.MOBILEGL_CI_DISAGGREGATED == 'ON' if: ${{ !cancelled() && env.MOBILEGL_CI_DISAGGREGATED == 'ON' }}
run: | run: |
node tools/trace_replay/render_retrace_summary.mjs --input retrace-artifacts/spawn-acceptance --output-dir android-retrace-summary/spawn-acceptance --title "MobileGL Android spawn acceptance" --group-label "Android Emulator / spawn acceptance" --html mobilegl-android-spawn-acceptance.html node tools/trace_replay/render_retrace_summary.mjs \
--cases "${REDUCED_LANE_CASES}" \
--input retrace-artifacts/spawn-acceptance \
--output-dir android-retrace-summary/spawn-acceptance \
--title "MobileGL Android spawn acceptance" \
--group-label "Android Emulator / spawn acceptance" \
--html mobilegl-android-spawn-acceptance.html
- name: Upload Android monolith control summary - name: Upload Android monolith control summary
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7 uses: actions/upload-artifact@v7
with: with:
name: MobileGL-android-monolith-control-summary-${{ github.sha }} name: MobileGL-android-monolith-control-summary-${{ github.sha }}
@@ -797,7 +884,7 @@ jobs:
if-no-files-found: error if-no-files-found: error
- name: Upload Android inproc acceptance summary - name: Upload Android inproc acceptance summary
if: env.MOBILEGL_CI_DISAGGREGATED == 'ON' if: ${{ !cancelled() && env.MOBILEGL_CI_DISAGGREGATED == 'ON' }}
uses: actions/upload-artifact@v7 uses: actions/upload-artifact@v7
with: with:
name: MobileGL-android-inproc-acceptance-summary-${{ github.sha }} name: MobileGL-android-inproc-acceptance-summary-${{ github.sha }}
@@ -806,7 +893,7 @@ jobs:
if-no-files-found: error if-no-files-found: error
- name: Upload Android spawn acceptance summary - name: Upload Android spawn acceptance summary
if: env.MOBILEGL_CI_DISAGGREGATED == 'ON' if: ${{ !cancelled() && env.MOBILEGL_CI_DISAGGREGATED == 'ON' }}
uses: actions/upload-artifact@v7 uses: actions/upload-artifact@v7
with: with:
name: MobileGL-android-spawn-acceptance-summary-${{ github.sha }} name: MobileGL-android-spawn-acceptance-summary-${{ github.sha }}
@@ -814,74 +901,23 @@ jobs:
archive: false archive: false
if-no-files-found: error if-no-files-found: error
remove-artifact-clutter: # The summaries embed every image they show, so the per-shard results are intermediates
name: remove artifact clutter # from here on. This is the last step of the workflow rather than a job of its own, which
runs-on: ubuntu-26.04 # would wait for a runner and start from an empty machine. It keeps what a failure needs,
needs: retrace-summary # read from the shard-status.json of the results downloaded above: the results of every
if: always() # shard that ran a case which did not pass somewhere, on all backends and transports, and
permissions: # every result when a job failed without saying which cases it ran. Deletes are paced
actions: write # below GitHub's limit for mutating requests, and a refused delete is a warning. Skipped
steps: # when a summary step failed, so that re-running this job still finds the results, and
# when a retrace job did not succeed in a run that was not canceled: every result stays
# then, so a shard stopped by its job limit keeps its diagnostics and a re-run of the
# failed jobs renders complete summaries.
- name: Delete intermediate Android retrace artifacts - name: Delete intermediate Android retrace artifacts
if: ${{ (success() && needs.retrace.result == 'success') || cancelled() }}
env: env:
GH_TOKEN: ${{ github.token }} GH_TOKEN: ${{ github.token }}
run: | run: |
# Fail before deletion if either paginated inventory cannot be read. python3 android-plugin/retrace_artifact_cleanup.py \
# 50 rows per page: the jobs endpoint answers HTTP 502 for 90+ rows on a re-run attempt of a run this size. --downloads retrace-artifacts/monolith-control \
gh api --paginate "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/jobs?per_page=50" \ --downloads retrace-artifacts/inproc-acceptance \
--jq '.jobs[] | select(.name | startswith("retrace (")) | select(.conclusion == "failure" or .conclusion == "canceled" or .conclusion == "timed_out" or .conclusion == "action_required") | .name' \ --downloads retrace-artifacts/spawn-acceptance
> "${RUNNER_TEMP}/cleanup-jobs.txt"
gh api --paginate "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/artifacts?per_page=100" \
--jq '.artifacts[] | select(.name | startswith("MobileGL-trace-fixture-") or startswith("MobileGL-android-retrace-result-") or startswith("trace-fixture-") or startswith("retrace-result-")) | [.id, .name] | @tsv' \
> "${RUNNER_TEMP}/cleanup-artifacts.txt"
declare -A failed_cases=()
while IFS= read -r job_name; do
case_name="${job_name#retrace (*, }"
case_name="${case_name%)}"
failed_cases["${case_name}"]=1
done < "${RUNNER_TEMP}/cleanup-jobs.txt"
if ((${#failed_cases[@]})); then
echo "Retaining fixtures and results for failed retrace case(s):"
printf ' %s\n' "${!failed_cases[@]}"
else
echo "All retrace jobs succeeded; nothing needs to be retained."
fi
deleted=0
retained=0
while IFS=$'\t' read -r artifact_id artifact_name; do
keep=0
if [[ "${artifact_name}" == MobileGL-trace-fixture-* ]]; then
case_name="${artifact_name#MobileGL-trace-fixture-}"
if [[ -v "failed_cases[${case_name}]" ]]; then
keep=1
fi
elif [[ "${artifact_name}" == MobileGL-android-retrace-result-* ]]; then
# The result artifact carries mobilegl.log, retrace.log, logcat,
# the emulator log and the actual/diff images - the only record of
# why a retrace failed. Its name ends in -<backend>-<case>, so a
# suffix match on the case name keeps both backends' results for a
# case that failed on either of them, which is what a comparison
# needs. The match is anchored at the end, so a case name that is a
# prefix of a longer one does not retain the longer one's results.
for case_name in "${!failed_cases[@]}"; do
if [[ "${artifact_name}" == *-"${case_name}" ]]; then
keep=1
break
fi
done
fi
if ((keep)); then
echo "Retaining ${artifact_name} (${artifact_id}) for failed retrace."
((retained += 1))
continue
fi
echo "Deleting ${artifact_name} (${artifact_id})"
gh api --method DELETE "repos/${GITHUB_REPOSITORY}/actions/artifacts/${artifact_id}"
((deleted += 1))
done < "${RUNNER_TEMP}/cleanup-artifacts.txt"
echo "Deleted ${deleted} intermediate Android artifact(s); retained ${retained} failed-retrace fixture(s)."
+2 -1
View File
@@ -18,7 +18,8 @@ jobs:
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
timeout-minutes: 15 timeout-minutes: 15
env: env:
TEXT_HYGIENE_BASE_BRANCH: feat/disaggregated # A new branch is checked for the commits it adds to the default branch.
TEXT_HYGIENE_BASE_BRANCH: ${{ github.event.repository.default_branch }}
steps: steps:
- name: Checkout repo - name: Checkout repo
uses: actions/checkout@v6 uses: actions/checkout@v6
+551 -522
View File
File diff suppressed because it is too large Load Diff
+2
View File
@@ -29,3 +29,5 @@ __pycache__/
*.py[cod] *.py[cod]
/.gradle /.gradle
/.trace-work/ /.trace-work/
# The restored trace fixture cache entry (.github/scripts/trace-fixture-bundle.sh)
/trace-fixture-bundle/
+2
View File
@@ -444,6 +444,7 @@ set(SOURCE_FILES
MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/LegalizeResourceArrayIndexPass.cpp MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/LegalizeResourceArrayIndexPass.cpp
MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/FlattenAtomicCounterBlockPass.cpp MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/FlattenAtomicCounterBlockPass.cpp
MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/DemotePointSizePass.cpp MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/DemotePointSizePass.cpp
MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/EmulateClipDistancePass.cpp
MobileGL/MG_Util/BackendLoaders/OpenGL/Loader.cpp MobileGL/MG_Util/BackendLoaders/OpenGL/Loader.cpp
MobileGL/MG_Util/BackendLoaders/Vulkan/Loader.cpp MobileGL/MG_Util/BackendLoaders/Vulkan/Loader.cpp
@@ -451,6 +452,7 @@ set(SOURCE_FILES
MobileGL/MG_Util/SelfTest/DriverBugProbes.cpp MobileGL/MG_Util/SelfTest/DriverBugProbes.cpp
MobileGL/MG_Util/SelfTest/PersistentBufferOrderingProbe.cpp MobileGL/MG_Util/SelfTest/PersistentBufferOrderingProbe.cpp
MobileGL/MG_Util/SelfTest/DriverPost.cpp MobileGL/MG_Util/SelfTest/DriverPost.cpp
MobileGL/MG_Util/SelfTest/DriverPostClipDistance.cpp
MobileGL/MG_Util/SelfTest/DriverPostIterationRPWitness.cpp MobileGL/MG_Util/SelfTest/DriverPostIterationRPWitness.cpp
MobileGL/MG_Util/SelfTest/PrimitivesGeneratedNoXfbProbe.cpp MobileGL/MG_Util/SelfTest/PrimitivesGeneratedNoXfbProbe.cpp
+1 -1
View File
@@ -81,7 +81,7 @@ Example:
## Checks ## Checks
`python3 scripts/check_text_hygiene.py ci` runs what CI runs. It checks the commits on the branch's own line of history since the check was added. The base branch it compares with is `feat/disaggregated`, which you can change with `TEXT_HYGIENE_BASE_BRANCH`. `python3 scripts/check_text_hygiene.py ci` runs what CI runs. It checks the commits on the branch's own line of history since the check was added. A push that moves a branch forward checks the commits it adds. A new branch, a force push and a run that is not a push check the commits the branch adds to the base branch, which is the repository's default branch `dev` (CI reads it from the event). Set `TEXT_HYGIENE_BASE_BRANCH` to compare with another branch. Another branch that adds nothing to the base branch has only its tree checked, and the base branch at its own tip checks its last commit.
These fail on any violation: These fail on any violation:
+27 -10
View File
@@ -214,16 +214,16 @@ namespace MobileGL {
Int MaxComputeImageUniforms = 8; Int MaxComputeImageUniforms = 8;
Int MaxDrawBuffers = 8; Int MaxDrawBuffers = 8;
Int MaxColorAttachments = 8; Int MaxColorAttachments = 8;
// GL_MAX_CLIP_DISTANCES. Zero is a legal answer here, not a placeholder, and a // GL_MAX_CLIP_DISTANCES. What the backend can actually CLIP with, natively or through
// backend that cannot host a clip distance MUST report it: advertising eight the // EmulatesClipDistance below - never a number the backend would then refuse:
// backend will refuse does not make gl_ClipDistance work, it only moves the failure // advertising eight that no stage can honor only moves the failure from query time to
// from an honest "unsupported" at query time to a backend shader-compile error the // a backend shader-compile error the frontend never surfaces. DirectGLES fills it from
// frontend never surfaces, after which every draw with that program silently renders // GL_EXT_clip_cull_distance, DirectVulkan from the shaderClipDistance device feature,
// nothing. DirectGLES fills it from GL_EXT_clip_cull_distance, DirectVulkan from the // and both answer 8 when they emulate. Zero stays a legal answer for a backend that
// shaderClipDistance device feature. The DEFAULT stays at the GL 4.3 core minimum // neither hosts nor emulates the built-in (MOBILEGL_CLIP_DISTANCE_EMULATION=0 on such
// because it describes the no-backend case (standalone shader compiles, unit tests), // a device). The DEFAULT is the GL 4.3 core minimum because it describes the
// where there is no device to be honest about and BuildTBuiltInResource still has to // no-backend case (standalone shader compiles, unit tests), where
// hand glslang a workable gl_MaxClipDistances. // BuildTBuiltInResource still has to hand glslang a workable gl_MaxClipDistances.
Int MaxClipDistances = 8; Int MaxClipDistances = 8;
// GL_MAX_CULL_DISTANCES and GL_MAX_COMBINED_CLIP_AND_CULL_DISTANCES, under exactly // GL_MAX_CULL_DISTANCES and GL_MAX_COMBINED_CLIP_AND_CULL_DISTANCES, under exactly
// the contract stated for MaxClipDistances above: ZERO IS A LEGAL ANSWER and a // the contract stated for MaxClipDistances above: ZERO IS A LEGAL ANSWER and a
@@ -349,6 +349,23 @@ namespace MobileGL {
Bool SupportsTessellationPointSize = true; Bool SupportsTessellationPointSize = true;
// The geometry-stage twin (OpCapability GeometryPointSize). // The geometry-stage twin (OpCapability GeometryPointSize).
Bool SupportsGeometryPointSize = true; Bool SupportsGeometryPointSize = true;
// The backend cannot clip with gl_ClipDistance natively - no
// GL_EXT_clip_cull_distance, an extension the POST clip probe caught not clipping or
// not honoring the per-distance enables, or shaderClipDistance == VK_FALSE - so
// ProgramSpirvTask lowers the built-in to varyings plus a fragment discard
// (ShaderCompiler::EmulateClipDistanceForProgram) and the source pipeline declares the
// reserved enables uniform the draw path feeds. MOBILEGL_CLIP_DISTANCE_EMULATION
// overrides the detection in either direction at backend init. Defaults FALSE: true
// ARMS a rewrite, so the no-backend answer is the one that leaves modules untouched.
Bool EmulatesClipDistance = false;
// The backend HOSTS gl_ClipDistance but has no per-distance enable state: Vulkan
// clips by every distance a shader writes, so GL_CLIP_DISTANCEi would mean nothing.
// The producing stage then writes `enabled(i) ? d[i] : 1.0` back into the built-in
// (EmulateClipDistanceOptions::nativeBuiltin) from the same reserved enables uniform
// the emulation uses. DirectVulkan sets it whenever it clips natively; DirectGLES
// never does - ES drivers it keeps native passed the POST probe, which checks the
// driver honors GL_CLIP_DISTANCEi_EXT itself. Exclusive with EmulatesClipDistance.
Bool MasksNativeClipDistances = false;
Uint64 MaxShaderStorageBlockSize = 128 * 1024 * 1024; Uint64 MaxShaderStorageBlockSize = 128 * 1024 * 1024;
Uint32 SubgroupSize = 0; Uint32 SubgroupSize = 0;
Uint32 SubgroupSupportedStages = 0; Uint32 SubgroupSupportedStages = 0;
+9 -1
View File
@@ -14,7 +14,7 @@ namespace MobileGL::MG_Config {
inline const String ProjectName = "MobileGL"; inline const String ProjectName = "MobileGL";
inline const String CoreName = "MobileGL Core"; inline const String CoreName = "MobileGL Core";
inline const String CoreVendor = "MobileGL-Dev (BZLZHH, Swung0x48, Tungsten)"; inline const String CoreVendor = "MobileGL-Dev (BZLZHH, Swung0x48, Tungsten)";
inline const Version CoreVersion = {26, 10, 0, "-dev", VersionType::Development}; inline const Version CoreVersion = {26, 11, 0, "-dev", VersionType::Development};
inline const VersionStringFormatAttrib DefaultVersionStringFormatAttrib = {2, 2, 0, true, true}; inline const VersionStringFormatAttrib DefaultVersionStringFormatAttrib = {2, 2, 0, true, true};
inline const Uint64 CacheVersion = 0; inline const Uint64 CacheVersion = 0;
@@ -237,6 +237,14 @@ namespace MobileGL::MG_Config {
// negative control). Cross-backend by design: the demotion runs in the shared // negative control). Cross-backend by design: the demotion runs in the shared
// phase-B chain, so one switch covers both. See DemotePointSizePass. // phase-B chain, so one switch covers both. See DemotePointSizePass.
QuirkOverride PointSizeDemotion = QuirkOverride::Auto; QuirkOverride PointSizeDemotion = QuirkOverride::Auto;
// MOBILEGL_CLIP_DISTANCE_EMULATION: lower gl_ClipDistance to ordinary varyings plus a
// fragment discard (ShaderCompiler::EmulateClipDistanceForProgram) on a backend that
// cannot clip natively - no GL_EXT_clip_cull_distance or a driver the POST clip probe
// caught misbehaving (DirectGLES), shaderClipDistance == VK_FALSE (DirectVulkan).
// Auto arms it exactly there. ForceOn exercises it on a healthy driver (the pinned
// integration lane); ForceOff restores the native-or-nothing answer, which reports
// GL_MAX_CLIP_DISTANCES = 0 on a device without native support.
QuirkOverride ClipDistanceEmulation = QuirkOverride::Auto;
// MOBILEGL_COHERENT_AS_FLUSH: app-compat for engines (e.g. Flywheel) that write // MOBILEGL_COHERENT_AS_FLUSH: app-compat for engines (e.g. Flywheel) that write
// GPU-read data through persistent GL_MAP_FLUSH_EXPLICIT_BIT maps they never // GPU-read data through persistent GL_MAP_FLUSH_EXPLICIT_BIT maps they never
// flush. Persistent FLUSH_EXPLICIT map requests are rewritten to coherent // flush. Persistent FLUSH_EXPLICIT map requests are rewritten to coherent
+1
View File
@@ -402,6 +402,7 @@ namespace MobileGL::MG_ConfigLoader {
features.EsprytAtomicSubgroupScratch = QueryEnvQuirkOverride("MOBILEGL_ESPRYT_ATOMIC_SUBGROUP_SCRATCH"); features.EsprytAtomicSubgroupScratch = QueryEnvQuirkOverride("MOBILEGL_ESPRYT_ATOMIC_SUBGROUP_SCRATCH");
features.EsprytInvalidImagePlaceholder = QueryEnvQuirkOverride("MOBILEGL_ESPRYT_INVALID_IMAGE_PLACEHOLDER"); features.EsprytInvalidImagePlaceholder = QueryEnvQuirkOverride("MOBILEGL_ESPRYT_INVALID_IMAGE_PLACEHOLDER");
features.PointSizeDemotion = QueryEnvQuirkOverride("MOBILEGL_POINT_SIZE_DEMOTION"); features.PointSizeDemotion = QueryEnvQuirkOverride("MOBILEGL_POINT_SIZE_DEMOTION");
features.ClipDistanceEmulation = QueryEnvQuirkOverride("MOBILEGL_CLIP_DISTANCE_EMULATION");
features.CoherentAsFlush = QueryEnvFlag("MOBILEGL_COHERENT_AS_FLUSH"); features.CoherentAsFlush = QueryEnvFlag("MOBILEGL_COHERENT_AS_FLUSH");
features.EsprytDisableUboRing = QueryEnvFlag("MOBILEGL_ESPRYT_DISABLE_UBO_RING"); features.EsprytDisableUboRing = QueryEnvFlag("MOBILEGL_ESPRYT_DISABLE_UBO_RING");
features.EsprytDisableUnpackRing = QueryEnvFlag("MOBILEGL_ESPRYT_DISABLE_UNPACK_RING"); features.EsprytDisableUnpackRing = QueryEnvFlag("MOBILEGL_ESPRYT_DISABLE_UNPACK_RING");
@@ -1996,12 +1996,18 @@ namespace MobileGL::MG_Backend::DirectGLES {
} }
m_dynamicParameters.MaxDrawBuffers = m_GLESCapabilities.MaxDrawBuffers; m_dynamicParameters.MaxDrawBuffers = m_GLESCapabilities.MaxDrawBuffers;
m_dynamicParameters.MaxColorAttachments = m_GLESCapabilities.MaxColorAttachments; m_dynamicParameters.MaxColorAttachments = m_GLESCapabilities.MaxColorAttachments;
m_dynamicParameters.MaxClipDistances = m_GLESCapabilities.MaxClipDistances; // Eight when emulated - the GL 3.0+ minimum, and what EmulateClipDistancePass carries -
// else the driver's own number (0 without the extension and with the emulation forced
// off: the native-or-nothing answer).
m_dynamicParameters.EmulatesClipDistance = m_GLESCapabilities.EmulatesClipDistance;
m_dynamicParameters.MaxClipDistances =
m_GLESCapabilities.EmulatesClipDistance ? 8 : m_GLESCapabilities.MaxClipDistances;
// The loader already gated both on GL_EXT_clip_cull_distance and left 0 without it, which // The loader already gated both on GL_EXT_clip_cull_distance and left 0 without it, which
// is the answer that keeps glslang from accepting a gl_CullDistance the ESSL compiler // is the answer that keeps glslang from accepting a gl_CullDistance the ESSL compiler
// would reject. // would reject.
m_dynamicParameters.MaxCullDistances = m_GLESCapabilities.MaxCullDistances; m_dynamicParameters.MaxCullDistances = m_GLESCapabilities.MaxCullDistances;
m_dynamicParameters.MaxCombinedClipAndCullDistances = m_GLESCapabilities.MaxCombinedClipAndCullDistances; m_dynamicParameters.MaxCombinedClipAndCullDistances =
std::max(m_GLESCapabilities.MaxCombinedClipAndCullDistances, m_dynamicParameters.MaxClipDistances);
m_dynamicParameters.MaxViewports = m_GLESCapabilities.MaxViewports; m_dynamicParameters.MaxViewports = m_GLESCapabilities.MaxViewports;
// Whatever the driver said about which vertex supplies gl_Layer, and GL_UNDEFINED_VERTEX // Whatever the driver said about which vertex supplies gl_Layer, and GL_UNDEFINED_VERTEX
// for gl_ViewportIndex on every driver without GL_OES_viewport_array - which is both test // for gl_ViewportIndex on every driver without GL_OES_viewport_array - which is both test
@@ -514,7 +514,7 @@ namespace MobileGL::MG_Backend::DirectGLES {
} }
} }
if (tailSpanDirty && g_GLESCapabilities.SupportsClipDistance) { if (tailSpanDirty && g_GLESCapabilities.SupportsClipDistance && !g_GLESCapabilities.EmulatesClipDistance) {
// gl_ClipDistance clipping is per-distance enable state in GL, and ES reaches it // gl_ClipDistance clipping is per-distance enable state in GL, and ES reaches it
// only through GL_EXT_clip_cull_distance. That extension reuses the desktop enum // only through GL_EXT_clip_cull_distance. That extension reuses the desktop enum
// values for CLIP_DISTANCE0_EXT..7_EXT, but the token is absent from the ES // values for CLIP_DISTANCE0_EXT..7_EXT, but the token is absent from the ES
@@ -1117,6 +1117,28 @@ namespace MobileGL::MG_Backend::DirectVulkan {
} }
} }
// Without the shaderClipDistance feature the built-in is emulated instead
// (EmulateClipDistancePass: varyings plus a fragment discard, which needs nothing from the
// device), so the frontend can still answer the GL 3.0+ minimum of eight. Every Mali device
// measured so far reports shaderClipDistance = VK_FALSE. MOBILEGL_CLIP_DISTANCE_EMULATION=1
// arms it on a device that has the feature (the pinned integration lane); =0 restores the
// plain zero.
static Bool DecideClipDistanceEmulation(const Bool deviceHasClipDistance) {
switch (MG_Config::Features.ClipDistanceEmulation) {
case MG_Config::QuirkOverride::ForceOn:
MGLOG_I("DirectVulkan: MOBILEGL_CLIP_DISTANCE_EMULATION=1 - emulating gl_ClipDistance even though "
"the device %s shaderClipDistance",
deviceHasClipDistance ? "has" : "lacks");
return true;
case MG_Config::QuirkOverride::ForceOff:
MGLOG_I("DirectVulkan: MOBILEGL_CLIP_DISTANCE_EMULATION=0 - no gl_ClipDistance emulation");
return false;
case MG_Config::QuirkOverride::Auto:
default:
return !deviceHasClipDistance;
}
}
void BackendObject_DirectVulkan::UpdateDynamicBackendParameters() { void BackendObject_DirectVulkan::UpdateDynamicBackendParameters() {
const auto mapShaderStages = [](Uint32 vkStages) { const auto mapShaderStages = [](Uint32 vkStages) {
Uint32 glStages = 0; Uint32 glStages = 0;
@@ -1343,8 +1365,21 @@ namespace MobileGL::MG_Backend::DirectVulkan {
// by every device, but declaring ClipDistance in a module needs the shaderClipDistance // by every device, but declaring ClipDistance in a module needs the shaderClipDistance
// FEATURE, which VulkanRenderer enables exactly where the physical device has it. Without // FEATURE, which VulkanRenderer enables exactly where the physical device has it. Without
// it the limit describes a capacity no shader may use, so report none. // it the limit describes a capacity no shader may use, so report none.
const Bool emulateClipDistance = DecideClipDistanceEmulation(m_vulkanCaps.SupportsShaderClipDistance);
m_dynamicParameters.EmulatesClipDistance = emulateClipDistance;
// Hosted natively, but Vulkan has no per-distance enable state: apply GL_CLIP_DISTANCEi
// in the producing stage instead (EmulateClipDistanceOptions::nativeBuiltin).
m_dynamicParameters.MasksNativeClipDistances = !emulateClipDistance && m_vulkanCaps.SupportsShaderClipDistance;
m_dynamicParameters.MaxClipDistances = m_dynamicParameters.MaxClipDistances =
m_vulkanCaps.SupportsShaderClipDistance ? std::max(m_vulkanCaps.MaxClipDistances, 0) : 0; emulateClipDistance
? 8
: (m_vulkanCaps.SupportsShaderClipDistance ? std::max(m_vulkanCaps.MaxClipDistances, 0) : 0);
MGLOG_I("DirectVulkan: gl_ClipDistance %s (GL_MAX_CLIP_DISTANCES = %d)",
emulateClipDistance
? "EMULATED (varyings + fragment discard)"
: (m_dynamicParameters.MasksNativeClipDistances ? "native, GL_CLIP_DISTANCEi applied in the shader"
: "unsupported"),
m_dynamicParameters.MaxClipDistances);
// The cull pair, gated on its own feature. shaderCullDistance is separate from // The cull pair, gated on its own feature. shaderCullDistance is separate from
// shaderClipDistance and VulkanRenderer enables it independently, so it gets its own // shaderClipDistance and VulkanRenderer enables it independently, so it gets its own
// gate rather than riding on the clip one. // gate rather than riding on the clip one.
@@ -1357,7 +1392,7 @@ namespace MobileGL::MG_Backend::DirectVulkan {
(m_vulkanCaps.SupportsShaderClipDistance || m_vulkanCaps.SupportsShaderCullDistance) (m_vulkanCaps.SupportsShaderClipDistance || m_vulkanCaps.SupportsShaderCullDistance)
? std::max({m_vulkanCaps.MaxCombinedClipAndCullDistances, m_dynamicParameters.MaxClipDistances, ? std::max({m_vulkanCaps.MaxCombinedClipAndCullDistances, m_dynamicParameters.MaxClipDistances,
m_dynamicParameters.MaxCullDistances}) m_dynamicParameters.MaxCullDistances})
: 0; : m_dynamicParameters.MaxClipDistances;
m_dynamicParameters.MaxViewports = m_vulkanCaps.MaxViewports; m_dynamicParameters.MaxViewports = m_vulkanCaps.MaxViewports;
// Assigned explicitly rather than left to the struct's defaults, like every other // Assigned explicitly rather than left to the struct's defaults, like every other
// parameter here, so a second fill cannot inherit a stale value. GL_UNDEFINED_VERTEX is // parameter here, so a second fill cannot inherit a stale value. GL_UNDEFINED_VERTEX is
@@ -132,6 +132,9 @@ namespace MobileGL::MG_Backend::DirectVulkan {
private: private:
VkTextureManager& m_manager; VkTextureManager& m_manager;
}; };
// True inside a PendingClearHold: the image being resolved is an attachment of the pass
// being prepared, which takes pending clears as its load op.
Bool PendingClearsHeld() const { return m_pendingClearHold != 0; }
// A whole-subresource clear that was recorded but has not run yet (VulkanRenderer::DeferWireClear). // A whole-subresource clear that was recorded but has not run yet (VulkanRenderer::DeferWireClear).
struct WirePendingClear { struct WirePendingClear {
Uint32 level = 0, layer = 0, layers = 1; Uint32 level = 0, layer = 0, layers = 1;
@@ -620,7 +620,35 @@ namespace MobileGL::MG_Backend::DirectVulkan {
Uint32 level = 0, layer = 0, layers = 1; Uint32 level = 0, layer = 0, layers = 1;
}; };
Vector<WireFoldClear> m_wireFoldClears; Vector<WireFoldClear> m_wireFoldClears;
// Attachments of the pass being prepared whose content is undefined: a default-framebuffer
// image first used since its present. Their load op becomes DONT_CARE (WireClearVariant).
Vector<Uint32> m_wireDiscardLoads;
static Bool WireImageCanDeferClear(const WireImage& image); static Bool WireImageCanDeferClear(const WireImage& image);
// A whole-image clear of the default framebuffer, held for the next pass that draws into
// the window (its load op) the way WirePendingClear holds a texture's. Any other use of the
// window images, and the present, run it first (MaterializeDefaultWirePendingClear).
struct DefaultWirePendingClear {
VkSwapchainKHR swapchain = VK_NULL_HANDLE;
Uint32 swapchainImageIndex = 0;
Bool color = false;
VkImageAspectFlags depthStencil = 0;
ClearAttachmentPayload colorPayload{};
// colorPayload against the window format and GL_FRAMEBUFFER_SRGB as they were at the
// glClear (PreCompensateSrgbClearColor): the standalone clear a present or another
// use of the window records later must not read that state again, after the verb.
ClearAttachmentPayload colorPayloadCompensated{};
ClearAttachmentPayload depthStencilPayload{};
Bool Pending() const { return color || depthStencil != 0; }
};
DefaultWirePendingClear m_defaultWirePendingClear;
GLbitfield DeferDefaultWireClear(const MG_Pipe::MGPFramebufferState& fbo, const ClearAttachmentPayload& payload,
GLint drawbuffer);
void MaterializeDefaultWirePendingClear();
void FoldDefaultWirePendingClear(RenderPassEntry& begun, Uint64& variantKey, VkClearValue* merged);
// Records a clear of `area` of one wire image in a single-attachment pass of its own.
// `colorPreCompensated`: the payload already went through PreCompensateSrgbClearColor.
void RecordWireImageClear(WireImage& image, VkImageAspectFlags aspect, const ClearAttachmentPayload& payload,
const VkRect2D& area, Bool textureSurface, Bool colorPreCompensated = false);
static void NoteWirePendingClear(const WireImage& image, VkImageAspectFlags aspects, const VkClearValue& value); static void NoteWirePendingClear(const WireImage& image, VkImageAspectFlags aspects, const VkClearValue& value);
VkRenderPass WireClearVariant(RenderPassEntry& begun, Uint64 variantKey); VkRenderPass WireClearVariant(RenderPassEntry& begun, Uint64 variantKey);
Bool BeginWireDrawPassFolding(FrameContext::FrameData& frame, RenderPassEntry& begun, Uint64 variantKey, Bool BeginWireDrawPassFolding(FrameContext::FrameData& frame, RenderPassEntry& begun, Uint64 variantKey,
@@ -16,6 +16,15 @@ namespace MobileGL::MG_Backend::DirectVulkan {
// Present has its own submit path. Tag its last wire pass before that // Present has its own submit path. Tag its last wire pass before that
// submission, rather than retaining it until a draw in the next frame. // submission, rather than retaining it until a draw in the next frame.
RetireWireDrawPass(); RetireWireDrawPass();
// A window color clear no pass drew after still has to reach the presented image. Its
// depth/stencil half is dropped: a swap leaves those undefined (see the content flags
// reset below). A frame that cannot be presented drops the whole clear.
if (m_swapchainObject.GetHandle() == VK_NULL_HANDLE || m_presentSuspended) {
m_defaultWirePendingClear = {};
} else {
m_defaultWirePendingClear.depthStencil = 0;
MaterializeDefaultWirePendingClear();
}
#endif #endif
if (m_swapchainObject.GetHandle() == VK_NULL_HANDLE || m_presentSuspended) { if (m_swapchainObject.GetHandle() == VK_NULL_HANDLE || m_presentSuspended) {
// No usable swapchain: the window was zero-area at initialization, or // No usable swapchain: the window was zero-area at initialization, or
@@ -9,6 +9,14 @@
#include "VulkanRendererInternal.h" #include "VulkanRendererInternal.h"
namespace MobileGL::MG_Backend::DirectVulkan { namespace MobileGL::MG_Backend::DirectVulkan {
namespace {
// A wire pass variant key holds two clear bits per attachment from bit 0 and, from this
// bit on, one bit per attachment whose content is undefined (WireClearVariant).
constexpr Uint32 kWireDiscardBit = 32;
static_assert(2 * (MG_Pipe::kMGPipeMaxColorAttachments + 1) <= kWireDiscardBit,
"the clear bits of a wire pass variant key overlap its discard bits");
} // namespace
// GL_EXT_fragment_shading_rate's pipeline rate and combiner ops for the draw about to be // GL_EXT_fragment_shading_rate's pipeline rate and combiner ops for the draw about to be
// recorded. The packed rate is log2(width) << 2 | log2(height), which is exactly the extent // recorded. The packed rate is log2(width) << 2 | log2(height), which is exactly the extent
// VK_KHR_fragment_shading_rate takes, and the combiner op indices follow // VK_KHR_fragment_shading_rate takes, and the combiner op indices follow
@@ -445,6 +453,7 @@ namespace MobileGL::MG_Backend::DirectVulkan {
WireDrawScratch& scratch, Bool passContinues, Bool& continuing) { WireDrawScratch& scratch, Bool passContinues, Bool& continuing) {
continuing = true; continuing = true;
m_wireFoldClears.clear(); m_wireFoldClears.clear();
m_wireDiscardLoads.clear();
if (passContinues) { if (passContinues) {
auto& store = MG_Record::ServerStagedTexture(); auto& store = MG_Record::ServerStagedTexture();
if (!m_wirePassMarksValid || m_wirePassMarksClearGeneration != store.GpuDirtyClearGeneration()) { if (!m_wirePassMarksValid || m_wirePassMarksClearGeneration != store.GpuDirtyClearGeneration()) {
@@ -536,6 +545,13 @@ namespace MobileGL::MG_Backend::DirectVulkan {
if (!image.isDefault && surface.Kind == MG_Pipe::kMGPipeSurfaceKindTexture) if (!image.isDefault && surface.Kind == MG_Pipe::kMGPipeSurfaceKindTexture)
scratch.gpuWriteMarks.push_back({image.storage, image.level, image.layer, image.layers}); scratch.gpuWriteMarks.push_back({image.storage, image.level, image.layer, image.layers});
if (image.isDefault) { if (image.isDefault) {
// A present leaves the default framebuffer undefined (VulkanRenderer::Present):
// the first pass after it has nothing to load.
const Bool defined =
aspect == VK_IMAGE_ASPECT_COLOR_BIT
? m_swapchainObject.IsImageContentDefined(image.swapchainImageIndex)
: m_swapchainObject.IsDepthStencilContentDefined(image.swapchainImageIndex);
if (!defined) m_wireDiscardLoads.push_back(static_cast<Uint32>(attachments.size() - 1));
if (aspect == VK_IMAGE_ASPECT_COLOR_BIT) if (aspect == VK_IMAGE_ASPECT_COLOR_BIT)
m_swapchainObject.SetImageContentDefined(image.swapchainImageIndex, true); m_swapchainObject.SetImageContentDefined(image.swapchainImageIndex, true);
else else
@@ -1361,9 +1377,18 @@ namespace MobileGL::MG_Backend::DirectVulkan {
if (entry.first == variantKey) return entry.second; if (entry.first == variantKey) return entry.second;
const Uint32 attachmentCount = static_cast<Uint32>(begun.wireAttachments.size()); const Uint32 attachmentCount = static_cast<Uint32>(begun.wireAttachments.size());
auto descriptions = begun.wireAttachments; auto descriptions = begun.wireAttachments;
// Bits 2a and 2a+1 clear attachment a (color or depth, and stencil); bit kWireDiscardBit + a
// marks its content undefined, so whatever is not cleared is not loaded either.
for (Uint32 a = 0; a < attachmentCount; ++a) { for (Uint32 a = 0; a < attachmentCount; ++a) {
if (variantKey >> (2 * a) & 1) descriptions[a].loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR; const Bool discard = variantKey >> (kWireDiscardBit + a) & 1;
if (variantKey >> (2 * a + 1) & 1) descriptions[a].stencilLoadOp = VK_ATTACHMENT_LOAD_OP_CLEAR; if (variantKey >> (2 * a) & 1)
descriptions[a].loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
else if (discard)
descriptions[a].loadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
if (variantKey >> (2 * a + 1) & 1)
descriptions[a].stencilLoadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
else if (discard)
descriptions[a].stencilLoadOp = VK_ATTACHMENT_LOAD_OP_DONT_CARE;
} }
VkSubpassDescription subpass{}; VkSubpassDescription subpass{};
subpass.pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS; subpass.pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS;
@@ -1382,6 +1407,142 @@ namespace MobileGL::MG_Backend::DirectVulkan {
return variant; return variant;
} }
// The default framebuffer's half of the deferred clears: a whole-window glClear waits for the
// next pass that draws into the window, which loads it with LOAD_OP_CLEAR
// (BeginWireDrawPassFolding) instead of storing a cleared window and loading it back. A pass
// open on the window ends first, since the in-pass clear does not take the window
// (MagmaInPassClearDefault); a present drops the depth/stencil half (Present). Returns the
// buffers it did not take.
GLbitfield VulkanRenderer::DeferDefaultWireClear(const MG_Pipe::MGPFramebufferState& fbo,
const ClearAttachmentPayload& payload, GLint drawbuffer) {
GLbitfield remaining = payload.mask & (GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
if (!remaining || !fbo.IsDefault || drawbuffer >= 0 || m_wirePreparationDepth != 0) return remaining;
if (GLScissorTestClips()) return remaining;
GLbitfield handled = 0;
Bool color = false;
VkImageAspectFlags depthStencil = 0;
if (remaining & GL_COLOR_BUFFER_BIT) {
const Int32 slot = fbo.DrawBuffers[0];
const auto mask = MG_Pipe::gPipeInputs->GetColorMaskIndexed(0);
const Bool none = slot < 0 || (!mask.r() && !mask.g() && !mask.b() && !mask.a());
if (none)
handled |= GL_COLOR_BUFFER_BIT;
else if (mask.r() && mask.g() && mask.b() && mask.a() &&
payload.colorEncoding == ClearColorEncoding::Float) {
handled |= GL_COLOR_BUFFER_BIT;
color = true;
}
}
// A window without a depth/stencil buffer clears nothing there.
const VkFormat depthStencilFormat = m_swapchainObject.GetDepthStencilFormat();
const Bool hasDepth = depthStencilFormat != VK_FORMAT_UNDEFINED;
const Bool packed = vkuFormatIsDepthAndStencil(depthStencilFormat);
if (remaining & GL_DEPTH_BUFFER_BIT) {
handled |= GL_DEPTH_BUFFER_BIT;
if (hasDepth && MG_Pipe::gPipeInputs->GetDepthMask()) depthStencil |= VK_IMAGE_ASPECT_DEPTH_BIT;
}
if (remaining & GL_STENCIL_BUFFER_BIT) {
const Uint32 mask = MG_Pipe::gPipeInputs->GetStencilState(StencilFace::Front).WriteMask & 255u;
if (mask == 0 || !packed)
handled |= GL_STENCIL_BUFFER_BIT;
else if (mask == 255u) {
handled |= GL_STENCIL_BUFFER_BIT;
depthStencil |= VK_IMAGE_ASPECT_STENCIL_BIT;
}
}
if (!color && depthStencil == 0) return remaining & ~handled;
if (VkRenderPassManager::GetActiveRenderPass() != nullptr && m_wireDrawPass && m_wireDrawPassKey.isDefault)
RetireWireDrawPass();
// The index the next drawing pass on the window will address (WireDraw.cpp's pass key).
const Uint32 index = DefaultFramebufferWriteIndex(true);
auto& pending = m_defaultWirePendingClear;
if (pending.Pending() &&
(pending.swapchain != m_swapchainObject.GetHandle() || pending.swapchainImageIndex != index))
MaterializeDefaultWirePendingClear();
pending.swapchain = m_swapchainObject.GetHandle();
pending.swapchainImageIndex = index;
if (color) {
pending.color = true;
pending.colorPayload = payload;
pending.colorPayloadCompensated = payload;
PreCompensateSrgbClearColor(pending.colorPayloadCompensated, m_swapchainObject.GetSurfaceFormat().format);
m_swapchainObject.SetImageContentDefined(index, true);
}
if (depthStencil & VK_IMAGE_ASPECT_DEPTH_BIT) pending.depthStencilPayload.depth = payload.depth;
if (depthStencil & VK_IMAGE_ASPECT_STENCIL_BIT) pending.depthStencilPayload.stencil = payload.stencil;
if (depthStencil) {
pending.depthStencil |= depthStencil;
m_swapchainObject.SetDepthStencilContentDefined(index, true);
}
return remaining & ~handled;
}
void VulkanRenderer::MaterializeDefaultWirePendingClear() {
if (!m_defaultWirePendingClear.Pending()) return;
const DefaultWirePendingClear pending = m_defaultWirePendingClear;
m_defaultWirePendingClear = {};
// A recreated swapchain took the images the clear was for.
if (pending.swapchain != m_swapchainObject.GetHandle() ||
pending.swapchainImageIndex >= m_swapchainObject.GetImageCount())
return;
const VkExtent2D extent = m_swapchainObject.GetExtent();
const VkRect2D area{{0, 0}, extent};
WireImage image{};
image.isDefault = true;
image.extent = extent;
image.swapchainImageIndex = pending.swapchainImageIndex;
if (pending.color) {
image.aspect = VK_IMAGE_ASPECT_COLOR_BIT;
image.image = m_swapchainObject.GetImage(pending.swapchainImageIndex);
image.format = m_swapchainObject.GetSurfaceFormat().format;
image.layout = m_swapchainObject.GetImageLayout(pending.swapchainImageIndex);
RecordWireImageClear(image, VK_IMAGE_ASPECT_COLOR_BIT, pending.colorPayloadCompensated, area, false,
/*colorPreCompensated=*/true);
}
if (pending.depthStencil && m_swapchainObject.GetDepthStencilImage(pending.swapchainImageIndex)) {
image.aspect = pending.depthStencil;
image.image = m_swapchainObject.GetDepthStencilImage(pending.swapchainImageIndex);
image.format = m_swapchainObject.GetDepthStencilFormat();
image.layout = m_swapchainObject.GetDepthStencilImageLayout(pending.swapchainImageIndex);
RecordWireImageClear(image, pending.depthStencil, pending.depthStencilPayload, area, false);
}
}
// A pending window clear (DeferDefaultWireClear) becomes the load op of the window pass that
// begins: its attachments are matched by image, since a continued pass key is not walked again.
// A clear the caller already put in `variantKey` is newer and wins.
void VulkanRenderer::FoldDefaultWirePendingClear(RenderPassEntry& begun, Uint64& variantKey, VkClearValue* merged) {
auto& pending = m_defaultWirePendingClear;
if (!pending.Pending() || !m_wireDrawPassKey.isDefault || m_wireDrawPassKey.swapchain != pending.swapchain ||
m_wireDrawPassKey.swapchainImageIndex != pending.swapchainImageIndex)
return;
const Uint32 attachmentCount =
static_cast<Uint32>(std::min<SizeT>(begun.wireAttachments.size(), m_wireDrawPassKey.attachments.size()));
const VkImage colorImage = m_swapchainObject.GetImage(pending.swapchainImageIndex);
const VkImage depthStencilImage = m_swapchainObject.GetDepthStencilImage(pending.swapchainImageIndex);
for (Uint32 a = 0; a < attachmentCount && a < MG_Pipe::kMGPipeMaxColorAttachments + 1; ++a) {
const auto& attachment = m_wireDrawPassKey.attachments[a];
if (pending.color && attachment.image == colorImage) {
if (!(variantKey >> (2 * a) & 1)) {
variantKey |= 1ull << (2 * a);
merged[a].color = MakeVkClearColorValue(pending.colorPayload, false);
}
pending.color = false;
} else if (pending.depthStencil && attachment.image == depthStencilImage) {
if ((pending.depthStencil & VK_IMAGE_ASPECT_DEPTH_BIT) && !(variantKey >> (2 * a) & 1)) {
variantKey |= 1ull << (2 * a);
merged[a].depthStencil.depth = pending.depthStencilPayload.depth;
}
if ((pending.depthStencil & VK_IMAGE_ASPECT_STENCIL_BIT) && !(variantKey >> (2 * a + 1) & 1)) {
variantKey |= 1ull << (2 * a + 1);
merged[a].depthStencil.stencil = pending.depthStencilPayload.stencil;
}
pending.depthStencil = 0;
}
}
// An aspect the pass does not attach stays pending: none of its draws write it.
}
// P15: EVERY DRAW PASS BEGINS HERE. The pending clears of its attachments // P15: EVERY DRAW PASS BEGINS HERE. The pending clears of its attachments
// (m_wireFoldClears, gathered by PrepareWireDrawPass) become their load op and leave the // (m_wireFoldClears, gathered by PrepareWireDrawPass) become their load op and leave the
// images; a clear the caller already put in `variantKey` (its own glClear, newer than any // images; a clear the caller already put in `variantKey` (its own glClear, newer than any
@@ -1426,6 +1587,10 @@ namespace MobileGL::MG_Backend::DirectVulkan {
} }
} }
m_wireFoldClears.clear(); m_wireFoldClears.clear();
FoldDefaultWirePendingClear(begun, variantKey, merged);
for (const Uint32 a : m_wireDiscardLoads)
if (a < attachmentCount) variantKey |= 1ull << (kWireDiscardBit + a);
m_wireDiscardLoads.clear();
if (variantKey == 0) return VkRenderPassManager::BeginRenderPass(frame.commandBuffer, begun); if (variantKey == 0) return VkRenderPassManager::BeginRenderPass(frame.commandBuffer, begun);
return VkRenderPassManager::BeginRenderPass(frame.commandBuffer, begun, WireClearVariant(begun, variantKey), return VkRenderPassManager::BeginRenderPass(frame.commandBuffer, begun, WireClearVariant(begun, variantKey),
merged, attachmentCount); merged, attachmentCount);
@@ -658,6 +658,10 @@ namespace MobileGL::MG_Backend::DirectVulkan {
WireImage out{}; WireImage out{};
out.aspect = aspect; out.aspect = aspect;
if (fbo.IsDefault) { if (fbo.IsDefault) {
// Only an attachment of the pass being prepared takes a pending window clear as its
// load op; every other use runs it first.
if (m_defaultWirePendingClear.Pending() && !m_textureManager->PendingClearsHeld())
MaterializeDefaultWirePendingClear();
out.isDefault = true; out.isDefault = true;
out.extent = m_swapchainObject.GetExtent(); out.extent = m_swapchainObject.GetExtent();
out.swapchainImageIndex = out.swapchainImageIndex =
@@ -1109,6 +1113,87 @@ namespace MobileGL::MG_Backend::DirectVulkan {
return remaining & ~handled; return remaining & ~handled;
} }
void VulkanRenderer::RecordWireImageClear(WireImage& image, VkImageAspectFlags aspect,
const ClearAttachmentPayload& payload, const VkRect2D& area,
Bool textureSurface, Bool colorPreCompensated) {
const Bool color = aspect == VK_IMAGE_ASPECT_COLOR_BIT;
const auto layout =
color ? VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL : VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;
TransitionWireImage(image, layout);
VkImageViewCreateInfo viewInfo{VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO};
viewInfo.image = image.image;
viewInfo.viewType = image.viewType;
viewInfo.format = image.format;
viewInfo.subresourceRange = {
color ? VK_IMAGE_ASPECT_COLOR_BIT
: (image.format == VK_FORMAT_D24_UNORM_S8_UINT || image.format == VK_FORMAT_D32_SFLOAT_S8_UINT
? VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT
: image.aspect),
image.level, 1, image.layer, image.layers};
VkImageView view = VK_NULL_HANDLE;
VK_VERIFY(vkCreateImageView(m_device, &viewInfo, nullptr, &view));
VkAttachmentDescription attachment{};
attachment.format = image.format;
attachment.samples = image.samples;
attachment.loadOp = VK_ATTACHMENT_LOAD_OP_LOAD;
attachment.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
attachment.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_LOAD;
attachment.stencilStoreOp = VK_ATTACHMENT_STORE_OP_STORE;
attachment.initialLayout = layout;
attachment.finalLayout = layout;
VkAttachmentReference ref{0, layout};
VkSubpassDescription subpass{};
subpass.pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS;
subpass.colorAttachmentCount = color ? 1 : 0;
subpass.pColorAttachments = color ? &ref : nullptr;
subpass.pDepthStencilAttachment = color ? nullptr : &ref;
VkRenderPassCreateInfo passInfo{VK_STRUCTURE_TYPE_RENDER_PASS_CREATE_INFO};
passInfo.attachmentCount = 1;
passInfo.pAttachments = &attachment;
passInfo.subpassCount = 1;
passInfo.pSubpasses = &subpass;
VkRenderPass pass = VK_NULL_HANDLE;
VK_VERIFY(vkCreateRenderPass(m_device, &passInfo, nullptr, &pass));
VkFramebufferCreateInfo fbInfo{VK_STRUCTURE_TYPE_FRAMEBUFFER_CREATE_INFO};
fbInfo.renderPass = pass;
fbInfo.attachmentCount = 1;
fbInfo.pAttachments = &view;
fbInfo.width = image.extent.width;
fbInfo.height = image.extent.height;
fbInfo.layers = image.layers;
VkFramebuffer fb = VK_NULL_HANDLE;
VK_VERIFY(vkCreateFramebuffer(m_device, &fbInfo, nullptr, &fb));
VkRenderPassBeginInfo begin{VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO};
begin.renderPass = pass;
begin.framebuffer = fb;
begin.renderArea = area;
auto clearPayload = payload;
if (color && !colorPreCompensated) PreCompensateSrgbClearColor(clearPayload, image.format);
VkClearAttachment clear{};
clear.aspectMask = aspect;
clear.clearValue.color = MakeVkClearColorValue(clearPayload, false);
if (!color) clear.clearValue.depthStencil = {payload.depth, payload.stencil};
VkClearRect rect{area, 0, image.layers};
auto& frame = m_frameContext.GetCurrent();
EndActiveRenderPassOn(frame.commandBuffer);
vkCmdBeginRenderPass(frame.commandBuffer, &begin, VK_SUBPASS_CONTENTS_INLINE);
vkCmdClearAttachments(frame.commandBuffer, 1, &clear, 1, &rect);
vkCmdEndRenderPass(frame.commandBuffer);
WireRetiredObjects retired{};
retired.framebuffers.push_back(fb);
retired.renderPasses.push_back(pass);
retired.imageViews.push_back(view);
RetireWireObjects(Move(retired));
if (!image.isDefault && textureSurface)
m_textureManager->MarkWireTextureGpuWritten(image.storage, image.level, image.layer, image.layers);
if (image.isDefault) {
if (color)
m_swapchainObject.SetImageContentDefined(image.swapchainImageIndex, true);
else
m_swapchainObject.SetDepthStencilContentDefined(image.swapchainImageIndex, true);
}
}
void VulkanRenderer::ClearWireFramebuffer(const MG_Pipe::MGPFramebufferState& fbo, void VulkanRenderer::ClearWireFramebuffer(const MG_Pipe::MGPFramebufferState& fbo,
const ClearAttachmentPayload& clearRequest, GLint drawbuffer) { const ClearAttachmentPayload& clearRequest, GLint drawbuffer) {
if (MG_Pipe::gPipeInputs->IsCapabilityEnabled(CapabilityInput::RasterizerDiscard)) return; if (MG_Pipe::gPipeInputs->IsCapabilityEnabled(CapabilityInput::RasterizerDiscard)) return;
@@ -1126,6 +1211,7 @@ namespace MobileGL::MG_Backend::DirectVulkan {
payload.mask = ClearWireFramebufferInPass(fbo, payload, drawbuffer); payload.mask = ClearWireFramebufferInPass(fbo, payload, drawbuffer);
if (payload.mask) payload.mask = ClearWireAttachmentsOfOpenPass(fbo, payload, drawbuffer); if (payload.mask) payload.mask = ClearWireAttachmentsOfOpenPass(fbo, payload, drawbuffer);
if (payload.mask) payload.mask = DeferWireClear(fbo, payload, drawbuffer); if (payload.mask) payload.mask = DeferWireClear(fbo, payload, drawbuffer);
if (payload.mask) payload.mask = DeferDefaultWireClear(fbo, payload, drawbuffer);
if (payload.mask) payload.mask = ClearWireFramebufferInPass(fbo, payload, drawbuffer); if (payload.mask) payload.mask = ClearWireFramebufferInPass(fbo, payload, drawbuffer);
if (!payload.mask) return; if (!payload.mask) return;
RetireWireDrawPass(); RetireWireDrawPass();
@@ -1177,82 +1263,7 @@ namespace MobileGL::MG_Backend::DirectVulkan {
m_swapchainObject.GetPreTransform(), &area.offset, &area.extent)) m_swapchainObject.GetPreTransform(), &area.offset, &area.extent))
MagmaWireFatal("clear-surface-transform"); MagmaWireFatal("clear-surface-transform");
} }
const Bool color = aspect == VK_IMAGE_ASPECT_COLOR_BIT; RecordWireImageClear(image, aspect, payload, area, surface.Kind == MG_Pipe::kMGPipeSurfaceKindTexture);
const auto layout =
color ? VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL : VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL;
TransitionWireImage(image, layout);
VkImageViewCreateInfo viewInfo{VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO};
viewInfo.image = image.image;
viewInfo.viewType = image.viewType;
viewInfo.format = image.format;
viewInfo.subresourceRange = {
color ? VK_IMAGE_ASPECT_COLOR_BIT
: (image.format == VK_FORMAT_D24_UNORM_S8_UINT || image.format == VK_FORMAT_D32_SFLOAT_S8_UINT
? VK_IMAGE_ASPECT_DEPTH_BIT | VK_IMAGE_ASPECT_STENCIL_BIT
: image.aspect),
image.level, 1, image.layer, image.layers};
VkImageView view = VK_NULL_HANDLE;
VK_VERIFY(vkCreateImageView(m_device, &viewInfo, nullptr, &view));
VkAttachmentDescription attachment{};
attachment.format = image.format;
attachment.samples = image.samples;
attachment.loadOp = VK_ATTACHMENT_LOAD_OP_LOAD;
attachment.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
attachment.stencilLoadOp = VK_ATTACHMENT_LOAD_OP_LOAD;
attachment.stencilStoreOp = VK_ATTACHMENT_STORE_OP_STORE;
attachment.initialLayout = layout;
attachment.finalLayout = layout;
VkAttachmentReference ref{0, layout};
VkSubpassDescription subpass{};
subpass.pipelineBindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS;
subpass.colorAttachmentCount = color ? 1 : 0;
subpass.pColorAttachments = color ? &ref : nullptr;
subpass.pDepthStencilAttachment = color ? nullptr : &ref;
VkRenderPassCreateInfo passInfo{VK_STRUCTURE_TYPE_RENDER_PASS_CREATE_INFO};
passInfo.attachmentCount = 1;
passInfo.pAttachments = &attachment;
passInfo.subpassCount = 1;
passInfo.pSubpasses = &subpass;
VkRenderPass pass = VK_NULL_HANDLE;
VK_VERIFY(vkCreateRenderPass(m_device, &passInfo, nullptr, &pass));
VkFramebufferCreateInfo fbInfo{VK_STRUCTURE_TYPE_FRAMEBUFFER_CREATE_INFO};
fbInfo.renderPass = pass;
fbInfo.attachmentCount = 1;
fbInfo.pAttachments = &view;
fbInfo.width = image.extent.width;
fbInfo.height = image.extent.height;
fbInfo.layers = image.layers;
VkFramebuffer fb = VK_NULL_HANDLE;
VK_VERIFY(vkCreateFramebuffer(m_device, &fbInfo, nullptr, &fb));
VkRenderPassBeginInfo begin{VK_STRUCTURE_TYPE_RENDER_PASS_BEGIN_INFO};
begin.renderPass = pass;
begin.framebuffer = fb;
begin.renderArea = area;
auto clearPayload = payload;
if (color) PreCompensateSrgbClearColor(clearPayload, image.format);
VkClearAttachment clear{};
clear.aspectMask = aspect;
clear.clearValue.color = MakeVkClearColorValue(clearPayload, false);
if (!color) clear.clearValue.depthStencil = {payload.depth, payload.stencil};
VkClearRect rect{area, 0, image.layers};
auto& frame = m_frameContext.GetCurrent();
EndActiveRenderPassOn(frame.commandBuffer);
vkCmdBeginRenderPass(frame.commandBuffer, &begin, VK_SUBPASS_CONTENTS_INLINE);
vkCmdClearAttachments(frame.commandBuffer, 1, &clear, 1, &rect);
vkCmdEndRenderPass(frame.commandBuffer);
WireRetiredObjects retired{};
retired.framebuffers.push_back(fb);
retired.renderPasses.push_back(pass);
retired.imageViews.push_back(view);
RetireWireObjects(Move(retired));
if (!image.isDefault && surface.Kind == MG_Pipe::kMGPipeSurfaceKindTexture)
m_textureManager->MarkWireTextureGpuWritten(image.storage, image.level, image.layer, image.layers);
if (image.isDefault) {
if (color)
m_swapchainObject.SetImageContentDefined(image.swapchainImageIndex, true);
else
m_swapchainObject.SetDepthStencilContentDefined(image.swapchainImageIndex, true);
}
}; };
if (payload.mask & GL_COLOR_BUFFER_BIT) { if (payload.mask & GL_COLOR_BUFFER_BIT) {
for (Uint32 i = 0; i < MG_Pipe::kMGPipeMaxColorAttachments; ++i) { for (Uint32 i = 0; i < MG_Pipe::kMGPipeMaxColorAttachments; ++i) {
@@ -108,6 +108,20 @@ namespace MobileGL::MG_Impl::GLImpl {
program->WriteReservedNumSamples(static_cast<Int>(std::max<GLint>(ResolveDrawFramebufferSampleCount(), 1))); program->WriteReservedNumSamples(static_cast<Int>(std::max<GLint>(ResolveDrawFramebufferSampleCount(), 1)));
} }
// The GL_CLIP_DISTANCE0..7 enables, for a program whose clip distances are emulated (see
// EmulateClipDistancePass): the producing stage selects 1.0 for every disabled distance, so
// the mask is a property of the render state at THIS draw, not of the program. Same cost
// model as the sample count above.
static void PublishClipDistanceEnables(const SharedPtr<MG_State::GLState::ProgramObject>& program) {
if (!program || !program->UsesReservedClipDistanceEnables()) return;
Uint32 enables = 0;
for (Uint i = 0; i < 8; ++i) {
const auto cap = static_cast<CapabilityInput>(static_cast<Uint>(CapabilityInput::ClipDistance0) + i);
if (MG_State::pGLContext->IsCapabilityEnabled(cap)) enables |= 1u << i;
}
program->WriteReservedClipDistanceEnables(enables);
}
// The one funnel every drawing command passes through. Order is load-bearing: validate first // The one funnel every drawing command passes through. Order is load-bearing: validate first
// (a rejected draw must leave state alone), then publish the sample count - which reads the // (a rejected draw must leave state alone), then publish the sample count - which reads the
// DRAW FRAMEBUFFER binding, so it has to run after the caller's framebuffer state is settled // DRAW FRAMEBUFFER binding, so it has to run after the caller's framebuffer state is settled
@@ -133,6 +147,7 @@ namespace MobileGL::MG_Impl::GLImpl {
const auto& currentProgram = MG_State::pGLContext->GetProgramForDraw(); const auto& currentProgram = MG_State::pGLContext->GetProgramForDraw();
if (!ValidateResolvedProgramForDraw(currentProgram, functionName)) return false; if (!ValidateResolvedProgramForDraw(currentProgram, functionName)) return false;
PublishDrawFramebufferSampleCount(currentProgram); PublishDrawFramebufferSampleCount(currentProgram);
PublishClipDistanceEnables(currentProgram);
return true; return true;
} }
+25 -12
View File
@@ -2197,7 +2197,7 @@ namespace MobileGL::MG_Impl::GLImpl {
// Allocate in TextureObject // Allocate in TextureObject
if (isProxy) { if (isProxy) {
MGLOG_D("%s: isProxy = true, not allocating", __func__); TextureImpl::pProxyTextureManager->RecordProxyLevel(textureUploadTarget, level, {width, height, depth});
} else { } else {
DiscardMipmapChainOnBaseRespecification(textureMipmapObject, textureUploadTarget, level); DiscardMipmapChainOnBaseRespecification(textureMipmapObject, textureUploadTarget, level);
textureMipmapObject->AllocateStorage(textureUploadTarget, level, {{width, height, depth}, internalBytes}); textureMipmapObject->AllocateStorage(textureUploadTarget, level, {{width, height, depth}, internalBytes});
@@ -2345,7 +2345,7 @@ namespace MobileGL::MG_Impl::GLImpl {
// Allocate in TextureObject // Allocate in TextureObject
if (isProxy) { if (isProxy) {
MGLOG_D("%s: isProxy = true, not allocating", __func__); TextureImpl::pProxyTextureManager->RecordProxyLevel(textureUploadTarget, level, {width, height, 1});
} else { } else {
MGLOG_D("%s: Allocating %d bytes at mip %d", __func__, internalBytes, level); MGLOG_D("%s: Allocating %d bytes at mip %d", __func__, internalBytes, level);
DiscardMipmapChainOnBaseRespecification(textureMipmapObject, textureUploadTarget, level); DiscardMipmapChainOnBaseRespecification(textureMipmapObject, textureUploadTarget, level);
@@ -2460,7 +2460,9 @@ namespace MobileGL::MG_Impl::GLImpl {
MOBILEGL_ASSERT(nullptr != static_cast<MG_State::GLState::TextureObjectMipmap*>(textureObject.get()), MOBILEGL_ASSERT(nullptr != static_cast<MG_State::GLState::TextureObjectMipmap*>(textureObject.get()),
"Texture object here should always be an object with mipmap"); "Texture object here should always be an object with mipmap");
auto textureMipmapObject = static_cast<MG_State::GLState::TextureObjectMipmap*>(textureObject.get()); auto textureMipmapObject = static_cast<MG_State::GLState::TextureObjectMipmap*>(textureObject.get());
if (!isProxy) { if (isProxy) {
TextureImpl::pProxyTextureManager->RecordProxyLevel(textureUploadTarget, level, {width, 1, 1});
} else {
DiscardMipmapChainOnBaseRespecification(textureMipmapObject, textureUploadTarget, level); DiscardMipmapChainOnBaseRespecification(textureMipmapObject, textureUploadTarget, level);
textureMipmapObject->AllocateStorage(textureUploadTarget, level, {{width, 1, 1}, internalBytes}); textureMipmapObject->AllocateStorage(textureUploadTarget, level, {{width, 1, 1}, internalBytes});
// After AllocateStorage, which clears the tag. No block-compressed format has a 1D // After AllocateStorage, which clears the tag. No block-compressed format has a 1D
@@ -2905,7 +2907,12 @@ namespace MobileGL::MG_Impl::GLImpl {
case GL_TEXTURE_HEIGHT: case GL_TEXTURE_HEIGHT:
case GL_TEXTURE_DEPTH: { case GL_TEXTURE_DEPTH: {
const SizeT axis = pname == GL_TEXTURE_WIDTH ? 0 : (pname == GL_TEXTURE_HEIGHT ? 1 : 2); const SizeT axis = pname == GL_TEXTURE_WIDTH ? 0 : (pname == GL_TEXTURE_HEIGHT ? 1 : 2);
if (storageType == TextureStorageType::Mipmap && mipmapObject != nullptr) { // A proxy level never took storage; its size lives with the proxy manager.
const auto proxySize =
isProxy ? TextureImpl::pProxyTextureManager->GetProxyLevelTexelSize(uploadTarget, level) : std::nullopt;
if (proxySize) {
value = (*proxySize)[axis];
} else if (storageType == TextureStorageType::Mipmap && mipmapObject != nullptr) {
value = mipmapObject->GetMipmapTexelSize(uploadTarget, level)[axis]; value = mipmapObject->GetMipmapTexelSize(uploadTarget, level)[axis];
} else if (storageType == TextureStorageType::Buffer) { } else if (storageType == TextureStorageType::Buffer) {
// A buffer texture is one-dimensional. // A buffer texture is one-dimensional.
@@ -3988,10 +3995,13 @@ namespace MobileGL::MG_Impl::GLImpl {
MG_Util::ConvertGLEnumToTextureInternalFormat(internalformat); MG_Util::ConvertGLEnumToTextureInternalFormat(internalformat);
textureObject->SetInternalFormat(textureInternalFormat); textureObject->SetInternalFormat(textureInternalFormat);
// A proxy records the format and nothing else - it must never take storage, and it must never // A proxy records the format and the level size and nothing else - it must never take
// be tagged compressed, or GL_TEXTURE_COMPRESSED_IMAGE_SIZE on a proxy would stop being // storage, and it must never be tagged compressed, or GL_TEXTURE_COMPRESSED_IMAGE_SIZE on a
// INVALID_OPERATION. // proxy would stop being INVALID_OPERATION.
if (isProxy) return; if (isProxy) {
TextureImpl::pProxyTextureManager->RecordProxyLevel(textureUploadTarget, level, {width, height, depth});
return;
}
const SizeT internalBpp = const SizeT internalBpp =
MG_Util::GetInternalBytesPerPixel(textureInternalFormat, TexturePixelDataType::UnsignedByte); MG_Util::GetInternalBytesPerPixel(textureInternalFormat, TexturePixelDataType::UnsignedByte);
@@ -4072,10 +4082,13 @@ namespace MobileGL::MG_Impl::GLImpl {
MG_Util::ConvertGLEnumToTextureInternalFormat(internalformat); MG_Util::ConvertGLEnumToTextureInternalFormat(internalformat);
textureObject->SetInternalFormat(textureInternalFormat); textureObject->SetInternalFormat(textureInternalFormat);
// A proxy records the format and nothing else - it must never take storage, and it must never // A proxy records the format and the level size and nothing else - it must never take
// be tagged compressed, or GL_TEXTURE_COMPRESSED_IMAGE_SIZE on a proxy would stop being // storage, and it must never be tagged compressed, or GL_TEXTURE_COMPRESSED_IMAGE_SIZE on a
// INVALID_OPERATION. // proxy would stop being INVALID_OPERATION.
if (isProxy) return; if (isProxy) {
TextureImpl::pProxyTextureManager->RecordProxyLevel(textureUploadTarget, level, {width, height, 1});
return;
}
const SizeT internalBpp = const SizeT internalBpp =
MG_Util::GetInternalBytesPerPixel(textureInternalFormat, TexturePixelDataType::UnsignedByte); MG_Util::GetInternalBytesPerPixel(textureInternalFormat, TexturePixelDataType::UnsignedByte);
@@ -7,6 +7,7 @@
// End of Source File Header // End of Source File Header
#include "ProxyTexture.h" #include "ProxyTexture.h"
#include <MG_Backend/BackendObjects.h>
#include <MG_State/GLState/TextureState/TextureObject1D.h> #include <MG_State/GLState/TextureState/TextureObject1D.h>
#include <MG_State/GLState/TextureState/TextureObject2D.h> #include <MG_State/GLState/TextureState/TextureObject2D.h>
#include <MG_State/GLState/TextureState/TextureObject2DCube.h> #include <MG_State/GLState/TextureState/TextureObject2DCube.h>
@@ -87,4 +88,51 @@ namespace MobileGL::MG_Impl::GLImpl::TextureImpl {
static SharedPtr<MG_State::GLState::ITextureObject> nullTextureObject = nullptr; static SharedPtr<MG_State::GLState::ITextureObject> nullTextureObject = nullptr;
return nullTextureObject; return nullTextureObject;
} }
namespace {
// Whether the implementation can hold a level of this size on this proxy target. Every
// dimension of a mipmapped image shrinks with the level; array layers do not.
Bool ProxyLevelFits(TextureUploadTarget target, GLint level, IntVec3 size) {
MG_Backend::DynamicBackendParameters limits{};
if (MG_Backend::pActiveBackendObject) limits = MG_Backend::pActiveBackendObject->GetDynamicParameters();
const auto levelLimit = [level](Int limit) { return level < 31 ? std::max(limit >> level, 1) : 1; };
const Int planar = levelLimit(limits.MaxTextureSize);
const Int cube = levelLimit(limits.MaxCubeMapTextureSize);
const Int volume = levelLimit(limits.Max3DTextureSize);
const Int layers = limits.MaxArrayTextureLayers;
switch (target) {
case TextureUploadTarget::ProxyTexture1D:
return size.x() <= planar;
case TextureUploadTarget::ProxyTexture1DArray:
return size.x() <= planar && size.y() <= layers;
case TextureUploadTarget::ProxyTexture2DArray:
return size.x() <= planar && size.y() <= planar && size.z() <= layers;
case TextureUploadTarget::ProxyTexture3D:
return size.x() <= volume && size.y() <= volume && size.z() <= volume;
case TextureUploadTarget::ProxyCubeMap:
return size.x() <= cube && size.y() <= cube;
case TextureUploadTarget::ProxyCubeMapArray:
return size.x() <= cube && size.y() <= cube && size.z() <= layers;
case TextureUploadTarget::ProxyTextureRectangle:
return level == 0 && size.x() <= limits.MaxTextureSize && size.y() <= limits.MaxTextureSize;
default:
return size.x() <= planar && size.y() <= planar;
}
}
} // namespace
void ProxyTextureManager::RecordProxyLevel(TextureUploadTarget target, GLint level, IntVec3 size) {
if (level < 0) return;
auto& levels = m_proxyLevelSizes[target];
if (levels.size() <= static_cast<SizeT>(level)) levels.resize(static_cast<SizeT>(level) + 1);
levels[level] = ProxyLevelFits(target, level, size) ? size : IntVec3{0, 0, 0};
}
Optional<IntVec3> ProxyTextureManager::GetProxyLevelTexelSize(TextureUploadTarget target, GLint level) const {
auto it = m_proxyLevelSizes.find(target);
if (it == m_proxyLevelSizes.end() || level < 0 || it->second.size() <= static_cast<SizeT>(level)) {
return std::nullopt;
}
return it->second[level];
}
} // namespace MobileGL::MG_Impl::GLImpl::TextureImpl } // namespace MobileGL::MG_Impl::GLImpl::TextureImpl
@@ -19,8 +19,15 @@ namespace MobileGL::MG_Impl::GLImpl::TextureImpl {
TextureUploadTarget target); TextureUploadTarget target);
const SharedPtr<MG_State::GLState::ITextureObject>& GetProxyTextureObject(TextureUploadTarget target); const SharedPtr<MG_State::GLState::ITextureObject>& GetProxyTextureObject(TextureUploadTarget target);
// The image a proxy glTexImage* or glCompressedTexImage* describes, kept here because a proxy
// never takes storage. A request past the implementation limits records all zeros (GL 4.6
// core 8.22), which is how glGetTexLevelParameter tells the application it cannot be had.
void RecordProxyLevel(TextureUploadTarget target, GLint level, IntVec3 size);
Optional<IntVec3> GetProxyLevelTexelSize(TextureUploadTarget target, GLint level) const;
private: private:
UnorderedMap<TextureUploadTarget, SharedPtr<MG_State::GLState::ITextureObject>> m_proxyTexturesMap; UnorderedMap<TextureUploadTarget, SharedPtr<MG_State::GLState::ITextureObject>> m_proxyTexturesMap;
UnorderedMap<TextureUploadTarget, Vector<Optional<IntVec3>>> m_proxyLevelSizes;
}; };
extern UniquePtr<ProxyTextureManager>& pProxyTextureManager; extern UniquePtr<ProxyTextureManager>& pProxyTextureManager;
+14 -2
View File
@@ -2602,8 +2602,20 @@ namespace MobileGL::MG_Pipe {
// boundary reads unpack state, and the staged-repack upload path does not even issue // boundary reads unpack state, and the staged-repack upload path does not even issue
// glPixelStorei (ARCHITECTURE.md 4.6 D5). // glPixelStorei (ARCHITECTURE.md 4.6 D5).
Uint64 EmitPixelPackState(GLContext& ctx) { Uint64 EmitPixelPackState(GLContext& ctx) {
MGPPixelPackState pack{}; // Field by field into zeroed bytes: the two bytes of padding after the Bools would
pack.Pack = ctx.GetPixelStoreParameters(false); // otherwise carry whatever the context's heap block held, onto the wire and into the
// recorder's hash.
MGPPixelPackState pack;
std::memset(&pack, 0, sizeof(pack));
const PixelStoreParameters live = ctx.GetPixelStoreParameters(false);
pack.Pack.SwapBytes = live.SwapBytes;
pack.Pack.LSBFirst = live.LSBFirst;
pack.Pack.RowLength = live.RowLength;
pack.Pack.ImageHeight = live.ImageHeight;
pack.Pack.SkipPixels = live.SkipPixels;
pack.Pack.SkipRows = live.SkipRows;
pack.Pack.SkipImages = live.SkipImages;
pack.Pack.Alignment = live.Alignment;
MGPipeRouteSetPixelPackState(pack); MGPipeRouteSetPixelPackState(pack);
return sizeof(MGPPixelPackState); return sizeof(MGPPixelPackState);
} }
@@ -7,9 +7,11 @@ Configuration-specific controls are selected separately and must pass without sk
## Unresolved behavior ## Unresolved behavior
- `ClipDistanceScenario.ADisabledClipDistanceRemovesNothing` and - `ClipDistanceScenario.ADisabledClipDistanceRemovesNothing` and
`TheEnablesAreIndependentPerDistance`: per-distance enable semantics are missing `TheEnablesAreIndependentPerDistance`: resolved on both backends. Espryt on
on the tested GLES and Vulkan paths. Neither backend supplies a passing control llvmpipe emulates clip distances (its native probe catches llvmpipe clipping by
on local llvmpipe/lavapipe. The strict list names the GLES and Magma Fm entries. disabled distances); Magma on lavapipe applies GL_CLIP_DISTANCEi in the shader
(Vulkan has no per-distance enable state). The entries stay listed only so a
stack that ever skips them again keeps a documented reason.
- `PartialUploadOverGpuContentScenario.OneDimensionalArraySubImageKeepsTheGpuClearedLayer`: - `PartialUploadOverGpuContentScenario.OneDimensionalArraySubImageKeepsTheGpuClearedLayer`:
the Vulkan path cannot attach the 1D array layer. GLES passes, which does not the Vulkan path cannot attach the 1D array layer. GLES passes, which does not
establish Vulkan coverage. establish Vulkan coverage.
@@ -26,16 +26,15 @@
// identical distances and the enable turned off must leave both sides painted, which is what // identical distances and the enable turned off must leave both sides painted, which is what
// says the pixels below are being removed by clipping and not by something else. // says the pixels below are being removed by clipping and not by something else.
// //
// HONEST LIMIT OF THIS FILE IN CI. Of the four cases, only EnableIsObservableThroughIsEnabled is // WHAT CI CAN AND CANNOT FALSIFY. llvmpipe's ES driver clips by EVERY written gl_ClipDistance
// falsifiable on the software rasterizers every automated lane runs on. llvmpipe and lavapipe // regardless of the enables, which the DirectGLES loader's native clip probe catches - so on
// clip by EVERY declared gl_ClipDistance regardless of the enables, so // that lane Espryt EMULATES clip distances (EmulateClipDistancePass: varyings, a fragment
// AnEnabledClipDistanceRemovesTheNegativeHalf goes green there against the broken tree as well, // discard and a per-draw enables uniform), and the per-distance cases below run and pass
// and the two cases that need real per-distance semantics skip (see // there. That is the emulation every Mali device takes. lavapipe hosts shaderClipDistance, so
// DriverHonorsPerDistanceEnables). What actually pins the behavior is Adreno, through // Magma clips natively; Vulkan has no per-distance enable state, so Magma applies GL_CLIP_DISTANCEi in
// KHR-GLxx.clip_distance.functional - whose "without dynamic redeclaration" variants declare all // the producing stage (BackendCaps::MasksNativeClipDistances) and the per-distance cases run
// gl_MaxClipDistances slots and enable only the first N, i.e. exactly the subset semantics these // green there too. The native ES path - Adreno, which honors GL_CLIP_DISTANCEi_EXT - is pinned on
// skipped cases assert. Read a green CI run here as "the state survives the frontend", not as // the device, through KHR-GLxx.clip_distance.functional.
// "clipping is correct"; the second claim is a device claim.
// //
// Every case disables all eight distances on entry rather than assuming they start off: // Every case disables all eight distances on entry rather than assuming they start off:
// XfbAfterClipDistanceScenario deliberately leaves one enabled for the rest of the process, and // XfbAfterClipDistanceScenario deliberately leaves one enabled for the rest of the process, and
@@ -83,13 +82,33 @@ void main() {
const char* const kFragmentSource = R"(#version 400 core const char* const kFragmentSource = R"(#version 400 core
out vec4 fragColor; out vec4 fragColor;
void main() { fragColor = vec4(0.0, 1.0, 0.0, 1.0); } void main() { fragColor = vec4(0.0, 1.0, 0.0, 1.0); }
)";
// The field report's shape (Mali-G615 / Espryt, where GL_MAX_CLIP_DISTANCES used to be 0
// and this did not even compile): a #version 330 core stage writing all eight distances
// from uniform planes, implicitly sized by use.
const char* const kEightPlaneVertexSource = R"(#version 330 core
uniform vec4 u_ClipPlane[8];
void main() {
vec2 positions[3] = vec2[3](vec2(-1.0, -1.0), vec2(3.0, -1.0), vec2(-1.0, 3.0));
vec4 eyePos = vec4(positions[gl_VertexID], 0.0, 1.0);
gl_Position = eyePos;
gl_ClipDistance[0] = dot(u_ClipPlane[0], eyePos);
gl_ClipDistance[1] = dot(u_ClipPlane[1], eyePos);
gl_ClipDistance[2] = dot(u_ClipPlane[2], eyePos);
gl_ClipDistance[3] = dot(u_ClipPlane[3], eyePos);
gl_ClipDistance[4] = dot(u_ClipPlane[4], eyePos);
gl_ClipDistance[5] = dot(u_ClipPlane[5], eyePos);
gl_ClipDistance[6] = dot(u_ClipPlane[6], eyePos);
gl_ClipDistance[7] = dot(u_ClipPlane[7], eyePos);
}
)"; )";
class ClipDistanceScenario : public ScenarioTest { class ClipDistanceScenario : public ScenarioTest {
protected: protected:
GLuint BuildProgram() { GLuint BuildProgram(const char* vertexSource = kVertexSource) {
const GLuint vs = glCreateShader(GL_VERTEX_SHADER); const GLuint vs = glCreateShader(GL_VERTEX_SHADER);
glShaderSource(vs, 1, &kVertexSource, nullptr); glShaderSource(vs, 1, &vertexSource, nullptr);
glCompileShader(vs); glCompileShader(vs);
GLint compiled = 0; GLint compiled = 0;
glGetShaderiv(vs, GL_COMPILE_STATUS, &compiled); glGetShaderiv(vs, GL_COMPILE_STATUS, &compiled);
@@ -178,11 +197,13 @@ void main() { fragColor = vec4(0.0, 1.0, 0.0, 1.0); }
// always. DirectVulkan therefore clips by a disabled distance. // always. DirectVulkan therefore clips by a disabled distance.
// - llvmpipe's ES driver behaves like Vulkan here. // - llvmpipe's ES driver behaves like Vulkan here.
// //
// Emulating GL's semantics on those two would mean forcing the disabled slots to a // EmulateClipDistancePass gives those semantics to every backend that EMULATES clip
// non-negative value inside the shader, which makes the enable mask part of the // distances (the disabled slots are forced to +1 from a per-draw uniform, so nothing
// pipeline key - a feature, not a fix, and deliberately not attempted here. The // joins the pipeline key), and the DirectGLES loader arms that emulation on a driver
// cases that need the real semantics gate on this probe and say so when they skip, // whose native clip distances ignore the enables. Magma's native path applies them in
// rather than being deleted or silently weakened. // the producing stage (MasksNativeClipDistances). The cases that need the real
// semantics still gate on this probe and say so if a stack ever skips, rather than
// being deleted or silently weakened.
bool DriverHonorsPerDistanceEnables(GLuint program, GLuint vao) const { bool DriverHonorsPerDistanceEnables(GLuint program, GLuint vao) const {
for (int i = 0; i < 8; ++i) { for (int i = 0; i < 8; ++i) {
glDisable(static_cast<GLenum>(GL_CLIP_DISTANCE0 + i)); glDisable(static_cast<GLenum>(GL_CLIP_DISTANCE0 + i));
@@ -403,4 +424,91 @@ void main() { fragColor = vec4(0.0, 1.0, 0.0, 1.0); }
gl.EndFrame(); gl.EndFrame();
} }
// GL 3.0+ requires eight clip distances, and the field report was a stack answering zero: a
// #version 330 core stage writing gl_ClipDistance[0..7] did not compile. Every backend now
// either hosts eight or emulates them (EmulateClipDistancePass), so this case requires eight
// and does not skip. Distance 2 clips the left half, the other seven are positive, and all
// eight are enabled - which clips identically whether or not the stack honors per-distance
// enables. The per-distance half (distance 5 negative on the right but DISABLED) only runs
// where the stack honors them, which every emulating backend does by construction.
TEST_F(ClipDistanceScenario, EightDistancesFromUniformPlanesClipLikeTheReport) {
if (!Ready()) return;
GLint maxClipDistances = 0;
glGetIntegerv(GL_MAX_CLIP_DISTANCES, &maxClipDistances);
ASSERT_GE(maxClipDistances, 8) << "GL 3.0+ requires GL_MAX_CLIP_DISTANCES >= 8";
HeadlessGL& gl = Gl();
const int width = gl.Width();
const int height = gl.Height();
ASSERT_GE(width, 8);
ASSERT_GE(height, 8);
GLuint vao = 0;
glGenVertexArrays(1, &vao);
const GLuint program = BuildProgram(kEightPlaneVertexSource);
ASSERT_NE(program, 0u) << "the eight-plane gl_ClipDistance program did not build: " << BuildLog();
const GLint planes = glGetUniformLocation(program, "u_ClipPlane");
ASSERT_GE(planes, 0);
BindDefaultFramebuffer();
glViewport(0, 0, width, height);
glDisable(GL_SCISSOR_TEST);
glDisable(GL_DEPTH_TEST);
glDisable(GL_CULL_FACE);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
// w = 1 everywhere, so (0,0,0,1) is a distance of +1 and keeps everything.
GLfloat values[8 * 4];
for (int i = 0; i < 8; ++i) {
values[i * 4 + 0] = 0.0f;
values[i * 4 + 1] = 0.0f;
values[i * 4 + 2] = 0.0f;
values[i * 4 + 3] = 1.0f;
}
values[2 * 4 + 0] = 1.0f; // distance 2 = x: negative on the left half
values[2 * 4 + 3] = 0.0f;
values[5 * 4 + 0] = -1.0f; // distance 5 = -x: negative on the right half
values[5 * 4 + 3] = 0.0f;
glUseProgram(program);
glUniform4fv(planes, 8, values);
DisableEveryClipDistance();
for (int i = 0; i < 8; ++i) {
if (i != 5) glEnable(static_cast<GLenum>(GL_CLIP_DISTANCE0 + i));
}
// First with distance 5 enabled too: both halves clipped, by different distances.
glEnable(static_cast<GLenum>(GL_CLIP_DISTANCE0 + 5));
DrawClippedTriangle(program, vao);
unsigned char left[4] = {0, 0, 0, 0};
unsigned char right[4] = {0, 0, 0, 0};
PixelAt(width / 4, height / 2, left);
PixelAt(width - 1 - width / 4, height / 2, right);
EXPECT_EQ(FirstGLError(), 0u);
EXPECT_TRUE(IsRed(left)) << "distance 2 is negative on the left and enabled";
EXPECT_TRUE(IsRed(right)) << "distance 5 is negative on the right and enabled";
// Then distance 5 disabled: on a stack with per-distance enables the right half returns.
glDisable(static_cast<GLenum>(GL_CLIP_DISTANCE0 + 5));
DrawClippedTriangle(program, vao);
PixelAt(width / 4, height / 2, left);
PixelAt(width - 1 - width / 4, height / 2, right);
EXPECT_EQ(FirstGLError(), 0u);
EXPECT_TRUE(IsRed(left)) << "distance 2 is still enabled";
const bool rightKept = IsGreen(right);
DisableEveryClipDistance();
const GLuint control = BuildProgram();
const bool honorsEnables = control != 0 && DriverHonorsPerDistanceEnables(control, vao);
if (control != 0) glDeleteProgram(control);
if (honorsEnables) {
EXPECT_TRUE(rightKept) << "distance 5 is negative on the right but GL_CLIP_DISTANCE5 is disabled, so the "
"right half must survive";
}
glUseProgram(0);
glBindVertexArray(0);
glDeleteProgram(program);
glDeleteVertexArrays(1, &vao);
gl.EndFrame();
}
} // namespace MGITest } // namespace MGITest
@@ -357,4 +357,35 @@ void main() { o0 = texture(uPalette, vec2(0.5, 0.5)); }
glDeleteProgram(paletteProgram); glDeleteProgram(paletteProgram);
} }
// A depth clear of the DEFAULT framebuffer while a pass is open on it (a game clearing depth
// before its hand or GUI): the in-pass clear does not take the window, so the clear ends the
// pass and becomes the load op of the next one, which the farther draw must see.
TEST_F(InPassClearScenario, ADefaultFramebufferDepthClearBetweenTestedDrawsDecidesTheSecondDraw) {
if (!Ready()) return;
HeadlessGL& gl = Gl();
const int width = gl.Width(), height = gl.Height();
Build();
for (int frame = 0; frame < 2; ++frame) {
BindDefaultFramebuffer();
glViewport(0, 0, width, height);
glDisable(GL_SCISSOR_TEST);
glDisable(GL_BLEND);
glDepthMask(GL_TRUE);
glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClearDepth(1.0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LESS);
Quad(1.0f, 0.0f, 0.0f, 1.0f, -0.5f);
glClear(GL_DEPTH_BUFFER_BIT);
Quad(0.0f, 1.0f, 0.0f, 1.0f, 0.0f);
glDisable(GL_DEPTH_TEST);
EXPECT_EQ(FirstGLError(), 0u);
const Image image = ReadPixels(width, height);
EXPECT_TRUE(Near(image.At(width / 2, height / 2), 0, 255, 0, 255))
<< "frame " << frame << ": the depth clear between the draws let the farther draw pass";
gl.EndFrame();
}
}
} // namespace MGITest } // namespace MGITest
+1 -1
View File
@@ -389,7 +389,7 @@
F(FragmentInterpolationOffsetBits) F(SupportsWideLines) \ F(FragmentInterpolationOffsetBits) F(SupportsWideLines) \
F(SupportsDistinctDepthStencilAttachments) F(PerLayerFramebufferAttachmentTargets) \ F(SupportsDistinctDepthStencilAttachments) F(PerLayerFramebufferAttachmentTargets) \
F(SupportsShaderFloat64) F(SupportsFloat64VertexAttributes) F(SupportsTessellationPointSize) \ F(SupportsShaderFloat64) F(SupportsFloat64VertexAttributes) F(SupportsTessellationPointSize) \
F(SupportsGeometryPointSize) F(MaxShaderStorageBlockSize) F(SubgroupSize) \ F(SupportsGeometryPointSize) F(EmulatesClipDistance) F(MasksNativeClipDistances) F(MaxShaderStorageBlockSize) F(SubgroupSize) \
F(SubgroupSupportedStages) F(SubgroupSupportedFeatures) F(SamplePositions) \ F(SubgroupSupportedStages) F(SubgroupSupportedFeatures) F(SamplePositions) \
F(SamplePositionCountMask) F(SubgroupQuadOperationsInAllStages) \ F(SamplePositionCountMask) F(SubgroupQuadOperationsInAllStages) \
F(GpuVendor) F(SupportsFragmentShadingRate) F(SupportsPrimitiveFragmentShadingRate) \ F(GpuVendor) F(SupportsFragmentShadingRate) F(SupportsPrimitiveFragmentShadingRate) \
@@ -660,6 +660,8 @@ inline constexpr WireLayoutMember kMGPipeWireLayoutMembers[] = {
{"DynamicBackendParameters.SupportsFloat64VertexAttributes", offsetof(DynamicBackendParameters, SupportsFloat64VertexAttributes), sizeof(((DynamicBackendParameters*)nullptr)->SupportsFloat64VertexAttributes)}, {"DynamicBackendParameters.SupportsFloat64VertexAttributes", offsetof(DynamicBackendParameters, SupportsFloat64VertexAttributes), sizeof(((DynamicBackendParameters*)nullptr)->SupportsFloat64VertexAttributes)},
{"DynamicBackendParameters.SupportsTessellationPointSize", offsetof(DynamicBackendParameters, SupportsTessellationPointSize), sizeof(((DynamicBackendParameters*)nullptr)->SupportsTessellationPointSize)}, {"DynamicBackendParameters.SupportsTessellationPointSize", offsetof(DynamicBackendParameters, SupportsTessellationPointSize), sizeof(((DynamicBackendParameters*)nullptr)->SupportsTessellationPointSize)},
{"DynamicBackendParameters.SupportsGeometryPointSize", offsetof(DynamicBackendParameters, SupportsGeometryPointSize), sizeof(((DynamicBackendParameters*)nullptr)->SupportsGeometryPointSize)}, {"DynamicBackendParameters.SupportsGeometryPointSize", offsetof(DynamicBackendParameters, SupportsGeometryPointSize), sizeof(((DynamicBackendParameters*)nullptr)->SupportsGeometryPointSize)},
{"DynamicBackendParameters.EmulatesClipDistance", offsetof(DynamicBackendParameters, EmulatesClipDistance), sizeof(((DynamicBackendParameters*)nullptr)->EmulatesClipDistance)},
{"DynamicBackendParameters.MasksNativeClipDistances", offsetof(DynamicBackendParameters, MasksNativeClipDistances), sizeof(((DynamicBackendParameters*)nullptr)->MasksNativeClipDistances)},
{"DynamicBackendParameters.MaxShaderStorageBlockSize", offsetof(DynamicBackendParameters, MaxShaderStorageBlockSize), sizeof(((DynamicBackendParameters*)nullptr)->MaxShaderStorageBlockSize)}, {"DynamicBackendParameters.MaxShaderStorageBlockSize", offsetof(DynamicBackendParameters, MaxShaderStorageBlockSize), sizeof(((DynamicBackendParameters*)nullptr)->MaxShaderStorageBlockSize)},
{"DynamicBackendParameters.SubgroupSize", offsetof(DynamicBackendParameters, SubgroupSize), sizeof(((DynamicBackendParameters*)nullptr)->SubgroupSize)}, {"DynamicBackendParameters.SubgroupSize", offsetof(DynamicBackendParameters, SubgroupSize), sizeof(((DynamicBackendParameters*)nullptr)->SubgroupSize)},
{"DynamicBackendParameters.SubgroupSupportedStages", offsetof(DynamicBackendParameters, SubgroupSupportedStages), sizeof(((DynamicBackendParameters*)nullptr)->SubgroupSupportedStages)}, {"DynamicBackendParameters.SubgroupSupportedStages", offsetof(DynamicBackendParameters, SubgroupSupportedStages), sizeof(((DynamicBackendParameters*)nullptr)->SubgroupSupportedStages)},
@@ -337,6 +337,12 @@ namespace MobileGL::MG_State::GLState {
// gate has to be answerable without joining the SPIR-V job, or every draw of every // gate has to be answerable without joining the SPIR-V job, or every draw of every
// program would pay a join to discover it has nothing to write. // program would pay a join to discover it has nothing to write.
Bool usesReservedNumSamples = false; Bool usesReservedNumSamples = false;
// A pre-rasterization stage of this program declared the reserved clip-distance enables
// uniform (ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME), which the source
// pipeline does only on a backend that emulates gl_ClipDistance, so the draw path owes it
// the GL_CLIP_DISTANCE0..7 enables before every draw. Phase A for the reason
// usesReservedNumSamples is.
Bool usesReservedClipDistanceEnables = false;
// A shader of this program declared an `es` #version (ShaderTranspiler::FindEsOrigin). // A shader of this program declared an `es` #version (ShaderTranspiler::FindEsOrigin).
// ESSL fixes an image uniform's unit at its binding qualifier, so glUniform* on one is // ESSL fixes an image uniform's unit at its binding qualifier, so glUniform* on one is
// INVALID_OPERATION for such a program and legal for a desktop one. // INVALID_OPERATION for such a program and legal for a desktop one.
@@ -415,6 +421,8 @@ namespace MobileGL::MG_State::GLState {
// uniformOffsets, because the member has no GL location at all: the link task keeps // uniformOffsets, because the member has no GL location at all: the link task keeps
// it out of the GL-visible uniform index space so no application can see or write it. // it out of the GL-visible uniform index space so no application can see or write it.
Uint reservedNumSamplesOffset = kInvalidUniformOffset; Uint reservedNumSamplesOffset = kInvalidUniformOffset;
// The same, for the reserved clip-distance enables uniform.
Uint reservedClipDistanceEnablesOffset = kInvalidUniformOffset;
// False for a program whose SPIR-V was never produced (phase B canceled at // False for a program whose SPIR-V was never produced (phase B canceled at
// teardown or by a relink) or whose optimizer run failed. GL has no way to // teardown or by a relink) or whose optimizer run failed. GL has no way to
// retract a LINK_STATUS it already reported true, so such a program stays // retract a LINK_STATUS it already reported true, so such a program stays
@@ -443,6 +451,11 @@ namespace MobileGL::MG_State::GLState {
// deliberately keeps answering "gl_PointSize": demotion happens after phase A, // deliberately keeps answering "gl_PointSize": demotion happens after phase A,
// so every query keeps the truthful GL spelling. // so every query keeps the truthful GL spelling.
Bool pointSizeDemoted = false; Bool pointSizeDemoted = false;
// Whether these modules had gl_ClipDistance lowered to varyings plus a fragment discard
// (ShaderCompiler::EmulateClipDistanceForProgram) because the backend cannot clip
// natively. Recorded for the same reason pointSizeDemoted is: the final bytes no longer
// declare the capability that armed it.
Bool clipDistanceEmulated = false;
}; };
// ---- the archive field tables (ARCHITECTURE.md §7): ONE table per type, serving both directions ---- // ---- the archive field tables (ARCHITECTURE.md §7): ONE table per type, serving both directions ----
@@ -579,6 +592,7 @@ namespace MobileGL::MG_State::GLState {
v("uniformBlocksWithoutBinding", a.uniformBlocksWithoutBinding); v("uniformBlocksWithoutBinding", a.uniformBlocksWithoutBinding);
v("activeUniformCount", a.activeUniformCount); v("activeUniformCount", a.activeUniformCount);
v("usesReservedNumSamples", a.usesReservedNumSamples); v("usesReservedNumSamples", a.usesReservedNumSamples);
v("usesReservedClipDistanceEnables", a.usesReservedClipDistanceEnables);
v("esOrigin", a.esOrigin); v("esOrigin", a.esOrigin);
v("maxUniformLocation", a.maxUniformLocation); v("maxUniformLocation", a.maxUniformLocation);
v("uniformNameMaxLength", a.uniformNameMaxLength); v("uniformNameMaxLength", a.uniformNameMaxLength);
@@ -614,10 +628,12 @@ namespace MobileGL::MG_State::GLState {
v("uniformOffsets", a.uniformOffsets); v("uniformOffsets", a.uniformOffsets);
v("globalUboScratch", a.globalUboScratch); v("globalUboScratch", a.globalUboScratch);
v("reservedNumSamplesOffset", a.reservedNumSamplesOffset); v("reservedNumSamplesOffset", a.reservedNumSamplesOffset);
v("reservedClipDistanceEnablesOffset", a.reservedClipDistanceEnablesOffset);
v("spirvStatus", a.spirvStatus); v("spirvStatus", a.spirvStatus);
v("nativeFloat64", a.nativeFloat64); v("nativeFloat64", a.nativeFloat64);
v("pointSizeDemoted", a.pointSizeDemoted); v("pointSizeDemoted", a.pointSizeDemoted);
} // 8 fields v("clipDistanceEmulated", a.clipDistanceEmulated);
} // 10 fields
// ---- trip wires ---- // ---- trip wires ----
// TypeFacts is a POD on every ABI: 13 Bool + 3 bytes of padding + 7 x 4-byte scalars. // TypeFacts is a POD on every ABI: 13 Bool + 3 bytes of padding + 7 x 4-byte scalars.
@@ -635,7 +651,7 @@ namespace MobileGL::MG_State::GLState {
// P7 OQ-8 added storageBlocks, one more Vector, in the disaggregated build ONLY. The pull // P7 OQ-8 added storageBlocks, one more Vector, in the disaggregated build ONLY. The pull
// number below is the one G1 measures and it has not moved. // number below is the one G1 measures and it has not moved.
#define MGL_LINKARTIFACTS_SIZE 1080 #define MGL_LINKARTIFACTS_SIZE 1080
#define MGL_SPIRVARTIFACTS_SIZE 88 #define MGL_SPIRVARTIFACTS_SIZE 96
#elif defined(_LIBCPP_VERSION) && (SIZE_MAX == UINT64_MAX) && defined(MGL_ARTIFACT_SIZES_LIBCXX_PINNED) #elif defined(_LIBCPP_VERSION) && (SIZE_MAX == UINT64_MAX) && defined(MGL_ARTIFACT_SIZES_LIBCXX_PINNED)
// The integrator pins these from the NDK build (brief C.4); until then this branch is inert. // The integrator pins these from the NDK build (brief C.4); until then this branch is inert.
#endif #endif
@@ -1135,7 +1135,9 @@ namespace MobileGL::MG_State::GLState {
for (const ProgramObject::XfbVarying& varying : artifacts.xfbVaryings) { for (const ProgramObject::XfbVarying& varying : artifacts.xfbVaryings) {
if (varying.name == "gl_PointSize") { if (varying.name == "gl_PointSize") {
spirvHandoff.captureRequestsPointSize = true; spirvHandoff.captureRequestsPointSize = true;
break; }
if (varying.name.rfind("gl_ClipDistance", 0) == 0) {
spirvHandoff.captureRequestsClipDistance = true;
} }
} }
// Phase B pairs this with its own SpirvArtifacts to insert the completed front end. // Phase B pairs this with its own SpirvArtifacts to insert the completed front end.
@@ -1501,6 +1503,21 @@ namespace MobileGL::MG_State::GLState {
}; };
for (Int i = 0; i < tProgramUniformCount; i++) { for (Int i = 0; i < tProgramUniformCount; i++) {
const auto& uniform = artifacts.program->getUniform(i); const auto& uniform = artifacts.program->getUniform(i);
// The clip-distance enables InjectClipDistanceEnablesShim declared: MobileGL's own
// per-draw input to the emulation, hidden from the GL surface on the same terms as
// the gl_NumSamples stand-in below. Checked BEFORE the dead-uniform filter on
// purpose: no application code reads it - only EmulateClipDistancePass does, after
// this reflection ran - so glslang reports it with stages == 0, and filtering it as
// dead would leave the draw path never writing it (every distance then selects 1.0
// and nothing clips, with no error anywhere).
if (isGlobalUboMember(uniform) &&
uniform.name == MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME) {
artifacts.usesReservedClipDistanceEnables = true;
MGLOG_D("ProgramObject %u: Reflection - reserved clip-distance enables '%s' hidden from the GL "
"uniform surface",
in.externalIndex, uniform.name.c_str());
continue;
}
if (isGlobalUboMember(uniform) && uniform.stages == 0) { if (isGlobalUboMember(uniform) && uniform.stages == 0) {
MGLOG_D("ProgramObject %u: Reflection - dead default-block uniform '%s' filtered from the GL " MGLOG_D("ProgramObject %u: Reflection - dead default-block uniform '%s' filtered from the GL "
"surface", "surface",
@@ -131,6 +131,10 @@ namespace MobileGL::MG_State::GLState {
// this bit. No new L1 key material: the key already covers // this bit. No new L1 key material: the key already covers
// `requestedXfbVaryings`, of which this is a function. // `requestedXfbVaryings`, of which this is a function.
Bool captureRequestsPointSize = false; Bool captureRequestsPointSize = false;
// The same derived bit for gl_ClipDistance (any element): the clip-distance
// emulation declines such a program, since the capture wants the unmasked value at a
// varying the emulation never creates. Also a function of `requestedXfbVaryings`.
Bool captureRequestsClipDistance = false;
// L1 shader-translation memo key for this program's SPIR-V (see // L1 shader-translation memo key for this program's SPIR-V (see
// MG_Util/ShaderTranspiler/TranslationCache.h). Built HERE, at the tail of phase // MG_Util/ShaderTranspiler/TranslationCache.h). Built HERE, at the tail of phase
@@ -799,6 +799,27 @@ namespace MobileGL::MG_State::GLState {
MarkUBOContentDirtyRange(offset, static_cast<Uint>(sizeof(Int))); MarkUBOContentDirtyRange(offset, static_cast<Uint>(sizeof(Int)));
return true; return true;
} }
// ---- the reserved clip-distance enables (ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME) ----
//
// Same contract as the gl_NumSamples pair above: phase-A gate, value-identical writes
// dropped without a content-version bump, false when there is nowhere to put it.
Bool UsesReservedClipDistanceEnables() const { return Artifacts().usesReservedClipDistanceEnables; }
Bool WriteReservedClipDistanceEnables(Uint32 enables) {
if (!UsesReservedClipDistanceEnables()) return false;
SpirvArtifacts& spirv = Spirv();
const Uint offset = spirv.reservedClipDistanceEnablesOffset;
if (offset == kInvalidUniformOffset) return false;
if (static_cast<SizeT>(offset) + sizeof(Uint32) > spirv.globalUboScratch.size()) return false;
Uint8* const slot = spirv.globalUboScratch.data() + offset;
Uint32 current = 0;
Memcpy(&current, slot, sizeof(Uint32));
if (current == enables) return true;
Memcpy(slot, &enables, sizeof(Uint32));
MarkUBOContentDirtyRange(offset, static_cast<Uint>(sizeof(Uint32)));
return true;
}
Bool ClipDistanceEmulated() const { return Spirv().clipDistanceEmulated; }
// ---- glUniform* inside the phase-A -> phase-B window ---- // ---- glUniform* inside the phase-A -> phase-B window ----
// //
// True while the program is fully linked and fully queryable but its uniform shadow's // True while the program is fully linked and fully queryable but its uniform shadow's
@@ -33,6 +33,12 @@ namespace MobileGL::MG_State::GLState {
} }
std::atomic_flag g_pointSizeDeclineReported; std::atomic_flag g_pointSizeDeclineReported;
std::atomic_flag g_pointSizeOptimizerFailureReported; std::atomic_flag g_pointSizeOptimizerFailureReported;
std::atomic_flag g_clipDistanceDeclineReported;
std::atomic_flag g_clipDistanceOptimizerFailureReported;
std::atomic_flag g_clipDistanceEmulationReported;
// Varying locations the clip carriers may reach. 32 vec4s is what both backends' target
// drivers expose (128 output components); a program past it keeps the decline.
constexpr Uint32 kClipDistanceCarrierLocationBudget = 32;
} // namespace } // namespace
void ProgramSpirvTask::DeferLog(String line, const Int level) { void ProgramSpirvTask::DeferLog(String line, const Int level) {
@@ -154,8 +160,12 @@ namespace MobileGL::MG_State::GLState {
m_phaseA->in.env != nullptr && m_phaseA->in.env->DemotesTessellationPointSize(); m_phaseA->in.env != nullptr && m_phaseA->in.env->DemotesTessellationPointSize();
const Bool demoteGeometryPointSize = const Bool demoteGeometryPointSize =
m_phaseA->in.env != nullptr && m_phaseA->in.env->DemotesGeometryPointSize(); m_phaseA->in.env != nullptr && m_phaseA->in.env->DemotesGeometryPointSize();
// Part of the frontend fingerprint (CompileEnv), so the L1 key already separates the two.
const Bool emulateClipDistance = m_phaseA->in.env != nullptr && m_phaseA->in.env->EmulatesClipDistance();
const Bool maskNativeClipDistance = m_phaseA->in.env != nullptr && m_phaseA->in.env->MasksNativeClipDistances();
GenerateSpirv(handoff, externalIndex, deferOutputValidationForDirectVulkan, enableSpirvValidation, GenerateSpirv(handoff, externalIndex, deferOutputValidationForDirectVulkan, enableSpirvValidation,
nativeFloat64, demoteTessellationPointSize, demoteGeometryPointSize); nativeFloat64, demoteTessellationPointSize, demoteGeometryPointSize, emulateClipDistance,
maskNativeClipDistance);
// GlslangToSpv was the only consumer of the parsed ASTs; everything after this point // GlslangToSpv was the only consumer of the parsed ASTs; everything after this point
// works on the SPIR-V and on the TProgram's own self-contained reflection pool. Drop // works on the SPIR-V and on the TProgram's own self-contained reflection pool. Drop
// them here rather than at the end of the body, which is ~87% of this node's runtime // them here rather than at the end of the body, which is ~87% of this node's runtime
@@ -214,7 +224,8 @@ namespace MobileGL::MG_State::GLState {
void ProgramSpirvTask::GenerateSpirv(const ProgramLinkTask::SpirvHandoff& handoff, const Uint externalIndex, void ProgramSpirvTask::GenerateSpirv(const ProgramLinkTask::SpirvHandoff& handoff, const Uint externalIndex,
const Bool deferOutputValidationForDirectVulkan, const Bool deferOutputValidationForDirectVulkan,
const Bool enableSpirvValidation, const Bool nativeFloat64, const Bool enableSpirvValidation, const Bool nativeFloat64,
const Bool demoteTessellationPointSize, const Bool demoteGeometryPointSize) { const Bool demoteTessellationPointSize, const Bool demoteGeometryPointSize,
const Bool emulateClipDistance, const Bool maskNativeClipDistance) {
/* As we passed first stage compilation/linking, /* As we passed first stage compilation/linking,
* we'll assume all the operations here should * we'll assume all the operations here should
* pass. We may be able to employ some optimizations * pass. We may be able to employ some optimizations
@@ -269,6 +280,51 @@ namespace MobileGL::MG_State::GLState {
} }
artifacts.nativeFloat64 = keepFloat64; artifacts.nativeFloat64 = keepFloat64;
// The clip-distance emulation, program-wide and BEFORE the sanitize chain, on glslang's
// raw output. It cannot run after it: a stage whose only default-block uniform is the
// reserved enables member (nothing in the application reads it - only this pass does)
// has its whole MGL_GLOBAL_UBO deleted by ADCE, and the producer would then fall back
// to clipping by every distance. Running first adds the read that keeps it alive. The
// point-size demotion below places its carrier past these carriers.
artifacts.clipDistanceEmulated = false;
if (emulateClipDistance || maskNativeClipDistance) {
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
if (!ShaderCompiler::EmulateClipDistanceForProgram(
artifacts.generatedSpirv, handoff.shaderTypes, handoff.captureRequestsClipDistance,
kClipDistanceCarrierLocationBudget, outcome, !deferOutputValidationForDirectVulkan,
enableSpirvValidation, maskNativeClipDistance)) {
DeferLog(std::format("ProgramObject {}: clip-distance emulation failed in the optimizer; the "
"program keeps gl_ClipDistance, which this backend cannot host",
externalIndex),
FirstTimeWarnLevel(g_clipDistanceOptimizerFailureReported));
} else if (outcome.emulated && maskNativeClipDistance) {
// Native masking: the built-in is still hosted, only the enables are applied in
// the shader. Not "emulated" for the artifact (nothing reads it differently).
const Bool first = !g_clipDistanceEmulationReported.test_and_set(std::memory_order_relaxed);
DeferLog(std::format("ProgramObject {}: GL_CLIP_DISTANCEi enables applied in the shader to {} "
"native gl_ClipDistance output(s){}",
externalIndex, outcome.distanceCount,
outcome.enablesFound ? "" : "; no enables uniform, every distance clips"),
first ? MOBILEGL_LOG_LEVEL_INFO : MOBILEGL_LOG_LEVEL_DEBUG);
} else if (outcome.emulated) {
artifacts.clipDistanceEmulated = true;
// THE ARMING SIGNAL, INFO and latched: the integration lane that pins
// MOBILEGL_CLIP_DISTANCE_EMULATION=1 asserts on this line, because a healthy
// driver renders the same pixels whether the emulation ran or not.
const Bool first = !g_clipDistanceEmulationReported.test_and_set(std::memory_order_relaxed);
DeferLog(std::format("ProgramObject {}: gl_ClipDistance emulated ({} distance(s) carried to the "
"fragment stage as varyings and clipped by discard{})",
externalIndex, outcome.distanceCount,
outcome.enablesFound ? "" : "; no enables uniform, every distance clips"),
first ? MOBILEGL_LOG_LEVEL_INFO : MOBILEGL_LOG_LEVEL_DEBUG);
} else if (!outcome.declineDetail.empty()) {
DeferLog(std::format("ProgramObject {}: clip-distance emulation declined ({}); the program keeps "
"gl_ClipDistance, which this backend cannot host",
externalIndex, outcome.declineDetail),
FirstTimeWarnLevel(g_clipDistanceDeclineReported));
}
}
// Linked SPIR-V generated, sanitize and optimize it // Linked SPIR-V generated, sanitize and optimize it
Bool allOptimized = true; Bool allOptimized = true;
{ {
@@ -348,6 +404,7 @@ namespace MobileGL::MG_State::GLState {
artifacts.uniformOffsets.clear(); artifacts.uniformOffsets.clear();
artifacts.globalUboScratch.clear(); artifacts.globalUboScratch.clear();
artifacts.reservedNumSamplesOffset = ProgramObject::kInvalidUniformOffset; artifacts.reservedNumSamplesOffset = ProgramObject::kInvalidUniformOffset;
artifacts.reservedClipDistanceEnablesOffset = ProgramObject::kInvalidUniformOffset;
// kInvalidUniformOffset marks locations that end up without global-UBO backing // kInvalidUniformOffset marks locations that end up without global-UBO backing
// (e.g. the optimizer eliminated every use of the uniform); the fallback pass // (e.g. the optimizer eliminated every use of the uniform); the fallback pass
// below gives those locations tail storage so glUniform* always has a target. // below gives those locations tail storage so glUniform* always has a target.
@@ -386,6 +443,10 @@ namespace MobileGL::MG_State::GLState {
// precisely so no application can address it - so the lookup below would find // precisely so no application can address it - so the lookup below would find
// nothing and log it as unbacked. Only the fragment stage declares it, and // nothing and log it as unbacked. Only the fragment stage declares it, and
// every stage's copy sits at the same offset in the one shared global UBO. // every stage's copy sits at the same offset in the one shared global UBO.
if (name == CLIP_DISTANCE_ENABLES_UNIFORM_NAME) {
artifacts.reservedClipDistanceEnablesOffset = offset;
continue;
}
if (name == NUM_SAMPLES_UNIFORM_NAME) { if (name == NUM_SAMPLES_UNIFORM_NAME) {
artifacts.reservedNumSamplesOffset = offset; artifacts.reservedNumSamplesOffset = offset;
MGLOG_D("ProgramObject %u: BuildGlobalUboRouting - reserved gl_NumSamples stand-in '%s' " MGLOG_D("ProgramObject %u: BuildGlobalUboRouting - reserved gl_NumSamples stand-in '%s' "
@@ -67,7 +67,8 @@ namespace MobileGL::MG_State::GLState {
void GenerateSpirv(const ProgramLinkTask::SpirvHandoff& handoff, Uint externalIndex, void GenerateSpirv(const ProgramLinkTask::SpirvHandoff& handoff, Uint externalIndex,
Bool deferOutputValidationForDirectVulkan, Bool enableSpirvValidation, Bool nativeFloat64, Bool deferOutputValidationForDirectVulkan, Bool enableSpirvValidation, Bool nativeFloat64,
Bool demoteTessellationPointSize, Bool demoteGeometryPointSize); Bool demoteTessellationPointSize, Bool demoteGeometryPointSize, Bool emulateClipDistance,
Bool maskNativeClipDistance);
void BuildGlobalUboRouting(const ProgramLinkTask::SpirvHandoff& handoff, Uint externalIndex); void BuildGlobalUboRouting(const ProgramLinkTask::SpirvHandoff& handoff, Uint externalIndex);
// Worker-side MGLOG replacement, replayed by the join on the GL thread. Same reason as // Worker-side MGLOG replacement, replayed by the join on the GL thread. Same reason as
+1
View File
@@ -45,6 +45,7 @@ enable_testing()
add_executable( add_executable(
SanityTest SanityTest
SanityTest.cpp SanityTest.cpp
SanityClipDistanceTest.cpp
) )
target_link_libraries( target_link_libraries(
SanityTest SanityTest
@@ -531,9 +531,9 @@ TEST(ProgramArtifactsCodec, TheTablesVisitEveryMemberExceptTheLiveProgram) {
// P7 OQ-8: 59 members, 58 visited. The 59th is still the live TProgram; the 58th is // P7 OQ-8: 59 members, 58 visited. The 59th is still the live TProgram; the 58th is
// storageBlocks, which follows the record arm's guard (ProgramArtifacts.h). esOrigin then // storageBlocks, which follows the record arm's guard (ProgramArtifacts.h). esOrigin then
// made it 60 and 59. // made it 60 and 59.
EXPECT_EQ(ProgramArtifactsVisitedFieldCount<LinkArtifacts>(), 59u); EXPECT_EQ(ProgramArtifactsVisitedFieldCount<LinkArtifacts>(), 60u);
EXPECT_EQ(ProgramArtifactsVisitedFieldCount<StorageBlockReflection>(), 3u); EXPECT_EQ(ProgramArtifactsVisitedFieldCount<StorageBlockReflection>(), 3u);
EXPECT_EQ(ProgramArtifactsVisitedFieldCount<SpirvArtifacts>(), 8u); EXPECT_EQ(ProgramArtifactsVisitedFieldCount<SpirvArtifacts>(), 10u);
EXPECT_EQ(ProgramArtifactsVisitedFieldCount<ResourceReflection>(), 14u); EXPECT_EQ(ProgramArtifactsVisitedFieldCount<ResourceReflection>(), 14u);
EXPECT_EQ(ProgramArtifactsVisitedFieldCount<XfbVarying>(), 11u); EXPECT_EQ(ProgramArtifactsVisitedFieldCount<XfbVarying>(), 11u);
EXPECT_EQ(ProgramArtifactsVisitedFieldCount<TypeFacts>(), 20u); EXPECT_EQ(ProgramArtifactsVisitedFieldCount<TypeFacts>(), 20u);
@@ -84,8 +84,8 @@ namespace {
// P7 OQ-8 added LinkArtifacts::storageBlocks and its own three-field table; since P13 W5 // P7 OQ-8 added LinkArtifacts::storageBlocks and its own three-field table; since P13 W5
// the member follows the record arm (ProgramArtifacts.h), which every library builds. // the member follows the record arm (ProgramArtifacts.h), which every library builds.
EXPECT_EQ(CountFields<StorageBlockReflection>(), 3u); EXPECT_EQ(CountFields<StorageBlockReflection>(), 3u);
EXPECT_EQ(CountFields<LinkArtifacts>(), 59u); EXPECT_EQ(CountFields<LinkArtifacts>(), 60u);
EXPECT_EQ(CountFields<SpirvArtifacts>(), 8u); EXPECT_EQ(CountFields<SpirvArtifacts>(), 10u);
} }
// A const walk hands the visitor const references, a non-const walk mutable ones: the one // A const walk hands the visitor const references, a non-const walk mutable ones: the one
+121
View File
@@ -0,0 +1,121 @@
// MobileGL - MobileGL/MG_Test/SanityClipDistanceTest.cpp
// Copyright (c) 2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
//
// The clip and cull distance limits each backend advertises: native where the device clips,
// the emulated eight where it does not, and cull only where it is hosted.
#include <gtest/gtest.h>
#include <MG_Backend/DirectGLES/BackendObject_DirectGLES.h>
#include <MG_Backend/DirectVulkan/BackendObject_DirectVulkan.h>
#include <MG_Util/BackendLoaders/OpenGL/Loader.h>
// maxClipDistances is a LIMIT every Vulkan device reports; declaring ClipDistance in a module
// needs the shaderClipDistance FEATURE, which is separate and which VulkanRenderer enables only
// where the physical device has it. Without the feature the device's limit is never forwarded:
// the backend EMULATES the built-in instead (EmulateClipDistancePass), which is what lets it
// answer GL 3.0's minimum of eight on every Mali device. With the feature it keeps its real
// number and the native path.
TEST(DirectVulkanSanity, GatesClipDistancesOnTheShaderClipDistanceFeature) {
using namespace MobileGL;
MG_Backend::DirectVulkan::BackendObject_DirectVulkan backend;
MG_External::VulkanCapabilities caps;
caps.MaxClipDistances = 8;
caps.SupportsShaderClipDistance = false;
caps.MaxClipDistances = 0; // what a Mali device reports alongside the missing feature
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_TRUE(backend.GetDynamicParameters().EmulatesClipDistance);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 8);
EXPECT_FALSE(backend.GetDynamicParameters().MasksNativeClipDistances);
caps.SupportsShaderClipDistance = true;
caps.MaxClipDistances = 8;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_FALSE(backend.GetDynamicParameters().EmulatesClipDistance);
// Vulkan has no per-distance enable state, so the native path masks in the shader.
EXPECT_TRUE(backend.GetDynamicParameters().MasksNativeClipDistances);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 8);
}
// The cull half of the same contract. shaderCullDistance is a SEPARATE feature from
// shaderClipDistance - VulkanRenderer enables each independently - so it gets its own gate, and
// the combined limit is gated on either being present because GL 4.6 core 11.1.3.10 makes it at
// least as large as both halves. These three used to be literal 8s inside BuildTBuiltInResource
// with no device consulted at all, which let glslang accept a gl_CullDistance write that then
// discarded every primitive it touched.
TEST(DirectVulkanSanity, GatesCullDistancesOnTheShaderCullDistanceFeature) {
using namespace MobileGL;
MG_Backend::DirectVulkan::BackendObject_DirectVulkan backend;
MG_External::VulkanCapabilities caps;
caps.MaxClipDistances = 8;
caps.MaxCullDistances = 8;
caps.MaxCombinedClipAndCullDistances = 8;
// Neither feature: cull stays zero (a per-primitive cull has no per-fragment substitute),
// while clip is emulated, so the combined limit describes the emulated eight.
caps.SupportsShaderClipDistance = false;
caps.SupportsShaderCullDistance = false;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
// Clip only: cull stays zero, and the combined limit still describes the clip capacity.
caps.SupportsShaderClipDistance = true;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
caps.SupportsShaderCullDistance = true;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
}
// DirectGLES reaches clip AND cull distances only through GL_EXT_clip_cull_distance, so the
// loader leaves all three at zero without it and the backend forwards that verbatim. Zero is the
// answer that stops a gl_CullDistance shader from reaching an ESSL compiler that would reject it.
TEST(DirectGLESSanity, ForwardsTheProbedClipAndCullDistanceLimits) {
using namespace MobileGL;
MG_Backend::DirectGLES::BackendObject_DirectGLES backend;
MG_External::GLESCapabilities caps;
backend.ApplyGLESCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 0);
caps.SupportsClipDistance = true;
caps.MaxClipDistances = 8;
caps.MaxCullDistances = 8;
caps.MaxCombinedClipAndCullDistances = 8;
backend.ApplyGLESCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
EXPECT_FALSE(backend.GetDynamicParameters().EmulatesClipDistance);
}
// The loader's verdict that the driver cannot clip natively (no extension, or the POST clip probe
// caught it misbehaving) turns the clip limit into the emulated eight - GL 3.0's minimum - while
// cull keeps the driver's own answer: a per-primitive cull has no per-fragment substitute.
TEST(DirectGLESSanity, AnEmulatingLoaderVerdictAdvertisesEightClipDistancesAndNoCull) {
using namespace MobileGL;
MG_Backend::DirectGLES::BackendObject_DirectGLES backend;
MG_External::GLESCapabilities caps; // no GL_EXT_clip_cull_distance, as on every Mali driver
caps.EmulatesClipDistance = true;
backend.ApplyGLESCapabilitiesForTesting(caps);
EXPECT_TRUE(backend.GetDynamicParameters().EmulatesClipDistance);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
}
-78
View File
@@ -696,84 +696,6 @@ TEST(DirectGLESSanity, PreservesHostPerStageImageUniformLimits) {
EXPECT_EQ(params.MaxComputeImageUniforms, 5); EXPECT_EQ(params.MaxComputeImageUniforms, 5);
} }
// maxClipDistances is a LIMIT every Vulkan device reports; declaring ClipDistance in a module
// needs the shaderClipDistance FEATURE, which is separate and which VulkanRenderer enables only
// where the physical device has it. Forwarding the limit without the feature advertises eight
// clip planes no shader may use - the same shape as the image-uniform limits above, and the same
// shape as the GL_EXT_clip_cull_distance lie on DirectGLES. Not a blanket zero: a device WITH the
// feature keeps its real number.
TEST(DirectVulkanSanity, GatesClipDistancesOnTheShaderClipDistanceFeature) {
using namespace MobileGL;
MG_Backend::DirectVulkan::BackendObject_DirectVulkan backend;
MG_External::VulkanCapabilities caps;
caps.MaxClipDistances = 8;
caps.SupportsShaderClipDistance = false;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 0);
caps.SupportsShaderClipDistance = true;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 8);
}
// The cull half of the same contract. shaderCullDistance is a SEPARATE feature from
// shaderClipDistance - VulkanRenderer enables each independently - so it gets its own gate, and
// the combined limit is gated on either being present because GL 4.6 core 11.1.3.10 makes it at
// least as large as both halves. These three used to be literal 8s inside BuildTBuiltInResource
// with no device consulted at all, which let glslang accept a gl_CullDistance write that then
// discarded every primitive it touched.
TEST(DirectVulkanSanity, GatesCullDistancesOnTheShaderCullDistanceFeature) {
using namespace MobileGL;
MG_Backend::DirectVulkan::BackendObject_DirectVulkan backend;
MG_External::VulkanCapabilities caps;
caps.MaxClipDistances = 8;
caps.MaxCullDistances = 8;
caps.MaxCombinedClipAndCullDistances = 8;
caps.SupportsShaderClipDistance = false;
caps.SupportsShaderCullDistance = false;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 0);
// Clip only: cull stays zero, and the combined limit still describes the clip capacity.
caps.SupportsShaderClipDistance = true;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
caps.SupportsShaderCullDistance = true;
backend.ApplyVulkanCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
}
// DirectGLES reaches clip AND cull distances only through GL_EXT_clip_cull_distance, so the
// loader leaves all three at zero without it and the backend forwards that verbatim. Zero is the
// answer that stops a gl_CullDistance shader from reaching an ESSL compiler that would reject it.
TEST(DirectGLESSanity, ForwardsTheProbedClipAndCullDistanceLimits) {
using namespace MobileGL;
MG_Backend::DirectGLES::BackendObject_DirectGLES backend;
MG_External::GLESCapabilities caps;
backend.ApplyGLESCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 0);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 0);
caps.SupportsClipDistance = true;
caps.MaxClipDistances = 8;
caps.MaxCullDistances = 8;
caps.MaxCombinedClipAndCullDistances = 8;
backend.ApplyGLESCapabilitiesForTesting(caps);
EXPECT_EQ(backend.GetDynamicParameters().MaxClipDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCullDistances, 8);
EXPECT_EQ(backend.GetDynamicParameters().MaxCombinedClipAndCullDistances, 8);
}
// GL_LAYER_PROVOKING_VERTEX / GL_VIEWPORT_INDEX_PROVOKING_VERTEX were a hard-coded // GL_LAYER_PROVOKING_VERTEX / GL_VIEWPORT_INDEX_PROVOKING_VERTEX were a hard-coded
// GL_LAST_VERTEX_CONVENTION for both backends, derived from nothing, and wrong on both test // GL_LAST_VERTEX_CONVENTION for both backends, derived from nothing, and wrong on both test
// devices in opposite directions. DirectGLES now forwards what its loader resolved; DirectVulkan // devices in opposite directions. DirectGLES now forwards what its loader resolved; DirectVulkan
@@ -31,6 +31,7 @@ add_executable(
WidenImageFormatsTest.cpp WidenImageFormatsTest.cpp
GlslangCaptureTest.cpp GlslangCaptureTest.cpp
DemotePointSizeTest.cpp DemotePointSizeTest.cpp
EmulateClipDistanceTest.cpp
FragCoordYFlipTest.cpp FragCoordYFlipTest.cpp
PixelCenterIntegerTest.cpp PixelCenterIntegerTest.cpp
ZeroLayerOutputTest.cpp ZeroLayerOutputTest.cpp
@@ -0,0 +1,441 @@
// MobileGL - MobileGL/MG_Test/ShaderTranspiler/EmulateClipDistanceTest.cpp
// Copyright (c) 2025-2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
#include <gtest/gtest.h>
#include <string>
#include <vector>
#include "Includes.h"
#include "Init.h"
#include <MG_Util/ShaderTranspiler/CompileEnv.h>
#include <MG_Util/ShaderTranspiler/ShaderCompiler.h>
#include <MG_Util/ShaderTranspiler/ShaderSourceProcessor.h>
#include <MG_Util/ShaderTranspiler/SpvcSession.h>
#include <MG_Util/ShaderTranspiler/Types.h>
#include "spirv-tools/libspirv.hpp"
using namespace MobileGL;
using MobileGL::MG_Util::ShaderTranspiler::CompileEnv;
using MobileGL::MG_Util::ShaderTranspiler::SessionUsageBit;
using MobileGL::MG_Util::ShaderTranspiler::ShaderCompiler;
using MobileGL::MG_Util::ShaderTranspiler::SpvcSession;
namespace {
// The exact bytes ProgramSpirvTask hands the emulation: linked, generated, sanitized.
Vector<Vector<Uint32>> CompileProgramToSpirv(const Vector<Pair<GLenum, const char*>>& stages,
Vector<GLenum>& types) {
using namespace MG_Util::ShaderTranspiler;
Vector<SharedPtr<glslang::TShader>> shaders;
types.clear();
for (const auto& [stage, source] : stages) {
ShaderAttrib shaderAttrib{.shaderType = stage, .sourceStr = source};
auto shaderResult = ShaderCompiler::CompileShader(shaderAttrib);
EXPECT_TRUE(shaderResult) << (shaderResult ? String{} : shaderResult.error().log);
if (!shaderResult) return {};
shaders.push_back(shaderResult.value());
types.push_back(stage);
}
ProgramAttrib programAttrib{.shaders = shaders};
auto programResult = ShaderCompiler::LinkProgram(programAttrib);
EXPECT_TRUE(programResult) << (programResult ? String{} : programResult.error().log);
if (!programResult) return {};
ProgramBinaryAttrib binaryAttrib{.shaderTypes = types, .program = *programResult.value()};
auto binaryResult = ShaderCompiler::GetSpirvBinaryFromProgram(binaryAttrib);
EXPECT_TRUE(binaryResult) << (binaryResult ? String{} : binaryResult.error().log);
if (!binaryResult) return {};
// RAW glslang output: ProgramSpirvTask runs the emulation BEFORE the sanitize chain.
return Move(binaryResult.value());
}
// What ProgramSpirvTask does right after the emulation, so every validity / transpile
// assertion is made against the bytes the backends really receive.
void SanitizeAll(Vector<Vector<Uint32>>& modules) {
for (auto& module : modules) {
EXPECT_TRUE(ShaderCompiler::SanitizeAndOptimizeBinary(module, module, true, true));
}
}
String Disassemble(const Vector<Uint32>& spirv) {
spvtools::SpirvTools tools(SPV_ENV_VULKAN_1_1);
String text;
EXPECT_TRUE(tools.Disassemble(spirv, &text));
return text;
}
String Transpile(const Vector<Uint32>& spirv) {
SpvcSession session(spirv, SessionUsageBit::Transpile);
auto essl = ShaderCompiler::DecompileShader(session);
EXPECT_TRUE(essl) << (essl ? String{} : essl.error().log);
return essl ? essl.value() : String{};
}
Bool Validates(const Vector<Uint32>& spirv) {
spvtools::SpirvTools tools(SPV_ENV_VULKAN_1_1);
String messages;
tools.SetMessageConsumer([&](spv_message_level_t, const char*, const spv_position_t&, const char* message) {
messages += message;
messages += '\n';
});
const Bool ok = tools.Validate(spirv);
EXPECT_TRUE(ok) << messages;
return ok;
}
Bool Contains(const String& haystack, const char* needle) {
return haystack.find(needle) != String::npos;
}
// The shape from the field report (Mali-G615, Espryt): a #version 330 core vertex stage
// writing all eight distances from uniform planes. The enables uniform is what
// InjectClipDistanceEnablesShim adds on an emulating backend; spelled out here because a
// unit test has no backend to arm the shim.
const char* kEightPlaneVertex = R"(#version 330 core
uniform uint mg_ClipDistanceEnables;
uniform vec4 u_ClipPlane[8];
uniform mat4 u_ModelView;
in vec4 a_Position;
void main() {
vec4 eyePos = u_ModelView * a_Position;
gl_Position = eyePos;
gl_ClipDistance[0] = dot(u_ClipPlane[0], eyePos);
gl_ClipDistance[1] = dot(u_ClipPlane[1], eyePos);
gl_ClipDistance[2] = dot(u_ClipPlane[2], eyePos);
gl_ClipDistance[3] = dot(u_ClipPlane[3], eyePos);
gl_ClipDistance[4] = dot(u_ClipPlane[4], eyePos);
gl_ClipDistance[5] = dot(u_ClipPlane[5], eyePos);
gl_ClipDistance[6] = dot(u_ClipPlane[6], eyePos);
gl_ClipDistance[7] = dot(u_ClipPlane[7], eyePos);
}
)";
const char* kPlainFragment = R"(#version 330 core
out vec4 o_color;
void main() { o_color = vec4(0.0, 1.0, 0.0, 1.0); }
)";
constexpr Uint32 kBudget = 32;
} // namespace
TEST(EmulateClipDistance, TheReportedEightPlaneVertexStageClipsByDiscard) {
Vector<GLenum> types;
auto modules =
CompileProgramToSpirv({{GL_VERTEX_SHADER, kEightPlaneVertex}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 2u);
ASSERT_TRUE(Contains(Disassemble(modules[0]), "OpCapability ClipDistance"));
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
SanitizeAll(modules);
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
EXPECT_EQ(outcome.distanceCount, 8u);
EXPECT_TRUE(outcome.enablesFound);
for (const auto& module : modules) {
EXPECT_TRUE(Validates(module));
EXPECT_FALSE(Contains(Disassemble(module), "OpCapability ClipDistance"));
}
const String vertexAsm = Disassemble(modules[0]);
EXPECT_TRUE(Contains(vertexAsm, "mg_ClipDistanceEmu0"));
EXPECT_TRUE(Contains(vertexAsm, "mg_ClipDistanceEmu1"));
EXPECT_TRUE(Contains(vertexAsm, "OpBitwiseAnd"));
const String fragmentAsm = Disassemble(modules[1]);
EXPECT_TRUE(Contains(fragmentAsm, "OpKill"));
EXPECT_TRUE(Contains(fragmentAsm, "mg_ClipDistanceEmu1"));
// The ES hop must no longer need GL_EXT_clip_cull_distance anywhere.
const String vertexEssl = Transpile(modules[0]);
const String fragmentEssl = Transpile(modules[1]);
EXPECT_FALSE(Contains(vertexEssl, "gl_ClipDistance")) << vertexEssl;
EXPECT_FALSE(Contains(vertexEssl, "clip_cull_distance")) << vertexEssl;
EXPECT_TRUE(Contains(vertexEssl, "mg_ClipDistanceEmu0")) << vertexEssl;
EXPECT_TRUE(Contains(fragmentEssl, "discard")) << fragmentEssl;
EXPECT_FALSE(Contains(fragmentEssl, "clip_cull_distance")) << fragmentEssl;
}
TEST(EmulateClipDistance, OneDistanceUsesOneCarrierAndNoEnablesMeansAllEnabled) {
const char* vertex = R"(#version 330 core
void main() {
gl_Position = vec4(float(gl_VertexID), 0.0, 0.0, 1.0);
gl_ClipDistance[0] = gl_Position.x;
}
)";
Vector<GLenum> types;
auto modules = CompileProgramToSpirv({{GL_VERTEX_SHADER, vertex}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 2u);
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
SanitizeAll(modules);
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
EXPECT_EQ(outcome.distanceCount, 1u);
EXPECT_FALSE(outcome.enablesFound);
for (const auto& module : modules)
EXPECT_TRUE(Validates(module));
const String vertexAsm = Disassemble(modules[0]);
EXPECT_TRUE(Contains(vertexAsm, "mg_ClipDistanceEmu0"));
EXPECT_FALSE(Contains(vertexAsm, "mg_ClipDistanceEmu1"));
}
TEST(EmulateClipDistance, AProgramThatNeverClipsIsUntouched) {
const char* vertex = R"(#version 330 core
void main() { gl_Position = vec4(float(gl_VertexID), 0.0, 0.0, 1.0); }
)";
Vector<GLenum> types;
auto modules = CompileProgramToSpirv({{GL_VERTEX_SHADER, vertex}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 2u);
const auto before = modules;
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
EXPECT_FALSE(outcome.emulated);
EXPECT_TRUE(outcome.declineDetail.empty());
EXPECT_EQ(modules, before);
}
TEST(EmulateClipDistance, AGeometryStageProducesBeforeEveryEmitAndTheVertexWriteIsRedirected) {
const char* vertex = R"(#version 330 core
uniform uint mg_ClipDistanceEnables;
void main() {
gl_Position = vec4(float(gl_VertexID), 0.0, 0.0, 1.0);
gl_ClipDistance[0] = -1.0;
}
)";
const char* geometry = R"(#version 330 core
uniform uint mg_ClipDistanceEnables;
layout(triangles) in;
layout(triangle_strip, max_vertices = 3) out;
void main() {
for (int i = 0; i < 3; ++i) {
gl_Position = gl_in[i].gl_Position;
gl_ClipDistance[0] = gl_in[i].gl_Position.x;
gl_ClipDistance[1] = 1.0;
EmitVertex();
}
EndPrimitive();
}
)";
Vector<GLenum> types;
auto modules = CompileProgramToSpirv(
{{GL_VERTEX_SHADER, vertex}, {GL_GEOMETRY_SHADER, geometry}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 3u);
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
SanitizeAll(modules);
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
EXPECT_EQ(outcome.distanceCount, 2u);
for (const auto& module : modules) {
EXPECT_TRUE(Validates(module));
EXPECT_FALSE(Contains(Disassemble(module), "OpCapability ClipDistance"));
}
// The vertex stage is not the producer: redirected, no carriers.
EXPECT_FALSE(Contains(Disassemble(modules[0]), "mg_ClipDistanceEmu0"));
EXPECT_TRUE(Contains(Disassemble(modules[1]), "mg_ClipDistanceEmu0"));
}
TEST(EmulateClipDistance, AFragmentStageReadOfGlClipDistanceIsServedFromTheCarriers) {
const char* vertex = R"(#version 330 core
void main() {
gl_Position = vec4(float(gl_VertexID), 0.0, 0.0, 1.0);
gl_ClipDistance[0] = 0.5;
}
)";
const char* fragment = R"(#version 330 core
out vec4 o_color;
void main() { o_color = vec4(gl_ClipDistance[0], 1.0, 0.0, 1.0); }
)";
Vector<GLenum> types;
auto modules = CompileProgramToSpirv({{GL_VERTEX_SHADER, vertex}, {GL_FRAGMENT_SHADER, fragment}}, types);
ASSERT_EQ(modules.size(), 2u);
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
SanitizeAll(modules);
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
for (const auto& module : modules) {
EXPECT_TRUE(Validates(module));
EXPECT_FALSE(Contains(Disassemble(module), "OpCapability ClipDistance"));
}
const String fragmentEssl = Transpile(modules[1]);
EXPECT_FALSE(Contains(fragmentEssl, "gl_ClipDistance")) << fragmentEssl;
EXPECT_TRUE(Contains(fragmentEssl, "mg_ClipDistanceShadow")) << fragmentEssl;
}
TEST(EmulateClipDistance, AGlInReadDeclinesAndLeavesEveryModuleAlone) {
const char* vertex = R"(#version 330 core
void main() {
gl_Position = vec4(float(gl_VertexID), 0.0, 0.0, 1.0);
gl_ClipDistance[0] = 1.0;
}
)";
const char* geometry = R"(#version 330 core
layout(points) in;
layout(points, max_vertices = 1) out;
void main() {
gl_Position = gl_in[0].gl_Position + vec4(gl_in[0].gl_ClipDistance[0]);
EmitVertex();
}
)";
Vector<GLenum> types;
auto modules = CompileProgramToSpirv(
{{GL_VERTEX_SHADER, vertex}, {GL_GEOMETRY_SHADER, geometry}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 3u);
const auto before = modules;
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
EXPECT_FALSE(outcome.emulated);
EXPECT_NE(outcome.declineDetail.find("gl_in"), String::npos) << outcome.declineDetail;
EXPECT_EQ(modules, before);
}
TEST(EmulateClipDistance, ACaptureOfGlClipDistanceDeclines) {
Vector<GLenum> types;
auto modules =
CompileProgramToSpirv({{GL_VERTEX_SHADER, kEightPlaneVertex}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 2u);
const auto before = modules;
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, true, kBudget, outcome, true, true));
EXPECT_FALSE(outcome.emulated);
EXPECT_FALSE(outcome.declineDetail.empty());
EXPECT_EQ(modules, before);
}
// End to end through the source pipeline: the shim's uniform must survive glslang and the
// sanitize chain so the producer finds it. Without this the draw path's mask has nowhere to
// land and every distance silently clips (the all-enabled fallback).
TEST(EmulateClipDistance, TheShimmedEnablesUniformReachesTheProducer) {
CompileEnv emulating{};
emulating.backend = BackendType::DirectGLES;
emulating.params.EmulatesClipDistance = true;
String vertex = R"(#version 400 core
out float gl_ClipDistance[2];
void main() {
vec2 positions[3] = vec2[3](vec2(-1.0, -1.0), vec2(3.0, -1.0), vec2(-1.0, 3.0));
vec2 p = positions[gl_VertexID];
gl_Position = vec4(p, 0.0, 1.0);
gl_ClipDistance[0] = p.x;
gl_ClipDistance[1] = p.y;
}
)";
String fragment = "#version 400 core\nout vec4 o;\nvoid main() { o = vec4(1.0); }\n";
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Vertex, vertex, emulating);
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Fragment, fragment, emulating);
ASSERT_TRUE(Contains(vertex, MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) << vertex;
Vector<GLenum> types;
auto modules =
CompileProgramToSpirv({{GL_VERTEX_SHADER, vertex.c_str()}, {GL_FRAGMENT_SHADER, fragment.c_str()}}, types);
ASSERT_EQ(modules.size(), 2u);
EXPECT_TRUE(Contains(Disassemble(modules[0]), "mg_ClipDistanceEnables")) << Disassemble(modules[0]);
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true));
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
EXPECT_TRUE(outcome.enablesFound) << vertex;
// And the read the producer added keeps the member alive through the sanitize chain.
SanitizeAll(modules);
for (const auto& module : modules)
EXPECT_TRUE(Validates(module));
EXPECT_TRUE(Contains(Disassemble(modules[0]), "mg_ClipDistanceEnables"));
}
// Native masking (Vulkan with shaderClipDistance): the producer keeps the built-in and the
// capability, writes `enabled(i) ? d[i] : 1.0` back into it, and nothing else changes - no
// carriers, no discard, the fragment stage byte-identical.
TEST(EmulateClipDistance, NativeMaskingWritesTheMaskedDistancesBackIntoTheBuiltIn) {
Vector<GLenum> types;
auto modules =
CompileProgramToSpirv({{GL_VERTEX_SHADER, kEightPlaneVertex}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 2u);
const auto fragmentBefore = modules[1];
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true,
/*nativeMask=*/true));
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
EXPECT_EQ(outcome.distanceCount, 8u);
EXPECT_TRUE(outcome.enablesFound);
EXPECT_EQ(modules[1], fragmentBefore);
const String vertexRaw = Disassemble(modules[0]);
EXPECT_TRUE(Contains(vertexRaw, "OpCapability ClipDistance"));
EXPECT_TRUE(Contains(vertexRaw, "OpBitwiseAnd"));
EXPECT_FALSE(Contains(vertexRaw, "mg_ClipDistanceEmu0"));
SanitizeAll(modules);
for (const auto& module : modules)
EXPECT_TRUE(Validates(module));
EXPECT_FALSE(Contains(Disassemble(modules[1]), "OpKill"));
}
// Native masking leaves gl_in[] reads native: a geometry stage that reads the vertex stage's
// distances and re-emits them is the producer, and the vertex stage is untouched.
TEST(EmulateClipDistance, NativeMaskingKeepsGlInReadsAndMasksOnlyTheProducer) {
const char* vertex = R"(#version 330 core
void main() {
gl_Position = vec4(float(gl_VertexID), 0.0, 0.0, 1.0);
gl_ClipDistance[0] = gl_Position.x;
}
)";
const char* geometry = R"(#version 330 core
uniform uint mg_ClipDistanceEnables;
layout(triangles) in;
layout(triangle_strip, max_vertices = 3) out;
void main() {
for (int i = 0; i < 3; ++i) {
gl_Position = gl_in[i].gl_Position;
gl_ClipDistance[0] = gl_in[i].gl_ClipDistance[0];
EmitVertex();
}
}
)";
Vector<GLenum> types;
auto modules = CompileProgramToSpirv(
{{GL_VERTEX_SHADER, vertex}, {GL_GEOMETRY_SHADER, geometry}, {GL_FRAGMENT_SHADER, kPlainFragment}}, types);
ASSERT_EQ(modules.size(), 3u);
const auto vertexBefore = modules[0];
ShaderCompiler::ClipDistanceEmulationOutcome outcome;
ASSERT_TRUE(ShaderCompiler::EmulateClipDistanceForProgram(modules, types, false, kBudget, outcome, true, true,
/*nativeMask=*/true));
EXPECT_TRUE(outcome.emulated) << outcome.declineDetail;
EXPECT_TRUE(outcome.enablesFound);
EXPECT_EQ(modules[0], vertexBefore);
SanitizeAll(modules);
for (const auto& module : modules)
EXPECT_TRUE(Validates(module));
}
TEST(EmulateClipDistance, TheEnablesShimArmsOnlyOnAnEmulatingBackendAndOnlyWhereItIsNamed) {
CompileEnv emulating{};
emulating.backend = BackendType::DirectGLES;
emulating.params.EmulatesClipDistance = true;
CompileEnv native = emulating;
native.params.EmulatesClipDistance = false;
const String vertex = "#version 330 core\nvoid main() { gl_Position = vec4(0.0); gl_ClipDistance[0] = 1.0; }\n";
const String fragment = "#version 330 core\nout vec4 o; void main() { o = vec4(1.0); }\n";
const String nonClipping = "#version 330 core\nvoid main() { gl_Position = vec4(0.0); }\n";
String source = vertex;
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Vertex, source, emulating);
EXPECT_TRUE(Contains(source, MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) << source;
source = vertex;
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Vertex, source, native);
EXPECT_FALSE(Contains(source, MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) << source;
source = fragment;
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Fragment, source, emulating);
EXPECT_FALSE(Contains(source, MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) << source;
source = nonClipping;
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Vertex, source, emulating);
EXPECT_FALSE(Contains(source, MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) << source;
// A native backend that masks in the shader (Magma) needs the uniform too.
CompileEnv masking = native;
masking.backend = BackendType::DirectVulkan;
masking.params.MasksNativeClipDistances = true;
source = vertex;
MG_Util::ShaderTranspiler::PreprocessShaderSource(ShaderStage::Vertex, source, masking);
EXPECT_TRUE(Contains(source, MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) << source;
}
+44 -4
View File
@@ -1204,18 +1204,58 @@ TEST_F(TextureTest, TextureParameterRejectsProxyTargetsThatGetTexLevelParameterS
ExpectSingleGlError(GL_INVALID_ENUM); ExpectSingleGlError(GL_INVALID_ENUM);
// The level query keeps its proxy support - that is the entire point of a proxy texture, and the // The level query keeps its proxy support - that is the entire point of a proxy texture, and the
// predicate deliberately does not gate it. Only the ERROR is asserted, not the width: MobileGL // predicate deliberately does not gate it: the query answers the size the proxy was given
// does not currently report a proxy level's dimensions back (it answers 0), which is a separate // instead of turning into GL_INVALID_ENUM alongside the setters.
// pre-existing gap in GetTexLevelParameter and not something this predicate decides. What
// matters here is that the query is not turned into GL_INVALID_ENUM alongside the setters.
GLint proxyWidth = -1; GLint proxyWidth = -1;
MG_Impl::GLImpl::GetTexLevelParameteriv(GL_PROXY_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &proxyWidth); MG_Impl::GLImpl::GetTexLevelParameteriv(GL_PROXY_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &proxyWidth);
EXPECT_EQ(MG_Impl::GLImpl::GetError(), GL_NO_ERROR) EXPECT_EQ(MG_Impl::GLImpl::GetError(), GL_NO_ERROR)
<< "the proxy target must still be accepted by GetTexLevelParameter"; << "the proxy target must still be accepted by GetTexLevelParameter";
EXPECT_EQ(proxyWidth, 4) << "a proxy level reports the width it was given";
DrainPendingGlErrors(); DrainPendingGlErrors();
} }
// GL 4.6 core 8.22: a proxy level reports the size it was given while the implementation can hold
// it, and all zeros - with no error - once a dimension passes the limit at that level. Applications
// probe GL_MAX_TEXTURE_SIZE this way, halving from the largest size until the width comes back.
TEST_F(TextureTest, ProxyTextureLevelReportsRequestedSizeOrZerosPastTheLimit) {
GLint maxTextureSize = 0;
MG_Impl::GLImpl::GetIntegerv(GL_MAX_TEXTURE_SIZE, &maxTextureSize);
ASSERT_GT(maxTextureSize, 1);
const auto levelSize = [](GLenum target, GLint level, GLenum pname) {
GLint value = -1;
MG_Impl::GLImpl::GetTexLevelParameteriv(target, level, pname, &value);
return value;
};
MG_Impl::GLImpl::TexImage2D(GL_PROXY_TEXTURE_2D, 0, GL_RGBA8, maxTextureSize, 16, 0, GL_RGBA, GL_UNSIGNED_BYTE,
nullptr);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_2D, 0, GL_TEXTURE_WIDTH), maxTextureSize);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT), 16);
MG_Impl::GLImpl::TexImage2D(GL_PROXY_TEXTURE_2D, 0, GL_RGBA8, maxTextureSize * 2, 16, 0, GL_RGBA, GL_UNSIGNED_BYTE,
nullptr);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_2D, 0, GL_TEXTURE_WIDTH), 0) << "past GL_MAX_TEXTURE_SIZE";
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT), 0) << "every dimension is cleared";
// The limit halves with each level.
MG_Impl::GLImpl::TexImage2D(GL_PROXY_TEXTURE_2D, 1, GL_RGBA8, maxTextureSize, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE,
nullptr);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_2D, 1, GL_TEXTURE_WIDTH), 0);
MG_Impl::GLImpl::TexImage2D(GL_PROXY_TEXTURE_2D, 1, GL_RGBA8, maxTextureSize / 2, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE,
nullptr);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_2D, 1, GL_TEXTURE_WIDTH), maxTextureSize / 2);
MG_Impl::GLImpl::TexImage1D(GL_PROXY_TEXTURE_1D, 0, GL_RGBA8, 64, 0, GL_RGBA, GL_UNSIGNED_BYTE, nullptr);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_1D, 0, GL_TEXTURE_WIDTH), 64);
MG_Impl::GLImpl::TexImage3D(GL_PROXY_TEXTURE_3D, 0, GL_RGBA8, 8, 4, 2, 0, GL_RGBA, GL_UNSIGNED_BYTE, nullptr);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_3D, 0, GL_TEXTURE_WIDTH), 8);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_3D, 0, GL_TEXTURE_HEIGHT), 4);
EXPECT_EQ(levelSize(GL_PROXY_TEXTURE_3D, 0, GL_TEXTURE_DEPTH), 2);
EXPECT_EQ(MG_Impl::GLImpl::GetError(), GL_NO_ERROR) << "an unsupported proxy size is not an error";
}
TEST_F(TextureTest, GenThenBindCreatesObjectForUnsizedPackedBgraSubImageUpload) { TEST_F(TextureTest, GenThenBindCreatesObjectForUnsizedPackedBgraSubImageUpload) {
GLuint texture = 0; GLuint texture = 0;
MG_Impl::GLImpl::GenTextures(1, &texture); MG_Impl::GLImpl::GenTextures(1, &texture);
+32 -6
View File
@@ -479,16 +479,30 @@ TEST(JobNodeException, AThrowingJobDoesNotPoisonTheWorkerForLaterJobs) {
TEST(ShaderCompilePoolDrain, StopAndDrainWithAThousandQueuedJobsLeavesNoneRunningOrPending) { TEST(ShaderCompilePoolDrain, StopAndDrainWithAThousandQueuedJobsLeavesNoneRunningOrPending) {
constexpr Uint kJobs = 1000; constexpr Uint kJobs = 1000;
ShaderCompilePool pool(kTestThreads); ShaderCompilePool pool(kTestThreads);
pool.SetMaxConcurrency(1); // keep the vast majority queued behind the budget pool.SetMaxConcurrency(1);
// The first job holds the only slot until the drain has taken the queue, so the other
// kJobs - 1 are still queued when it does; with empty bodies alone, a test thread that is
// descheduled after posting lets a worker run every job before StopAndDrain is reached.
// While the first job holds the slot, only the drain can settle the last one, so the
// opener waits for that; its timeout turns a drain that never settles it into failures
// rather than a hang.
Gate gate;
Vector<SharedPtr<TestJob>> jobs; Vector<SharedPtr<TestJob>> jobs;
jobs.reserve(kJobs); jobs.reserve(kJobs);
for (Uint i = 0; i < kJobs; ++i) { jobs.push_back(MakeShared<TestJob>([&](TestJob&) { gate.Wait(); }));
pool.Post(jobs.back());
for (Uint i = 1; i < kJobs; ++i) {
jobs.push_back(MakeShared<TestJob>()); jobs.push_back(MakeShared<TestJob>());
pool.Post(jobs.back()); pool.Post(jobs.back());
} }
std::thread opener([&] {
WaitUntil([&] { return jobs.back()->IsTerminal(); });
gate.Open();
});
pool.StopAndDrain(); pool.StopAndDrain();
opener.join();
// Every node is terminal, so nothing can be waiting on a worker that will never come. // Every node is terminal, so nothing can be waiting on a worker that will never come.
Uint complete = 0; Uint complete = 0;
@@ -501,10 +515,14 @@ TEST(ShaderCompilePoolDrain, StopAndDrainWithAThousandQueuedJobsLeavesNoneRunnin
} }
EXPECT_EQ(complete + canceled, kJobs); EXPECT_EQ(complete + canceled, kJobs);
EXPECT_GT(canceled, 0u); // the drain really did abandon queued work rather than run it EXPECT_GT(canceled, 0u); // the drain really did abandon queued work rather than run it
// Only the first job can have run its body: it completes if it was running when the
// drain began and is canceled if it was still waiting for a worker.
EXPECT_GE(canceled, kJobs - 1);
} }
TEST(ShaderCompilePoolDrain, StopAndDrainWaitsForARunningBodyToReturn) { TEST(ShaderCompilePoolDrain, StopAndDrainWaitsForARunningBodyToReturn) {
ShaderCompilePool pool(kTestThreads); ShaderCompilePool pool(kTestThreads);
pool.SetMaxConcurrency(1);
Gate gate; Gate gate;
std::atomic<Bool> entered{false}; std::atomic<Bool> entered{false};
@@ -514,21 +532,29 @@ TEST(ShaderCompilePoolDrain, StopAndDrainWaitsForARunningBodyToReturn) {
gate.Wait(); gate.Wait();
left.store(true, std::memory_order_release); left.store(true, std::memory_order_release);
}); });
// Queued behind the running job, so only the drain can settle it: the opener waits for
// that, which keeps the body running until StopAndDrain has started, however the
// threads are scheduled.
auto queued = MakeShared<TestJob>();
pool.Post(job); pool.Post(job);
pool.Post(queued);
ASSERT_TRUE(WaitUntil([&] { return entered.load(); })); ASSERT_TRUE(WaitUntil([&] { return entered.load(); }));
std::thread opener([&] { std::thread opener([&] {
std::this_thread::sleep_for(std::chrono::milliseconds(20)); WaitUntil([&] { return queued->IsTerminal(); });
gate.Open(); gate.Open();
}); });
pool.StopAndDrain(); pool.StopAndDrain();
// This is the guarantee library teardown relies on: once StopAndDrain returns, no worker
// is still inside a body that could touch glslang's process globals. Read it before
// joining the opener, since by then the body would have finished even without the drain.
const Bool leftWhenDrainReturned = left.load(std::memory_order_acquire);
opener.join(); opener.join();
// This is the guarantee library teardown relies on: once StopAndDrain returns, no worker EXPECT_TRUE(leftWhenDrainReturned);
// is still inside a body that could touch glslang's process globals.
EXPECT_TRUE(left.load());
EXPECT_TRUE(job->IsTerminal()); EXPECT_TRUE(job->IsTerminal());
EXPECT_TRUE(queued->IsCanceled());
} }
TEST(ShaderCompilePoolDrain, JobsPostedAfterADrainStillRun) { TEST(ShaderCompilePoolDrain, JobsPostedAfterADrainStillRun) {
@@ -1059,6 +1059,43 @@ namespace MobileGL::MG_Util::BackendLoader {
caps.MaxFragmentShadingRateAttachmentTexelHeight); caps.MaxFragmentShadingRateAttachmentTexelHeight);
} }
// Native clip distances or MobileGL's emulation. Native only where the driver both advertises
// GL_EXT_clip_cull_distance AND passes the probe (its own control first): an extension that is
// listed but does not clip, or clips by disabled distances, is a wrong answer the application
// sees, and the emulation answers it correctly. The probe is skipped entirely without the
// extension - there is nothing to measure.
static void DecideClipDistanceEmulation(MG_External::GLESCapabilities& caps,
const MG_External::GLESFunctionsTable& glesFuncs) {
if (caps.SupportsClipDistance && caps.MaxClipDistances > 0) {
const SelfTest::NativeClipDistanceMeasurement& clip = SelfTest::NativeClipDistance(glesFuncs);
caps.NativeClipDistanceClips = clip.ran && clip.clips;
caps.NativeClipDistanceHonorsEnables = clip.ran && clip.honorsEnables;
const String verdict = !clip.ran ? "inconclusive (" + clip.failure + ")"
: !clip.clips ? "an enabled negative distance did NOT clip"
: !clip.honorsEnables ? "clips, but by DISABLED distances too"
: "clips, and honors the per-distance enables";
MGLOG_I(" Native gl_ClipDistance probe: %s", verdict.c_str());
}
caps.EmulatesClipDistance = !(caps.SupportsClipDistance && caps.MaxClipDistances > 0 &&
caps.NativeClipDistanceClips && caps.NativeClipDistanceHonorsEnables);
switch (MG_Config::Features.ClipDistanceEmulation) {
case MG_Config::QuirkOverride::ForceOn:
caps.EmulatesClipDistance = true;
MGLOG_I(" gl_ClipDistance: emulation forced ON by MOBILEGL_CLIP_DISTANCE_EMULATION");
break;
case MG_Config::QuirkOverride::ForceOff:
caps.EmulatesClipDistance = false;
MGLOG_I(" gl_ClipDistance: emulation forced OFF by MOBILEGL_CLIP_DISTANCE_EMULATION");
break;
case MG_Config::QuirkOverride::Auto:
default:
break;
}
MGLOG_I(" gl_ClipDistance: %s", caps.EmulatesClipDistance
? "EMULATED (varyings + fragment discard; 8 distances advertised)"
: "native (GL_EXT_clip_cull_distance)");
}
Bool FillInGLESCapabilities(MG_External::GLESCapabilities& caps, const MG_External::GLESFunctionsTable& glesFuncs) { Bool FillInGLESCapabilities(MG_External::GLESCapabilities& caps, const MG_External::GLESFunctionsTable& glesFuncs) {
if (!glesFuncs.glGetString || !glesFuncs.glGetIntegerv) { if (!glesFuncs.glGetString || !glesFuncs.glGetIntegerv) {
MGLOG_E("Required GLES functions are not loaded, cannot query capabilities"); MGLOG_E("Required GLES functions are not loaded, cannot query capabilities");
@@ -1997,6 +2034,7 @@ namespace MobileGL::MG_Util::BackendLoader {
MGLOG_I(" GL_MAX_COMPUTE_IMAGE_UNIFORMS: %d", caps.MaxComputeImageUniforms); MGLOG_I(" GL_MAX_COMPUTE_IMAGE_UNIFORMS: %d", caps.MaxComputeImageUniforms);
MGLOG_I(" GL_MAX_DRAW_BUFFERS: %d", caps.MaxDrawBuffers); MGLOG_I(" GL_MAX_DRAW_BUFFERS: %d", caps.MaxDrawBuffers);
MGLOG_I(" GL_MAX_COLOR_ATTACHMENTS: %d", caps.MaxColorAttachments); MGLOG_I(" GL_MAX_COLOR_ATTACHMENTS: %d", caps.MaxColorAttachments);
DecideClipDistanceEmulation(caps, glesFuncs);
// Worth spelling the reason out for the same reason the per-stage storage block counts // Worth spelling the reason out for the same reason the per-stage storage block counts
// are: a zero here is what stops an application's gl_ClipDistance from ever clipping, and // are: a zero here is what stops an application's gl_ClipDistance from ever clipping, and
// reading it back from an artifact is the difference between "MobileGL dropped my draw" // reading it back from an artifact is the difference between "MobileGL dropped my draw"
@@ -1274,6 +1274,19 @@ namespace MobileGL {
// compile and the per-distance enables have nowhere to go - clipping silently never // compile and the per-distance enables have nowhere to go - clipping silently never
// happens, which is exactly what KHR-GLxx.clip_distance.functional catches. // happens, which is exactly what KHR-GLxx.clip_distance.functional catches.
Bool SupportsClipDistance = false; Bool SupportsClipDistance = false;
// What SelfTest::ProbeNativeClipDistance measured - only run where the extension is
// advertised: an enabled negative distance removes its fragments, and a disabled one
// removes nothing.
Bool NativeClipDistanceClips = false;
Bool NativeClipDistanceHonorsEnables = false;
// DirectGLES lowers gl_ClipDistance to varyings plus a fragment discard
// (ShaderCompiler::EmulateClipDistanceForProgram) instead of handing it to the
// driver: no extension, no distances behind it, or a probe that caught the driver
// not clipping or not honoring the enables. MOBILEGL_CLIP_DISTANCE_EMULATION
// overrides. While set, the CLIP_DISTANCEi enables are NOT forwarded to the driver -
// no emulated module writes the built-in, and an enabled distance nobody writes is
// undefined clipping.
Bool EmulatesClipDistance = false;
// GL_OES_viewport_array is present: the driver knows gl_ViewportIndex in ESSL - and // GL_OES_viewport_array is present: the driver knows gl_ViewportIndex in ESSL - and
// only then. ESSL has no core spelling for it at ANY version, while SPIRV-Cross prints // only then. ESSL has no core spelling for it at ANY version, while SPIRV-Cross prints
// the identifier bare and requests nothing for it (contrast gl_Layer, which it backs // the identifier bare and requests nothing for it (contrast gl_Layer, which it backs
@@ -410,8 +410,12 @@ namespace MobileGL {
return TextureUploadTarget::Texture2D; return TextureUploadTarget::Texture2D;
case GL_TEXTURE_3D: case GL_TEXTURE_3D:
return TextureUploadTarget::Texture3D; return TextureUploadTarget::Texture3D;
case GL_PROXY_TEXTURE_1D:
return TextureUploadTarget::ProxyTexture1D;
case GL_PROXY_TEXTURE_2D: case GL_PROXY_TEXTURE_2D:
return TextureUploadTarget::ProxyTexture2D; return TextureUploadTarget::ProxyTexture2D;
case GL_PROXY_TEXTURE_3D:
return TextureUploadTarget::ProxyTexture3D;
case GL_TEXTURE_1D_ARRAY: case GL_TEXTURE_1D_ARRAY:
return TextureUploadTarget::Texture1DArray; return TextureUploadTarget::Texture1DArray;
case GL_PROXY_TEXTURE_1D_ARRAY: case GL_PROXY_TEXTURE_1D_ARRAY:
@@ -317,8 +317,12 @@ namespace MobileGL {
return GL_TEXTURE_1D; return GL_TEXTURE_1D;
case TextureUploadTarget::Texture3D: case TextureUploadTarget::Texture3D:
return GL_TEXTURE_3D; return GL_TEXTURE_3D;
case TextureUploadTarget::ProxyTexture1D:
return GL_PROXY_TEXTURE_1D;
case TextureUploadTarget::ProxyTexture2D: case TextureUploadTarget::ProxyTexture2D:
return GL_PROXY_TEXTURE_2D; return GL_PROXY_TEXTURE_2D;
case TextureUploadTarget::ProxyTexture3D:
return GL_PROXY_TEXTURE_3D;
case TextureUploadTarget::Texture1DArray: case TextureUploadTarget::Texture1DArray:
return GL_TEXTURE_1D_ARRAY; return GL_TEXTURE_1D_ARRAY;
case TextureUploadTarget::ProxyTexture1DArray: case TextureUploadTarget::ProxyTexture1DArray:
@@ -2066,6 +2066,168 @@ namespace MobileGL::MG_Util::SelfTest {
return measurement; return measurement;
} }
namespace {
// ===================== NATIVE CLIP DISTANCE =====================
constexpr const char* kClipDistanceProbeName = "clip distance";
// Four pixels across: pixel 0 is centred at NDC x = -0.75, pixel 3 at +0.75, so the
// distance gl_ClipDistance[0] = x is clearly negative on one and positive on the other.
constexpr GLsizei kClipDistanceProbeWidth = 4;
// GL_CLIP_DISTANCE0_EXT / 1_EXT: the desktop values, absent from the ES headers.
constexpr GLenum kClipDistance0 = 0x3000;
constexpr GLenum kClipDistance1 = 0x3001;
const char* const kClipDistanceProbeVertexSource =
"#version 300 es\n"
"#extension GL_EXT_clip_cull_distance : require\n"
"precision highp float;\n"
"void main() {\n"
" vec2 mg_p = vec2((gl_VertexID == 1) ? 3.0 : -1.0, (gl_VertexID == 2) ? 3.0 : -1.0);\n"
" gl_Position = vec4(mg_p, 0.0, 1.0);\n"
" gl_ClipDistance[0] = mg_p.x;\n"
" gl_ClipDistance[1] = 1.0;\n"
"}\n";
// The control: the same stage with a distance that is positive everywhere.
const char* const kClipDistanceProbeControlVertexSource =
"#version 300 es\n"
"#extension GL_EXT_clip_cull_distance : require\n"
"precision highp float;\n"
"void main() {\n"
" vec2 mg_p = vec2((gl_VertexID == 1) ? 3.0 : -1.0, (gl_VertexID == 2) ? 3.0 : -1.0);\n"
" gl_Position = vec4(mg_p, 0.0, 1.0);\n"
" gl_ClipDistance[0] = 1.0;\n"
" gl_ClipDistance[1] = 1.0;\n"
"}\n";
const char* const kClipDistanceProbeFragmentSource =
"#version 300 es\n"
"precision mediump float;\n"
"layout(location = 0) out vec4 mg_probeColor;\n"
"void main() { mg_probeColor = vec4(0.0, 1.0, 0.0, 1.0); }\n";
Bool IsProbeGreen(const GLubyte* pixel) {
return pixel[0] < 64 && pixel[1] > 192;
}
// One draw into the bound 4x1 target, cleared red first. Returns false when the draw or
// the read-back raised an error (inconclusive).
Bool DrawClipDistanceProbe(const GLESFunctionsTable& gl, Bool enableDistance0, Bool& leftPainted,
Bool& rightPainted) {
if (enableDistance0) {
gl.glEnable(kClipDistance0);
} else {
gl.glDisable(kClipDistance0);
}
gl.glDisable(kClipDistance1);
gl.glClearColor(1.0f, 0.0f, 0.0f, 1.0f);
gl.glClear(GL_COLOR_BUFFER_BIT);
Drain(gl);
gl.glDrawArrays(GL_TRIANGLES, 0, 3);
if (gl.glGetError() != GL_NO_ERROR) return false;
GLubyte row[kClipDistanceProbeWidth * 4] = {};
gl.glReadPixels(0, 0, kClipDistanceProbeWidth, 1, GL_RGBA, GL_UNSIGNED_BYTE, row);
if (gl.glGetError() != GL_NO_ERROR) return false;
leftPainted = IsProbeGreen(row);
rightPainted = IsProbeGreen(row + (kClipDistanceProbeWidth - 1) * 4);
return true;
}
} // namespace
NativeClipDistanceMeasurement ProbeNativeClipDistance(const GLESFunctionsTable& gl) {
NativeClipDistanceMeasurement measurement;
if (!HasIoBlockProbeEntryPoints(gl)) {
measurement.failure = "a probe entry point is missing";
return measurement;
}
SavedState saved;
Save(gl, saved);
GLboolean savedColorMask[4] = {GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE};
const Bool canMaskColor = gl.glColorMask != nullptr && gl.glGetBooleanv != nullptr;
if (canMaskColor) {
gl.glGetBooleanv(GL_COLOR_WRITEMASK, savedColorMask);
gl.glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
}
GLuint vao = 0;
gl.glGenVertexArrays(1, &vao);
gl.glBindVertexArray(vao);
PrepareForProbeDraw(gl);
Vector<StageSource> stages;
stages.push_back({GL_VERTEX_SHADER, kClipDistanceProbeVertexSource, "vertex"});
stages.push_back({GL_FRAGMENT_SHADER, kClipDistanceProbeFragmentSource, "fragment"});
const ProgramBuild build = BuildProgram(gl, stages, kClipDistanceProbeName);
Vector<StageSource> controlStages;
controlStages.push_back({GL_VERTEX_SHADER, kClipDistanceProbeControlVertexSource, "control vertex"});
controlStages.push_back({GL_FRAGMENT_SHADER, kClipDistanceProbeFragmentSource, "fragment"});
const ProgramBuild control = BuildProgram(gl, controlStages, kClipDistanceProbeName);
if (!build.compiled || !control.compiled) {
measurement.failure = "the probe's gl_ClipDistance vertex stage did not compile";
} else if (!build.linked || !control.linked) {
measurement.failure =
"the probe program did not link: " + FirstLine(!build.linked ? build.infoLog : control.infoLog);
} else {
GLuint renderbuffer = 0;
GLuint framebuffer = 0;
gl.glGenRenderbuffers(1, &renderbuffer);
gl.glBindRenderbuffer(GL_RENDERBUFFER, renderbuffer);
gl.glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8, kClipDistanceProbeWidth, 1);
gl.glGenFramebuffers(1, &framebuffer);
gl.glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
gl.glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, renderbuffer);
if (gl.glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
measurement.failure = "the probe framebuffer is incomplete";
} else {
gl.glViewport(0, 0, kClipDistanceProbeWidth, 1);
Bool left = false;
Bool right = false;
// THE CONTROL first: a positive distance, enabled, so both halves must paint.
gl.glUseProgram(control.program);
const Bool controlRan = DrawClipDistanceProbe(gl, true, left, right);
gl.glUseProgram(build.program);
if (!controlRan || !left || !right) {
measurement.failure = "the control draw (a positive distance, enabled) did not paint the target";
} else if (!DrawClipDistanceProbe(gl, true, left, right) || !right) {
measurement.failure = "the clipped draw raised an error or lost its kept half";
} else {
measurement.clips = !left;
Bool leftAfter = false;
Bool rightAfter = false;
if (DrawClipDistanceProbe(gl, false, leftAfter, rightAfter) && rightAfter) {
measurement.ran = true;
measurement.honorsEnables = leftAfter;
} else {
measurement.failure = "the re-disabled draw raised an error or lost its kept half";
}
}
}
gl.glDisable(kClipDistance0);
gl.glDisable(kClipDistance1);
if (framebuffer != 0) gl.glDeleteFramebuffers(1, &framebuffer);
if (renderbuffer != 0) gl.glDeleteRenderbuffers(1, &renderbuffer);
}
if (build.program != 0) gl.glDeleteProgram(build.program);
if (control.program != 0) gl.glDeleteProgram(control.program);
if (vao != 0) {
gl.glBindVertexArray(0);
gl.glDeleteVertexArrays(1, &vao);
}
if (canMaskColor) {
gl.glColorMask(savedColorMask[0], savedColorMask[1], savedColorMask[2], savedColorMask[3]);
}
Restore(gl, saved);
Drain(gl);
if (!measurement.failure.empty()) {
MGLOG_I("[driver-bug] %s probe: %s", kClipDistanceProbeName, measurement.failure.c_str());
}
return measurement;
}
const NativeClipDistanceMeasurement& NativeClipDistance(const GLESFunctionsTable& gl) {
static const NativeClipDistanceMeasurement measurement = ProbeNativeClipDistance(gl);
return measurement;
}
namespace { namespace {
// ===================== PACKED16 COPY-IMAGE FIELD ORDER ===================== // ===================== PACKED16 COPY-IMAGE FIELD ORDER =====================
@@ -95,6 +95,38 @@ namespace MobileGL::MG_Util::SelfTest {
// ProbeLocatedIoBlocksLosePayload(), evaluated at most once per process. // ProbeLocatedIoBlocksLosePayload(), evaluated at most once per process.
const LocatedIoBlockMeasurement& LocatedIoBlocksLosePayload(const MG_External::GLESFunctionsTable& gl); const LocatedIoBlockMeasurement& LocatedIoBlocksLosePayload(const MG_External::GLESFunctionsTable& gl);
// What the native clip-distance probe measured. Not a driver-bug finding by itself: the
// loader turns it into the DirectGLES choice between the driver's gl_ClipDistance and
// MobileGL's emulation (EmulateClipDistancePass), and the POST "Clip distances" row reports
// that choice.
struct NativeClipDistanceMeasurement {
// The probe built its program and drew it; every field below is meaningless otherwise.
Bool ran = false;
// GL_CLIP_DISTANCE0_EXT enabled, distance negative on the left half: the left half was
// removed and the right half painted.
Bool clips = false;
// GL_CLIP_DISTANCE0_EXT disabled, same distances: nothing was removed.
Bool honorsEnables = false;
// The driver's first info-log line (or a short reason) when the probe could not run.
String failure;
};
// Draws full-viewport triangles under GL_EXT_clip_cull_distance. THE CONTROL first: a
// program whose gl_ClipDistance[0] is +1 everywhere, with distance 0 enabled - both halves
// must paint whatever the driver does with enables, or the extension's draw itself is broken
// and the probe has nothing to say. Then the SUBJECT, writing gl_ClipDistance[0] = x: with
// distance 0 enabled the left half must vanish, and with it disabled nothing may vanish
// (the llvmpipe ES driver clips by every written distance regardless of the enables, a wrong
// answer GL applications see).
//
// Requires the extension; the caller only runs it where the extension string names it.
// Restores every piece of GL state it touches and leaves both distances disabled - the
// state DirectGLES's enable forwarding assumes a fresh context is in.
NativeClipDistanceMeasurement ProbeNativeClipDistance(const MG_External::GLESFunctionsTable& gl);
// ProbeNativeClipDistance(), evaluated at most once per process.
const NativeClipDistanceMeasurement& NativeClipDistance(const MG_External::GLESFunctionsTable& gl);
// Blits one layer of an RGBA8 2D array onto another array's layer 1 and reports whether the // Blits one layer of an RGBA8 2D array onto another array's layer 1 and reports whether the
// copy landed where it was asked to. Returns true only when the destination layer is ignored // copy landed where it was asked to. Returns true only when the destination layer is ignored
// while the control lands correctly. // while the control lands correctly.
+22
View File
@@ -7,6 +7,7 @@
// End of Source File Header // End of Source File Header
#include "DriverPost.h" #include "DriverPost.h"
#include "DriverPostClipDistance.h"
#include "DriverPostIterationRPWitness.h" #include "DriverPostIterationRPWitness.h"
#include "DriverPostIterationRPWitnessSpv.h" #include "DriverPostIterationRPWitnessSpv.h"
#include "PrimitivesGeneratedNoXfbProbe.h" #include "PrimitivesGeneratedNoXfbProbe.h"
@@ -340,6 +341,22 @@ namespace MobileGL::MG_Util::SelfTest {
} }
} }
void EmitClipDistanceRows(ReportBuilder& builder, Vector<ClipDistancePostRow> rows) {
for (ClipDistancePostRow& row : rows) {
switch (row.verdict) {
case ClipDistancePostRow::Verdict::Pass:
builder.Pass(Move(row.name), Move(row.detail));
break;
case ClipDistancePostRow::Verdict::Warn:
builder.Warn(Move(row.name), Move(row.detail));
break;
case ClipDistancePostRow::Verdict::FailOptional:
builder.FailOptional(Move(row.name), Move(row.detail));
break;
}
}
}
void EvaluateGlesChecklist(ReportBuilder& builder, const MG_External::GLESCapabilities& caps, void EvaluateGlesChecklist(ReportBuilder& builder, const MG_External::GLESCapabilities& caps,
const MG_External::GLESFunctionsTable& glesFuncs) { const MG_External::GLESFunctionsTable& glesFuncs) {
const Int major = caps.GLESVersion.Major; const Int major = caps.GLESVersion.Major;
@@ -388,6 +405,7 @@ namespace MobileGL::MG_Util::SelfTest {
"hard-fails, and a second fragment output cannot be produced any other " "hard-fails, and a second fragment output cannot be produced any other "
"way"); "way");
} }
EmitClipDistanceRows(builder, GlesClipDistancePostRows(caps, glesFuncs));
if (es31) { if (es31) {
GLint maxVertexSsboBlocks = 0; GLint maxVertexSsboBlocks = 0;
@@ -2776,6 +2794,10 @@ namespace MobileGL::MG_Util::SelfTest {
builder.FailOptional("dualSrcBlend", "unsupported; a draw using a GL_SRC1_* blend factor hard-fails, and a " builder.FailOptional("dualSrcBlend", "unsupported; a draw using a GL_SRC1_* blend factor hard-fails, and a "
"second fragment output cannot be produced any other way"); "second fragment output cannot be produced any other way");
} }
EmitClipDistanceRows(builder, VulkanClipDistancePostRows(features.shaderClipDistance == VK_TRUE,
features.shaderCullDistance == VK_TRUE,
properties.limits.maxClipDistances,
properties.limits.maxCullDistances));
// The Magma counterpart of the GLES "Buffer textures" row, so the two sections can be // The Magma counterpart of the GLES "Buffer textures" row, so the two sections can be
// read side by side. Vulkan has no optional-feature bit here: a uniform texel buffer is // read side by side. Vulkan has no optional-feature bit here: a uniform texel buffer is
// core, and maxTexelBufferElements has a spec floor of 65536 - exactly the GL 3.1 floor // core, and maxTexelBufferElements has a spec floor of 65536 - exactly the GL 3.1 floor
@@ -0,0 +1,118 @@
// MobileGL - MobileGL/MG_Util/SelfTest/DriverPostClipDistance.cpp
// Copyright (c) 2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
#include "DriverPostClipDistance.h"
#include "DriverBugProbes.h"
#include <Config.h>
#include <format>
namespace MobileGL::MG_Util::SelfTest {
namespace {
using Verdict = ClipDistancePostRow::Verdict;
// What the emulation still cannot do, shared by both backends' WARN rows.
constexpr const char* kClipDistanceEmulationResidue =
"MobileGL emulates 8 distances: gl_ClipDistance becomes two ordinary varyings and the "
"fragment stage discards where a distance is negative, honoring GL_CLIP_DISTANCEi. It costs "
"two varying locations and early depth/stencil for programs that clip; transform-feedback "
"capture of gl_ClipDistance and gl_in[].gl_ClipDistance reads are not emulated";
// Culling is per primitive, which a per-fragment discard cannot reproduce.
constexpr const char* kNoCullDistance =
"GL_MAX_CULL_DISTANCES = 0. Culling is per primitive, which a per-fragment discard cannot "
"reproduce, so a gl_CullDistance shader fails to compile";
String GlesEmulationReason(const MG_External::GLESCapabilities& caps, const String& probeFailure) {
if (!caps.SupportsClipDistance) return "no GL_EXT_clip_cull_distance";
if (caps.MaxClipDistances <= 0) return "GL_EXT_clip_cull_distance reports no distances";
if (!probeFailure.empty()) {
return "GL_EXT_clip_cull_distance is listed, but the native probe was inconclusive (" + probeFailure +
")";
}
if (!caps.NativeClipDistanceClips) {
return "GL_EXT_clip_cull_distance is listed, but in the probe an enabled negative distance did not "
"clip";
}
if (!caps.NativeClipDistanceHonorsEnables) return "the driver clips by DISABLED distances too (probe)";
return "forced by MOBILEGL_CLIP_DISTANCE_EMULATION=1";
}
} // namespace
Vector<ClipDistancePostRow> GlesClipDistancePostRows(const MG_External::GLESCapabilities& caps,
const MG_External::GLESFunctionsTable& glesFuncs) {
Vector<ClipDistancePostRow> rows;
// Only meaningful where the loader ran it (extension present); cached per process.
String probeFailure;
if (caps.SupportsClipDistance && caps.MaxClipDistances > 0) {
const NativeClipDistanceMeasurement& probe = NativeClipDistance(glesFuncs);
if (!probe.ran) probeFailure = probe.failure;
}
const Bool native = caps.SupportsClipDistance && caps.MaxClipDistances > 0 && caps.NativeClipDistanceClips &&
caps.NativeClipDistanceHonorsEnables;
if (!caps.EmulatesClipDistance && native) {
rows.push_back({Verdict::Pass, "Clip distances",
std::format("native via GL_EXT_clip_cull_distance ({} distances); probe: an enabled "
"distance clips and a disabled one does not",
caps.MaxClipDistances)});
} else if (caps.EmulatesClipDistance) {
rows.push_back(
{Verdict::Warn, "Clip distances",
std::format("{}; {}", GlesEmulationReason(caps, probeFailure), kClipDistanceEmulationResidue)});
} else {
rows.push_back({Verdict::FailOptional, "Clip distances",
caps.SupportsClipDistance
? "GL_EXT_clip_cull_distance is listed but misbehaves in the probe, and the emulation "
"is disabled (MOBILEGL_CLIP_DISTANCE_EMULATION=0)"
: "no GL_EXT_clip_cull_distance and the emulation is disabled "
"(MOBILEGL_CLIP_DISTANCE_EMULATION=0): GL_MAX_CLIP_DISTANCES = 0, so a "
"gl_ClipDistance shader fails to compile"});
}
if (caps.SupportsClipDistance && caps.MaxCullDistances > 0) {
rows.push_back({Verdict::Pass, "Cull distances",
std::format("native via GL_EXT_clip_cull_distance ({} distances)", caps.MaxCullDistances)});
} else {
rows.push_back({Verdict::FailOptional, "Cull distances",
std::format("no GL_EXT_clip_cull_distance: {}", kNoCullDistance)});
}
return rows;
}
Vector<ClipDistancePostRow> VulkanClipDistancePostRows(const Bool hasClipDistance, const Bool hasCullDistance,
const Uint32 maxClipDistances,
const Uint32 maxCullDistances) {
Vector<ClipDistancePostRow> rows;
Bool emulate = !hasClipDistance;
if (MG_Config::Features.ClipDistanceEmulation == MG_Config::QuirkOverride::ForceOn) emulate = true;
if (MG_Config::Features.ClipDistanceEmulation == MG_Config::QuirkOverride::ForceOff) emulate = false;
if (!emulate && hasClipDistance) {
rows.push_back({Verdict::Pass, "shaderClipDistance",
std::format("native ({} distances); Vulkan has no per-distance enable state, so MobileGL "
"applies GL_CLIP_DISTANCEi in the last pre-rasterization stage (a disabled "
"distance is written as +1)",
maxClipDistances)});
} else if (emulate) {
rows.push_back({Verdict::Warn, "shaderClipDistance",
std::format("{}; {}",
hasClipDistance ? "forced by MOBILEGL_CLIP_DISTANCE_EMULATION=1"
: "unsupported by the device",
kClipDistanceEmulationResidue)});
} else {
rows.push_back({Verdict::FailOptional, "shaderClipDistance",
"unsupported, and the emulation is disabled (MOBILEGL_CLIP_DISTANCE_EMULATION=0): "
"GL_MAX_CLIP_DISTANCES = 0, so a gl_ClipDistance shader fails to compile"});
}
if (hasCullDistance) {
rows.push_back(
{Verdict::Pass, "shaderCullDistance", std::format("native ({} distances)", maxCullDistances)});
} else {
rows.push_back(
{Verdict::FailOptional, "shaderCullDistance", std::format("unsupported: {}", kNoCullDistance)});
}
return rows;
}
} // namespace MobileGL::MG_Util::SelfTest
@@ -0,0 +1,40 @@
// MobileGL - MobileGL/MG_Util/SelfTest/DriverPostClipDistance.h
// Copyright (c) 2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
//
// The POST rows for clip and cull distances on both backends. GL_MAX_CLIP_DISTANCES >= 8 is a
// GL 3.0 requirement: PASS where the backend clips natively, WARN where MobileGL emulates the
// built-in, FAIL where it does neither.
#pragma once
#include <Includes.h>
#include <MG_Util/BackendLoaders/OpenGL/Loader.h>
namespace MobileGL::MG_Util::SelfTest {
struct ClipDistancePostRow {
enum class Verdict {
Pass,
Warn,
FailOptional
};
Verdict verdict = Verdict::Pass;
String name;
String detail;
};
// "Clip distances" and "Cull distances" for the GLES backend, from the loader's decision and
// its cached native probe.
Vector<ClipDistancePostRow> GlesClipDistancePostRows(const MG_External::GLESCapabilities& caps,
const MG_External::GLESFunctionsTable& glesFuncs);
// "shaderClipDistance" and "shaderCullDistance" for the Vulkan backend, mirroring
// BackendObject_DirectVulkan's decision: emulate exactly when the feature is absent, unless
// MOBILEGL_CLIP_DISTANCE_EMULATION says otherwise.
Vector<ClipDistancePostRow> VulkanClipDistancePostRows(Bool hasClipDistance, Bool hasCullDistance,
Uint32 maxClipDistances, Uint32 maxCullDistances);
} // namespace MobileGL::MG_Util::SelfTest
@@ -75,6 +75,11 @@ namespace MobileGL::MG_Util::ShaderTranspiler {
// and expands gl_MaxClipDistances from the same number, so it decides both whether a // and expands gl_MaxClipDistances from the same number, so it decides both whether a
// shader compiles at all and what a module that reads the constant generates. // shader compiles at all and what a module that reads the constant generates.
HashValue(state, env.params.MaxClipDistances); HashValue(state, env.params.MaxClipDistances);
// Unlike the point-size demotion bits this IS frontend material: when it is set the
// source pipeline declares the reserved enables uniform in every pre-rasterization
// stage that names gl_ClipDistance, so glslang parses a different source.
HashValue(state, env.params.EmulatesClipDistance);
HashValue(state, env.params.MasksNativeClipDistances);
// The cull-distance pair, added when the GL 4.6 API-surface wave made them env-derived: // The cull-distance pair, added when the GL 4.6 API-surface wave made them env-derived:
// they were bare literals (8/8) in BuildTBuiltInResource while no backend had ever been // they were bare literals (8/8) in BuildTBuiltInResource while no backend had ever been
// asked whether it can host a cull distance. Exactly the MaxClipDistances class - glslang // asked whether it can host a cull distance. Exactly the MaxClipDistances class - glslang
@@ -165,6 +165,16 @@ namespace MobileGL::MG_Util::ShaderTranspiler {
// fingerprint: glslang produces the same thing either way. // fingerprint: glslang produces the same thing either way.
Bool DemotesTessellationPointSize() const { return HasBackend() && !params.SupportsTessellationPointSize; } Bool DemotesTessellationPointSize() const { return HasBackend() && !params.SupportsTessellationPointSize; }
Bool DemotesGeometryPointSize() const { return HasBackend() && !params.SupportsGeometryPointSize; } Bool DemotesGeometryPointSize() const { return HasBackend() && !params.SupportsGeometryPointSize; }
// The backend cannot clip with gl_ClipDistance natively and emulates it
// (BackendCaps::EmulatesClipDistance). Part of the frontend fingerprint, not a separate
// L1 bit: it changes the parsed source (the reserved enables uniform), not only phase B.
Bool EmulatesClipDistance() const { return HasBackend() && params.EmulatesClipDistance; }
// BackendCaps::MasksNativeClipDistances, same fingerprint reasoning.
Bool MasksNativeClipDistances() const {
return HasBackend() && params.MasksNativeClipDistances && !params.EmulatesClipDistance;
}
// Either way the program needs the reserved enables uniform.
Bool NeedsClipDistanceEnables() const { return EmulatesClipDistance() || MasksNativeClipDistances(); }
// Matches the historical rule exactly: with no active backend every extension counts // Matches the historical rule exactly: with no active backend every extension counts
// as advertised, because the frontend then has nothing to gate against. // as advertised, because the frontend then has nothing to gate against.
Bool IsExtensionAdvertised(GLExtension extension) const { Bool IsExtensionAdvertised(GLExtension extension) const {
@@ -61,6 +61,7 @@
#include "SpirvPasses/LegalizeResourceArrayIndexPass.h" #include "SpirvPasses/LegalizeResourceArrayIndexPass.h"
#include "SpirvPasses/FlattenAtomicCounterBlockPass.h" #include "SpirvPasses/FlattenAtomicCounterBlockPass.h"
#include "SpirvPasses/DemotePointSizePass.h" #include "SpirvPasses/DemotePointSizePass.h"
#include "SpirvPasses/EmulateClipDistancePass.h"
#include "spirv-tools/libspirv.h" #include "spirv-tools/libspirv.h"
#include "spirv-tools/optimizer.hpp" #include "spirv-tools/optimizer.hpp"
#include "source/opt/build_module.h" #include "source/opt/build_module.h"
@@ -1215,6 +1216,140 @@ namespace MobileGL {
return true; return true;
} }
Bool ShaderCompiler::EmulateClipDistanceForProgram(
Vector<Vector<Uint32>>& modules, const Vector<GLenum>& shaderTypes,
const Bool captureRequestsClipDistance, const Uint32 maxVaryingLocations,
ClipDistanceEmulationOutcome& outcome, const bool validateOutput, const bool enableSpirvValidation,
const bool nativeMask) {
outcome = {};
// Pipeline order: VS, TCS, TES, GS, FS.
Int stageIndex[5] = {-1, -1, -1, -1, -1};
for (SizeT i = 0; i < shaderTypes.size() && i < modules.size(); ++i) {
switch (shaderTypes[i]) {
case GL_VERTEX_SHADER:
stageIndex[0] = static_cast<Int>(i);
break;
case GL_TESS_CONTROL_SHADER:
stageIndex[1] = static_cast<Int>(i);
break;
case GL_TESS_EVALUATION_SHADER:
stageIndex[2] = static_cast<Int>(i);
break;
case GL_GEOMETRY_SHADER:
stageIndex[3] = static_cast<Int>(i);
break;
case GL_FRAGMENT_SHADER:
stageIndex[4] = static_cast<Int>(i);
break;
default:
break;
}
}
ClipDistanceModuleProbe probes[5];
Bool anyAccess = false;
Uint32 carrierLocation = 0;
for (Int stage = 0; stage < 5; ++stage) {
if (stageIndex[stage] < 0) continue;
const Vector<Uint32>& module = modules[stageIndex[stage]];
probes[stage] = EmulateClipDistancePass::ProbeModule(module);
if (!probes[stage].parsed) return true; // broken for other reasons; owns its failure
anyAccess |= probes[stage].declaresClipDistanceCapability;
const PointSizeModuleProbe locations = ProbePointSizeModule(module);
carrierLocation = std::max(carrierLocation, locations.locationSlotEnd);
}
if (!anyAccess) return true;
if (captureRequestsClipDistance) {
outcome.declineDetail = "transform feedback captures gl_ClipDistance";
return true;
}
// The producer: the last pre-rasterization stage present, and only when it
// actually writes the built-in - an earlier stage's writes are dead in GL unless
// a later stage reads gl_in[].gl_ClipDistance, which the pass declines.
Int producer = -1;
for (Int stage = 3; stage >= 0; --stage) {
if (stage == 1) continue; // a control stage is never last
if (stageIndex[stage] >= 0) {
producer = stage;
break;
}
}
Uint32 distanceCount = 0;
if (producer >= 0 && probes[producer].declaresClipDistanceCapability) {
distanceCount = probes[producer].outputDistanceCount;
}
if (distanceCount > EmulateClipDistancePass::kMaxCarriers * 4) {
outcome.declineDetail =
std::format("{} clip distances, more than the eight the carriers hold", distanceCount);
return true;
}
const Uint32 carrierCount = nativeMask ? 0 : (distanceCount + 3) / 4;
if (carrierCount > 0 && carrierLocation + carrierCount > maxVaryingLocations) {
outcome.declineDetail = std::format(
"the program's varyings already reach location {}, leaving no room for {} clip carrier(s) "
"under the {}-location budget",
carrierLocation, carrierCount, maxVaryingLocations);
return true;
}
Vector<Vector<Uint32>> rewritten(modules.size());
Bool rewrote[5] = {false, false, false, false, false};
Bool enablesFound = false;
for (Int stage = 0; stage < 5; ++stage) {
const Int moduleIndex = stageIndex[stage];
if (moduleIndex < 0) continue;
EmulateClipDistanceOptions options;
options.location = carrierLocation;
// Native masking touches the producer alone: every other stage hosts the
// built-in, and only the producer's values reach the clipper.
if (nativeMask && !(stage == producer && distanceCount > 0)) continue;
options.nativeBuiltin = nativeMask;
if (stage == 4) {
options.role = EmulateClipDistanceOptions::Role::Consumer;
options.distanceCount = distanceCount;
if (distanceCount == 0 && !probes[stage].declaresClipDistanceCapability) continue;
} else if (stage == producer && distanceCount > 0) {
options.role = EmulateClipDistanceOptions::Role::Producer;
options.enablesMemberName = CLIP_DISTANCE_ENABLES_UNIFORM_NAME;
} else {
options.role = EmulateClipDistanceOptions::Role::Redirect;
if (!probes[stage].declaresClipDistanceCapability) continue;
}
EmulateClipDistanceReport report;
spvtools::Optimizer optimizer(SPV_ENV_VULKAN_1_1);
optimizer.RegisterPass(EmulateClipDistancePass::CreateEmulateClipDistancePass(options, &report));
if (!RunOptimizerChecked("EmulateClipDistanceForProgram", optimizer, modules[moduleIndex],
rewritten[moduleIndex], validateOutput, enableSpirvValidation)) {
return false; // modules untouched: nothing was committed
}
if (report.declined) {
outcome.declineDetail = Move(report.declineReason);
return true;
}
if (options.role == EmulateClipDistanceOptions::Role::Producer) {
if (report.distanceCount != distanceCount) {
outcome.declineDetail =
std::format("the producing stage carries {} clip distances where {} were planned",
report.distanceCount, distanceCount);
return true;
}
enablesFound = report.enablesFound;
}
rewrote[stage] = report.rewrote;
}
for (Int stage = 0; stage < 5; ++stage) {
if (!rewrote[stage]) continue;
modules[stageIndex[stage]] = Move(rewritten[stageIndex[stage]]);
}
outcome.emulated = true;
outcome.distanceCount = distanceCount;
outcome.enablesFound = enablesFound;
return true;
}
Bool ShaderCompiler::ModuleDeclaresFloat64(const Vector<Uint32>& spirv) { Bool ShaderCompiler::ModuleDeclaresFloat64(const Vector<Uint32>& spirv) {
if (spirv.empty()) { if (spirv.empty()) {
// Same reasoning as ModuleDeclaresBufferTextureSampler: a stage that produced // Same reasoning as ModuleDeclaresBufferTextureSampler: a stage that produced
@@ -650,6 +650,44 @@ namespace MobileGL {
Bool demoteGeometry, Bool captureRequestsPointSize, PointSizeDemotionOutcome& outcome, Bool demoteGeometry, Bool captureRequestsPointSize, PointSizeDemotionOutcome& outcome,
bool validateOutput = true, bool enableSpirvValidation = false); bool validateOutput = true, bool enableSpirvValidation = false);
// What the program-scoped clip-distance emulation left behind. `emulated` false
// with an empty detail means the program never touches gl_ClipDistance; false
// WITH a detail is a shape the pass cannot express (modules byte-identical, so the
// program keeps the built-in and the backend's own refusal of it).
struct ClipDistanceEmulationOutcome {
Bool emulated = false;
// How many distances the producing stage carries to the fragment stage.
Uint32 distanceCount = 0;
// False: the producer had no enables uniform, so every distance clips.
Bool enablesFound = false;
String declineDetail;
};
// Emulates gl_ClipDistance across a WHOLE program on a backend that cannot clip
// natively (BackendCaps::EmulatesClipDistance): every stage's accesses move onto
// a Private shadow, the last pre-rasterization stage carries
// `enabled(i) ? d[i] : 1.0` to the fragment stage in ceil(N/4) vec4 varyings
// placed past every location the program uses, and the fragment stage discards
// where a carried distance is negative. See EmulateClipDistancePass for the
// per-module rewrite. Runs BEFORE the sanitize chain (ProgramSpirvTask): the
// reserved enables uniform is unread until this pass reads it, and ADCE would
// otherwise delete a global UBO that holds nothing else. Atomic: every stage is
// rewritten or none is.
// `captureRequestsClipDistance` declines (a capture would need the unmasked
// value at a varying the pass does not create); `maxVaryingLocations` is the
// carrier budget. Returns false only when the optimizer itself failed.
//
// `nativeMask`: the backend HOSTS gl_ClipDistance but has no per-distance enable
// state (Vulkan). Only the producing stage is rewritten - it writes the masked
// values back into the built-in (EmulateClipDistanceOptions::nativeBuiltin) - and
// no carrier, location or fragment discard is involved.
static Bool EmulateClipDistanceForProgram(Vector<Vector<Uint32>>& modules,
const Vector<GLenum>& shaderTypes,
Bool captureRequestsClipDistance, Uint32 maxVaryingLocations,
ClipDistanceEmulationOutcome& outcome,
bool validateOutput = true,
bool enableSpirvValidation = false, bool nativeMask = false);
// True when the module still declares a 64-bit float type. After // True when the module still declares a 64-bit float type. After
// SanitizeAndOptimizeBinary that can only mean DemoteFloat64Pass declined the // SanitizeAndOptimizeBinary that can only mean DemoteFloat64Pass declined the
// module (see its header for the two operations that make it decline), which is // module (see its header for the two operations that make it decline), which is
@@ -1880,6 +1880,28 @@ namespace {
source.insert(afterVersion.Get(source), shim); source.insert(afterVersion.Get(source), shim);
} }
// On a backend that emulates gl_ClipDistance, the stage that ends up producing the clip
// carriers needs GL's per-distance enables, and the only thing a module can read per draw
// without a pipeline key is a uniform. Declared in every pre-rasterization stage that names
// the built-in, because which of them is LAST is a link-time fact this per-shader pass cannot
// know; the stages that are not the producer simply never read it, and glslang folds the one
// name into the shared MGL_GLOBAL_UBO for every stage of the program alike.
void InjectClipDistanceEnablesShim(MobileGL::ShaderStage stage, MobileGL::String& source,
AfterVersionAnchor& afterVersion) {
if (stage != MobileGL::ShaderStage::Vertex && stage != MobileGL::ShaderStage::TessControl &&
stage != MobileGL::ShaderStage::TessEval && stage != MobileGL::ShaderStage::Geometry) {
return;
}
if (source.find("gl_ClipDistance") == MobileGL::String::npos) return;
const MobileGL::String masked = MaskCommentsAndQuotedText(source);
if (!MaskedSourceHasIdentifier(masked, "gl_ClipDistance")) return;
if (MaskedSourceHasIdentifier(masked,
MobileGL::MG_Util::ShaderTranspiler::CLIP_DISTANCE_ENABLES_UNIFORM_NAME)) {
return;
}
source.insert(afterVersion.Get(source), "uniform uint mg_ClipDistanceEnables;\n");
}
// The uniforms `text` declares with the external sampler type, and (into `others`) the names it // The uniforms `text` declares with the external sampler type, and (into `others`) the names it
// declares with any other type - the same name can be both in a source whose #if branches pick // declares with any other type - the same name can be both in a source whose #if branches pick
// one (a compositor's shader generator does exactly that). // one (a compositor's shader generator does exactly that).
@@ -2020,6 +2042,7 @@ namespace MobileGL {
ModernizeLegacyGLSL(stage, source, afterVersion); ModernizeLegacyGLSL(stage, source, afterVersion);
InjectDepthRangeBuiltinShim(stage, source, afterVersion); InjectDepthRangeBuiltinShim(stage, source, afterVersion);
InjectNumSamplesBuiltinShim(stage, originalLanguage, source, afterVersion); InjectNumSamplesBuiltinShim(stage, originalLanguage, source, afterVersion);
if (env.NeedsClipDistanceEnables()) InjectClipDistanceEnablesShim(stage, source, afterVersion);
} }
Vector<String> ExternalSamplerUniformNames(const String& preprocessedSource) { Vector<String> ExternalSamplerUniformNames(const String& preprocessedSource) {
@@ -0,0 +1,814 @@
// MobileGL - MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/EmulateClipDistancePass.cpp
// Copyright (c) 2025-2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
#include "EmulateClipDistancePass.h"
#include "spirv.hpp"
#include "source/opt/basic_block.h"
#include "source/opt/build_module.h"
#include "source/opt/constants.h"
#include "source/opt/def_use_manager.h"
#include "source/opt/function.h"
#include "source/opt/instruction.h"
#include "source/opt/ir_builder.h"
#include "source/opt/ir_context.h"
#include "source/opt/module.h"
#include "source/opt/type_manager.h"
#include "source/opt/types.h"
#include "source/util/make_unique.h"
#include "source/util/string_utils.h"
#include <format>
#include <unordered_set>
#include <vector>
namespace MobileGL {
namespace MG_Util {
namespace ShaderTranspiler {
namespace {
using spvtools::opt::BasicBlock;
using spvtools::opt::Function;
using spvtools::opt::Instruction;
using spvtools::opt::InstructionBuilder;
using spvtools::opt::IRContext;
using spvtools::opt::Operand;
namespace analysis = spvtools::opt::analysis;
const IRContext::Analysis kBuilderAnalyses = static_cast<IRContext::Analysis>(
IRContext::kAnalysisDefUse | IRContext::kAnalysisInstrToBlockMapping);
Instruction* EntryPoint(IRContext* ctx) {
for (Instruction& ep : ctx->module()->entry_points()) {
return &ep;
}
return nullptr;
}
Function* EntryFunction(IRContext* ctx, Instruction* entryPoint) {
const uint32_t id = entryPoint->GetSingleWordInOperand(1);
for (auto& function : *ctx->module()) {
if (function.result_id() == id) return &function;
}
return nullptr;
}
uint32_t PointeeOf(IRContext* ctx, uint32_t pointerTypeId) {
Instruction* ptrType = ctx->get_def_use_mgr()->GetDef(pointerTypeId);
if (ptrType == nullptr || ptrType->opcode() != spv::Op::OpTypePointer) return 0;
return ptrType->GetSingleWordInOperand(1);
}
bool PlainConstantValue(IRContext* ctx, uint32_t id, uint32_t& outValue) {
Instruction* def = ctx->get_def_use_mgr()->GetDef(id);
if (def == nullptr || def->opcode() != spv::Op::OpConstant) return false;
if (def->NumInOperands() != 1) return false;
outValue = def->GetSingleWordInOperand(0);
return true;
}
// Length of an OpTypeArray of 32-bit float with a plain constant length, else 0.
uint32_t FloatArrayLength(IRContext* ctx, uint32_t typeId) {
Instruction* t = ctx->get_def_use_mgr()->GetDef(typeId);
if (t == nullptr || t->opcode() != spv::Op::OpTypeArray) return 0;
Instruction* element = ctx->get_def_use_mgr()->GetDef(t->GetSingleWordInOperand(0));
if (element == nullptr || element->opcode() != spv::Op::OpTypeFloat ||
element->GetSingleWordInOperand(0) != 32) {
return 0;
}
uint32_t length = 0;
if (!PlainConstantValue(ctx, t->GetSingleWordInOperand(1), length)) return 0;
return length;
}
void AddNameFor(IRContext* ctx, uint32_t id, const String& name) {
std::vector<Operand> operands;
operands.push_back({SPV_OPERAND_TYPE_ID, {id}});
operands.push_back({SPV_OPERAND_TYPE_LITERAL_STRING, spvtools::utils::MakeVector(name)});
ctx->AddDebug2Inst(spvtools::MakeUnique<Instruction>(ctx, spv::Op::OpName, 0, 0, operands));
}
uint32_t AddGlobalVariable(IRContext* ctx, uint32_t pointeeTypeId, spv::StorageClass storage,
const String& name) {
const uint32_t ptrTypeId = ctx->get_type_mgr()->FindPointerToType(pointeeTypeId, storage);
if (ptrTypeId == 0) return 0;
const uint32_t varId = ctx->TakeNextId();
auto var = spvtools::MakeUnique<Instruction>(
ctx, spv::Op::OpVariable, ptrTypeId, varId,
std::initializer_list<Operand>{
{SPV_OPERAND_TYPE_STORAGE_CLASS, {static_cast<uint32_t>(storage)}}});
Instruction* varInst = var.get();
ctx->AddGlobalValue(std::move(var));
ctx->get_def_use_mgr()->AnalyzeInstDefUse(varInst);
AddNameFor(ctx, varId, name);
return varId;
}
void AddLocationDecoration(IRContext* ctx, uint32_t id, uint32_t location) {
ctx->AddAnnotationInst(spvtools::MakeUnique<Instruction>(
ctx, spv::Op::OpDecorate, 0, 0,
std::initializer_list<Operand>{
{SPV_OPERAND_TYPE_ID, {id}},
{SPV_OPERAND_TYPE_DECORATION, {static_cast<uint32_t>(spv::Decoration::Location)}},
{SPV_OPERAND_TYPE_LITERAL_INTEGER, {location}}}));
}
void RemoveFromInterface(Instruction* entryPoint, uint32_t id) {
Instruction::OperandList kept;
for (uint32_t i = 0; i < entryPoint->NumInOperands(); ++i) {
const Operand& operand = entryPoint->GetInOperand(i);
if (i >= 3 && operand.type == SPV_OPERAND_TYPE_ID && operand.words.size() == 1 &&
operand.words[0] == id) {
continue;
}
kept.push_back(operand);
}
entryPoint->SetInOperands(Move(kept));
}
// Whether every use of a pointer into the built-in is one the retype below can
// follow: loads, stores (as the POINTER), copies, and further access chains.
bool PointerUsesAreRetypable(IRContext* ctx, Instruction* pointer, String& reason) {
bool ok = true;
ctx->get_def_use_mgr()->ForEachUse(pointer, [&](Instruction* user, uint32_t operandIndex) {
if (!ok) return;
switch (user->opcode()) {
case spv::Op::OpName:
case spv::Op::OpDecorate:
case spv::Op::OpEntryPoint:
return;
case spv::Op::OpLoad:
case spv::Op::OpCopyMemory:
return;
case spv::Op::OpStore:
// Operand 0 of OpStore is the pointer; anything else would be
// storing the pointer itself, which logical addressing forbids.
if (operandIndex != 0) {
ok = false;
reason = "a pointer into gl_ClipDistance stored as a value";
}
return;
case spv::Op::OpAccessChain:
case spv::Op::OpInBoundsAccessChain:
// Base operand: the instruction's first id after result type/id.
if (operandIndex != 2) {
ok = false;
reason = "a pointer into gl_ClipDistance used as an access-chain index";
return;
}
if (!PointerUsesAreRetypable(ctx, user, reason)) ok = false;
return;
default:
ok = false;
reason = std::format("a pointer into gl_ClipDistance reaching SPIR-V opcode {}",
static_cast<uint32_t>(user->opcode()));
return;
}
});
return ok;
}
// Re-points every access chain hanging off `pointer` (which now lives in Private
// storage) at a Private pointer type of the same pointee.
void RetypeChainUsers(IRContext* ctx, Instruction* pointer) {
std::vector<Instruction*> chains;
ctx->get_def_use_mgr()->ForEachUser(pointer, [&](Instruction* user) {
if ((user->opcode() == spv::Op::OpAccessChain ||
user->opcode() == spv::Op::OpInBoundsAccessChain) &&
user->GetSingleWordInOperand(0) == pointer->result_id()) {
chains.push_back(user);
}
});
for (Instruction* chain : chains) {
const uint32_t pointee = PointeeOf(ctx, chain->type_id());
chain->SetResultType(
ctx->get_type_mgr()->FindPointerToType(pointee, spv::StorageClass::Private));
ctx->get_def_use_mgr()->AnalyzeInstUse(chain);
RetypeChainUsers(ctx, chain);
}
}
// A scalar constant of the given 32-bit integer type.
uint32_t IntConstantOfType(IRContext* ctx, uint32_t intTypeId, uint32_t value) {
const analysis::Type* type = ctx->get_type_mgr()->GetType(intTypeId);
const analysis::Integer* integer = type != nullptr ? type->AsInteger() : nullptr;
if (integer != nullptr && integer->IsSigned()) {
return ctx->get_constant_mgr()->GetSIntConstId(static_cast<int32_t>(value));
}
return ctx->get_constant_mgr()->GetUIntConstId(value);
}
bool IsInt32Type(IRContext* ctx, uint32_t typeId) {
const analysis::Type* type = ctx->get_type_mgr()->GetType(typeId);
const analysis::Integer* integer = type != nullptr ? type->AsInteger() : nullptr;
return integer != nullptr && integer->width() == 32;
}
struct ClipSites {
// struct type id -> member index of its BuiltIn ClipDistance member
std::vector<std::pair<uint32_t, uint32_t>> members;
std::unordered_set<uint32_t> standaloneVars;
Instruction* capability = nullptr;
};
ClipSites FindClipSites(IRContext* ctx) {
ClipSites sites;
for (Instruction& ann : ctx->annotations()) {
if (ann.opcode() == spv::Op::OpMemberDecorate && ann.NumInOperands() >= 4 &&
static_cast<spv::Decoration>(ann.GetSingleWordInOperand(2)) == spv::Decoration::BuiltIn &&
static_cast<spv::BuiltIn>(ann.GetSingleWordInOperand(3)) == spv::BuiltIn::ClipDistance) {
sites.members.emplace_back(ann.GetSingleWordInOperand(0), ann.GetSingleWordInOperand(1));
} else if (ann.opcode() == spv::Op::OpDecorate && ann.NumInOperands() >= 3 &&
static_cast<spv::Decoration>(ann.GetSingleWordInOperand(1)) ==
spv::Decoration::BuiltIn &&
static_cast<spv::BuiltIn>(ann.GetSingleWordInOperand(2)) ==
spv::BuiltIn::ClipDistance) {
Instruction* var = ctx->get_def_use_mgr()->GetDef(ann.GetSingleWordInOperand(0));
if (var != nullptr && var->opcode() == spv::Op::OpVariable) {
sites.standaloneVars.insert(var->result_id());
}
}
}
for (Instruction& capability : ctx->module()->capabilities()) {
if (capability.NumInOperands() >= 1 &&
static_cast<spv::Capability>(capability.GetSingleWordInOperand(0)) ==
spv::Capability::ClipDistance) {
sites.capability = &capability;
}
}
return sites;
}
bool ClipMemberOf(const ClipSites& sites, uint32_t structId, uint32_t& member) {
for (const auto& [id, index] : sites.members) {
if (id == structId) {
member = index;
return true;
}
}
return false;
}
// Every redirected access must agree on one gl_ClipDistance array type.
bool NoteShadowType(uint32_t& shadowTypeId, uint32_t typeId) {
if (shadowTypeId == 0) {
shadowTypeId = typeId;
return true;
}
return shadowTypeId == typeId;
}
// A chain that stops short of the gl_PerVertex member (a whole block or gl_in[i]
// pointer) is harmless unless something then reaches the clip member through it or
// copies the whole struct.
bool ShortChainTouchesClipMember(IRContext* ctx, Instruction* chain, uint32_t clipMember) {
bool touches = false;
ctx->get_def_use_mgr()->ForEachUser(chain, [&](Instruction* next) {
if (next->opcode() == spv::Op::OpAccessChain ||
next->opcode() == spv::Op::OpInBoundsAccessChain) {
uint32_t member = 0;
if (next->NumInOperands() < 2 ||
!PlainConstantValue(ctx, next->GetSingleWordInOperand(1), member) ||
member == clipMember) {
touches = true;
}
} else if (next->opcode() != spv::Op::OpName) {
touches = true; // whole-struct load/store/copy
}
});
return touches;
}
struct BlockVariable {
Instruction* var = nullptr;
spv::StorageClass storage = spv::StorageClass::Output;
bool arrayed = false;
uint32_t clipMember = 0;
uint32_t memberTypeId = 0;
};
// Classifies every use of a gl_PerVertex block variable: chains onto the clip member
// of a non-arrayed block are collected, anything else that reaches it declines.
bool CollectBlockClipChains(IRContext* ctx, const BlockVariable& block, uint32_t& shadowTypeId,
std::vector<Instruction*>& memberChains, String& reason) {
const uint32_t memberOperand = block.arrayed ? 2u : 1u; // [vertex,] member, ...
bool declined = false;
ctx->get_def_use_mgr()->ForEachUser(block.var, [&](Instruction* user) {
if (declined) return;
switch (user->opcode()) {
case spv::Op::OpEntryPoint:
case spv::Op::OpName:
case spv::Op::OpDecorate:
return;
case spv::Op::OpAccessChain:
case spv::Op::OpInBoundsAccessChain: {
if (user->NumInOperands() <= memberOperand) {
if (ShortChainTouchesClipMember(ctx, user, block.clipMember)) {
declined = true;
reason = "a split access chain or whole-gl_PerVertex copy reaching gl_ClipDistance";
}
return;
}
uint32_t member = 0;
if (!PlainConstantValue(ctx, user->GetSingleWordInOperand(memberOperand), member)) {
declined = true;
reason = "a non-constant gl_PerVertex member index";
return;
}
if (member != block.clipMember) return; // another member
if (block.arrayed) {
declined = true;
reason = block.storage == spv::StorageClass::Input ? "a gl_in[].gl_ClipDistance read"
: "a gl_out[].gl_ClipDistance write";
return;
}
if (!NoteShadowType(shadowTypeId, block.memberTypeId)) {
declined = true;
reason = "two differently sized gl_ClipDistance arrays";
return;
}
if (!PointerUsesAreRetypable(ctx, user, reason)) {
declined = true;
return;
}
memberChains.push_back(user);
return;
}
case spv::Op::OpLoad:
case spv::Op::OpStore:
case spv::Op::OpCopyMemory:
declined = true;
reason = "a whole-aggregate load/store/copy of the gl_PerVertex interface";
return;
default:
declined = true;
reason = std::format("SPIR-V opcode {} reaching the gl_PerVertex interface",
static_cast<uint32_t>(user->opcode()));
return;
}
});
return !declined;
}
// The clip member type of the stage's gl_PerVertex OUTPUT block, or 0. A producer with
// nothing redirected may still have declared the length the driver planned with.
uint32_t DeclaredOutputClipType(IRContext* ctx, const ClipSites& sites) {
for (Instruction& inst : ctx->module()->types_values()) {
if (inst.opcode() != spv::Op::OpVariable ||
static_cast<spv::StorageClass>(inst.GetSingleWordInOperand(0)) !=
spv::StorageClass::Output) {
continue;
}
Instruction* type = ctx->get_def_use_mgr()->GetDef(PointeeOf(ctx, inst.type_id()));
uint32_t member = 0;
if (type != nullptr && type->opcode() == spv::Op::OpTypeStruct &&
ClipMemberOf(sites, type->result_id(), member)) {
return type->GetSingleWordInOperand(member);
}
}
return 0;
}
struct EnablesUniform {
uint32_t varId = 0;
uint32_t memberIndex = 0;
uint32_t typeId = 0;
};
// The producer's enables uniform: a 32-bit integer member, by name, of a
// Uniform-storage struct.
EnablesUniform FindEnablesUniform(IRContext* ctx, const String& memberName) {
EnablesUniform found;
for (Instruction& debug : ctx->module()->debugs2()) {
if (debug.opcode() != spv::Op::OpMemberName || debug.NumInOperands() < 3) continue;
if (debug.GetOperand(2).AsString() != memberName) continue;
const uint32_t structId = debug.GetSingleWordInOperand(0);
const uint32_t member = debug.GetSingleWordInOperand(1);
Instruction* structType = ctx->get_def_use_mgr()->GetDef(structId);
if (structType == nullptr || structType->opcode() != spv::Op::OpTypeStruct ||
member >= structType->NumInOperands()) {
continue;
}
const uint32_t memberType = structType->GetSingleWordInOperand(member);
if (!IsInt32Type(ctx, memberType)) continue;
for (Instruction& var : ctx->module()->types_values()) {
if (var.opcode() == spv::Op::OpVariable &&
static_cast<spv::StorageClass>(var.GetSingleWordInOperand(0)) ==
spv::StorageClass::Uniform &&
PointeeOf(ctx, var.type_id()) == structId) {
return {var.result_id(), member, memberType};
}
}
}
return found;
}
// Where the producer writes its distances out: before each vertex a geometry stage
// emits, else at the entry function's returns.
std::vector<Instruction*> FlushSites(IRContext* ctx, Instruction* entryPoint, bool isGeometry) {
std::vector<Instruction*> sites;
if (isGeometry) {
for (auto& function : *ctx->module()) {
function.ForEachInst([&](Instruction* inst) {
if (inst->opcode() == spv::Op::OpEmitVertex ||
inst->opcode() == spv::Op::OpEmitStreamVertex) {
sites.push_back(inst);
}
});
}
} else if (Function* entry = EntryFunction(ctx, entryPoint)) {
entry->ForEachInst([&](Instruction* inst) {
if (inst->opcode() == spv::Op::OpReturn || inst->opcode() == spv::Op::OpReturnValue) {
sites.push_back(inst);
}
});
}
return sites;
}
struct ScalarIds {
uint32_t floatType = 0;
uint32_t vec4Type = 0;
uint32_t boolType = 0;
uint32_t one = 0;
uint32_t zero = 0;
uint32_t ptrPrivateFloat = 0;
};
ScalarIds MakeScalarIds(IRContext* ctx) {
auto* typeMgr = ctx->get_type_mgr();
ScalarIds ids;
analysis::Float floatTy(32);
ids.floatType = typeMgr->GetTypeInstruction(&floatTy);
analysis::Vector vec4Ty(typeMgr->GetRegisteredType(&floatTy), 4);
ids.vec4Type = typeMgr->GetTypeInstruction(&vec4Ty);
analysis::Bool boolTy;
ids.boolType = typeMgr->GetTypeInstruction(&boolTy);
ids.one = ctx->get_constant_mgr()->GetFloatConstId(1.0f);
ids.zero = ctx->get_constant_mgr()->GetFloatConstId(0.0f);
ids.ptrPrivateFloat = typeMgr->FindPointerToType(ids.floatType, spv::StorageClass::Private);
return ids;
}
// Fragment stage: load the carriers at the top of the entry function, serve the
// shadow's gl_ClipDistance reads from them, and kill on any negative distance.
void EmitCarrierDiscard(IRContext* ctx, Instruction* entryPoint, const ScalarIds& ids,
Uint32 consumedCount, Uint32 location, uint32_t shadowId,
uint32_t shadowLength) {
Function* entry = EntryFunction(ctx, entryPoint);
if (entry == nullptr || entry->begin() == entry->end()) return;
auto* constMgr = ctx->get_constant_mgr();
const Uint32 carrierCount = (consumedCount + 3) / 4;
uint32_t carriers[EmulateClipDistancePass::kMaxCarriers] = {0, 0};
for (Uint32 k = 0; k < carrierCount; ++k) {
carriers[k] = AddGlobalVariable(ctx, ids.vec4Type, spv::StorageClass::Input,
EmulateClipDistancePass::CarrierName(k));
AddLocationDecoration(ctx, carriers[k], location + k);
entryPoint->AddOperand({SPV_OPERAND_TYPE_ID, {carriers[k]}});
}
BasicBlock* head = &*entry->begin();
auto split = head->begin();
while (split != head->end() && split->opcode() == spv::Op::OpVariable)
++split;
BasicBlock* rest = head->SplitBasicBlock(ctx, ctx->TakeNextId(), split);
InstructionBuilder b(ctx, head, kBuilderAnalyses);
uint32_t loaded[EmulateClipDistancePass::kMaxCarriers] = {0, 0};
for (Uint32 k = 0; k < carrierCount; ++k) {
loaded[k] = b.AddLoad(ids.vec4Type, carriers[k])->result_id();
}
uint32_t anyNegative = 0;
for (Uint32 i = 0; i < consumedCount; ++i) {
const uint32_t component =
b.AddCompositeExtract(ids.floatType, loaded[i / 4], {i % 4})->result_id();
if (shadowId != 0 && i < shadowLength) {
Instruction* ptr =
b.AddAccessChain(ids.ptrPrivateFloat, shadowId, {constMgr->GetUIntConstId(i)});
b.AddStore(ptr->result_id(), component);
}
const uint32_t negative =
b.AddBinaryOp(ids.boolType, spv::Op::OpFOrdLessThan, component, ids.zero)->result_id();
anyNegative =
anyNegative == 0
? negative
: b.AddBinaryOp(ids.boolType, spv::Op::OpLogicalOr, anyNegative, negative)->result_id();
}
const uint32_t killLabel = ctx->TakeNextId();
b.AddConditionalBranch(anyNegative, killLabel, rest->id(), rest->id());
auto killBlock = spvtools::MakeUnique<BasicBlock>(spvtools::MakeUnique<Instruction>(
ctx, spv::Op::OpLabel, 0, killLabel, std::initializer_list<Operand>{}));
killBlock->AddInstruction(spvtools::MakeUnique<Instruction>(ctx, spv::Op::OpKill, 0, 0,
std::initializer_list<Operand>{}));
entry->InsertBasicBlockAfter(std::move(killBlock), head);
}
} // namespace
ClipDistanceModuleProbe EmulateClipDistancePass::ProbeModule(const Vector<Uint32>& spirv) {
ClipDistanceModuleProbe probe;
if (spirv.empty()) {
probe.parsed = true;
return probe;
}
std::unique_ptr<IRContext> ctx = spvtools::BuildModule(
SPV_ENV_VULKAN_1_1, [](spv_message_level_t, const char*, const spv_position_t&, const char*) {},
spirv.data(), spirv.size());
if (!ctx) return probe;
probe.parsed = true;
const ClipSites sites = FindClipSites(ctx.get());
probe.declaresClipDistanceCapability = sites.capability != nullptr;
for (Instruction& inst : ctx->module()->types_values()) {
if (inst.opcode() != spv::Op::OpVariable) continue;
if (static_cast<spv::StorageClass>(inst.GetSingleWordInOperand(0)) != spv::StorageClass::Output) {
continue;
}
const uint32_t pointee = PointeeOf(ctx.get(), inst.type_id());
if (sites.standaloneVars.count(inst.result_id()) != 0) {
probe.outputDistanceCount = FloatArrayLength(ctx.get(), pointee);
break;
}
Instruction* type = ctx->get_def_use_mgr()->GetDef(pointee);
uint32_t member = 0;
if (type != nullptr && type->opcode() == spv::Op::OpTypeStruct &&
ClipMemberOf(sites, type->result_id(), member)) {
probe.outputDistanceCount = FloatArrayLength(ctx.get(), type->GetSingleWordInOperand(member));
break;
}
}
return probe;
}
spvtools::opt::Pass::Status EmulateClipDistancePass::Process() {
IRContext* ctx = context();
auto* defUse = ctx->get_def_use_mgr();
auto* typeMgr = ctx->get_type_mgr();
auto* constMgr = ctx->get_constant_mgr();
using Role = EmulateClipDistanceOptions::Role;
Instruction* entryPoint = EntryPoint(ctx);
if (entryPoint == nullptr) return Status::SuccessWithoutChange;
const auto model = static_cast<spv::ExecutionModel>(entryPoint->GetSingleWordInOperand(0));
const bool isFragment = model == spv::ExecutionModel::Fragment;
const bool isGeometry = model == spv::ExecutionModel::Geometry;
const bool isPreRaster = model == spv::ExecutionModel::Vertex ||
model == spv::ExecutionModel::TessellationControl ||
model == spv::ExecutionModel::TessellationEvaluation || isGeometry;
const auto decline = [&](String reason) {
if (m_report != nullptr) {
m_report->declined = true;
m_report->declineReason = Move(reason);
}
return Status::SuccessWithoutChange;
};
if (m_options.role == Role::Consumer && !isFragment)
return decline("consumer role on a non-fragment stage");
if (m_options.role != Role::Consumer && !isPreRaster) {
return decline("producer/redirect role on a non-pre-rasterization stage");
}
if (m_options.role == Role::Producer && model == spv::ExecutionModel::TessellationControl) {
return decline("a control stage cannot be the last pre-rasterization stage");
}
const bool native = m_options.nativeBuiltin;
if (native && m_options.role != Role::Producer) return decline("native masking outside the producer");
const ClipSites sites = FindClipSites(ctx);
// ---- Phase 1: ANALYSIS ONLY. A decline leaves the module byte-identical. ----
std::vector<Instruction*> memberChains; // [clipMember, rest...] on a non-arrayed block
std::vector<Instruction*> standaloneVars; // standalone ClipDistance variables
// Native masking: where the masked values are written back to.
Instruction* nativeBlockVar = nullptr;
uint32_t nativeBlockMember = 0;
Instruction* nativeStandaloneVar = nullptr;
uint32_t shadowTypeId = 0;
String reason;
for (Instruction& inst : ctx->module()->types_values()) {
if (inst.opcode() != spv::Op::OpVariable) continue;
const auto storage = static_cast<spv::StorageClass>(inst.GetSingleWordInOperand(0));
if (storage != spv::StorageClass::Input && storage != spv::StorageClass::Output) continue;
// Native masking leaves every INPUT alone: gl_in[] reads are hosted natively.
if (native && storage == spv::StorageClass::Input) continue;
const uint32_t pointeeId = PointeeOf(ctx, inst.type_id());
if (sites.standaloneVars.count(inst.result_id()) != 0) {
if (native) nativeStandaloneVar = &inst;
if (FloatArrayLength(ctx, pointeeId) == 0) {
return decline("a standalone gl_ClipDistance that is not a sized float array");
}
// A pre-rasterization stage's standalone INPUT would be gl_in data,
// which a non-arrayed variable cannot be; only the fragment stage reads
// gl_ClipDistance this way.
if (storage == spv::StorageClass::Input && !isFragment) {
return decline("a standalone gl_ClipDistance input outside the fragment stage");
}
if (!NoteShadowType(shadowTypeId, pointeeId)) {
return decline("two differently sized gl_ClipDistance arrays");
}
if (!PointerUsesAreRetypable(ctx, &inst, reason)) return decline(Move(reason));
standaloneVars.push_back(&inst);
continue;
}
Instruction* pointee = defUse->GetDef(pointeeId);
if (pointee == nullptr) continue;
bool arrayed = false;
if (pointee->opcode() == spv::Op::OpTypeArray) {
arrayed = true;
pointee = defUse->GetDef(pointee->GetSingleWordInOperand(0));
if (pointee == nullptr) continue;
}
BlockVariable block{&inst, storage, arrayed, 0, 0};
if (pointee->opcode() != spv::Op::OpTypeStruct ||
!ClipMemberOf(sites, pointee->result_id(), block.clipMember)) {
continue;
}
block.memberTypeId = pointee->GetSingleWordInOperand(block.clipMember);
if (native && !arrayed && storage == spv::StorageClass::Output) {
nativeBlockVar = &inst;
nativeBlockMember = block.clipMember;
}
if (!CollectBlockClipChains(ctx, block, shadowTypeId, memberChains, reason)) {
return decline(Move(reason));
}
}
const bool hasAccesses = !memberChains.empty() || !standaloneVars.empty();
if (shadowTypeId == 0 && m_options.role == Role::Producer) {
shadowTypeId = DeclaredOutputClipType(ctx, sites);
}
const uint32_t shadowLength = shadowTypeId != 0 ? FloatArrayLength(ctx, shadowTypeId) : 0;
if (shadowTypeId != 0 && shadowLength == 0)
return decline("gl_ClipDistance is not a sized float array");
Uint32 producedCount = 0;
if (m_options.role == Role::Producer) {
producedCount = std::min<Uint32>(shadowLength, kMaxCarriers * 4);
if (shadowLength > kMaxCarriers * 4) return decline("more than eight clip distances");
}
const Uint32 consumedCount =
m_options.role == Role::Consumer ? std::min<Uint32>(m_options.distanceCount, kMaxCarriers * 4) : 0;
const EnablesUniform enables = producedCount > 0 && !m_options.enablesMemberName.empty()
? FindEnablesUniform(ctx, m_options.enablesMemberName)
: EnablesUniform{};
const uint32_t enablesVarId = enables.varId;
const uint32_t enablesTypeId = enables.typeId;
if (!hasAccesses && sites.capability == nullptr && producedCount == 0 && consumedCount == 0) {
return Status::SuccessWithoutChange;
}
if (native && producedCount > 0 && nativeBlockVar == nullptr && nativeStandaloneVar == nullptr) {
return decline("no gl_ClipDistance output to write the masked distances back into");
}
// ---- Phase 2: MUTATION. Nothing below may decline. ----
const bool interfaceListsAllGlobals = ctx->module()->version() >= 0x00010400u;
uint32_t shadowId = 0;
if (shadowTypeId != 0 && (hasAccesses || producedCount > 0)) {
shadowId = AddGlobalVariable(ctx, shadowTypeId, spv::StorageClass::Private, kShadowName);
if (interfaceListsAllGlobals) entryPoint->AddOperand({SPV_OPERAND_TYPE_ID, {shadowId}});
}
for (Instruction* chain : memberChains) {
if (chain->NumInOperands() == 2) {
// The chain IS the whole array: every use now addresses the shadow.
ctx->ReplaceAllUsesWith(chain->result_id(), shadowId);
ctx->KillInst(chain);
continue;
}
Instruction::OperandList operands;
operands.push_back({SPV_OPERAND_TYPE_ID, {shadowId}});
for (uint32_t i = 2; i < chain->NumInOperands(); ++i) {
operands.push_back(chain->GetInOperand(i));
}
chain->SetInOperands(Move(operands));
chain->SetResultType(
typeMgr->FindPointerToType(PointeeOf(ctx, chain->type_id()), spv::StorageClass::Private));
defUse->AnalyzeInstUse(chain);
RetypeChainUsers(ctx, chain);
}
for (Instruction* var : standaloneVars) {
const uint32_t varId = var->result_id();
if (native) {
// The variable stays the built-in the flush writes; only the shader's
// own accesses move to the shadow.
ctx->ReplaceAllUsesWithPredicate(varId, shadowId, [](Instruction* user) {
return user->opcode() != spv::Op::OpDecorate && user->opcode() != spv::Op::OpName &&
user->opcode() != spv::Op::OpEntryPoint;
});
continue;
}
ctx->KillNamesAndDecorates(varId);
RemoveFromInterface(entryPoint, varId);
ctx->ReplaceAllUsesWith(varId, shadowId);
ctx->KillInst(var);
}
if (shadowId != 0) RetypeChainUsers(ctx, defUse->GetDef(shadowId));
if (sites.capability != nullptr && !native) ctx->KillInst(sites.capability);
const ScalarIds ids = MakeScalarIds(ctx);
const uint32_t floatTypeId = ids.floatType;
const uint32_t vec4TypeId = ids.vec4Type;
const uint32_t boolTypeId = ids.boolType;
const uint32_t oneId = ids.one;
// ---- producer: write the carriers ----
if (producedCount > 0) {
const Uint32 carrierCount = native ? 0 : (producedCount + 3) / 4;
uint32_t carriers[kMaxCarriers] = {0, 0};
const uint32_t ptrOutputFloat = typeMgr->FindPointerToType(floatTypeId, spv::StorageClass::Output);
const uint32_t nativeMemberConst =
nativeBlockVar != nullptr ? constMgr->GetSIntConstId(static_cast<int32_t>(nativeBlockMember))
: 0;
for (Uint32 k = 0; k < carrierCount; ++k) {
carriers[k] = AddGlobalVariable(ctx, vec4TypeId, spv::StorageClass::Output, CarrierName(k));
AddLocationDecoration(ctx, carriers[k], m_options.location + k);
entryPoint->AddOperand({SPV_OPERAND_TYPE_ID, {carriers[k]}});
}
uint32_t ptrUniformEnables = 0;
uint32_t enablesMemberConst = 0;
if (enablesVarId != 0) {
ptrUniformEnables = typeMgr->FindPointerToType(enablesTypeId, spv::StorageClass::Uniform);
enablesMemberConst = constMgr->GetSIntConstId(static_cast<int32_t>(enables.memberIndex));
if (interfaceListsAllGlobals) {
bool listed = false;
for (uint32_t i = 3; i < entryPoint->NumInOperands(); ++i) {
if (entryPoint->GetSingleWordInOperand(i) == enablesVarId) listed = true;
}
if (!listed) entryPoint->AddOperand({SPV_OPERAND_TYPE_ID, {enablesVarId}});
}
}
for (Instruction* site : FlushSites(ctx, entryPoint, isGeometry)) {
InstructionBuilder b(ctx, site, kBuilderAnalyses);
uint32_t maskId = 0;
if (enablesVarId != 0) {
Instruction* ptr = b.AddAccessChain(ptrUniformEnables, enablesVarId, {enablesMemberConst});
maskId = b.AddLoad(enablesTypeId, ptr->result_id())->result_id();
}
std::vector<uint32_t> values;
for (Uint32 i = 0; i < producedCount; ++i) {
Instruction* ptr =
b.AddAccessChain(ids.ptrPrivateFloat, shadowId, {constMgr->GetUIntConstId(i)});
uint32_t value = b.AddLoad(floatTypeId, ptr->result_id())->result_id();
if (maskId != 0) {
const uint32_t bit = b.AddBinaryOp(enablesTypeId, spv::Op::OpBitwiseAnd, maskId,
IntConstantOfType(ctx, enablesTypeId, 1u << i))
->result_id();
const uint32_t on = b.AddBinaryOp(boolTypeId, spv::Op::OpINotEqual, bit,
IntConstantOfType(ctx, enablesTypeId, 0u))
->result_id();
value = b.AddSelect(floatTypeId, on, value, oneId)->result_id();
}
values.push_back(value);
}
if (native) {
for (Uint32 i = 0; i < producedCount; ++i) {
const uint32_t index = constMgr->GetUIntConstId(i);
Instruction* ptr =
nativeBlockVar != nullptr
? b.AddAccessChain(ptrOutputFloat, nativeBlockVar->result_id(),
{nativeMemberConst, index})
: b.AddAccessChain(ptrOutputFloat, nativeStandaloneVar->result_id(), {index});
b.AddStore(ptr->result_id(), values[i]);
}
}
while (values.size() % 4 != 0)
values.push_back(oneId);
for (Uint32 k = 0; k < carrierCount; ++k) {
const uint32_t composite =
b.AddCompositeConstruct(vec4TypeId, {values[k * 4], values[k * 4 + 1],
values[k * 4 + 2], values[k * 4 + 3]})
->result_id();
b.AddStore(carriers[k], composite);
}
}
if (m_report != nullptr) {
m_report->distanceCount = producedCount;
m_report->enablesFound = enablesVarId != 0;
}
}
// ---- consumer: read the carriers, discard, serve gl_ClipDistance reads ----
if (m_options.role == Role::Consumer && consumedCount > 0) {
EmitCarrierDiscard(ctx, entryPoint, ids, consumedCount, m_options.location, shadowId, shadowLength);
}
if (m_report != nullptr) m_report->rewrote = true;
ctx->InvalidateAnalysesExceptFor(IRContext::kAnalysisNone);
return Status::SuccessWithChange;
}
spvtools::Optimizer::PassToken EmulateClipDistancePass::CreateEmulateClipDistancePass(
EmulateClipDistanceOptions options, EmulateClipDistanceReport* report) {
return spvtools::Optimizer::PassToken(MakeUnique<EmulateClipDistancePass>(Move(options), report));
}
} // namespace ShaderTranspiler
} // namespace MG_Util
} // namespace MobileGL
@@ -0,0 +1,140 @@
// MobileGL - MobileGL/MG_Util/ShaderTranspiler/SpirvPasses/EmulateClipDistancePass.h
// Copyright (c) 2025-2026 MobileGL-Dev
// Licensed under the GNU Lesser General Public License v3.0:
// https://www.gnu.org/licenses/gpl-3.0.txt
// https://www.gnu.org/licenses/lgpl-3.0.txt
// SPDX-License-Identifier: LGPL-3.0-only
// End of Source File Header
#pragma once
#include "source/opt/pass.h"
#include "spirv-tools/optimizer.hpp"
#include <Includes.h>
namespace MobileGL {
namespace MG_Util {
namespace ShaderTranspiler {
// Emulates gl_ClipDistance on a backend that cannot host the built-in at all: an ES
// driver without GL_EXT_clip_cull_distance, or a Vulkan device with
// shaderClipDistance == VK_FALSE (every Mali GPU measured so far is both). GL 3.0+
// requires GL_MAX_CLIP_DISTANCES >= 8, so a frontend that reports GL 4.6 cannot
// honestly answer 0 there, and answering 8 without this pass makes every clipping
// draw either fail to compile or render unclipped.
//
// WHAT THE EMULATION IS. Clip distance d is linear in clip space, so its
// perspective-correct interpolant at a fragment is exactly the distance the
// rasterizer's clipper would have evaluated there; discarding the fragments where it
// is negative removes the same pixels hardware clipping removes. So:
// - every access to the ClipDistance built-in, in every stage, is redirected onto
// a Private float[N] shadow (mg_ClipDistanceShadow) and the ClipDistance
// capability is stripped - the gl_PerVertex member stays declared but unaccessed,
// which is the shape glslang emits for every program that never clips;
// - the LAST pre-rasterization stage (the PRODUCER) writes
// `enabled(i) ? shadow[i] : 1.0` into ceil(N/4) ordinary vec4 varyings
// (mg_ClipDistanceEmu0/1) at every return of its entry function - before every
// EmitVertex in a geometry stage, which is when one vertex's outputs latch.
// enabled(i) is bit i of the reserved default-block uniform the source pipeline
// declared (CLIP_DISTANCE_ENABLES_UNIFORM_NAME), which the draw path refreshes
// from GL_CLIP_DISTANCE0..7 before every draw. Applying the enables HERE keeps
// the varyings free of pipeline-key state and makes a disabled distance - even
// one the shader writes negative - clip nothing, as GL requires;
// - the fragment stage (the CONSUMER) reads the carriers before anything else and
// executes OpKill when any carried distance is negative. Its own gl_ClipDistance
// reads are served from the same carriers (GLSL leaves the disabled elements
// undefined there, so the 1.0 they carry is a legal value).
// - every other stage that touches the built-in (a vertex stage ahead of a
// geometry stage, say) only has its accesses redirected: in GL those writes are
// dead unless a later stage reads gl_in[].gl_ClipDistance.
//
// WHAT IT COSTS: two varying locations past every location the program uses, and a
// discard in the fragment stage of every clipping program (early depth/stencil is
// lost for those programs). Cull distance is NOT emulated: culling is per primitive,
// which a per-fragment discard cannot reproduce.
//
// WHAT DECLINES (the module is left byte-identical and the report says why):
// gl_in[].gl_ClipDistance reads and gl_out[].gl_ClipDistance writes (arrayed
// per-vertex access in tessellation/geometry stages), a non-constant gl_PerVertex
// member index, whole-gl_PerVertex loads/stores/copies, and a pointer into the
// built-in that escapes into an opcode the pass cannot retype (a function argument).
//
// NATIVE MASKING (EmulateClipDistanceOptions::nativeBuiltin) is the same producer
// rewrite for a backend that hosts the built-in but cannot switch distances off: the
// masked values go back into gl_ClipDistance instead of into carriers.
//
// One module per run; the program-wide contract (one shared location, which stage
// is the producer, every stage rewritten or none) is owned by
// ShaderCompiler::EmulateClipDistanceForProgram, the only caller.
struct EmulateClipDistanceOptions {
enum class Role : Uint8 {
// Redirect the accesses and strip the capability; no carriers.
Redirect,
// The last pre-rasterization stage: redirect, then write the carriers.
Producer,
// The fragment stage: read the carriers, discard, serve gl_ClipDistance reads.
Consumer,
};
Role role = Role::Redirect;
// The first carrier location; carrier k sits at location + k.
Uint32 location = 0;
// Consumer only: how many distances the producer carries (0 = none, in which
// case the stage only has its own reads redirected).
Uint32 distanceCount = 0;
// Producer only: the default-block uniform member holding the enable bits. When
// the module has no such member every distance counts as enabled.
String enablesMemberName;
// Producer only: the backend HOSTS the built-in (a Vulkan device with
// shaderClipDistance) but has no per-distance enable state - Vulkan clips by
// every distance a shader writes. Instead of carriers, the producer writes
// `enabled(i) ? d[i] : 1.0` back into gl_ClipDistance itself at the same flush
// points, so hardware clipping stays and a disabled distance clips nothing. The
// capability is kept, nothing else in the program is touched, and gl_in /
// fragment-stage reads stay native.
Bool nativeBuiltin = false;
};
struct EmulateClipDistanceReport {
Bool declined = false;
String declineReason;
Bool rewrote = false;
// Producer: how many distances it now carries.
Uint32 distanceCount = 0;
// Producer: whether the enables uniform was found (false = all-enabled fallback).
Bool enablesFound = false;
};
// What one module looks like to the program-wide driver, from a single parse.
struct ClipDistanceModuleProbe {
Bool parsed = false;
// glslang declares the capability on first ACCESS, so this is the trigger.
Bool declaresClipDistanceCapability = false;
// The length of the module's non-arrayed ClipDistance OUTPUT (block member or
// standalone variable), or 0 when it declares none.
Uint32 outputDistanceCount = 0;
};
class EmulateClipDistancePass : public spvtools::opt::Pass {
public:
static constexpr const char* kShadowName = "mg_ClipDistanceShadow";
static constexpr Uint32 kMaxCarriers = 2;
static const char* CarrierName(Uint32 index) {
return index == 0 ? "mg_ClipDistanceEmu0" : "mg_ClipDistanceEmu1";
}
EmulateClipDistancePass(EmulateClipDistanceOptions options, EmulateClipDistanceReport* report)
: m_options(Move(options)), m_report(report) {}
const char* name() const override { return "mobilegl-emulate-clip-distance"; }
Status Process() override;
static ClipDistanceModuleProbe ProbeModule(const Vector<Uint32>& spirv);
static spvtools::Optimizer::PassToken CreateEmulateClipDistancePass(EmulateClipDistanceOptions options,
EmulateClipDistanceReport* report);
private:
EmulateClipDistanceOptions m_options;
EmulateClipDistanceReport* m_report;
};
} // namespace ShaderTranspiler
} // namespace MG_Util
} // namespace MobileGL
@@ -28,6 +28,14 @@ namespace MobileGL {
// uses gl_NumSamples the link task will still hide its uniform. That is the same // uses gl_NumSamples the link task will still hide its uniform. That is the same
// bargain every mg_-prefixed rewrite in this pipeline strikes. // bargain every mg_-prefixed rewrite in this pipeline strikes.
inline const char* NUM_SAMPLES_UNIFORM_NAME = "mg_NumSamples"; inline const char* NUM_SAMPLES_UNIFORM_NAME = "mg_NumSamples";
// The default-block uniform InjectClipDistanceEnablesShim declares, on a backend that
// emulates gl_ClipDistance (BackendCaps::EmulatesClipDistance), in every
// pre-rasterization stage that names the built-in: bit i is GL_CLIP_DISTANCE0 + i's
// enable. EmulateClipDistancePass reads it where the producing stage writes its
// carriers; the draw path refreshes it from the render state before every draw, and
// the link task hides it from the GL uniform surface exactly as it hides
// NUM_SAMPLES_UNIFORM_NAME. Reserved on the same terms.
inline const char* CLIP_DISTANCE_ENABLES_UNIFORM_NAME = "mg_ClipDistanceEnables";
// glslang's Vulkan-relaxed parse rewrites every atomic_uint into a member of a // glslang's Vulkan-relaxed parse rewrites every atomic_uint into a member of a
// synthesized storage block named "<this>_<GL atomic-counter binding>" // synthesized storage block named "<this>_<GL atomic-counter binding>"
// (ParseContextBase::growAtomicCounterBlock). That block IS the GL atomic counter // (ParseContextBase::growAtomicCounterBlock). That block IS the GL atomic counter
+132
View File
@@ -74,3 +74,135 @@ adb shell am start -a top.mobilegl.plugin.TRACE_REPLAY \
--es ssim_threshold 0.99 --es ssim_threshold 0.99
adb shell run-as top.mobilegl.plugin.espryt.trace cat files/trace-replay/output/result.json adb shell run-as top.mobilegl.plugin.espryt.trace cat files/trace-replay/output/result.json
``` ```
## Emulator shards in CI
The APK workflow replays its cases through `trace-replay-shard.sh`: one emulator boot, one APK
install, then an ordered list of cases for one lane, transport and backend. Each case still runs
through `trace-replay-ci.sh`, with the arguments and environment of a one-case job and the
transport proof of its lane (see [Transport proofs](#transport-proofs)).
```sh
sh android-plugin/trace-replay-shard.sh \
--apk-file MobileGL-plugin-trace-release.apk \
--backend DirectVulkan --gpu lavapipe \
--transport inproc --lane inproc-acceptance \
--cases-file cases.json --deadline-seconds 7200
```
The case list is a JSON array of case objects: the `cases` member of a row of
`trace_cases.py --ci --format github-apk-transport-shard-matrix`, which is what the workflow
passes, or the `case` members of `--format github-apk-transport-matrix` rows. `--dry-run` prints
what each case would be run with and touches no device.
The workflow's matrix is that shard matrix: one row per emulator job, holding some of the cases
of one lane, transport and backend. `--shard-seconds` (`RETRACE_SHARD_SECONDS` in `apk.yml`) sets
how much hinted replay time one shard aims for, and `trace_cases.py` refuses to print a shard
matrix whose shards, taken apart, are not exactly the per-case matrix. The fixtures of a shard's
cases must be in `tools/trace_replay/fixtures` before the runner starts; the workflow restores
them from one cache entry (`.github/scripts/trace-fixture-bundle.sh`).
Unless a manual run sets `android_full_matrix`, the workflow passes `--android-lanes
inproc-full`: the inproc lane, the transport the launcher ships, replays every CI case, and the
monolith and spawn lanes replay the cases `trace_cases.json` lists under `android_smoke`. The
Linux retrace jobs of `test.yml` replay every case on all three transports, so those two
emulator lanes keep the cases that prove the Android side of each arm: the server started from
the APK's native library directory, the pbuffer surface of the spawn lane, the monolith build
of the same commit as a control, and every per-case replay setting at least once. A manual run
with `android_full_matrix` replays every case on all three lanes. A branch that runs only the
monolith lane replays every case on it.
Between cases the runner force-stops the app, checks that no process of the app's uid is left,
clears the app data, removes the case's files from `/data/local/tmp` and from the host, and
checks that the device still answers. It restarts the emulator when it does not, before the
case that follows one that did not pass, and before the single retry of a case that
`trace-replay-ci.sh` charged to the infrastructure (exit 75).
`MOBILEGL_SHARD_RESTART_EMULATOR_EACH_CASE=1` cold boots before every case.
The cases of a shard share the job's time. `--deadline-seconds` stops the runner that long after
it starts, so that the job can still upload what it has: a case that is running then, or has not
started, is recorded as `not-run`. `retrace_shard_budget.py` reads a shard matrix and adds two
fields to every shard: `deadline_seconds`, the time in which every case of the shard could use
its whole time budget (`timeout_seconds` plus 300 s) when the cases before it used all of
theirs, with one infrastructure retry of the longest budget and a cold boot per case on top,
and `timeout_minutes`, the limit of the shard's job. `--cap-seconds`
(`RETRACE_SHARD_DEADLINE_CAP_SECONDS` in `apk.yml`) bounds the deadline, so a shard whose
replays all hang cannot hold a runner for hours; a capped shard loses verdicts only when several
of its replays hang to the end of their budgets.
A failing case does not stop the shard. Every case ends with `<case>-<backend>/case-status.json`
(`passed`, `failed` or `not-run`), the result root gets `shard-status.json`, the job log gets one
`::error` per case that did not pass, and the exit status is non-zero when a case did not pass or
a verdict is missing. `--finalize` writes the summary inputs and a `not-run` verdict for every
case without one, without a device; the workflow runs it after the replay step whatever happened.
### Transport proofs
Every lane passes `trace-replay-ci.sh` the flag that requires the proof of its transport. The
check runs after a replay that passed, reads the library's own log (`mobilegl.log`, the client
role's log followed by the server role's), writes `transport-proof.json` and prints one
`Android retrace: ...` line. A case whose proof fails is a failed case.
| transport | flag | what the library log must show |
| --- | --- | --- |
| monolith | `--require-monolith` | nothing a split transport writes: neither `Config: MOBILEGL_TRANSPORT=... - the MGPipe record stream` sentence, no `Config: IPC` line, no `spawn ARMED` line, no library line in a server role log |
| inproc | `--require-inproc` | the inproc transport sentence, `strict=1`, `role-split-state=1` and `run-ahead=1` on every `Config: IPC` line, no `Fatal{` record |
| spawn | `--require-spawn` | the spawn transport sentence, `spawn ARMED - the server role runs in pid N`, the same settings on the client's `Config: IPC` line, no `Fatal{` record |
The monolith proof is one of absence. The library logs a transport sentence only when it
resolves inproc or spawn, and a library built without the split transports has no such code, so
the check fails only on what a split run writes. Its line says how much the log proves:
- `Android retrace: monolith confirmed: ...` when the log shows the configuration load
(`Config loaded`), during which the transport sentence and the `Config: IPC` line are
written, and nothing of the table row above.
- `Android retrace: monolith not confirmed (...)` when the log is missing or lacks the load. A
lost log says nothing about the transport, so the case still passes; its
`transport-proof.json` has `"confirmed": false` and the reason.
The check reads a split APK run with `MOBILEGL_TRANSPORT=monolith` and an APK built without
the split transports alike. That build writes one log, `mobilegl.log` itself, which the script
pulls whenever no role log arrived.
### What the job log holds
The workflow deletes the result artifacts of a green run, so the job log repeats what the
result files say. The shard starts with one `trace-replay-shard: every run: {...}` line, and
each case's group holds:
```text
trace-replay-shard: run <case>: {"arguments": [...], "case": "<case>", "environment": {...}, "timeout": N}
(the output of trace-replay-ci.sh, ending with its "Android retrace: ..." proof line)
trace-replay-shard: library-log <case>: mobilegl.log: <a line of the library log>
trace-replay-shard: phases <case>: {"fixture-pushed": 2.5, "apk-installed": 0.0, ...}
trace-replay-shard: record <case>: {"attempts": 1, "case": "<case>", "exit_codes": [0], ...}
trace-replay-shard: emulator boots after <case>: 1
trace-replay-shard: <case>: passed
```
- `every run` is the replay command, the environment the runner adds to every case, and
under `inherited` each further variable `trace-replay-ci.sh` reads (`ADB`, `PYTHON`,
`MOBILEGL_RETRACE_USE_PBUFFER`, `MOBILEGL_TRACE_ENV`) with the value it has in the runner's
own environment, which a replay inherits, or `null` when it is not set there.
- `run` is the line `--dry-run` prints for the case: the argument list `trace-replay-ci.sh` is
started with, the case's replay environment and its time budget. The same object starts the
replay, so the two cannot differ.
- `library-log` quotes the lines of the pulled library log that say which transport ran: the
backend and `Config loaded` lines, every line naming `MOBILEGL_TRANSPORT`, the `Config: IPC`
and `spawn ARMED` lines, and `Fatal{` records. At most 12 lines are quoted per case, each cut
to 1000 characters with characters outside ASCII escaped, and a further line says how many
were left out. The log is that of the attempt that counts. A failed case is quoted too, and
a case whose replay script ended before its own pulls is quoted from the logs the runner
pulls: the two role logs, or the one log of a build without the split transports.
- `phases` is the seconds between one mark of `phase-timings.txt` and the next, named by the
later mark, for the attempt that counts.
- `record` is `case-status.json`: status, reason, attempts, exit codes, seconds and notes.
- The verdict line is the last line of the group.
The shard ends with `trace-replay-shard: shard-status: {...}`, the content of
`shard-status.json`, and then the tally line. `--finalize` prints the same line for the file as
it leaves it.
`python3 android-plugin/test_trace_replay_shard.py` drives the runner with a fake adb, replay
script and AVD script, and runs `trace-replay-ci.sh` itself against an adb that serves a host
directory.
+1 -1
View File
@@ -80,7 +80,7 @@ val releaseSigningReady = signingStoreFile.exists()
val debuggableRelease = (findProperty("mobilegl.debuggableRelease") ?: "false").toString().toBoolean() val debuggableRelease = (findProperty("mobilegl.debuggableRelease") ?: "false").toString().toBoolean()
val mobileGlVersionMajor = 26 val mobileGlVersionMajor = 26
val mobileGlVersionMinor = 10 val mobileGlVersionMinor = 11
val mobileGlGitShortHash = runGit("rev-parse", "--short=7", "HEAD") ?: "nogit" val mobileGlGitShortHash = runGit("rev-parse", "--short=7", "HEAD") ?: "nogit"
val mobileGlMonthlyRevision = runGit( val mobileGlMonthlyRevision = runGit(
"rev-list", "rev-list",
+235
View File
@@ -0,0 +1,235 @@
#!/usr/bin/env python3
# MobileGL - android-plugin/retrace_artifact_cleanup.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""Delete the intermediate artifacts of an APK workflow run, keeping what a failure needs.
The retrace summary embeds every image it shows, so once it is uploaded the per-shard result
artifacts are only worth keeping for cases that did not pass:
- a result artifact stays when its shard-status.json is missing, when its job failed, or when
it holds a case that did not pass in any shard (so both backends and all transports of a
failed case stay comparable);
- every result artifact stays when a job failed without leaving a shard-status.json to say
which cases it was running.
A canceled job retains nothing by itself; a failed verdict inside it still does. The trace
fixtures are not artifacts: the retrace jobs restore them from a cache entry.
"""
import argparse
import json
import os
import shlex
import subprocess
import sys
import threading
import time
from pathlib import Path
RESULT_PREFIXES = ("MobileGL-android-retrace-result-", "retrace-result-")
FAILED_CONCLUSIONS = ("failure", "timed_out", "action_required")
# GitHub's secondary rate limit allows about 180 mutating REST calls a minute. Requests start
# at most this often, whatever the number of workers.
DELETE_INTERVAL_SECONDS = 60.0 / 170.0
DELETE_WORKERS = 3
MAX_REFUSED_DELETES = 5
def pause(seconds):
time.sleep(seconds)
def read_rows(text, columns):
rows = []
for line in text.splitlines():
if line.strip():
fields = line.split("\t")
rows.append(fields + [""] * (columns - len(fields)))
return rows
def shard_statuses(download_roots):
"""Artifact name -> its shard-status.json, for every downloaded result artifact that has one."""
statuses = {}
for root in download_roots:
root = Path(root)
if not root.is_dir():
continue
for directory in root.iterdir():
try:
status = json.loads((directory / "shard-status.json").read_text(encoding="utf-8"))
except (OSError, ValueError):
continue
if isinstance(status, dict) and isinstance(status.get("cases"), list):
statuses[directory.name] = status
return statuses
def plan(artifacts, jobs, statuses):
"""Returns (ids to delete, lines explaining what stays)."""
failed_job_ids = {job_id for job_id, conclusion, _ in jobs if conclusion in FAILED_CONCLUSIONS}
status_job_ids = {str(status.get("job_id") or "") for status in statuses.values()} - {""}
# Failed jobs that left no status file: nothing says which cases they were running.
unexplained = failed_job_ids - status_job_ids
if not status_job_ids and failed_job_ids:
# The statuses carry no job ids to match against. Count instead: each failed job must
# be explained by an artifact that shows a case without a pass.
explaining = sum(1 for status in statuses.values()
if any(record.get("status") != "passed" for record in status["cases"]))
unexplained = set() if explaining >= len(failed_job_ids) else failed_job_ids
failed_cases = set() # did not pass somewhere: every result that holds one stays
for status in statuses.values():
not_passed = [record for record in status["cases"] if record.get("status") != "passed"]
if status_job_ids:
job_failed = str(status.get("job_id") or "") in failed_job_ids
else:
# Without job ids, a status that shows a case without a pass stands for a failed job.
job_failed = bool(failed_job_ids) and bool(not_passed)
for record in not_passed:
if record.get("status") == "failed" or job_failed:
failed_cases.add(record.get("case"))
delete, kept = [], []
for artifact_id, name in artifacts:
if not name.startswith(RESULT_PREFIXES):
continue
reason = ""
status = statuses.get(name)
if unexplained:
reason = f"{len(unexplained)} failed job(s) left no shard status"
elif status is None:
reason = "no shard status was downloaded for it"
elif str(status.get("job_id") or "") in failed_job_ids:
reason = "its job failed"
else:
cases = sorted(failed_cases.intersection(record.get("case") for record in status["cases"]))
if cases:
reason = "holds failed case(s): " + ", ".join(cases)
if reason:
kept.append(f"{name} ({artifact_id}): {reason}")
else:
delete.append((artifact_id, name))
return delete, kept
class Deleter:
"""Runs `command` once per artifact, with the artifact id appended to its last word."""
def __init__(self, command):
self.command = command
self.lock = threading.Lock()
self.next_start = 0.0
self.refused = []
self.deleted = 0
def wait_for_slot(self):
with self.lock:
now = time.monotonic()
start = max(now, self.next_start)
self.next_start = start + DELETE_INTERVAL_SECONDS
if start > now:
pause(start - now)
def delete(self, artifact_id, name):
for attempt in (1, 2):
with self.lock:
if len(self.refused) >= MAX_REFUSED_DELETES:
return
self.wait_for_slot()
done = subprocess.run(self.command[:-1] + [self.command[-1] + artifact_id],
capture_output=True, text=True)
if done.returncode == 0:
with self.lock:
self.deleted += 1
return
if attempt == 1:
# Give a rate limit time to clear before the one retry.
with self.lock:
self.next_start = max(self.next_start, time.monotonic() + 30)
with self.lock:
self.refused.append(f"{name} ({artifact_id}): {' '.join(done.stderr.split())[:200]}")
def run(self, entries):
pending = list(entries)
def work():
while True:
with self.lock:
if not pending:
return
artifact_id, name = pending.pop(0)
self.delete(artifact_id, name)
threads = [threading.Thread(target=work) for _ in range(min(DELETE_WORKERS, len(pending)))]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
def gh_rows(path, query, columns):
done = subprocess.run(["gh", "api", "--paginate", path, "--jq", query], capture_output=True, text=True)
if done.returncode != 0:
raise RuntimeError(f"gh api {path} failed: {done.stderr.strip()}")
return read_rows(done.stdout, columns)
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--downloads", action="append", default=[],
help="Directory holding one sub-directory per downloaded result artifact.")
parser.add_argument("--artifacts-file", help="Tab-separated id and name per line, instead of asking gh.")
parser.add_argument("--jobs-file", help="Tab-separated id, conclusion and name per line, instead of asking gh.")
parser.add_argument("--dry-run", action="store_true", help="Print the plan, delete nothing.")
options = parser.parse_args(argv)
repository = os.environ.get("GITHUB_REPOSITORY", "")
run_id = os.environ.get("GITHUB_RUN_ID", "")
try:
if options.artifacts_file:
artifacts = read_rows(Path(options.artifacts_file).read_text(encoding="utf-8"), 2)
else:
artifacts = gh_rows(f"repos/{repository}/actions/runs/{run_id}/artifacts?per_page=100",
".artifacts[] | [.id, .name] | @tsv", 2)
if options.jobs_file:
jobs = read_rows(Path(options.jobs_file).read_text(encoding="utf-8"), 3)
else:
# 50 rows per page: the jobs endpoint answers HTTP 502 for larger pages of a re-run
# attempt of a run this size.
jobs = gh_rows(f"repos/{repository}/actions/runs/{run_id}/jobs?per_page=50",
'.jobs[] | [.id, .conclusion // "", .name] | @tsv', 3)
except (OSError, RuntimeError) as error:
print(f"::error::Could not read the run's artifacts and jobs; nothing was deleted: {error}")
return 1
delete, kept = plan([(row[0], row[1]) for row in artifacts], [tuple(row[:3]) for row in jobs],
shard_statuses(options.downloads))
for line in kept:
print(f"Retaining {line}")
if options.dry_run:
for artifact_id, name in delete:
print(f"Would delete {name} ({artifact_id})")
return 0
deleter = Deleter(shlex.split(os.environ.get("MOBILEGL_CLEANUP_DELETE_COMMAND") or
f"gh api --method DELETE repos/{repository}/actions/artifacts/"))
started = time.monotonic()
deleter.run(delete)
for line in deleter.refused:
print(f"::warning::Could not delete intermediate artifact {line}")
if len(deleter.refused) >= MAX_REFUSED_DELETES:
print("::warning::Stopped deleting intermediate artifacts after repeated refusals; the rest stay until they expire.")
print(f"Deleted {deleter.deleted} intermediate Android artifact(s) in {time.monotonic() - started:.0f}s; "
f"retained {len(kept)}.")
return 0
if __name__ == "__main__":
sys.exit(main())
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
# MobileGL - android-plugin/retrace_shard_budget.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""Give every emulator shard the deadline and the job timeout its own cases need.
The cases of a shard share one job, so one case that hangs spends time the cases after it
need. This reads a shard matrix of trace_cases.py (--format github-apk-shard-matrix or
github-apk-transport-shard-matrix) on standard input, or from --matrix-file, and prints it
with two fields added to every shard: deadline_seconds, which the retrace job passes to
trace-replay-shard.sh, and timeout_minutes, the limit of that job. The deadline is what the
shard needs for every case to use its whole time budget (trace_replay_shard.needed_seconds),
up to --cap-seconds. A capped shard still replays every case; only when several of its
replays hang to the end of their budgets are the cases after them recorded as not-run.
"""
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import trace_replay_shard # noqa: E402 (a sibling file, found through the line above)
def shard_label(row):
backend = row.get("backend")
backend = backend.get("name", "") if isinstance(backend, dict) else str(backend or "")
return f"{backend} / {row.get('transport') or 'monolith'}, shard {row.get('shard')}/{row.get('shards')}"
def shard_needs(matrix):
"""(label, case count, seconds needed) for every shard of the matrix."""
rows = matrix.get("include") if isinstance(matrix, dict) else None
if not isinstance(rows, list) or not rows:
raise ValueError("the matrix has no shard rows under 'include'")
needs = []
for row in rows:
cases = row.get("cases") if isinstance(row, dict) else None
if not isinstance(cases, list) or not cases:
raise ValueError(f"a shard row has no 'cases' list: {str(row)[:200]}")
needs.append((shard_label(row), len(cases), trace_replay_shard.needed_seconds(cases)))
return needs
# Minutes a retrace job has around its shard runner: the setup before the replays and the
# uploads after them.
JOB_OVERHEAD_MINUTES = 15
# A hosted job may run six hours.
HOSTED_JOB_LIMIT_MINUTES = 360
def with_limits(matrix, cap_seconds):
"""The matrix with deadline_seconds and timeout_minutes on every shard, and the report."""
needs = shard_needs(matrix)
rows = []
report = []
for row, (label, cases, seconds) in zip(matrix["include"], needs):
deadline = min(seconds, cap_seconds)
rows.append({**row, "deadline_seconds": deadline,
"timeout_minutes": -(-deadline // 60) + JOB_OVERHEAD_MINUTES})
if seconds > cap_seconds:
report.append(f"retrace shard budget: {label} would need {seconds} s for all of its "
f"{cases} case(s) to use their whole budgets; capped at {cap_seconds} s")
largest = max(needs, key=lambda entry: entry[2])
report.append(f"retrace shard budget: {len(needs)} shard(s), {len(report)} capped at "
f"{cap_seconds} s; the most a shard needs is {largest[2]} s ({largest[0]}, "
f"{largest[1]} case(s))")
return {**matrix, "include": rows}, report
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--cap-seconds", type=int, required=True,
help="The longest deadline a shard is given.")
parser.add_argument("--matrix-file", help="The shard matrix; standard input when not given.")
options = parser.parse_args(argv)
limit = (HOSTED_JOB_LIMIT_MINUTES - JOB_OVERHEAD_MINUTES) * 60
if not 0 < options.cap_seconds <= limit:
print(f"retrace shard budget: --cap-seconds must be between 1 and {limit}", file=sys.stderr)
return 2
if options.matrix_file:
text = Path(options.matrix_file).read_text(encoding="utf-8")
else:
text = sys.stdin.read()
try:
matrix, report = with_limits(json.loads(text), options.cap_seconds)
except (ValueError, KeyError, TypeError) as error:
print(f"retrace shard budget: the shard matrix cannot be read: {error}", file=sys.stderr)
return 2
for line in report:
print(line, file=sys.stderr)
print(json.dumps(matrix, separators=(",", ":")))
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,178 @@
#!/usr/bin/env python3
# MobileGL - android-plugin/test_retrace_artifact_cleanup.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""Pin which intermediate APK artifacts the cleanup keeps, and how fast it deletes the rest."""
import contextlib
import importlib.util
import io
import json
import os
import pathlib
import sys
import tempfile
import unittest
from unittest import mock
HERE = pathlib.Path(__file__).resolve().parent
_spec = importlib.util.spec_from_file_location("retrace_artifact_cleanup", HERE / "retrace_artifact_cleanup.py")
cleanup = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(cleanup)
RESULT = "MobileGL-android-retrace-result-"
def status(job_id, **cases):
return {"job_id": job_id, "cases": [{"case": name, "status": verdict} for name, verdict in cases.items()]}
def names(entries):
return sorted(name for _, name in entries)
class PlanTest(unittest.TestCase):
ARTIFACTS = [("1", RESULT + "mono-gles"), ("2", RESULT + "inproc-gles"), ("3", RESULT + "spawn-vk"),
("4", "mobilegl-android-monolith-control.html"), ("5", "MobileGL-plugin-release-abc.apk"),
("6", "MobileGL-plugin-trace-release-abc.apk")]
def test_a_green_run_deletes_every_intermediate_and_nothing_else(self):
statuses = {RESULT + "mono-gles": status("11", a="passed", b="passed"),
RESULT + "inproc-gles": status("12", a="passed", b="passed"),
RESULT + "spawn-vk": status("13", c="passed")}
jobs = [("11", "success", "x"), ("12", "success", "y"), ("13", "success", "z"), ("14", "", "retrace summary")]
delete, kept = cleanup.plan(self.ARTIFACTS, jobs, statuses)
self.assertEqual(kept, [])
self.assertEqual(names(delete), sorted(name for _, name in self.ARTIFACTS[:3]))
def test_a_failed_case_keeps_every_shard_that_ran_it(self):
statuses = {RESULT + "mono-gles": status("11", a="passed", b="passed"),
RESULT + "inproc-gles": status("12", a="passed", b="failed"),
RESULT + "spawn-vk": status("13", c="passed")}
jobs = [("11", "success", "x"), ("12", "failure", "y"), ("13", "success", "z")]
delete, kept = cleanup.plan(self.ARTIFACTS, jobs, statuses)
# Only b's results are worth comparing, so the monolith shard stays for b and the
# spawn shard goes.
self.assertEqual(names(delete), [RESULT + "spawn-vk"])
self.assertEqual(len(kept), 2)
self.assertTrue(any("inproc-gles" in line and "its job failed" in line for line in kept))
self.assertTrue(any("mono-gles" in line and "holds failed case(s): b" in line for line in kept))
def test_cases_a_failed_job_never_reached_are_kept(self):
statuses = {RESULT + "mono-gles": status("11", a="passed", b="not-run"),
RESULT + "spawn-vk": status("13", c="passed")}
jobs = [("11", "timed_out", "x"), ("13", "success", "z")]
delete, kept = cleanup.plan(self.ARTIFACTS[:1] + self.ARTIFACTS[2:], jobs, statuses)
self.assertEqual(names(delete), [RESULT + "spawn-vk"])
self.assertEqual(len(kept), 1)
def test_a_job_that_failed_after_its_cases_passed_keeps_its_results(self):
statuses = {RESULT + "mono-gles": status("11", a="passed", b="passed"),
RESULT + "spawn-vk": status("13", c="passed")}
jobs = [("11", "failure", "upload failed"), ("13", "success", "z")]
delete, kept = cleanup.plan(self.ARTIFACTS[:1] + self.ARTIFACTS[2:], jobs, statuses)
self.assertEqual(names(delete), [RESULT + "spawn-vk"])
self.assertTrue("mono-gles" in kept[0] and "its job failed" in kept[0])
def test_a_canceled_job_keeps_nothing_by_itself(self):
statuses = {RESULT + "mono-gles": status("11", a="passed", b="not-run"),
RESULT + "inproc-gles": status("12", a="failed", b="not-run")}
jobs = [("11", "cancelled", "x"), ("12", "cancelled", "y")]
delete, kept = cleanup.plan(self.ARTIFACTS[:2], jobs, statuses)
# Case a failed in the inproc shard, so both shards that ran it stay.
self.assertEqual(delete, [])
self.assertEqual(len(kept), 2)
only_not_run = {RESULT + "mono-gles": status("11", a="passed", b="not-run")}
delete, kept = cleanup.plan(self.ARTIFACTS[:1], [("11", "cancelled", "x")], only_not_run)
self.assertEqual((names(delete), kept), ([RESULT + "mono-gles"], []))
def test_a_result_without_a_status_is_kept(self):
statuses = {RESULT + "mono-gles": status("11", a="passed")}
delete, kept = cleanup.plan(self.ARTIFACTS[:2], [("11", "success", "x")], statuses)
self.assertEqual(names(delete), [RESULT + "mono-gles"])
self.assertIn("no shard status", kept[0])
def test_a_failed_job_without_a_status_keeps_everything(self):
statuses = {RESULT + "mono-gles": status("11", a="passed", b="passed")}
jobs = [("11", "success", "x"), ("12", "failure", "died before its upload")]
delete, kept = cleanup.plan(self.ARTIFACTS, jobs, statuses)
self.assertEqual(delete, [])
self.assertEqual(len(kept), 3)
def test_statuses_without_job_ids_fall_back_to_counting(self):
artifacts = self.ARTIFACTS[:1] + self.ARTIFACTS[2:]
explained = {RESULT + "mono-gles": status("", a="passed", b="not-run"),
RESULT + "spawn-vk": status("", c="passed")}
delete, _ = cleanup.plan(artifacts, [("12", "failure", "y")], explained)
# The one status with a case that did not pass stands for the one failed job.
self.assertEqual(names(delete), [RESULT + "spawn-vk"])
unexplained = {RESULT + "mono-gles": status("", a="passed", b="passed"),
RESULT + "spawn-vk": status("", c="passed")}
delete, _ = cleanup.plan(artifacts, [("12", "failure", "y")], unexplained)
self.assertEqual(delete, [])
class RunTest(unittest.TestCase):
@staticmethod
def passed(artifacts):
"""A shard status with one passed case for every artifact, each from a job of its own."""
return {name: status(f"9{artifact_id}", a="passed") for artifact_id, name in artifacts}
def run_cleanup(self, artifacts, jobs, statuses, exit_code=0, dry_run=False):
with tempfile.TemporaryDirectory() as directory:
root = pathlib.Path(directory)
for name, value in statuses.items():
(root / "downloads" / "lane" / name).mkdir(parents=True)
(root / "downloads" / "lane" / name / "shard-status.json").write_text(json.dumps(value))
(root / "artifacts.tsv").write_text("".join(f"{i}\t{n}\n" for i, n in artifacts))
(root / "jobs.tsv").write_text("".join("\t".join(job) + "\n" for job in jobs))
# One file per call: the deletes run in parallel processes.
(root / "deleted").mkdir()
script = root / "delete.py"
script.write_text("import pathlib, sys\n"
"(pathlib.Path(sys.argv[1]) / sys.argv[2].replace('/', '_')).write_text('')\n"
f"sys.exit({exit_code})\n")
python = pathlib.Path(sys.executable).as_posix()
command = f'"{python}" "{script.as_posix()}" "{(root / "deleted").as_posix()}" artifacts/'
pauses = []
output = io.StringIO()
with mock.patch.dict(os.environ, {"MOBILEGL_CLEANUP_DELETE_COMMAND": command}), \
mock.patch.object(cleanup, "pause", pauses.append), contextlib.redirect_stdout(output):
code = cleanup.main(["--downloads", str(root / "downloads" / "lane"),
"--artifacts-file", str(root / "artifacts.tsv"),
"--jobs-file", str(root / "jobs.tsv")] + (["--dry-run"] if dry_run else []))
deleted = sorted(entry.name.replace("_", "/") for entry in (root / "deleted").iterdir())
return code, output.getvalue(), deleted, pauses
def test_deletes_are_spaced_under_the_secondary_rate_limit(self):
artifacts = [(str(index), f"{RESULT}shard{index}") for index in range(12)]
code, output, deleted, pauses = self.run_cleanup(artifacts, [], self.passed(artifacts))
self.assertEqual(code, 0)
self.assertEqual(deleted, sorted(f"artifacts/{index}" for index in range(12)))
self.assertIn("Deleted 12 intermediate Android artifact(s)", output)
# Twelve starts are booked at least 11 intervals apart, so the rate stays under 180 a minute.
self.assertLess(60 / cleanup.DELETE_INTERVAL_SECONDS, 180)
self.assertGreater(len(pauses), 0)
def test_refused_deletes_warn_and_leave_the_run_green(self):
artifacts = [(str(index), f"{RESULT}shard{index}") for index in range(8)]
code, output, _, _ = self.run_cleanup(artifacts, [], self.passed(artifacts), exit_code=1)
self.assertEqual(code, 0)
self.assertIn("::warning::Could not delete intermediate artifact", output)
self.assertIn("Stopped deleting", output)
self.assertIn("Deleted 0 intermediate", output)
def test_a_dry_run_deletes_nothing(self):
artifacts = [("1", RESULT + "shard1")]
code, output, deleted, _ = self.run_cleanup(artifacts, [], self.passed(artifacts), dry_run=True)
self.assertEqual((code, deleted), (0, []))
self.assertIn(f"Would delete {RESULT}shard1 (1)", output)
if __name__ == "__main__":
unittest.main()
File diff suppressed because it is too large Load Diff
+186 -29
View File
@@ -35,6 +35,7 @@ Usage:
[--env "K=V;K=V"] \ [--env "K=V;K=V"] \
[--require-inproc] \ [--require-inproc] \
[--require-spawn] \ [--require-spawn] \
[--require-monolith] \
[--benchmark] \ [--benchmark] \
[--benchmark-tail-frames N] \ [--benchmark-tail-frames N] \
[--benchmark-finish 0|1] \ [--benchmark-finish 0|1] \
@@ -62,6 +63,9 @@ copies benchmark.json (per-frame times plus mean/median/p95) out of the app, and
"passed" only means the replay reached the end of the trace without an error. "passed" only means the replay reached the end of the trace without an error.
Pass --reuse-fixture to skip re-extracting and re-pushing the trace, for repeat Pass --reuse-fixture to skip re-extracting and re-pushing the trace, for repeat
runs of a case whose fixture is already in /data/local/tmp. runs of a case whose fixture is already in /data/local/tmp.
Set MOBILEGL_TRACE_POLL_INTERVAL to the seconds between result polls (for example
0.5) to wait on a wall clock: the wait then ends --timeout-seconds plus a fifth after
the app starts, which is about how long that many one-second polls take.
Pass --env "K=V;K=V" (or set MOBILEGL_TRACE_ENV) to hand arbitrary environment Pass --env "K=V;K=V" (or set MOBILEGL_TRACE_ENV) to hand arbitrary environment
variables to the replay process. They are applied last, immediately before variables to the replay process. They are applied last, immediately before
libMobileGL.so is loaded, so they override every flag above; an entry with no "=" libMobileGL.so is loaded, so they override every flag above; an entry with no "="
@@ -72,6 +76,11 @@ sentence a monolith fallback can never write: the pid of the server process.
Pass --require-inproc to require the library's real transport-resolution log, Pass --require-inproc to require the library's real transport-resolution log,
strict errors and separate role state, plus zero Fatal records. This checks the strict errors and separate role state, plus zero Fatal records. This checks the
APK process after replay; a host environment echo or an SSIM-only pass is not proof. APK process after replay; a host environment echo or an SSIM-only pass is not proof.
Pass --require-monolith to fail a run whose library log shows a split transport:
either transport sentence, an IPC configuration line, a spawned server or a log of
a server role. A log with the configuration load and none of these is reported as
monolith confirmed; a run whose log is missing or lacks the load passes and is
reported as not confirmed.
EOF EOF
} }
@@ -93,6 +102,21 @@ adb_device_path() {
MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*' "${ADB}" "$@" MSYS_NO_PATHCONV=1 MSYS2_ARG_CONV_EXCL='*' "${ADB}" "$@"
} }
# Milliseconds since the epoch; whole seconds where date has no %N.
now_ms() {
stamp="$(date +%s%3N 2>/dev/null || true)"
case "${stamp}" in
''|*[!0-9]*) stamp="$(($(date +%s) * 1000))" ;;
esac
printf '%s' "${stamp}"
}
# Appends "<phase> <ms>" to phase-timings.txt in the result directory, so a run records where
# its wall time went: push, install, replay, log pulls.
mark_phase() {
printf '%s %s\n' "$1" "$(now_ms)" >> "${result_dir}/phase-timings.txt" 2>/dev/null || true
}
next_arg() { next_arg() {
if [ "$#" -lt 2 ]; then if [ "$#" -lt 2 ]; then
die "$1 requires a value" die "$1 requires a value"
@@ -134,6 +158,7 @@ texture_2d_dumps=""
env_overrides="${MOBILEGL_TRACE_ENV:-}" env_overrides="${MOBILEGL_TRACE_ENV:-}"
require_inproc=0 require_inproc=0
require_spawn=0 require_spawn=0
require_monolith=0
benchmark=0 benchmark=0
benchmark_tail_frames=200 benchmark_tail_frames=200
benchmark_finish=1 benchmark_finish=1
@@ -182,6 +207,7 @@ while [ "$#" -gt 0 ]; do
--env) env_overrides="$(next_arg "$@")"; shift 2 ;; --env) env_overrides="$(next_arg "$@")"; shift 2 ;;
--require-inproc) require_inproc=1; shift 1 ;; --require-inproc) require_inproc=1; shift 1 ;;
--require-spawn) require_spawn=1; shift 1 ;; --require-spawn) require_spawn=1; shift 1 ;;
--require-monolith) require_monolith=1; shift 1 ;;
--benchmark) benchmark=1; shift 1 ;; --benchmark) benchmark=1; shift 1 ;;
--benchmark-tail-frames) benchmark_tail_frames="$(next_arg "$@")"; shift 2 ;; --benchmark-tail-frames) benchmark_tail_frames="$(next_arg "$@")"; shift 2 ;;
--benchmark-finish) benchmark_finish="$(next_arg "$@")"; shift 2 ;; --benchmark-finish) benchmark_finish="$(next_arg "$@")"; shift 2 ;;
@@ -353,7 +379,16 @@ collect_role_logs() {
rm -f "${role_dst}" rm -f "${role_dst}"
fi fi
done done
[ "${wrote_any}" -eq 1 ] || rm -f "${dest}" if [ "${wrote_any}" -eq 1 ]; then
return 0
fi
# A library built without the split transports has one role and writes one log, at the base
# path itself. Pulled only when no role log arrived, and judged by content as above, so a
# replay of such a build leaves its library log in <dest> whether it passed or failed.
adb_device_path exec-out run-as "${package_name}" cat "${base_source}" > "${dest}" 2>/dev/null || true
if [ ! -s "${dest}" ] || head -1 "${dest}" | grep -q '^cat: .*: No such file'; then
rm -f "${dest}"
fi
} }
copy_texture_2d_dumps() { copy_texture_2d_dumps() {
@@ -411,6 +446,81 @@ copy_fixture_to_app() {
fi fi
} }
# Waits for result.json with one poll per second; --timeout-seconds counts the polls.
wait_for_result_by_count() {
# One failed pidof is not an exit: under memory pressure the adb shell call
# itself can fail while the app is alive (seen with the process still in
# pidof and the activity still initializing right after the "exit").
missed_app_polls=0
for _ in $(seq 1 "${timeout_seconds}"); do
if adb_device_path shell run-as "${package_name}" ls "${app_dir}/output/result.json" >/dev/null 2>&1; then
break
fi
if adb_device_path shell pidof "${package_name}" >/dev/null 2>&1; then
saw_app_process=1
missed_app_polls=0
else
poll_adb_state="$(adb_device_path get-state 2>/dev/null | tr -d '\r' || true)"
if [ "${poll_adb_state}" != "device" ]; then
printf '%s\n' "${poll_adb_state:-unknown}" > "${result_dir}/adb-disconnected.txt"
break
fi
if [ "${saw_app_process}" -eq 1 ]; then
missed_app_polls=$((missed_app_polls + 1))
if [ "${missed_app_polls}" -ge 3 ]; then
app_exited=1
break
fi
fi
fi
sleep 1
done
}
# The same wait on a wall clock, selected by MOBILEGL_TRACE_POLL_INTERVAL (seconds between
# polls, fractions allowed). One adb call per poll asks for result.json and for the app
# process. The budget is --timeout-seconds of wall time plus a fifth: a poll of the
# one-second loop takes about 1.2 s with its two adb calls, so a replay gets the same time
# to finish under either loop. An exit needs three missed polls spread over at least three
# seconds, so a short interval keeps the one-second loop's tolerance for adb calls that fail
# under memory pressure.
wait_for_result_by_clock() {
poll_deadline=$((poll_started + timeout_seconds + timeout_seconds / 5))
missed_app_polls=0
first_missed_at=0
while [ "$(date +%s)" -lt "${poll_deadline}" ]; do
poll_output="$(adb_device_path shell "run-as ${package_name} ls ${app_dir}/output/result.json >/dev/null 2>&1 && echo mobilegl-poll-result; pidof ${package_name} >/dev/null 2>&1 && echo mobilegl-poll-alive; echo mobilegl-poll-end" 2>/dev/null | tr -d '\r' || true)"
case "${poll_output}" in
*mobilegl-poll-result*)
break
;;
*mobilegl-poll-alive*)
saw_app_process=1
missed_app_polls=0
;;
*)
poll_adb_state="$(adb_device_path get-state 2>/dev/null | tr -d '\r' || true)"
if [ "${poll_adb_state}" != "device" ]; then
printf '%s\n' "${poll_adb_state:-unknown}" > "${result_dir}/adb-disconnected.txt"
break
fi
if [ "${saw_app_process}" -eq 1 ]; then
poll_now="$(date +%s)"
if [ "${missed_app_polls}" -eq 0 ]; then
first_missed_at="${poll_now}"
fi
missed_app_polls=$((missed_app_polls + 1))
if [ "${missed_app_polls}" -ge 3 ] && [ "$((poll_now - first_missed_at))" -ge 3 ]; then
app_exited=1
break
fi
fi
;;
esac
sleep "${MOBILEGL_TRACE_POLL_INTERVAL}"
done
}
run_retrace() { run_retrace() {
result_dir="${result_root}/${safe_case}-${backend}" result_dir="${result_root}/${safe_case}-${backend}"
alternate_golden_app_path="" alternate_golden_app_path=""
@@ -427,13 +537,17 @@ run_retrace() {
fi fi
mkdir -p "${result_dir}" mkdir -p "${result_dir}"
printf 'script-started %s\n' "${script_started_ms}" > "${result_dir}/phase-timings.txt" 2>/dev/null || true
mark_phase fixture-pushed
# A repeat run of a case the previous invocation already installed and pushed only needs # A repeat run of a case the previous invocation already installed and pushed only needs
# the on-device copy back into a fresh app output directory. # the on-device copy back into a fresh app output directory.
# MOBILEGL_TRACE_SKIP_INSTALL=1: the caller installed the APK itself (ColorOS shows an install-confirmation # MOBILEGL_TRACE_SKIP_INSTALL=1: the caller installed the APK itself (ColorOS shows an install-confirmation
# dialog whose tap drops adb for a few seconds, so the install has to be done and settled outside a run). # dialog whose tap drops adb for a few seconds, so the install has to be done and settled outside a run;
# trace-replay-shard.sh installs once per emulator boot).
if [ "${reuse_fixture}" -eq 0 ] && [ "${MOBILEGL_TRACE_SKIP_INSTALL:-0}" != "1" ]; then if [ "${reuse_fixture}" -eq 0 ] && [ "${MOBILEGL_TRACE_SKIP_INSTALL:-0}" != "1" ]; then
"${ADB}" install -r "$(host_path_for_adb "${apk_file}")" "${ADB}" install -r "$(host_path_for_adb "${apk_file}")"
fi fi
mark_phase apk-installed
copy_fixture_to_app copy_fixture_to_app
adb_device_path shell am force-stop "${package_name}" adb_device_path shell am force-stop "${package_name}"
"${ADB}" logcat -c "${ADB}" logcat -c
@@ -502,39 +616,20 @@ run_retrace() {
--ei crop_width "${crop_width}" \ --ei crop_width "${crop_width}" \
--ei crop_height "${crop_height}" --ei crop_height "${crop_height}"
adb_device_path shell "$@" adb_device_path shell "$@"
mark_phase app-started
app_exited=0 app_exited=0
saw_app_process=0 saw_app_process=0
# One failed pidof is not an exit: under memory pressure the adb shell call
# itself can fail while the app is alive (seen with the process still in
# pidof and the activity still initializing right after the "exit").
missed_app_polls=0
poll_started="$(date +%s)" poll_started="$(date +%s)"
for _ in $(seq 1 "${timeout_seconds}"); do if [ -n "${MOBILEGL_TRACE_POLL_INTERVAL:-}" ]; then
if adb_device_path shell run-as "${package_name}" ls "${app_dir}/output/result.json" >/dev/null 2>&1; then wait_for_result_by_clock
break else
fi wait_for_result_by_count
if adb_device_path shell pidof "${package_name}" >/dev/null 2>&1; then fi
saw_app_process=1 mark_phase wait-ended
missed_app_polls=0
else
poll_adb_state="$(adb_device_path get-state 2>/dev/null | tr -d '\r' || true)"
if [ "${poll_adb_state}" != "device" ]; then
printf '%s\n' "${poll_adb_state:-unknown}" > "${result_dir}/adb-disconnected.txt"
break
fi
if [ "${saw_app_process}" -eq 1 ]; then
missed_app_polls=$((missed_app_polls + 1))
if [ "${missed_app_polls}" -ge 3 ]; then
app_exited=1
break
fi
fi
fi
sleep 1
done
collect_run_diagnostics "${result_dir}" collect_run_diagnostics "${result_dir}"
mark_phase diagnostics-collected
if ! adb_device_path shell run-as "${package_name}" ls "${app_dir}/output/result.json" >/dev/null 2>&1; then if ! adb_device_path shell run-as "${package_name}" ls "${app_dir}/output/result.json" >/dev/null 2>&1; then
if [ -f "${result_dir}/adb-disconnected.txt" ]; then if [ -f "${result_dir}/adb-disconnected.txt" ]; then
echo "trace-replay-ci.sh: device disconnected while waiting for result.json" >&2 echo "trace-replay-ci.sh: device disconnected while waiting for result.json" >&2
@@ -574,6 +669,7 @@ run_retrace() {
copy_app_artifact "${app_dir}/output/benchmark.json" "${result_dir}/benchmark.json" copy_app_artifact "${app_dir}/output/benchmark.json" "${result_dir}/benchmark.json"
fi fi
copy_texture_2d_dumps copy_texture_2d_dumps
mark_phase artifacts-copied
# A replay that wrote result.json but did not pass used to print nothing but # A replay that wrote result.json but did not pass used to print nothing but
# the JSON, which for a non-zero statusCode says only "retrace failed with # the JSON, which for a non-zero statusCode says only "retrace failed with
@@ -692,10 +788,71 @@ if not passed:
sys.exit("trace replay failed actual spawn/second-process/zero-Fatal proof: " + json.dumps(proof)) sys.exit("trace replay failed actual spawn/second-process/zero-Fatal proof: " + json.dumps(proof))
print("Android retrace: real spawn transport, server role in pid " print("Android retrace: real spawn transport, server role in pid "
f"{proof['server_pid']}, and zero Fatal records confirmed") f"{proof['server_pid']}, and zero Fatal records confirmed")
PY
fi
if [ "${require_monolith}" -eq 1 ]; then
"${PYTHON}" - "$(host_path_for_adb "${result_dir}/mobilegl.log")" "$(host_path_for_adb "${result_dir}/transport-proof.json")" <<'PY'
import json
from pathlib import Path
import re
import sys
log_path, proof_path = map(Path, sys.argv[1:])
text = log_path.read_text(encoding="utf-8", errors="replace") if log_path.is_file() else ""
# A monolith run has no sentence of its own: ConfigLoader logs a transport line only when it
# resolves inproc or spawn, and a library built without the split transports has no such
# code. So the check fails only on what a split run writes: either transport sentence, an IPC
# configuration line (logged for every transport but monolith), a spawned server, or library
# lines in a server role's log. The sentence and the IPC line are written during the
# configuration load, so a log that shows the load and none of these confirms monolith. A log
# that is missing or lacks the load says nothing about the transport: the run passes and is
# reported as not confirmed.
loaded = "Config loaded" in text
markers = {transport: f"Config: MOBILEGL_TRANSPORT={transport} - the MGPipe record stream" in text
for transport in ("inproc", "spawn")}
configs = re.findall(r"Config: IPC[^\r\n]*", text)
armed = re.search(r"spawn ARMED - the server role runs in pid (\d+)", text)
server_log = log_path.with_name(log_path.stem + ".server" + log_path.suffix)
server_text = server_log.read_text(encoding="utf-8", errors="replace") if server_log.is_file() else ""
# Library lines only, known by the "[...thread/LEVEL]: " header each one starts with, so that
# what a device answers when the file is not there is never read as a server role.
server_lines = len(re.findall(r"^\[[^\r\n]*?/(?:DEBUG|INFO|WARN|ERROR|FATAL)\]: ", server_text, re.M))
problems = [f"the library log has the {transport} transport marker"
for transport, present in markers.items() if present]
if configs:
problems.append("the library log has an IPC configuration line")
if armed:
problems.append("the library log names a spawned server process")
if server_lines:
problems.append("a server role wrote a log of its own")
if loaded:
unconfirmed = ""
elif text.strip():
unconfirmed = "the library log does not show the configuration load"
else:
unconfirmed = "the library log is missing or empty"
proof = {"required_transport": "monolith", "library_log": str(log_path),
"configuration_loaded": loaded, "split_transport_markers": markers,
"ipc_config_lines": configs, "spawned_server_pid": int(armed.group(1)) if armed else None,
"server_role_log_lines": server_lines, "problems": problems, "passed": not problems,
"confirmed": not problems and loaded, "not_confirmed_because": unconfirmed}
proof_path.write_text(json.dumps(proof, indent=2), encoding="utf-8")
if problems:
sys.exit("trace replay failed actual monolith/no-split-transport proof (" + "; ".join(problems) + "): "
+ json.dumps(proof))
if unconfirmed:
print(f"Android retrace: monolith not confirmed ({unconfirmed}): nothing a split transport writes "
"was found, so the run is not failed")
else:
print("Android retrace: monolith confirmed: the library loaded its configuration and logged no inproc "
"or spawn transport, no IPC configuration, no spawned server and no server role")
PY PY
fi fi
} }
script_started_ms="$(now_ms)"
mkdir -p "${fixture_root}" "${result_root}" mkdir -p "${fixture_root}" "${result_root}"
if [ "${reuse_fixture}" -eq 0 ]; then if [ "${reuse_fixture}" -eq 0 ]; then
prepare_fixture prepare_fixture
+28
View File
@@ -0,0 +1,28 @@
#!/bin/sh
# MobileGL - android-plugin/trace-replay-shard.sh
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
# Replays an ordered list of trace cases for one (lane, transport, backend) on one emulator
# boot, each through trace-replay-ci.sh. The loop, the per-case settings and the verdict files
# are in trace_replay_shard.py; `--help` lists the arguments.
#
# sh android-plugin/trace-replay-shard.sh \
# --apk-file FILE --backend DirectGLES|DirectVulkan --gpu GPU \
# --transport monolith|inproc|spawn --lane LANE \
# --cases-env VARIABLE | --cases-file FILE | --cases-json JSON \
# [--deadline-seconds N] [--dry-run | --finalize]
#
# Environment:
# AVD_NAME, EMULATOR_LOG, EMULATOR_PID_FILE the emulator to restart when it stops answering
# MOBILEGL_GPU_HANG_BUDGET_MS forwarded to every replay (required)
# MOBILEGL_SHARD_RESTART_EMULATOR_EACH_CASE 1 cold boots the emulator before every case
# MOBILEGL_SHARD_POLL_INTERVAL seconds between result polls (default 0.5);
# empty selects trace-replay-ci.sh's 1 s loop
set -eu
exec "${PYTHON:-python3}" "$(dirname "$0")/trace_replay_shard.py" "$@"
+871
View File
@@ -0,0 +1,871 @@
#!/usr/bin/env python3
# MobileGL - android-plugin/trace_replay_shard.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""Replay an ordered list of trace cases on one emulator boot.
One shard is one (lane, transport, backend). Each case runs through trace-replay-ci.sh with
the per-case and per-transport settings of the APK workflow, in a fresh app process with
cleared app data. A failing case does not stop the shard: the case after it starts on a cold
boot, every case gets a verdict file, and the exit status is non-zero when a case failed or a
verdict is missing.
Layout under --result-root (the directory the workflow uploads):
<case>-<backend>/ what trace-replay-ci.sh writes, plus lane.json, the golden
copies the summary renders, and case-status.json
<case>-<backend>/first-attempt/ files of an attempt that was retried
shard-status.json every case of the shard with its verdict
diagnostics/ emulator logs kept across restarts
The job log repeats what those files say, since the workflow deletes them after a green run:
what each case is run with, the transport lines of its library log, its phase timings and its
record, and shard-status.json at the end.
"""
import argparse
import json
import os
import re
import shlex
import shutil
import signal
import subprocess
import sys
import time
from pathlib import Path
INFRASTRUCTURE_FAILURE_EXIT_CODE = 75
TIMEOUT_EXIT_CODE = 124
BACKENDS = ("DirectGLES", "DirectVulkan")
TRANSPORTS = ("monolith", "inproc", "spawn")
PASSED, FAILED, NOT_RUN = "passed", "failed", "not-run"
# Cases with settings of their own. Bliss needs the newer packaged ANGLE build and, on
# DirectGLES, sampler mipmap min-filter downgrading on ANGLE llvmpipe; iterationRP needs its
# SPIR-V repairs forced on.
BLISS_CASE = "minecraft-1.21.4-fabric-iris-bliss-in-world"
ITERATIONRP_CASE = "minecraft-1.21.4-fabric-iris-iterationrp-in-world"
DEFAULT_ANGLE_VARIANT = "ec889e6ea831"
BLISS_ANGLE_VARIANT = "90a62123d794"
# The inproc and spawn lanes pass the same client settings, so they differ only in which
# process runs the server role.
SPLIT_CLIENT_SETTINGS = "MOBILEGL_IPC_ROLE_SPLIT_STATE=1;MOBILEGL_IPC_STRICT_ERRORS=1;MOBILEGL_IPC_RUN_AHEAD=1"
REQUIRED_CASE_KEYS = ("name", "trace_archive", "trace_file", "golden", "target_call", "width", "height",
"crop_x", "crop_y", "crop_width", "crop_height", "timeout_seconds")
# Seconds trace-replay-ci.sh gets beyond the case's own budget for install, push and log pulls.
REPLAY_GRACE_SECONDS = 300
BOOT_TIMEOUT_SECONDS = 300
# Seconds a case may take around its replays: the app reset, the failure diagnostics and the
# cold boot that follows a case that did not pass.
CASE_OVERHEAD_SECONDS = 180
# What marks the lines of a library log that say which transport ran: the configuration load
# every run logs, and what the --require-* checks of trace-replay-ci.sh read. The job log
# quotes at most LIBRARY_LOG_LINE_LIMIT of them per case, cut to LIBRARY_LOG_LINE_LENGTH.
LIBRARY_LOG_EVIDENCE = ("Config: Active backend type set to", "Config loaded", "MOBILEGL_TRANSPORT",
"Config: IPC", "spawn ARMED", "Fatal{")
LIBRARY_LOG_LINE_LIMIT = 12
LIBRARY_LOG_LINE_LENGTH = 1000
# What trace-replay-ci.sh reads from its environment and the runner does not set. A replay
# inherits the runner's environment, so the job log names the value each one has there.
INHERITED_REPLAY_SETTINGS = ("ADB", "PYTHON", "MOBILEGL_RETRACE_USE_PBUFFER", "MOBILEGL_TRACE_ENV")
def say(message):
print(message, flush=True)
def pause(seconds):
time.sleep(seconds)
def safe_case(name):
return re.sub(r"[^A-Za-z0-9._-]", "_", name)
def as_text(value):
if value is True:
return "true"
if value is False:
return "false"
return "" if value is None else str(value)
def is_true(value):
return value is True or value == "true"
def load_cases(text, backend):
"""The case objects of the APK matrix, as text ready for the command line."""
try:
# Numbers stay the literal text of the matrix, so 0.985 is passed on as 0.985.
cases = json.loads(text, parse_float=str, parse_int=str)
except json.JSONDecodeError as error:
raise ValueError(f"the case list is not JSON: {error}") from error
if not isinstance(cases, list) or not cases:
raise ValueError("the case list must be a non-empty JSON array of case objects")
names = set()
for index, case in enumerate(cases):
if not isinstance(case, dict):
raise ValueError(f"case {index} is not an object; pass the case objects of the APK matrix")
missing = [key for key in REQUIRED_CASE_KEYS if case.get(key) in (None, "")]
if missing:
raise ValueError(f"case {case.get('name', index)} lacks {', '.join(missing)}")
name = case["name"]
if name in names:
raise ValueError(f"case {name} is listed twice")
names.add(name)
backends = case.get("ci_backends")
if backends is not None and backend not in backends:
raise ValueError(f"case {name} does not run on {backend} (ci_backends: {backends})")
return cases
def needed_seconds(cases):
"""Wall time in which every case of a shard can use its whole time budget.
It covers each case running until its replay script is stopped, one infrastructure retry
that does so again for the longest budget, and CASE_OVERHEAD_SECONDS per case. A shard
deadline at or above this never stops a case that is within its own budget, unless more
than one case needs a retry as long as its first attempt.
"""
budgets = [int(as_text(case["timeout_seconds"])) + REPLAY_GRACE_SECONDS for case in cases]
return sum(budgets) + max(budgets) + CASE_OVERHEAD_SECONDS * len(budgets)
def replay_environment(backend, case_name):
return {
"MOBILEGL_ESPRYT_USE_ANGLE": "1" if backend == "DirectGLES" else "0",
"MOBILEGL_TRACE_ANGLE_VARIANT": BLISS_ANGLE_VARIANT if case_name == BLISS_CASE else DEFAULT_ANGLE_VARIANT,
"MOBILEGL_MAGMA_R11G11B10F_FALLBACK": "1" if backend == "DirectVulkan" else "0",
"MOBILEGL_ITRP_FIX": "ForceOn" if case_name == ITERATIONRP_CASE else "Auto",
}
def transport_arguments(transport, hang_budget_ms):
hang_budget = f"MOBILEGL_GPU_HANG_BUDGET_MS={hang_budget_ms}"
if transport == "inproc":
return ["--env", f"MOBILEGL_TRANSPORT=inproc;{SPLIT_CLIENT_SETTINGS};{hang_budget}", "--require-inproc"]
if transport == "spawn":
# No MOBILEGL_IPC_SERVER_PATH: only the app can spell nativeLibraryDir, and
# TraceReplayActivity resolves it when the environment asks for spawn. The pbuffer is
# required: a spawned server cannot be handed the client's window.
return ["--env", f"MOBILEGL_TRANSPORT=spawn;{SPLIT_CLIENT_SETTINGS};{hang_budget}",
"--use-pbuffer", "--require-spawn"]
return ["--env", f"MOBILEGL_TRANSPORT=monolith;{hang_budget}", "--require-monolith"]
def replay_arguments(options, case, hang_budget_ms):
name = case["name"]
arguments = [
"--apk-file", options.apk_file,
"--package", options.package,
"--backend", options.backend,
"--result-root", options.result_root,
"--fixture-root", options.fixture_root,
"--case", name,
"--trace-archive", as_text(case["trace_archive"]),
"--trace-file", as_text(case["trace_file"]),
"--golden", as_text(case["golden"]),
"--alternate-golden", as_text(case.get("alternate_golden") or ""),
"--target-call", as_text(case["target_call"]),
"--width", as_text(case["width"]),
"--height", as_text(case["height"]),
"--ssim-threshold", as_text(case.get("ssim_threshold") or "0.99"),
"--crop-x", as_text(case["crop_x"]),
"--crop-y", as_text(case["crop_y"]),
"--crop-width", as_text(case["crop_width"]),
"--crop-height", as_text(case["crop_height"]),
"--timeout-seconds", as_text(case["timeout_seconds"]),
]
arguments += transport_arguments(options.transport, hang_budget_ms)
if options.backend == "DirectGLES" and name == BLISS_CASE:
arguments.append("--avoid-angle-llvmpipe-sampler-mipmap-min-filter")
if options.backend == "DirectGLES" and is_true(case.get("avoid_angle_llvmpipe_explicit_lod_bias")):
arguments.append("--avoid-angle-llvmpipe-explicit-lod-bias")
if is_true(case.get("coherent_as_flush")):
arguments.append("--coherent-as-flush")
return arguments
def read_json(path):
try:
return json.loads(Path(path).read_text(encoding="utf-8"))
except (OSError, ValueError):
return None
def write_json(path, value):
Path(path).write_text(json.dumps(value, indent=2) + "\n", encoding="utf-8")
def one_line(text):
return " ".join(str(text).split())
class Shard:
def __init__(self, options, cases, environ):
self.options = options
self.cases = cases
self.environ = dict(environ)
self.adb_command = shlex.split(environ.get("ADB", "adb"))
directory = Path(__file__).resolve().parent
self.replay_command = shlex.split(environ.get("MOBILEGL_SHARD_REPLAY_COMMAND") or
f"sh {shlex.quote(str(directory / 'trace-replay-ci.sh'))}")
self.avd_command = shlex.split(environ.get("MOBILEGL_SHARD_AVD_COMMAND") or
f"sh {shlex.quote(str(directory / 'run-avd-ci.sh'))}")
self.hang_budget_ms = environ.get("MOBILEGL_GPU_HANG_BUDGET_MS")
self.restart_each_case = environ.get("MOBILEGL_SHARD_RESTART_EMULATOR_EACH_CASE", "0") == "1"
self.poll_interval = environ.get("MOBILEGL_SHARD_POLL_INTERVAL", "0.5")
self.avd_name = environ.get("AVD_NAME", "")
self.emulator_log = environ.get("EMULATOR_LOG", "")
self.emulator_pid_file = environ.get("EMULATOR_PID_FILE", "")
self.result_root = Path(options.result_root)
self.diagnostics = self.result_root / "diagnostics"
self.started = time.monotonic()
self.boots = 1
self.install_seconds = []
self.records = {}
self.aborted = ""
# The case before the current one when it did not pass; the next case gets a cold boot.
self.unclean_case = ""
# ---- bookkeeping -------------------------------------------------------------------
def result_dir(self, case):
return self.result_root / f"{safe_case(case['name'])}-{self.options.backend}"
def record_for(self, case, status, reason, **extra):
record = {"case": case["name"], "backend": self.options.backend, "transport": self.options.transport,
"lane": self.options.lane, "status": status, "reason": reason}
record.update(extra)
return record
def write_summary_inputs(self, case):
"""lane.json and the golden copies the retrace summary reads next to each result."""
directory = self.result_dir(case)
directory.mkdir(parents=True, exist_ok=True)
write_json(directory / "lane.json", {"lane": self.options.lane, "transport": self.options.transport})
stem = f"{safe_case(case['name'])}-{self.options.backend}"
for key, suffix in (("golden", "golden"), ("alternate_golden", "alternate-golden")):
source = as_text(case.get(key) or "")
if source and os.path.isfile(source) and os.path.getsize(source) > 0:
shutil.copyfile(source, directory / f"{stem}-{suffix}.png")
def save_record(self, case, record):
self.records[case["name"]] = record
self.write_summary_inputs(case)
write_json(self.result_dir(case) / "case-status.json", record)
self.write_shard_status()
def plan(self, case):
"""What a replay of the case is run with. --dry-run prints it; a run prints it and uses it."""
return {"case": case["name"],
"environment": replay_environment(self.options.backend, case["name"]),
"arguments": replay_arguments(self.options, case, self.hang_budget_ms),
"timeout": int(as_text(case["timeout_seconds"])) + REPLAY_GRACE_SECONDS}
def shared_environment(self):
"""What every replay of the shard gets on top of its case's environment."""
# The shard installs the APK once per boot.
return {"MOBILEGL_TRACE_SKIP_INSTALL": "1", "MOBILEGL_TRACE_POLL_INTERVAL": self.poll_interval}
def say_shard_status(self):
status = read_json(self.result_root / "shard-status.json")
say(f"trace-replay-shard: shard-status: {json.dumps(status, sort_keys=True)}")
def write_shard_status(self, complete=False):
self.result_root.mkdir(parents=True, exist_ok=True)
records = [self.records.get(case["name"]) or
self.record_for(case, NOT_RUN, "the shard has not reached this case") for case in self.cases]
write_json(self.result_root / "shard-status.json", {
"schema": 1, "lane": self.options.lane, "transport": self.options.transport,
"backend": self.options.backend, "complete": complete,
# The Actions job that ran the shard, so the artifact cleanup can tell whose job failed.
"job_id": self.environ.get("MOBILEGL_SHARD_JOB_ID", ""),
"expected_cases": [case["name"] for case in self.cases], "cases": records,
"emulator_boots": self.boots, "install_seconds": self.install_seconds})
def remaining_seconds(self):
if self.options.deadline_seconds <= 0:
return None
return self.options.deadline_seconds - (time.monotonic() - self.started)
def deadline_passed(self):
remaining = self.remaining_seconds()
return remaining is not None and remaining <= 0
# ---- processes ---------------------------------------------------------------------
def run(self, command, timeout=None, capture=False, extra_env=None):
"""Returns (exit status, output). A timeout kills the whole process group and reads as 124."""
environment = dict(self.environ)
environment.update(extra_env or {})
own_group = os.name == "posix"
try:
process = subprocess.Popen(command, env=environment, start_new_session=own_group,
stdout=subprocess.PIPE if capture else None,
stderr=subprocess.STDOUT if capture else None)
except OSError as error:
return 127, str(error)
def kill():
if own_group:
try:
os.killpg(process.pid, signal.SIGKILL)
except OSError:
pass
process.kill()
try:
output, _ = process.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
kill()
output, _ = process.communicate()
return TIMEOUT_EXIT_CODE, (output or b"").decode("utf-8", "replace")
except BaseException:
# The job was canceled or timed out. The child is in a session of its own, so it
# would otherwise keep driving the device behind the steps that still run.
kill()
raise
return process.returncode, (output or b"").decode("utf-8", "replace")
def adb(self, *arguments, timeout=60):
status, output = self.run(self.adb_command + list(arguments), timeout=timeout, capture=True)
return status, output.replace("\r", "")
# ---- emulator ----------------------------------------------------------------------
def can_restart(self):
return bool(self.avd_name and self.options.gpu and self.emulator_log and self.emulator_pid_file)
def emulator_pid(self):
try:
return int(Path(self.emulator_pid_file).read_text().strip())
except (OSError, ValueError):
return None
def emulator_process_alive(self):
pid = self.emulator_pid()
if pid is None:
return True # nothing to check against, for example a real device
try:
os.kill(pid, 0)
except OSError:
return False
return True
def health_problem(self):
"""Empty when the device answers; otherwise what did not."""
if not self.emulator_process_alive():
return "the emulator process is gone"
status, output = self.adb("get-state", timeout=15)
if status != 0 or output.strip() != "device":
return f"adb state is {output.strip() or 'unknown'}"
status, output = self.adb("shell", "getprop", "sys.boot_completed", timeout=15)
if status != 0 or output.strip() != "1":
return "sys.boot_completed is not 1"
# A framework call: boot_completed stays 1 across a system_server restart.
status, output = self.adb("shell", "pm", "path", self.options.package, timeout=30)
if status != 0 or "package:" not in output:
return "the package manager does not list the trace APK"
return ""
def keep_emulator_log(self, label):
if self.emulator_log and os.path.isfile(self.emulator_log):
self.diagnostics.mkdir(parents=True, exist_ok=True)
shutil.copyfile(self.emulator_log, self.diagnostics / f"emulator-boot-{self.boots}-{label}.log")
def prepare_booted_device(self):
# A shard outlives the screen timeout a single replay never reached.
self.adb("shell", "svc", "power", "stayon", "true", timeout=30)
return self.install()
def install(self):
started = time.monotonic()
status, output = self.adb("install", "-r", self.options.apk_file, timeout=300)
self.install_seconds.append(round(time.monotonic() - started, 1))
say(output.strip())
return status == 0
def restart_emulator(self, reason):
"""Stop, cold boot and reinstall. False when the emulator did not come back."""
if not self.can_restart():
say(f"trace-replay-shard: cannot restart the emulator ({reason}): no AVD is configured")
return False
say(f"trace-replay-shard: restarting the emulator: {reason}")
common = ["--avd-name", self.avd_name, "--emulator-log", self.emulator_log,
"--pid-file", self.emulator_pid_file]
for attempt in (1, 2):
# The start truncates the log, and the boot that failed is the one worth reading.
self.keep_emulator_log(f"attempt-{attempt}" if attempt > 1 else "ended")
old_pid = self.emulator_pid()
self.run(self.avd_command + ["stop"] + common, timeout=120)
self.wait_for_exit(old_pid)
self.adb("kill-server", timeout=30)
pause(2)
self.boots += 1
status, _ = self.run(self.avd_command + ["start"] + common +
["--gpu", self.options.gpu, "--boot-timeout", str(BOOT_TIMEOUT_SECONDS)],
timeout=2 * BOOT_TIMEOUT_SECONDS + 60)
if status == 0 and self.prepare_booted_device():
return True
say(f"trace-replay-shard: emulator restart attempt {attempt} failed")
return False
def wait_for_exit(self, pid):
if pid is None or os.name != "posix":
return
for _ in range(60):
try:
os.kill(pid, 0)
except OSError:
return
pause(0.5)
try:
os.kill(pid, signal.SIGKILL)
except OSError:
pass
# ---- app state ---------------------------------------------------------------------
def leftover_processes(self):
"""Processes still running under the app's uid, as 'pid name' strings."""
status, user = self.adb("shell", "run-as", self.options.package, "id", "-un", timeout=30)
user = user.strip()
if status != 0 or not re.fullmatch(r"[A-Za-z0-9_]+", user):
return None
status, listing = self.adb("shell", "ps", "-A", "-o", "USER,PID,NAME", timeout=30)
if status != 0:
return None
found = []
for line in listing.splitlines():
fields = line.split()
if len(fields) >= 3 and fields[0] == user and fields[1].isdigit():
found.append(f"{fields[1]} {fields[2]}")
return found
def settled_leftover_processes(self, polls):
"""leftover_processes once the app's uid is empty, or as it still is after `polls` looks.
A killed process stays listed for a moment, so one look right after the kill would
report every force-stop as incomplete.
"""
found = self.leftover_processes()
for _ in range(polls - 1):
if not found:
break
pause(0.5)
found = self.leftover_processes()
return found
def reset_app_state(self):
"""Fresh process and fresh app data, as a new install gives. Returns notes for the record."""
notes = []
package = self.options.package
self.adb("shell", "am", "force-stop", package, timeout=60)
leftovers = self.settled_leftover_processes(polls=4)
if leftovers is None:
notes.append("the app's processes could not be listed after force-stop")
if leftovers:
# A force-stop is expected to take a spawned server with it; say so when it did not.
notes.append("processes left after force-stop: " + ", ".join(leftovers))
self.adb("shell", "run-as", package, "kill", "-9", *[entry.split()[0] for entry in leftovers],
timeout=30)
if self.settled_leftover_processes(polls=6):
notes.append("the leftover processes could not be killed")
if self.restart_emulator("app processes survived force-stop and kill"):
notes.append("the emulator was restarted to remove them")
status, output = self.adb("shell", "pm", "clear", package, timeout=60)
probe_status, _ = self.adb("shell", "run-as", package, "ls", f"/data/user/0/{package}", timeout=30)
if status != 0 or "Success" not in output or probe_status != 0:
# The state a fresh install leaves, reached the other way.
notes.append("pm clear did not leave a usable app data directory; reinstalled the APK")
self.adb("uninstall", package, timeout=120)
self.install()
return notes
def remove_case_leftovers(self, case):
name = safe_case(case["name"])
base = f"/data/local/tmp/mobilegl-{name}"
self.adb("shell", "rm", "-f", f"{base}.trace", f"{base}.golden.png", f"{base}.alternate-golden.png",
timeout=60)
self.adb("shell", "am", "force-stop", self.options.package, timeout=60)
shutil.rmtree(Path(self.options.fixture_root) / name, ignore_errors=True)
# ---- one case ----------------------------------------------------------------------
def reason_lines(self):
try:
return (self.result_root / "infrastructure-failure-reason.txt").read_text().splitlines()
except OSError:
return []
def run_attempt(self, plan):
"""Returns (exit status, seconds, how the shard deadline cut the attempt short or "")."""
case_budget = plan["timeout"]
budget = case_budget
remaining = self.remaining_seconds()
if remaining is not None and remaining < case_budget:
budget = max(remaining, 1.0)
started = time.monotonic()
status, _ = self.run(self.replay_command + plan["arguments"], timeout=budget,
extra_env={**plan["environment"], **self.shared_environment()})
elapsed = round(time.monotonic() - started, 1)
cut = ""
if status == TIMEOUT_EXIT_CODE and budget < case_budget:
# The case never had its own budget, so the timeout says nothing about the case.
cut = f"cut by the shard deadline after {elapsed:.0f} s of a {case_budget} s budget"
return status, elapsed, cut
def keep_first_attempt(self, case):
"""Move the retried attempt's files aside so the retry starts from an empty directory.
result.json is renamed: the summary keys rows on any result.json below a directory
that names the case, and the retry's is the one that counts.
"""
directory = self.result_dir(case)
if not directory.is_dir():
return
kept = directory / "first-attempt"
kept.mkdir(exist_ok=True)
for entry in directory.iterdir():
if entry.is_file():
name = "result.first-attempt.json" if entry.name == "result.json" else entry.name
entry.replace(kept / name)
def verdict(self, case, status):
directory = self.result_dir(case)
if status == 0:
result = read_json(directory / "result.json")
if not isinstance(result, dict) or result.get("passed") is not True:
return FAILED, "the replay exited 0 without a passed result.json"
proof = read_json(directory / "transport-proof.json")
if not (isinstance(proof, dict) and proof.get("passed") is True and
proof.get("required_transport") == self.options.transport):
return FAILED, f"the replay exited 0 without a passed {self.options.transport} transport proof"
return PASSED, ""
if status == INFRASTRUCTURE_FAILURE_EXIT_CODE:
return FAILED, "the emulator infrastructure failed again on the retry"
if status == TIMEOUT_EXIT_CODE:
return FAILED, "the replay script exceeded its time budget"
result = read_json(directory / "result.json")
if not isinstance(result, dict):
return FAILED, f"exit {status} without a result.json"
if result.get("passed") is True:
# The picture matched, so what failed is a check that runs after it.
return FAILED, f"exit {status} after a passed result.json: the transport proof or a log check failed"
return FAILED, f"exit {status}: {one_line(result.get('message', ''))}"
def collect_failure_diagnostics(self, case):
directory = self.result_dir(case)
directory.mkdir(parents=True, exist_ok=True)
logcat = directory / "logcat.txt"
if not logcat.is_file() or logcat.stat().st_size == 0:
# trace-replay-ci.sh pulls this itself unless it was killed first.
_, output = self.adb("logcat", "-d", "-t", "20000", timeout=30)
logcat.write_text(output, encoding="utf-8", errors="replace")
app_output = f"/data/user/0/{self.options.package}/files/trace-replay/output"
# The two role logs of a split build, and the one log of a build without the split
# transports; the device has one kind or the other.
for name in ("retrace.log", "mobilegl.client.log", "mobilegl.server.log", "mobilegl.log"):
status, output = self.adb("exec-out", "run-as", self.options.package, "cat",
f"{app_output}/{name}", timeout=30)
if status == 0 and output and not output.startswith("cat: "):
(directory / name).write_text(output, encoding="utf-8", errors="replace")
if self.emulator_log and os.path.isfile(self.emulator_log):
shutil.copyfile(self.emulator_log, directory / "emulator.log")
# A vanished emulator looks the same whether the host OOM killer took it or the
# renderer faulted. These two say which.
_, memory = self.run(["free", "-h"], timeout=30, capture=True)
(directory / "host-memory.txt").write_text(memory, encoding="utf-8")
_, dmesg = self.run(["sudo", "-n", "dmesg", "-T"], timeout=30, capture=True)
(directory / "host-dmesg.txt").write_text("\n".join(dmesg.splitlines()[-300:]) + "\n", encoding="utf-8")
def say_library_log(self, case):
"""Quote the lines of the case's pulled library log that say which transport ran."""
directory = self.result_dir(case)
label = f"trace-replay-shard: library-log {case['name']}:"
# mobilegl.log is what the proofs of trace-replay-ci.sh read: the client role's log and
# then the server role's, or the one log of a build without the split transports. A
# split build has only its role logs when the runner pulled them itself.
sources = [directory / "mobilegl.log"]
if not sources[0].is_file():
sources = [directory / "mobilegl.client.log", directory / "mobilegl.server.log"]
read = shown = matched = 0
for source in sources:
try:
lines = source.read_text(encoding="utf-8", errors="replace").splitlines()
except OSError:
continue
read += 1
for line in lines:
if any(mark in line for mark in LIBRARY_LOG_EVIDENCE):
matched += 1
if shown < LIBRARY_LOG_LINE_LIMIT:
shown += 1
# Escaped, so that no character of a device log can fail the job log's encoding.
quoted = line[:LIBRARY_LOG_LINE_LENGTH].encode("ascii", "backslashreplace").decode("ascii")
say(f"{label} {source.name}: {quoted}")
if not read:
say(f"{label} no library log was pulled")
elif not matched:
say(f"{label} the library log has none of the lines looked for")
elif matched > shown:
say(f"{label} {matched - shown} more such line(s) are not shown")
def say_phases(self, case):
"""Seconds from each mark of the last attempt's phase-timings.txt to the next, named by the later."""
try:
text = (self.result_dir(case) / "phase-timings.txt").read_text(encoding="utf-8", errors="replace")
except OSError:
return
marks = [line.split() for line in text.splitlines()]
marks = [(mark[0], int(mark[1])) for mark in marks if len(mark) == 2 and mark[1].isdecimal()]
phases = {later[0]: round((later[1] - earlier[1]) / 1000, 1) for earlier, later in zip(marks, marks[1:])}
if phases:
say(f"trace-replay-shard: phases {case['name']}: {json.dumps(phases)}")
def run_case(self, index, case):
name = case["name"]
notes = []
seconds = {}
case_started = time.monotonic()
if self.deadline_passed():
return self.record_for(case, NOT_RUN, "the shard deadline passed before this case started")
plan = self.plan(case)
say(f"trace-replay-shard: run {name}: {json.dumps(plan, sort_keys=True)}")
restarted_for = ""
if self.restart_each_case and index > 0:
restarted_for = "MOBILEGL_SHARD_RESTART_EMULATOR_EACH_CASE=1"
elif self.unclean_case and self.can_restart():
# A crashed, hung or killed replay can leave the guest or the host renderer in a
# state no adb query shows, and this case's verdict must not depend on it.
restarted_for = f"the case before it, {self.unclean_case}, did not pass"
else:
restarted_for = self.health_problem()
if restarted_for:
notes.append(f"emulator restarted before the case: {restarted_for}")
if not self.restart_emulator(restarted_for):
self.aborted = "the emulator could not be restarted"
return self.record_for(case, NOT_RUN, self.aborted, notes=notes)
notes += self.reset_app_state()
seconds["reset"] = round(time.monotonic() - case_started, 1)
reasons_before = len(self.reason_lines())
exit_codes = []
replay_seconds = []
status, elapsed, cut = self.run_attempt(plan)
exit_codes.append(status)
replay_seconds.append(elapsed)
surface_lost = 0
if status == INFRASTRUCTURE_FAILURE_EXIT_CODE:
say(f"::warning::Android emulator infrastructure failed; restarting it and retrying {name} once.")
new_reasons = self.reason_lines()[reasons_before:]
# A lost surface is retried, so it is counted where a rate spike shows.
surface_lost = sum(1 for line in new_reasons if "angle-surface-lost" in line)
if surface_lost:
self.append_step_summary(f"surface-lost retries: {surface_lost} ({self.options.backend}, {name})\n")
notes.append("infrastructure retry: " + (", ".join(new_reasons) or "no reason recorded"))
self.keep_first_attempt(case)
if not self.restart_emulator(f"infrastructure failure in {name}"):
self.aborted = "the emulator could not be restarted"
elif self.deadline_passed():
cut = "the shard deadline passed before the infrastructure retry could start"
else:
notes += self.reset_app_state()
status, elapsed, cut = self.run_attempt(plan)
exit_codes.append(status)
replay_seconds.append(elapsed)
seconds["replay"] = replay_seconds
finish_started = time.monotonic()
if self.aborted:
verdict, reason = FAILED, "the emulator did not come back for the infrastructure retry"
elif cut:
verdict, reason = NOT_RUN, cut
else:
verdict, reason = self.verdict(case, status)
if verdict != PASSED:
self.collect_failure_diagnostics(case)
self.say_library_log(case)
self.say_phases(case)
if not self.aborted:
self.remove_case_leftovers(case)
seconds["finish"] = round(time.monotonic() - finish_started, 1)
seconds["total"] = round(time.monotonic() - case_started, 1)
return self.record_for(case, verdict, reason, attempts=len(exit_codes), exit_codes=exit_codes,
surface_lost_retries=surface_lost, seconds=seconds, notes=notes)
# ---- the shard ---------------------------------------------------------------------
def append_step_summary(self, text):
path = self.environ.get("GITHUB_STEP_SUMMARY")
if path:
with open(path, "a", encoding="utf-8") as summary:
summary.write(text)
def run_all(self):
self.write_shard_status()
say(f"trace-replay-shard: {len(self.cases)} case(s) on {self.options.backend} / "
f"{self.options.transport} ({self.options.lane})")
say("trace-replay-shard: every run: " + json.dumps(
{"command": self.replay_command, "environment": self.shared_environment(),
"inherited": {name: self.environ.get(name) for name in INHERITED_REPLAY_SETTINGS}}, sort_keys=True))
if not self.prepare_booted_device():
if not self.restart_emulator("the trace APK did not install"):
self.aborted = "the trace APK could not be installed"
for index, case in enumerate(self.cases):
say(f"::group::{case['name']} ({index + 1}/{len(self.cases)})")
if self.aborted:
record = self.record_for(case, NOT_RUN, self.aborted)
else:
try:
record = self.run_case(index, case)
except Exception as error: # one case's bookkeeping must not cost the others their verdicts
record = self.record_for(case, FAILED, f"the shard runner failed: {one_line(error)}")
self.unclean_case = "" if record["status"] == PASSED else case["name"]
self.save_record(case, record)
say(f"trace-replay-shard: record {case['name']}: {json.dumps(record, sort_keys=True)}")
say(f"trace-replay-shard: emulator boots after {case['name']}: {self.boots}")
say(f"trace-replay-shard: {case['name']}: {record['status']}"
+ (f" ({record['reason']})" if record["reason"] else ""))
say("::endgroup::")
return self.finish()
def finish(self):
"""Tally from the files on disk, so a verdict the loop never wrote counts as missing."""
label = f"{self.options.backend} / {self.options.transport}"
rows = []
verdicts = 0
failed = []
for case in self.cases:
record = read_json(self.result_dir(case) / "case-status.json")
if not isinstance(record, dict) or record.get("case") != case["name"]:
record = self.record_for(case, NOT_RUN, "no verdict file was written")
if record["status"] in (PASSED, FAILED):
verdicts += 1
if record["status"] == PASSED:
result = read_json(self.result_dir(case) / "result.json")
if not isinstance(result, dict) or result.get("passed") is not True:
record = dict(record, status=FAILED, reason="the verdict is passed but result.json is not")
if record["status"] != PASSED:
failed.append(record)
result = read_json(self.result_dir(case) / "result.json")
ssim = result.get("ssim", "") if isinstance(result, dict) else ""
seconds = record.get("seconds", {}).get("total", "")
note = one_line("; ".join([record.get("reason", "")] + record.get("notes", [])).strip("; "))
rows.append(f"| {case['name']} | {record['status']} | {record.get('attempts', 0)} | {seconds} | "
f"{ssim} | {note.replace('|', '/')} |")
self.write_shard_status(complete=verdicts == len(self.cases))
summary = [f"### Android retrace: {label} ({self.options.lane})", "",
f"{len(self.cases) - len(failed)} of {len(self.cases)} passed, {self.boots} emulator boot(s).", "",
"| case | status | attempts | seconds | SSIM | note |", "| --- | --- | --- | --- | --- | --- |"]
self.append_step_summary("\n".join(summary + rows) + "\n\n")
for record in failed:
say(f"::error title=Android retrace {record['status']} ({label})::"
f"{record['case']}: {record.get('reason') or record['status']}")
if verdicts != len(self.cases):
say(f"::error title=Android retrace shard incomplete ({label})::"
f"{verdicts} verdict(s) for {len(self.cases)} case(s)")
self.say_shard_status()
say(f"trace-replay-shard: {len(self.cases) - len(failed)} passed, {len(failed)} not passed, "
f"{verdicts} verdict(s) for {len(self.cases)} case(s)")
return 0 if not failed and verdicts == len(self.cases) else 1
def finalize(self):
"""Give every case its summary inputs and a verdict file without touching the device.
Runs after the replay step whatever its outcome, so a shard that was killed or never
started still uploads one named row per case.
"""
for case in self.cases:
record = read_json(self.result_dir(case) / "case-status.json")
if isinstance(record, dict) and record.get("case") == case["name"]:
self.records[case["name"]] = record
self.write_summary_inputs(case)
else:
self.save_record(case, self.record_for(case, NOT_RUN, "the shard did not reach this case"))
previous = read_json(self.result_root / "shard-status.json")
if isinstance(previous, dict):
self.boots = previous.get("emulator_boots", self.boots)
self.install_seconds = previous.get("install_seconds", self.install_seconds)
self.write_shard_status(
complete=all(self.records[case["name"]]["status"] in (PASSED, FAILED) for case in self.cases))
self.say_shard_status()
return 0
def dry_run(self):
for case in self.cases:
say(json.dumps(self.plan(case), sort_keys=True))
return 0
def parse_arguments(argv):
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--apk-file", required=True, help="Trace APK to install once per emulator boot.")
parser.add_argument("--package", default="top.mobilegl.plugin.trace")
parser.add_argument("--backend", required=True, choices=BACKENDS)
parser.add_argument("--transport", required=True, choices=TRANSPORTS)
parser.add_argument("--lane", required=True, help="Lane name written to lane.json, e.g. inproc-acceptance.")
parser.add_argument("--gpu", default="", help="Emulator -gpu mode, needed to restart the emulator.")
source = parser.add_mutually_exclusive_group(required=True)
source.add_argument("--cases-json", help="JSON array of the APK matrix's case objects.")
source.add_argument("--cases-file", help="File holding that JSON array.")
source.add_argument("--cases-env", help="Name of the environment variable holding that JSON array.")
parser.add_argument("--result-root", default="android-retrace-result")
parser.add_argument("--fixture-root", default="android-retrace-fixture")
parser.add_argument("--deadline-seconds", type=int, default=0,
help="Stop starting cases, and cut a running one, this long after the start; both are "
"recorded as not-run. 0: no limit. retrace_shard_budget.py says what a shard needs.")
mode = parser.add_mutually_exclusive_group()
mode.add_argument("--dry-run", action="store_true",
help="Print each case's environment and trace-replay-ci.sh arguments, run nothing.")
mode.add_argument("--finalize", action="store_true",
help="Only write the summary inputs and a not-run verdict for cases without one.")
return parser.parse_args(argv)
def main(argv=None):
options = parse_arguments(argv)
if options.cases_json is not None:
text = options.cases_json
elif options.cases_file is not None:
text = Path(options.cases_file).read_text(encoding="utf-8")
else:
text = os.environ.get(options.cases_env, "")
try:
cases = load_cases(text, options.backend)
except ValueError as error:
say(f"::error title=Android retrace shard misconfigured::{error}")
return 2
shard = Shard(options, cases, os.environ)
if shard.hang_budget_ms is None:
say("::error title=Android retrace shard misconfigured::MOBILEGL_GPU_HANG_BUDGET_MS is not set")
return 2
if options.dry_run:
return shard.dry_run()
if options.finalize:
return shard.finalize()
if not os.path.isfile(options.apk_file):
say(f"::error title=Android retrace shard misconfigured::APK does not exist: {options.apk_file}")
return 2
# A termination request unwinds through Shard.run, which kills the running child first.
signal.signal(signal.SIGTERM, lambda number, frame: sys.exit(128 + number))
return shard.run_all()
if __name__ == "__main__":
sys.exit(main())
+158
View File
@@ -0,0 +1,158 @@
---
name: mali-gpu-counters
description: Capture and compare Mali GPU hardware counters (fragment/vertex cycles, tiles, ZS, bandwidth, tile write elimination) on a rooted Android phone through perfetto and the vendor's libgpudataproducer, to find where one renderer spends more GPU time than another. Use when a GPU-bound workload is slower on Magma (DirectVulkan) than on the device's GLES driver, or slower after a change, and you need to know which part of the GPU pipeline the gap is in.
---
# Mali GPU performance counters
Use this to answer "where does Magma spend the GPU time that the system GLES driver (or the previous build) doesn't?" on a Mali phone. Two captures, each normalized per frame, show which pipeline stage is responsible: vertex/tiling, fragment shading, depth testing, or memory bandwidth. Do this before guessing at shader or pass-structure fixes.
The workflow was built and verified on a MediaTek MT6893 (Mali-G77 MC9, r32 driver, TrebleDroid). The vendor library path and counter IDs can differ on other devices; step 2 shows how to list them.
## Prerequisites
- A rooted phone (`adb root` or `su`). Check that `/vendor/lib64/libgpudataproducer.so` exists. This is the vendor library behind Android GPU Inspector's counters.
- The workload must be **GPU-bound**. Check `/sys/kernel/ged/hal/gpu_utilization` on MediaTek (first field ≥ 90%). Otherwise counter differences say nothing about FPS.
- Fixed clocks, so cycles are comparable across runs. On MT6893:
- GPU: `echo 836000 > /proc/gpufreq/gpufreq_opp_freq`.
- CPU: `echo 7 1 > /proc/ppm/policy_status`, then `echo <cluster> <kHz>` into `/proc/ppm/policy/userlimit_{min,max}_cpu_freq`.
- The sysfs `scaling_*_freq` files are read-only there.
- Restore everything afterwards.
- Host: the NDK, `protoc`, and a venv with `pip install perfetto` (TraceProcessor downloads its shell binary on first use).
## 1. Start the counter producer
`gpuprod.c` dlopens the vendor library and calls its exported `start()`, which registers the perfetto data source `gpu.counters`.
```sh
$NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang -O2 -o gpuprod gpuprod.c
adb push gpuprod /data/local/tmp/ && adb shell chmod 755 /data/local/tmp/gpuprod
adb shell 'su -c "getenforce"' # note the mode to restore later
adb shell 'su -c "setenforce 0; nohup /data/local/tmp/gpuprod > /data/local/tmp/gpuprod.log 2>&1 &"'
adb shell perfetto --query | grep gpu.counters # must list /data/local/tmp/gpuprod
```
It needs SELinux permissive. **Afterwards**, kill the producer (`su -c "kill $(pidof gpuprod)"`) and restore the mode (`setenforce 1` if it was Enforcing).
## 2. List the counters this GPU offers
A capture without `counter_ids` records nothing, and the IDs are device-specific. Read them from the producer's descriptor:
```sh
adb exec-out perfetto --query-raw > qraw.bin
protoc --decode_raw < qraw.bin > qraw.txt
python3 - <<'P'
import re
s = open('qraw.txt').read()
for i, n in re.findall(r'\n\s*1: (\d+)\n\s*2: "([^"\n]{2,90})"\n\s*3: "', s):
print(i, n)
P
```
IDs below 65535 are raw hardware counters. IDs 65535 and up are derived expressions (percentages, rates). Prefer raw counters: they add up over time, while ratios don't.
On the Mali-G77 these raw IDs were the useful set:
| ID | Counter | Use |
|---|---|---|
| 6 | GPU active cycles | total GPU cost |
| 196 / 214 | Fragment / non-fragment active cycles | which side the gap is on |
| 68 | Tiler active cycles | vertex + binning cost |
| 160 / 175 | Output external read / write beats | DRAM bandwidth |
| 210 | Tiles | how many tile passes the frame runs (≈ passes × screen tiles) |
| 211 | Constant tiles killed | tile write-backs eliminated because the tile was unchanged |
| 254 | Tile buffer write beats to L2 | tile write-back traffic |
| 203 / 202 | Rasterized / partial rasterized quads | fragment workload size |
| 204 / 206 | Early ZS tested / killed quads | early depth test working |
| 207 / 208 | Late ZS tested / killed quads | shaders forced to late ZS (discard, depth writes, clears via shader) |
| 212 | FPK occluder quads | forward pixel kill |
| 234 | Texture quads | quads that sample textures |
| 219 / 220 / 221 / 222 | FMA / CVT / SFU / message instructions | shader instruction mix |
| 242 / 243 | 32-bit / 16-bit interpolation active | fp32 vs fp16 varyings |
| 209 | Warps using more than 32 registers | register pressure |
| 70 / 75 | Triangle / visible primitives | geometry identity check |
| 85 / 101 | IDVS position / varying shading requests | vertex shading |
`counters.cfg` captures this set: 5 s at a 20 ms sampling period.
## 3. Capture each renderer
Get each renderer into the same steady scene: same camera, frozen world time, same resolution. Then capture while it runs:
```sh
adb push counters.cfg /data/local/tmp/
adb shell 'cat /data/local/tmp/counters.cfg | perfetto --txt -c - -o /data/misc/perfetto-traces/cnt.pftrace'
adb pull /data/misc/perfetto-traces/cnt.pftrace A.pftrace
```
- Record each run's FPS during the capture (for example, the on-screen FPS counter in a screenshot taken right after).
- Change only the renderer between captures.
- Cool the phone between runs and keep the clocks locked.
## 4. Compare per frame
```sh
python compare.py A.pftrace <fpsA> B.pftrace <fpsB>
```
It sums each counter over the capture, converts to per second, divides by FPS, and prints A/frame, B/frame and B/A, sorted by A's per-frame cost. Many values come out fractional, because the producer averages per-core counters across shader cores. Only the ratios matter.
## 5. Read the result
Work top-down:
1. **GPU active cycles B/A**: the size of the gap. It should roughly match the FPS ratio when GPU-bound.
2. **Fragment vs non-fragment active cycles.** Whichever ratio is above 1 holds the gap. Equal vertex counters (non-fragment, tiler, primitives, IDVS) mean the geometry and vertex shading are identical, so look only at fragment work.
3. **Tiles.** B/A ≈ 1 + k means B renders about k extra full-screen passes per frame. Typical causes:
- a standalone clear pass;
- a pass split by a mid-frame upload or barrier;
- a blit, copy or clear outside a render pass (Mali runs those as fragment jobs too).
Find them with a per-frame Vulkan command census (see [Census of Magma's Vulkan commands](#census-of-magmas-vulkan-commands)).
4. **Rasterized quads, late ZS, texture quads.**
- With equal tiles: more rasterized quads mean extra full-screen draws inside a pass. For example, `vkCmdClearAttachments` mid-pass costs about one screen of quads each time.
- High late-ZS counts mean fragments lost the early depth test: shaders with `discard`, `gl_FragDepth` or `gl_SampleMask` writes, or clears implemented as shader draws.
- More early-ZS kills with equal visible primitives often mean the same thing seen from the other side.
5. **External write beats vs tile buffer write beats.**
- If B's extra external writes ≈ extra tile write beats × shader-core count, the difference is tile write-back.
- Check `Constant tiles killed`: the GLES driver eliminates write-back of unchanged tiles. A Vulkan path with a much lower count is losing that elimination.
- In a static benchmark scene this is a big effect. With a moving camera it is a small one.
6. **Instruction mix and interpolation.** More FMA or CVT instructions with equal quads point at shader code quality. Much more 32-bit than 16-bit interpolation points at precision: for example, ESSL `mediump` lost when a backend recompiles the shader as desktop GLSL.
## Census of Magma's Vulkan commands
When the tile count says Magma runs extra passes, find out which ones. Use a temporary, never-committed build:
- **Wrappers:** write a header that declares counting wrappers and maps each `vkCmd*` entry point to its wrapper with a function-like macro (`#define vkCmdBeginRenderPass(...) Census_BeginRenderPass(__VA_ARGS__)`). Function-like macros leave function-pointer assignments such as `fns.vkCmdBeginRenderPass = vkCmdBeginRenderPass;` alone.
- Include `VK_USE_PLATFORM_ANDROID_KHR` before `<vulkan/vulkan.h>` in that header.
- Map at least render pass begin, render pass creation (to record each pass's load/store ops), blit, copy, clear (including `vkCmdClearAttachments`), resolve, dispatch, barrier, submit and present.
- **Force-include:** apply it only to the DirectVulkan sources:
```cmake
file(GLOB _census ${CMAKE_CURRENT_SOURCE_DIR}/MobileGL/MG_Backend/DirectVulkan/*.cpp
${CMAKE_CURRENT_SOURCE_DIR}/MobileGL/MG_Backend/DirectVulkan/Renderer/*.cpp)
list(FILTER _census EXCLUDE REGEX "Census") # the wrapper definitions must call the real entry points
set_source_files_properties(${_census} PROPERTIES COMPILE_OPTIONS "-include;<path>/Census.h")
```
- **Recording:** in each wrapper, key the count by kind, the caller's offset in the library (`__builtin_return_address(0)` minus `dladdr`'s `dli_fbase`) and details: render area plus load/store ops for passes, area for clears and blits. In the present wrapper, every few seconds, log the counts divided by the number of frames to logcat.
- **Symbolize** the offsets against the unstripped library: `llvm-addr2line -f -C -i -e build/intermediates/cxx/RelWithDebInfo/*/obj/arm64-v8a/libMobileGL.so 0x<offset>`. The library under `stripped_native_libs` has no symbols. If a wrapper's caller is a shared helper, log `__builtin_return_address(1..5)` inside that helper; arm64 Android keeps frame pointers.
## Example: Magma vs the system driver
Minecraft 1.12.2 with OptiFine at 2400×1080 on a Mali-G77, clocks locked, SFPEW over each backend:
| Finding | Counters | Cause | Fix |
|---|---|---|---|
| Fragment active cycles 1.33×, tiles 1.46× | rasterized quads, late ZS and tiles all up; vertex counters equal | one extra full-screen depth-only pass per frame (census: `vkCmdBeginRenderPass` with only `D24S8` LOAD/STORE, plus a full-screen `vkCmdClearAttachments`) | a window clear with a window pass open now ends the pass and folds into the next one (`VulkanRenderer::DeferDefaultWireClear`) |
| Extra external writes 1.7× | tile buffer write beats ≈ 23×, constant tiles killed 0.07× | the GLES driver eliminates the write-back of unchanged tiles in a static scene; Magma's attachments get none | still open |
| Fragment cycles still 1.18× with equal tiles | rasterized quads 1.26×, late ZS 1.44× | the game's two in-pass depth clears per frame, recorded as full-screen `vkCmdClearAttachments` | under investigation |
Separately, SFPEW hands MobileGL ESSL with `precision mediump float;` inside `#ifdef GL_ES`. The source processor rewrites ESSL to desktop GLSL with `GL_ES` undefined, so that guard silently turned the shaders fp32. 16-bit interpolation counters and the RelaxedPrecision decorations in the SPIR-V show this kind of loss.
## Pitfalls
- **Unlocked clocks.** Cycle counts are only comparable at the same GPU frequency, and FPS also depends on the CPU clock when the workload is not fully GPU-bound.
- **Not checking the build.** Before comparing builds, verify that each run actually ran the build you think it did. `adb install -r` of an APK with a lower versionCode fails with `INSTALL_FAILED_VERSION_DOWNGRADE`, and the run silently uses whatever was installed.
- **Different scenes.** Counters from different scenes or camera angles are not comparable at all. Freeze world time and camera position between runs.
- **Leaving the phone modified.** The producer and permissive SELinux stay in place until you undo them. Restore both, and the clock locks, when you are done.
@@ -0,0 +1,34 @@
"""Per-frame comparison of two Mali counter captures.
usage: python compare.py A.pftrace FPS_A B.pftrace FPS_B
Needs `pip install perfetto`.
"""
import sys
from perfetto.trace_processor import TraceProcessor
def per_second(path):
tp = TraceProcessor(trace=path)
rows = tp.query(
"select t.name, sum(c.value) s, (max(c.ts) - min(c.ts)) / 1e9 dur "
"from gpu_counter_track t join counter c on c.track_id = t.id group by t.id")
out = {r.name: r.s / r.dur for r in rows if r.dur > 0}
tp.close()
return out
def main():
if len(sys.argv) != 5:
sys.exit(__doc__)
a, fps_a, b, fps_b = sys.argv[1], float(sys.argv[2]), sys.argv[3], float(sys.argv[4])
A, B = per_second(a), per_second(b)
print(f"{'counter':52s} {'A/frame':>14s} {'B/frame':>14s} {'B/A':>6s}")
for name in sorted(A, key=lambda k: -A[k] / fps_a):
pa, pb = A[name] / fps_a, B.get(name, 0.0) / fps_b
ratio = f"{pb / pa:6.2f}" if pa else " n/a"
print(f"{name:52s} {pa:14.0f} {pb:14.0f} {ratio}")
if __name__ == "__main__":
main()
@@ -0,0 +1,50 @@
# Mali-G77 raw counters (IDs from step 2 of SKILL.md); 5 s at a 20 ms period.
buffers { size_kb: 131072 fill_policy: DISCARD }
data_sources { config { name: "gpu.counters" gpu_counter_config { counter_period_ns: 20000000
counter_ids: 6
counter_ids: 10
counter_ids: 18
counter_ids: 196
counter_ids: 214
counter_ids: 68
counter_ids: 160
counter_ids: 175
counter_ids: 218
counter_ids: 219
counter_ids: 220
counter_ids: 221
counter_ids: 222
counter_ids: 245
counter_ids: 201
counter_ids: 216
counter_ids: 203
counter_ids: 202
counter_ids: 204
counter_ids: 206
counter_ids: 207
counter_ids: 208
counter_ids: 212
counter_ids: 210
counter_ids: 211
counter_ids: 254
counter_ids: 250
counter_ids: 251
counter_ids: 231
counter_ids: 234
counter_ids: 241
counter_ids: 242
counter_ids: 243
counter_ids: 247
counter_ids: 248
counter_ids: 249
counter_ids: 70
counter_ids: 75
counter_ids: 85
counter_ids: 101
counter_ids: 209
counter_ids: 8
counter_ids: 16
counter_ids: 226
counter_ids: 223
} } }
duration_ms: 5000
@@ -0,0 +1,26 @@
/*
* Starts the vendor Mali counter producer so perfetto can record the
* "gpu.counters" data source. Run as root, with SELinux permissive.
*
* Build: aarch64-linux-android30-clang -O2 -o gpuprod gpuprod.c
*/
#include <dlfcn.h>
#include <stdio.h>
#include <unistd.h>
int main(void) {
void* lib = dlopen("/vendor/lib64/libgpudataproducer.so", RTLD_NOW);
if (!lib) {
fprintf(stderr, "dlopen: %s\n", dlerror());
return 1;
}
void (*start)(void) = (void (*)(void))dlsym(lib, "start");
if (!start) {
fprintf(stderr, "libgpudataproducer.so exports no start()\n");
return 1;
}
fprintf(stderr, "starting producer\n");
start();
for (;;)
pause();
}
+6
View File
@@ -64,6 +64,12 @@ them wrong.
| 2026-10-18 | `Negative control A (corrupted snapshot)` and `Keep lane logs before controls` | Test / `integration-verify-split` | the first re-ran the ten `VerifySplit.ClearThenReadPixelsScenario` entries of both backends with the same knob and accepted any red; the second only kept the lane's logs from being overwritten by that rerun | the in-lane `VerifySplitCorrupted` and `VerifySplitReadCorrupted` entries of both backends, required by name (`Verify-split control entries`); that the ambient entries arm is the per-entry arm proof, and that they run fatal is `VerifySplitArming.PipeVerifyArmingScenario.Armed` (the ambient environment plus its marker and log), which requires `fatal=1` on the arming line | | 2026-10-18 | `Negative control A (corrupted snapshot)` and `Keep lane logs before controls` | Test / `integration-verify-split` | the first re-ran the ten `VerifySplit.ClearThenReadPixelsScenario` entries of both backends with the same knob and accepted any red; the second only kept the lane's logs from being overwritten by that rerun | the in-lane `VerifySplitCorrupted` and `VerifySplitReadCorrupted` entries of both backends, required by name (`Verify-split control entries`); that the ambient entries arm is the per-entry arm proof, and that they run fatal is `VerifySplitArming.PipeVerifyArmingScenario.Armed` (the ambient environment plus its marker and log), which requires `fatal=1` on the arming line |
| 2026-10-18 | unit `PipeInputsTest.{PoisonOmitKnobArmsTheOmission, BadPoisonOmitKnobIsFatalNamingTheKnob}`, and the `MOBILEGL_PIPE_POISON_OMIT` knob: `Features.PipePoisonOmit`, its parser on every verb (`MGPipeValidateForVerb`'s `ParsePoisonOmissionKnob`, compiled into every build) and the server stamp's own parse of it | Test / unit (every poison lane); product code | the knob existed only for negative control B, and the two cases pinned its parser and its `Fatal{PipeVerifyBadKnob}` | the PoisonOmitted and VerifySplitPoisonOmitted entries set the harness marker `MGITEST_POISON_OMIT` to the same pair, and the re-executed child arms it through `MGPipeSetPoisonOmission`, compiled only with `-DMOBILEGL_PIPE_VERIFY=ON` (`Harness/PoisonOmissionArm.h`); `OmittingOneFieldForOneVerbLeavesExactlyThatFieldStale`, `ReadingAnOmittedFieldAbortsNamingTheVerb` and `TheMutationNoticeRefreshesTheValueButNotTheStamp` withhold the stamp test-side and still run in every poison lane; the setter's scope to its verb, not its class, is checked by `OmittingOneFieldForOneVerbLeavesExactlyThatFieldStale` in a forked child, on the verify runtime only | | 2026-10-18 | unit `PipeInputsTest.{PoisonOmitKnobArmsTheOmission, BadPoisonOmitKnobIsFatalNamingTheKnob}`, and the `MOBILEGL_PIPE_POISON_OMIT` knob: `Features.PipePoisonOmit`, its parser on every verb (`MGPipeValidateForVerb`'s `ParsePoisonOmissionKnob`, compiled into every build) and the server stamp's own parse of it | Test / unit (every poison lane); product code | the knob existed only for negative control B, and the two cases pinned its parser and its `Fatal{PipeVerifyBadKnob}` | the PoisonOmitted and VerifySplitPoisonOmitted entries set the harness marker `MGITEST_POISON_OMIT` to the same pair, and the re-executed child arms it through `MGPipeSetPoisonOmission`, compiled only with `-DMOBILEGL_PIPE_VERIFY=ON` (`Harness/PoisonOmissionArm.h`); `OmittingOneFieldForOneVerbLeavesExactlyThatFieldStale`, `ReadingAnOmittedFieldAbortsNamingTheVerb` and `TheMutationNoticeRefreshesTheValueButNotTheStamp` withhold the stamp test-side and still run in every poison lane; the setter's scope to its verb, not its class, is checked by `OmittingOneFieldForOneVerbLeavesExactlyThatFieldStale` in a forked child, on the verify runtime only |
| 2026-10-18 | the `serial == resets` assertion of `CtWireScenario.ApplierResetCrossesAtThePrimedEdgeInSerialOrder`, and the `MOBILEGL_TEST_SERVER_COUNTERS` knob: `ServerMain`'s `MGPipe server counters:` line on every LogFlush and `ServerVerbSink`'s test-only tallies (`ApplierResets`, `ObjectDeaths`, `ExpectedApplierResetSerial`) | Test / `integration-split`, `integration-split-strict`, `integration-verify-split`, `integration (inproc)`; product code | the knob existed only so the spawn and tcp arms could read the sink's tallies; the serial assertion could not fail, since `OnApplierReset` moves the count and the serial together and any other serial latches `Fatal{ProtocolCorruption, "ApplierReset.ContextSerial"}` | `ar` / `od` on the server role's `MGPipe stats:` line, counted in `ServerVerbSink::OnApplierReset` / `OnObjectDeath`; every CtWire entry that counts them sets `MOBILEGL_PIPE_STATS=1` with period 1 and reads them on every arm, inproc included, over a window closed by a swap and a readback; the unit cases `ApplierResetWireRoundTrip.TheSerialSequenceIsAssertedAndTheServerResetRuns` and `ObjectDeathWireRoundTrip.TheTextureAndFramebufferTwinsReleaseByHandleAndNeverByAStaleGeneration` read the same counters, and the first shows the expected serial moving by having serial 2 accepted | | 2026-10-18 | the `serial == resets` assertion of `CtWireScenario.ApplierResetCrossesAtThePrimedEdgeInSerialOrder`, and the `MOBILEGL_TEST_SERVER_COUNTERS` knob: `ServerMain`'s `MGPipe server counters:` line on every LogFlush and `ServerVerbSink`'s test-only tallies (`ApplierResets`, `ObjectDeaths`, `ExpectedApplierResetSerial`) | Test / `integration-split`, `integration-split-strict`, `integration-verify-split`, `integration (inproc)`; product code | the knob existed only so the spawn and tcp arms could read the sink's tallies; the serial assertion could not fail, since `OnApplierReset` moves the count and the serial together and any other serial latches `Fatal{ProtocolCorruption, "ApplierReset.ContextSerial"}` | `ar` / `od` on the server role's `MGPipe stats:` line, counted in `ServerVerbSink::OnApplierReset` / `OnObjectDeath`; every CtWire entry that counts them sets `MOBILEGL_PIPE_STATS=1` with period 1 and reads them on every arm, inproc included, over a window closed by a swap and a readback; the unit cases `ApplierResetWireRoundTrip.TheSerialSequenceIsAssertedAndTheServerResetRuns` and `ObjectDeathWireRoundTrip.TheTextureAndFramebufferTwinsReleaseByHandleAndNeverByAStaleGeneration` read the same counters, and the first shows the expected serial moving by having serial 2 accepted |
| 2026-11-15 | per-case jobs `retrace (<backend>, <case>)`, `retrace verify (<backend>, <case>)`, `retrace split (<backend>, <case>, <transport>)` | Test / `retrace`, `retrace-verify`, `retrace-split` | one runner per case spent most of its time on setup; no case, knob, assertion or arm proof is dropped | shard jobs that replay the same rows one after another through `scripts/ci/retrace_shard.py`, one ctest entry and one verdict per case; `trace_cases.py --self-test-shard-matrices` checks that the shards hold exactly the per-case rows |
| 2026-11-15 | per-case emulator jobs `retrace (<backend>, <case>)` of each lane | MobileGL APK / `retrace` | one emulator boot per replay; no case, setting or arm proof is dropped | shard jobs that replay the cases of one lane, transport and backend on one boot through `android-plugin/trace_replay_shard.py`, with app data cleared between cases, a cold boot after a case that did not pass, and one verdict per case |
| 2026-11-15 | per-case `trace fixture (<case>)` jobs, their per-case cache entries and artifacts, `.github/scripts/trace-fixture-cache.sh` | Test and MobileGL APK / `trace-fixtures` | 40 jobs and 40 cache entries per workflow for files that change rarely | one content-addressed cache entry for all Git LFS fixtures (`.github/scripts/trace-fixture-bundle.sh`); a replay job restores it and downloads a case only when the entry does not hold it |
| 2026-11-15 | `remove-artifact-clutter` job | Test and MobileGL APK | it deleted the fixture artifacts and the green result artifacts after the run; the fixture artifacts no longer exist | the delete step at the end of `retrace-summary`; on Linux `retention-days: 1` on the summary inputs as well, `retrace-split` and `retrace-shipped` upload results only for a failed shard (their green results were deleted unread before), and `retrace-verify` keeps its results for every shard as before |
| 2026-11-15 | emulator replays of the cases outside `android_smoke` on the monolith control and spawn acceptance lanes (56 of 79 rows per lane; the 12 smoke cases and the whole inproc lane stay) | MobileGL APK / `retrace` | the 237 emulator replays of a push held the 20 runner slots for most of an hour, and the launcher ships inproc; what goes is the same-commit Android monolith control and the Android spawn evidence (server exec from the native library directory, the pbuffer surface) for those 56 rows | Test replays every case on monolith, inproc and spawn on Linux; the Android inproc lane replays every case; `workflow_dispatch` with `android_full_matrix` runs the full emulator matrix; `trace_cases.py --self-test-transport-matrices` checks that the reduced lanes are the full matrix less exactly those rows |
| 2026-11-15 | the three separate ctest invocations of `Split lanes (inproc, spawn, TCP)` (`-L integration-split`, `-L integration-spawn`, `-L '^integration-tcp$'`) | Test / `integration-split` | every TCP entry holds one resource lock, so the TCP invocation ran one entry at a time while the other cores were idle; the three selections share no entry | one invocation of the union of the three selections: each arm is still discovered with its own label regex, `lane_selection.py union` writes the exact names and fails when an arm selects nothing, and `junit_tally.py --select-from/--write-selected` carves each arm's JUnit out of the run under its old stem and fails when a selected name is missing or reported twice; the per-arm tallies, the entries required by name, the run-ahead proof and the skip-census rows are unchanged |
## Kept after investigation ## Kept after investigation
+70 -17
View File
@@ -54,7 +54,8 @@ CONTENT_MAX = 72
SCOPE_ABBREVIATE_OVER = 3 SCOPE_ABBREVIATE_OVER = 3
CONTINUATION_MAX = 2 CONTINUATION_MAX = 2
COMMENT_BLOCK_MAX = 15 COMMENT_BLOCK_MAX = 15
BASE_BRANCH = os.environ.get("TEXT_HYGIENE_BASE_BRANCH", "feat/disaggregated") # The branch a new branch is compared with: the repository's default branch, which CI passes in.
BASE_BRANCH = os.environ.get("TEXT_HYGIENE_BASE_BRANCH") or "dev"
SUBJECT_RE = re.compile(r"^\[(?P<types>[^\]]+)\] \((?P<scopes>[^)]+)\): (?P<content>\S.*)$") SUBJECT_RE = re.compile(r"^\[(?P<types>[^\]]+)\] \((?P<scopes>[^)]+)\): (?P<content>\S.*)$")
# Ids from the phased planning process: decision ids, contract documents, lettered phases, and a # Ids from the phased planning process: decision ids, contract documents, lettered phases, and a
@@ -610,25 +611,51 @@ def check_format(rev_range):
# ------------------------------------------------------------------------------------------ CI # ------------------------------------------------------------------------------------------ CI
def ci_range(): def push_before():
"""The commits a CI run is about: the pushed ones, or what this branch adds to the base branch.""" """The pushed branch's previous tip; all zeros when the push created the branch, empty when the
head = git("rev-parse", "HEAD").strip() run is not a push."""
before = ""
event = os.environ.get("GITHUB_EVENT_PATH") event = os.environ.get("GITHUB_EVENT_PATH")
if os.environ.get("GITHUB_EVENT_NAME") == "push" and event and os.path.exists(event): if os.environ.get("GITHUB_EVENT_NAME") != "push" or not event or not os.path.exists(event):
with open(event, encoding="utf-8") as fh: return ""
before = json.load(fh).get("before") or "" with open(event, encoding="utf-8") as fh:
if (before and set(before) != {"0"} and git("cat-file", "-t", before, check=False).strip() == "commit" return json.load(fh).get("before") or ""
and subprocess.run(["git", "-C", ROOT, "merge-base", "--is-ancestor", before, head]).returncode == 0):
def ci_range(repo=ROOT, base_branch=None, before=None, ref_name=None):
"""(base, range) for a CI run: the pushed commits when the push moved an existing branch forward,
else what the branch adds to the base branch. A branch that adds nothing to it, such as a new one
at its tip, checks only the tree; the base branch at its own tip checks its last commit."""
base_branch = BASE_BRANCH if base_branch is None else base_branch
before = push_before() if before is None else before
if ref_name is None:
ref_name = (os.environ.get("GITHUB_REF_NAME")
or git("rev-parse", "--abbrev-ref", "HEAD", check=False, repo=repo).strip())
head = git("rev-parse", "HEAD", repo=repo).strip()
if (before and set(before) != {"0"} and git("cat-file", "-t", before, check=False, repo=repo).strip() == "commit"
and is_ancestor(before, head, repo)):
return before, "%s..HEAD" % before return before, "%s..HEAD" % before
for ref in ("origin/" + BASE_BRANCH, "origin/HEAD"): tip_of = None
base = git("merge-base", "HEAD", ref, check=False).strip() for ref in ("origin/" + base_branch, "origin/HEAD"):
base = git("merge-base", "HEAD", ref, check=False, repo=repo).strip()
if base and base != head: if base and base != head:
return base, "%s..HEAD" % base return base, "%s..HEAD" % base
if base == head: if base == head:
# The branch the ref stands for: origin/HEAD names the remote's default branch.
tip_of = (git("symbolic-ref", "--short", "-q", "refs/remotes/" + ref, check=False, repo=repo).strip()
or ref)
if ref_name != tip_of.split("/", 1)[1]:
notice("%s adds no commits to %s; checking only the tree" % (ref_name or "HEAD", tip_of))
return head, "%s..HEAD" % head
break break
parent = git("rev-parse", "HEAD^").strip() parent = git("rev-parse", "--verify", "--quiet", "HEAD^", check=False, repo=repo).strip()
notice("no base branch to compare with; checking only the last commit") if not parent:
# A shallow or single-commit history: HEAD^! is HEAD alone, and there is no older baseline.
notice("no parent commit to compare with; checking only the last commit")
return None, "%s^!" % head
if tip_of:
notice("%s is at the tip of %s; checking only the last commit" % (ref_name, tip_of))
else:
notice("no base branch to compare with; checking only the last commit")
return parent, "%s..HEAD" % parent return parent, "%s..HEAD" % parent
@@ -781,7 +808,7 @@ def self_test():
"diff --git a/b.cpp b/b.cpp\n--- /dev/null\n+++ b/b.cpp\n@@ -0,0 +1,4 @@\n+int f();\n+int g();\n" "diff --git a/b.cpp b/b.cpp\n--- /dev/null\n+++ b/b.cpp\n@@ -0,0 +1,4 @@\n+int f();\n+int g();\n"
"+ int f();\n+- x\n") "+ int f();\n+- x\n")
expect("format: moved lines are not checked", unmoved_ranges(moved) == {"b.cpp": [(2, 3)]}) expect("format: moved lines are not checked", unmoved_ranges(moved) == {"b.cpp": [(2, 3)]})
expect("range: first parent and cutoff", range_self_test()) expect("range: first parent, cutoff and CI base", range_self_test())
for name in failures: for name in failures:
print("self-test FAILED: %s" % name) print("self-test FAILED: %s" % name)
@@ -791,7 +818,8 @@ def self_test():
def range_self_test(): def range_self_test():
"""In a scratch repository: commits before the script landed and commits a merge brings in are """In a scratch repository: commits before the script landed and commits a merge brings in are
not checked; the merge itself and later first-parent commits are.""" not checked; the merge itself and later first-parent commits are. Then the CI range for new,
pushed, force-pushed and up-to-date branches."""
tmp = tempfile.mkdtemp() tmp = tempfile.mkdtemp()
try: try:
def g(*args): def g(*args):
@@ -818,7 +846,32 @@ def range_self_test():
merge = g("rev-parse", "HEAD").strip() merge = g("rev-parse", "HEAD").strip()
after = commit("MobileGL/MG_A/B/c.cpp", "[Fix] (MG_A/B): change c") after = commit("MobileGL/MG_A/B/c.cpp", "[Fix] (MG_A/B): change c")
checked = commits_to_check("%s..HEAD" % old, tmp) checked = commits_to_check("%s..HEAD" % old, tmp)
return checked == [lint, merge, after] and side not in checked if checked != [lint, merge, after] or side in checked:
return False
# ci_range on main at `after`: a new branch is compared with the base branch, a branch at
# the base tip adds nothing, and the base branch at its own tip checks its last commit.
zeros = "0" * 40
dropped = g("commit-tree", "-p", lint, "-m", "a force push dropped this", lint + "^{tree}").strip()
g("update-ref", "refs/remotes/origin/main", merge)
cases = [(("main", zeros, "topic"), (merge, merge + "..HEAD")),
(("main", lint, "main"), (lint, lint + "..HEAD")),
(("main", dropped, "topic"), (merge, merge + "..HEAD"))]
out = io.StringIO()
with contextlib.redirect_stdout(out):
got = [ci_range(tmp, *args) for args, _ in cases]
g("update-ref", "refs/remotes/origin/main", after)
got += [ci_range(tmp, "main", zeros, "topic"), ci_range(tmp, "main", "", "main"),
ci_range(tmp, "gone", zeros, "topic")]
g("symbolic-ref", "refs/remotes/origin/HEAD", "refs/remotes/origin/main")
got += [ci_range(tmp, "gone", zeros, "topic"), ci_range(tmp, "gone", zeros, "main")]
want = [w for _, w in cases] + [(after, after + "..HEAD"), (merge, merge + "..HEAD"),
(merge, merge + "..HEAD"), (after, after + "..HEAD"),
(merge, merge + "..HEAD")]
# The missing base ref alone reports no base branch; main at its own tip gets the tip notice
# through origin/main and through origin/HEAD.
log = out.getvalue()
return got == want and log.count("no base branch") == 1 and log.count("is at the tip of") == 2
finally: finally:
shutil.rmtree(tmp, ignore_errors=True) shutil.rmtree(tmp, ignore_errors=True)
+121
View File
@@ -0,0 +1,121 @@
#!/bin/bash
# MobileGL - scripts/ci/checkout_submodules.sh
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
# The submodules a CI build reads, checked out shallow and in parallel, in place of a
# recursive checkout of every submodule in the tree.
# library what the root CMakeLists.txt configures for the library, its tests and benchmarks.
# retrace the library set plus the trace replay tool's: apitrace and the eight directories
# its thirdparty/CMakeLists.txt requires.
# Each row names a submodule and one file the build reads from it. A row whose file is missing
# after the update fails the script: the configure would otherwise go on without it, and for
# 3rdparty/flatbuffers the root CMakeLists.txt then builds a library without MG_Remote instead
# of stopping. A submodule the build starts to use needs a row here.
#
# usage: checkout_submodules.sh <library|retrace> (from the repository root)
# checkout_submodules.sh --list <library|retrace> prints the rows, checks out nothing
set -euo pipefail
usage="usage: checkout_submodules.sh [--list] <library|retrace>"
list_only=false
if [ "${1:-}" = --list ]; then
list_only=true
shift
fi
set_name="${1:?${usage}}"
# <submodule path> <file the build reads from it>
library=(
"3rdparty/glslang CMakeLists.txt"
"3rdparty/SPIRV-Cross CMakeLists.txt"
"3rdparty/SPIRV-Reflect CMakeLists.txt"
"3rdparty/VulkanMemoryAllocator CMakeLists.txt"
"3rdparty/Vulkan-Headers CMakeLists.txt"
"3rdparty/Vulkan-Utility-Libraries CMakeLists.txt"
"3rdparty/xxHash build/cmake/CMakeLists.txt"
"3rdparty/asio include/asio.hpp"
"3rdparty/flatbuffers include/flatbuffers/flatbuffers.h"
"include/ska flat_hash_map.hpp"
)
retrace=(
"3rdparty/apitrace thirdparty/CMakeLists.txt"
)
# Inside 3rdparty/apitrace: the files its thirdparty/CMakeLists.txt tests before it configures.
apitrace=(
"thirdparty/brotli LICENSE"
"thirdparty/gtest LICENSE"
"thirdparty/libbacktrace LICENSE"
"thirdparty/libpng LICENSE"
"thirdparty/snappy COPYING"
"thirdparty/zlib README"
"thirdparty/zstd LICENSE"
"thirdparty/directxmath LICENSE"
)
case "${set_name}" in
library) top=("${library[@]}"); nested=() ;;
retrace) top=("${library[@]}" "${retrace[@]}"); nested=("${apitrace[@]}") ;;
*)
echo "checkout_submodules.sh: unknown set '${set_name}' (library or retrace)" >&2
echo "${usage}" >&2
exit 2
;;
esac
if [ "${list_only}" = true ]; then
printf '%s\n' "${top[@]}"
if [ "${#nested[@]}" -gt 0 ]; then
printf '3rdparty/apitrace/%s\n' "${nested[@]}"
fi
exit 0
fi
# update <directory> <row>...: one shallow, parallel update of the rows' submodules. The update
# is idempotent, so a fetch that failed on the network is tried again, three times in all.
update() {
local dir="$1"
shift
local paths=() row attempt
for row in "$@"; do
paths+=("${row%% *}")
done
for attempt in 1 2 3; do
if git -C "${dir}" submodule update --init --depth=1 --jobs 8 -- "${paths[@]}"; then
return 0
fi
echo "checkout_submodules.sh: the submodule update in ${dir} failed (attempt ${attempt} of 3)" >&2
if [ "${attempt}" -lt 3 ]; then
sleep $((attempt * 5))
fi
done
return 1
}
# verify <directory> <row>...: every row's file exists; prints each missing one and returns 1.
verify() {
local dir="$1" missing=0 row path file
shift
for row in "$@"; do
path="${row%% *}"
file="${row#* }"
if [ ! -f "${dir}/${path}/${file}" ]; then
echo "::error::checkout_submodules.sh: ${dir}/${path}/${file} is missing after the submodule update, so the '${set_name}' build cannot use ${path}"
missing=1
fi
done
return "${missing}"
}
update . "${top[@]}"
verify . "${top[@]}"
if [ "${#nested[@]}" -gt 0 ]; then
update 3rdparty/apitrace "${nested[@]}"
verify 3rdparty/apitrace "${nested[@]}"
fi
echo "checkout_submodules.sh: the '${set_name}' set is checked out ($(( ${#top[@]} + ${#nested[@]} )) submodules)"
+9
View File
@@ -25,12 +25,17 @@ variant="${1:?usage: configure_build.sh <primary|split> <build dir>}"
build_dir="${2:?usage: configure_build.sh <primary|split> <build dir>}" build_dir="${2:?usage: configure_build.sh <primary|split> <build dir>}"
shift 2 shift 2
# ENABLE_PCH is glslang's option, on by default. ccache cannot cache a compile that uses a
# precompiled header, so with it on every glslang translation unit is compiled again in every
# run. It is in the common part so that both lines stay the same configuration for
# build_fingerprint.py.
common=( common=(
-S . -B "${build_dir}" -G Ninja -S . -B "${build_dir}" -G Ninja
-DCMAKE_C_COMPILER="${MOBILEGL_CI_CC}" -DCMAKE_C_COMPILER="${MOBILEGL_CI_CC}"
-DCMAKE_CXX_COMPILER="${MOBILEGL_CI_CXX}" -DCMAKE_CXX_COMPILER="${MOBILEGL_CI_CXX}"
-DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
-DENABLE_PCH=OFF
) )
case "${variant}" in case "${variant}" in
@@ -74,4 +79,8 @@ case "${variant}" in
;; ;;
esac esac
# The command line goes to the job log, so the log shows which options a build was made with.
printf 'configure_build.sh:'
printf ' %q' cmake "${common[@]}" "${flags[@]}" "$@"
echo
cmake "${common[@]}" "${flags[@]}" "$@" cmake "${common[@]}" "${flags[@]}" "$@"
+138
View File
@@ -0,0 +1,138 @@
#!/usr/bin/env python3
# MobileGL - scripts/ci/lane_selection.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""One ctest run for several lanes: the exact names of every lane, as one --tests-from-file list.
A lane is a `ctest -L <label regex>` selection. Running the lanes of one job as one ctest
invocation lets a lane that holds a resource lock (the TCP arm runs one entry at a time) share
the run with the others. The lanes keep their own selections: each is discovered with its own
label regex, this script writes the union of the discovered names, ctest runs that list, and
junit_tally.py --select-from/--write-selected carves each lane's JUnit back out, which fails
when a lane's name is missing from the run or reported twice.
lane_selection.py union OUT.txt LANE=DISCOVERY.json [LANE=DISCOVERY.json ...]
lane_selection.py --self-test
DISCOVERY.json is `ctest -L <regex> --show-only=json-v1`. union fails when a lane selects
nothing (what --no-tests=error says for a lane run on its own), names an entry twice, or holds a
name a tests-from-file line cannot carry. It prints each lane's count and the size of the union.
"""
import json
from pathlib import Path
import sys
import tempfile
def lane_names(label, path):
"""The names one discovery selects, in order; raises ValueError when the lane is unusable."""
names = [test['name'] for test in json.loads(Path(path).read_text()).get('tests', [])]
if not names:
raise ValueError(f'lane {label}: {path} selects no test')
if len(names) != len(set(names)):
twice = sorted({name for name in names if names.count(name) > 1})
raise ValueError(f'lane {label}: {path} names {len(twice)} entr(ies) twice, e.g. {twice[0]}')
# ctest reads the list one name per line and trims each line.
bad = [name for name in names if name != name.strip() or '\n' in name or '\r' in name]
if bad:
raise ValueError(f'lane {label}: {len(bad)} name(s) cannot be written one per line, e.g. {bad[0]!r}')
return names
def union(lanes):
"""(sorted union of names, {label: count}, number of names more than one lane selects)."""
counts, seen, shared = {}, set(), set()
for label, path in lanes:
if label in counts:
raise ValueError(f'lane {label} is given twice')
names = lane_names(label, path)
counts[label] = len(names)
shared |= seen & set(names)
seen |= set(names)
return sorted(seen), counts, len(shared)
def parse_lanes(arguments):
lanes = []
for argument in arguments:
label, separator, path = argument.partition('=')
if not separator or not label or not path:
raise ValueError(f'expected LANE=DISCOVERY.json, got {argument!r}')
lanes.append((label, path))
if not lanes:
raise ValueError('no lane given')
return lanes
def run(argv):
if len(argv) < 3 or argv[0] != 'union':
print(__doc__, file=sys.stderr)
return 2
try:
names, counts, shared = union(parse_lanes(argv[2:]))
except (ValueError, OSError, json.JSONDecodeError) as error:
print(f'::error::lane_selection.py: {error}')
return 1
Path(argv[1]).write_text(''.join(name + '\n' for name in names))
summary = ', '.join(f'{label} {count}' for label, count in counts.items())
print(f'lane_selection.py: {len(names)} entr(ies) in one run ({summary}; {shared} in more than one lane)')
return 0
def self_test():
import contextlib
import io
def quiet(argv):
# The negative controls print ::error:: lines on purpose; keep them off a green run.
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
return run(argv)
with tempfile.TemporaryDirectory() as tmp:
tmp = Path(tmp)
def discovery(name, names):
path = tmp / name
path.write_text(json.dumps({'tests': [{'name': n} for n in names]}))
return str(path)
out = tmp / 'tests.txt'
split = discovery('split.json', ['DirectGLES.Split.A', 'DirectGLES.Split.B'])
spawn = discovery('spawn.json', ['DirectGLES.Spawn.A', 'SpawnLane.Control'])
tcp = discovery('tcp.json', ['TcpServer.Start', 'DirectGLES.Tcp.A', 'TcpServer.Stop'])
assert quiet(['union', str(out), f'split={split}', f'spawn={spawn}', f'tcp={tcp}']) == 0
assert out.read_text().splitlines() == sorted(
['DirectGLES.Split.A', 'DirectGLES.Split.B', 'DirectGLES.Spawn.A', 'SpawnLane.Control',
'TcpServer.Start', 'DirectGLES.Tcp.A', 'TcpServer.Stop']), 'the union is every lane, each name once'
# A name two lanes select is written once; each lane still carves its own copy.
shared = discovery('shared.json', ['DirectGLES.Split.A', 'Shared.Fixture'])
assert quiet(['union', str(out), f'split={split}', f'other={shared}']) == 0
assert out.read_text().splitlines() == ['DirectGLES.Split.A', 'DirectGLES.Split.B', 'Shared.Fixture']
out.unlink()
controls = {
'a lane that selects nothing': [f'split={split}', f"spawn={discovery('empty.json', [])}"],
'a lane that names an entry twice': [f"split={discovery('twice.json', ['A', 'A'])}"],
'a name with a trailing blank': [f"split={discovery('blank.json', ['A '])}"],
'a name with a line break': [f"split={discovery('break.json', ['A' + chr(10) + 'B'])}"],
'a lane given twice': [f'split={split}', f'split={spawn}'],
'a missing discovery file': [f"split={tmp / 'absent.json'}"],
'an argument without a lane name': [split],
}
for label, arguments in controls.items():
assert quiet(['union', str(out)] + arguments) == 1, f'{label} must fail'
assert not out.exists(), f'{label} must not leave a list behind'
assert quiet(['union', str(out)]) == 2, 'no lane is a usage error'
assert quiet(['nonsense', str(out), f'split={split}']) == 2, 'an unknown command is a usage error'
print('lane_selection self-test: ok')
return 0
if __name__ == '__main__':
if sys.argv[1:] == ['--self-test']:
sys.exit(self_test())
sys.exit(run(sys.argv[1:]))
+689
View File
@@ -0,0 +1,689 @@
#!/usr/bin/env python3
# MobileGL - scripts/ci/retrace_shard.py
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
"""Replay the cases of one retrace shard, one after another, and give each its own verdict.
A shard is some of the cases of one (lane, transport, backend) cell, replayed by one CI job
(tools/trace_replay/trace_cases.py, shard_matrix). Each case runs through its own ctest entry
with the environment and the checks of its lane (case_plan), in a process group and a driver
cache directory of its own, so no case sees what another left behind. A failing case does not
stop the shard. The shard fails when a case failed, or when the verdicts it recorded are not
exactly the cases it was given.
For every case it prints a log group, appends a row to the step summary and records a verdict
in <result-dir>/results/shard-results.json; a failed case also gets an error annotation.
The replay output the lane uploads is moved to <result-dir>/results/<case>/, the JUnit files
are in <result-dir>/junit/, and the extracted trace is deleted.
Usage:
retrace_shard.py run --profile {retrace,split,verify} --backend B [--transport T] [--lane L]
--build-dir DIR --result-dir DIR [--names-env VAR | --names JSON]
[--split-reuse] [--server-path PATH] [--shard N/M]
retrace_shard.py --self-test
"""
import argparse
import json
import os
import re
import shutil
import signal
import subprocess
import sys
import tempfile
import time
import xml.etree.ElementTree as ET
from pathlib import Path
SCRIPTS = Path(__file__).resolve().parent
ITERATIONRP_CASE = "minecraft-1.21.4-fabric-iris-iterationrp-in-world"
BLENDED_DEPTH_WRITE_CASE = "improved-transparency-minecraft-26.3"
WIRE_EVIDENCE_CASE = "OpenRA"
CONTROL_CASE = "OpenRA"
CONTROL_BACKEND = "DirectGLES"
CONTROL_NAME = "OpenRA (corrupted-snapshot control)"
# The comparator runs the long cases several times slower than a plain replay, and a spawned
# server can take as long, so the verify and split lanes raise ctest's 1500 s default.
LONG_TIMEOUT = ("--timeout", "10800")
SPLIT_REUSE_EXPECT_IPC = "strict=0 role-split-state=1 run-ahead=1"
SPLIT_REUSE_EXPECT_IPC_REASON = (
"retrace-split dropped its inproc rows because the inproc-acceptance retrace rows replay the "
"same library with strict=1 role-split-state=1 run-ahead=1, which differs from this leg's "
"defaults only in strict while these hold. A default changed: restore the inproc rows (the "
"trace-cases job's split-spawn-shards without --transports spawn) or update the expectation.")
# GitHub shows about ten error annotations of a step. The first failures are annotated as they
# happen, so a shard killed later still shows them; the rest wait for the roll-up that names
# every failed case.
IMMEDIATE_ANNOTATIONS = 8
def test_regex(case, backend):
return rf"^MobileGLTraceReplay\.{case}\.{backend}$"
def test_name(case, backend):
return f"MobileGLTraceReplay.{case}.{backend}"
def case_plan(profile, transport, backend, case, *, split_reuse=False, server_path=""):
"""What one case of a lane runs with: the environment it adds, ctest's arguments, its checks.
retrace the monolith-control and inproc-acceptance lanes of the retrace job. The job itself
exports MOBILEGL_TRANSPORT and the three IPC knobs for every case alike.
split the retrace-split job, inproc and spawn.
verify the retrace-verify job; its library has no transport.
"""
if profile not in ("retrace", "split", "verify"):
raise ValueError(f"unknown profile: {profile}")
env = {}
if backend == "DirectVulkan":
env["MOBILEGL_MAGMA_R11G11B10F_FALLBACK"] = "1"
# The blended depth-write quirk enables itself only on Qualcomm, which no runner has,
# so the case it exists for forces it on; ForceOn bypasses only the vendor gate.
if case == BLENDED_DEPTH_WRITE_CASE:
env["MOBILEGL_MAGMA_DISABLE_BLENDED_DEPTH_WRITE"] = "1"
# The iterationRP repairs default to device detection and the software driver is on no
# list, so the one case that needs them forces them. The verify lane does not set it.
if case == ITERATIONRP_CASE and profile != "verify":
env["MOBILEGL_ITRP_FIX"] = "ForceOn"
ctest = ["-V", "--no-tests=error"]
evidence = None
census = False
if profile == "retrace":
if transport == "inproc":
# These rows stand in for retrace-split's dropped inproc rows, so they prove the
# configuration they ran with from the client's own log.
env["MGTRACE_EXPECT_IPC"] = "strict=1 role-split-state=1 run-ahead=1"
census = True
if transport != "monolith":
evidence = []
elif profile == "split":
if transport not in ("inproc", "spawn"):
raise ValueError(f"the split profile runs inproc or spawn, not {transport!r}")
if not server_path.startswith("/"):
# The replay's working directory is not the workspace and the launcher resolves the
# image relative to nothing. The inproc arm reads it only to decide it needs none.
raise ValueError("the split profile needs an absolute --server-path")
env["MOBILEGL_TRANSPORT"] = transport
env["MOBILEGL_IPC_SERVER_PATH"] = server_path
evidence = []
if case == WIRE_EVIDENCE_CASE:
# This case also counts the wire's draw records and the server's draws. That turns
# on the library's per-frame counters, so the others keep the plain configuration.
env["MGTRACE_WIRE_EVIDENCE"] = "1"
evidence = ["--require-wire"]
if split_reuse:
env["MGTRACE_EXPECT_IPC"] = SPLIT_REUSE_EXPECT_IPC
env["MGTRACE_EXPECT_IPC_REASON"] = SPLIT_REUSE_EXPECT_IPC_REASON
ctest += LONG_TIMEOUT
census = True
else:
env["MOBILEGL_PIPE_VERIFY"] = "1"
ctest += LONG_TIMEOUT
ctest += ["-R", test_regex(case, backend)]
return {"env": env, "ctest": ctest, "evidence": evidence, "census": census,
"control": has_control(profile, backend, case)}
def control_plan():
"""The verify lane's negative control: the same replay with one snapshot field corrupted.
It must fail, and for the comparator's reason. It adds only these two variables and raises
no core limit, since the failure is the expected outcome.
"""
return {"env": {"MOBILEGL_PIPE_VERIFY": "1",
"MOBILEGL_PIPE_VERIFY_CORRUPT": "GetRenderStateParameters"},
"ctest": ["-V", "--no-tests=error", *LONG_TIMEOUT,
"-R", test_regex(CONTROL_CASE, CONTROL_BACKEND)]}
def has_control(profile, backend, case):
return profile == "verify" and case == CONTROL_CASE and backend == CONTROL_BACKEND
def expected_results(profile, backend, names):
"""The verdicts a shard owes, in order."""
out = []
for name in names:
out.append(name)
if has_control(profile, backend, name):
out.append(CONTROL_NAME)
return out
def step_ignores_sigpipe():
"""Whether the process that started this script, the step's shell, ignores SIGPIPE."""
try:
with open(f"/proc/{os.getppid()}/status", encoding="ascii") as status:
for line in status:
if line.startswith("SigIgn:"):
return bool(int(line.split()[1], 16) & (1 << (signal.SIGPIPE - 1)))
except (OSError, ValueError, IndexError):
pass
return False
def raise_core_limit():
# What `ulimit -c unlimited` does; the runner's core handler honors each process's limit.
import resource
resource.setrlimit(resource.RLIMIT_CORE, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))
class Shard:
def __init__(self, args, names, base_env):
self.args = args
self.names = names
self.base_env = dict(base_env)
self.build_dir = Path(args.build_dir).resolve()
self.result_dir = Path(args.result_dir).resolve()
self.results_root = self.result_dir / "results"
self.junit_dir = self.result_dir / "junit"
self.log_dir = self.result_dir / "logs"
self.cache_root = self.result_dir / "driver-cache"
self.records = []
self.failures = 0
self.child = None
self.lane = args.lane or "-"
self.transport = args.transport or "-"
self.where = f"{self.lane} / {self.transport} / {args.backend}"
# --- output -------------------------------------------------------------------------------
def say(self, text=""):
print(text, flush=True)
def summary(self, text):
path = os.environ.get("GITHUB_STEP_SUMMARY")
if path:
with open(path, "a", encoding="utf-8") as file:
file.write(text + "\n")
def annotate(self, title, message):
escaped = message.replace("%", "%25").replace("\r", "").replace("\n", "%0A")
# A property value also escapes the two characters that delimit properties.
title = title.replace("%", "%25").replace(":", "%3A").replace(",", "%2C")
self.say(f"::error title={title}::{escaped}")
def write_results(self):
self.results_root.mkdir(parents=True, exist_ok=True)
payload = {"profile": self.args.profile, "lane": self.args.lane, "transport": self.args.transport,
"backend": self.args.backend, "shard": self.args.shard, "cases": self.names,
"results": self.records}
target = self.results_root / "shard-results.json"
scratch = target.with_suffix(".tmp")
scratch.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
scratch.replace(target)
def record(self, name, status, seconds, reasons, *, ssim="", cores=()):
"""One verdict: "passed", "failed", or "skipped" for a control whose case already failed."""
entry = {"case": name, "status": status, "seconds": round(seconds, 1), "ssim": ssim,
"reasons": list(reasons), "cores": list(cores), "annotated": False}
if status == "failed":
self.failures += 1
entry["annotated"] = self.failures <= IMMEDIATE_ANNOTATIONS
self.records.append(entry)
self.write_results()
note = "; ".join(reasons)
if cores:
note = (note + "; " if note else "") + "core: " + ", ".join(cores)
shown = "**failed**" if status == "failed" else status
self.summary(f"| {name} | {shown} | {seconds:.0f} | {ssim} | {note.replace('|', '/')[:300]} |")
if entry["annotated"]:
self.annotate(f"retrace failed: {self.where} / {name}", note or "failed")
# --- one process ----------------------------------------------------------------------------
def run_ctest(self, plan, *, junit=None, transcript=None, core_limit=True, cache_home=None):
env = dict(self.base_env)
env.update(plan["env"])
if cache_home is not None:
env["XDG_CACHE_HOME"] = str(cache_home)
command = [self.args.ctest, *plan["ctest"]]
if junit is not None:
command += ["--output-junit", str(junit)]
self.say("+ " + " ".join(f"{key}={value!r}" for key, value in sorted(plan["env"].items())))
self.say("+ " + " ".join(command))
options = {"cwd": self.build_dir, "env": env, "start_new_session": True}
if core_limit:
options["preexec_fn"] = raise_core_limit
# Python ignores SIGPIPE itself and by default hands a child the default action. The
# replay gets what a step's own child would get instead, so a write to a closed pipe
# or socket ends the same way as when the step ran ctest directly.
options["restore_signals"] = not step_ignores_sigpipe()
sys.stdout.flush()
if transcript is None:
self.child = subprocess.Popen(command, **options)
status = self.child.wait()
else:
# The transcript is both shown and kept, as `2>&1 | tee` would.
self.child = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **options)
with open(transcript, "wb") as file:
for chunk in iter(lambda: self.child.stdout.read1(65536), b""):
file.write(chunk)
sys.stdout.buffer.write(chunk)
sys.stdout.buffer.flush()
status = self.child.wait()
self.reap()
return status
def reap(self, grace=5.0):
"""End whatever the finished replay left running, so it cannot run under the next case.
A spawned server exits by itself when its client is gone, so the group gets a moment
first. True when something had to be killed.
"""
child, self.child = self.child, None
if child is None:
return False
deadline = time.monotonic() + grace
try:
while True:
os.killpg(child.pid, 0)
if time.monotonic() >= deadline:
break
time.sleep(0.25)
os.killpg(child.pid, signal.SIGKILL)
except (ProcessLookupError, PermissionError):
return False
self.say(f"killed the processes ctest (pid {child.pid}) left running")
return True
# --- one case -------------------------------------------------------------------------------
def fresh_cache_home(self):
"""An empty driver cache directory, as on a runner that has replayed nothing yet.
The software drivers keep compiled shaders on disk under the user cache directory. Each
case gets a new one, so a case never starts from shaders another case compiled. On a
hosted runner the default location is emptied as well, in case a process ignores the
redirect; a self-hosted machine's own cache is left alone.
"""
self.cache_root.mkdir(parents=True, exist_ok=True)
cache_home = Path(tempfile.mkdtemp(prefix="case-", dir=self.cache_root))
if self.base_env.get("RUNNER_ENVIRONMENT") != "self-hosted":
default_home = Path(self.base_env.get("HOME", "/nonexistent")) / ".cache"
for stale in default_home.glob("mesa_shader_cache*"):
self.say(f"removing {stale} left by an earlier process")
shutil.rmtree(stale, ignore_errors=True)
return cache_home
def drop_cache_home(self, cache_home):
entries = sum(1 for path in cache_home.rglob("*") if path.is_file())
self.say(f"driver cache: {entries} file(s) written under {cache_home}, removed")
shutil.rmtree(cache_home, ignore_errors=True)
def junit_problem(self, junit, name):
"""Why this JUnit file does not show exactly the one entry that was asked for, or ''."""
if not junit.is_file():
return f"ctest wrote no {junit.name}"
try:
entries = [case.get("name") for case in ET.parse(junit).getroot().iter("testcase")]
except ET.ParseError as error:
return f"{junit.name} does not parse: {error}"
if entries != [name]:
return f"ctest ran {entries or 'no entry'} where exactly {name} was asked for"
return ""
def check_evidence(self, plan, case):
"""Run the lane's evidence check; its error lines are folded into the case's annotation."""
evidence = self.build_dir / case / self.args.backend / "output" / "split-evidence.txt"
command = [sys.executable, str(SCRIPTS / "retrace_split_evidence.py"), str(evidence),
"--transport", self.args.transport, "--case", case, "--backend", self.args.backend,
*plan["evidence"]]
done = subprocess.run(command, cwd=self.build_dir, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True)
text = done.stdout.strip()
self.say(re.sub(r"^::error::", "split evidence: ", text, flags=re.M))
if done.returncode == 0:
return ""
return "split evidence: " + re.sub(r"^::error::", "", text, flags=re.M).replace("\n", " ")[:600]
def census(self, case):
"""The refusal census from the library's own logs, recorded and never a verdict.
Both roles are read: under spawn the applier's refusals are raised in the server's
process. ctest's transcript cannot stand in, since the console sink is compiled out.
"""
output = self.build_dir / case / self.args.backend / "output"
log = output / "mobilegl.client.log"
server_log = output / "mobilegl.server.log"
evidence = output / "split-evidence.txt"
self.say(evidence.read_text(errors="replace").rstrip() if evidence.is_file() else "no split-evidence.txt")
if not log.is_file():
self.say(f"no {log} - the replay wrote no library log")
return
files = [log] + ([server_log] if server_log.is_file() else [])
self.say("census over: " + " ".join(str(path) for path in files))
lines = [line for path in files for line in path.read_text(errors="replace").splitlines()]
self.say(f"Fatal{{ lines: {sum(1 for line in lines if 'Fatal{' in line)}")
counts = {}
for line in lines:
for match in re.findall(r'Fatal\{[A-Za-z]*, "[^"]*"', line):
counts[match] = counts.get(match, 0) + 1
for match, count in sorted(counts.items(), key=lambda item: (-item[1], item[0]))[:20]:
self.say(f"{count:7d} {match}")
transport_line = next((line for line in log.read_text(errors="replace").splitlines()
if "MOBILEGL_TRANSPORT=" in line), None)
self.say(transport_line or f"no transport line in {log}")
def collect(self, case):
"""Move what the lane uploads into the result directory and delete the extracted trace."""
source = self.build_dir / case
target = self.results_root / case
shutil.rmtree(source / self.args.backend / "input", ignore_errors=True)
for relative in ("actual-images", f"{self.args.backend}/output"):
if (source / relative).is_dir():
(target / relative).parent.mkdir(parents=True, exist_ok=True)
shutil.rmtree(target / relative, ignore_errors=True)
shutil.move(str(source / relative), str(target / relative))
shutil.rmtree(source, ignore_errors=True)
def ssim_of(self, case):
try:
result = json.loads((self.build_dir / case / self.args.backend / "output" / "result.json")
.read_text(errors="replace"))
except (OSError, ValueError):
return ""
value = result.get("ssim")
return "" if value is None else str(value)
def run_case(self, case):
plan = case_plan(self.args.profile, self.args.transport, self.args.backend, case,
split_reuse=self.args.split_reuse, server_path=self.args.server_path)
self.say(f"::group::{case} ({self.where})")
started = time.monotonic()
cores_before = set(Path("/tmp").glob("core.*"))
junit = self.junit_dir / f"{case}.xml"
reasons = []
cache_home = self.fresh_cache_home()
try:
try:
status = self.run_ctest(plan, junit=junit, cache_home=cache_home)
except (OSError, subprocess.SubprocessError) as error:
# The case still gets its verdict, and the cases after it still run.
self.child = None
status = None
reasons.append(f"ctest could not be run: {error}")
if status is not None and status > 0:
reasons.append(f"ctest exit {status}")
elif status is not None and status < 0:
reasons.append(f"ctest ended by signal {-status}")
problem = self.junit_problem(junit, test_name(case, self.args.backend))
if problem:
reasons.append(problem)
if plan["evidence"] is not None:
# The transport assertions ran on this case and found their evidence. A failed
# replay is already red, and its missing evidence file would only repeat that.
if status == 0:
problem = self.check_evidence(plan, case)
if problem:
reasons.append(problem)
if plan["census"]:
try:
self.census(case)
except OSError as error:
self.say(f"refusal census skipped: {error}")
finally:
self.drop_cache_home(cache_home)
ssim = self.ssim_of(case)
self.collect(case)
cores = sorted(path.name for path in set(Path("/tmp").glob("core.*")) - cores_before)
seconds = time.monotonic() - started
self.say(f"{case}: {'passed' if not reasons else 'FAILED: ' + '; '.join(reasons)} in {seconds:.0f} s")
self.say("::endgroup::")
self.record(case, "failed" if reasons else "passed", seconds, reasons, ssim=ssim, cores=cores)
return not reasons
def run_control(self, verified_passed):
"""With a snapshot field corrupted the same replay must fail, or the lane proves nothing.
It replays into the case directory the verified run used. That run's output has already
been moved to the result directory, and the corrupted run's output is deleted, so only
the verified run is uploaded under the case's name.
"""
self.say(f"::group::{CONTROL_NAME}")
if not verified_passed:
self.say("not run: the verified replay it is compared with failed")
self.say("::endgroup::")
self.record(CONTROL_NAME, "skipped", 0, ["not run: the verified replay failed"])
return
started = time.monotonic()
reasons = []
transcript = self.log_dir / "verify-control.txt"
cache_home = self.fresh_cache_home()
status = None
try:
status = self.run_ctest(control_plan(), transcript=transcript, core_limit=False,
cache_home=cache_home)
except (OSError, subprocess.SubprocessError) as error:
self.child = None
reasons.append(f"ctest could not be run: {error}")
finally:
self.drop_cache_home(cache_home)
shutil.rmtree(self.build_dir / CONTROL_CASE, ignore_errors=True)
if status is None:
pass
elif status == 0:
reasons.append(
"MOBILEGL_PIPE_VERIFY_CORRUPT=GetRenderStateParameters left the OpenRA retrace "
"green, so the comparator is not comparing and the verify retrace lane proves nothing")
else:
# Its red must carry the comparator's divergence on the corrupted field.
done = subprocess.run([sys.executable, str(SCRIPTS / "retrace_verify_reason.py"), str(transcript)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
text = re.sub(r"^::error::", "", done.stdout.strip(), flags=re.M)
self.say(text)
if done.returncode != 0:
reasons.append(text.replace("\n", " ")[:600])
else:
self.say(f"the corrupted field turned the retrace red, as it must (ctest exit {status})")
seconds = time.monotonic() - started
self.say("::endgroup::")
self.record(CONTROL_NAME, "failed" if reasons else "passed", seconds, reasons)
# --- the shard --------------------------------------------------------------------------------
def run(self):
for directory in (self.results_root, self.junit_dir, self.log_dir):
directory.mkdir(parents=True, exist_ok=True)
expected = expected_results(self.args.profile, self.args.backend, self.names)
self.say(f"retrace shard {self.args.shard or ''} ({self.where}): {len(self.names)} case(s), "
f"{len(expected)} verdict(s) owed")
self.summary(f"### retrace shard {self.args.shard or ''} ({self.where})\n\n"
"| case | result | seconds | SSIM | note |\n|---|---|---|---|---|")
self.write_results()
for case in self.names:
passed = self.run_case(case)
if case_plan(self.args.profile, self.args.transport, self.args.backend, case,
split_reuse=self.args.split_reuse, server_path=self.args.server_path)["control"]:
self.run_control(passed)
return self.finish(expected)
def finish(self, expected):
problems = verdict_problems(expected, self.records)
# Counted again from what ctest wrote, apart from the loop's own bookkeeping.
ran = sum(1 for case in self.names
if not self.junit_problem(self.junit_dir / f"{case}.xml", test_name(case, self.args.backend)))
passed_cases = sum(1 for entry in self.records
if entry["status"] == "passed" and entry["case"] in self.names)
if ran < passed_cases:
problems.append(f"{passed_cases} case(s) recorded as passed but only {ran} JUnit file(s) "
f"show their entry ran")
failed = [entry for entry in self.records if entry["status"] == "failed"]
if failed:
names = ", ".join(entry["case"] for entry in failed)
self.annotate(f"retrace shard {self.args.shard or ''}: {len(failed)} of {len(expected)} failed",
f"{self.where}: {names}")
for entry in failed:
if not entry["annotated"]:
self.annotate(f"retrace failed: {self.where} / {entry['case']}",
"; ".join(entry["reasons"]) or "failed")
for problem in problems:
self.annotate(f"retrace shard {self.args.shard or ''}: verdicts do not match the cases", problem)
passed = sum(1 for entry in self.records if entry["status"] == "passed")
closing = f"{passed} passed, {len(failed)} failed, {len(expected)} owed"
if problems:
closing += "; " + "; ".join(problems)
self.summary(f"\n{closing}\n")
self.say(f"retrace shard ({self.where}): {closing}")
return 1 if failed or problems else 0
def verdict_problems(expected, records):
"""Why the recorded verdicts are not exactly the ones owed; empty when they are."""
recorded = [entry["case"] for entry in records]
problems = []
if len(recorded) != len(expected):
problems.append(f"{len(recorded)} verdict(s) recorded for {len(expected)} owed")
if recorded != expected:
missing = [name for name in expected if name not in recorded]
extra = [name for name in recorded if name not in expected]
if missing:
problems.append("no verdict for " + ", ".join(missing))
if extra:
problems.append("verdict for a case that was not assigned: " + ", ".join(extra))
if not missing and not extra and len(recorded) == len(expected):
problems.append("verdicts are out of order or repeated")
return problems
def run(args):
if args.names is not None:
names = json.loads(args.names)
else:
names = json.loads(os.environ.get(args.names_env) or "null")
if not isinstance(names, list) or not names or not all(isinstance(name, str) and name for name in names):
print(f"::error::retrace shard: no case list (looked in --names and ${args.names_env})")
return 2
if len(set(names)) != len(names):
print(f"::error::retrace shard: the case list repeats a case: {names}")
return 2
if args.profile == "retrace":
# The job exports the transport for this lane; the plan and the library must agree on it.
exported = os.environ.get("MOBILEGL_TRANSPORT", "")
if exported != args.transport:
print(f"::error::retrace shard: --transport is {args.transport!r} but the job exports "
f"MOBILEGL_TRANSPORT={exported!r}")
return 2
try:
case_plan(args.profile, args.transport, args.backend, names[0],
split_reuse=args.split_reuse, server_path=args.server_path)
except ValueError as error:
print(f"::error::retrace shard: {error}")
return 2
shard = Shard(args, names, os.environ)
def stop(signum, _frame):
shard.reap(grace=0)
shard.say(f"retrace shard: stopped by signal {signum} after {len(shard.records)} verdict(s)")
sys.exit(128 + signum)
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
return shard.run()
def self_test():
"""The plan of every lane against the values the lanes are defined by, and the verdict check."""
oit, itrp, other = BLENDED_DEPTH_WRITE_CASE, ITERATIONRP_CASE, "minecraft-1.21.4-in-world"
server = "/w/build-linux/libMobileGLServer.so"
regex = lambda case, backend: ["-R", rf"^MobileGLTraceReplay\.{case}\.{backend}$"]
plain, long = ["-V", "--no-tests=error"], ["-V", "--no-tests=error", "--timeout", "10800"]
vk = {"MOBILEGL_MAGMA_R11G11B10F_FALLBACK": "1"}
expect_strict = {"MGTRACE_EXPECT_IPC": "strict=1 role-split-state=1 run-ahead=1"}
reuse = {"MGTRACE_EXPECT_IPC": SPLIT_REUSE_EXPECT_IPC, "MGTRACE_EXPECT_IPC_REASON": SPLIT_REUSE_EXPECT_IPC_REASON}
split = lambda transport: {"MOBILEGL_TRANSPORT": transport, "MOBILEGL_IPC_SERVER_PATH": server}
table = [
# profile, transport, backend, case, reuse -> env, ctest, evidence, census, control
("retrace", "monolith", "DirectGLES", other, False, {}, plain, None, False, False),
("retrace", "monolith", "DirectVulkan", other, False, vk, plain, None, False, False),
("retrace", "monolith", "DirectVulkan", oit, False,
{**vk, "MOBILEGL_MAGMA_DISABLE_BLENDED_DEPTH_WRITE": "1"}, plain, None, False, False),
("retrace", "monolith", "DirectGLES", oit, False, {}, plain, None, False, False),
("retrace", "monolith", "DirectVulkan", itrp, False, {**vk, "MOBILEGL_ITRP_FIX": "ForceOn"},
plain, None, False, False),
("retrace", "inproc", "DirectGLES", other, False, expect_strict, plain, [], True, False),
("retrace", "inproc", "DirectVulkan", itrp, False,
{**vk, "MOBILEGL_ITRP_FIX": "ForceOn", **expect_strict}, plain, [], True, False),
("retrace", "inproc", "DirectGLES", "OpenRA", False, expect_strict, plain, [], True, False),
("split", "spawn", "DirectGLES", other, True, {**split("spawn"), **reuse}, long, [], True, False),
("split", "spawn", "DirectGLES", other, False, split("spawn"), long, [], True, False),
("split", "spawn", "DirectVulkan", oit, True,
{**vk, "MOBILEGL_MAGMA_DISABLE_BLENDED_DEPTH_WRITE": "1", **split("spawn"), **reuse},
long, [], True, False),
("split", "spawn", "DirectVulkan", itrp, True,
{**vk, "MOBILEGL_ITRP_FIX": "ForceOn", **split("spawn"), **reuse}, long, [], True, False),
("split", "spawn", "DirectGLES", "OpenRA", True,
{**split("spawn"), "MGTRACE_WIRE_EVIDENCE": "1", **reuse}, long, ["--require-wire"], True, False),
("split", "inproc", "DirectVulkan", "OpenRA", True,
{**vk, **split("inproc"), "MGTRACE_WIRE_EVIDENCE": "1", **reuse}, long, ["--require-wire"], True, False),
("split", "inproc", "DirectGLES", other, False, split("inproc"), long, [], True, False),
("verify", "", "DirectGLES", other, False, {"MOBILEGL_PIPE_VERIFY": "1"}, long, None, False, False),
("verify", "", "DirectGLES", "OpenRA", False, {"MOBILEGL_PIPE_VERIFY": "1"}, long, None, False, True),
("verify", "", "DirectVulkan", "OpenRA", False, {"MOBILEGL_PIPE_VERIFY": "1", **vk}, long, None, False, False),
("verify", "", "DirectVulkan", oit, False,
{"MOBILEGL_PIPE_VERIFY": "1", **vk, "MOBILEGL_MAGMA_DISABLE_BLENDED_DEPTH_WRITE": "1"},
long, None, False, False),
("verify", "", "DirectVulkan", itrp, False, {"MOBILEGL_PIPE_VERIFY": "1", **vk}, long, None, False, False),
]
for profile, transport, backend, case, split_reuse, env, ctest, evidence, census, control in table:
plan = case_plan(profile, transport, backend, case, split_reuse=split_reuse, server_path=server)
wanted = {"env": env, "ctest": ctest + regex(case, backend), "evidence": evidence,
"census": census, "control": control}
assert plan == wanted, f"{profile} {transport} {backend} {case}: {plan} != {wanted}"
control = control_plan()
assert control["env"] == {"MOBILEGL_PIPE_VERIFY": "1", "MOBILEGL_PIPE_VERIFY_CORRUPT": "GetRenderStateParameters"}
assert control["ctest"] == long + regex("OpenRA", "DirectGLES")
for bad in (("split", "spawn", ""), ("split", "spawn", "relative/server"), ("split", "monolith", server),
("pull", "monolith", server)):
try:
case_plan(bad[0], bad[1], "DirectGLES", other, server_path=bad[2])
except ValueError:
continue
raise AssertionError(f"case_plan accepted {bad}")
assert expected_results("verify", "DirectGLES", ["a", "OpenRA", "b"]) == ["a", "OpenRA", CONTROL_NAME, "b"]
assert expected_results("verify", "DirectVulkan", ["OpenRA"]) == ["OpenRA"]
assert expected_results("split", "DirectGLES", ["OpenRA"]) == ["OpenRA"]
entry = lambda name: {"case": name, "status": "passed"}
assert verdict_problems(["a", "b"], [entry("a"), entry("b")]) == []
assert verdict_problems(["a", "b"], [entry("a")]), "a case without a verdict passed the check"
assert verdict_problems(["a", "b"], []), "a shard that ran nothing passed the check"
assert verdict_problems(["a"], [entry("a"), entry("a")]), "a repeated verdict passed the check"
assert verdict_problems(["a", "b"], [entry("a"), entry("c")]), "an unassigned case passed the check"
assert verdict_problems(["a", "b"], [entry("b"), entry("a")]), "out-of-order verdicts passed the check"
print(f"retrace_shard: {len(table)} plans, the control plan and the verdict check OK")
return 0
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("command", nargs="?", choices=("run",))
parser.add_argument("--self-test", action="store_true")
parser.add_argument("--profile", choices=("retrace", "split", "verify"))
parser.add_argument("--backend")
parser.add_argument("--transport", default="")
parser.add_argument("--lane", default="")
parser.add_argument("--shard", default="", help="N/M, for the report only")
parser.add_argument("--build-dir", help="the directory ctest runs in")
parser.add_argument("--result-dir", help="where verdicts, replay output and JUnit files go")
parser.add_argument("--names", help="the cases, as a JSON list")
parser.add_argument("--names-env", default="SHARD_NAMES", help="the variable holding the JSON list")
parser.add_argument("--split-reuse", action="store_true",
help="split profile: the library is the retrace job's, so prove the IPC defaults")
parser.add_argument("--server-path", default="", help="split profile: absolute path of the server image")
parser.add_argument("--ctest", default="ctest")
args = parser.parse_args(argv)
if args.self_test:
return self_test()
if args.command != "run" or not (args.profile and args.backend and args.build_dir and args.result_dir):
parser.error("run needs --profile, --backend, --build-dir and --result-dir")
return run(args)
if __name__ == "__main__":
sys.exit(main())
+120
View File
@@ -0,0 +1,120 @@
#!/bin/bash
# MobileGL - scripts/ci/set_swap_space.sh
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
# Replaces the runner's swap with one swap file of SWAP_SIZE_GB GiB (.github/actions/set-swap-space).
# Hosted images differ: some ship a /swapfile, some no swap, and a swap partition or zram device is
# possible. Every active swap area is turned off, but only a swap file is replaced in place; with
# none the file is /swapfile, or /mnt/swapfile when that file system has more room. The file is
# sized down to what fits with SWAP_MARGIN_MIB to spare, and the step fails only below 1 GiB.
# df only informs the size: when it reports nothing usable, the requested size is tried as is.
#
# usage: set_swap_space.sh (SWAP_SIZE_GB from the action input; SWAP_MARGIN_MIB and
# SWAP_ALT_DIR, the /mnt fallback directory, are optional)
set -euo pipefail
size_gb="${SWAP_SIZE_GB:-}"
margin_mib="${SWAP_MARGIN_MIB:-2048}"
alt_dir="${SWAP_ALT_DIR:-/mnt}"
floor_gb=1
case "${size_gb}" in
'' | *[!0-9]* | 0)
echo "::error::swap-size-gb must be a positive whole number of GiB, got '${size_gb}'"
exit 1
;;
esac
# Prints the MiB available on the file system that holds $1, or nothing when df fails or prints
# something else.
avail_mib() {
local value
value="$(df --output=avail -B1M "$1" 2> /dev/null | tail -n 1 | tr -d ' ' || true)"
case "${value}" in
'' | *[!0-9]*) ;;
*) printf '%s\n' "${value}" ;;
esac
}
show_state() {
echo "Swap $1:"
swapon --show || true
free -h || true
}
show_state before
swap_file=""
swap_names=()
while read -r name type _; do
[ -n "${name}" ] || continue
swap_names+=("${name}")
if [ "${type}" = file ]; then
swap_file="${name}"
fi
done < <(swapon --show=NAME,TYPE --noheadings)
swap_file="${swap_file:-/swapfile}"
for name in "${swap_names[@]}"; do
sudo swapoff "${name}"
done
sudo rm -f "${swap_file}"
want_mib=$((size_gb * 1024))
swap_dir="$(dirname "${swap_file}")"
avail="$(avail_mib "${swap_dir}")"
if [ -z "${avail}" ]; then
echo "df did not report the space available on ${swap_dir}; trying the requested ${size_gb} GiB"
elif [ "${avail}" -lt $((want_mib + margin_mib)) ] && [ "${swap_dir}" != "${alt_dir}" ] &&
[ -d "${alt_dir}" ]; then
alt_avail="$(avail_mib "${alt_dir}")"
if [ -n "${alt_avail}" ] && [ "${alt_avail}" -gt "${avail}" ]; then
echo "${swap_dir} has ${avail} MiB available and ${alt_dir} has ${alt_avail} MiB; using ${alt_dir}/swapfile"
swap_file="${alt_dir}/swapfile"
swap_dir="${alt_dir}"
avail="${alt_avail}"
sudo rm -f "${swap_file}"
fi
fi
df -h "${swap_dir}" || true
if [ -n "${avail}" ] && [ "${avail}" -lt $((want_mib + margin_mib)) ]; then
# df only shrinks the file; whether even the floor fits is left to fallocate, which may use
# blocks df does not count as available.
fit_gb=$(((avail - margin_mib) / 1024))
if [ "${fit_gb}" -lt "${floor_gb}" ]; then
fit_gb="${floor_gb}"
fi
echo "::warning::${swap_dir} has ${avail} MiB available; the swap file is ${fit_gb} GiB instead of the requested ${size_gb} GiB"
size_gb="${fit_gb}"
fi
# A refusal that df did not predict halves the size down to the floor, printing what the file
# system looked like each time.
df_note="df reported ${avail} MiB available"
if [ -z "${avail}" ]; then
df_note="df did not report the available space"
fi
while ! sudo fallocate -l "${size_gb}G" "${swap_file}"; do
sudo rm -f "${swap_file}"
df -h "${swap_dir}" || true
findmnt -T "${swap_dir}" || true
lsblk || true
if [ "${size_gb}" -le "${floor_gb}" ]; then
echo "::error::fallocate of ${size_gb} GiB at ${swap_file} failed; ${df_note}"
exit 1
fi
next_gb=$((size_gb / 2 > floor_gb ? size_gb / 2 : floor_gb))
echo "::warning::fallocate of ${size_gb} GiB at ${swap_file} failed; ${df_note}; trying ${next_gb} GiB"
size_gb="${next_gb}"
done
sudo chmod 600 "${swap_file}"
sudo mkswap "${swap_file}"
sudo swapon "${swap_file}"
show_state after
df -h "${swap_dir}" || true
+148
View File
@@ -0,0 +1,148 @@
#!/bin/bash
# MobileGL - scripts/ci/test_set_swap_space.sh
# Copyright (c) 2025-2026 MobileGL-Dev
# Licensed under the GNU Lesser General Public License v3.0:
# https://www.gnu.org/licenses/gpl-3.0.txt
# https://www.gnu.org/licenses/lgpl-3.0.txt
# SPDX-License-Identifier: LGPL-3.0-only
# End of Source File Header
# Runs scripts/ci/set_swap_space.sh against stubbed sudo (which logs swapoff, fallocate and the
# other privileged commands instead of running them), swapon, df, free, findmnt and lsblk, so no
# case touches the host's swap. STUB_SWAPS lists the active swap areas as "NAME TYPE" lines and
# STUB_FS the file systems as "mount=available-MiB" words; fallocate fails with ENOSPC when the
# size does not fit, as a small devtmpfs on /dev does, or exceeds STUB_FALLOCATE_MAX_GB, and
# STUB_DF=fail or STUB_DF=garbage makes df fail or print no number. The /mnt fallback is a
# temporary directory, so the result does not depend on the host's /mnt.
# Each case checks the exit status and the privileged commands the script ran.
#
# usage: test_set_swap_space.sh [script] (defaults to set_swap_space.sh next to this file)
set -euo pipefail
script="${1:-$(cd "$(dirname "$0")" && pwd)/set_swap_space.sh}"
stubs="$(mktemp -d)"
trap 'rm -rf "${stubs}"' EXIT
cat > "${stubs}/stub-lib" << 'EOF'
mount_of() {
local path="$1" best="" mount
for entry in ${STUB_FS}; do
mount="${entry%%=*}"
case "${path}" in
"${mount}" | "${mount%/}"/*)
if [ "${#mount}" -gt "${#best}" ]; then
best="${mount}"
fi
;;
esac
done
printf '%s\n' "${best}"
}
avail_of() {
local mount entry
mount="$(mount_of "$1")"
for entry in ${STUB_FS}; do
if [ "${entry%%=*}" = "${mount}" ]; then
printf '%s\n' "${entry#*=}"
fi
done
}
EOF
stub() {
printf '#!/bin/bash\n. "%s/stub-lib"\n%s\n' "${stubs}" "$2" > "${stubs}/$1"
chmod +x "${stubs}/$1"
}
stub sudo 'echo "$*" >> "${STUB_LOG}"
if [ "$1" = fallocate ]; then
want="${3%G}"
if [ $((want * 1024)) -gt "$(avail_of "$(dirname "$4")")" ] ||
[ "${want}" -gt "${STUB_FALLOCATE_MAX_GB:-${want}}" ]; then
echo "fallocate: fallocate failed: No space left on device" >&2
exit 1
fi
fi'
stub swapon 'if [ "$1" = --show=NAME,TYPE ] && [ "${2:-}" = --noheadings ]; then
printf "%s\n" "${STUB_SWAPS}" | sed "/^$/d"
elif [ "$1" = --show=NAME ] && [ "${2:-}" = --noheadings ]; then
printf "%s\n" "${STUB_SWAPS}" | sed "/^$/d" | cut -d" " -f1
else
printf "%s\n" "${STUB_SWAPS}"
fi'
stub df 'path="${*: -1}"
if [ "${STUB_DF:-}" = fail ]; then
echo "df: stub failure" >&2
exit 1
elif [ "${STUB_DF:-}" = garbage ]; then
echo "Avail"
echo "-"
elif [ "$1" = --output=avail ]; then
echo "1M-blocks"
avail_of "${path}"
else
echo "Filesystem Avail Mounted on"
echo "stub $(avail_of "${path}")M $(mount_of "${path}")"
fi'
stub free 'echo "Mem: stub"'
stub findmnt 'true'
stub lsblk 'true'
failures=0
# expect <case> <exit> <expected sudo log, ";"-separated> <swaps> <file systems> [size]
expect() {
local name="$1" want_exit="$2" want_log="$3" got_exit=0 got_log
: > "${stubs}/log"
PATH="${stubs}:${PATH}" STUB_LOG="${stubs}/log" STUB_SWAPS="$4" STUB_FS="$5" \
SWAP_SIZE_GB="${6:-8}" SWAP_ALT_DIR="${SWAP_ALT_DIR:-${mnt}}" \
bash -euo pipefail "${script}" > "${stubs}/out" 2>&1 || got_exit=$?
got_log="$(paste -sd ';' "${stubs}/log")"
if [ "${got_exit}" -ne "${want_exit}" ] || [ "${got_log}" != "${want_log}" ]; then
echo "NOT OK ${name}: exit ${got_exit} (expected ${want_exit})"
echo " ran: ${got_log}"
echo " expected: ${want_log}"
sed 's/^/ | /' "${stubs}/out"
failures=$((failures + 1))
fi
}
# The /mnt fallback directory; ${mnt}/missing stands for a host without /mnt.
mnt="${stubs}/mnt"
mkdir "${mnt}"
roomy="/=93000 /dev=4 ${mnt}=70000"
replace() {
echo "rm -f $1;fallocate -l $2G $1;chmod 600 $1;mkswap $1;swapon $1"
}
expect "image swap file" 0 "swapoff /swapfile;$(replace /swapfile 8)" "/swapfile file" "${roomy}"
expect "no swap" 0 "$(replace /swapfile 32)" "" "${roomy}" 32
expect "swap file on /mnt" 0 "swapoff ${mnt}/swapfile;$(replace "${mnt}/swapfile" 8)" \
"${mnt}/swapfile file" "${roomy}"
expect "zram device" 0 "swapoff /dev/zram0;$(replace /swapfile 8)" "/dev/zram0 partition" "${roomy}"
expect "swap partition and file" 0 "swapoff /dev/sdb2;swapoff /swapfile;$(replace /swapfile 8)" \
"/dev/sdb2 partition
/swapfile file" "${roomy}"
expect "short root, roomy /mnt" 0 "rm -f /swapfile;$(replace "${mnt}/swapfile" 16)" \
"" "/=6000 /dev=4 ${mnt}=70000" 16
expect "short root, no /mnt file system" 0 "$(replace /swapfile 3)" "" "/=6000 /dev=4" 16
SWAP_ALT_DIR="${mnt}/missing" expect "short root, no /mnt directory" 0 "$(replace /swapfile 3)" \
"" "/=6000 /dev=4 ${mnt}/missing=70000" 16
expect "room for the floor only" 0 "$(replace /swapfile 1)" "" "/=2500 /dev=4" 8
expect "no room anywhere" 1 "rm -f /swapfile;fallocate -l 1G /swapfile;rm -f /swapfile" "" \
"/=900 /dev=4" 8
expect "size not a number" 1 "" "" "${roomy}" 8G
STUB_DF=fail expect "df fails" 0 "swapoff /swapfile;$(replace /swapfile 8)" "/swapfile file" "${roomy}"
STUB_DF=garbage expect "df prints no number" 0 "$(replace /swapfile 8)" "" "${roomy}"
STUB_DF=fail STUB_FALLOCATE_MAX_GB=4 expect "df fails and fallocate refuses" 0 \
"rm -f /swapfile;fallocate -l 8G /swapfile;$(replace /swapfile 4)" "" "${roomy}"
STUB_FALLOCATE_MAX_GB=2 expect "fallocate refuses although df has room" 0 \
"rm -f /swapfile;fallocate -l 8G /swapfile;rm -f /swapfile;fallocate -l 4G /swapfile;$(replace /swapfile 2)" \
"" "${roomy}"
STUB_FALLOCATE_MAX_GB=0 expect "fallocate refuses every size" 1 \
"rm -f /swapfile;fallocate -l 2G /swapfile;rm -f /swapfile;fallocate -l 1G /swapfile;rm -f /swapfile" \
"" "${roomy}" 2
if [ "${failures}" -ne 0 ]; then
echo "set_swap_space self-test: FAILED"
exit 1
fi
echo "set_swap_space self-test: ok"
+2 -1
View File
@@ -48,7 +48,8 @@ public final class SpawnServerPathTest {
public static void main(String[] args) { public static void main(String[] args) {
// ---- the shapes the lanes really pass ------------------------------------------------- // ---- the shapes the lanes really pass -------------------------------------------------
// .github/workflows/apk.yml's spawn acceptance lane, verbatim. // The spawn acceptance lane of android-plugin/trace_replay_shard.py (transport_arguments()),
// verbatim.
String spawnLane = "MOBILEGL_TRANSPORT=spawn;MOBILEGL_IPC_ROLE_SPLIT_STATE=1;" String spawnLane = "MOBILEGL_TRANSPORT=spawn;MOBILEGL_IPC_ROLE_SPLIT_STATE=1;"
+ "MOBILEGL_IPC_STRICT_ERRORS=1;MOBILEGL_IPC_RUN_AHEAD=1"; + "MOBILEGL_IPC_STRICT_ERRORS=1;MOBILEGL_IPC_RUN_AHEAD=1";
expect("the spawn acceptance lane still gets its server path appended", expect("the spawn acceptance lane still gets its server path appended",
+10 -2
View File
@@ -16,14 +16,15 @@ const BACKENDS = ["DirectGLES", "DirectVulkan"];
const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url)); const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
const DEFAULT_FIXTURE_ROOT = path.join(SCRIPT_DIR, "fixtures"); const DEFAULT_FIXTURE_ROOT = path.join(SCRIPT_DIR, "fixtures");
const TRACE_CASES = JSON.parse(readFileSync(path.join(SCRIPT_DIR, "trace_cases.json"), "utf8")); const TRACE_CASES = JSON.parse(readFileSync(path.join(SCRIPT_DIR, "trace_cases.json"), "utf8"));
const CASES = TRACE_CASES.cases.map((traceCase) => traceCase.name); // Every case of the manifest, or the ones --cases names when a lane replays a subset.
let CASES = TRACE_CASES.cases.map((traceCase) => traceCase.name);
function usage() { function usage() {
console.error(`Usage: console.error(`Usage:
node tools/trace_replay/render_retrace_summary.mjs \\ node tools/trace_replay/render_retrace_summary.mjs \\
--input DIR [--input DIR ...] \\ --input DIR [--input DIR ...] \\
--output-dir DIR \\ --output-dir DIR \\
[--title TITLE] [--group-label LABEL] [--html NAME]`); [--title TITLE] [--group-label LABEL] [--html NAME] [--cases NAME,NAME,...]`);
} }
function parseArgs(argv) { function parseArgs(argv) {
@@ -52,6 +53,13 @@ function parseArgs(argv) {
} else if (arg === "--html" && value) { } else if (arg === "--html" && value) {
args.htmlName = value; args.htmlName = value;
i += 1; i += 1;
} else if (arg === "--cases" && value !== undefined) {
// An empty list keeps every case, so a caller can pass the option unconditionally.
const names = value.split(",").filter((name) => name);
const unknown = names.filter((name) => !CASES.includes(name));
if (unknown.length) throw new Error(`--cases names unknown trace case(s): ${unknown.join(", ")}`);
if (names.length) CASES = CASES.filter((name) => names.includes(name));
i += 1;
} else if (arg === "--help" || arg === "-h") { } else if (arg === "--help" || arg === "-h") {
usage(); usage();
process.exit(0); process.exit(0);
@@ -650,7 +650,8 @@ def transport_proof_args(args):
return {"inproc": ["--require-inproc"], "spawn": ["--require-spawn"]}.get(args.transport, []) return {"inproc": ["--require-inproc"], "spawn": ["--require-spawn"]}.get(args.transport, [])
# The knobs .github/workflows/apk.yml hands BOTH acceptance lanes. They are kept here verbatim so # The knobs android-plugin/trace_replay_shard.py (transport_arguments()) hands both acceptance
# lanes. They are kept here verbatim so
# a local run is the same run: the arm proof below reads the library's own `Config: IPC` line and # a local run is the same run: the arm proof below reads the library's own `Config: IPC` line and
# requires strict/role-split-state/run-ahead to be 1, so a local invocation that quietly omitted # requires strict/role-split-state/run-ahead to be 1, so a local invocation that quietly omitted
# them would fail a gate CI passes - or worse, pass a weaker one. # them would fail a gate CI passes - or worse, pass a weaker one.
+14
View File
@@ -1,4 +1,18 @@
{ {
"android_smoke": [
"OpenRA",
"minecraft-1.21.4-startup",
"minecraft-1.21.4-main-menu",
"minecraft-1.21.4-fabric-sodium-in-world",
"minecraft-1.21.4-fabric-iris-bsl-in-world",
"minecraft-1.21.4-fabric-iris-complementary-reimagined-in-world",
"minecraft-1.21.4-fabric-iris-bliss-in-world",
"minecraft-1.21.4-fabric-iris-sundial-lite-in-world",
"minecraft-1.21.1-neoforge-create-indirect-in-world-align1024",
"minecraft-1.21.1-neoforge-create-instancing-in-world",
"improved-transparency-minecraft-26.3",
"minecraft-1.21.4-fabric-iris-iterationrp-in-world"
],
"defaults": { "defaults": {
"trace_file": "trace.trace", "trace_file": "trace.trace",
"width": 854, "width": 854,
+415 -5
View File
@@ -14,6 +14,9 @@ from pathlib import Path
TRACE_CASES_JSON = Path(__file__).with_name("trace_cases.json") TRACE_CASES_JSON = Path(__file__).with_name("trace_cases.json")
# The top-level keys of the manifest. An unknown one is refused, as an unknown case key is.
KNOWN_MANIFEST_KEYS = frozenset({"defaults", "cases", "android_smoke"})
CI_BACKENDS = ("DirectGLES", "DirectVulkan") CI_BACKENDS = ("DirectGLES", "DirectVulkan")
# Every key a case or the defaults block may carry. An UNKNOWN key is a hard error rather than a # Every key a case or the defaults block may carry. An UNKNOWN key is a hard error rather than a
@@ -127,7 +130,26 @@ def load_trace_case_manifest(path=TRACE_CASES_JSON):
merged["split"] = split merged["split"] = split
validate_backend_overrides(merged) validate_backend_overrides(merged)
cases.append(merged) cases.append(merged)
return {"defaults": defaults, "cases": cases} unknown_top = sorted(set(manifest) - KNOWN_MANIFEST_KEYS)
if unknown_top:
raise ValueError(f"unknown top-level key(s) in the manifest: {', '.join(unknown_top)}")
# "android_smoke" names the cases the emulator replays on the monolith and spawn lanes
# when those lanes are reduced. A name that is not a CI case would shrink the subset with
# no red, so it is refused here.
smoke = manifest.get("android_smoke", [])
if not isinstance(smoke, list) or not all(isinstance(name, str) for name in smoke):
raise ValueError("android_smoke must be a list of case names")
if len(set(smoke)) != len(smoke):
raise ValueError("android_smoke repeats a case name")
ci_names = {case["name"] for case in cases if case.get("ci", True)}
not_ci = [name for name in smoke if name not in ci_names]
if not_ci:
raise ValueError(f"android_smoke names case(s) that are not CI cases: {', '.join(not_ci)}")
return {"defaults": defaults, "cases": cases, "android_smoke": smoke}
def android_smoke_names(path=TRACE_CASES_JSON):
return load_trace_case_manifest(path)["android_smoke"]
def validate_backend_overrides(case): def validate_backend_overrides(case):
@@ -400,11 +422,21 @@ def github_apk_matrix(cases):
} }
def github_transport_matrix(cases, *, apk=False): # The Android lanes of the transport matrix. "full" replays every case on every transport.
# "inproc-full" replays every case on inproc, the transport the launcher ships, and only the
# manifest's android_smoke cases on monolith and spawn: the Linux retrace jobs replay every
# case on all three transports, so those two emulator lanes keep the cases that prove the
# Android side of each arm.
ANDROID_LANES = ("full", "inproc-full")
def github_transport_matrix(cases, *, apk=False, android_smoke=None):
"""Run the entire original matrix as control and as acceptance, once per transport. """Run the entire original matrix as control and as acceptance, once per transport.
The transports use the same D/P artifact and each case's existing backend The transports use the same D/P artifact and each case's existing backend
list. The legacy split:true subset does not limit this main acceptance matrix. list. The legacy split:true subset does not limit this main acceptance matrix.
With `android_smoke`, an APK matrix keeps every case on inproc and only the named
cases on monolith and spawn.
Nested APK case/backend metadata is deliberately left untouched. Nested APK case/backend metadata is deliberately left untouched.
THE SPAWN LANE IS APK-ONLY, and that is a statement about where the evidence is missing THE SPAWN LANE IS APK-ONLY, and that is a statement about where the evidence is missing
@@ -423,12 +455,19 @@ def github_transport_matrix(cases, *, apk=False):
build = github_apk_matrix if apk else github_test_matrix build = github_apk_matrix if apk else github_test_matrix
if not cases: if not cases:
raise ValueError("transport matrix needs at least one trace case") raise ValueError("transport matrix needs at least one trace case")
reduced = cases
if android_smoke is not None:
if not apk:
raise ValueError("only the APK transport matrix has reduced lanes")
reduced = [case for case in cases if case["name"] in android_smoke]
if not reduced:
raise ValueError("the reduced Android lanes are empty: no case is in android_smoke")
lanes = [ lanes = [
(cases, "monolith-control", "monolith"), (reduced, "monolith-control", "monolith"),
(cases, "inproc-acceptance", "inproc"), (cases, "inproc-acceptance", "inproc"),
] ]
if apk: if apk:
lanes.append((cases, "spawn-acceptance", "spawn")) lanes.append((reduced, "spawn-acceptance", "spawn"))
return {"include": [ return {"include": [
{**row, "lane": lane, "transport": transport} {**row, "lane": lane, "transport": transport}
for selected, lane, transport in lanes for selected, lane, transport in lanes
@@ -481,6 +520,29 @@ def self_test_transport_matrices():
print(f"transport matrix {'APK' if apk else 'Linux'}: {len(controls)} unchanged controls, " print(f"transport matrix {'APK' if apk else 'Linux'}: {len(controls)} unchanged controls, "
f"{len(acceptance)} inproc acceptance entries, {len(spawned)} spawn acceptance " f"{len(acceptance)} inproc acceptance entries, {len(spawned)} spawn acceptance "
f"entries; exact metadata and negative controls OK") f"entries; exact metadata and negative controls OK")
# The reduced Android lanes are the full matrix less the monolith and spawn rows of the
# cases outside android_smoke: nothing else is dropped, changed or added.
smoke = android_smoke_names()
assert smoke, "android_smoke is empty, so the reduced monolith and spawn lanes would be too"
full = github_transport_matrix(cases, apk=True)["include"]
reduced = github_transport_matrix(cases, apk=True, android_smoke=smoke)["include"]
kept = [row for row in full if row["transport"] == "inproc" or row["case"]["name"] in smoke]
assert reduced == kept, "the reduced Android lanes are not the full matrix filtered"
for transport in ("monolith", "spawn"):
names = {row["case"]["name"] for row in reduced if row["transport"] == transport}
assert names == set(smoke), f"the reduced {transport} lane is not the smoke cases: {names}"
full_inproc = [row for row in full if row["transport"] == "inproc"]
assert [row for row in reduced if row["transport"] == "inproc"] == full_inproc, \
"the inproc lane was reduced"
for bad in ({"apk": False, "android_smoke": smoke},
{"apk": True, "android_smoke": ["no-such-case"]}):
try:
github_transport_matrix(cases, **bad)
except ValueError:
continue
raise AssertionError(f"github_transport_matrix accepted {bad}")
print(f"transport matrix APK reduced: {len(reduced)} of {len(full)} rows; inproc keeps all "
f"{len(full_inproc)}, monolith and spawn keep the {len(smoke)} smoke cases")
def self_test_split_matrix(cases=None): def self_test_split_matrix(cases=None):
@@ -518,6 +580,318 @@ def self_test_split_matrix(cases=None):
f"kept); all {len(dropped)} dropped inproc rows are inproc-acceptance rows") f"kept); all {len(dropped)} dropped inproc rows are inproc-acceptance rows")
# --- Shards: several cases of one cell replayed by one CI job ---------------------------------
#
# A cell is everything a per-case matrix row carries except its case: the backend, and the lane
# and transport where the matrix has them. A shard row keeps the cell's keys unchanged and adds
# the cases it replays, so the job that runs it reads the same fields a per-case leg reads.
# Every shard matrix is built from the per-case matrix it replaces and is refused unless its
# rows, taken apart again, are exactly that matrix.
TRACE_DURATION_HINTS_JSON = Path(__file__).with_name("trace_duration_hints.json")
# Shards per cell on Linux, the one tunable of that layout. A cell with little to replay gets
# fewer: a shard costs about half a minute of setup, so it is not split below
# MIN_SHARD_SECONDS of replay.
DEFAULT_SHARDS_PER_CELL = 4
MIN_SHARD_SECONDS = 90
# Replay seconds one emulator shard aims for, the one tunable of the Android layout. A cell is
# split into as many shards as that takes.
DEFAULT_ANDROID_SHARD_SECONDS = 580
# The hint a row without a transport uses. The verify lane has timings of its own.
HINT_KIND_MONOLITH = "monolith"
HINT_KIND_VERIFY = "verify"
# A shard's case names reach a shell command line, so the characters are restricted.
SHARD_SAFE_NAME = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-")
def load_duration_hints(platform, path=TRACE_DURATION_HINTS_JSON):
"""{case: {backend: {kind: seconds}}} for "linux" or "android"; empty when there is none."""
path = Path(path)
if not path.is_file():
return {}
with path.open("r", encoding="utf-8") as file:
table = json.load(file).get(platform, {}).get("seconds", {})
for case, backends in table.items():
for backend, kinds in backends.items():
for kind, seconds in kinds.items():
if isinstance(seconds, bool) or not isinstance(seconds, (int, float)) or seconds <= 0:
raise ValueError(f"duration hint for {case} {backend} {kind} must be a positive "
f"number: {seconds!r}")
return table
def median(values):
ordered = sorted(values)
if not ordered:
return None
middle = len(ordered) // 2
if len(ordered) % 2:
return ordered[middle]
return (ordered[middle - 1] + ordered[middle]) / 2
def hint_seconds(hints, case, backend, kind):
"""The hinted replay time of one row, or the median of its backend and kind, or of everything."""
try:
return hints[case][backend][kind]
except KeyError:
pass
same_cell = [kinds[kind] for backends in hints.values()
for name, kinds in backends.items() if name == backend and kind in kinds]
everything = [seconds for backends in hints.values() for kinds in backends.values()
for seconds in kinds.values()]
return median(same_cell) or median(everything) or 30
def row_case_name(row):
case = row["case"]
return case["name"] if isinstance(case, dict) else case
def row_backend_name(row):
backend = row["backend"]
return backend["name"] if isinstance(backend, dict) else backend
def row_cell(row):
return {key: value for key, value in row.items() if key != "case"}
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"))
def shard_rows_as_cases(shard_rows):
"""Take shard rows apart into the per-case rows they stand for."""
added = ("shard", "shards", "names", "cases", "seconds")
rows = []
for shard in shard_rows:
cell = {key: value for key, value in shard.items() if key not in added}
for case in shard.get("cases", shard["names"]):
rows.append({**cell, "case": case})
return rows
def check_shard_union(rows, shard_rows):
"""Refuse a shard matrix that is not exactly the per-case matrix `rows`, regrouped."""
expected = sorted(canonical(row) for row in rows)
actual = sorted(canonical(row) for row in shard_rows_as_cases(shard_rows))
if expected != actual:
missing = sorted(set(expected) - set(actual))
extra = sorted(set(actual) - set(expected))
raise ValueError(
f"the shard matrix is not the per-case matrix regrouped: {len(expected)} per-case rows, "
f"{len(actual)} rows in the shards; missing {missing[:3]}, unexpected {extra[:3]}")
if len(set(expected)) != len(expected):
raise ValueError("the per-case matrix repeats a row, so a shard would replay it twice")
cells = {}
for shard in shard_rows:
if not shard["names"]:
raise ValueError(f"empty shard: {shard}")
if "cases" in shard and [case["name"] for case in shard["cases"]] != shard["names"]:
raise ValueError(f"a shard's names and cases disagree: {shard['names']}")
cell = canonical({key: value for key, value in shard.items()
if key not in ("shard", "shards", "names", "cases", "seconds")})
cells.setdefault(cell, []).append((shard["shard"], shard["shards"]))
for cell, numbers in cells.items():
count = len(numbers)
if sorted(numbers) != [(index, count) for index in range(1, count + 1)]:
raise ValueError(f"shards of {cell} are not numbered 1..{count} of {count}: {numbers}")
def shard_matrix(rows, hints, *, default_kind=HINT_KIND_MONOLITH, shards=None, shard_seconds=None):
"""Regroup a per-case matrix into shards of one cell each.
Exactly one of `shards` (shards per cell, capped so that none holds less than
MIN_SHARD_SECONDS of hinted replay) and `shard_seconds` (hinted replay per shard) sets the
layout. Cases go longest first into the shard with the least hinted time so far, which is
deterministic and keeps the shards of a cell even; inside a shard they run in matrix order.
"""
if (shards is None) == (shard_seconds is None):
raise ValueError("give exactly one of shards and shard_seconds")
if shards is not None and shards < 1:
raise ValueError(f"shards per cell must be at least 1: {shards}")
if shard_seconds is not None and shard_seconds < 1:
raise ValueError(f"seconds per shard must be at least 1: {shard_seconds}")
cells = {}
for index, row in enumerate(rows):
name = row_case_name(row)
if not name or not set(name) <= SHARD_SAFE_NAME:
raise ValueError(f"case name {name!r} cannot be passed on a command line; allowed "
f"characters are letters, digits, '.', '_' and '-'")
seconds = hint_seconds(hints, name, row_backend_name(row),
row.get("transport", default_kind))
cells.setdefault(canonical(row_cell(row)), []).append((index, row, seconds))
out = []
for members in cells.values():
total = sum(seconds for _, _, seconds in members)
if shards is not None:
count = min(shards, int(total // MIN_SHARD_SECONDS))
else:
count = -(-int(total) // shard_seconds)
count = max(1, min(count, len(members)))
bins = [[] for _ in range(count)]
loads = [0] * count
for member in sorted(members, key=lambda item: (-item[2], item[0])):
target = loads.index(min(loads))
bins[target].append(member)
loads[target] += member[2]
for number, members_of_shard in enumerate(bins, start=1):
members_of_shard.sort(key=lambda item: item[0])
shard = {**row_cell(members_of_shard[0][1]), "shard": number, "shards": count,
"names": [row_case_name(row) for _, row, _ in members_of_shard]}
if isinstance(members_of_shard[0][1]["case"], dict):
shard["cases"] = [row["case"] for _, row, _ in members_of_shard]
shard["seconds"] = round(sum(seconds for _, _, seconds in members_of_shard))
out.append(shard)
check_shard_union(rows, out)
return {"include": out}
def per_case_matrix(format_name, cases, *, transports=SPLIT_TRANSPORTS, keep_inproc=(),
android_smoke=None):
"""The per-case matrix a shard format regroups, and the hint kind of its transport-less rows."""
if format_name == "github-test-shard-matrix":
return github_test_matrix(cases), HINT_KIND_MONOLITH
if format_name == "github-test-transport-shard-matrix":
return github_transport_matrix(cases), HINT_KIND_MONOLITH
if format_name == "github-verify-shard-matrix":
return github_verify_matrix(cases), HINT_KIND_VERIFY
if format_name == "github-split-shard-matrix":
return github_split_matrix(cases, transports, keep_inproc), HINT_KIND_MONOLITH
if format_name == "github-apk-shard-matrix":
return github_apk_matrix(cases), HINT_KIND_MONOLITH
if format_name == "github-apk-transport-shard-matrix":
return (github_transport_matrix(cases, apk=True, android_smoke=android_smoke),
HINT_KIND_MONOLITH)
raise ValueError(f"not a shard format: {format_name}")
LINUX_SHARD_FORMATS = (
"github-test-shard-matrix",
"github-test-transport-shard-matrix",
"github-verify-shard-matrix",
"github-split-shard-matrix",
)
ANDROID_SHARD_FORMATS = (
"github-apk-shard-matrix",
"github-apk-transport-shard-matrix",
)
def github_shard_matrix(format_name, cases, *, shards=DEFAULT_SHARDS_PER_CELL,
shard_seconds=DEFAULT_ANDROID_SHARD_SECONDS,
transports=SPLIT_TRANSPORTS, keep_inproc=(), hints_path=TRACE_DURATION_HINTS_JSON,
android_smoke=None):
"""A shard format: the Linux ones lay out by shards per cell, the Android ones by seconds."""
matrix, default_kind = per_case_matrix(format_name, cases, transports=transports,
keep_inproc=keep_inproc, android_smoke=android_smoke)
if format_name in ANDROID_SHARD_FORMATS:
return shard_matrix(matrix["include"], load_duration_hints("android", hints_path),
default_kind=default_kind, shard_seconds=shard_seconds)
return shard_matrix(matrix["include"], load_duration_hints("linux", hints_path),
default_kind=default_kind, shards=shards)
def self_test_shard_matrices():
"""Every shard format replays exactly the rows of the per-case matrix it replaces."""
cases = ci_trace_cases(load_trace_cases())
narrowed = {"transports": ("spawn",), "keep_inproc": ("OpenRA",)}
layouts = [(name, {}, {"shards": count}) for name in LINUX_SHARD_FORMATS for count in (1, 4, 100)]
layouts += [("github-split-shard-matrix", narrowed, {"shards": count}) for count in (1, 4, 100)]
layouts += [(name, {}, {"shard_seconds": seconds}) for name in ANDROID_SHARD_FORMATS
for seconds in (60, DEFAULT_ANDROID_SHARD_SECONDS, 10 ** 9)]
layouts += [("github-apk-transport-shard-matrix", {"android_smoke": android_smoke_names()},
{"shard_seconds": seconds}) for seconds in (60, DEFAULT_ANDROID_SHARD_SECONDS, 10 ** 9)]
for name, selection, layout in layouts:
matrix, _ = per_case_matrix(name, cases, **selection)
rows = matrix["include"]
built = github_shard_matrix(name, cases, **selection, **layout)["include"]
again = github_shard_matrix(name, cases, **selection, **layout)["include"]
assert built == again, f"{name} {layout}: the layout is not deterministic"
# Compared here as well as inside shard_matrix, with nothing shared but the JSON form.
replayed = sorted(
canonical({**{key: value for key, value in shard.items()
if key not in ("shard", "shards", "names", "cases", "seconds")},
"case": case})
for shard in built for case in shard.get("cases", shard["names"]))
assert replayed == sorted(canonical(row) for row in rows), \
f"{name} {layout}: the shards do not replay the per-case matrix"
assert sum(len(shard["names"]) for shard in built) == len(rows)
cells = {canonical(row_cell(row)) for row in rows}
assert len({canonical({key: value for key, value in shard.items()
if key not in ("shard", "shards", "names", "cases", "seconds")})
for shard in built}) == len(cells), f"{name} {layout}: a shard spans two cells"
if layout.get("shards") == 1 or layout.get("shard_seconds") == 10 ** 9:
assert len(built) == len(cells), f"{name} {layout}: expected one shard per cell"
seconds = [shard["seconds"] for shard in built]
shown = {key: (f"{len(value)} smoke cases" if key == "android_smoke" else value)
for key, value in selection.items()}
print(f"shard matrix {name} {shown or ''} {layout}: {len(rows)} rows in {len(built)} "
f"shards over {len(cells)} cells, {min(seconds)}-{max(seconds)} s hinted per shard")
# The check itself must be able to fail: a dropped, a repeated and a moved case.
if len(built) >= 2 and any(len(shard["names"]) >= 2 for shard in built):
index = next(i for i, shard in enumerate(built) if len(shard["names"]) >= 2)
for label, damage in (("dropped", lambda names: names[1:]),
("repeated", lambda names: names + names[:1])):
broken = [dict(shard) for shard in built]
broken[index]["names"] = damage(list(broken[index]["names"]))
if "cases" in broken[index]:
broken[index]["cases"] = damage(list(broken[index]["cases"]))
try:
check_shard_union(rows, broken)
except ValueError:
pass
else:
raise AssertionError(f"{name} {layout}: a {label} case passed the union check")
other = next((i for i, shard in enumerate(built)
if row_backend_name(shard) != row_backend_name(built[index])), None)
if other is not None:
broken = [dict(shard) for shard in built]
for key in ("names", "cases"):
if key in broken[index]:
moved = list(broken[index][key])
broken[other][key] = list(broken[other][key]) + moved[:1]
broken[index][key] = moved[1:]
try:
check_shard_union(rows, broken)
except ValueError:
pass
else:
raise AssertionError(f"{name} {layout}: a case moved to another backend's "
f"shard passed the union check")
# Hints only weigh the packing. With none at all, and with a case no hint names, the rows
# replayed are the same.
rows = github_transport_matrix(cases)["include"]
for hints in ({}, {"no-such-case": {"DirectGLES": {"monolith": 5}}}):
check_shard_union(rows, shard_matrix(rows, hints, shards=4)["include"])
assert hint_seconds({}, "x", "DirectGLES", "monolith") == 30
assert hint_seconds({"a": {"DirectGLES": {"monolith": 10}}, "b": {"DirectGLES": {"monolith": 20}},
"c": {"DirectVulkan": {"monolith": 90}}}, "x", "DirectGLES", "monolith") == 15
for bad in ({"shards": 0}, {"shard_seconds": 0}, {}, {"shards": 4, "shard_seconds": 60}):
try:
shard_matrix(rows, {}, **bad)
except ValueError:
continue
raise AssertionError(f"shard_matrix accepted {bad}")
try:
shard_matrix([{"backend": "DirectGLES", "case": "bad name; rm"}], {}, shards=1)
except ValueError:
pass
else:
raise AssertionError("a case name with shell characters was accepted")
# The verify lane's corrupted-snapshot control replays OpenRA on DirectGLES, so that row has
# to be in the verify matrix for the control to have a shard to run in.
verify_rows = github_verify_matrix(cases)["include"]
assert {"backend": "DirectGLES", "case": "OpenRA"} in verify_rows, \
"the verify matrix has no OpenRA DirectGLES row, so its negative control would not run"
print("shard matrices: union, numbering, determinism and negative controls OK")
def cmake_quote(value): def cmake_quote(value):
return '"' + str(value).replace("\\", "/").replace('"', '\\"') + '"' return '"' + str(value).replace("\\", "/").replace('"', '\\"') + '"'
@@ -570,15 +944,27 @@ def parse_args():
parser.add_argument("--self-test-transport-matrices", action="store_true") parser.add_argument("--self-test-transport-matrices", action="store_true")
parser.add_argument("--self-test-split-matrix", action="store_true", parser.add_argument("--self-test-split-matrix", action="store_true",
help="prove the spawn-only split matrix drops only inproc-acceptance rows") help="prove the spawn-only split matrix drops only inproc-acceptance rows")
parser.add_argument("--self-test-shard-matrices", action="store_true",
help="prove every shard matrix replays exactly its per-case matrix")
parser.add_argument("--fixture-root", default="tools/trace_replay/fixtures") parser.add_argument("--fixture-root", default="tools/trace_replay/fixtures")
parser.add_argument("--transports", default=",".join(SPLIT_TRANSPORTS), parser.add_argument("--transports", default=",".join(SPLIT_TRANSPORTS),
help="github-split-matrix: the comma-separated split transports to emit") help="github-split-matrix: the comma-separated split transports to emit")
parser.add_argument("--keep-inproc", default="", parser.add_argument("--keep-inproc", default="",
help="github-split-matrix: comma-separated split cases whose inproc rows stay") help="github-split-matrix: comma-separated split cases whose inproc rows stay")
parser.add_argument("--shards", type=int, default=DEFAULT_SHARDS_PER_CELL,
help="Linux shard formats: shards per (lane, transport, backend) cell")
parser.add_argument("--shard-seconds", type=int, default=DEFAULT_ANDROID_SHARD_SECONDS,
help="Android shard formats: hinted replay seconds one shard aims for")
parser.add_argument("--android-lanes", choices=ANDROID_LANES, default="full",
help="APK transport formats: inproc-full keeps every case on inproc and "
"the manifest's android_smoke cases on monolith and spawn")
parser.add_argument("--duration-hints", default=str(TRACE_DURATION_HINTS_JSON),
help="shard formats: the JSON of replay seconds the packing weighs cases by")
parser.add_argument( parser.add_argument(
"--format", "--format",
choices=( choices=(
"names", "names",
"android-smoke-names",
"github-test-matrix", "github-test-matrix",
"github-test-transport-matrix", "github-test-transport-matrix",
"github-verify-matrix", "github-verify-matrix",
@@ -586,7 +972,10 @@ def parse_args():
"github-apk", "github-apk",
"github-apk-matrix", "github-apk-matrix",
"github-apk-transport-matrix", "github-apk-transport-matrix",
*LINUX_SHARD_FORMATS,
*ANDROID_SHARD_FORMATS,
"fixture-files", "fixture-files",
"fixture-manifest",
"cmake", "cmake",
), ),
default="names", default="names",
@@ -602,11 +991,17 @@ def main():
if args.self_test_split_matrix: if args.self_test_split_matrix:
self_test_split_matrix() self_test_split_matrix()
return 0 return 0
if args.self_test_shard_matrices:
self_test_shard_matrices()
return 0
cases = load_trace_cases() cases = load_trace_cases()
if args.ci: if args.ci:
cases = ci_trace_cases(cases) cases = ci_trace_cases(cases)
android_smoke = android_smoke_names() if args.android_lanes == "inproc-full" else None
if args.format == "names": if args.format == "names":
print(json.dumps([case["name"] for case in cases], separators=(",", ":"))) print(json.dumps([case["name"] for case in cases], separators=(",", ":")))
elif args.format == "android-smoke-names":
print(",".join(android_smoke_names()))
elif args.format == "github-test-matrix": elif args.format == "github-test-matrix":
print(json.dumps(github_test_matrix(cases), separators=(",", ":"))) print(json.dumps(github_test_matrix(cases), separators=(",", ":")))
elif args.format == "github-test-transport-matrix": elif args.format == "github-test-transport-matrix":
@@ -622,7 +1017,22 @@ def main():
elif args.format == "github-apk-matrix": elif args.format == "github-apk-matrix":
print(json.dumps(github_apk_matrix(cases), separators=(",", ":"))) print(json.dumps(github_apk_matrix(cases), separators=(",", ":")))
elif args.format == "github-apk-transport-matrix": elif args.format == "github-apk-transport-matrix":
print(json.dumps(github_transport_matrix(cases, apk=True), separators=(",", ":"))) print(json.dumps(github_transport_matrix(cases, apk=True, android_smoke=android_smoke),
separators=(",", ":")))
elif args.format in LINUX_SHARD_FORMATS + ANDROID_SHARD_FORMATS:
transports = tuple(t for t in args.transports.split(",") if t)
keep = tuple(c for c in args.keep_inproc.split(",") if c)
print(json.dumps(
github_shard_matrix(args.format, cases, shards=args.shards,
shard_seconds=args.shard_seconds, transports=transports,
keep_inproc=keep, hints_path=args.duration_hints,
android_smoke=android_smoke),
separators=(",", ":")))
elif args.format == "fixture-manifest":
# One "<case><TAB><file>" line per fixture file, for the fixture bundle script.
for case in cases:
for file in fixture_files(case, args.fixture_root):
print(f"{case['name']}\t{file}")
elif args.format == "fixture-files": elif args.format == "fixture-files":
if not args.case_name: if not args.case_name:
print("--case is required for --format fixture-files", file=sys.stderr) print("--case is required for --format fixture-files", file=sys.stderr)
@@ -0,0 +1,99 @@
{
"description": [
"Replay seconds per case, backend and transport, used only to pack retrace shards evenly",
"(tools/trace_replay/trace_cases.py, shard_matrix). A missing entry falls back to the median of",
"its backend and transport, so a new case needs no entry here. Nothing asserts these values;",
"a stale number only makes shards uneven. To refresh, copy the seconds column of the shard",
"step summaries of a green run."
],
"linux": {
"source": "Test run 43666299896 (2026-11-15), step 'Retrace and validate' of each one-case job, two gaps filled from run 43700779712; the verify key is the retrace-verify job",
"seconds": {
"OpenRA": {"DirectGLES": {"inproc": 3, "monolith": 3, "spawn": 3, "verify": 3}, "DirectVulkan": {"inproc": 3, "monolith": 3, "spawn": 2, "verify": 4}},
"improved-transparency-minecraft-26.3": {"DirectGLES": {"inproc": 86, "monolith": 51, "spawn": 78, "verify": 81}, "DirectVulkan": {"inproc": 82, "monolith": 104, "spawn": 107, "verify": 85}},
"minecraft-1.17-main-menu-854": {"DirectGLES": {"inproc": 4, "monolith": 6, "spawn": 3, "verify": 3}, "DirectVulkan": {"inproc": 5, "monolith": 5, "spawn": 3, "verify": 3}},
"minecraft-1.20.1-fabric-iris-iterationt-in-world": {"DirectGLES": {"inproc": 59, "monolith": 70, "spawn": 72}, "DirectVulkan": {"inproc": 62, "monolith": 69, "spawn": 68}},
"minecraft-1.21.1-neoforge-create-indirect-in-world-align1024": {"DirectGLES": {"inproc": 9, "monolith": 18, "spawn": 16}, "DirectVulkan": {"inproc": 13, "monolith": 13, "spawn": 8}},
"minecraft-1.21.1-neoforge-create-instancing-in-world": {"DirectGLES": {"inproc": 12, "monolith": 14, "spawn": 14}, "DirectVulkan": {"inproc": 10, "monolith": 14, "spawn": 10}},
"minecraft-1.21.11-main-menu": {"DirectGLES": {"inproc": 5, "monolith": 5, "spawn": 3, "verify": 6}, "DirectVulkan": {"inproc": 7, "monolith": 6, "spawn": 6, "verify": 5}},
"minecraft-1.21.4-fabric-common-mods-in-world": {"DirectGLES": {"inproc": 13, "monolith": 15, "spawn": 15}, "DirectVulkan": {"inproc": 19, "monolith": 17, "spawn": 14}},
"minecraft-1.21.4-fabric-common-mods-inventory": {"DirectGLES": {"inproc": 20, "monolith": 13, "spawn": 11}, "DirectVulkan": {"inproc": 22, "monolith": 21, "spawn": 22}},
"minecraft-1.21.4-fabric-iris-bliss-in-world": {"DirectGLES": {"inproc": 11, "monolith": 23, "spawn": 16}, "DirectVulkan": {"inproc": 17, "monolith": 17, "spawn": 16}},
"minecraft-1.21.4-fabric-iris-bsl-esc-menu-854": {"DirectGLES": {"inproc": 126, "monolith": 127, "spawn": 124}, "DirectVulkan": {"inproc": 130, "monolith": 131, "spawn": 134}},
"minecraft-1.21.4-fabric-iris-bsl-in-world": {"DirectGLES": {"inproc": 15, "monolith": 18, "spawn": 17}, "DirectVulkan": {"inproc": 16, "monolith": 15, "spawn": 17}},
"minecraft-1.21.4-fabric-iris-chocapic-v6-lite-in-world": {"DirectGLES": {"inproc": 10, "monolith": 15, "spawn": 14}, "DirectVulkan": {"inproc": 14, "monolith": 13, "spawn": 15}},
"minecraft-1.21.4-fabric-iris-complementary-reimagined-in-world": {"DirectGLES": {"inproc": 27, "monolith": 35, "spawn": 29}, "DirectVulkan": {"inproc": 41, "monolith": 31, "spawn": 28}},
"minecraft-1.21.4-fabric-iris-complementary-unbound-in-world": {"DirectGLES": {"inproc": 41, "monolith": 33, "spawn": 38}, "DirectVulkan": {"inproc": 26, "monolith": 31, "spawn": 25}},
"minecraft-1.21.4-fabric-iris-derivative-main-d24.4.14-in-world": {"DirectGLES": {"inproc": 23, "monolith": 33, "spawn": 24}, "DirectVulkan": {"inproc": 20, "monolith": 26, "spawn": 27}},
"minecraft-1.21.4-fabric-iris-iterationrp-in-world": {"DirectVulkan": {"inproc": 42, "monolith": 42, "spawn": 41}},
"minecraft-1.21.4-fabric-iris-iterationt-in-world": {"DirectGLES": {"inproc": 26, "monolith": 27, "spawn": 23}, "DirectVulkan": {"inproc": 23, "monolith": 20, "spawn": 19}},
"minecraft-1.21.4-fabric-iris-iterationt-nodsa-in-world": {"DirectGLES": {"inproc": 24, "monolith": 19, "spawn": 19}, "DirectVulkan": {"inproc": 19, "monolith": 13, "spawn": 21}},
"minecraft-1.21.4-fabric-iris-makeup-ultrafast-in-world": {"DirectGLES": {"inproc": 13, "monolith": 9, "spawn": 10}, "DirectVulkan": {"inproc": 11, "monolith": 8, "spawn": 8}},
"minecraft-1.21.4-fabric-iris-mellow-in-world": {"DirectGLES": {"inproc": 7, "monolith": 11, "spawn": 9}, "DirectVulkan": {"inproc": 9, "monolith": 9, "spawn": 7}},
"minecraft-1.21.4-fabric-iris-nostalgia-in-world": {"DirectGLES": {"inproc": 25, "monolith": 21, "spawn": 27}, "DirectVulkan": {"inproc": 24, "monolith": 26, "spawn": 16}},
"minecraft-1.21.4-fabric-iris-photon-v1.1-in-world": {"DirectGLES": {"inproc": 25, "monolith": 34, "spawn": 21}, "DirectVulkan": {"inproc": 24, "monolith": 24, "spawn": 33}},
"minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world": {"DirectGLES": {"inproc": 29, "monolith": 36, "spawn": 29}, "DirectVulkan": {"inproc": 30, "monolith": 27, "spawn": 36}},
"minecraft-1.21.4-fabric-iris-sundial-lite-in-world": {"DirectGLES": {"inproc": 35, "monolith": 27, "spawn": 21}, "DirectVulkan": {"inproc": 31, "monolith": 20, "spawn": 30}},
"minecraft-1.21.4-fabric-iris-super-duper-vanilla-in-world": {"DirectGLES": {"inproc": 11, "monolith": 22, "spawn": 21}, "DirectVulkan": {"inproc": 14, "monolith": 20, "spawn": 14}},
"minecraft-1.21.4-fabric-journeymap-in-world": {"DirectGLES": {"inproc": 43, "monolith": 66, "spawn": 65}, "DirectVulkan": {"inproc": 41, "monolith": 60, "spawn": 90}},
"minecraft-1.21.4-fabric-journeymap-in-world-normal-world": {"DirectGLES": {"inproc": 10, "monolith": 12, "spawn": 14}, "DirectVulkan": {"inproc": 15, "monolith": 12, "spawn": 14}},
"minecraft-1.21.4-fabric-modernui-inventory": {"DirectGLES": {"inproc": 54, "monolith": 50, "spawn": 67}, "DirectVulkan": {"inproc": 50, "monolith": 66, "spawn": 70}},
"minecraft-1.21.4-fabric-modernui-inventory-normal-world": {"DirectGLES": {"inproc": 15, "monolith": 17, "spawn": 9}, "DirectVulkan": {"inproc": 18, "monolith": 13, "spawn": 10}},
"minecraft-1.21.4-fabric-rei-inventory": {"DirectGLES": {"inproc": 72, "monolith": 63, "spawn": 90}, "DirectVulkan": {"inproc": 79, "monolith": 84, "spawn": 49}},
"minecraft-1.21.4-fabric-rei-inventory-normal-world": {"DirectGLES": {"inproc": 15, "monolith": 22, "spawn": 17}, "DirectVulkan": {"inproc": 20, "monolith": 18, "spawn": 19}},
"minecraft-1.21.4-fabric-sodium-in-world": {"DirectGLES": {"inproc": 19, "monolith": 15, "spawn": 26, "verify": 25}, "DirectVulkan": {"inproc": 29, "monolith": 32, "spawn": 23, "verify": 16}},
"minecraft-1.21.4-fabric-xaero-minimap-in-world": {"DirectGLES": {"inproc": 41, "monolith": 51, "spawn": 64}, "DirectVulkan": {"inproc": 55, "monolith": 69, "spawn": 77}},
"minecraft-1.21.4-fabric-xaero-minimap-in-world-normal-world": {"DirectGLES": {"inproc": 12, "monolith": 13, "spawn": 8}, "DirectVulkan": {"inproc": 9, "monolith": 10, "spawn": 13}},
"minecraft-1.21.4-fabric-xaero-world-map-in-world": {"DirectGLES": {"inproc": 41, "monolith": 40, "spawn": 52}, "DirectVulkan": {"inproc": 42, "monolith": 43, "spawn": 59}},
"minecraft-1.21.4-fabric-xaero-world-map-in-world-normal-world": {"DirectGLES": {"inproc": 6, "monolith": 11, "spawn": 8}, "DirectVulkan": {"inproc": 13, "monolith": 9, "spawn": 10}},
"minecraft-1.21.4-in-world": {"DirectGLES": {"inproc": 17, "monolith": 10, "spawn": 11, "verify": 11}, "DirectVulkan": {"inproc": 10, "monolith": 9, "spawn": 12, "verify": 11}},
"minecraft-1.21.4-main-menu": {"DirectGLES": {"inproc": 7, "monolith": 4, "spawn": 9, "verify": 6}, "DirectVulkan": {"inproc": 5, "monolith": 10, "spawn": 6, "verify": 6}},
"minecraft-1.21.4-startup": {"DirectGLES": {"inproc": 5, "monolith": 3, "spawn": 1, "verify": 3}, "DirectVulkan": {"inproc": 1, "monolith": 4, "spawn": 2, "verify": 2}}
}
},
"android": {
"source": "in-shard seconds of MobileGL APK runs 43732107527 and 43735077666 (2026-11-15), one APK install per shard; the monolith and spawn rows outside android_smoke are one-case samples of run 43666299861 scaled to in-shard time",
"seconds": {
"OpenRA": {"DirectGLES": {"inproc": 17, "monolith": 21, "spawn": 18}, "DirectVulkan": {"inproc": 16, "monolith": 22, "spawn": 15}},
"improved-transparency-minecraft-26.3": {"DirectGLES": {"inproc": 116, "monolith": 144, "spawn": 98}, "DirectVulkan": {"inproc": 182, "monolith": 200, "spawn": 138}},
"minecraft-1.17-main-menu-854": {"DirectGLES": {"inproc": 25, "monolith": 20, "spawn": 18}, "DirectVulkan": {"inproc": 18, "monolith": 19, "spawn": 17}},
"minecraft-1.20.1-fabric-iris-iterationt-in-world": {"DirectGLES": {"inproc": 116, "monolith": 108, "spawn": 93}, "DirectVulkan": {"inproc": 93, "monolith": 96, "spawn": 82}},
"minecraft-1.21.1-neoforge-create-indirect-in-world-align1024": {"DirectGLES": {"inproc": 92, "monolith": 61, "spawn": 79}, "DirectVulkan": {"inproc": 21, "monolith": 42, "spawn": 18}},
"minecraft-1.21.1-neoforge-create-instancing-in-world": {"DirectGLES": {"inproc": 74, "monolith": 86, "spawn": 89}, "DirectVulkan": {"inproc": 19, "monolith": 24, "spawn": 20}},
"minecraft-1.21.11-main-menu": {"DirectGLES": {"inproc": 28, "monolith": 28, "spawn": 20}, "DirectVulkan": {"inproc": 34, "monolith": 24, "spawn": 27}},
"minecraft-1.21.4-fabric-common-mods-in-world": {"DirectGLES": {"inproc": 54, "monolith": 38, "spawn": 35}, "DirectVulkan": {"inproc": 45, "monolith": 47, "spawn": 35}},
"minecraft-1.21.4-fabric-common-mods-inventory": {"DirectGLES": {"inproc": 56, "monolith": 65, "spawn": 35}, "DirectVulkan": {"inproc": 68, "monolith": 53, "spawn": 40}},
"minecraft-1.21.4-fabric-iris-bliss-in-world": {"DirectGLES": {"inproc": 48, "monolith": 43, "spawn": 37}, "DirectVulkan": {"inproc": 28, "monolith": 60, "spawn": 26}},
"minecraft-1.21.4-fabric-iris-bsl-esc-menu-854": {"DirectGLES": {"inproc": 197, "monolith": 134, "spawn": 107}, "DirectVulkan": {"inproc": 134, "monolith": 172, "spawn": 153}},
"minecraft-1.21.4-fabric-iris-bsl-in-world": {"DirectGLES": {"inproc": 26, "monolith": 27, "spawn": 42}, "DirectVulkan": {"inproc": 25, "monolith": 30, "spawn": 37}},
"minecraft-1.21.4-fabric-iris-chocapic-v6-lite-in-world": {"DirectGLES": {"inproc": 30, "monolith": 37, "spawn": 30}, "DirectVulkan": {"inproc": 24, "monolith": 34, "spawn": 25}},
"minecraft-1.21.4-fabric-iris-complementary-reimagined-in-world": {"DirectGLES": {"inproc": 91, "monolith": 89, "spawn": 59}, "DirectVulkan": {"inproc": 52, "monolith": 63, "spawn": 67}},
"minecraft-1.21.4-fabric-iris-complementary-unbound-in-world": {"DirectGLES": {"inproc": 81, "monolith": 77, "spawn": 63}, "DirectVulkan": {"inproc": 48, "monolith": 81, "spawn": 75}},
"minecraft-1.21.4-fabric-iris-derivative-main-d24.4.14-in-world": {"DirectGLES": {"inproc": 38, "monolith": 69, "spawn": 49}, "DirectVulkan": {"inproc": 34, "monolith": 63, "spawn": 42}},
"minecraft-1.21.4-fabric-iris-iterationrp-in-world": {"DirectVulkan": {"inproc": 66, "monolith": 77, "spawn": 53}},
"minecraft-1.21.4-fabric-iris-iterationt-in-world": {"DirectGLES": {"inproc": 41, "monolith": 51, "spawn": 32}, "DirectVulkan": {"inproc": 30, "monolith": 51, "spawn": 44}},
"minecraft-1.21.4-fabric-iris-iterationt-nodsa-in-world": {"DirectGLES": {"inproc": 51, "monolith": 54, "spawn": 28}, "DirectVulkan": {"inproc": 23, "monolith": 45, "spawn": 44}},
"minecraft-1.21.4-fabric-iris-makeup-ultrafast-in-world": {"DirectGLES": {"inproc": 32, "monolith": 32, "spawn": 33}, "DirectVulkan": {"inproc": 16, "monolith": 35, "spawn": 28}},
"minecraft-1.21.4-fabric-iris-mellow-in-world": {"DirectGLES": {"inproc": 33, "monolith": 31, "spawn": 29}, "DirectVulkan": {"inproc": 15, "monolith": 31, "spawn": 28}},
"minecraft-1.21.4-fabric-iris-nostalgia-in-world": {"DirectGLES": {"inproc": 63, "monolith": 59, "spawn": 53}, "DirectVulkan": {"inproc": 38, "monolith": 54, "spawn": 49}},
"minecraft-1.21.4-fabric-iris-photon-v1.1-in-world": {"DirectGLES": {"inproc": 51, "monolith": 86, "spawn": 59}, "DirectVulkan": {"inproc": 42, "monolith": 81, "spawn": 55}},
"minecraft-1.21.4-fabric-iris-photon-v1.3b-in-world": {"DirectGLES": {"inproc": 75, "monolith": 82, "spawn": 52}, "DirectVulkan": {"inproc": 46, "monolith": 87, "spawn": 58}},
"minecraft-1.21.4-fabric-iris-sundial-lite-in-world": {"DirectGLES": {"inproc": 43, "monolith": 46, "spawn": 49}, "DirectVulkan": {"inproc": 44, "monolith": 59, "spawn": 35}},
"minecraft-1.21.4-fabric-iris-super-duper-vanilla-in-world": {"DirectGLES": {"inproc": 49, "monolith": 53, "spawn": 34}, "DirectVulkan": {"inproc": 28, "monolith": 53, "spawn": 37}},
"minecraft-1.21.4-fabric-journeymap-in-world": {"DirectGLES": {"inproc": 281, "monolith": 175, "spawn": 112}, "DirectVulkan": {"inproc": 212, "monolith": 199, "spawn": 138}},
"minecraft-1.21.4-fabric-journeymap-in-world-normal-world": {"DirectGLES": {"inproc": 19, "monolith": 26, "spawn": 44}, "DirectVulkan": {"inproc": 25, "monolith": 48, "spawn": 32}},
"minecraft-1.21.4-fabric-modernui-inventory": {"DirectGLES": {"inproc": 289, "monolith": 219, "spawn": 97}, "DirectVulkan": {"inproc": 254, "monolith": 251, "spawn": 126}},
"minecraft-1.21.4-fabric-modernui-inventory-normal-world": {"DirectGLES": {"inproc": 29, "monolith": 45, "spawn": 43}, "DirectVulkan": {"inproc": 31, "monolith": 45, "spawn": 48}},
"minecraft-1.21.4-fabric-rei-inventory": {"DirectGLES": {"inproc": 264, "monolith": 241, "spawn": 106}, "DirectVulkan": {"inproc": 237, "monolith": 210, "spawn": 131}},
"minecraft-1.21.4-fabric-rei-inventory-normal-world": {"DirectGLES": {"inproc": 30, "monolith": 44, "spawn": 37}, "DirectVulkan": {"inproc": 35, "monolith": 54, "spawn": 49}},
"minecraft-1.21.4-fabric-sodium-in-world": {"DirectGLES": {"inproc": 111, "monolith": 118, "spawn": 57}, "DirectVulkan": {"inproc": 145, "monolith": 154, "spawn": 60}},
"minecraft-1.21.4-fabric-xaero-minimap-in-world": {"DirectGLES": {"inproc": 219, "monolith": 172, "spawn": 95}, "DirectVulkan": {"inproc": 224, "monolith": 201, "spawn": 112}},
"minecraft-1.21.4-fabric-xaero-minimap-in-world-normal-world": {"DirectGLES": {"inproc": 51, "monolith": 33, "spawn": 35}, "DirectVulkan": {"inproc": 23, "monolith": 41, "spawn": 33}},
"minecraft-1.21.4-fabric-xaero-world-map-in-world": {"DirectGLES": {"inproc": 171, "monolith": 195, "spawn": 75}, "DirectVulkan": {"inproc": 165, "monolith": 199, "spawn": 90}},
"minecraft-1.21.4-fabric-xaero-world-map-in-world-normal-world": {"DirectGLES": {"inproc": 32, "monolith": 40, "spawn": 30}, "DirectVulkan": {"inproc": 21, "monolith": 37, "spawn": 45}},
"minecraft-1.21.4-in-world": {"DirectGLES": {"inproc": 62, "monolith": 44, "spawn": 29}, "DirectVulkan": {"inproc": 40, "monolith": 47, "spawn": 33}},
"minecraft-1.21.4-main-menu": {"DirectGLES": {"inproc": 41, "monolith": 37, "spawn": 27}, "DirectVulkan": {"inproc": 47, "monolith": 53, "spawn": 29}},
"minecraft-1.21.4-startup": {"DirectGLES": {"inproc": 15, "monolith": 11, "spawn": 12}, "DirectVulkan": {"inproc": 13, "monolith": 12, "spawn": 7}}
}
}
}
+1 -1
View File
@@ -68,7 +68,7 @@ frame. It exists for sharing one GPU-owning server between processes, sandboxes
- `GL_RENDERER` starts with `Espryt (MobileGL Core)` or `Magma (MobileGL Core)`. - `GL_RENDERER` starts with `Espryt (MobileGL Core)` or `Magma (MobileGL Core)`.
- `GL_VENDOR` contains `MobileGL-Dev`. - `GL_VENDOR` contains `MobileGL-Dev`.
- `GL_VERSION` looks like `4.6 MobileGL 26.10.0-dev..., Direct (GLES) Backend, GIT@<hash>` (or - `GL_VERSION` looks like `4.6.0 MobileGL 26.11-dev, Direct (GLES) Backend, GIT@<hash>` (or
`Direct (Vulkan)`). `Direct (Vulkan)`).
- In Minecraft, the F3 screen shows these strings. - In Minecraft, the F3 screen shows these strings.
- The log says `Config: Active backend type set to DirectGLES` (or `DirectVulkan`). Logs go to - The log says `Config: Active backend type set to DirectGLES` (or `DirectVulkan`). Logs go to
+1 -1
View File
@@ -75,7 +75,7 @@ latency.
The plugin's launcher environment includes `MOBILEGL_TRANSPORT=inproc` **only if** the APK was built The plugin's launcher environment includes `MOBILEGL_TRANSPORT=inproc` **only if** the APK was built
with `-Pmobilegl.buildDisaggregated=ON` or `-Pmobilegl.buildDisaggregatedInproc=ON` passed explicitly with `-Pmobilegl.buildDisaggregated=ON` or `-Pmobilegl.buildDisaggregatedInproc=ON` passed explicitly
(or the matching `MOBILEGL_BUILD_DISAGGREGATED` / `MOBILEGL_BUILD_DISAGGREGATED_INPROC` environment (or the matching `MOBILEGL_BUILD_DISAGGREGATED` / `MOBILEGL_BUILD_DISAGGREGATED_INPROC` environment
variable set to `ON`; the CI build of `feat/disaggregated` does this). A plain `gradle :app:assemblePluginRelease` variable set to `ON`; the CI builds of `dev` and `feat/disaggregated` do this, so their APKs default to `inproc` and carry the render server). A plain `gradle :app:assemblePluginRelease`
still builds the split layer into the library, but the launcher environment then omits the transport still builds the split layer into the library, but the launcher environment then omits the transport
and the plugin runs monolith. Either rebuild with the property, or add and the plugin runs monolith. Either rebuild with the property, or add
`MOBILEGL_TRANSPORT=inproc` in the launcher's environment editor. `MOBILEGL_TRANSPORT=inproc` in the launcher's environment editor.