WIP: perf(vk): the async present ring and event-gated readback delivery (RX_ASYNC_RENDER, RX_WB_RB_EVENTS), ported to current main #2910

Draft
blessedplummer wants to merge 7 commits from blessedplummer/Hookshot:async-port into main
First-time contributor

Why: the game's job threads stall on readbacks — invalidate()'s wait goes through the replay queue's catch-up, and every readback rode whole submissions until the per-part wait landed. This carries the async-render work as optional, default-off switches so it can be measured alone: a deeper present ring whose presenter paces presents on the display's vblank grid, and readback delivery that copies written binds into guest memory right after their pass, event-gated, so the CPU polls an event instead of waiting the submission.

What it does (every switch default OFF — with them off, main's paths run untouched):

  • RX_ASYNC_RENDER=1 (+ RX_ASYNC_FIF, 3..8, default 4): the present ring keeps N frame slots, the game's release event fires from its present (host/nvn_native.c, rx_release_due), the presenter thread lands those presents on the display's vblank grid, and invalidate()'s readbacks wait only their own write-back's timeline value — per-value ordering, never the replay queue. A phase-2 retry lets a submission parked at an event wait go on before the wait re-runs. A per-frame poll-only ledger (vk: async f=) measures every remaining wait; a per-frame summary reports fence/acquire waits and busy.
  • RX_WB_RB_EVENTS=1: written binds whose range touches the readback history (the meters and counters the game reads every frame) are delivered into guest memory right after the pass that binds them — their write-back copy made early, each with its own event (memory_rb_deliver, memory_writeback_event); the CPU's readback polls the event before waiting the covering value.
  • Also carried, default off, kept for separate measurement: RX_VK_CAP_EAGER (the capture's pool list includes the mirror-wanted and not-yet-bound pools — closes the mirror-sync live-read race at import), RX_WB_RB_DIRECT (prototype: readback pages copy straight from the mirror), RX_VK_RB_EVER_PAGES, RX_VK_COND_CPU_WORDS.

Testing status: incomplete — a draft on purpose.

  • Suite: 518 passed. test_readback_part.py's gate follows the ported choice block in this branch (the fixture stubs the async additions with the switch off; every assertion on the per-part wait is unchanged). test_appimage_package_retry fails in this Linux container — not yet checked against main on the same box, so its cause is unattributed so far.
  • Mechanism health (1080p, 3 x 360 s runs, RX 9070 XT / RADV 26.2.0-devel): 21,566 async-ledger frames, present-fence max 0.032 ms all-time (0 frames over 2 ms), acquire max 0.079 ms, fifo_max 1; no faults, no device lost, no validation lines.
  • What is not yet measured here: the fps verdict on current main. The first A/B (1080p, 3 x 360 s) was confounded: the presenter paces presents to the display's vblank grid (console-faithful vi2 behavior) while main's presents run unpaced (RX_VK_IMMEDIATE) and above the 60 Hz refresh on this GPU — main 65.30 median, port 60.00 flat in every run. That pair says nothing about the readback win; a 4K A/B where both arms sit below refresh is running now and its numbers will land in this description as a comment.
  • History and provenance: this is the re-based continuation of the async-render campaign's branch. On its original base (before the readback-aware submit flushing and the occlusion-count fix) the full config measured +4.33% median fps (95% CI [+3.05..+5.66], 131 pooled gameplay windows over 3 runs, RX 9070 XT, RADV), zero corruption, coherence intact (0 of 26,319 readback lookups fell back to the submission wait). That measurement does not carry to current main — which is exactly why the 4K A/B is the claim that matters. The original draft PR was orphaned when the project moved off GitHub.
  • A design note that fell out of the confound: with RX_ASYNC_RENDER on, presents pace to the display's refresh; at loads above refresh that reads as a cap next to main's unpaced presents. An opt-out (present unpaced) may be worth a switch of its own if wanted.

Row added to docs/tested-configurations.md for the 1080p runs.

**Why:** the game's job threads stall on readbacks — invalidate()'s wait goes through the replay queue's catch-up, and every readback rode whole submissions until the per-part wait landed. This carries the async-render work as optional, default-off switches so it can be measured alone: a deeper present ring whose presenter paces presents on the display's vblank grid, and readback delivery that copies written binds into guest memory right after their pass, event-gated, so the CPU polls an event instead of waiting the submission. **What it does** (every switch default OFF — with them off, main's paths run untouched): - `RX_ASYNC_RENDER=1` (+ `RX_ASYNC_FIF`, 3..8, default 4): the present ring keeps N frame slots, the game's release event fires from its present (host/nvn_native.c, `rx_release_due`), the presenter thread lands those presents on the display's vblank grid, and invalidate()'s readbacks wait only their own write-back's timeline value — per-value ordering, never the replay queue. A phase-2 retry lets a submission parked at an event wait go on before the wait re-runs. A per-frame poll-only ledger (`vk: async f=`) measures every remaining wait; a per-frame summary reports fence/acquire waits and busy. - `RX_WB_RB_EVENTS=1`: written binds whose range touches the readback history (the meters and counters the game reads every frame) are delivered into guest memory right after the pass that binds them — their write-back copy made early, each with its own event (`memory_rb_deliver`, `memory_writeback_event`); the CPU's readback polls the event before waiting the covering value. - Also carried, default off, kept for separate measurement: `RX_VK_CAP_EAGER` (the capture's pool list includes the mirror-wanted and not-yet-bound pools — closes the mirror-sync live-read race at import), `RX_WB_RB_DIRECT` (prototype: readback pages copy straight from the mirror), `RX_VK_RB_EVER_PAGES`, `RX_VK_COND_CPU_WORDS`. **Testing status: incomplete — a draft on purpose.** - Suite: 518 passed. `test_readback_part.py`'s gate follows the ported choice block in this branch (the fixture stubs the async additions with the switch off; every assertion on the per-part wait is unchanged). `test_appimage_package_retry` fails in this Linux container — not yet checked against main on the same box, so its cause is unattributed so far. - Mechanism health (1080p, 3 x 360 s runs, RX 9070 XT / RADV 26.2.0-devel): 21,566 async-ledger frames, present-fence max 0.032 ms all-time (0 frames over 2 ms), acquire max 0.079 ms, fifo_max 1; no faults, no device lost, no validation lines. - What is **not** yet measured here: the fps verdict on current main. The first A/B (1080p, 3 x 360 s) was confounded: the presenter paces presents to the display's vblank grid (console-faithful vi2 behavior) while main's presents run unpaced (`RX_VK_IMMEDIATE`) and above the 60 Hz refresh on this GPU — main 65.30 median, port 60.00 flat in every run. That pair says nothing about the readback win; a 4K A/B where both arms sit below refresh is running now and its numbers will land in this description as a comment. - History and provenance: this is the re-based continuation of the async-render campaign's branch. On its original base (before the readback-aware submit flushing and the occlusion-count fix) the full config measured **+4.33% median fps (95% CI [+3.05..+5.66], 131 pooled gameplay windows over 3 runs, RX 9070 XT, RADV)**, zero corruption, coherence intact (0 of 26,319 readback lookups fell back to the submission wait). That measurement does not carry to current main — which is exactly why the 4K A/B is the claim that matters. The original draft PR was orphaned when the project moved off GitHub. - A design note that fell out of the confound: with `RX_ASYNC_RENDER` on, presents pace to the display's refresh; at loads above refresh that reads as a cap next to main's unpaced presents. An opt-out (present unpaced) may be worth a switch of its own if wanted. Row added to docs/tested-configurations.md for the 1080p runs.
The CPU frame loop round-tripped on a 3-frame-old present fence and on the
48-slot recording ring, and nothing reported how far ahead of the GPU the CPU
actually ran, so "is the replay asynchronous?" could only be argued, not read.

With RX_ASYNC_RENDER=1 the present path keeps RX_ASYNC_FIF frame slots (default
4, clamped 3..8; 3 when the switch is unset), so the fence a present waits for
belongs to a slot several frames back, and every wait this path can take is
timed on its thread and reported once a frame ("vk: async f= ... key=value",
plus a 60-frame "vk: async summary" in the format results/glm's async-analyze.py
already reads): present fence, acquire, recording-slot, nvnSyncWait,
nvnQueueFinish, the readback's write-back wait, and the replay's own phase
totals, with progress polled from timeline/rec_timeline (never waited on).

Hardware: nvdrv.c's "GPFIFO submissions run as they are made" with a deep
command-buffer queue (Lichii NX2_NOTES row 59: ~96 CBs in flight), cross-queue
sync kept GPU-side as timeline releases, and nvnQueueWaitSync staying an
ordered no-op for the CPU.

Measured (RX 9070 XT, 6,200+ gameplay frames per run): lagv (command buffers
CPU ahead of GPU) >= 4 for 4,172 consecutive frames, fifo >= 1 throughout; the
render path's worst blocking wait 0.663 ms (budget 2 ms); the host's own
"nvn: GPU waits in 5 s" showing 0 sync waits / 0 ms; stock GPU-bound wait
0.1 ms both with and without the flag; fps at parity (median 51.9 vs 50.5).

Unset is stock: nothing else changes, frames_n stays 3, no timers run, no
per-frame line prints, and the suite matches pristine 55b976c0 exactly
(440 passed, 1 failed (upstream's own test_env_on), 33 skipped). Two test
needles were updated for the loop's renamed index variable and to stub the
ledger's timing line; their checks are unchanged.

Co-authored-by: GLM-5.3-Flash <[email protected]>
The async-render race's best measured pieces, merged into one flag-gated branch
(default OFF = byte-identical to stock; every piece measured or measurable alone):

- RX_ASYNC_RENDER (mimo2 7dc628df + the phase-3 port): the present ring keeps
  RX_ASYNC_FIF frame slots (default 4, clamped 3..8), the game's present lands on
  the display's vblank grid on the presenter thread, the frame-release event fires
  from the present, and invalidate()'s buffer and texture paths wait only their
  own write-back's timeline value (per-value ordering, never the replay queue),
  with the phase-2 retry for a submission parked at an event wait. A per-frame
  poll-only ledger (`vk: async f=`) measures every remaining wait.

- RX_WB_MODE=pervalue (the wb-minimize prototype): in sync mode the readback skips
  the queue-wide catch-up when its covering value is known and no stopped work
  holds the queue; a bounded (5x) re-lookup follows the value while recording
  proceeds. `off` is the measurement-only no-wait arm.

- RX_WB_RB_DIRECT=1 (variant B2): the pages the CPU reads back copy straight from
  the mirror out of the write-back's diff pass — a partition of the runs, never a
  dropped byte. A pool's shadow decision sees the readback share first (no shadow
  churn for readback pools), and the write-back's tail barrier makes the copies'
  TRANSFER writes host-visible explicitly (the stock path relies on its
  vkWaitSemaphores and stays byte-identical).

- RX_WB_RB_EVENTS=1 (the wb-minimize prototype): written binds whose range touches
  the readback history are delivered right after their producing pass, each with
  an event signaled then; the readback polls the event (<=30 ms) before falling
  back to the timeline wait, and the end-of-submission write-back skips delivered
  ranges so the game's post-submit meter reset is not stomped. The event lookup
  scans newest-first (two deliveries of one submission: the last one holds its
  final words for the range). Off under a parallel replay's segments.

- RX_VK_RB_EVER_PAGES=1: tier two of the readback history — a page whose 600-epoch
  TTL expired stays known per pool and refreshes again while the GPU binds that
  pool, closing the pause/menu window where a long-unread page's untracked CPU
  write (the exposure meter's reset) would miss the refresh.

- RX_VK_COND_CPU_WORDS=1: render conditions read cpu_words-first — the replay's
  own CPU-written words live, anything else as-submitted, never racing the game's
  post-submit rewrite of the variable.

Excluded on measurement: the full replay-thread split (-4% as built), the
present-to-release ~0 ms claim (window-sum artifact; the mechanism stays, the
claim does not).

Tests: 440 passed, 1 failed (upstream's own test_env_on/HOS_FPS_LOG), 33 skipped
= the pristine 55b976c0 baseline exactly; test_recorder_sync covers the new
vkCmdSetEvent2 packet and the vkResetEvent boundary, test_parallel_rb_refresh and
test_direct_buffer_writes carry the new bind-tail stubs.

Co-authored-by: GLM-5.3-Flash <[email protected]>
A pool's first submission writes its pages while it is still mirror-less (the capture that
ran at that submission's start did not know it — the pool was not imported yet), and when
its mirror is made, mirror_sync live-reads those writes from guest memory, racing the game's
threads: the tearing the captures exist to prevent. Measured: the scene-load windows
live-read 1.5-4.1k pages while every steady-state window reads 0. The eager list publishes at
the import (both the background drain and the on-demand path); MIRROR_REFUSED stays out (the
GPU binds its guest memory, coherent by construction).

Co-authored-by: GLM-5.3-Flash <[email protected]>
Serving a stale snapshot for the tiny slow-adapting sums before the submission's delivery
lands would let the game read its own reset (it writes 0 right after every read), and the
late delivery then stomps that reset — the read -> reset -> rb_refresh accumulation chain
breaks and the meter's sum diverges. The console's counter lives GPU-side (a DMA'd
snapshot); ours lives in the guest memory the delivery writes. The edit was withdrawn
before any run; the comment keeps the analysis where the next implementer will find it.

Co-authored-by: GLM-5.3-Flash <[email protected]>
# Conflicts:
#	gpu/vk/vk_draw.cpp
#	gpu/vk/vk_internal.h
#	gpu/vk/vk_memory.cpp
#	gpu/vk/vk_replay.cpp
#	gpu/vk/vk_work.h
#	tests/direct_buffer_writes_test.cpp
The port adds the async additions (AsyncTimer ledger timing, the RX_WB_RB_EVENTS
early-delivery poll, the parked-wait retry) to the block this test cuts out; the
marker follows the new text and the fixture stubs the additions with the switch
off, so the fixture still exercises the sync per-part wait it exists for.
Every assertion on the choice block is unchanged.

Co-authored-by: GLM-5.3-Flash <[email protected]>
docs(tested): the 1080p A/B rows for the async-port — mechanism health recorded, the pacing confound stated, the 4K verdict pending
Some checks failed
Linux AppImage / appimage (pull_request) Blocked by required conditions
Linux AppImage / Release asset (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (windows-2022, part 1/2) (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (windows-2022, part 2/2) (pull_request) Blocked by required conditions
Game-free source regressions / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
Game-free source regressions / Vulkan fixtures (lavapipe, validation) (pull_request) Blocked by required conditions
Game-free source regressions / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Windows builder / Hookshot Builder package and setup (pull_request) Blocked by required conditions
Windows builder / Windows Vulkan back end (llvm-mingw) (pull_request) Blocked by required conditions
Windows builder / dav1d for Windows (llvm-mingw) (pull_request) Blocked by required conditions
Windows builder / Release assets (pull_request) Blocked by required conditions
PR title / Conventional Commit title (pull_request_target) Has been cancelled
4b60fb3d9c
Co-authored-by: GLM-5.3-Flash <[email protected]>
Author
First-time contributor

4K interim results (1 of 3 pairs) — parity so far.

At 4K both arms sit below the display's refresh, so the vblank-pacing confound from the 1080p runs (above) does not apply. First pair, 360 s each, warp-normalized Hyrule Field (windows = D >= 1000 draws):

arm windows median fps
main db35b122 (base-r1) 115 51.30
this branch, switches on (port-r1) 116 51.30

Delta +0.00%, bootstrap 95% CI [-0.19 .. +0.20] (20k resamples). 19,205 async-ledger frames; mechanism health at 4K: present-fence max 0.047 ms all-time (0 frames over 2 ms), acquire max 0.160 ms, fifo_max 1, busy 69%; no faults, no device lost, no validation lines.

So far the readback-delivery win this branch carried from its original base does not reproduce on current main — consistent with main having absorbed the stall since (#2422's readback-aware submit flushing, the per-part wait, the occlusion-count fix). Two more pairs are running; the pooled 3-pair verdict lands here when they finish.

**4K interim results (1 of 3 pairs) — parity so far.** At 4K both arms sit below the display's refresh, so the vblank-pacing confound from the 1080p runs (above) does not apply. First pair, 360 s each, warp-normalized Hyrule Field (windows = D >= 1000 draws): | arm | windows | median fps | |---|---|---| | main db35b122 (base-r1) | 115 | 51.30 | | this branch, switches on (port-r1) | 116 | 51.30 | Delta +0.00%, bootstrap 95% CI [-0.19 .. +0.20] (20k resamples). 19,205 async-ledger frames; mechanism health at 4K: present-fence max 0.047 ms all-time (0 frames over 2 ms), acquire max 0.160 ms, fifo_max 1, busy 69%; no faults, no device lost, no validation lines. So far the readback-delivery win this branch carried from its original base does not reproduce on current main — consistent with main having absorbed the stall since (#2422's readback-aware submit flushing, the per-part wait, the occlusion-count fix). Two more pairs are running; the pooled 3-pair verdict lands here when they finish.
Author
First-time contributor

Measurement stopped here (author's call) — 1 of 3 pairs, and the verdict it gives is the honest one: parity.

The remaining pairs were cut: the first 4K pair (above) is +0.00% with a 95% CI of [-0.19 .. +0.20] over 231 pooled windows, and no further run was going to move a bound that tight. The readback-stall win this branch carried from its original base (+4.33% there) does not reproduce on current main — main absorbed the stall since, with #2422's readback-aware submit flushing, the per-part wait, and the occlusion-count fix. That is main getting better, and this branch declining to claim otherwise.

What the branch still offers, for the maintainers' judgment:

  • The mechanism is real and healthy: 40k+ async-ledger frames across both resolutions with present-fence max 0.047 ms, 0 frames over 2 ms, no faults, no validation lines, all switches default-off with main's paths untouched (suite 518 passed; the readback-part gate follows the ported block in-branch).
  • Console-faithful present pacing: with RX_ASYNC_RENDER on, presents land on the display's vblank grid (vi2 behavior). At loads above refresh that reads as a cap next to main's unpaced presents; an unpaced opt-out would be the switch to want if this ever comes back.

Leaving this open as WIP rather than closing: the port recipe (single merge onto main, one resolution pass), the gate fix and the pacing finding are on the record, and the branch is one command away from a re-measure if main's balance shifts again. If it is just noise at this point, close it with a clear conscience — the numbers above are the whole claim.

**Measurement stopped here (author's call) — 1 of 3 pairs, and the verdict it gives is the honest one: parity.** The remaining pairs were cut: the first 4K pair (above) is +0.00% with a 95% CI of [-0.19 .. +0.20] over 231 pooled windows, and no further run was going to move a bound that tight. The readback-stall win this branch carried from its original base (+4.33% there) does not reproduce on current main — main absorbed the stall since, with #2422's readback-aware submit flushing, the per-part wait, and the occlusion-count fix. That is main getting better, and this branch declining to claim otherwise. What the branch still offers, for the maintainers' judgment: - The mechanism is real and healthy: 40k+ async-ledger frames across both resolutions with present-fence max 0.047 ms, 0 frames over 2 ms, no faults, no validation lines, all switches default-off with main's paths untouched (suite 518 passed; the readback-part gate follows the ported block in-branch). - Console-faithful present pacing: with RX_ASYNC_RENDER on, presents land on the display's vblank grid (vi2 behavior). At loads above refresh that reads as a cap next to main's unpaced presents; an unpaced opt-out would be the switch to want if this ever comes back. Leaving this open as WIP rather than closing: the port recipe (single merge onto main, one resolution pass), the gate fix and the pacing finding are on the record, and the branch is one command away from a re-measure if main's balance shifts again. If it is just noise at this point, close it with a clear conscience — the numbers above are the whole claim.
Author
First-time contributor

In conclusion, with the current, there is probably not a whole lot to gain performance-wise from this PR. I'll leave it here for reference in case any ideas here have some value. Otherwise, feel free to close it.

In conclusion, with the current, there is probably not a whole lot to gain performance-wise from this PR. I'll leave it here for reference in case any ideas here have some value. Otherwise, feel free to close it.
Some checks failed
Linux AppImage / appimage (pull_request) Blocked by required conditions
Linux AppImage / Release asset (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (ubuntu-24.04, part 1/2) (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (ubuntu-24.04, part 2/2) (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (windows-2022, part 1/2) (pull_request) Blocked by required conditions
Game-free source regressions / Python source checks (windows-2022, part 2/2) (pull_request) Blocked by required conditions
Game-free source regressions / SPIR-V fixtures (global narrow load/store, direct surface atomics, F2I immediate) (pull_request) Blocked by required conditions
Game-free source regressions / Vulkan fixtures (lavapipe, validation) (pull_request) Blocked by required conditions
Game-free source regressions / Workflow lint (actionlint) (pull_request) Blocked by required conditions
Windows builder / Hookshot Builder package and setup (pull_request) Blocked by required conditions
Windows builder / Windows Vulkan back end (llvm-mingw) (pull_request) Blocked by required conditions
Windows builder / dav1d for Windows (llvm-mingw) (pull_request) Blocked by required conditions
Windows builder / Release assets (pull_request) Blocked by required conditions
PR title / Conventional Commit title (pull_request_target) Has been cancelled
This pull request has changes conflicting with the target branch.
  • docs/development.md
  • gpu/vk/vk_internal.h
  • gpu/vk/vk_memory.cpp
  • gpu/vk/vk_replay.cpp
  • host/nvn_native.c
  • tests/test_readback_part.py
Some workflows are waiting to be reviewed.
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u 'async-port:blessedplummer-async-port'
git switch 'blessedplummer-async-port'

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 'main'
git merge --no-ff 'blessedplummer-async-port'
git switch 'blessedplummer-async-port'
git rebase 'main'
git switch 'main'
git merge --ff-only 'blessedplummer-async-port'
git switch 'blessedplummer-async-port'
git rebase 'main'
git switch 'main'
git merge --no-ff 'blessedplummer-async-port'
git switch 'main'
git merge --squash 'blessedplummer-async-port'
git switch 'main'
git merge --ff-only 'blessedplummer-async-port'
git switch 'main'
git merge 'blessedplummer-async-port'
git push origin 'main'
Sign in to join this conversation.
No description provided.