Skip to content

GX fog on the NV2A (per-vertex, from GX's own fog registers) - #1

Merged
TechProGabe merged 1 commit into
mainfrom
claude/ecstatic-dijkstra-12uzub
Sep 30, 2026
Merged

TechProGabe merged 1 commit into
mainfrom
claude/ecstatic-dijkstra-12uzub

Conversation

@TechProGabe

Copy link
Copy Markdown
Owner

What

Until now GXSetFog recorded its arguments and the back end ignored them. This PR draws every GX fog type (GX_FOG_PERSP_* / GX_FOG_ORTHO_* × LIN, EXP, EXP2, REVEXP, REVEXP2; GX_FOG_NONE is off).

GX semantics (libogc GX_SetFog, Dolphin PixelShaderGen::WriteFog)

  • GXSetFog writes A and C as floats cut to 11 mantissa bits. Perspective fog also gets B as a 24-bit b_mag (B_mant * 8388638) plus a b_shift.
  • Per pixel, from the 24-bit EFB depth Zs:
    • perspective: ze = A·2²⁴ / (b_mag − (Zs >> b_shift)), which is eye depth / (end − start) when fog near/far match the projection's;
    • ortho: ze = A·Zs/2²⁴.
  • Then x = clamp(ze − C, 0, 1) and the curve: LIN x, EXP 1−2^(−8x), EXP2 1−2^(−8x²), REVEXP 2^(−8(1−x)), REVEXP2 2^(−8(1−x)²).
  • Fog is applied after TEV, to RGB only: (rgb·(256−F₂₅₆) + fog·F₂₅₆) >> 8.
  • HSD passes the current camera's near/far, so fog and projection always match in Melee.

Mapping onto the NV2A

  • nv2a_fog.c (new). Rebuilds exactly the registers GXSetFog would write. Zs/2²⁴ = (depth row · P)/(w row · P), so clamp input ze − C (or 1 − (ze − C) for REV) is a ratio of two linear functions of the view-space position. Those two rows plus a curve row go into constants VPC_FOG..+2 (134–136).
  • Vertex program. VpKey.fog (off / linear / exp / exp2, taking a former pad byte) adds 4 (LIN) to 10 (EXP2) instructions writing oFog.x. EXP types clamp and use expp.
  • Fog unit. FOG_MODE LINEAR with FOG_PARAMS (1, 1, 0) passes oFog.x through. The linear mode computes p0 + p1·c − 1; this is the setup nxdk_pgraph_tests uses for vertex-shader fog, and it's what xemu implements. The gen mode doesn't matter with a vertex program. The NV2A's EXP modes aren't used: they're evaluated per vertex as well, and hardware measures EXP's zero point at ~1.51.
  • Final combiner. CW0 becomes A = FOG.a, B = FOG.rgb, C = PREV, so the result is F·fog + (1−F)·prev. Alpha (CW1's G) is untouched. SET_FOG_COLOR is sent ABGR with alpha 0.
  • Cost. Fog state goes out only on XGX_DIRTY_FOG, or on a projection change while fog is on. GXSetFog now returns early on unchanged values like the other setters, because HSD calls it per camera pass and per particle kind and each call used to flush the waiting batch. No per-draw CPU cost is added. xgx_draw, the vertex-buffer paths, gx_vtx.c and gx_tex.c are untouched: the fog work hangs off emit_fixed, build_proj, derive_vk and emit_combiners.
  • Unfogged output is unchanged. CW0 without fog is the compiled rp->cw0, byte for byte. Fog-off vertex programs are identical to before: I checked 200,000 random keys, old generator against new.

Known limits

  • Fog is per vertex. GX evaluates it per pixel, the NV2A interpolates. For LIN this is exact when GX's ze is affine in eye depth. With Melee's 0.1..16384 game camera, GX's own b_mag rounding (8388638/2²³) grows its denominator by ~57% from near to far. A polygon spanning a long depth range then gets less fog mid-span than on the GameCube: up to ~23/255 on an edge from depth 2000 to 14000 through a narrow fog range. EXP curves are linear between vertices.
  • Replicating the registers exactly is still right. That same rounding makes GX's ze up to 1.57× smaller at the far plane than the textbook formula, and the port matches that at every vertex.
  • GXSetFogRangeAdj isn't applied. It's a horizontal radial-distance correction that HSD sets when a fog has a FogAdj descriptor.
  • A program already near the 136-instruction limit drops lights slightly sooner with fog.

Tests

  • New tools/xbox/test_fog.py + tests/xbox/test_fog.c.
    • Reference GX: libogc's register math plus Dolphin's per-pixel formula on the integer 24-bit depth.
    • Port: build_proj's rows → fog_setup → the vertex program's float arithmetic → the NV2A LINEAR factor.
    • Coverage: every type, perspective and ortho, 6 cameras including Melee's, 5 fog ranges including reversed ones, Z24 and Z16, a non-0..1 viewport depth range, and the odd cases (start == end).
    • At every vertex the port is within 1/255 of GX over one of GX's own depth steps. With Melee's camera GX has only 99 steps at depth 5000 and 41 at the far plane. The worst excess is 0.29/255.
    • LIN edges are checked with perspective-correct interpolation wherever b_mag doesn't bend ze (<1%). The Melee-camera edge deviation is reported, not asserted.
    • The final combiner equation is checked against GX's 8-bit blend: worst 1.40/255.
    • 613,944 checks, 0 failed. Added to CI.
  • test_vp_encoder.py now also covers expp and a MAD reading one constant row twice: 17 instructions match nv2a-vsh.
  • test_tex_convert.py passes.
  • tools/lower/test_lower.py was not run. It needs LLVM 21 + disc_lower; the host has LLVM 18, disc_lower doesn't build against 18's API, and apt.llvm.org is blocked by this environment's egress policy. This PR doesn't touch tools/lower or any lowered game code.

Build

tools/xbox/docker/build.sh was not run, so the XBE link is unverified here. The SDK image build failed: Docker Hub rate-limited debian:trixie, and apt inside the container was blocked by the proxy. This PR's CI builds default.xbe and runs the host tests, so the link check happens there.

As a substitute I compiled against nxdk at the pinned SHA using host clang 18 with nxdk-cc's flags and the project's hw flags:

  • nv2a.c (with and without -DXGX_DEBUG_TRACE), nv2a_fog.c, nv2a_vp.c and nv2a_rc.c compile warning-free under -Wall for i386-pc-win32. nv2a_fog.obj needs only __fltused.
  • gx_state.c compiles with the SDK's game-triple flags.

No imported game code was edited (no PORT: changes), and xgx.h is unchanged.

Please check on the console against Dolphin

-DXGX_DEBUG_TRACE now logs each draw's fog state.

  1. Stages. Ground_801C1E94 loads each stage's fog descriptor (start/end scaled by the stage's param->y; its colour is also the clear colour), and Ground_801C1E2C applies it per camera pass. Compare distant backgrounds, large floor or background polygons (the per-vertex limit above) and particles on several stages. Final Destination turns its fog off during its transition (grlast.c). Particles whose kind has DispFog (psdisp.c) switch fog per kind.
  2. Title screen (ScTitle_fog).
  3. Classic mode "vs." intros (gm_1832.c) and the All-Star intro (gm_186E.c).
  4. Adventure cutscenes: vi0102, vi0401, vi0501, vi0502, vi0801, and vi1201v1/v2 (Bowser).
  5. Tournament mode screens (gmtou_*.c), the 1P ending display (gmregenddisp.c), the staff roll.
  6. Trophy display (ScMenDisplay_fog, tydisplay.c). toy.c's own white fog is debug-ROM only.
  7. Regression check. A standard Green Greens 4-CPU run should look identical wherever that stage has no fog, and [PERF] shouldn't move.

Docs: renderer.md (new Fog section, constant rows), decisions.md, testing.md, roadmap.md, CLAUDE.md test list.

🤖 Generated with Claude Code

https://claude.ai/code/session_016pxfm8RyF7fCMgQdaKigij


Generated by Claude Code

GXSetFog was recorded and ignored (fog off). Now every GX fog type is drawn:

- nv2a_fog.c rebuilds the registers GXSetFog writes (A and C cut to 11
  mantissa bits, b_mag, b_shift; libogc GX_SetFog) and turns them into two
  vertex-program rows: GX's clamp(ze - C) is a ratio of two linear
  functions of the view-space position, since the EFB depth is
  (depth row . P) / (w row . P). The vertex program (VpKey.fog: linear,
  exp, exp2) evaluates it per vertex and, for the EXP types, the curve with
  expp; rows VPC_FOG..+2.
- The NV2A's fog unit runs LINEAR with FOG_PARAMS (1, 1, 0), which passes
  oFog.x through (the linear mode subtracts 1), clamped per pixel; the final
  combiner's CW0 becomes F * fog colour + (1 - F) * PREV, alpha untouched.
- Fog state is sent only when GXSetFog changed something (it now returns
  early on equal values like the other setters) or the projection changed
  while fog is on. Unfogged vertex programs and CW0 are unchanged.
- tools/xbox/test_fog.py checks the port against a reference GX (libogc's
  registers, Dolphin's per-pixel formula on the 24-bit depth) for every
  type, both projections and several cameras including Melee's 0.1..16384:
  within 1/255 of GX over one of its depth steps at every vertex; LIN edges
  where GX's b_mag rounding doesn't bend ze; the final combiner's blend.
- test_vp_encoder.py covers expp and a MAD reading one constant twice.

GXSetFogRangeAdj stays unapplied. Docs: renderer.md (Fog section, rows),
decisions.md, testing.md, roadmap.md; CI runs the new test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pxfm8RyF7fCMgQdaKigij
@TechProGabe
TechProGabe merged commit 3848012 into main Sep 30, 2026
1 of 2 checks passed
@TechProGabe
TechProGabe deleted the claude/ecstatic-dijkstra-12uzub branch September 30, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants