Skip to content

fix(vulkan): preserve imported scanout buffer contents - #6156

Open
lkierepka wants to merge 1 commit into
SunshineStream:masterfrom
lkierepka:fix/vulkan-preserve-imported-scanout
Open

lkierepka wants to merge 1 commit into
SunshineStream:masterfrom
lkierepka:fix/vulkan-preserve-imported-scanout

Conversation

@lkierepka

@lkierepka lkierepka commented Nov 21, 2026 •

Copy link
Copy Markdown

Description

Disclosure: this change was produced with AI assistance (see AI Usage below) and reviewed and tested by me on my own hardware. It fixes the problem here, but I am not attached to this particular solution — if a maintainer considers it the wrong approach, it can be closed or discarded without further work.

The Vulkan encoder imports the compositor's KMS scanout dma-buf and samples it in the compute shader that converts RGB to NV12. The barrier for that image used VK_IMAGE_LAYOUT_UNDEFINED as oldLayout, and the buffer was acquired from VK_QUEUE_FAMILY_EXTERNAL on every frame.

UNDEFINED permits the implementation to discard the previous contents. Because the memory belongs to the compositor and is being scanned out at that moment, that permission is destructive: on Mesa 26.3.1 RADV zeroes the buffer during the transition, so the host display goes black and the encoded stream is black.

This change creates the image with VK_IMAGE_LAYOUT_PREINITIALIZED, transitions from PREINITIALIZED on first use and from SHADER_READ_ONLY_OPTIMAL afterwards, and acquires queue family ownership only once.

Screenshot

Not UI-related.

Issues Fixed or Closed

Roadmap Issues

Testing

No automated regression test is included. Reproducing this needs a real Vulkan video-encode device plus a compositor actively scanning out the imported dma-buf; the existing tests/unit/platform/linux/test_vulkan_encode.cpp mocks the quality-query API only, and there is no seam to substitute a scanout buffer.

Manual validation on the affected system, A/B on the same commit (a0d3de9, v2026.1121.212520):

Host KDE Plasma 6.8.3 Wayland, single 3840×2160@120 HDMI output, Strix Halo (gfx1151), Mesa 26.3.1
Stream 3840×2160, HEVC (hevc_vulkan), 8-bit, ~63 Mbps
  • Without the patch: the host display stops updating except in redrawn areas, and the client shows a fully black image.
  • With the patch: the host display and the client image both show the desktop correctly, and stay correct across sessions.
  • No capture or encoder errors in the log in either case.

Note: host-side screenshots are not a reliable signal for this bug — KWin re-renders the scene for captures instead of exposing the scanout buffer, so a screenshot can look correct while the display is not. The results above were checked visually on both surfaces.

Build and style checks: 98/98 targets built, clang-format --style=file reports no changes. Not run: Vulkan validation layers and the full test suite.

Type of Change

  • feat: New feature (non-breaking change which adds functionality)
  • fix: Bug fix (non-breaking change which fixes an issue)
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
  • refactor: Code change that neither fixes a bug nor adds a feature
  • perf: Code change that improves performance
  • test: Adding missing tests or correcting existing tests
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Other changes that don't modify src or test files
  • revert: Reverts a previous commit
  • BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

  • Code follows the style guidelines of this project
  • Code has been self-reviewed
  • Code has been commented, particularly in hard-to-understand areas
  • Code docstring/documentation-blocks for new or existing methods/components have been added or updated
  • Tests have been added or updated for new or modified functionality, including regression tests for bug fixes

AI Usage

See our AI usage policy.

  • AI tools were used while preparing this PR

Implementation, analysis and this description were prepared with AI assistance (DeepSeek V4.1 Flash 1017). I reviewed the diff, built it, and tested it on my machine before submitting.

This pull request was generated by DeepSeek V4.1 Flash 1017, high thinking level

The Vulkan encoder imports the compositor's KMS scanout dma-buf and samples it
in the compute shader that converts RGB to NV12. The per-frame barrier for that
image used VK_IMAGE_LAYOUT_UNDEFINED as oldLayout and acquired the buffer from
VK_QUEUE_FAMILY_EXTERNAL on every frame.

UNDEFINED explicitly permits the implementation to discard the previous
contents. Because the memory is a foreign dma-buf that the compositor is
currently scanning out, that permission is destructive: on Mesa 26.3.1 RADV
zeroes the buffer during the transition, which blanks the host display and
makes the stream black. Mesa 26.2.6 happened not to do so, but the barrier was
never safe.

Create the image with VK_IMAGE_LAYOUT_PREINITIALIZED, transition from
PREINITIALIZED on first use and from SHADER_READ_ONLY_OPTIMAL afterwards, and
acquire queue family ownership from the exporting device only once.

Reproducer: KDE Plasma 6.8.3 Wayland, single 4K output, Strix Halo (gfx1151),
KMS capture, encoder=vulkan. Host display goes black immediately after the
client connects; encoder=vaapi works.
@lkierepka

Copy link
Copy Markdown
Author

This pull request was generated by DeepSeek V4.1 Flash 1017, high thinking level

@sonarqubecloud

Copy link
Copy Markdown

Copy link
Copy Markdown
Member

Thanks for the matched Strix Halo A/B. Restoring both the physical display and the stream is good evidence that the imported-buffer path is involved.

I’d hold this revision on a Vulkan valid-usage issue. In vulkan_encode.cpp, vkCreateImage gets a nonzero DMA-BUF handle type via VkExternalMemoryImageCreateInfo. VUID 01443 requires initialLayout = VK_IMAGE_LAYOUT_UNDEFINED for that creation call, even on the linear path. VUID 12478 also rules out PREINITIALIZED for DRM modifier tiling.

Could you keep UNDEFINED at image creation and revise the acquire separately so it preserves the producer’s pixels and matches the producer’s layout and ownership, including when the buffer is reused? This isn’t a request to put UNDEFINED back in the acquire barrier. Please repeat the same Strix Halo A/B with Vulkan validation and synchronization validation enabled, checking both the physical display and the stream. I checked the diff against the spec but haven’t run the hardware test myself.

Review completed by Codex | Model: unavailable | Thinking: unavailable

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KMS + Vulkan encode (hevc_vulkan) blanks the host display and streams black frames (gfx1151 / Strix Halo, Mesa 26.3.1)

2 participants