Repository navigation
Conversation
The Vulkan encoder imports the compositor's KMS scanout dma-buf and samples it in the compute shader that converts RGB to NV12. The per-frame barrier for that image used VK_IMAGE_LAYOUT_UNDEFINED as oldLayout and acquired the buffer from VK_QUEUE_FAMILY_EXTERNAL on every frame. UNDEFINED explicitly permits the implementation to discard the previous contents. Because the memory is a foreign dma-buf that the compositor is currently scanning out, that permission is destructive: on Mesa 26.3.1 RADV zeroes the buffer during the transition, which blanks the host display and makes the stream black. Mesa 26.2.6 happened not to do so, but the barrier was never safe. Create the image with VK_IMAGE_LAYOUT_PREINITIALIZED, transition from PREINITIALIZED on first use and from SHADER_READ_ONLY_OPTIMAL afterwards, and acquire queue family ownership from the exporting device only once. Reproducer: KDE Plasma 6.8.3 Wayland, single 4K output, Strix Halo (gfx1151), KMS capture, encoder=vulkan. Host display goes black immediately after the client connects; encoder=vaapi works.
|
|
|
Thanks for the matched Strix Halo A/B. Restoring both the physical display and the stream is good evidence that the imported-buffer path is involved. I’d hold this revision on a Vulkan valid-usage issue. In Could you keep Review completed by Codex | Model: unavailable | Thinking: unavailable |



Description
Disclosure: this change was produced with AI assistance (see AI Usage below) and reviewed and tested by me on my own hardware. It fixes the problem here, but I am not attached to this particular solution — if a maintainer considers it the wrong approach, it can be closed or discarded without further work.
The Vulkan encoder imports the compositor's KMS scanout dma-buf and samples it in the compute shader that converts RGB to NV12. The barrier for that image used
VK_IMAGE_LAYOUT_UNDEFINEDasoldLayout, and the buffer was acquired fromVK_QUEUE_FAMILY_EXTERNALon every frame.UNDEFINEDpermits the implementation to discard the previous contents. Because the memory belongs to the compositor and is being scanned out at that moment, that permission is destructive: on Mesa 26.3.1 RADV zeroes the buffer during the transition, so the host display goes black and the encoded stream is black.This change creates the image with
VK_IMAGE_LAYOUT_PREINITIALIZED, transitions fromPREINITIALIZEDon first use and fromSHADER_READ_ONLY_OPTIMALafterwards, and acquires queue family ownership only once.Screenshot
Not UI-related.
Issues Fixed or Closed
Roadmap Issues
Testing
No automated regression test is included. Reproducing this needs a real Vulkan video-encode device plus a compositor actively scanning out the imported dma-buf; the existing
tests/unit/platform/linux/test_vulkan_encode.cppmocks the quality-query API only, and there is no seam to substitute a scanout buffer.Manual validation on the affected system, A/B on the same commit (
a0d3de9,v2026.1121.212520):hevc_vulkan), 8-bit, ~63 MbpsNote: host-side screenshots are not a reliable signal for this bug — KWin re-renders the scene for captures instead of exposing the scanout buffer, so a screenshot can look correct while the display is not. The results above were checked visually on both surfaces.
Build and style checks: 98/98 targets built,
clang-format --style=filereports no changes. Not run: Vulkan validation layers and the full test suite.Type of Change
Checklist
AI Usage
See our AI usage policy.
Implementation, analysis and this description were prepared with AI assistance (DeepSeek V4.1 Flash 1017). I reviewed the diff, built it, and tested it on my machine before submitting.