Dewarping without DirectShow

[Aidan63]

Hello,

We're starting development on a new product which will be using an Axis camera and are planning to integrating the dewarping into our existing playback software. The solution provided by the SDK being based on DirectXShow poses some problems.

• We have our own video pipeline which uses libav, we don't really want to have to pull in legacy DirectX libraries for dewarping.
• We're looking to moving to a full GPU drive video pipeline, based on modern D3D (11 & 12). Needing to round trip to the system memory or figure out how to integrate DirectXShow into new D3D sounds like it's going to be a pain.
• At some point we will likely want to perform dewarping on non-Windows systems, d3d9 shader seem to be a key part of the dewarping which is a blocker for this.

I have a few questions based on these concerns.

• Does the D3D9 shader perform the "core" of the dewarping and is it possible to directly use it? We currently use WPF for UI which is D3D9 based, so if we knew what the inputs were could we create a WPF shader which performs the dewarping?
• Would it be possible to get D3D11 and D3D12 compatible versions of the shaders? I don't want to have to use the D3D9Ex sharing more than I have to.
• Is there a way to get a CPU version of the dewarping, specifically one which would run on non-Windows systems (maybe something opencv based)? A Cuda kernel or SPIR-V bytecode would also work here for accelerated non-Windows solutions.

Having access to the HLSL shader source code would probably solve most of these, I'd imagine it would be fairly straight forward to port to newer pixel shader versions and get SPIR-V code using the new D3D shader compiler.

Thanks,
Aidan

[github-actions[bot]]

This automatically generated reply acts as a friendly reminder.

Answers to your questions will most often come from the community, from developers like yourself. You will, from time to time, find that Axis employees answers some of the questions, but this is not a guarantee. Think of the discussion forum as a complement to other support channels, not a replacement to any of them.

If your question remains unanswered for a period of time, please revisit it to see whether it can be improved by following the guidelines listed in Axis support guidelines.

[vivekatoffice]

Hi Aidan,
Hi @Aidan63 ,

Thanks for the detailed questions. There may be a few options worth considering in the current AXIS Dewarping SDK before assuming that DirectShow needs to be part of the overall video pipeline.

The SDK does not require DirectShow for all integration scenarios. In addition to the DirectShow integration through the AXIS Dewarping Transform DMO, the SDK provides an image I/O library interface and includes pre-compiled pixel shaders for Direct3D integration. The SDK also includes C++ and C#/.NET sample code demonstrating these integration paths.

In particular, the pre-compiled pixel shaders are intended to allow the dewarping operation to make use of the graphics card, while the AXIS Dewarping Transform can be integrated as a software library rather than necessarily being tied to a DirectShow graph.

Therefore, if you already have your own libav/FFmpeg-based decoding pipeline, it may be worth looking at the image I/O and Direct3D integration examples in the SDK rather than using the DirectShow transform.

Regarding the D3D11/D3D12 and cross-platform questions: the publicly distributed SDK provides the pre-compiled shaders and documented integration interfaces, but the shader source/internal implementation details are not part of the public SDK interface. Consequently, Axis would not recommend relying on the D3D9 shader as a portable dewarping implementation.

The SDK download also contains the relevant samples and test recordings, which should help determine whether its image I/O or Direct3D integration can fit your existing pipeline.

For requirements that are not covered by the public SDK interfaces, please contact Axis integration helpdesk support or Professional Services so that we can assess the specific integration scenario.

Axis integration helpdesk support

Professional Services

[Aidan63]

Thanks for the quick response @vivekatoffice,

I initially skipped over the DewarpDirect3D sample as it uses the old D3DX library which is no longer part of the modern Windows (8+) SDK and is a bit of a pain to get running. Having now had a closer look it does seem like it should do what I want, I'm hoping it will be fairly simple to integrate into WPFs shader support which is also D3D9/PS 2.0.

Modern D3D pixel or compute shaders would still be very useful for us as there are several situations where we're not ultimately interfacing with D3D9 so would need to do D3D11 -> D3D9 sharing interop which is do-able but not ideal. I'd open an integration help desk ticket but I don't seem to have that option?

Non-Windows dewarping isn't an initial requirement for us but is likely to be useful down the road, so I'm happy with the D3D solution provided right now.

Thanks!

[vivekatoffice]

Hi @Aidan63,
I reached to one of the senior R&D team for your query. Please find the response below and I will make sure to provide update on this soon:

We had a similar case about a year ago where a customer made the same request. We discussed it back and forth, but unfortunately, we did not reach a conclusion at the time.

Several years ago, we updated VideoTransform.dll from DirectX 9 to DirectX 11, but the sample code was never updated accordingly.

There is some work required to bring the sample code up to date, but it should be relatively straightforward. Another option would be to share the HLSL source code. Personally, I believe this would be the best solution, although we would first need to investigate whether we can and want to share source code of this nature.

I will discuss this again next week with AXIS Dewarping SDK's Product Owner and the respective R&D Manager to determine how we should proceed.

---

What problem you are solving for End Customer or the product name you working on?

Regards,
@vivekatoffice
LinkedIn

[Aidan63]

Thanks @vivekatoffice,

Promising to hear internally there is updated code. I'm a software developer for EVCam and we've started working on a successor to our "RT120" system which we're planning on using one of your panoramic cameras for.
I'm probably not supposed to say anything in public but I'm happy to go into more details of system and the service it will provide as well as how we're planning on using one of the cameras over emails. I can be reached at "aidan.lee@evcam.com".

Thanks,
Aidan

[Aidan63]

Bit of an update of where I'm at.

None of the pixel shaders can be loaded by WPF even though they use a compatible shader model, all of them throw vague "Invalid user specified pixel shader" with no further details. So some argument passed to fxc when they were compiled has made them incompatible. Next I tried disassembling one of the shaders and re-compiling it with fxc, WPF was then able to load this but even when passing the lens, ptz, etc values copied from the examples I just get a black output. D3D9 is now so old that all of the current graphics debugging tools have no support for it, so that was a dead end.

I started and very quickly gave up manually using the DMO object to pass in buffers for dewarping for a few reasons. Modern Win32 C# projections such as Vortice do not expose DirectShow and other ancient libraries. DirectShowLib seems to be the nearest thing but it's very old, doesn't use modern C# interop features, and it's not clear how some bits of code in the C++ DewarpDMO sample would map onto this library. I don't want to add C++ to out projects as we've worked to remove all of it in favour of modern C# interop features over the last few years. Finally, I don't want to pull in a dependency to DirectShow and have to make sure our installers setup the dewarping COM dll properly.

My current working idea is to use D3D9 directly via Vortice to use the provided shaders to render to a quad and then download that back to system memory for encoding and share with WPF using the D3DImage control. The D3D sample uses the deprecated D3DX library which abstracts stuff like the register bindings, part of the previously mentioned disassembly did reveal these though, for anyone else who wants to know they are as follows for the main shader.

sampler2D input : register(s0);
float3 LensProfile : register(c5);
float2 imageCenter : register(c3);
float lambdaOffset : register(c2);
float3 rotateData : register(c4);
float2 size : register(c0);

This has all been very frustrating, while there's good functionality in the SDK it's reliance on Windows technologies which are ancient at this point makes it a pain to use and integrate with modern Windows APIs / C# / tooling. This SDK was probably state of the art in the Windows XP era, now it feels out of date and forgotten about and leaves a poor impression.
I wish I looked closer at this stuff when initially evaluating the axis camera, I probably would have pushed for a different solution if I realised all of this sooner (or at least gave myself more time). Sorry for having a whinge.

[Aidan63]

I seem to have something working, I also gave up with the D3D9 idea as PIX and RenderDoc no longer have D3D9 support and instead took the disassembled D3D9 shader and updated it to shader model 5.0 (basically just putting the constants into a cbuffer) so I could use it in D3D11, seems to work!

That's a RenderDoc view of the output render target, the source program is a small C# program I wrote using Vortice.Windows to provide D3D11 bindings (we use this library in our existing applications). Next step is to share the ID3D11Texture2D and display in WPF which should be trivial as we already have code for this. After that is the RTSP stream -> D3D11 VA Decode -> D3D11 Dewarp -> WPF pipeline which should also be pretty simple as we already have RTSP systems in use.