Testing HLSL with the LLVM Offload Test Suite in WSL 2

If you are a developer working with HLSL or GPU compilers you likely prefer working in Linux. You might have felt sidelined because Linux lacks a dedicated DirectX device. However, if you are on Windows, you have a powerful tool at your disposal: Windows Subsystem for Linux (WSL2).

Since 2020, WSL 2 has supported DirectX, exposing the DxCore (libdxcore.so) and D3D12 (libd3d12.so) APIs. This bridge allows you to run the Offload test suite directly within your Linux environment by leveraging the underlying Windows hardware.

What is the Offload Test Suite?

The LLVM Offload Test Suite is an experimental runtime test suite designed specifically for HLSL (High-Level Shader Language). It provides a framework for verifying the correctness of GPU code generated by compilers like Clang and DXC.

By using a YAML-based pipeline description, it allows developers to define GPU workloads, buffers, and expected results, which are then executed across various backends including DirectX 12, Vulkan, and Metal. It is a critical tool for ensuring that HLSL features function correctly on real hardware or software emulators.

Requirements and Setup

Before diving in, ensure you are using a WSL 2 distribution (glibc-based, such as Ubuntu).

1. Install DirectX Headers

You need the WSL-specific headers and libraries to bridge the gap between Linux and Windows D3D12. Specifically, you need libDirectX-Guids.a and libd3dx12-format-properties.a.

• Via Package Manager: Install the directx-headers-dev package.
• Via Source: Alternatively, build them from the DirectX-Headers repository.

2. Build the LLVM Suite

Follow the standard LLVM instructions to add the experimental runtime test suite for HLSL to your LLVM build. Ensure your environment can see the pre-compiled user-mode binaries shipped by Windows at: /usr/lib/wsl/lib

Known Issues and Workarounds

Developing in a bridged environment like WSL isn't without its quirks. Here are the common hurdles and how to resolve them:

1. The D3D12 Debug Layer Crash

Currently, the D3D12 Debug Layer appears to be incompatible with WSL. Calling D3D12GetDebugInterface typically results in a segmentation fault.

Tip: You will have to rely on standard logging, as the debug layer is currently unavailable in this environment.

2. PSO Creation Failures (Unsigned Shaders)

If your Pipeline State Objects (PSOs) fail to create, it is likely because the shaders are unsigned.

• The Fix: Use a newer version of the DirectX Shader Compiler ( DXC ) that includes the validator binary, dxv.

3. Memory Corruption on Exit (double free or corruption)

You might encounter a double free or corruption (!prev) error when api-query or offloader exit. This often leads to an infinite stall rather than a clean termination.

This issue stems from how Microsoft::WRL::ComPtr::InternalRelease behaves in WSL when more than one device is present.

Workarounds:

• Manual Clear: Manually clearing DeviceContext::Devices before the program exits usually eliminates the crash.
• The "Single Device" Hack: To cover all exit points, edit lib/API/DX/Device.cpp to force the system to select only one device. Add a break; at the end of the first iteration of the for-loop in initializeDXDevices().
• Targeted Selection: Run api-query first to find the index of your preferred GPU, then edit the AdapterIndex in the for-loop to target that specific device index.

Conclusion

By utilizing the DirectX bridge in WSL 2, you can effectively test HLSL offloading for the DirectX target without ever needing to interact with the host Windows OS. While the memory management in the WSL/D3D12 bridge can be finicky, these workarounds provide a stable path for your experimental compiler development.

Happy testing!

First published April 14th at blog.farzon.org