Compiling GLSL to SPIR-V at build time

2018-06-23 · Gfx-hal Tutorials

When I last left off, I had decided to switch away from OpenGL and start learning gfx-hal. Progress has been good so far - I’m more or less back to where I was before, but with slightly more portable code. I’m planning to go into that soon, but since I’m on holiday, I wanted to write something smaller and simpler.

One of the many changes between old gfx and gfx-hal is that you now have to supply your shaders in SPIR-V format. Thankfully, you can still author them in GLSL, and use Khronos’ compiler to convert them to SPIR-V, so in practice this just adds another step to your build process.

But manual steps are lame and not fun! A much better approach is to use a build script to automate this.

Setup

First of all, we’ll need some GLSL shaders to convert. I’m using the following folder structure:

source_assets/
  shaders/
    simple.vert
    simple.frag
assets/
  gen/
    shaders/
      ... our generated shaders will go here
src/
build.rs
Cargo.toml
...

Vertex shader:

// simple.vert
#version 450
#extension GL_ARB_separate_shader_objects : enable

layout(location = 0) in vec3 vertex_position;
layout(location = 0) out vec4 varying_color;

void main() {
    varying_color = vec4(vertex_position, 1.0);
    gl_Position = vec4(vertex_position, 1.0);
}

And fragment shader:

// simple.frag
#version 450
#extension GL_ARB_separate_shader_objects : enable

layout(location = 0) in vec4 varying_color;
layout(location = 0) out vec4 output_color;

void main() {
    output_color = varying_color;
}

We can also use the glsl-to-spirv Rust crate, rather than compiling Khronos’ glslangValidator ourselves. To do so, add it as a build dependency to your Cargo.toml file:

[build-dependencies]
glsl-to-spirv = "0.1.5"

Finally, you should probably add the output path to your .gitignore file.

build.rs

If you have a source file named build.rs in the root of your project, then cargo will invoke it before compiling your crate. This is where we’ll compile our shaders. I’ll include the full build script here, but I’ll also break it down a bit in this post.

To begin with, we’ll get something running:

extern crate glsl_to_spirv;

use std::error::Error;
use glsl_to_spirv::ShaderType;

fn main() -> Result<(), Box<Error>> {
    // Tell the build script to only run again if we change our source shaders
    println!("cargo:rerun-if-changed=source_assets/shaders");

    // Create destination path if necessary
    std::fs::create_dir_all("assets/gen/shaders")?;
    ...
    Ok(())
}

For reasons I don’t fully understand, you can give instructions about the build script to cargo by printing them to stdout. Without the println! above, we would be recompiling our shaders every time we compile our code, even if they hadn’t changed.

Next, we loop over all of our GLSL source shaders, ignoring anything that isn’t a file, and determine the type of shader based on its filename extension:

    for entry in std::fs::read_dir("source_assets/shaders")? {
        let entry = entry?;

        if entry.file_type()?.is_file() {
            let in_path = entry.path();

            // Support only vertex and fragment shaders currently
            let shader_type = in_path.extension().and_then(|ext| {
                match ext.to_string_lossy().as_ref() {
                    "vert" => Some(ShaderType::Vertex),
                    "frag" => Some(ShaderType::Fragment),
                    _ => None,
                }
            });
        }
        ...
    }

Assuming we can determine a shader type, we can then invoke the shader compiler:

            if let Some(shader_type) = shader_type {
                use std::io::Read;

                let source = std::fs::read_to_string(&in_path)?;
                let mut compiled_file = glsl_to_spirv::compile(&source, shader_type)?;
                ...
            }

The result of this compilation is a temporary file containing the SPIR-V binary. We then want to copy that data into our desired output location:

                // Read the binary data from the compiled file
                let mut compiled_bytes = Vec::new();
                compiled_file.read_to_end(&mut compiled_bytes)?;

                // Determine the output path based on the input name
                let out_path = format!(
                    "assets/gen/shaders/{}.spv",
                    in_path.file_name().unwrap().to_string_lossy()
                );

                std::fs::write(&out_path, &compiled_bytes)?;

If this works you should be able to run any cargo command and see two new files: siple.vert.spv and simple.frag.spv. You can now read these directly in your application, and they’ll be recompiled whenever you change them.

Next steps

There are a few things you could do from here:

  1. Improve error handling and reporting. As things stand, making a mistake in your shader code will probably give you an ugly error like this:

    error: failed to run custom build command for `superior v0.1.0 (file:///Users/user/superior)`
    process didn't exit successfully: `/Users/user/superior/target/debug/build/
        superior-876c6153f46c1599/build-script-build` (exit code: 1)
    --- stdout
    cargo:rerun-if-changed=source_assets/shaders
    
    --- stderr
    Error: StringError("/var/folders/rn/l191w8g17qng7q6w71kx1h600000gp/T/.tmpbA5DPX/
    0.vert\nERROR: /var/folders/rn/l191w8g17qng7q6w71kx1h600000gp/T/.tmpbA5DPX/0.vert:8:
    \'\' :  syntax error, unexpected IDENTIFIER\nERROR: 1 compilation errors.
    No code generated.\n\n\nERROR: Linking vertex stage: Missing entry point:
    Each stage requires one entry point\n\nSPIR-V is not generated for failed
    compile or link\n")
    

    Some manual formatting of that might be required.

  2. While you probably do want to pre-compile your shaders in release mode, it might be useful in development to be able to tweak and recompile shaders at runtime. I’ll likely end up doing this at some point, so there may be a post on that in future.