Skinning Pipeline

I'd like to talk a bit about the work I do to write skinning shaders for OpenGL. The OpenGL I'm targeting is version 4.5; the modern-ish concepts that will be thrown around without regard for background knowledge include:

vertex array objects (VAOs) for storing OpenGL vertex attribute bind locations
buffer objects
vertex data (GL_ARRAY_BUFFER)
element arrays (GL_ELEMENT_ARRAY_BUFFER)
transform feedback (GL_TRANSFORM_FEEDBACK_BUFFER)
generic shader storage (GL_SHADER_STORAGE_BUFFER)
shader stages: vertex, geometry, tessellation control and evaluation, and fragment
Geometry buffers (G-Buffers), framebuffer objects (FBOs), and render-to-texture uses
Customized rendering pipeline
separation of skinning and displaying)
accessing GPU skinning results on CPU

But above all else, I'm going to be talking about shaders. Shaders are programs that run on graphics hardware to render 3D objects at interactive rates. Most exciting of all, you can write your own shaders! In earlier days of OpenGL, the 3D rendering pipeline was rather static. The rendering state was programmable to the extent that you could toggle a bunch of options to control the rendering - lighting properties, texture parameters, transparency options, etc. - but the general behavior of the internal pipeline was immutable. Shaders changed that by allowing the user to perform arbitrary commands of their own in each stage, having impressive control over most of the rendering process.

In the time I've spent writing shaders, I've come to truly despise something about the process. Let's look at this simple vertex shader:

#version 450

uniform mat4 mvp;            // model-view-projection matrix
uniform mat3 normalMatrix;   // inverse transpose of model-view matrix 

// Each vertex in the mesh contains these fields
in vec3 position;
in vec3 normal;
in vec3 uv;

// The output values of the vertex shader, to be used by the next stage 
out vec3 v_normal;
out vec2 v_uv;

void main()
{
    gl_Position = mvp * vec4(position, 1.0);
    v_normal = normalize(normalMatrix * normal);
    v_uv = uv;
}

This does what any good vertex shader should do: transforms the vertex's position by the scene's model * view * projection matrix and the vertex's normal by the inverse transpose model * view matrix, then assigns the vertex's texture coordinate to the corresponding output variable. The two matrices are uniforms: every vertex passing through the shader sees the same value for the uniforms. In between rendering frames, the host application can update a uniform's value. OpenGL shader programs have introspection commands that make this possible. For example, to set the mvp matrix, we would call the following code in our application:

GLint mvp_loc = glGetUniformLocation(shaderProgram, ""mvp"");
if (mvp_loc != -1)
{
    // Compute the mesh's model-view-projection matrix
    glm::mat4 mvp = camera->projection * camera->view * model->transform;
    glUniformMatrix4fv(mvp_loc, 1, GL_FALSE, (const GLfloat*)glm::value_ptr(mvp));
}

Works fine, performs well - but the drawback is that the host application needs to perform some work to identify uniform locations. There are alternates that let you specify these locations within the shader source code, which is nice, but the host application must still be informed about these positions in order to use them, which is not nice.

This problem occurs with vertex attributes, too. Values for the attributes(position, normal, and uv) are defined for every vertex in the mesh. This data is loaded in some format to the GPU, and the hosting application is responsible for binding the attributes correctly. We might store our mesh in such a way:

struct Vertex
{
    glm::vec3 position;
    glm::vec3 normal;
    glm::vec2 uv;
}

// Contains all vertices in a mesh
std::vector<Vertex> vertices;

// Every 3 elements are indices for a triangle in the mesh. Indices are based off of location in the vertices vector above.
std::vector<GLuint> indices;

Putting the mesh data on the GPU is easy. Do this once during startup:

// Vertex array object let us bind vertex attributes just once. Then we just bind a specific VAO when we want to render the data associated with it. 
GLuint meshVAO;
glGenVertexArrays(1, &meshVAO);
glBindVertexArray(meshVAO);

// Vertex buffer objects store our mesh data on the GPU and let us bind shader vertex attributes to them
GLuint meshVBO;
GLuint indexVBO;
glGenBuffers(1, &meshVBO);
glGenBuffers(1, &indexVBO);

// VAOs also contain the element array index binding, so we can do that once here and forget about it afterward!
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexVBO);
glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(GLuint) * indices.size(), (const GLvoid*)indices.data(), GL_STATIC_DRAW);

// Now for the mesh's data. 
glBindBuffer(GL_ARRAY_BUFFER, meshVBO);
glBufferData(GL_ARRAY_BUFFER, sizeof(Vertex) * vertices.size(), (const GLvoid*)vertices.data(), GL_STATIC_DRAW);

// With the VAO and VBO bound, let's bind our attributes!

GLint pos_loc = glGetAttribLocation(shaderProgram, ""position"");
glVertexAttribPointer(pos_loc, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (GLvoid*)offsetof(Vertex, position));
glEnableVertexAttribArray(pos_loc);

GLint nrm_loc = glGetAttribLocation(shaderProgram, ""normal"");
glVertexAttribPointer(nrm_loc, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (GLvoid*)offsetof(Vertex, normal));
glEnableVertexAttribArray(nrm_loc);

GLint uv_loc = glGetAttribLocation(shaderProgram, ""uv"");
glVertexAttribPointer(uv_loc, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (GLvoid*)offsetof(Vertex, uv));
glEnableVertexAttribArray(uv_loc);

Once we've done that, rendering something is really simple:


glBindVertexArray(meshVAO);
glUseProgram(shaderProgram);

GLint mvp_loc = glGetUniformLocation(shaderProgram, ""mvp"");
if (mvp_loc != -1)
{
    glm::mat4 mvp = camera->projection * camera->view * model->transform;
    glUniformMatrix4fv(mvp_loc, 1, GL_FALSE, (const GLfloat*)glm::value_ptr(mvp));
}

// Plus any other uniforms we want to update

glDrawElements(GL_TRIANGLES, indices.size(), GL_UNSIGNED_INT, 0);

That takes care of the application side of things, as well as the vertex shader's work, but how about the actual appearance of the mesh on the screen? For that, we turn to the fragment shader.

Between the vertex stage and the fragment stage of the programmable shader pipeline, a not-so-programmable rasterizer stage finds all of the potential fragments contained inside a mesh triangle's area (on-screen) and provides each fragment with values computed from the vertex shader's output (the built-in gl_Position, plus our very own v_normal and v_uv) based on the fragment's barycentric coordinates. The fragment shader gets per-fragment attributes just like the vertex shader gets per-vertex attributes, but luckily we don't have to do any binding for that :) We just have to make sure that the output from one stage corresponds to the input for the following stage.

For whichever fragments are marked as visible on the screen, a fragment shader uses its input attributes to calculate an appropriate color. This approach allows for dynamic lightning, smooth shading across surfaces, texture mapping, and so much more!

So let's say this is our fragment shader:

#version 450

uniform sampler2D tex;

uniform vec3 diffuseColor;
uniform vec3 lightDirection;

in vec3 v_normal;
in vec2 v_uv;

out vec4 fragColor;

void main()
{
    // Basic diffuse light contribution computation using the dot product between the surface normal and the light direction
    vec3 diffuse = diffuseColor * max(0.0, dot(v_normal, lightDirection));

    fragColor = texture2D(tex, v_uv) * vec4(diffuse, 1.0);
}

This is fairly simple too - it computes a lighting contribution using the available data and applies it to the result of a texture lookup.

The code snippets above provide a rough skeleton for modern OpenGL rendering (I didn't show how to set up and bind a texture for use in the fragment shader. I will later, but it won't be used for texturing 3D models!). With variations on this, and a reliable method for creating or importing 3D meshes, you can put any kind of junk you want on the screen! But here's where you begin making choices that make you feel dirty.

Shader programs are compiled from text during runtime in OpenGL. I think this is actually pretty cool! First, you can dump your regularly-used shader code into a file and just tell your application to read from it whenever the shader needs to be created. A nice move for two reasons:

it removes a bunch of GLSL shader source code from the application source code.
It allows you to edit the shader's source file during runtime and recompile the shader at-will, providing instant feedback for experimentation or comparison purposes. Seriously, this is one of the greatest things about writing shaders.

The great drawback to text-based shader compilation is encountered in this question: how does the application source code ""know"" about the shader's dependencies? More specifically, How does it know which uniforms and vertex attributes to bind, and to what locations? In the example code above, the answer is simple: the shader source and application code were written together, so it is straightforward to pattern the two off of each other.

But this demonstrates the inherent weakness of shaders. Although shader code is not considered part of the source code, the application must be somewhat aware of the shader's structure. Given the dependency, the situation produces tremendous temptation to generate shader code directly from the application. This could help with several issues:

Reduces redundancy.

Generating shader text for a desired uniform variable or vertex attribute can occur alongside construction of an object responsible for its state:

std::string shaderText = ""#version 450\n"";

shaderText += addUniform(""mat4"", ""mvp"", [this](GLint loc)
{
    // Compute the mesh's model-view-projection matrix
    glm::mat4 mvp = camera->projection * camera->view * model->transform;
    glUniformMatrix4fv(mvp_loc, 1, GL_FALSE, (const GLfloat*)glm::value_ptr(mvp));
});

shaderText += addAttribute(""vec3"", ""position"", [this](GLint loc)
{
    glBindBuffer(GL_ARRAY_BUFFER, meshVBO);
    glVertexAttribPointer(loc, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const GLvoid*)offsetof(Vertex, position));
    glEnableVertexAttribArray(loc);
});

Consistent binding

The logic to update a uniform's value for the shader accompanies its text generation. This removes lines of code from render(), and ensures that any future changes to a uniform's behavior are not needlessly scattered throughout separate parts of the application.