This blog series is a part of the write-up assignments of our Game Engineering II class in the Master of Entertainment Arts & Engineering program at University of Utah.

In this assignment, we are modifying our mesh builder from just copying the project from source to target to actually build binary mesh files from our Lua mesh file so we can use in our engine.

Binary File

Let’s look at our cylinder.mesh file. We see that the vertices count is 200 and our indices count is 240. Also, note that I deleted some of the vertex attributes from the file that we’re not actually using in the engine right now, that’s why you’re seeing multiple identical vertices.CylinderLua.PNG

CylinderMeshLuaIndex.PNG

Now let’s look at our built binary mesh file. We can see that the first two bytes are the indices count, the next two bytes are vertices count, and then it’s the index array and vertex array. If you look at the picture below, the F0 00 is our indices count which is 240 in decimal, and C8 00 is 200. Also, be aware that this is the little-endian system so F0 00 shown is actually a value 00 F0. After that, we see the index array starts with 00 00, 2C 00, and 30 00 which correlate to index 0, 44, and 48 in our index array. If we go down a bit more and look at the first vertex, it is 57 78 F3 3F, 00 00 00 00, 7A 37 1E BF, and 00 FF 76 FF. We can convert the first three parts into 4 bytes float which are 1.90211, 0, and -0.618034. 00 FF 76 FF becomes 0, 255, 118, 255 which correlates to the color 0, 1, 0.4615, 1 in our first vertex.

The one important part about this format is that the counts for both indices and vertices need to come before the counter array data, so we can directly set pointers to the correct location without having to iterate through all of them!

cylinderbinaryNewOutline.png

Note that this is in OpenGL order (right-handed). When building meshes, my codes check if it’s under OpenGL or Direct3D settings and fill in every second and third index differently (in reverse) so that we don’t have to iterate and swap all of them after the array is populated.

Binary File Extraction

Extracting the mesh data from binary files is relatively easy. After we loaded the binary file in, they are all in our memory, all we need to do is just have the pointer point to the correct location and reinterpret the data as the formats that we know (because we created the binary files). The codes for that function look like this.

eae6320::cResult eae6320::Graphics::cMesh::Initialize(const char* const i_path, const eae6320::Platform::sDataFromFile& i_loadedMesh) {
	// read from Binary files
	auto result = Results::Success;

	auto currentOffset = reinterpret_cast(i_loadedMesh.data);
	const auto finalOffset = currentOffset + i_loadedMesh.size;

	// point the pointers to correct location in i_loadedMesh's void ptr
	const auto indicesCount = *reinterpret_cast(currentOffset);
	currentOffset += sizeof(indicesCount);
	const auto actualVerticesCount = (*reinterpret_cast(currentOffset));
	currentOffset += sizeof(actualVerticesCount);
	const auto* const indexData = reinterpret_cast(currentOffset);
	currentOffset += sizeof(*indexData) * indicesCount;
	const auto* const vertices = reinterpret_cast(currentOffset);

	if (!(result == Initialize(indicesCount, actualVerticesCount, vertices, indexData))) {
		return Results::Failure;
	}

	return result;
}

Performance

Let’s look at some differences in loading and file size between our original human-readable file and the new binary file.

EngIIAS8Wolf.PNG

Let’s take this wolf (dog?) shown above for example.

Lua file size: 2016 KB

Binary file size: 109 KB

That means the Lua format takes 17.5 times more disk usage comparing to the binary file!

To calculate loading/processing time, we first time how long it took our MeshBuilder to read the Lua file, process it, and extract all the data it needs. We stop the timing right before our builder writes the data into a binary file. And it shows that it took around 0.12 seconds to process that Lua file.

For the binary file, we calculate the time of loading in the mesh file, and then populating the mesh class with the information. The time used for the binary file was around 0.00064 seconds on average, which is roughly 186 times faster!

Next, let’s look at something bigger. In order to do so, I’m gonna need to change my vertex count and index count from using uint16_t to unsigned int to support larger size meshes. This can change many pieces of our mesh pipeline and also our draw call, we need to change our D3D format from

DXGI_FORMAT_R16_UINT to DXGI_FORMAT_R32_UINT;

And our OpenGL glDrawElements type from

GL_UNSIGNED_SHORT to GL_UNSIGNED_INT

Now we can compare the sizes and load speed for our new helix mesh!

GameRunAS8Helix.PNG

Lua file size: 23,435 KB (22.8 MB)

Binary file size: 1,465 KB (1.43 MB)

Lua file takes around 15 times more disk space than the binary file.

Lua process time: 1.129745 seconds

Binary process time: 0.0099402 seconds

This means that our binary file format is around 112 times faster than the Lua format! However, the binary process time ranges from 0.003929 to 0.020986, that’s probably affected by my computer’s condition at the time, so I closed my browsers and other programs as many as I could and ran the tests again.

New Lua process time: 1.1014018 seconds

New binary process time: 0.0038373 seconds

Now binary time is 286 times faster than Lua file since we remove all the spikes in the original data set!

Game Result

GameResult.PNG

Optional Challenges

Binary File Extension

We can easily change our built binary file extension in our AssetBuildFunction.lua to distinguish them from the original human-readable format.

NewAssetTypeInfo( "meshes",
	{
		ConvertSourceRelativePathToBuiltRelativePath = function( i_sourceRelativePath )
			-- Change the source file extension to the binary version
			local relativeDirectory, file = i_sourceRelativePath:match( "(.-)([^/\\]+)$" )
			local fileName, extensionWithPeriod = file:match( "([^%.]+)(.*)" )
			local binaryExtension = "bin"
			return relativeDirectory .. fileName .. extensionWithPeriod .. binaryExtension
		end,
		GetBuilderRelativePath = function()
			return "MeshBuilder.exe"
		end
	}
)

Dynamic Size Indices

Now we want to try to support uint16_t or unsigned int type indices, so we need to handle these dynamically. Our indices count will still be unsigned int though.

In order to do this, we need to figure out how many vertices are there during build time. After we read the vertices count, we can just check if it’s larger than 0xFFFF or not, and determine the correct offset (2 bytes or 4 bytes) to use when we use memcpy to just copy the data into our array. However, since we won’t be able to declare different pointer types at run-time, we can just use char*.

When we read the binary files in to initialize our meshes, we know that the vertex and index counts will come first, and then we can do pretty much the same thing. We determine the correct offset, move our pointers accordingly and assign to our indices and vertices pointers.

We also need to make sure that our OpenGL and D3D is receiving the correct size for buffers since we previously use sizeof(*i_indexData), now we need to multiply it with the offsets because otherwise, the sizeof will just be 1 (char*).

Draw calls can also be problematic, take OpenGL for example, we need to dynamically determine whether to use GL_UNSIGNED_SHORT or GL_UNSIGNED_INT when using glDrawElements!

Binary Data Alignment

Now let’s think about the alignments for our binary files. We want the data to be aligned perfectly when we read it in. Since I ordered the writing order to be unsigned int, unsigned int, and then a number of unsigned int or uint16_t, they will all be aligned nicely. Now I only need to worry about the vertices. The misalignment might happen after I wrote out all of the indices. By looking at the debugger, we can see that the vertex’s alignment is 4 bytes which is what I expected (3 floats + 4 uint8_t). However, this may change in the future so that means there might be a possibility that I need to add paddings into the sMesh struct.

sMeshAlignment.PNG

Basically, what I’m doing now is before I write/read out/in the vertices data, I make sure whether it is on the correct alignment or not and add some paddings (just 0).

Executables

Hold down “Z” to change the wolf object into a helix. Use “WASD” to move the camera, “J” and “L” to rotate the camera horizontally. “Arrow keys” to move the wolf object. “E” and “R” to rotate the cylinder object.

Windows x86 (OpenGL)
Windows x64 (Direct 3D)