LoadGLBBuffers depricated?

[alexwennstrom]

I'm currently evaluating the fastgltf library and wanted to express my appreciation for the great work you've done on making GLTF parsing fast and efficient.

While exploring the API, I noticed that the LoadGLBBuffers option has been marked as deprecated. I understand that APIs evolve over time, but I wanted to mention that this specific functionality, controlling the loading of binary buffer data, was exactly what I was looking for in my use case.

Would you be open to sharing the reasoning behind the deprecation, or perhaps suggest an alternative approach for achieving similar behavior with the current version of the API?

[spnda]

The simdjson API doesn't allow me to parse JSON on the fly in the sense that I can't load the file piece by piece into memory and parsing individual chunks at a time. Therefore the whole file needs to be loaded into memory before parsing can begin.

This is fine with .gltf files that are only made up of JSON. However, with GLB it is hard to implement an interface that conveys the information where the JSON is in the file, since the parser needs to know about the chunk headers before and after the JSON chunk. And since I want to support virtual file systems and other niches I cannot just take in a std::FILE* or std::ifstream.

I decided to just not do that, and instead to just always put the entire file into memory including the GLB data, since (1) it makes the parsing code & interface a lot simpler, and allows for validation, and (2) the GLB will need to be loaded regardless. I am aware that this is suboptimal, and I am interested in suggestions on how I could make this better.

So for now the option is deprecated/ignored since it's the default behaviour. All other buffers/images are still not loaded by default, though, this only affects the optional embedded buffer.

I hope you understand the design idea I'm trying to get at and why certain things block me from efficiently implementing a system that would only load the GLB without the buffer.

[alexwennstrom]

I'm not entirely sure I understand the API correctly or how the virtual file system is expected to work, but I don’t see why simdjson would need the entire GLB file loaded—aside from the actual JSON data in the first chunk. I've specifically looked for this behavior, and it's also supported by the GLB specification:

"By reading this chunk first, an implementation is able to progressively retrieve resources from subsequent chunks. This way, it is also possible to read only a selected subset of resources from a Binary glTF asset."

Our tool takes a scene description file and merges multiple glTF files into a single one. Until now, we’ve used tinygltf as the loader. For a future version of the tool, our idea is to avoid loading the full buffer data for every glTF file. Instead, we’d only read the gltf information (JSON sectio) initially, and defer loading buffer data until the final merged glTF is being written. At that point, buffer data would be selectively loaded and copied from the included glTF files. This approach ensures that only the necessary buffer data is ever accessed.

I think it would be helpful to update the documentation to clarify that LoadGLBBuffers option is not used.

[spnda]

I decided to read through how I designed the GltfDataGetter system since I wrote that a while back and didn't remember the details exactly. The issue was never about simdjson being the limiter here, but I designed the getter thing around the fact I want to load everything anyway.

fastgltf/src/fastgltf.cpp

Lines 3970 to 4000 in 42d26b2

	// Is there enough room for another chunk header?
	if (header.length > (data.bytesRead() + sizeof(BinaryGltfChunk))) {
	auto binaryChunk = readBinaryChunk(data);
	if (binaryChunk.chunkType != binaryGltfDataChunkMagic) {
	return Error::InvalidGLB;
	}
	// TODO: Somehow allow skipping the binary part in the future?
	if (binaryChunk.chunkLength != 0) {
	if (config.mapCallback != nullptr) {
	auto info = config.mapCallback(binaryChunk.chunkLength, config.userPointer);
	if (info.mappedMemory != nullptr) {
	data.read(info.mappedMemory, binaryChunk.chunkLength);
	if (config.unmapCallback != nullptr) {
	config.unmapCallback(&info, config.userPointer);
	}
	glbBuffer = sources::CustomBuffer{info.customId, MimeType::None};
	}
	} else {
	StaticVector<std::byte> binaryData(binaryChunk.chunkLength);
	data.read(binaryData.data(), binaryChunk.chunkLength);
	sources::Array vectorData = {
	std::move(binaryData),
	MimeType::GltfBuffer,
	};
	glbBuffer = std::move(vectorData);
	}
	}
	}

Now actually reading it there is no real reason not to, I think. I could just skip the loading I linked above, which would work for GltfFileStream, but not for things like GltfDataBuffer. I'll have to think about it some more, but maybe it is doable and I just forgot about it?

[alexwennstrom]

I think this would be a great API feature that could really make this GLTF loader stand out. That said, similar functionality might also be achievable with TinyGLTF—or even this loader—by using a dummy memory allocation callback. However, I'm not sure if that approach would be a valid workaround without potentially breaking other parts of the loader unrelated to accessors and image data.

Personally, I would prefer functionality that aligns exactly with the intent of the option flags—specifically, the ability to load just the JSON chunk and use the API purely for accessing GLTF structural information, without automatically loading buffer or image data. Ideally, there would also be an option to load those buffers on demand.

[alexwennstrom]

That approach would only be effective if the input data is a file stream. If the input is a memory blob, it likely wouldn't make much sense, since you'd need to keep the memory pointer valid in case the buffer is accessed later—which might be impractical or even impossible in certain scenarios.

[spnda]

Personally, I would prefer functionality that aligns exactly with the intent of the option flags—specifically, the ability to load just the JSON chunk and use the API purely for accessing GLTF structural information, without automatically loading buffer or image data. Ideally, there would also be an option to load those buffers on demand.

That is exactly how it worked up until v0.8. And just to be absolutely clear here, this does nothing beyond loading the bytes. Accessors and image data are not touched at all within the library.

I think this would be a great API feature that could really make this GLTF loader stand out

This has been one of the many reason this loader has been standing out for its entire lifetime. It does nothing with buffers nor images at any point in time within the parser. It's all up to you, and it only provides other facilities to simplify the process. See https://fastgltf.readthedocs.io/latest/tools.html.

The only concern I have is that when for example you use GltfDataBuffer you might be shooting yourself in the foot if you don't know about the LoadGLBBuffers option or think it loads it automatically, whereas with GltfFileStream I feel like it kinda makes more sense that it doesn't get loaded. And then the very important question is how this would be represented in DataSource. Would I want to give a std::filesystem::path + offset + length, just an offset + length, a std::span<std::byte>, ...? Because the input can vary so broadly, I have a hard time thinking of what an appropriate data structure should be. Do you have any input regarding this generally speaking?