For a very long time I have been interested in gnome-nds-thumbnailer, a NDS roms thumbnailer created for use with GNOME. While initially I used it to thumbnail my own collection of NDS roms, later on I developed the interest in trying to rewrite it in Rust (RIIR?).

Ok, but if I were to RIIR and only support NDS roms, it might not be as useful, so what to do? Well, I own a CFW 3DS and I have a collection of several files useful to it (most of them in CIA format), why not add support for the 3DS file formats as well?
So, I would have coverage for 2 Nintendo handheld systems (GB/GBC/GBA don’t have icon data, DSi is supported as DS and Switch is a hybrid console) and it would be much more than a simple Rust rewrite.

Initially v0.1.0 of bign-handheld-thumbnailer only supported SMDH, 3DSX and CIA formats, but v0.9.0 added CXI and CCI as well as initial packaging. And, finally, 1.0.0 focused on optimizations.
“Wait, wait, wait, what is a CIA?”, you might ask if you are unfamiliar with the 3DS and it formats, “Is is that CIA?”.

The 3DS file formats

Well, I believe we should take a look at the 3DS file formats before continuing any further:

• SMDH: the 3DS icon and metadata file format, this contains the file icon (in small and large variants) along with other info such as game name, developer, publisher and even age rating and region lock data; basically the thumbnailer end goal is to find this starting from any other 3DS file fromat and extract the contained large icon
• CIA: the 3DS installer file format, this file allows you to install the application on the 3DS, if the Meta section is present, it usually contains the SMDH data (as a fun fact, you install those with a tool called FBI on a CFW 3DS)
• 3DSX: the 3DS homebrew file format, any homebrew you can run on homebrew launcher will come in this format, some homebrew (usually older ones) might ship a .smdh alongside and some might have a .cia format option as well
• CCI : the 3DS cartridge dump file format (most commonly in the .3ds extension), a specialization of the NCSD container; contains 8 partitions, from which partition 0 is executable content in the CXI format
• CXI: the 3DS executable file format, a specialization of the NCCH, will be decrypted only if ncchflag[7] is set to NoCrypto, impacting on whether ExeFS and other sections are encrypted or not
• ExeFS: a small filesystem internal to the CXI file fomat, contains up to 10 files, usually including a icon file in SMDH format

Due to handling the CCI and CXI files being considered harder at the time of 0.1.0 release, implementing support for them was postponed until 0.9.0. But, with that done, basically all the important file types are supported (being of note that only decrypted CCI and CXI are supported).

More information about 3DS file formats can be found on 3dbrew.

v0.1.0: The Basics

Version 0.1.0 focused on the very basics: making the thumbnailer support NDS files and the initial 3DS files (initially focused on CIAs, but later extended support to SMDH and 3DSX) as well as integrating into the file manager thumbnailer system.

CIA support was originally intended to be the only supported file format, but since I already had code to decode SMDH files, it was easy to just support standalone .smdh files.
3DSX ended being implemented due to being a simple matter of checking the extended header for info on how to grab the SMDH.

Version 0.1.0 made its way into that week’s This Week of GNOME (TWIG), so consider reading there as well for some details.

It’s worth noting that the thumbnailer requires some extra mime type definition (borrowed from Citra/Lime) to work.
Note also that nautilus runs the thumbnailer under a sandbox, which means you likely will have to install the thumbnailer with the package manager for it to wrok.

v0.9.0: CXI and CCI support and COPR repo

After a few months, bign-handheld-thumbnailer gets a new version: 0.9.0!
While 0.1.0 was the initial code upload, 0.9.0 is the first actual proper release, and a COPR is now available for Fedora users to easily install it!

Some other recent improvements from this version include:

• Improvements to BGR555 and RGB565 code (dropping a external dependency by rolling my own implementation)
• Usage of bitstream-io crate for replacing byte slicing code (suggested as a way to improve performance )
• Several under-the-hood improvements

Some functionality that is planned for the future:

• Trying to get a SMDH from the CIA the proper way: via extracting the contents (a NCCH)
• Allowing the user to provide decryption keys that can’t be distributed.

The image above shows the many file types and their respective thumbnails, take note the detailed file type for each file.
Notice how it contains both official games as well as homebrew in the supported DS and 3DS formats (note that both CCI and CXI are decrypted).

Proper support for the mime types required for the thumbnailer is borrowed from Citra/Lime and an issue was created for proper support upstream on xdg’s shared-mime-info repo.

0.9.0 was intended to have been featured in that week’s This Week in Matrix but due some performance issues it was postponed and a whole weekend spent on optimizing.

v1.0.0: Optimizations whenever possible

As you might remember, Rust is a memory-safe language so, coming from C or Java, I don’t really have to worry with pointers or null values; instead I can count on the expressive type system and on the compiler and its borrow checker.
Even then I did manage to create a binary with atrocious resource usage due to a simple mistake.

Before we go on, let’s see my perfomance on 0.9.0:

Those stats were from trying to thumbnail my 3DS games folder, which we can see below:

Another detail that complements the resource usage image is that, not only you would hear the fans on my beefed up laptop spin up but the entire processing would take a few minutes until it finishes and all thumbnails are shown.

IO mistake

It’s time to explain the atrocious resource usage. Just keep in mind that it’s very likely a rookie mistake.

When I started parsing the several format, I did so via byte slicing. So, as a starting point I wanted a byte array from the file:

let f = File::for_path(file_path);

let content = f.load_bytes(None::<&Cancellable>)?;
let content = content.0;

When I was programming what would eventually become 0.9.0, people suggested the bitstream-io crate, and so I tried to take a look into it. The problem is that I did this:

fn extract_exefs(exefs_bytes: &[u8]) -> Result<ExeFSContent, Box<dyn std::error::Error>> {
    let mut reader = ByteReader::endian(exefs_bytes, LittleEndian);
    let exefs_header = reader.read_to_vec(0x200)?;
    let exefs_header = &exefs_header[..];

    let mut reader = ByteReader::endian(exefs_header, LittleEndian);
    let file_headers = reader.read_to_vec(0xA0)?;
    let file_headers = &file_headers[..];

    ...
}

Not only I was creating the ByteReader around the byte slice (therefore negating any benefits from the crate), but I was so unsure on how to properly seek that I ended up prefering to recreate the reader in many cases. So, I accidentally kept the bad performance for that release 🫠.

Though in some scenarios the type conversion utility for that create were very useful:

let mut reader = ByteReader::endian(file_header_bytes, LittleEndian);

let file_name = reader.read_to_vec(0x8)?;
let file_name = &file_name[..];
let file_name = String::from_utf8(file_name.to_vec())?;
let file_name = file_name.trim_matches(char::from(0)).to_owned();

let file_offset = reader.read_as::<LittleEndian, u32>()?;
let file_size = reader.read_as::<LittleEndian, u32>()?;

At some point, it was pointed to me that I was loading the entire file into memory (official 3DS games can get very close to 4 GB in size, and I’m pretty sure I don’t need all that for getting the icon) and, that by instead using a std::fs::File and seeking and only using byte slices when needed, I could get much better performance and resource usage.

3DS Matryoshka mistake

This is not something I am completely sure was a mistake, but probably the code is much better to read now that that is gone. Esssentially, there are a few of the 3DS formats that I consider entrypoints (i.e. you can start from them for a thumbnail), some of them being also able to be found within other files and also file formats that I only treat as internal (ExeFS and related).

The way I initially mapped the formats was something like this:

• SMDH -> large icon
• 3DSX -> (SMDH, if 3DSX has extended header) -> large icon
• CIA -> (Cia Meta, if present) -> SMDH -> large icon
• CXI -> (ExeFS, if CXI not encrypted) -> (SMDH := ExeFS icon file, if exists) -> large icon
• CCI -> CXI (partition 0) -> (ExeFS, if CXI not encrypted) -> (SMDH := ExeFS icon file, if exists) -> large icon

That would mean several structs useful only for wrapping some item, when the ony thing I really care about is the final icon. Luckly I was convinced to create a single SMDHIcon struct with several from_<file_format> methods, just needed to share the file reference when parsing the inner file and seek properly so it works for any entry point.

Note that some of the internal structs were kept separately, such as the one for handling CCI partition data and some related to the ExeFS.

Other improvements

• Use fixed size arrays instead of Vecs whenever possible
• Remove unneeded conversion to strings from bytes inside files (file magic, ExeFS file name; direct byte comparison is used instead)
• Restructure crates (removal of mod.rs files and removal of prefixes in subcrates)
• Better error handling with thiserror crate
• Moved the previous BGR555 and RGB565 parsing code to from_<color_format>_bytes on a new Rgb888 struct
• Made icon scaling optional
• Added --version support
• Use const whenever possible (includes proper names for some constants)
• Apply cargo clippy suggestions
• Apply most of clippy::pendantic suggestions
• Replace gdk-pixbuf (selected because it’s what C thumbnailers use) with image-rs crate (suggested as a better Rust alternative).

The end result

This is what is the current performance on opening the CIA games folder on 1.0.0:

Now not only the thumbnails would load almost instantly but with minimal resource usage.

Conclusion

Well, it took some time but not only I went from nothing to a 1.0.0 in a few months, but also from nothing to a packaged RPM on COPR.

Many people have helped, so I will add “Thank You” section below for them.

In any case, this was my first real Rust project and it was actually more complex than I thought. I do have some basic C skills (my language of choice is Dart but I have been taking up Kotlin), but I couldn’t do it on C with my current skill level. And a thumbnailer would mainly require good performance, and that’s why I chose Rust (which makes it even more ironic that I made a bad performing one from the start 😅).

So, kids, avoid loading entire files into memory when you need just a few bytes at a time, people with low RAM will be thankful!
Not requiring a NASA computer to run your app is also a good selling point!

Thank you

Many thanks to the fellow people for helping with this:

• DaKnig - has been helping me with this project since basically the start, initially with reducing binary size and later with performance
• The people from #rust:matrix.org - help with general Rust questions
• The people from #rust:gnome.org - help with Rust in relation Glib-related stuff
• The people from #gtk:gnome.org - help with stuff related to Glib (such as the interaction between gio and mime types)
• Adrien Plazas - help with trying to interact with xdg upstream for upstreaming the extra 3DS mime types
• The people from #rust:fedoraproject.org - help with generating an RPM (basically, clarification on the rust2rpm generated file and how to build)

Links

bign-handheld-thumbnailer source code - Licensed under GPLv2+, hosted on GitHub
COPR repo - COPR repo with support for Fedora 39, 40 and Rawhide (in both x86_64 and Aarch64 variants)