Shrinking The Linux Kernel And File System For IoT

At last year’s Embedded Linux Conference Europe, Sony’s Tim Bird warned that the stalled progress in reducing Linux kernel size meant that Linux was ceding the huge market in IoT edge nodes to real-time operating systems (RTOSes). At this February’s ELC North America event, another figure who has long been at the center of the ELC scene — Free Electrons’  Michael Opdenacker — summed up the latest kernel shrinkage schemes as well as future possibilities. Due perhaps to Tim Bird’s exhortations, ELC 2017 had several presentations on reducing footprint, including Rob Landley’s Tutorial: Building the Simplest Possible Linux System.

Like Bird, Opdenacker bemoaned the lack of progress, but said there are plenty of ways for embedded Linux developers to reduce footprint. These range from using newer technologies such as musl, toybox, and Clang to revisiting other approaches that developers sometimes overlook.

In his talk, Opdenacker explained that the traditional motivator for shrinking the kernel was to speed boot time or copy a Linux image from low-capacity storage. In today’s IoT world, this has been joined with meeting the requirement for very small endpoints with limited resources. These aren’t the only reasons, however. “Some want to run Linux as a bootloader so they don’t have to re-create bootloader drivers, and some want to run to the whole system in internal RAM or cache,” said Opdenacker. “A small kernel can also reduce the attack surface to improve security.”

Stalled efforts such as the Linux Kernel Tinification project have largely done their job, said Opdenacker. Although size has edged up slightly over the years, you can still call upon a variety of techniques to run your kernel in as little as 4MB of RAM.

“You’d think that since the Tinification project is not that active that the kernel would grow exponentially, but it’s still under control, so maybe we could reverse the trend,” said Opdenacker. “With more aggressive work, 2-3MB may be achievable. Still, there has not been much new in this area since ELC Europe 2015.”

Shrinking the kernel

Meanwhile, there are still plenty of ways to reduce footprint. One of the easiest is to shrink kernel size during compile. First, use a recent compiler, said Opdenacker. For example, gcc 6.2 gives you almost a half percentage point reduction over gcc 4.7 with ARM versatile Linux 4.10. That may not be much, but, “every byte can count,” he added.

Then there are compiler optimizations. With gcc, for example, you can use the -Os option to reduce size. Since gcc 4.7, users have also been able to run optional Link Time Optimizations that can reduce unused code when applied at the end of the compile “when linking all the object files together to optimize things like inlining across various objects,” said Opdenacker. In one test, running gcc 6.2 with LTO reduced the size of the stripped variable by 2.6 percent (x86_64) to 2.8 percent (32-bit ARM).

A few years ago, there was keen interest in an LLVM Linux project that used the Clang front end to the LLVM compiler to compile the Linux kernel for performance and size optimizations. “It is possibly better than what you can get with gcc LTO today, but the project has been stalled since 2015,” said Opdenacker. In response, an audience member suggested the project was still alive.

Using the Clang front end for the LLVM compiler brings even more footprint savings than gcc LTO. Opdenacker ran some tests using a program called OggEnc that consists of a single C program. He then compared Clang 3.8.1 with gcc 6.2 on x86_64, and saw a 5 percent reduction “out of the box without doing anything.” Gcc, however, can offer greater reductions when compiling very small programs, he added.

User space reductions

To reduce user space footprint on simpler programs, Opdenacker suggests that instead of busybox, developers should try the toybox set of Linux command line utilities, which is now baked into Android. “Toybox has the same applications and mostly the same features as busybox, but uses only 84KB instead of 100KB,” he added. “If you just want a shell with just a few command line utilities, toybox could save you a few thousands of bytes, though it’s less configurable.”

Another technique is to switch C/POSIX standard library implementations. The newer musl libc uses less space than uclibc and glibc. Opdenacker described one test on the hello.c program under gcc 6.3 and busybox in which musl used only 7.3KB vs. 67KB for uclibc-ng 1.0.22 vs. 49KB using glibc with with gcc 6.2.

For reducing file system size, Opdenacker recommends booting on initramfs for small file systems. “It lets you boot earlier because you don’t have to initialize file-system and storage drivers.” For bigger RAM sizes, he suggests using compression file systems such as SquashFS, JFFS2, or ZRAM.

“There’s still significant room for improvement in user and kernel space reduction,” concluded Opdenacker. However, when he asked if the community should resurrect the Kernel Tinification project, he was met with a somewhat tepid response.

“These days you can’t even buy an 8MB RAM card,” said one attendee. “It’s an interesting exercise, but I don’t know that there’s a whole lot of payback.” Another developer noted that one problem with the Tinification project was that it “removed things that weren’t needed in really small memory configurations, but the minute you go to the cloud all that stuff is required.”