Ken YapIn another page, I described my adventures converting an Ubuntu DEB package to an RPM package. This provided me with a native toolchain for my Linux workstation.
I wished though that there was a solution that would allow far more Linux users, no matter what distro they use, to have a gcc-ia16 toolchain. Just as much for my self-education, I looked at the main contenders for portable packages, which are: flatpak, snap, and appimage. In this project I'll detail my adventures building a flatpak package.
Flatpak technology
Flatpak is based on Linux namespaces which allow processes to run in a container, which is a modified environment from processes in the host environment. Think of chroot on steroids as a first image. But don't hold onto that image, because containers are far more than that; more capable and more secure.
Containers are also called sandboxes. For example one can expose only part of the host filesystem (a filesystem is a type of namespace) to the container. It took many versions of the Linux kernel to develop all the available types of namespaces, but the set is now complete. You will find containers in development and production situations (think of a server hosting multiple containers), but closer to home you will find that the Firefox and Chrome browsers use containers to enhance security. Just do a lsns(1) on your Linux system and you will see those browsers listed.
This barely scratches the surface of namespace and container technology and you can find documents for deeper study.
Flatpak runtimes
Flatpak is based on a set of standard runtimes. There are runtimes for a generic desktop environment, and for GNOME and KDE. SDKs are provided for all these runtimes to build an application. The completed application will run in an environment containing the specified runtime, plus any accompanying resources for the application itself. To make it concrete, imagine that your GIMP application when running has access to an environment containing the standard system devices and libraries, plus any resources it needs to run, e.g. additional libraries, icons, data, scripts, etc., all inside the container's filesystem.
Flatpak prerequisites
First of all the flatpak tools should be installed. Most mainstream distros have packages for this.
Next the desired runtime and SDK needs to be installed from the Internet. Runtimes are versioned and generally new ones issued each year. In the case of this project, org.freedesktop.Platform and org.freedesktop.Sdk are needed. Unfortunately there isn't anything more spartan, say only for command line applications of which the gcc toolchain is one, available so we choose the simplest one.
Building a flatpak
Flatpaks are described by a manifest, which is a YAML or JSON (your choice) file describing how the application is built from the sources. Generally the sources are fetched from the Internet as tarballs, zip files, or Git downloads.
Unfortunately documentation is where Flatpak technology is deficient. If you go to their website you can find a Hello World example which will get you started. But very quickly you will be searching for "how do I" explanations with frustration. The documentation is lacking in a roadmap that will help users go from a Hello World build to a real-world build. Also the documentation writers seem to be diagram and graphics adverse. Assuming you have the prerequisites, that is the flatpak tools and the runtime platform installed, here are the steps in outline:
- Mkdir a build directory. This is usually called build-dir in the documents, and that name is as good as any. I put mine in /tmp because it's on a solid state disk so compilation is fast.
- Choose an application ID. This is a reverse domain name that should be unique. For example I chose io.github.gcc_ia16.Gcc_ia16. Why the last name twice? The first is my project's name, hopefully nobody else will claim it, and the second is the application within that project. So I could create other applications later.
- Initialise the build directory with flatpak-build-init(1).
- Write your manifest. I did mine using JSON. JSON is very strict. You can't even have a trailing comma on the last element of an array or object, like C allows for array initialisers. And nowhere to put comments. You will iterate this step many times, not only to fix errors, but also to add or modify modules in it. I will go through the manifest for this flatpak below.
Where should I get the sources for the modules? I could pull them down with git from the base project tkchia/gcc-ia16. The extra shared libraries can be downloaded as tarballs from source repos. However a git clone of the base project would include the entire history and be much larger. I hit upon the idea of using the source tarballs built for Ubuntu in the gcc-ia16 PPA. - Build your application using flatpak-build using the manifest. Repeat steps 4 and 5 until the build succeeds with no errors. The manifest isn't very large, mine's only 144 lines but boy did I edit them so many times.
I'll discuss the final steps later.
The manifest in detail
Here is the manifest, line numbered for reference.
1 {
2 "app-id": "io.github.gcc_ia16.Gcc_ia16",
3 "runtime": "org.freedesktop.Platform",
4 "runtime-version": "22.08",
5 "sdk": "org.freedesktop.Sdk",
6 "command": "ia16-elf-gcc",
7 "finish-args": [
8 "--filesystem=home"
9 ],
10 "modules": [
11 {
12 "name": "binutils",
13 "buildsystem": "autotools",
14 "config-opts": [
15 "--target=ia16-elf",
16 "--enable-ld=default",
17 "--enable-gold-yes",
18 "--enable-targets=ia16-elf",
19 "--enable-x86-hpa-segelf=yes",
20 "--disable-werror",
21 "--disable-libctf",
22 "--disable-gdb",
23 "--disable-libdecnumber",
24 "--disable-readline",
25 "--disable-sim",
26 "--disable-nls"
27 ],
28 "no-autogen": true,
29 "sources": [
30 {
31 "type": "archive",
32 "url": "https://launchpad.net/~tkchia/+archive/ubuntu/build-ia16/+sourcefiles/binutils-ia16-elf/2.39-20230319.15-ppa230319153~jammy/binutils-ia16-elf_2.39-20230319.15.orig.tar.xz",
33 "sha256": "13048b65aba9b467bfc36a73e268dcc8f44b9b5528af48b7f83ee62d8b652b9c"
34 }
35 ]
36 },
37 {
38 "name": "gmp",
39 "buildsystem": "autotools",
40 "no-autogen": true,
41 "sources": [
42 {
43 "type": "archive",
44 "url": "https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2",
45 "sha256": "5275bb04f4863a13516b2f39392ac5e272f5e1bb8057b18aec1c9b79d73d8fb2"
46 }
47 ]
48 },
49 {
50 "name": "mpfr",
51 "buildsystem": "autotools",
52 "no-autogen": true,
53 "sources": [
54 {
55 "type": "archive",
56 "url": "https://ftp.gnu.org/gnu/mpfr/mpfr-3.1.5.tar.bz2",
57 "sha256": "ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658"
58 }
59 ]
60 },
61 {
62 "name": "mpc",
63 "buildsystem": "autotools",
64 "no-autogen": true,
65 "sources": [
66 {
67 "type": "archive",
68 "url": "https://ftp.gnu.org/gnu/mpc/mpc-1.0.3.tar.gz",
69 "sha256": "617decc6ea09889fb08ede330917a00b16809b8db88c29c31bfbb49cbf88ecc3"
70 }
71 ]
72 },
73 {
74 "name": "isl",
75 "buildsystem": "autotools",
76 "no-autogen": true,
77 "sources": [
78 {
79 "type": "archive",
80 "url": "https://gcc.gnu.org/pub/gcc/infrastructure/isl-0.16.1.tar.bz2",
81 "sha256": "412538bb65c799ac98e17e8cfcdacbb257a57362acfaaff254b0fcae970126d2"
82 }
83 ]
84 },
85 {
86 "name": "gcc1",
87 "buildsystem": "autotools",
88 "config-opts": [
89 "--target=ia16-elf",
90 "--without-headers",
91 "--with-newlib",
92 "--enable-languages=c",
93 "--disable-libssp",
94 "--disable-libquadmath",
95 "--disable-libstdcxx"
96 ],
97 "no-autogen": true,
98 "sources": [
99 {
100 "type": "archive",
101 "url": "https://launchpad.net/~tkchia/+archive/ubuntu/build-ia16/+sourcefiles/gcc-ia16-elf/6.3.0-20230219.07-ppa230219074~jammy/gcc-ia16-elf_6.3.0-20230219.07.orig.tar.xz",
102 "sha256": "a8db1ea5603ed7d1c832ee452f3d7fa1093e00aa194146b6b15db55b830d3d15"
103 },
104 {
105 "type": "patch",
106 "path": "gcc-ia16-Makefile-in.patch"
107 }
108 ]
109 },
110 {
111 "name": "libnewlib",
112 "buildsystem": "autotools",
113 "config-opts": [
114 "--target=ia16-elf",
115 "--enable-newlib-elix-level=2",
116 "--disable-elks-libc",
117 "--disable-freestanding",
118 "--disable-newlib-wide-orient",
119 "--enable-newlib-nano-malloc",
120 "--disable-newlib-multithread",
121 "--enable-newlib-global-atexit",
122 "--enable-newlib-reent-small",
123 "--disable-newlib-fseek-optimization",
124 "--disable-newlib-unbuf-stream-opt",
125 "--enable-target-optspace",
126 "--enable-newlib-io-c99-formats",
127 "--enable-newlib-mb",
128 "--enable-newlib-iconv",
129 "--enable-newlib-iconv-encodings=utf_8,utf_16,cp850,cp852,koi8_uni"
130 ],
131 "no-autogen": true,
132 "build-options": {
133 "cflags": "-g -Os -D_IEEE_LIBM"
134 },
135 "sources": [
136 {
137 "type": "archive",
138 "url": "https://launchpad.net/~tkchia/+archive/ubuntu/build-ia16/+sourcefiles/libnewlib-ia16-elf/2.4.0-20230316.17-stage1gcc6.3.0-20230311.22-binutils2.39-20221221.21-ppa230316171~jammy/libnewlib-ia16-elf_2.4.0-20230316.17-stage1gcc6.3.0-20230311.22-binutils2.39-20221221.21.orig.tar.xz",
139 "sha256": "3177a1c4ab7b0952158a8adc2025973c7a5b56f5465247e0cdf1a4c3dc8706dc"
140 }
141 ]
142 }
143 ]
144 }
Lines 2-6 declare the name of the app, the runtime and SDK needed to build. Command is the default command that is run if the flatpak is run without specifying one.
Lines 7-9 specify the permissions that are allowed to the flatpak. Here files under /home can be processed.
The rest of the file is an array of modules.
The first module, binutils, lines 11-36 is built using GNU autotools. Flatpak knows about several build systems. This module will be built by first fetching the compressed tarball in the sources section from the Ubuntu PPA for gcc-ia16, unpacking it, entering the top directory, running ./configure with the arguments in config-opts, then make and make install. Note that flatpak-build will automatically add a configure argument of --prefix=/app which is the top directory when the flatpak is run.
The next 4 modules, gmp, mpfr, mpc and isl, lines 37-84, are all support libraries needed by the gcc compiler. These modules are build from tarballs fetched from repos on the Internet, then running ./configure, make and make install.
The next module, gcc, the compiler, lines 85-109, has two sources, the tarball, as well as a patch I created to make the compilation under flatpak work.
The last module, newlib, is the runtime libraries linked into the executables created by the compiler, lines 110-142, contains build-options which contains cflags which are extra arguments sent to the C compiler.
Build progress
There isn't much to comment on the build step, except that I ran it many times to iron out errors. I actually added a module at a time and made that build before adding the next. Fortunately a successful build of a module gets cached, and those modules doesn't have to be rebuilt in subsequent runs. Also the downloads are cached, which prevents multiple fetches.
Testing the build
When the build has completed, the builder will finalize the build directory. This means moving all the generated files to the correct relative positions in the directory and writing metadata description files. If flatpak-builder doesn't do this, run flatpak build-finish. It's an idempotent operation so multiple runs will harm nothing. You can run the commands in the flatpak. I cd'ed to /tmp, and did:
flatpak-builder --run build-dir io.github.gcc_ia16.Gcc_ia16.20230319_2208.json ia16-elf-gcc -o hello.com hello.c
This runs ia16-elf-gcc to compile hello.c to hello.com.
ia16-elf-gcc: error: hello.c: No such file or directory ia16-elf-gcc: fatal error: no input files compilation terminated.
Oops, remember the sandbox only allows for files in /home to be visible. Try again in my home directory.
flatpak-builder --run build-dir io.github.gcc_ia16.Gcc_ia16.20230319_2208.json ia16-elf-gcc -o ~/hello.com ~/hello.c
Ok, that worked, no error messages and ~/hello.com is generated. Try running it in dosbox:
Success.
Packaging the flatpak
Now we want to give our build to an unsuspecting world. This is where I got very frustrated with flatpak documentation.
Note, in what follows, I describe the command by the man page name. However a command like flatpak-build-bundle is actually typed in as flatpak build-bundle.
The primary means of distribution is from a web server containing a tree of the built files. Why not a package? Well if you think about it, many flatpaks will contain the same objects from the runtime. By downloading at the granularity of objects a lot of duplication can be avoided.
I can set up a webserver, but I am loath to do this. All I want is to put my build where people can fetch it. Using the official flathub requires registration and getting your flatpak approved. All I want to do is generate a self-contained flatpak!
Working backwards, I figured out that the command I wanted was flatpak-build-bundle which generates a .flatpak file. Digression: The publishing options of flatpak reveal a little about the world view of the project. The primary avenue of publishing is repositories. Yoiu subscribe to a repository and then you can install applications and get updates later. You can also generate .flatpak files, as mentioned. Finally you can also generate USB sticks and send them out. Surely a strange choice of technology which will eventually become dated. It probably says something about what was rumbling in the heads of the designers of the Flatpak project. Sure, USB drives are still manufactured and in use, but tell a young technologist they could be used to disseminate applications, and they might go Wut? Well maybe it's still useful in some countries. But these days everything is going online, and at least .flatpaks will be available a bit longer.
Back to bundles, they require a repository to generate from. Eventually I discovered that when I installed the flatpak tools not only was a system repo for the runtimes created, but I had a personal repo in ~/.local/share/flatpak/repo.
So now I need to propagate my build to a repo. flatpak-build-export seems to be the command I need. Ok, I exported from build-dir to my personal repo. But flatpak-list didn't show my application. Eventually I worked out that I should have exported to a staging repo, then registered it with flatpak-remote-add, then used flatpak-install to install to my personal repo. It was just lucky that I could use my personal repo as the staging repo too, which saved a lot of copying as the resources would have the same object IDs and checksums. As I wrote before, it would be nice if the documentation provided an overview of the publishing process with diagrams. The technology flatpak repos is built on, OSTree, is modelled after git, where content is stored in versioned objects.
Testing the applcation in the my personal repo
Again I ran ia16-elf-gcc to compile. This time I only needed to specify the application by name since it's been registered in my personal repo.
flatpak run io.github.gcc_ia16.Gcc_ia16 -o hello.com hello.c
That worked. Recall that the default command is ia16-elf-gcc. You can also explicitly name the command to be run using the --command= option which would allow you to access the other tools like say ar or ld. This seems a lot of typing to run a command, and here is where some shell aliases or scripts can help.
Publishing the .flatpak, finally
So finally I can export the toolchain as a bundle.
flatpak build-bundle ~/.local/share/flatpak/repo /tmp/export.flatpak io.github.gcc_ia16.Gcc_ia16
The result in /tmp/export.flatpak was 162MB, which meant that compression had been done on the objects in it. Another gripe, I couldn't find a way to verify what was in the bundle. It's said to comprise metadata followed by the content. I think there are utilities to examine bundles, but not in the core flatpak tools. So I took the tool's word for it.
The .flatpak file, suitably renamed, was then uploaded to the Github repository as a binary asset for a release. So, cut to the chase, you can fetch the toolchain flatpak from the Github repository in the links section, and install it using the flatpak tools. As it requires the freedesktop runtime, it will also pull that in.
Signing with GPG key
It is also possible to sign a .flatpak with a GPG key. The problem was I couldn't figure out which keystore an install verifies the key against. Perhaps it's the key of the repository, or the key signature is looked up in a public keystore (but what happens if the install is done offline?)
So I elected not to sign the .flatpak. Signing the .flatpak doesn't warrant that it's safe, just that you can verify who published it. So it's still caveat installer.