Brainstorming on AppImage downloadable frameworks
Among the different cross-distribution application packaging systems available for Linux, the major ones being Flatpak, Snap and AppImage, I've chosen the latter for my own needs for the simple reason that it provides me the easiest way to ship my application and have it work out of the box. Flatpak and Snap will eventually get there, but for the time being universal availability and features like drag and drop are still to come.
However, the AppImage for PhotoTeleport is huge: due to having me ship many Qt5 modules, and especially the enormous QtWebEngine with all of its dependencies, the final package is almost 100 megabytes in size. One way to reduce that would be to require the user system to already have a Qt5 installation available, and therefore remove these libraries from the AppImage file. And if someone ran PhotoTeleport in a system where these libraries are not available, a small wrapper tool (based on xmessage maybe, and whatever Wayland equivalent exists) could inform the user of the problem and invite him to install the missing dependency. And I'm sure you certainly agree that this would be a horrible user experience: since we are so smart to tell that a dependency is missing, why not go the extra mile and install it ourselves?
Unfortunately, this “extra mile” is filled with subtle issues and there's the risk of delivering a solution that works in only a few lucky cases. For this reason, I would like to get the help of other developers and try to find a solution that takes into account all (or most) the subtleties that we need consider for different applications and target distributions. So, what follows is only the result of my initial brainstorming, and could be totally changed (or abandoned, if deemed unfeasible) depending on your welcome feedback.
Requirements
- Smaller size: indeed, this is the whole point of this effort!
- Optional: we don't want to make life harder for all developers: if you are happy with the size of your AppImage file, you should continue to be able to produce it the way you do now, and users should be able to continue to install it in the same way
- Easy but not magic: installing the missing dependencies should be easy, but should not happen without the user's consent. At the very least, we must check that an internet connection is available and make sure that the user is fine with us downloading the extra stuff we need.
The UI problem
No matter how the solution gets implemented, we can be certain of one point: we'll need a way to show some UI to the user. At the very least, this would be a dialog box or a wizard, which would inform the user of what we are doing. We need to find a solution that works both on X11 and Wayland, and that is small enough not to totally destroy the size gain. xmessage is probably present in every Linux distribution, but there doesn't appear to be a Wayland equivalent that is as much available (wlmessage even seems to be unmaintained). Of course, the possibility of shipping whatever tool we need in the AppImage itself always exists and is indeed in the philosophy of AppImage, but then we need to consider how much this costs in term of size.
On the other hand, AppImage packages already assume that some libraries are present in the target system: the excludelist file in the AppImages project lists those libraries. Among them we find the XCB library (to create surfaces on X11 systems) and FreeFont (to render font glyphs), which should give us the possibility of creating some UI in X11; the wayland client libraries are not included, but on the other hand we are talking of less than 100 kilobytes. The next question is whether we want just rely on these libraries, and therefore do all the drawing ourselves, or make use of some other drawing library which would make our task easier. A simple drawing library such as libcairo is already 1 megabyte in size, and if we move to a more complete solution such as Gtk or QtWidgets, the size grows considerably (something between 10 and 20 MBs, from a quick rough estimation). I don't have an answer to this question, and I'd like to spend some more time investigating other options before jumping to code the installer and drawing it pixel by pixel. :-)
The possibility of having different installers, depending on the UI toolkit used by the application shipped in the AppImage (so that most of the dependencies can be shared), should also be considered. This would of course reduce the size gain — for a Qt application, for example, we would have anyway to ship QtCore and QtWidgets because the installer needs them — and would introduce additional complexity because we'd need to make sure that indeed both the application and the installer can work with the version of the toolkit we are shipping.
The developer story
Let's put the UI issue aside for a while, and think about how this could work from the application developer point of view. We have a bunch of dependencies that we'd like to take out of our AppImage; for example, Qt 5.9 with QtQuick and QtWebEngine, and OpenCV. Ultimately, the information we need to pass to the installer is the exact name of the libraries (like libQt5Core.so.5.9, libQt5Gui.so.5.9, etc.) as well as their soname as used by the application (like libQt5Core.so.5, libQt5Gui.so.5, etc.), possibly an md5 signature for each of them, and the URL where we can get them from, in case they are missing from the user system. Writing this information is boring, so the AppImage project might store some ready-made configuration snippets in its git repository.
An additional simplification could be the creation of “frameworks”: bundles of closely related libraries (for example, have a QtBase framework containing QtCore, QtGui, QtWidgets, etc. with all their dependencies).
The user story
When the user installs our AppImage on his system and launches the application, this will first execute the installer: the installer will check that all the downloadable libraries declared by the developer are present on the system, and if any of them are missing it would show its UI and guide the user to install them (here I figure a wizard-like UI, with progress reporting and asking as few questions as possible). The libraries would be installed somewhere in the user home directory, without requiring admin rights.
The installer story
This is where all the magic would happen, and where all the dragons are hiding. It's hard to describe all the complexity in a coherent text, therefore I'll just write down a bullet list of whatever passes through my mind (it's a brainstorming, after all):
- The downloaded libraries should be stored somewhere under the user's ~/.cache/ directory: the reason is that AppImage does not have an uninstallation procedure, and we've always been telling users that the way to uninstall an AppImage application is just to remove the corresponding binary. So, given that the user will not know that he has to remove all the downloaded libraries, the most polite thing we can do is to put them in the cache directory, and hope that if the system goes short on disk space, somehow the cache directory will be cleaned.
- The installer will add the location where the downloadable libraries are expected to be installed (for example, this could be ~/.cache/appimage/lib/<app-name>/) to the LD_LIBRARY_PATH environment variable.
- The installer will check for the library presence in the directory above, by using their soname; if all libraries are found, it will start the application.
- If some library is not found, the installer will look for it in the parent directory, ~/.cache/appimage/lib/, this time without using the soname but the full name; if found, a symlink will be created in ~/.cache/appimage/lib/<app-name>/ linking the library soname to the actual binary library. For example: ~/.cache/appimage/lib/MyApp/libQt5Core.5 -> ~/.cache/appimage/lib/libQt5Core.5.9.
- If the library is not found there, system locations could explored, too. And again, a symlink would be created in ~/.cache/appimage/lib/<app-name>/. The application developer should be able to configure the installer to skip this step, in case it's known that some target distributions ship a version of the desired library that is somehow incompatible with the app (because the distro has patched it in some unexpected way).
- If a library is not found at all, the installer UI shall appear, to inform the user that some libraries need to be downloaded. As the user agrees to proceed, the installer will download the missing libraries.
- Are either of curl or wget installed by default on every distribution? It would be simpler to use them from a shell script, instead of using libcurl from C.
- The library must be made available online. The developer could do that on his own website, and/or the effort could be coordinated among AppImage developers to create and maintain common repositories for open source libraries.
The way forward
While this is a lot of work, I don't immediately see a great complexity or any real blockers. The thing that scares me the most is the idea of writing the installer UI without using any comfortable UI toolkit. Do you see some other big issues that I haven't covered?
While I put more thoughts into this, I'm tempted to start an effort to solve this problem for PhotoTeleport only: the easiest way would be to write an installer program with QtWidgets, which is already a dependency I'm shipping, and have the installer take care of downloading QtDeclarative, a few other QML modules and QtWebEngine. I could write it in such a way that it would be reusable at least for other Qt-based AppImages, and on the other hand this could be a proof of concept to tell whether this whole idea is feasible or not.
Just to highlight it: I wrote that I'm tempted to play with this idea; whether I'll do it for real is all another story. :-) I'm still pondering whether the whole thing is worth the effort, given that I haven't received a single complaint from users so far.
Comments
There's also webmention support.