Mardy (Posts about QBS)

QHtmlParser: writing an HTML parser with your brain switched off

Alberto Mardegan — Sun, 10 Oct 2021 19:57:28 GMT

While developing MiTubo I've recently felt the need of parsing HTML pages: the first problem I wanted to solve was implementing proper RSS feed detection when the user entered a website URL into MiTubo's search box, so that MiTubo would parse the site's HTML, look for <link rel="alternate"...> URLs in the HEAD section, and let the user subscribe to any video feeds found there.

A quick search in the internet did not provide a clear answer: I found a Qt HTML parser in (stalled) development, and a few other C++ or C parsers (among the latters, lexbor is the most inspiring), but all of them seem to take the approach of parsing the HTML file into a DOM tree, while I was hoping to find a lightweight SAX-like parser. Pretty much like Python's html.parser.

Anyway, I don't remember how it happened, but at a certain point I found myself looking at html.parser source code, and I was surprised to see how compact it was (apart, of course, for the long list of character references for the HTML entities!). Upon a closer look, it also appeared that the code was not making much use of Python's dynamic typing, so, I thought, maybe I could give it a try to rewrite that into a Qt class. And a few hours later QHtmlParser was born.

As this post's title suggests, the process of rewriting html.parser with Qt was quite straightforward, and the nice thing about it is that I didn't have to spend any time reading the HTML standard or trying to figure out how to implement the parser: I just had to translate Python code into C++ code, and thanks to the nice API of QString (which in many ways resembles Python's — or vice versa) this was not too hard. I even left most of the original code comments untouched, and reused quite a few tests from the test suite.

It was time well spent. :-)

If you think you might need an HTML parser for your Qt application, you are welcome to give it a try. It's not a library, just a set of files that you can import into your project; for the time being I only have a build file for QBS, but I'll happily accept contributions to make it easier to use QHtmlParser with projects built using other build systems. You can see here the changes I made in MiTubo to start using it and detect RSS feed in a webpage's HEAD.

That's all for now. And in case you missed the link before, you can find QHtmlParser here.

Announcing CuteVNC

Alberto Mardegan — Sun, 13 Dec 2020 13:38:48 GMT

Having a single screen connected to my PC, I was wondering if I could use my nice BQ M10 tablet running UBports as an additional screen. The idea was that if I could convince the X server to create a new virtual display, and then run the x11vnc server on it, I could then connect the tablet to it and enjoy my desktop on two screens. Yes, the M10 has not such a big display, but it couls still be useful to host a window with a mail or a chat client, so that I would not need to switch between applications just to check if new emails or chat messages have arrived (and I hate instant notifications, so I tend to disable them in every program that allows me to).

Anyway, I digress. I found some articles, and specifically this question, that show how one can add a new virtual screen and connect to it via VNC. I tried it and it kind of worked, but there were some visual artifacts that annoyed me (most likely due to the AMDGPU drivers), so I'll defer the creation of a demo video till when I'll have figured out a better setup. But the main point is that the idea, in principle, can work.

CuteVNC

So I sat down and started writing a VNC client for UBports, since it appears that there wasn't one yet in the store. I use the past tense, because after a week or so of work, CuteVNC is there. It's still a very alpha version, and does seem to get confused if the VNC server is using caching, but overall it's working. Its impressive feature list includes:

Can connect to a VNC server, and show the remote display
Can send mouse events
Windowed and fullscreen mode
Pinch to zoom

There's a lot more that could be done, but for a first release this can be OK.

Development of a UBports Qt/QML app with QBS

If, like me, you are a fan of QBS, you might want to have a look at the source code of CuteVNC (yes, the project was born as "Lomiri VNC", but since it's a generic QtQuick.Controls 2 application, there was no need to tie it to the Lomiri environment); there you can find a QBS module for UBports applications, which can help in filling in the manifest file and deploying other Ubuntu Touch specific files into their proper install location.

You can also see the configuration file to make clickable build a QBS-based project. In my mental task backlog I still have the idea of making clickable support QBS out of the box, but the integration of the two technologies is so easy even now, that I gave this task a quite low priority.

A class for computing viewport transformations

Last but not least, a developer might find useful the generic Scaler class I wrote to compute the viewport transformations: it's a simple, QtGui-only C++ class which takes an input structure with the viewport information, and returns a similar structure containing a QTransform matrix to map item coordinates to coordinates in the source object coordinates (in my case, the source object is the remote screen served by VNC) and some other useful parameters, such as the scale, center offset, and the painted area rectangle.

The class has a 100% line and branch coverage in the unit tests, so I hope I can rely on it.

New website for Mappero Geotagger, and cross-compiling stuff

Alberto Mardegan — Sun, 12 Apr 2020 09:20:55 GMT

Mappero Geotagger has now moved from its previous page from this site to a new, separate website built with the awesome Nikola static website generator.

The main reason for this change is that I didn't have an online space where to host the application binaries, and I wanted to experiment with a different selling method. Now, downloads are (poorly) hidden behind a payment page, whereas in multiple places of the website I also mention that I can provide the application for free to whomever asks for it. While it might seem weird at first, I do honestly believe that this will not stop people from buying it: first of all, many people just think it's fair to pay for a software applications, and secondly, for some people writing an e-mail and establishing a personal contact with a stranger is actually harder than paying a small amount of money. And in all sincerity, the majority of the income I've had so far for Mappero Geotagger came from donations, rather than purchases; so, not much to lose here.

QBS and MXE

Anyway, since this is primarily a technical blog, I want to share my experiences with cross-building from Linux to Windows. As you might remember, some time ago I switched the build system of Mappero from qmake to QBS, and I haven't regretted it at all. I've managed to build the application in Linux (of course), macOS, as a Debian package on the Ubuntu PPA builders, on Windows with AppVeyor and, last but not least, on Linux for Windows using the mingw setup provided by the MXE project.

QBS worked surprisingly well also in this case, though I had to fight with a small bug on the toolchain detection, which is hopefully going to be fixed soon. For the few of you who are interested in achieving something similar, here's the steps I ran to configure QBS for mingw:

    MXE_BASE=<path-to-mxe>
    MXE_TARGET=x86_64-w64-mingw32.shared # 32 bit or static targets are also available

    MXE_PROFILE="mxe"
    QT_PROFILE="${MXE_PROFILE}-qt"
    qbs setup-toolchains "${MXE_BASE}/usr/bin/${MXE_TARGET}-g++" $MXE_PROFILE
    qbs config profiles.$MXE_PROFILE.cpp.toolchainPrefix "${MXE_TARGET}-" # temporary workaround
    qbs setup-qt "$MXE_BASE/usr/$MXE_TARGET/qt5/bin/qmake" ${QT_PROFILE}
    qbs config profiles.${QT_PROFILE}.baseProfile $MXE_PROFILE

Sorry for using that many environment variables ☺. After qbs is configured, it's just a matter of running

    qbs profile:$QT_PROFILE

to build the application. You will get a nice window binary and, once you collect all the needed library dependencies, you'll be able to run it on Windows. Or WINE ☺.

As part of this effort, I also had to build libraw, so I didn't miss the occasion to contribute its recipe to MXE. I'm also trying to get a change accepted, that would make MXE support the dynamic OpenGL selection available since Qt 5.4.

Qbs and code coverage reports

Alberto Mardegan — Mon, 01 Jul 2019 13:42:20 GMT

You know that I'm not an early adopter. That's why it was only a couple of weeks ago when I decided to give Qbs a try, by using the good old Mappero (and its spin-off, Mappero Geotagger) as a test bench. Yes, I know that the Qt company is not going to maintain Qbs anymore in the future, but the little I knew about Qbs was enough to convince me that it's a project worth supporting. So, better late than never -- and hopefully the community (me included) will do a good job in keeping Qbs thriving.

Having Mappero build with Qbs was the simplest thing ever. The only issue I met was in building the unit tests, because I'm used to set the rpath on test executables in order to make it easy to run them uninstalled, and with qmake I achieved that with this:

QMAKE_RPATHDIR = $${QMAKE_LIBDIR}

In turns out that with Qbs you can do it in almost the same way, but for some reason I couldn't figure it out and I even reported a bug to which I got some nice suggestions, before eventually settling on this:

import qbs 1.0

Test {
    name: "path-test"

    files: [
        "path-test.cpp",
        "path-test.h",
        "paths.qrc",
    ]

    Depends { name: "Mappero" }
    cpp.rpaths: cpp.libraryPaths    // <-- this does the trick!
}

It's surprisingly similar to how it's done in qmake, so it's not clear even to me why I didn't guess that immediately. Anyway, that was literally my only problem, and you can see the whole set of Qbs files I wrote by having a look at this commit.

Given how easy the migration was, I thought I should also try to add a code coverage report; that's not something I had in my qmake build either, but it's something I really want to have in all my newer projects.

Teaching Qbs to make a code coverage report

Unfortunately, my search for examples on how to have Qbs prepare a coverage report was mostly insuccessful, but thanks to some amazing help from Christian in the #qbs IRC channel, this was not hard to achieve. So, I hope to be of some help myself too, by sharing how this works.

First of all, it must be said that Qbs doesn't know anything about code coverage, at all. However, it's possible (and often easy) to extend Qbs by adding your own Product with its own set of build rules, so here's the CoverageReport item for Mappero (though, it should be general enough to be reusable in your own project):

import qbs

Product {
    name: "coverage"

    property string outputDirectory: "coverage-html"
    property stringList extractPatterns: []

    builtByDefault: false
    files: ["**"]
    type: ["coverage.html"]

    Depends { productTypes: ["autotest-result"] }

    Rule {
        multiplex: true
        explicitlyDependsOnFromDependencies: ["autotest-result"]
        outputFileTags: "coverage.html"
        requiresInputs: false
        prepare: {
            var commands = []
            var captureCmd = new Command("lcov", [
                "--directory", project.sourceDirectory,
                "--capture",
                "--output-file", "coverage.info",
                "--no-checksum",
                "--compat-libtool",
            ]);
            captureCmd.description = "Collecting coverage data";
            captureCmd.highlight = "coverage";
            captureCmd.silent = false;
            commands.push(captureCmd);

            var extractArgs = []
            for (var i = 0; i < product.extractPatterns.length; i++) {
                extractArgs.push("--extract");
                extractArgs.push("coverage.info");
                extractArgs.push(product.extractPatterns[i]);
            }
            if (product.extractPatterns.length > 0) {
                extractArgs.push("-o");
                extractArgs.push("coverage.info");
                var extractCmd = new Command("lcov", extractArgs);
                extractCmd.description = "Extracting coverage data";
                extractCmd.highlight = "coverage";
                extractCmd.silent = false;
                commands.push(extractCmd);
            }

            var filterCmd = new Command("lcov", [
                "--remove", "coverage.info", 'moc_*.cpp',
                "--remove", "coverage.info", 'qrc_*.cpp',
                "--remove", "coverage.info", '*/tests/*',
                "-o", "coverage.info",
            ]);
            filterCmd.description = "Filtering coverage data";
            filterCmd.highlight = "coverage";
            filterCmd.silent = false;
            commands.push(filterCmd);

            var genhtmlCmd = new Command("genhtml", [
                "--prefix", project.sourceDirectory,
                "--output-directory", product.outputDirectory,
                "--title", "Code coverage",
                "--legend",
                "--show-details",
                "coverage.info",
            ]);
            genhtmlCmd.description = "Generate HTML coverage report";
            genhtmlCmd.highlight = "coverage";
            genhtmlCmd.silent = false;
            commands.push(genhtmlCmd);

            return commands;
        }
    }
}

The most important thing here are the references to the autotest-result tag: this is the tag used by the AutotestRunner Qbs item, which is responsible for running the unit tests. Referencing its product's tag in the Depends item and in the explicitlyDependsOnFromDependencies properties ensures that "building" our product will cause the unit tests to run. Other needed bits are the requiresInputs: false property, which means that our rule doesn't have any required inputs, and the builtByDefault: false property, which says that our coverage report should not be generated when just typing qbs. Instead, to run the tests and get the code coverage report one will have to request it explicitly, by typing

qbs -p coverage

The prepare property of the Rule is where the commands to generate the code coverage report are defined. Here we can use the Command item to invoke external programs, and we return a list of such items, so that the commands will be executed in sequence. Note that here I'm using lcov and expecting to find the coverage data produced by gcov, so this is probably not portable outside of Linux/gcc.

Using the CoverageReport item is quite easy: you just need to declare it, and specify which paths contain the coverage data that you are interested in (otherwise, lcov will collect data from all object files that it find under the build directory, which might not be what you desire):

    CoverageReport {
        condition: project.enableCoverage
        extractPatterns: [ '*/src/*.cpp', '*/lib/*.cpp' ]
    }

There's little more than that to be done. Of course, you need to find a way to pass the --coverage option to gcc when building your products, and for this I created a small buildconfig module in qbs/modules/buildconfig/BuildConfig.qbs which I depend on in all products which I wish to build with coverage enabled:

import qbs

Module {
    cpp.cxxFlags: project.enableCoverage ? ["--coverage"] : undefined
    cpp.dynamicLibraries: project.enableCoverage ? ["gcov"] : undefined

    Depends { name: "cpp" }
}

If all this looks scary, you should probably have a look at the diff which shows how I added code coverage reporting to qbs: hopefully you'll find that it's not that complex, after all.

I hope that Qbs users will find this interesting, and possibly improving my setup. Ideally we should try to get something like this part of Qbs itself, but portability outside of Linux / gcc is going to be an issue.