Ubuntu Touch porting notes for the Redmi Note 7 Pro, part 2

This is the second part of my porting odyssey; for the first part, follow this link.

The good news is that I've done some progress; the bad news is that there are still plenty of issues, and solving them involves deep diving into nearly all components and technologies that make up the core of an Android device, so completing the porting is going to take quite some time. On the other hand, I'm learning a lot of new stuff, and I might be able to share it by writing some documentation. And, who knows, maybe work on some other device port.

Anyway, enough with the introduction! Let's see what progress I've been doing so far.

The new device tree

While asking for help to debug the audio issue I was facing (more about that later), I was also told that the lavender tree, which I was using as a reference, was obsolete. The new one was in gitlab, and was build with a totally different system, described here.

So, I picked the lavender tree and adapted it for violet: I changed the deviceinfo file to point to my kernel tree, use my kernel configuration, and use the same boot command line as before. By the way, here's my current "new" device tree. The build failed, with errors like:

In function 'memcpy',
    inlined from 'proc_ipc_auto_msgmni.part.1' at /home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/ipc/ipc_sysctl.c:82:2:
/home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/include/linux/string.h:340:4: error: call to '__read_overflow2' declared with attribute error: detected read beyond size of object passed as 2nd parameter
    __read_overflow2();
    ^

I then replayed the kernel build using the old system, and noticed that it was using clang as the compiler; so I changed the related flag in the deviceinfo file, and the build went past that point. It failed later, though:

/home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/drivers/staging/qcacld-3.0/core/sap/src/sap_fsm.c:1498:26: error: cast to smaller integer type 'eSapStatus' from 'void *' [-Werror,-Wvoid-pointer-to-enum-cast]
                bss_complete->status = (eSapStatus) context;
                                       ^~~~~~~~~~~~~~~~~~~~

I ended up editing the Kbuild file in the module source directory, and removed the -Werror from there (as well in another place that failed a bit later). This made the build proceed until the end, where it failed because the device tree file was not found:

+ cp -v /home/mardy/projects/port/xiaomi-violet/bd/downloads/KERNEL_OBJ/arch/arm64/boot/dts/qcom/sm8150-mtp-overlay.dtbo /home/mardy/projects/port/xiaomi-violet/bd/tmp/partitions/dtbo.img
cp: cannot stat '/home/mardy/projects/port/xiaomi-violet/bd/downloads/KERNEL_OBJ/arch/arm64/boot/dts/qcom/sm8150-mtp-overlay.dtbo': No such file or directory

I quickly realized that this was due to an error of mine: the Xiaomi violet is a sm6150, not a sm8150 as mentioned in my deviceinfo file! But what overlay file should I use then, since there isn't a sm6150-mtp-overlay.dts in my source tree? Having a loot at other deviceinfo files, I saw that the deviceinfo_kernel_dtb_overlay line is not always there, so I tried commenting it out, and it helped.

The next step was getting flashable images, of course. While the device is not supported by the UBports system-image server, we can use use some scripts to create a fake OTA (Over The Air update) and generate flashable images starting from it. The steps can be read in the devel-flashable target of the .gitlab-ci.yml file (if this step is not present, we should try to find another device repository which has it. They are the following:

./build/prepare-fake-ota.sh out/device_violet.tar.xz ota
./build/system-image-from-ota.sh ota/ubuntu_command out

Once these commands have completed their execution, these commands will push the images to the device:

fastboot flash boot out/boot.img; fastboot flash system out/system.img

There's also fastboot flash recovery out/recovery.img, but I left it out since I was not interested in the recovery image at this stage. And unless you are ready to submit your port for inclusion into the "supported devices" list, I'd recommend not flashing the UT recovery image since TWRP is more powerful and will likely help you in recover your device from a broken image.

Kernel command line

It's important that the kernel command line contains the systempart parameter. In my case, the first boot failed because the command line was longer than 512 bytes, and this parameter was getting truncated. So one thing to be careful about is the length of the kernel command line.

This was fixed by removing some unnecessary kernel parameters from the deviceinfo file.

Missing thumbnails in the Gallery app, content-hub not working

Another issue I noticed is that photo thumbnails were all black in the Gallery app, and the content-hub also was not working. I noticed a relevant commit in the lavender kernel tree and found a launchpad bug which mentioned the issues I was seeing. In the Telegram channel I was told that patch is a forward port of a commit from kernel 3.4 that was present in all of the cores devices, and that it was indeed needed to have the ContentHub working.

The patch did not apply cleanly on top of my kernel tree, but luckily it was just an offset issue: adjusting the patch was easy, and indeed after applying it thumbnails started appearing in the Gallery and Imaginario could import photos again via the ContentHub.

Time and date

Time was always reset to a far away date after a reboot. This is a common issue on Qualcomm devices, and can be fixed by disabling the time service in the Android container.

https://github.com/Halium/android_device_halium_halium_arm64/pull/3

For some reason, at a certain point my override stopped working (or, more likely, the override never worked, but I happened to have fixed the issue directly modifying the vendor init file). I had to copy /android/vendor/etc/init/hw/init.qcom.rc into /usr/share/halium-overrides/, modify it and bind mount the modified file in order to get it working.

This actually seems to match my understanding of the Android init documentation, because according to the path priorities a configuration file stored under /system/ will never be able to override one stored under /vendor.

Fixing the audio configuration

Audio was not working at all. The sound indicator icon was not shown, only the raw (unstranslated) "indicator-sound" text was shown in the panel. pulseaudio was not running. Trying to run it manually (as the phablet user, since pulseaudio is run in the user session) led to this:

phablet@ubuntu-phablet:~$ pulseaudio -n -vvv -F /etc/pulse/touch-android9.pa
I: [pulseaudio] main.c: setrlimit(RLIMIT_NICE, (31, 31)) failed: Operation not permitted
I: [pulseaudio] main.c: setrlimit(RLIMIT_RTPRIO, (9, 9)) failed: Operation not permitted
...
D: [pulseaudio] cli-command.c: Parsing script '/etc/pulse/touch-android9.pa'
D: [pulseaudio] database-tdb.c: Opened TDB database '/home/phablet/.config/pulse/ubuntu-phablet-stream-volumes.tdb'
I: [pulseaudio] module-stream-restore.c: Successfully opened database file '/home/phablet/.config/pulse/ubuntu-phablet-stream-volumes'.
...
D: [pulseaudio] module.c: Checking for existence of '/usr/lib/pulse-8.0/modules/module-droid-card-28.so': success
I: [pulseaudio] module-droid-card.c: Create new droid-card
D: [pulseaudio] droid-util.c: No configuration provided for opening module with id primary
I: [pulseaudio] config-parser-xml.c: Failed to open file (/odm/etc/audio_policy_configuration.xml): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /odm/etc/audio_policy_configuration.xml
D: [pulseaudio] config-parser-xml.c: Read /vendor/etc/audio/audio_policy_configuration.xml ...
D: [pulseaudio] config-parser-xml.c: New module: "primary"
W: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio/audio_policy_configuration.xml:78] Could not find element attribute "samplingRates"
E: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio/audio_policy_configuration.xml:78] Failed to parse element <profile>
E: [pulseaudio] config-parser-xml.c: parsing aborted at line 78
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio/audio_policy_configuration.xml
D: [pulseaudio] config-parser-xml.c: Read /vendor/etc/audio_policy_configuration.xml ...
D: [pulseaudio] config-parser-xml.c: New module: "primary"
W: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio_policy_configuration.xml:78] Could not find element attribute "samplingRates"
E: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio_policy_configuration.xml:78] Failed to parse element <profile>
E: [pulseaudio] config-parser-xml.c: parsing aborted at line 78
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio_policy_configuration.xml
I: [pulseaudio] config-parser-legacy.c: Failed to open config file (/vendor/etc/audio_policy.conf): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio_policy.conf
I: [pulseaudio] config-parser-xml.c: Failed to open file (/system/etc/audio_policy_configuration.xml): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /system/etc/audio_policy_configuration.xml
I: [pulseaudio] config-parser-legacy.c: Failed to open config file (/system/etc/audio_policy.conf): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /system/etc/audio_policy.conf
E: [pulseaudio] droid-config.c: Failed to parse any configuration.
...

Indeed, line 78 of /vendor/etc/audio_policy_configuration.xml had an error, where a property was spelt like simplingRate instead of samplingRate. However, the "vendor" partition is read-only, so I couldn't change that file directly. Another option could have been creating a fixed copy of the file and place it with /system/etc/audio_policy_configuration.xml, but the "system" is also read-only (there are ways to modify these partitions, of course, but I couldn't find a clean way to do it from the device tree scripts. So I went for the bind-mount approach: I would ship the fixed file in some other directory of the file-system, and then modify the /etc/init/mount-android.conf file (this is the job that upstart executes before starting the Android LXC container) to bind-mount the file onto /vendor/etc/audio_policy_configuration.xml.

This worked, but my joy was short-lived: audio was coming up only once every 5 boots or so. I will not list here all the things I tried, as they were plenty of them; and more than once I went to sleep happy and convinced of having fixed the issue for good, until the next day the device booted without audio. It was clearly a timing issue occurring in the early boot, because one thing I clearly noticed very early on is that in those cases when the audio was booting, the following lines appeared in the kernel log:

[    7.130057] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-ldo-rxtx: vol=[1800000 1800000]uV, curr=[25000]uA, ond 0
[    7.130068] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vddpx-1: vol=[1800000 1800000]uV, curr=[10000]uA, ond 0
[    7.130076] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-mic-bias: vol=[3296000 3296000]uV, curr=[25000]uA, ond 0
[    7.130084] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-buck: vol=[1800000 1800000]uV, curr=[650000]uA, ond 1
[    7.137759] wcd937x_codec wcd937x-codec: bound wcd937x-slave.1170224 (ops cleanup_module [wcd937x_slave_dlkm])
[    7.138065] wcd937x_codec wcd937x-codec: bound wcd937x-slave.1170223 (ops cleanup_module [wcd937x_slave_dlkm])

I started adding some printk to the kernel driver, and modified it slightly to register itself with the module_driver() macro instead of the simpler, but logless, module_platform_driver(). This showed that the driver was always loaded at about 7 seconds, but the platform_driver_register() method only called the driver's bind method (wcd937x_bind()) in those boots where audio was working.

After more debugging into platform_driver_register(), I stumbled upon the platform_match() function, added some debugging message in there to print the device name and the driver name, and observed how in those boots where audio was failing this function was called to find a driver for the wcd937x_codec device before the wcd937x_codec driver (provided by the wcd937x_dlmk module) was available. So, I tried adding wcd937x_dlmk to /etc/modules-load.d/modules.conf and this caused the driver to be loaded at about 3 seconds, and apparently fixed the audio issue. At least, till the time of writing this, I never had my phone boot without audio anymore.

Not all is fine with the audio, unfortunately: the mic records a background noise along with the actual sounds, and the recording volume is quite low. This also affects the call quality. On the other hand, the noise disappears when recording happens via the earphones. But I've yet to investigate this; I hope to give you some updates in part three.