Mardy (Entratas super Ubports)http://mardy.it/ia/categories/ubports.atom2024-02-02T20:11:01ZAlberto MardeganNikolaDebugging the “Factory mode” of BQ devices in Ubuntu Touchhttp://mardy.it/ia/blog/2021/12/debugging-the-factory-mode-of-bq-devices-in-ubuntu-touch.html2021-12-25T10:13:54+03:002021-12-25T10:13:54+03:00Alberto Mardegan<p>As you know, I'm trying to <a href="http://mardy.it/ia/blog/2021/12/enabling-the-fm-radio-in-ubuntu-touch.html">get the FM radio to work in Ubuntu
Touch</a>, and I basically have
it working on the Redmi Note 7 Pro. But then I remembered that the BQ Aquaris
E4.5 (which is the first commercial device officially supporting Ubuntu Touch)
also comes with an FM radio, so I decided to put some effort in getting that to
work, too. You might think it's a waste of time, but as a matter of fact this
device is built on a Mediatek SoC, and FM radio support is exposed to userspace
in a very similar way across all Mediatek devices — so this work should be
covering other devices as well.</p>
<p>It was relatively easy to get the FM radio to work on this phone: I can now
tune to a frequency and see that the RDS data is received, but I cannot get any
sound out of the speakers or headphones, which makes the whole radio experience
a bit, uhm… suboptimal, let's say.</p>
<p>The problem is not new: back in the days when Ubuntu Touch was still supported
by Canonical, <a href="http://www.lieberbiber.de/">sturmflut</a> was trying to get the FM
radio to work, and he met the same issues (see
<a href="https://lists.launchpad.net/ubuntu-phone/msg17256.html">this</a> and
<a href="https://lists.launchpad.net/ubuntu-phone/msg17340.html">this</a> messages in the
mailing list). He also spent some time investigating how the <a href="http://sturmflut.github.io/ubuntu/bq/2015/05/04/hacking-the-bq-part-2-factory-mode/">Factory
mode</a>
works (the FM radio is indeed one of the features that can be tested via the
Factory mode, and it <em>does</em> work) and he mentioned that he could get <code>gdb</code> to
attach to the factory mode program and could see the various <code>ioctl</code>s being
executed. Yesterday I tried to follow the same steps, but I failed quite soon:
I simply could not connect with <code>adb shell</code> while the device was in factory
mode, so no chances of debugging for me. ☹</p>
<p>Out of dispair, I tried just to manually run the program <code>/system/bin/factory</code>
inside a Lomiri session and, to my surprise, it overlayed its yellow “Factory
Mode” title on the screen — just to immediately quit afterwards. I tried
running it with <code>strace</code>, and noticed these lines:</p>
<div class="code"><pre class="code literal-block">open("/sys/class/BOOT/BOOT/boot/boot_mode", O_RDWR|O_LARGEFILE) = 15
read(15, "0\n", 4) = 2
close(15) = 0
writev(5, [{"\3", 1}, {"FTM\0", 4}, {"[FTM UTILS] Unsupported factory "..., 39}], 3) = 44
writev(5, [{"\6", 1}, {"NVRAM\0", 6}, {"[MAIN] Unsupported Factory mode\n"..., 33}], 3) = 40
</pre></div>
<p>Of course, the <code>/sys/class/BOOT/BOOT/boot/boot_mode</code> file is read-only, so I
couldn't just write <code>1</code> into it, but could a bind-mount work? Indeed it did!
And after a few attempts, I verified that writing a value of <code>4</code> in the
<code>boot_mode</code> file made the factory program happy. It was still unusable because
it was acting as if the volume down button was being constantly pressed, so the
cursor was always moving downwards, but killing the Lomiri session did the
trick.
In short, these are the steps you need to follow in order to run the Factory
Mode from an ordinary boot session:</p>
<div class="code"><pre class="code literal-block">sudo<span class="w"> </span>-i<span class="w"> </span><span class="c1"># become root</span>
<span class="nb">echo</span><span class="w"> </span><span class="m">4</span><span class="w"> </span>><span class="w"> </span>/tmp/trick
mount<span class="w"> </span>-o<span class="w"> </span><span class="nb">bind</span><span class="w"> </span>/tmp/trick<span class="w"> </span>/sys/class/BOOT/BOOT/boot/boot_mode
service<span class="w"> </span>lightdm<span class="w"> </span>stop<span class="w"> </span><span class="c1"># wait a few seconds until this returns</span>
<span class="nb">unset</span><span class="w"> </span>LD_PRELOAD
/system/bin/factory
</pre></div>
<p>What is more surprising, is that the FM radio test is working even in this
environment, and you can actually hear the sounds (for some reason, the FM
radio item takes about 40 seconds to initialize when run under these
conditions, but it eventually works). At this point, not only I could run <code>gdb</code>
(which I didn't), but I could even run the <code>factory</code> program under strace and
collect the logs.</p>
<p>At the moment of writing this post, I haven't yet examined the logs, so I'm not
at all sure that they'll be enough to make audio work (I suspect that the
factory mode binary might be playing some tricks that are somehow not
replicable in a proper Linux system with Pulseaudio and lots of other services
running), but I do have enough information to make at least a few attempts.</p>
<p>Stay tuned, and have a Merry Christmas!</p><p>As you know, I'm trying to <a href="http://mardy.it/ia/blog/2021/12/enabling-the-fm-radio-in-ubuntu-touch.html">get the FM radio to work in Ubuntu
Touch</a>, and I basically have
it working on the Redmi Note 7 Pro. But then I remembered that the BQ Aquaris
E4.5 (which is the first commercial device officially supporting Ubuntu Touch)
also comes with an FM radio, so I decided to put some effort in getting that to
work, too. You might think it's a waste of time, but as a matter of fact this
device is built on a Mediatek SoC, and FM radio support is exposed to userspace
in a very similar way across all Mediatek devices — so this work should be
covering other devices as well.</p>
<p>It was relatively easy to get the FM radio to work on this phone: I can now
tune to a frequency and see that the RDS data is received, but I cannot get any
sound out of the speakers or headphones, which makes the whole radio experience
a bit, uhm… suboptimal, let's say.</p>
<p>The problem is not new: back in the days when Ubuntu Touch was still supported
by Canonical, <a href="http://www.lieberbiber.de/">sturmflut</a> was trying to get the FM
radio to work, and he met the same issues (see
<a href="https://lists.launchpad.net/ubuntu-phone/msg17256.html">this</a> and
<a href="https://lists.launchpad.net/ubuntu-phone/msg17340.html">this</a> messages in the
mailing list). He also spent some time investigating how the <a href="http://sturmflut.github.io/ubuntu/bq/2015/05/04/hacking-the-bq-part-2-factory-mode/">Factory
mode</a>
works (the FM radio is indeed one of the features that can be tested via the
Factory mode, and it <em>does</em> work) and he mentioned that he could get <code>gdb</code> to
attach to the factory mode program and could see the various <code>ioctl</code>s being
executed. Yesterday I tried to follow the same steps, but I failed quite soon:
I simply could not connect with <code>adb shell</code> while the device was in factory
mode, so no chances of debugging for me. ☹</p>
<p>Out of dispair, I tried just to manually run the program <code>/system/bin/factory</code>
inside a Lomiri session and, to my surprise, it overlayed its yellow “Factory
Mode” title on the screen — just to immediately quit afterwards. I tried
running it with <code>strace</code>, and noticed these lines:</p>
<div class="code"><pre class="code literal-block">open("/sys/class/BOOT/BOOT/boot/boot_mode", O_RDWR|O_LARGEFILE) = 15
read(15, "0\n", 4) = 2
close(15) = 0
writev(5, [{"\3", 1}, {"FTM\0", 4}, {"[FTM UTILS] Unsupported factory "..., 39}], 3) = 44
writev(5, [{"\6", 1}, {"NVRAM\0", 6}, {"[MAIN] Unsupported Factory mode\n"..., 33}], 3) = 40
</pre></div>
<p>Of course, the <code>/sys/class/BOOT/BOOT/boot/boot_mode</code> file is read-only, so I
couldn't just write <code>1</code> into it, but could a bind-mount work? Indeed it did!
And after a few attempts, I verified that writing a value of <code>4</code> in the
<code>boot_mode</code> file made the factory program happy. It was still unusable because
it was acting as if the volume down button was being constantly pressed, so the
cursor was always moving downwards, but killing the Lomiri session did the
trick.
In short, these are the steps you need to follow in order to run the Factory
Mode from an ordinary boot session:</p>
<div class="code"><pre class="code literal-block">sudo<span class="w"> </span>-i<span class="w"> </span><span class="c1"># become root</span>
<span class="nb">echo</span><span class="w"> </span><span class="m">4</span><span class="w"> </span>><span class="w"> </span>/tmp/trick
mount<span class="w"> </span>-o<span class="w"> </span><span class="nb">bind</span><span class="w"> </span>/tmp/trick<span class="w"> </span>/sys/class/BOOT/BOOT/boot/boot_mode
service<span class="w"> </span>lightdm<span class="w"> </span>stop<span class="w"> </span><span class="c1"># wait a few seconds until this returns</span>
<span class="nb">unset</span><span class="w"> </span>LD_PRELOAD
/system/bin/factory
</pre></div>
<p>What is more surprising, is that the FM radio test is working even in this
environment, and you can actually hear the sounds (for some reason, the FM
radio item takes about 40 seconds to initialize when run under these
conditions, but it eventually works). At this point, not only I could run <code>gdb</code>
(which I didn't), but I could even run the <code>factory</code> program under strace and
collect the logs.</p>
<p>At the moment of writing this post, I haven't yet examined the logs, so I'm not
at all sure that they'll be enough to make audio work (I suspect that the
factory mode binary might be playing some tricks that are somehow not
replicable in a proper Linux system with Pulseaudio and lots of other services
running), but I do have enough information to make at least a few attempts.</p>
<p>Stay tuned, and have a Merry Christmas!</p>Enabling the FM radio in Ubuntu Touchhttp://mardy.it/ia/blog/2021/12/enabling-the-fm-radio-in-ubuntu-touch.html2021-12-07T19:41:26+03:002021-12-07T19:41:26+03:00Alberto Mardegan<p>I recently realized that my Xiaomi Redmi Note 7 Pro, on which <a href="http://mardy.it/ia/blog/2021/01/ubuntu-touch-porting-notes-for-the-redmi-note-7-pro.html">I installed
Ubuntu Touch not so long
ago</a>, has a
working FM radio. One of the many psychological bugs of mine is the irrational
urge I feel of having my hardware, no matter whether I use it or not,
<a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8c47311d34eccedb06bc60fc9435a53bd4aff392">supported by
Linux</a>.
So, the fact that I never listen to the radio is unfortunately not a reason to
dissuade me from wasting time on getting the FM radio working in Ubuntu Touch.</p>
<p>This post is a quick summary of my investigation, which should serve as a note
keeper for when I'll actually get to implement the feature.</p>
<h4>The Android story</h4>
<p>It was a bit surprising to find out that Android does not offer an API to use
the FM radio, meaning that we cannot simply expose an API via libhybris. Every
manufacturer is therefore choosing their own API, and Android applications are
most often developed for a specific chipset family, or need to carry the code
to support all the various devices.</p>
<p>For example, I found the <a href="https://github.com/vladislav805/RFM-Radio">RFM Radio Android
application</a> which supports a few
Qualcomm Snapdragon processors, for which the FM radio functionality is exposed
by a kernel driver via the V4L API. This means that the FM radio is probably
working out of the box with <a href="https://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/plugins/v4l?h=5.12">QtMultimedia's
QRadioTuner</a>.
Unfortunately the Note 7 Pro uses a newer Snapdragon, and even after some days
of investigations <a href="https://www.spinics.net/lists/linux-arm-msm/msg98952.html">I couldn't find out how the radio driver communicates to the
userspace</a>; but more
on this below. Other chipsets offer other APIs, and I was glad to find that
someone already wrote a <a href="https://github.com/Venji10/fmradio-volla">Ubuntu Touch FM radio application for the Volla
phone</a>, which has a Mediatek board.</p>
<p>Anyway, the lack of a unified FM radio API is probably the reason why most of
the so-called “FM radio” applications on Android are not really using the FM
radio but rather streaming audio from the internet.</p>
<h4>The FM radio in the Note 7 Pro</h4>
<p>Before I start talking about a phone that no one cares about, let me say that
what I'm going to write applies to several other Snapdragon chipsets and could
be relevant to other phones. For one, the Redmi Note 9 Pro uses the very same
bluetooth chipset as the Note 7 Pro (and in case you are wondering why I
mentioned bluetooth, it's because the FM radio functionality is delivered by
the same BT chip), so all what I'm going to write here is also relevant for
that phone.</p>
<p>In order to figure out how the radio worked on this device, I took the drastic
decision to reflash the stock Android (well, MIUI), started the preinstalled FM
radio application, and meanwhile looked at the logcat messages (I'm not sure if
this is needed, but before doing so I went to the “Developer options” in the
system settings and set the debugging level to the maximum). Among a lot of
noise, this showed me lines like these:</p>
<div class="code"><pre class="code literal-block">I android_hardware_fm: Opened fm_helium.so shared object library successfully
I android_hardware_fm: Obtaining handle: 'FM_HELIUM_LIB_INTERFACE' to the shared object library...
D FmReceiverJNI: init native called
D android_hardware_fm: BT soc is cherokee
I android_hardware_fm: Init native called
I android_hardware_fm: Initializing the FM HAL module & registering the JNI callback functions...
D radio_helium: ++hal_init
D fm_hci : ++fm_hci_init
I fm_hci : hci_initialize
</pre></div>
<p>Well, even without knowing nothing about all what these lines meant, I had
something I could search the internet for. So I found <a href="https://source.codeaurora.org/quic/la/platform/vendor/qcom-opensource/fm-commonsys">the Qualcomm FM code in
CodeAurora</a>,
and search for the code relative to my Snapdragon 675 (aka <code>sm6150</code>). I quickly
gave up on trying to make some sense out of the git tag naming in that
repository, and just tried to search for a tag which could be referring to my
device. I found one, and started browsing <a href="https://source.codeaurora.org/quic/la/platform/vendor/qcom-opensource/fm-commonsys/tree/?h=LA.UM.7.9.r1-11400-sm6150.0">its source
tree</a>.</p>
<p>It turns out that Qualcomm provides a Java package which applications can use,
and which internally <code>dlopen()</code>s the <code>fm_helium.so</code> library, which in turn
depends on the <code>libfm_hci.so</code> library. I had a quick look at the source code of
these libraries, which are also present in the repository, but decided that I
would have had more chances of success if I just tried to follow the JNI code,
and in particular the
<a href="https://source.codeaurora.org/quic/la/platform/vendor/qcom-opensource/fm-commonsys/tree/jni/android_hardware_fm.cpp?h=LA.UM.7.9.r1-11400-sm6150.0"><code>android_hardware_fm.cpp</code></a>
file. I'm not sure why this code is not using the C structure types defined in
the headers provided by the helium library, and instead redefines all the
constants and accesses the character buffers by offsets — it might be just for
historical reasons — but in any case I decided to follow along.</p>
<h4>The <code>fm-bridge</code> program</h4>
<p>Since we have a rather net separation between the Ubuntu Touch and the Android
worlds (the Android services are running inside an LXC container, with all
their Android libs and dependencies), one should not attempt to write an Ubuntu
process that loads the Android libraries, because the libc used in Android is
different, so things are likely not to work. But we can have Ubuntu and Android
processes communicate over a socket or other kind of IPC; so, what I decided to
go for, is writing a small C program that will live in the Android side, it
will talk to the FM radio (via <code>helium_fm</code>), and accept commands / give replies
via its <code>stdin</code> / <code>stdout</code>.</p>
<p>I unimaginatively called it “<code>fm-bridge</code>”, and you can look at its horrible
code <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/fm-bridge">here</a>. Really, I
just said it was terrible, so why did you look at it? I definitely need to
rewrite it from scratch, possibly using the helium headers, but as a proof of
concept this also works. Then I carefully examined the logcat output while
using the MIUI FM radio application in Android, and figured out what was the
command sequence I had to input into <code>fm-bridge</code>'s standard input in order to
have it tune onto a given frequency. I'm publishing the commands here too,
should I ever lose my notes:</p>
<div class="code"><pre class="code literal-block">enableSlimbus 1
setControl 0x8000004 1
enableSoftMute 1
setControl 0x8000029 0
setControl 0x800000c 1
setControl 0x800000d 1
setControl 0x800000e 1
setControl 0x800002b 0
setControl 0x8000007 4
setControl 0x8000006 0x40
setControl 0x8000006 0x40
setControl 0x8000011 0
setControl 0x800000f 1
getControl 0x8000010
setControl 0x8000010 0xef
setControl 0x800000f 1
setControl 0x800001b 1
setControl 0x8000012 0
setFreq 89300
setMonoStereo 1
</pre></div>
<p>I'm sure that not all of them are needed, but I'll figure out the optimal
sequence later. In order to use this program on Ubuntu Touch, I had to alter
the vendor partition to add this program, but also the <code>fm_helium.so</code> and
<code>libfm_hci.so</code> libraries (more on that below).</p>
<p>When feeding the above commands to the <code>fm-bridge</code> in Ubuntu, I saw that I was
getting a logcat output similar to the one from Android, which was mildly
comforting. No sound was comint out of the speaker or out of the earplugs, but
I was hardly expecting it all to work at the first try. And I got convinced
that the FM tuner was indeed working, because typing the command “<code>startSearch
1</code>” made a new frequency appear in the logs, proving that the tuner had found
another station and tuned onto it.</p>
<h4>Getting the sound out</h4>
<p>This was actually the easiest of the steps, thanks to the Ubuntu Touch FM radio
application we have for the Volla: its source code mentions a few pulseaudio
commands that worked perfectly in the Note 7 Pro too, despite the fact that the
underlying chipset is totally different. This should not be as surprising as it
might sound like, given that Android has a common audio API.</p>
<p>Just for my future reference, the commands are these:</p>
<div class="code"><pre class="code literal-block">pacmd set-source-port 1 input-fm_tuner
pactl load-module module-loopback source=1 sink=0
</pre></div>
<p>Ta-daaa! The radio was now playing from the phone loudspeakers! It was indeed
quite loud, and the volume buttons did not seem to have any effect on it, but
the volume can be controlled with pulseaudio:</p>
<div class="code"><pre class="code literal-block">pactl set-source-volume 1 50%
</pre></div>
<p>Of course, if we ever manage to make this into an Ubuntu Touch feature, we'll
have to find a way to make the volume respond to the volume buttons.</p>
<h4>Addind the needed files to the vendor partition</h4>
<p>The simplest approach (and the one I took initially) is that of downloading the
<code>vendor.img</code> into your PC, loop-mounting it, adding the <code>fm_helium.so</code>,
<code>libfm_hci.so</code> and <code>fm-bridge</code> files to it and then umount the partition and
reflash it (remembering to converting it from/to a sparse image before
downloading/uploading it). This approach works flawlessly, but I'm wondering if
one might incur into issues if the version of the NDK used to compile fm-bridge
is different from the one that was used to compile the other vendor binaries,
so I decided to give it a try to build the whole vendor partition myself.</p>
<p>This turned out to be a non trivial process, because I was using the Halium
tree to build the vendor image, and not the LineageOS which was used to build
the vendor image for my device: I could make an image, but it took some time
before I figured out which were the needed packages that somehow got lost
because of the Halium changes and that <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_vendor_xiaomi_violet/-/commit/495c83065c1f135d84c8e08563cece4fd4e4d3a5">had to be added to the
Makefile</a>.</p>
<p>To help my weak memory, I expanded the <a href="https://gitlab.com/ubports/community-ports/android9/xiaomi-redmi-note-7-pro/xiaomi-violet/-/blob/master/README.md">README file in the <code>violet</code>
port</a>
with the steps needed in order to build the vendor image.</p>
<h4>A system service for the FM radio</h4>
<p>While it could be possible for Ubuntu Touch applications to directly access the
FM radio device in the same way that Android applications do, this is
suboptimal for a few reasons. Even if we provided a shared library to deal with
the various radio chipset implementations, the application would either need to
be unconfined, or we'd had to provide an ever-changing AppArmor profile that
peeks new holes every time that a new device implementation is added (and what
if this implemenation uses a generic kernel device, which could be used for
other goals too?) and in any case we'd have to make this policy restricted,
since the RDS data provided by the radio stations would reveal the user
location (well, the city at least) to the application. Not to talk about
concurrent access to the radio device if two applications attempt to use it.</p>
<p>Therefore, my proposition (and what I'll implement, if I'll live long enough or
if someone doesn't beat me to it) is to have a system service deal with the
various hardware differences and expose a D-Bus API that will be hooked up as a
QRadioTunerControl plugin, so that Qt applications will be able to just use the
QtMultimedia APIs to access the radio.</p>
<p>The service would also need to talk to the trust-store, to let the user decide
whether the application should really be granted access to the FM receiver (and
when using the turst-store, this decision is remembered, and revocable from the
System Settings's Security panel). Of course we'll also need to add a
<code>fm-radio</code> AppArmor policy to let applications use this service.</p><p>I recently realized that my Xiaomi Redmi Note 7 Pro, on which <a href="http://mardy.it/ia/blog/2021/01/ubuntu-touch-porting-notes-for-the-redmi-note-7-pro.html">I installed
Ubuntu Touch not so long
ago</a>, has a
working FM radio. One of the many psychological bugs of mine is the irrational
urge I feel of having my hardware, no matter whether I use it or not,
<a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8c47311d34eccedb06bc60fc9435a53bd4aff392">supported by
Linux</a>.
So, the fact that I never listen to the radio is unfortunately not a reason to
dissuade me from wasting time on getting the FM radio working in Ubuntu Touch.</p>
<p>This post is a quick summary of my investigation, which should serve as a note
keeper for when I'll actually get to implement the feature.</p>
<h4>The Android story</h4>
<p>It was a bit surprising to find out that Android does not offer an API to use
the FM radio, meaning that we cannot simply expose an API via libhybris. Every
manufacturer is therefore choosing their own API, and Android applications are
most often developed for a specific chipset family, or need to carry the code
to support all the various devices.</p>
<p>For example, I found the <a href="https://github.com/vladislav805/RFM-Radio">RFM Radio Android
application</a> which supports a few
Qualcomm Snapdragon processors, for which the FM radio functionality is exposed
by a kernel driver via the V4L API. This means that the FM radio is probably
working out of the box with <a href="https://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/plugins/v4l?h=5.12">QtMultimedia's
QRadioTuner</a>.
Unfortunately the Note 7 Pro uses a newer Snapdragon, and even after some days
of investigations <a href="https://www.spinics.net/lists/linux-arm-msm/msg98952.html">I couldn't find out how the radio driver communicates to the
userspace</a>; but more
on this below. Other chipsets offer other APIs, and I was glad to find that
someone already wrote a <a href="https://github.com/Venji10/fmradio-volla">Ubuntu Touch FM radio application for the Volla
phone</a>, which has a Mediatek board.</p>
<p>Anyway, the lack of a unified FM radio API is probably the reason why most of
the so-called “FM radio” applications on Android are not really using the FM
radio but rather streaming audio from the internet.</p>
<h4>The FM radio in the Note 7 Pro</h4>
<p>Before I start talking about a phone that no one cares about, let me say that
what I'm going to write applies to several other Snapdragon chipsets and could
be relevant to other phones. For one, the Redmi Note 9 Pro uses the very same
bluetooth chipset as the Note 7 Pro (and in case you are wondering why I
mentioned bluetooth, it's because the FM radio functionality is delivered by
the same BT chip), so all what I'm going to write here is also relevant for
that phone.</p>
<p>In order to figure out how the radio worked on this device, I took the drastic
decision to reflash the stock Android (well, MIUI), started the preinstalled FM
radio application, and meanwhile looked at the logcat messages (I'm not sure if
this is needed, but before doing so I went to the “Developer options” in the
system settings and set the debugging level to the maximum). Among a lot of
noise, this showed me lines like these:</p>
<div class="code"><pre class="code literal-block">I android_hardware_fm: Opened fm_helium.so shared object library successfully
I android_hardware_fm: Obtaining handle: 'FM_HELIUM_LIB_INTERFACE' to the shared object library...
D FmReceiverJNI: init native called
D android_hardware_fm: BT soc is cherokee
I android_hardware_fm: Init native called
I android_hardware_fm: Initializing the FM HAL module & registering the JNI callback functions...
D radio_helium: ++hal_init
D fm_hci : ++fm_hci_init
I fm_hci : hci_initialize
</pre></div>
<p>Well, even without knowing nothing about all what these lines meant, I had
something I could search the internet for. So I found <a href="https://source.codeaurora.org/quic/la/platform/vendor/qcom-opensource/fm-commonsys">the Qualcomm FM code in
CodeAurora</a>,
and search for the code relative to my Snapdragon 675 (aka <code>sm6150</code>). I quickly
gave up on trying to make some sense out of the git tag naming in that
repository, and just tried to search for a tag which could be referring to my
device. I found one, and started browsing <a href="https://source.codeaurora.org/quic/la/platform/vendor/qcom-opensource/fm-commonsys/tree/?h=LA.UM.7.9.r1-11400-sm6150.0">its source
tree</a>.</p>
<p>It turns out that Qualcomm provides a Java package which applications can use,
and which internally <code>dlopen()</code>s the <code>fm_helium.so</code> library, which in turn
depends on the <code>libfm_hci.so</code> library. I had a quick look at the source code of
these libraries, which are also present in the repository, but decided that I
would have had more chances of success if I just tried to follow the JNI code,
and in particular the
<a href="https://source.codeaurora.org/quic/la/platform/vendor/qcom-opensource/fm-commonsys/tree/jni/android_hardware_fm.cpp?h=LA.UM.7.9.r1-11400-sm6150.0"><code>android_hardware_fm.cpp</code></a>
file. I'm not sure why this code is not using the C structure types defined in
the headers provided by the helium library, and instead redefines all the
constants and accesses the character buffers by offsets — it might be just for
historical reasons — but in any case I decided to follow along.</p>
<h4>The <code>fm-bridge</code> program</h4>
<p>Since we have a rather net separation between the Ubuntu Touch and the Android
worlds (the Android services are running inside an LXC container, with all
their Android libs and dependencies), one should not attempt to write an Ubuntu
process that loads the Android libraries, because the libc used in Android is
different, so things are likely not to work. But we can have Ubuntu and Android
processes communicate over a socket or other kind of IPC; so, what I decided to
go for, is writing a small C program that will live in the Android side, it
will talk to the FM radio (via <code>helium_fm</code>), and accept commands / give replies
via its <code>stdin</code> / <code>stdout</code>.</p>
<p>I unimaginatively called it “<code>fm-bridge</code>”, and you can look at its horrible
code <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/fm-bridge">here</a>. Really, I
just said it was terrible, so why did you look at it? I definitely need to
rewrite it from scratch, possibly using the helium headers, but as a proof of
concept this also works. Then I carefully examined the logcat output while
using the MIUI FM radio application in Android, and figured out what was the
command sequence I had to input into <code>fm-bridge</code>'s standard input in order to
have it tune onto a given frequency. I'm publishing the commands here too,
should I ever lose my notes:</p>
<div class="code"><pre class="code literal-block">enableSlimbus 1
setControl 0x8000004 1
enableSoftMute 1
setControl 0x8000029 0
setControl 0x800000c 1
setControl 0x800000d 1
setControl 0x800000e 1
setControl 0x800002b 0
setControl 0x8000007 4
setControl 0x8000006 0x40
setControl 0x8000006 0x40
setControl 0x8000011 0
setControl 0x800000f 1
getControl 0x8000010
setControl 0x8000010 0xef
setControl 0x800000f 1
setControl 0x800001b 1
setControl 0x8000012 0
setFreq 89300
setMonoStereo 1
</pre></div>
<p>I'm sure that not all of them are needed, but I'll figure out the optimal
sequence later. In order to use this program on Ubuntu Touch, I had to alter
the vendor partition to add this program, but also the <code>fm_helium.so</code> and
<code>libfm_hci.so</code> libraries (more on that below).</p>
<p>When feeding the above commands to the <code>fm-bridge</code> in Ubuntu, I saw that I was
getting a logcat output similar to the one from Android, which was mildly
comforting. No sound was comint out of the speaker or out of the earplugs, but
I was hardly expecting it all to work at the first try. And I got convinced
that the FM tuner was indeed working, because typing the command “<code>startSearch
1</code>” made a new frequency appear in the logs, proving that the tuner had found
another station and tuned onto it.</p>
<h4>Getting the sound out</h4>
<p>This was actually the easiest of the steps, thanks to the Ubuntu Touch FM radio
application we have for the Volla: its source code mentions a few pulseaudio
commands that worked perfectly in the Note 7 Pro too, despite the fact that the
underlying chipset is totally different. This should not be as surprising as it
might sound like, given that Android has a common audio API.</p>
<p>Just for my future reference, the commands are these:</p>
<div class="code"><pre class="code literal-block">pacmd set-source-port 1 input-fm_tuner
pactl load-module module-loopback source=1 sink=0
</pre></div>
<p>Ta-daaa! The radio was now playing from the phone loudspeakers! It was indeed
quite loud, and the volume buttons did not seem to have any effect on it, but
the volume can be controlled with pulseaudio:</p>
<div class="code"><pre class="code literal-block">pactl set-source-volume 1 50%
</pre></div>
<p>Of course, if we ever manage to make this into an Ubuntu Touch feature, we'll
have to find a way to make the volume respond to the volume buttons.</p>
<h4>Addind the needed files to the vendor partition</h4>
<p>The simplest approach (and the one I took initially) is that of downloading the
<code>vendor.img</code> into your PC, loop-mounting it, adding the <code>fm_helium.so</code>,
<code>libfm_hci.so</code> and <code>fm-bridge</code> files to it and then umount the partition and
reflash it (remembering to converting it from/to a sparse image before
downloading/uploading it). This approach works flawlessly, but I'm wondering if
one might incur into issues if the version of the NDK used to compile fm-bridge
is different from the one that was used to compile the other vendor binaries,
so I decided to give it a try to build the whole vendor partition myself.</p>
<p>This turned out to be a non trivial process, because I was using the Halium
tree to build the vendor image, and not the LineageOS which was used to build
the vendor image for my device: I could make an image, but it took some time
before I figured out which were the needed packages that somehow got lost
because of the Halium changes and that <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_vendor_xiaomi_violet/-/commit/495c83065c1f135d84c8e08563cece4fd4e4d3a5">had to be added to the
Makefile</a>.</p>
<p>To help my weak memory, I expanded the <a href="https://gitlab.com/ubports/community-ports/android9/xiaomi-redmi-note-7-pro/xiaomi-violet/-/blob/master/README.md">README file in the <code>violet</code>
port</a>
with the steps needed in order to build the vendor image.</p>
<h4>A system service for the FM radio</h4>
<p>While it could be possible for Ubuntu Touch applications to directly access the
FM radio device in the same way that Android applications do, this is
suboptimal for a few reasons. Even if we provided a shared library to deal with
the various radio chipset implementations, the application would either need to
be unconfined, or we'd had to provide an ever-changing AppArmor profile that
peeks new holes every time that a new device implementation is added (and what
if this implemenation uses a generic kernel device, which could be used for
other goals too?) and in any case we'd have to make this policy restricted,
since the RDS data provided by the radio stations would reveal the user
location (well, the city at least) to the application. Not to talk about
concurrent access to the radio device if two applications attempt to use it.</p>
<p>Therefore, my proposition (and what I'll implement, if I'll live long enough or
if someone doesn't beat me to it) is to have a system service deal with the
various hardware differences and expose a D-Bus API that will be hooked up as a
QRadioTunerControl plugin, so that Qt applications will be able to just use the
QtMultimedia APIs to access the radio.</p>
<p>The service would also need to talk to the trust-store, to let the user decide
whether the application should really be granted access to the FM receiver (and
when using the turst-store, this decision is remembered, and revocable from the
System Settings's Security panel). Of course we'll also need to add a
<code>fm-radio</code> AppArmor policy to let applications use this service.</p>Ubuntu Touch porting notes for the Redmi Note 7 Pro, part 2http://mardy.it/ia/blog/2021/02/ubuntu-touch-porting-notes-for-the-redmi-note-7-pro-part-2.html2021-02-04T23:34:28+03:002021-02-04T23:34:28+03:00Alberto Mardegan<p>This is the second part of my porting odyssey; for the first part, follow <a href="http://mardy.it/ia/blog/2021/01/ubuntu-touch-porting-notes-for-the-redmi-note-7-pro.html">this
link</a>.</p>
<p>The good news is that I've done some progress; the bad news is that there are
still plenty of issues, and solving them involves deep diving into nearly all
components and technologies that make up the core of an Android device, so
completing the porting is going to take quite some time. On the other hand, I'm
learning a lot of new stuff, and I might be able to share it by writing some
documentation. And, who knows, maybe work on some other device port.</p>
<p>Anyway, enough with the introduction! Let's see what progress I've been doing
so far.</p>
<h3>The new device tree</h3>
<p>While asking for help to debug the audio issue I was facing (more about that
later), I was also told that the <code>lavender</code> tree, which I was using as a
reference, was obsolete. The new one was <a href="https://gitlab.com/ubports/community-ports/android9/xiaomi-redmi-note-7/xiaomi-lavender">in
gitlab</a>,
and was build with a totally different system, described
<a href="https://github.com/ubports/porting-notes/wiki/GitLab-CI-builds-for-devices-based-on-halium_arm64-(Halium-9)">here</a>.</p>
<p>So, I picked the <code>lavender</code> tree and adapted it for <code>violet</code>: I changed the
<code>deviceinfo</code> file to point to my kernel tree, use my kernel configuration, and
use the same boot command line as before. By the way, <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/halium-xiaomi-violet">here's my current "new"
device
tree</a>. The
build failed, with errors like:</p>
<div class="code"><pre class="code literal-block">In function 'memcpy',
inlined from 'proc_ipc_auto_msgmni.part.1' at /home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/ipc/ipc_sysctl.c:82:2:
/home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/include/linux/string.h:340:4: error: call to '__read_overflow2' declared with attribute error: detected read beyond size of object passed as 2nd parameter
__read_overflow2();
^
</pre></div>
<p>I then replayed the kernel build using the old system, and noticed that it was
using clang as the compiler; so I changed the related flag in the <code>deviceinfo</code>
file, and the build went past that point. It failed later, though:</p>
<div class="code"><pre class="code literal-block">/home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/drivers/staging/qcacld-3.0/core/sap/src/sap_fsm.c:1498:26: error: cast to smaller integer type 'eSapStatus' from 'void *' [-Werror,-Wvoid-pointer-to-enum-cast]
bss_complete->status = (eSapStatus) context;
^~~~~~~~~~~~~~~~~~~~
</pre></div>
<p>I ended up editing the Kbuild file in the module source directory, and
<a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet/-/commit/58ee86d2faa154e40faa5787d4166384b3e41ee3">removed</a>
the <code>-Werror</code> from there (as well in another place that failed a bit later).
This made the build proceed until the end, where it failed because the device
tree file was not found:</p>
<div class="code"><pre class="code literal-block">+ cp -v /home/mardy/projects/port/xiaomi-violet/bd/downloads/KERNEL_OBJ/arch/arm64/boot/dts/qcom/sm8150-mtp-overlay.dtbo /home/mardy/projects/port/xiaomi-violet/bd/tmp/partitions/dtbo.img
cp: cannot stat '/home/mardy/projects/port/xiaomi-violet/bd/downloads/KERNEL_OBJ/arch/arm64/boot/dts/qcom/sm8150-mtp-overlay.dtbo': No such file or directory
</pre></div>
<p>I quickly realized that this was due to an error of mine: the Xiaomi <code>violet</code>
is a <code>sm6150</code>, not a <code>sm8150</code> as mentioned in my <code>deviceinfo</code> file! But what
overlay file should I use then, since there isn't a <code>sm6150-mtp-overlay.dts</code> in
my source tree? Having a loot at <a href="https://gitlab.com/ubports/community-ports/android9/xiaomi-poco-f1/xiaomi-beryllium/-/blob/master/deviceinfo">other deviceinfo
files</a>,
I saw that the <code>deviceinfo_kernel_dtb_overlay</code> line is not always there, so I
tried commenting it out, and it helped.</p>
<p>The next step was getting flashable images, of course. While the device is not
supported by the UBports system-image server, we can use use some scripts to
create a fake OTA (Over The Air update) and generate flashable images starting
from it. The steps can be read in the <code>devel-flashable</code> target of the
<code>.gitlab-ci.yml</code> file (if this step is not present, we should try to find
<a href="https://gitlab.com/ubports/community-ports/android9">another device
repository</a> which has it.
They are the following:</p>
<div class="code"><pre class="code literal-block">./build/prepare-fake-ota.sh<span class="w"> </span>out/device_violet.tar.xz<span class="w"> </span>ota
./build/system-image-from-ota.sh<span class="w"> </span>ota/ubuntu_command<span class="w"> </span>out
</pre></div>
<p>Once these commands have completed their execution, these commands will push
the images to the device:</p>
<div class="code"><pre class="code literal-block">fastboot<span class="w"> </span>flash<span class="w"> </span>boot<span class="w"> </span>out/boot.img<span class="p">;</span><span class="w"> </span>fastboot<span class="w"> </span>flash<span class="w"> </span>system<span class="w"> </span>out/system.img
</pre></div>
<p>There's also <code>fastboot flash recovery out/recovery.img</code>, but I left it out
since I was not interested in the recovery image at this stage. And unless you
are ready to submit your port for inclusion into the "supported devices" list,
I'd recommend not flashing the UT recovery image since TWRP is more powerful
and will likely help you in recover your device from a broken image.</p>
<h4>Kernel command line</h4>
<p>It's important that the kernel command line contains the <code>systempart</code>
parameter. In my case, the first boot failed because the command line was
longer than 512 bytes, and this parameter was getting truncated. So one thing
to be careful about is the length of the kernel command line.</p>
<p>This was fixed by <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/halium-xiaomi-violet/-/commit/25075c7e913501e2a142143f09e1e0598ecd6380">removing some unnecessary kernel
parameters</a>
from the <code>deviceinfo</code> file.</p>
<h3>Missing thumbnails in the Gallery app, content-hub not working</h3>
<p>Another issue I noticed is that photo thumbnails were all black in the Gallery
app, and the content-hub also was not working.
I noticed a <a href="https://github.com/ubuntu-touch-lavender/kernel/commit/2d0cd516735aaa88733dfc251e1e7dce9e549197">relevant commit in the lavender kernel
tree</a>
and found a <a href="https://bugs.launchpad.net/apparmor/+bug/1620635">launchpad bug</a>
which mentioned the issues I was seeing. In the Telegram channel I was told
that patch is a <em>forward</em> port of a commit from kernel 3.4 that was present in
all of the cores devices, and that it was indeed needed to have the ContentHub
working.</p>
<p>The patch did not apply cleanly on top of my kernel tree, but luckily it was
just an offset issue: <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet/-/commit/fba5fc33aee3147146199f8a3185b10b5c85b46c">adjusting the
patch</a>
was easy, and indeed after applying it thumbnails started appearing in the
Gallery and
<a href="http://imaginario.mardy.it">Imaginario</a> could import photos again via the
ContentHub.</p>
<h3>Time and date</h3>
<p>Time was always reset to a far away date after a reboot. This is a common issue
on Qualcomm devices, and can be fixed by disabling the time service in the
Android container.</p>
<p>https://github.com/Halium/android_device_halium_halium_arm64/pull/3</p>
<p>For some reason, at a certain point my override stopped working (or, more
likely, the override never worked, but I happened to have fixed the issue
directly modifying the vendor init file). I had to copy
<code>/android/vendor/etc/init/hw/init.qcom.rc</code> into <code>/usr/share/halium-overrides/</code>,
modify it and bind mount the modified file in order to get it working.</p>
<p>This actually seems to match my understanding of the <a href="https://android.googlesource.com/platform/system/core/+/master/init/README.md#imports">Android init
documentation</a>,
because according to the path priorities a configuration file stored under
<code>/system/</code> will never be able to override one stored under <code>/vendor</code>.</p>
<h3>Fixing the audio configuration</h3>
<p>Audio was not working at all. The sound indicator icon was not shown, only the
raw (unstranslated) "indicator-sound" text was shown in the panel. <code>pulseaudio</code>
was not running. Trying to run it manually (as the phablet user, since
pulseaudio is run in the user session) led to this:</p>
<div class="code"><pre class="code literal-block">phablet@ubuntu-phablet:~$ pulseaudio -n -vvv -F /etc/pulse/touch-android9.pa
I: [pulseaudio] main.c: setrlimit(RLIMIT_NICE, (31, 31)) failed: Operation not permitted
I: [pulseaudio] main.c: setrlimit(RLIMIT_RTPRIO, (9, 9)) failed: Operation not permitted
...
D: [pulseaudio] cli-command.c: Parsing script '/etc/pulse/touch-android9.pa'
D: [pulseaudio] database-tdb.c: Opened TDB database '/home/phablet/.config/pulse/ubuntu-phablet-stream-volumes.tdb'
I: [pulseaudio] module-stream-restore.c: Successfully opened database file '/home/phablet/.config/pulse/ubuntu-phablet-stream-volumes'.
...
D: [pulseaudio] module.c: Checking for existence of '/usr/lib/pulse-8.0/modules/module-droid-card-28.so': success
I: [pulseaudio] module-droid-card.c: Create new droid-card
D: [pulseaudio] droid-util.c: No configuration provided for opening module with id primary
I: [pulseaudio] config-parser-xml.c: Failed to open file (/odm/etc/audio_policy_configuration.xml): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /odm/etc/audio_policy_configuration.xml
D: [pulseaudio] config-parser-xml.c: Read /vendor/etc/audio/audio_policy_configuration.xml ...
D: [pulseaudio] config-parser-xml.c: New module: "primary"
W: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio/audio_policy_configuration.xml:78] Could not find element attribute "samplingRates"
E: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio/audio_policy_configuration.xml:78] Failed to parse element <profile>
E: [pulseaudio] config-parser-xml.c: parsing aborted at line 78
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio/audio_policy_configuration.xml
D: [pulseaudio] config-parser-xml.c: Read /vendor/etc/audio_policy_configuration.xml ...
D: [pulseaudio] config-parser-xml.c: New module: "primary"
W: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio_policy_configuration.xml:78] Could not find element attribute "samplingRates"
E: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio_policy_configuration.xml:78] Failed to parse element <profile>
E: [pulseaudio] config-parser-xml.c: parsing aborted at line 78
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio_policy_configuration.xml
I: [pulseaudio] config-parser-legacy.c: Failed to open config file (/vendor/etc/audio_policy.conf): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio_policy.conf
I: [pulseaudio] config-parser-xml.c: Failed to open file (/system/etc/audio_policy_configuration.xml): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /system/etc/audio_policy_configuration.xml
I: [pulseaudio] config-parser-legacy.c: Failed to open config file (/system/etc/audio_policy.conf): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /system/etc/audio_policy.conf
E: [pulseaudio] droid-config.c: Failed to parse any configuration.
...
</pre></div>
<p>Indeed, line 78 of <code>/vendor/etc/audio_policy_configuration.xml</code> had an error,
where a property was spelt like <code>simplingRate</code> instead of <code>samplingRate</code>.
However, the "vendor" partition is read-only, so I couldn't change that file
directly. Another option could have been creating a fixed copy of the file and
place it with <code>/system/etc/audio_policy_configuration.xml</code>, but the "system" is
also read-only (there are ways to modify these partitions, of course, but I
couldn't find a clean way to do it from the device tree scripts. So I went for
the bind-mount approach: I would ship the fixed file in some other directory of
the file-system, and then modify the <code>/etc/init/mount-android.conf</code> file (this
is the job that upstart executes before starting the Android LXC container) to
bind-mount the file onto <code>/vendor/etc/audio_policy_configuration.xml</code>.</p>
<p>This worked, but my joy was short-lived: audio was coming up only once every 5
boots or so. I will not list here all the things I tried, as they were plenty
of them; and more than once I went to sleep happy and convinced of having fixed
the issue for good, until the next day the device booted without audio. It was
clearly a timing issue occurring in the early boot, because one thing I clearly
noticed very early on is that in those cases when the audio was booting, the
following lines appeared in the kernel log:</p>
<div class="code"><pre class="code literal-block">[ 7.130057] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-ldo-rxtx: vol=[1800000 1800000]uV, curr=[25000]uA, ond 0
[ 7.130068] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vddpx-1: vol=[1800000 1800000]uV, curr=[10000]uA, ond 0
[ 7.130076] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-mic-bias: vol=[3296000 3296000]uV, curr=[25000]uA, ond 0
[ 7.130084] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-buck: vol=[1800000 1800000]uV, curr=[650000]uA, ond 1
[ 7.137759] wcd937x_codec wcd937x-codec: bound wcd937x-slave.1170224 (ops cleanup_module [wcd937x_slave_dlkm])
[ 7.138065] wcd937x_codec wcd937x-codec: bound wcd937x-slave.1170223 (ops cleanup_module [wcd937x_slave_dlkm])
</pre></div>
<p>I started adding some printk to the kernel driver, and modified
it slightly to register itself with the <code>module_driver()</code> macro instead of the
simpler, but logless, <code>module_platform_driver()</code>. This showed that the driver
was always loaded at about 7 seconds, but the <code>platform_driver_register()</code>
method only called the driver's bind method (<code>wcd937x_bind()</code>) in those boots
where audio was working.</p>
<p>After more debugging into <code>platform_driver_register()</code>, I stumbled upon the
<a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet/-/blob/halium-9.0/drivers/base/platform.c#L965"><code>platform_match()</code></a>
function, added some debugging message in there to print the device name and
the driver name, and observed how in those boots where audio was failing this
function was called to find a driver for the <code>wcd937x_codec</code> device <em>before</em>
the <code>wcd937x_codec</code> driver (provided by the <code>wcd937x_dlmk</code> module) was
available. So, I tried adding <code>wcd937x_dlmk</code> to
<code>/etc/modules-load.d/modules.conf</code> and this caused the driver to be loaded at
about 3 seconds, and apparently fixed the audio issue. At least, till the time
of writing this, I never had my phone boot without audio anymore.</p>
<p>Not all is fine with the audio, unfortunately: the mic records a background
noise along with the actual sounds, and the recording volume is quite low. This
also affects the call quality. On the other hand, the noise disappears when
recording happens via the earphones. But I've yet to investigate this; I hope
to give you some updates in part three.</p><p>This is the second part of my porting odyssey; for the first part, follow <a href="http://mardy.it/ia/blog/2021/01/ubuntu-touch-porting-notes-for-the-redmi-note-7-pro.html">this
link</a>.</p>
<p>The good news is that I've done some progress; the bad news is that there are
still plenty of issues, and solving them involves deep diving into nearly all
components and technologies that make up the core of an Android device, so
completing the porting is going to take quite some time. On the other hand, I'm
learning a lot of new stuff, and I might be able to share it by writing some
documentation. And, who knows, maybe work on some other device port.</p>
<p>Anyway, enough with the introduction! Let's see what progress I've been doing
so far.</p>
<h3>The new device tree</h3>
<p>While asking for help to debug the audio issue I was facing (more about that
later), I was also told that the <code>lavender</code> tree, which I was using as a
reference, was obsolete. The new one was <a href="https://gitlab.com/ubports/community-ports/android9/xiaomi-redmi-note-7/xiaomi-lavender">in
gitlab</a>,
and was build with a totally different system, described
<a href="https://github.com/ubports/porting-notes/wiki/GitLab-CI-builds-for-devices-based-on-halium_arm64-(Halium-9)">here</a>.</p>
<p>So, I picked the <code>lavender</code> tree and adapted it for <code>violet</code>: I changed the
<code>deviceinfo</code> file to point to my kernel tree, use my kernel configuration, and
use the same boot command line as before. By the way, <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/halium-xiaomi-violet">here's my current "new"
device
tree</a>. The
build failed, with errors like:</p>
<div class="code"><pre class="code literal-block">In function 'memcpy',
inlined from 'proc_ipc_auto_msgmni.part.1' at /home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/ipc/ipc_sysctl.c:82:2:
/home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/include/linux/string.h:340:4: error: call to '__read_overflow2' declared with attribute error: detected read beyond size of object passed as 2nd parameter
__read_overflow2();
^
</pre></div>
<p>I then replayed the kernel build using the old system, and noticed that it was
using clang as the compiler; so I changed the related flag in the <code>deviceinfo</code>
file, and the build went past that point. It failed later, though:</p>
<div class="code"><pre class="code literal-block">/home/mardy/projects/port/xiaomi-violet/bd/downloads/android_kernel_xiaomi_violet/drivers/staging/qcacld-3.0/core/sap/src/sap_fsm.c:1498:26: error: cast to smaller integer type 'eSapStatus' from 'void *' [-Werror,-Wvoid-pointer-to-enum-cast]
bss_complete->status = (eSapStatus) context;
^~~~~~~~~~~~~~~~~~~~
</pre></div>
<p>I ended up editing the Kbuild file in the module source directory, and
<a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet/-/commit/58ee86d2faa154e40faa5787d4166384b3e41ee3">removed</a>
the <code>-Werror</code> from there (as well in another place that failed a bit later).
This made the build proceed until the end, where it failed because the device
tree file was not found:</p>
<div class="code"><pre class="code literal-block">+ cp -v /home/mardy/projects/port/xiaomi-violet/bd/downloads/KERNEL_OBJ/arch/arm64/boot/dts/qcom/sm8150-mtp-overlay.dtbo /home/mardy/projects/port/xiaomi-violet/bd/tmp/partitions/dtbo.img
cp: cannot stat '/home/mardy/projects/port/xiaomi-violet/bd/downloads/KERNEL_OBJ/arch/arm64/boot/dts/qcom/sm8150-mtp-overlay.dtbo': No such file or directory
</pre></div>
<p>I quickly realized that this was due to an error of mine: the Xiaomi <code>violet</code>
is a <code>sm6150</code>, not a <code>sm8150</code> as mentioned in my <code>deviceinfo</code> file! But what
overlay file should I use then, since there isn't a <code>sm6150-mtp-overlay.dts</code> in
my source tree? Having a loot at <a href="https://gitlab.com/ubports/community-ports/android9/xiaomi-poco-f1/xiaomi-beryllium/-/blob/master/deviceinfo">other deviceinfo
files</a>,
I saw that the <code>deviceinfo_kernel_dtb_overlay</code> line is not always there, so I
tried commenting it out, and it helped.</p>
<p>The next step was getting flashable images, of course. While the device is not
supported by the UBports system-image server, we can use use some scripts to
create a fake OTA (Over The Air update) and generate flashable images starting
from it. The steps can be read in the <code>devel-flashable</code> target of the
<code>.gitlab-ci.yml</code> file (if this step is not present, we should try to find
<a href="https://gitlab.com/ubports/community-ports/android9">another device
repository</a> which has it.
They are the following:</p>
<div class="code"><pre class="code literal-block">./build/prepare-fake-ota.sh<span class="w"> </span>out/device_violet.tar.xz<span class="w"> </span>ota
./build/system-image-from-ota.sh<span class="w"> </span>ota/ubuntu_command<span class="w"> </span>out
</pre></div>
<p>Once these commands have completed their execution, these commands will push
the images to the device:</p>
<div class="code"><pre class="code literal-block">fastboot<span class="w"> </span>flash<span class="w"> </span>boot<span class="w"> </span>out/boot.img<span class="p">;</span><span class="w"> </span>fastboot<span class="w"> </span>flash<span class="w"> </span>system<span class="w"> </span>out/system.img
</pre></div>
<p>There's also <code>fastboot flash recovery out/recovery.img</code>, but I left it out
since I was not interested in the recovery image at this stage. And unless you
are ready to submit your port for inclusion into the "supported devices" list,
I'd recommend not flashing the UT recovery image since TWRP is more powerful
and will likely help you in recover your device from a broken image.</p>
<h4>Kernel command line</h4>
<p>It's important that the kernel command line contains the <code>systempart</code>
parameter. In my case, the first boot failed because the command line was
longer than 512 bytes, and this parameter was getting truncated. So one thing
to be careful about is the length of the kernel command line.</p>
<p>This was fixed by <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/halium-xiaomi-violet/-/commit/25075c7e913501e2a142143f09e1e0598ecd6380">removing some unnecessary kernel
parameters</a>
from the <code>deviceinfo</code> file.</p>
<h3>Missing thumbnails in the Gallery app, content-hub not working</h3>
<p>Another issue I noticed is that photo thumbnails were all black in the Gallery
app, and the content-hub also was not working.
I noticed a <a href="https://github.com/ubuntu-touch-lavender/kernel/commit/2d0cd516735aaa88733dfc251e1e7dce9e549197">relevant commit in the lavender kernel
tree</a>
and found a <a href="https://bugs.launchpad.net/apparmor/+bug/1620635">launchpad bug</a>
which mentioned the issues I was seeing. In the Telegram channel I was told
that patch is a <em>forward</em> port of a commit from kernel 3.4 that was present in
all of the cores devices, and that it was indeed needed to have the ContentHub
working.</p>
<p>The patch did not apply cleanly on top of my kernel tree, but luckily it was
just an offset issue: <a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet/-/commit/fba5fc33aee3147146199f8a3185b10b5c85b46c">adjusting the
patch</a>
was easy, and indeed after applying it thumbnails started appearing in the
Gallery and
<a href="http://imaginario.mardy.it">Imaginario</a> could import photos again via the
ContentHub.</p>
<h3>Time and date</h3>
<p>Time was always reset to a far away date after a reboot. This is a common issue
on Qualcomm devices, and can be fixed by disabling the time service in the
Android container.</p>
<p>https://github.com/Halium/android_device_halium_halium_arm64/pull/3</p>
<p>For some reason, at a certain point my override stopped working (or, more
likely, the override never worked, but I happened to have fixed the issue
directly modifying the vendor init file). I had to copy
<code>/android/vendor/etc/init/hw/init.qcom.rc</code> into <code>/usr/share/halium-overrides/</code>,
modify it and bind mount the modified file in order to get it working.</p>
<p>This actually seems to match my understanding of the <a href="https://android.googlesource.com/platform/system/core/+/master/init/README.md#imports">Android init
documentation</a>,
because according to the path priorities a configuration file stored under
<code>/system/</code> will never be able to override one stored under <code>/vendor</code>.</p>
<h3>Fixing the audio configuration</h3>
<p>Audio was not working at all. The sound indicator icon was not shown, only the
raw (unstranslated) "indicator-sound" text was shown in the panel. <code>pulseaudio</code>
was not running. Trying to run it manually (as the phablet user, since
pulseaudio is run in the user session) led to this:</p>
<div class="code"><pre class="code literal-block">phablet@ubuntu-phablet:~$ pulseaudio -n -vvv -F /etc/pulse/touch-android9.pa
I: [pulseaudio] main.c: setrlimit(RLIMIT_NICE, (31, 31)) failed: Operation not permitted
I: [pulseaudio] main.c: setrlimit(RLIMIT_RTPRIO, (9, 9)) failed: Operation not permitted
...
D: [pulseaudio] cli-command.c: Parsing script '/etc/pulse/touch-android9.pa'
D: [pulseaudio] database-tdb.c: Opened TDB database '/home/phablet/.config/pulse/ubuntu-phablet-stream-volumes.tdb'
I: [pulseaudio] module-stream-restore.c: Successfully opened database file '/home/phablet/.config/pulse/ubuntu-phablet-stream-volumes'.
...
D: [pulseaudio] module.c: Checking for existence of '/usr/lib/pulse-8.0/modules/module-droid-card-28.so': success
I: [pulseaudio] module-droid-card.c: Create new droid-card
D: [pulseaudio] droid-util.c: No configuration provided for opening module with id primary
I: [pulseaudio] config-parser-xml.c: Failed to open file (/odm/etc/audio_policy_configuration.xml): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /odm/etc/audio_policy_configuration.xml
D: [pulseaudio] config-parser-xml.c: Read /vendor/etc/audio/audio_policy_configuration.xml ...
D: [pulseaudio] config-parser-xml.c: New module: "primary"
W: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio/audio_policy_configuration.xml:78] Could not find element attribute "samplingRates"
E: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio/audio_policy_configuration.xml:78] Failed to parse element <profile>
E: [pulseaudio] config-parser-xml.c: parsing aborted at line 78
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio/audio_policy_configuration.xml
D: [pulseaudio] config-parser-xml.c: Read /vendor/etc/audio_policy_configuration.xml ...
D: [pulseaudio] config-parser-xml.c: New module: "primary"
W: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio_policy_configuration.xml:78] Could not find element attribute "samplingRates"
E: [pulseaudio] config-parser-xml.c: [/vendor/etc/audio_policy_configuration.xml:78] Failed to parse element <profile>
E: [pulseaudio] config-parser-xml.c: parsing aborted at line 78
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio_policy_configuration.xml
I: [pulseaudio] config-parser-legacy.c: Failed to open config file (/vendor/etc/audio_policy.conf): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /vendor/etc/audio_policy.conf
I: [pulseaudio] config-parser-xml.c: Failed to open file (/system/etc/audio_policy_configuration.xml): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /system/etc/audio_policy_configuration.xml
I: [pulseaudio] config-parser-legacy.c: Failed to open config file (/system/etc/audio_policy.conf): No such file or directory
D: [pulseaudio] droid-config.c: Failed to parse configuration from /system/etc/audio_policy.conf
E: [pulseaudio] droid-config.c: Failed to parse any configuration.
...
</pre></div>
<p>Indeed, line 78 of <code>/vendor/etc/audio_policy_configuration.xml</code> had an error,
where a property was spelt like <code>simplingRate</code> instead of <code>samplingRate</code>.
However, the "vendor" partition is read-only, so I couldn't change that file
directly. Another option could have been creating a fixed copy of the file and
place it with <code>/system/etc/audio_policy_configuration.xml</code>, but the "system" is
also read-only (there are ways to modify these partitions, of course, but I
couldn't find a clean way to do it from the device tree scripts. So I went for
the bind-mount approach: I would ship the fixed file in some other directory of
the file-system, and then modify the <code>/etc/init/mount-android.conf</code> file (this
is the job that upstart executes before starting the Android LXC container) to
bind-mount the file onto <code>/vendor/etc/audio_policy_configuration.xml</code>.</p>
<p>This worked, but my joy was short-lived: audio was coming up only once every 5
boots or so. I will not list here all the things I tried, as they were plenty
of them; and more than once I went to sleep happy and convinced of having fixed
the issue for good, until the next day the device booted without audio. It was
clearly a timing issue occurring in the early boot, because one thing I clearly
noticed very early on is that in those cases when the audio was booting, the
following lines appeared in the kernel log:</p>
<div class="code"><pre class="code literal-block">[ 7.130057] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-ldo-rxtx: vol=[1800000 1800000]uV, curr=[25000]uA, ond 0
[ 7.130068] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vddpx-1: vol=[1800000 1800000]uV, curr=[10000]uA, ond 0
[ 7.130076] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-mic-bias: vol=[3296000 3296000]uV, curr=[25000]uA, ond 0
[ 7.130084] wcd937x_codec wcd937x-codec: msm_cdc_dt_parse_vreg_info: cdc-vdd-buck: vol=[1800000 1800000]uV, curr=[650000]uA, ond 1
[ 7.137759] wcd937x_codec wcd937x-codec: bound wcd937x-slave.1170224 (ops cleanup_module [wcd937x_slave_dlkm])
[ 7.138065] wcd937x_codec wcd937x-codec: bound wcd937x-slave.1170223 (ops cleanup_module [wcd937x_slave_dlkm])
</pre></div>
<p>I started adding some printk to the kernel driver, and modified
it slightly to register itself with the <code>module_driver()</code> macro instead of the
simpler, but logless, <code>module_platform_driver()</code>. This showed that the driver
was always loaded at about 7 seconds, but the <code>platform_driver_register()</code>
method only called the driver's bind method (<code>wcd937x_bind()</code>) in those boots
where audio was working.</p>
<p>After more debugging into <code>platform_driver_register()</code>, I stumbled upon the
<a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet/-/blob/halium-9.0/drivers/base/platform.c#L965"><code>platform_match()</code></a>
function, added some debugging message in there to print the device name and
the driver name, and observed how in those boots where audio was failing this
function was called to find a driver for the <code>wcd937x_codec</code> device <em>before</em>
the <code>wcd937x_codec</code> driver (provided by the <code>wcd937x_dlmk</code> module) was
available. So, I tried adding <code>wcd937x_dlmk</code> to
<code>/etc/modules-load.d/modules.conf</code> and this caused the driver to be loaded at
about 3 seconds, and apparently fixed the audio issue. At least, till the time
of writing this, I never had my phone boot without audio anymore.</p>
<p>Not all is fine with the audio, unfortunately: the mic records a background
noise along with the actual sounds, and the recording volume is quite low. This
also affects the call quality. On the other hand, the noise disappears when
recording happens via the earphones. But I've yet to investigate this; I hope
to give you some updates in part three.</p>Ubuntu Touch porting notes for the Redmi Note 7 Prohttp://mardy.it/ia/blog/2021/01/ubuntu-touch-porting-notes-for-the-redmi-note-7-pro.html2021-01-17T22:15:01+03:002021-01-17T22:15:01+03:00Alberto Mardegan<p>In case you have a sense of <em>deja-vu</em> when reading this post, it's because
indeed this is not the first time I try porting a device to Ubuntu Touch. <a href="http://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html">The
previous
attempt</a>,
however, was with another phone model (and manufacturer), and did not have a
happy ending. This time it went better, although the real ending is still far
away; but at least I have something to celebrate.</p>
<h3>The phone</h3>
<p>I made myself a Christmas present and bought a Xiaomi Redmi Note 7 Pro, a
dual-SIM phone from 2019 with 6GB of RAM and 128GB of flash storage. To be
totally honest, I bought it by mistake: the phone I really wanted to buy is the
Redmi Note 7 (without the "Pro"), because it's a modern phone that is working
reasonable well with Ubuntu Touch. The online shop where I bought it from let
me choose some options, including the RAM size, so I chose the maximum
available (6GB) without being aware that this would mean that I would be buying
the "Pro" device — the shop did not alter the item name, so I couldn't really
know. Unfortunately, the two versions are two rather different beasts, powered
by different SoC; both are produced by Qualcomm, so they are not <em>that</em>
different, but it's enough to make the installation of Ubuntu Touch impossible.</p>
<p>But between the choice of retuning the phone to the shop and begin a new
porting adventure, I stood firm and went for the latter. Hopefully I won't
regret it (if everything goes bad, I can still use it with LineageOS, which
runs perfectly on it).</p>
<p>Moreover, there already exist a port of Ubuntu Touch for this phone, which
actually works reasonably well (I tried it briefly, and many things were
working), but the author claims to be a novice and indeed did not follow the
best git practices when working on the source code, so it's hard to understand
what was changed and why. But if you are looking for a quick way to get Ubuntu
Touch working on this phone, you are welcome to have a look <a href="https://t.me/note7prom">at this Telegram
channel</a>.</p>
<p>What follows are the raw notes of my attempts. They are here so that search
engines can index the error messages and the various logs, and hopefully help
someone hitting similar errors on other devices to find his way out.</p>
<h3>Getting Halium and the device source code</h3>
<p>Installed the dependencies like in the first step. <code>repo</code> was not found in the
Ubuntu 20.04 archives, but I had it installed anyway due to my work on a Yocto
device.</p>
<p>Since my device has Android 9 on it, I went for Halium 9:</p>
<div class="code"><pre class="code literal-block">repo<span class="w"> </span>init<span class="w"> </span>-u<span class="w"> </span>git://github.com/Halium/android.git<span class="w"> </span>-b<span class="w"> </span>halium-9.0<span class="w"> </span>--depth<span class="o">=</span><span class="m">1</span>
repo<span class="w"> </span>sync<span class="w"> </span>-c<span class="w"> </span>-j<span class="w"> </span><span class="m">16</span>
</pre></div>
<p>The official LineageOS repository for the Xiaomi Redmi Note 7 Pro is
<a href="https://github.com/LineageOS/android_device_xiaomi_violet">android_device_xiaomi_violet</a>,
but the <a href="https://download.lineageos.org/violet">page with the official build</a>
has been taken down (some DMCA violation, <a href="https://forum.xda-developers.com/t/rom-official-9-0-0-violet-lineageos-16-0-wrappedkey-fbe.3951524/post-81799135">if you believe the
forums</a>)
and no development has been happening since last year. A <a href="https://forum.xda-developers.com/t/unofficial-rom-10-0-9-0-lineageos-17-1-16-0-violet-q-pie.4050059/">more active forum
thread</a>
uses another repository which seems to be receiving more frequent updates, so I
chose to base my port on that.</p>
<p>I actually tested that LineageOS image on my phone, and verified that all the
hardware was working properly.</p>
<p>Initially, I created forks of the relevant repository under my own gitlab
account, but then I though (especially looking at the <a href="https://github.com/ubuntu-touch-lavender">Note 7 port</a>)
that creating a group just for this port would make people's life easier,
because they wouldn't need to navigate through my 1000 personal projects
to find what is relevant for this port. So, I created these forks:</p>
<ul>
<li><a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_device_xiaomi_violet">gitlab.com/ubuntu-touch-xiaomi-violet/android_device_xiaomi_violet</a></li>
<li><a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet">gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet</a></li>
<li><a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_vendor_xiaomi_violet">gitlab.com/ubuntu-touch-xiaomi-violet/android_vendor_xiaomi_violet</a></li>
</ul>
<p>Next, created the <code>halium/devices/manifests/xiaomi_violet.xml</code> file with this content:</p>
<div class="code"><pre class="code literal-block"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><manifest></span>
<span class="w"> </span><span class="cm"><!-- Remotes --></span>
<span class="w"> </span><span class="nt"><remote</span><span class="w"> </span><span class="na">name=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span>
<span class="w"> </span><span class="na">fetch=</span><span class="s">"https://gitlab.com/ubuntu-touch-xiaomi-violet"</span>
<span class="w"> </span><span class="na">revision=</span><span class="s">"halium-9.0"</span><span class="nt">/></span>
<span class="w"> </span><span class="cm"><!-- Device Tree --></span>
<span class="w"> </span><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"device/xiaomi/violet"</span>
<span class="w"> </span><span class="na">name=</span><span class="s">"android_device_xiaomi_violet"</span>
<span class="w"> </span><span class="na">remote=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span><span class="w"> </span><span class="nt">/></span>
<span class="w"> </span><span class="cm"><!-- Kernel --></span>
<span class="w"> </span><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"kernel/xiaomi/violet"</span>
<span class="w"> </span><span class="na">name=</span><span class="s">"android_kernel_xiaomi_violet"</span>
<span class="w"> </span><span class="na">remote=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span><span class="w"> </span><span class="nt">/></span>
<span class="w"> </span><span class="cm"><!-- Proprietary/Vendor blobs --></span>
<span class="w"> </span><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/xiaomi/violet"</span>
<span class="w"> </span><span class="na">name=</span><span class="s">"android_vendor_xiaomi_violet"</span>
<span class="w"> </span><span class="na">remote=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"></manifest></span>
</pre></div>
<p>Fetching all the sources mentioned in the manifest:</p>
<div class="code"><pre class="code literal-block">./halium/devices/setup violet
</pre></div>
<p>Full output:</p>
<div class="code"><pre class="code literal-block">*****************************************
I: Configuring for device xiaomi_violet
*****************************************
Fetching projects: 100% (393/393), done.
hardware/qcom/audio-caf/apq8084: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8916: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8952: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8960: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8974: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8994: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8996: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8998: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/sdm845: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/sm8150: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio/default: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/display: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/apq8084: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8916: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8952: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8960: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8974: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8994: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8996: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8998: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/sdm845: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/sm8150: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/media: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/apq8084: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8916: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8952: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8960: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8974: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8994: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8996: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8998: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/sdm845: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/sm8150: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/ril: Shared project LineageOS/android_hardware_ril found, disabling pruning.
hardware/ril-caf: Shared project LineageOS/android_hardware_ril found, disabling pruning.
hardware/qcom/wlan: Shared project LineageOS/android_hardware_qcom_wlan found, disabling pruning.
hardware/qcom/wlan-caf: Shared project LineageOS/android_hardware_qcom_wlan found, disabling pruning.
hardware/qcom/bt: Shared project LineageOS/android_hardware_qcom_bt found, disabling pruning.
hardware/qcom/bt-caf: Shared project LineageOS/android_hardware_qcom_bt found, disabling pruning.
Updating files: 100% (67031/67031), done.nel/testsUpdating files: 36% (24663/67031)
lineage/scripts/: discarding 1 commits
Updating files: 100% (1406/1406), done.ineageOS/android_vendor_qcom_opensource_thermal-engineUpdating files: 47% (666/1406)
Checking out projects: 100% (392/392), done.
*******************************************
I: Refreshing device vendor repository: device/xiaomi/violet
I: Processing proprietary blob file: device/xiaomi/violet/./proprietary-files.txt
I: Processing fstab file: device/xiaomi/violet/./rootdir/etc/fstab.qcom
I: Removing components relying on SettingsLib from: device/xiaomi/violet
*******************************************
</pre></div>
<h3>Starting the build</h3>
<p>Setting up the environment:</p>
<div class="code"><pre class="code literal-block">$ source build/envsetup.sh
including device/xiaomi/violet/vendorsetup.sh
including vendor/lineage/vendorsetup.sh
$
</pre></div>
<p>Running the breakfast command:</p>
<div class="code"><pre class="code literal-block">$ breakfast violet
including vendor/lineage/vendorsetup.sh
Trying dependencies-only mode on a non-existing device tree?
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=9
LINEAGE_VERSION=16.0-20210109-UNOFFICIAL-violet
TARGET_PRODUCT=lineage_violet
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm64
TARGET_ARCH_VARIANT=armv8-a
TARGET_CPU_VARIANT=kryo300
TARGET_2ND_ARCH=arm
TARGET_2ND_ARCH_VARIANT=armv8-a
TARGET_2ND_CPU_VARIANT=cortex-a75
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-5.4.0-58-generic-x86_64-Ubuntu-20.04.1-LTS
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=PQ3A.190801.002
OUT_DIR=/home/mardy/projects/port/halium/out
PRODUCT_SOONG_NAMESPACES= hardware/qcom/audio-caf/sm8150 hardware/qcom/display-caf/sm8150 hardware/qcom/media-caf/sm8150
============================================
</pre></div>
<h4>Building the kernel</h4>
<p>The configuration needs to be adapted for Halium. Locating the kernel config:</p>
<div class="code"><pre class="code literal-block">$ grep "TARGET_KERNEL_CONFIG" device/xiaomi/violet/BoardConfig.mk
TARGET_KERNEL_CONFIG := vendor/violet-perf_defconfig
$
</pre></div>
<p>Getting the Mer checker tool and running it:</p>
<div class="code"><pre class="code literal-block">git clone https://github.com/mer-hybris/mer-kernel-check
cd mer-kernel-check
./mer_verify_kernel_config ../kernel/xiaomi/violet/arch/arm64/configs/vendor/violet-perf_defconfig
</pre></div>
<p>Prints lots of warnings, starting with:</p>
<div class="code"><pre class="code literal-block">WARNING: kernel version missing, some reports maybe misleading
</pre></div>
<p>To help the tool, we need to let him know the kernel version. It can be seen at
the beginning of the kernel Makefile, located in
<code>../kernel/xiaomi/violet/Makefile</code>; in my case it was</p>
<div class="code"><pre class="code literal-block">VERSION = 4
PATCHLEVEL = 14
SUBLEVEL = 83
EXTRAVERSION =
</pre></div>
<p>So I edited the configuration file
<code>../kernel/xiaomi/violet/arch/arm64/configs/vendor/violet-perf_defconfig</code> and
added this line at the beginning:</p>
<div class="code"><pre class="code literal-block"># Version 4.14.83
</pre></div>
<p>then ran the checker tool again. This time the output was a long list of kernel
options that needed to be fixed, but as I went asking for some explanation in
the Telegram channel for Ubuntu Touch porters, I was told that I could/should
skip this test and instead use the <code>check-kernel-config</code> tool <a href="https://raw.githubusercontent.com/Halium/halium-boot/halium-9.0/check-kernel-config">from
Halium</a>.
I downloaded it, made it executable (<code>chmod +x check-kernel-config</code>) and ran
it:</p>
<div class="code"><pre class="code literal-block">./check-kernel-config kernel/xiaomi/violet/arch/arm64/configs/vendor/violet-perf_defconfig
[...lots of red and green lines...]
Config file checked, found 288 errors that I did not fix.
</pre></div>
<p>I ran it again with the <code>-w</code> option, and it reportedly fixed 287 errors. Weird,
does that mean that an error was still left? I ran the tool again (without
<code>-w</code>), and it reported 2 errors. Ran it once more in write mode, and if fixed
them. So, one might need to run it twice.</p>
<p>Next, added the line</p>
<div class="code"><pre class="code literal-block">BOARD_KERNEL_CMDLINE += console=tty0
</pre></div>
<p>in <code>device/xiaomi/violet/BoardConfig.mk</code>. One more thing that needs to be done
before starting the build is <a href="https://docs.ubports.com/en/latest/porting/halium_7-1/Building.html#fix-mount-points">fixing the mount
points</a>,
so I opened <code>device/xiaomi/violet/rootdir/etc/fstab.qcom</code> and changed the type
of the userdata partition from <code>f2fs</code> to <code>ext4</code>. There were no lines with the
<code>context</code> option, so that was the only change I needed to do.</p>
<p>At this point, while browsing throught the documentation, I found a link to
<a href="https://github.com/ubports/porting-notes/wiki/Halium-9">this page</a> which
contains some notes on Halium 9 porting, which are substantially different from
the official porting guide in halium.org (which I was already told in Telegram
to be obsolete in several points).</p>
<p>So, following the instructions from this new link I ran</p>
<div class="code"><pre class="code literal-block">hybris-patches/apply-patches.sh --mb
</pre></div>
<p>which completed successfully.</p>
<p>Then, continuing following the points from this page, I edited <code>device/xiaomi/violet/lineage_violet.mk</code>, commented out the lines</p>
<div class="code"><pre class="code literal-block"><span class="k">$(</span><span class="nv">call</span> <span class="nv">inherit-product</span>, <span class="k">$(</span><span class="nv">SRC_TARGET_DIR</span><span class="k">)</span>/<span class="nv">product</span>/<span class="nv">full_base_telephony.mk</span><span class="k">)</span>
<span class="c"># ...</span>
<span class="k">$(</span><span class="nv">call</span> <span class="nv">inherit-product</span>, <span class="nv">vendor</span>/<span class="nv">lineage</span>/<span class="nv">config</span>/<span class="nv">common_full_phone.mk</span><span class="k">)</span>
</pre></div>
<p>and replaced the first one with a similar line pointing to <code>halium.mk</code>. For the
record, a <code>find</code> revealed that the <code>SRC_TARGET_DIR</code> variable in my case was
<code>build/make/target/</code>. It contained also the <code>halium.mk</code> file, which was created
by the hybris patches before. As for removing the Java dependencies, I cound't
find any modules similar to those listed <a href="https://github.com/ubports-on-fxtec-pro1/proprietary_vendor_idealte/commit/65cfe7583aaa886709b6142d8b79c276a3028ad9">in this
commit</a>
in any of the makefiles in my source tree, so I just started the build:</p>
<div class="code"><pre class="code literal-block"><span class="nb">source</span><span class="w"> </span>build/envsetup.sh<span class="w"> </span><span class="o">&&</span><span class="w"> </span>breakfast<span class="w"> </span>violet
make<span class="w"> </span>halium-boot
</pre></div>
<p>This failed the first time with the compiler being killed (out of memory, most
likely), but it succeeded on the second run. There was a suspicious warning,
though:</p>
<div class="code"><pre class="code literal-block">drivers/input/touchscreen/Kconfig:1290:warning: multi-line strings not supported
</pre></div>
<p>And indeed my <code>kernel/xiaomi/violet/drivers/input/touchscreen/Kconfig</code> had this line:</p>
<div class="code"><pre class="code literal-block">source "drivers/input/touchscreen/ft8719_touch_f7b/Kconfig
</pre></div>
<p>(notice the unterminated string quote). I don't know if this had any impact on
the build, but just to be on the safe side I added the missing quote and
rebuilt.</p>
<h4>Building the system image</h4>
<p>Running</p>
<div class="code"><pre class="code literal-block">make systemimage
</pre></div>
<p>failed pretty soon:</p>
<div class="code"><pre class="code literal-block">ninja: error: '/home/mardy/projects/port/halium/out/soong/host/linux-x86/framework/turbine.jar', needed by '/home/mardy/projects/port/halium/out/soong/.intermediates/libcore/core-oj/android_common/turbine/core-oj.jar', missing and no known rule to make it
</pre></div>
<p>The error is due to all the Java-related stuff that I should have disabled but
couldn't find. So, I tried to have a look at the <a href="https://github.com/ubuntu-touch-lavender/android_device_xiaomi_lavender/commit/ad640e121d0dc75017f31d860332327f8acd5d36">changes made on another
xiaomi
device</a>
(<code>lavender</code>, the Redmi note 7, which might not be that different, I thought)
and started editing <code>device/xiaomi/violet/device.mk</code> and removing a couple of
Android packages. Eventually the build proceeded, just to stop at a python
error:</p>
<div class="code"><pre class="code literal-block"> File "build/make/tools/check_radio_versions.py", line 56
print "*** Error opening \"%s.sha1\"; can't verify %s" % (fn, key)
^
SyntaxError: invalid syntax
</pre></div>
<p>Yes, it's the python3 vs python2 issue, since in my system <code>python</code> is python
version 3. In order to fix it, I created a virtual environment:</p>
<div class="code"><pre class="code literal-block">virtualenv --python 2.7 ../python27 # adjust the path to your prefs
source ../python27/bin/activate
</pre></div>
<p>Remember that the second line must be run every time you'll need to setup the
Halium build environment (that is, every time you run <code>breakfast</code>).</p>
<p>The build then proceeded for several minutes, until it failed due to some unresolved symbols:</p>
<div class="code"><pre class="code literal-block">prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/aarch64-linux-android/bin/ld.gold: error: /home/mardy/projects/port/halium/out/target/product/violet/obj/STATIC_LIBRARIES/lib_driver_cmd_qcwcn_intermediates/lib_driver_cmd_qcwcn.a: member at 7694 is not an ELF object
external/wpa_supplicant_8/hostapd/src/drivers/driver_nl80211.c:7936: error: undefined reference to 'wpa_driver_set_p2p_ps'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_set_ap_wps_p2p_ie'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_get_p2p_noa'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_set_p2p_noa'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_nl80211_driver_cmd'
</pre></div>
<p>As people told me in the Telegram channel, this driver is not used since "our
<code>wpa_supplicant</code> talks to kernel directly". OK, so I simply disabled the
driver, copying again from the <a href="https://github.com/ubuntu-touch-lavender/android_device_xiaomi_lavender/commit/ad640e121d0dc75017f31d860332327f8acd5d36"><code>lavender</code> Halium
changes</a>:
commented out the <code>BOARD_WLAN_DEVICE := qcwcn</code> line (and all lines referring
this variable) from <code>device/xiaomi/violet/BoardConfig.mk</code> and ran <code>make
systemimage</code> again. This time, surprisingly, it all worked.</p>
<h4>Try it out</h4>
<p>I first tried to flash the kernel only. I rebooted into fastboot, and on my PC ran this:</p>
<div class="code"><pre class="code literal-block">cout
fastboot<span class="w"> </span>flash<span class="w"> </span>boot<span class="w"> </span>halium-boot.img
</pre></div>
<p>I then rebooted my phone, but after showing the boot logo for a few seconds it
would jump to the fastboot screen. Indeed, flashing the previous kernel would
restore the normal boot, so there had to be something wrong with my own kernel.</p>
<p>While looking at what the problem could be, I noticed that in the <code>lavender</code>
port the author did not modify the lineageos kernel config file in place, but
instead created a new one and changed the <code>BoardConfig.mk</code> file to point to his
new copy. Since it sounded like a good idea, I did the same and created
<code>kernel/xiaomi/violet/arch/arm64/configs/vendor/violet_halium_defconfig</code> for
the Halium changes. And then the line in the board config file became</p>
<div class="code"><pre class="code literal-block">TARGET_KERNEL_CONFIG := vendor/violet_halium_defconfig
</pre></div>
<p>I then continued investigating the boot issue, and I was told that it might
have been due to an Android option, <code>skip_initramfs</code>, which is set by the
bootloader and causes our Halium boot to fail. The fix is to just disable this
option in the kernel, by editing <code>init/initramfs.c</code> and change the
<code>skip_initramfs_param</code> function to always set the <code>do_skip_initramfs</code> variable
to <code>0</code>, rather than to <code>1</code>. After doing this, the boot proceeded to show the
Ubuntu splash screen with the five dots being lit, but it didn't proceed from
there.</p>
<h5>Setting up the udev rules</h5>
<p>Even in this state, the device was detected by my host PC and these lines
appeared in the system log:</p>
<div class="code"><pre class="code literal-block">kernel: usb 1-3: new high-speed USB device number 26 using xhci_hcd
kernel: usb 1-3: New USB device found, idVendor=0fce, idProduct=7169, bcdDevice= 4.14
kernel: usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
kernel: usb 1-3: Product: Unknown
kernel: usb 1-3: Manufacturer: GNU/Linux Device
kernel: usb 1-3: SerialNumber: GNU/Linux Device on usb0 10.15.19.82
</pre></div>
<p>Indeed, I guess the reason I could do this without even flashing my systemimage
is because I had first flashed another UT system image from another porter. I'm
not sure if I'd had the same results with my own image. Anyway, the USB
networking was there, so I connected and ran the following commands to generate
the udev rules:</p>
<div class="code"><pre class="code literal-block">ssh<span class="w"> </span>phablet@10.15.19.82
<span class="c1"># used 0000 as password</span>
sudo<span class="w"> </span>-i
<span class="c1"># same password again</span>
<span class="nb">cd</span><span class="w"> </span>/home/phablet
<span class="nv">DEVICE</span><span class="o">=</span>violet
cat<span class="w"> </span>/var/lib/lxc/android/rootfs/ueventd*.rc<span class="w"> </span>/vendor/ueventd*.rc<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^/dev<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/^\/dev\///'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{printf "ACTION==\"add\", KERNEL==\"%s\", OWNER=\"%s\", GROUP=\"%s\", MODE=\"%s\"\n",$1,$3,$4,$2}'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/\r//'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>>70-<span class="nv">$DEVICE</span>.rules
</pre></div>
<p>I then copied (with <code>scp</code>) this file to my host PC, and I moved it to
<code>device/xiaomi/violet/ubuntu/70-violet.rules</code> (I had to create the <code>ubuntu</code>
directory first). Then I edited the <code>device/xiaomi/violet/device.mk</code> file and
added these lines at the end:</p>
<div class="code"><pre class="code literal-block"><span class="c">### Ubuntu Touch ###</span>
<span class="nv">PRODUCT_COPY_FILES</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="k">$(</span>LOCAL_PATH<span class="k">)</span>/ubuntu/70-violet.rules:system/halium/lib/udev/rules.d/70-android.rules
<span class="c">### End Ubuntu Touch ###</span>
</pre></div>
<p>It was now time to try my own system image. I rebooted my device into TWRP, I
cloned the <a href="https://gitlab.com/JBBgameich/halium-install">halium-install
repository</a> into my halium build
dir, downloaded <a href="https://ci.ubports.com/job/xenial-hybris-android9-rootfs-arm64/107/">a rootfs for
Halium9</a>,
and ran</p>
<div class="code"><pre class="code literal-block">./halium-install/halium-install<span class="w"> </span>-p<span class="w"> </span>ut<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>~/Downloads/ubuntu-touch-android9-arm64.tar.gz<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>out/target/product/violet/system.img
</pre></div>
<p>The first time this failed because the <code>simg2img</code> tool was not installed. The
second time it proceeded to create the image, asked me for a password for the
phablet user (gave "0000") and pushed the image onto the device, into the
<code>/data/</code> partition. I then rebooted.</p>
<h5>My first Ubuntu Touch image</h5>
<p>Upon reboot, the usual splash screen appeared, followed by several seconds of
black screen. It definitely didn't look right, but at least it proved that
<em>something</em> had been flashed. After some more seconds, to my big surprise, the
Ubuntu boot screen appeared, just with a smaller logo than how it used to be
before, which also confimed that my system image was being used — in fact, I
did not adjust the <code>GRID_UNIT_PX</code> variable before. Since this was an easy fix,
I chose to focus on that, rather than fix the boot issues (indeed, my device
did not move on from the Ubuntu boot screen). SSH was working.</p>
<p>I took the scaling.conf file from the <a href="https://github.com/ubuntu-touch-lavender/android_device_xiaomi_lavender/commit/e02875e15a09e96635d32aaf9427fcd057d7a9ae">lavender
changes</a>,
put it in <code>device/xiaomi/violet/ubuntu/</code> and added this line in the
<code>PRODUCT_COPY_FILES</code> in <code>device.mk</code>:</p>
<div class="code"><pre class="code literal-block">$(LOCAL_PATH)/ubuntu/scaling.conf:system/halium/etc/ubuntu-touch-session.d/android.conf
</pre></div>
<p>I initially used <code>system/ubuntu/etc/...</code> as the target destination for the
config file, like in the lavender commit, but this didn't work out for me. Then
I changed the path to start with <code>system/halium/</code>, like it's mentioned in the
ubports documentation, but it apparently had no effect either.</p>
<p>After a couple of days spent trying to understand why my files were not
appearing under <code>/etc</code>, it turned out that the device was not using my system
image at all: with <code>halium-install</code> I had my image installed in
<code>/userdata/system.img</code>, while the correct path for Halium 9 devices is
<code>/userdata/android-rootfs.img</code>. I was told that the option <code>-s</code> of
<code>halium-install</code> would do the trick. Instead of re-running the script, I took
the shortcut of renaming <code>/userdata/system.img</code> to
<code>/userdata/android-rootfs.img</code> and after rebooting I could see that the Ubuntu
logo was displayed at the correct size. And indeed my system image was being
used.</p>
<p>So I started to debug why unity8 didn't start. The logs terminated with this line:</p>
<div class="code"><pre class="code literal-block">terminate called after throwing an instance of 'std::runtime_error'
what(): org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying
initctl: Event failed
</pre></div>
<p>It means, that unity8 did not handle correctly the situation where another
D-Bus service crashed while handing a call from unity8. The problem now was how
to figure out which service it was. I ran <code>dbus-monitor</code> and restarted unity8
(<code>initctl start unity8</code>), then examined the dbus logs; I saw the point where
unity8 got disconnected from the bus, but before that point I didn't find any
failed D-Bus calls. So it had to be the system bus. I did exactly the same
steps, just this time after running <code>dbus-monitor --system</code> as root, I found
the place where unity8 got disconnected, and found this D-bus error shortly
before that:</p>
<div class="code"><pre class="code literal-block">method call time=1605676421.968667 sender=:1.306 -> destination=com.ubuntu.biometryd.Service serial=3 path=/default_device; interface=com.ubuntu.biometryd.Device; member=Identifier
method return time=1605676421.970222 sender=:1.283 -> destination=:1.306 serial=4 reply_serial=3
object path "/default_device/identifier"
...
method call time=1605676421.974218 sender=:1.283 -> destination=org.freedesktop.DBus serial=6 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=GetConnectionAppArmorSecurityContext
string ":1.306"
error time=1605676421.974278 sender=org.freedesktop.DBus -> destination=:1.283 error_name=org.freedesktop.DBus.Error.AppArmorSecurityContextUnknown reply_serial=6
string "Could not determine security context for ':1.306'"
...
signal time=1605676421.989074 sender=org.freedesktop.DBus -> destination=:1.283 serial=6 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameLost
string "com.ubuntu.biometryd.Service"
...
error time=1605676421.989302 sender=org.freedesktop.DBus -> destination=:1.306 error_name=org.freedesktop.DBus.Error.NoReply reply_serial=4
string "Message recipient disconnected from message bus without replying"
</pre></div>
<p>So, it looks like unity8 (<code>:1.306</code>) made a request to biometryd, who asked the
D-Bus daemon what was the AppArmor label of the caller, but since I didn't
backport the D-Bus mediation patches for AppArmor, the label could not be
resolved. <code>biometryd</code> decided to crash instead of properly handling the error,
and so did unity8.</p>
<p>So I backported the AppArmor patches. I took them from the <a href="https://kernel.ubuntu.com/git/jj/linux-apparmor-backports/log/?h=v4.15%2b-apparmor-backport-to-v4.14-presquash">Canonical kernel
repo</a>,
but they did not apply cleanly because they are meant to be applied on top of a
pristine 4.14 branch, whereas the Android kernel had already some AppArmor
security fixes backported from later releases. So I set and quickly inspected
the contents of each patch, and found out that a couple of them had already
been applied, and the last patch had to be applied as the first (yeah, it's
hard to explain, but it all depends on other patches that Android has
backported from newer kernels). Anyway, after rebuilding the kernel and
reflashing it, my phone could finally boot to unity8!</p>
<h3>Conclusion (of the first episode)</h3>
<p>The actual porting, as I've been told, starts here. What has been documented
here are only the very first steps of the bring-up; what awaits me now is to
make all the hardware subsystems work properly, and this, according to people
more experienced in porting, is the harder part.</p>
<p>So far, very few things work, to the point that it's faster to me to list the
things that <em>do work</em>; it's safe to assume that all what is not listed here is
not working:</p>
<ul>
<li>Mir, with graphics and input: Unity8 starts and is usable</li>
<li>Camera: can take photos; video recording start but an error appears when the
stop button is pressed</li>
<li>Flashlight</li>
<li>Fingerprint reader: surprisingly, this worked out of the box</li>
<li>Screen brightness (though it can be changed only manually)</li>
</ul>
<p>For all the rest, please keep an eye on this blog: I'll write, when I make some
progress!</p><p>In case you have a sense of <em>deja-vu</em> when reading this post, it's because
indeed this is not the first time I try porting a device to Ubuntu Touch. <a href="http://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html">The
previous
attempt</a>,
however, was with another phone model (and manufacturer), and did not have a
happy ending. This time it went better, although the real ending is still far
away; but at least I have something to celebrate.</p>
<h3>The phone</h3>
<p>I made myself a Christmas present and bought a Xiaomi Redmi Note 7 Pro, a
dual-SIM phone from 2019 with 6GB of RAM and 128GB of flash storage. To be
totally honest, I bought it by mistake: the phone I really wanted to buy is the
Redmi Note 7 (without the "Pro"), because it's a modern phone that is working
reasonable well with Ubuntu Touch. The online shop where I bought it from let
me choose some options, including the RAM size, so I chose the maximum
available (6GB) without being aware that this would mean that I would be buying
the "Pro" device — the shop did not alter the item name, so I couldn't really
know. Unfortunately, the two versions are two rather different beasts, powered
by different SoC; both are produced by Qualcomm, so they are not <em>that</em>
different, but it's enough to make the installation of Ubuntu Touch impossible.</p>
<p>But between the choice of retuning the phone to the shop and begin a new
porting adventure, I stood firm and went for the latter. Hopefully I won't
regret it (if everything goes bad, I can still use it with LineageOS, which
runs perfectly on it).</p>
<p>Moreover, there already exist a port of Ubuntu Touch for this phone, which
actually works reasonably well (I tried it briefly, and many things were
working), but the author claims to be a novice and indeed did not follow the
best git practices when working on the source code, so it's hard to understand
what was changed and why. But if you are looking for a quick way to get Ubuntu
Touch working on this phone, you are welcome to have a look <a href="https://t.me/note7prom">at this Telegram
channel</a>.</p>
<p>What follows are the raw notes of my attempts. They are here so that search
engines can index the error messages and the various logs, and hopefully help
someone hitting similar errors on other devices to find his way out.</p>
<h3>Getting Halium and the device source code</h3>
<p>Installed the dependencies like in the first step. <code>repo</code> was not found in the
Ubuntu 20.04 archives, but I had it installed anyway due to my work on a Yocto
device.</p>
<p>Since my device has Android 9 on it, I went for Halium 9:</p>
<div class="code"><pre class="code literal-block">repo<span class="w"> </span>init<span class="w"> </span>-u<span class="w"> </span>git://github.com/Halium/android.git<span class="w"> </span>-b<span class="w"> </span>halium-9.0<span class="w"> </span>--depth<span class="o">=</span><span class="m">1</span>
repo<span class="w"> </span>sync<span class="w"> </span>-c<span class="w"> </span>-j<span class="w"> </span><span class="m">16</span>
</pre></div>
<p>The official LineageOS repository for the Xiaomi Redmi Note 7 Pro is
<a href="https://github.com/LineageOS/android_device_xiaomi_violet">android_device_xiaomi_violet</a>,
but the <a href="https://download.lineageos.org/violet">page with the official build</a>
has been taken down (some DMCA violation, <a href="https://forum.xda-developers.com/t/rom-official-9-0-0-violet-lineageos-16-0-wrappedkey-fbe.3951524/post-81799135">if you believe the
forums</a>)
and no development has been happening since last year. A <a href="https://forum.xda-developers.com/t/unofficial-rom-10-0-9-0-lineageos-17-1-16-0-violet-q-pie.4050059/">more active forum
thread</a>
uses another repository which seems to be receiving more frequent updates, so I
chose to base my port on that.</p>
<p>I actually tested that LineageOS image on my phone, and verified that all the
hardware was working properly.</p>
<p>Initially, I created forks of the relevant repository under my own gitlab
account, but then I though (especially looking at the <a href="https://github.com/ubuntu-touch-lavender">Note 7 port</a>)
that creating a group just for this port would make people's life easier,
because they wouldn't need to navigate through my 1000 personal projects
to find what is relevant for this port. So, I created these forks:</p>
<ul>
<li><a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_device_xiaomi_violet">gitlab.com/ubuntu-touch-xiaomi-violet/android_device_xiaomi_violet</a></li>
<li><a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet">gitlab.com/ubuntu-touch-xiaomi-violet/android_kernel_xiaomi_violet</a></li>
<li><a href="https://gitlab.com/ubuntu-touch-xiaomi-violet/android_vendor_xiaomi_violet">gitlab.com/ubuntu-touch-xiaomi-violet/android_vendor_xiaomi_violet</a></li>
</ul>
<p>Next, created the <code>halium/devices/manifests/xiaomi_violet.xml</code> file with this content:</p>
<div class="code"><pre class="code literal-block"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><manifest></span>
<span class="w"> </span><span class="cm"><!-- Remotes --></span>
<span class="w"> </span><span class="nt"><remote</span><span class="w"> </span><span class="na">name=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span>
<span class="w"> </span><span class="na">fetch=</span><span class="s">"https://gitlab.com/ubuntu-touch-xiaomi-violet"</span>
<span class="w"> </span><span class="na">revision=</span><span class="s">"halium-9.0"</span><span class="nt">/></span>
<span class="w"> </span><span class="cm"><!-- Device Tree --></span>
<span class="w"> </span><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"device/xiaomi/violet"</span>
<span class="w"> </span><span class="na">name=</span><span class="s">"android_device_xiaomi_violet"</span>
<span class="w"> </span><span class="na">remote=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span><span class="w"> </span><span class="nt">/></span>
<span class="w"> </span><span class="cm"><!-- Kernel --></span>
<span class="w"> </span><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"kernel/xiaomi/violet"</span>
<span class="w"> </span><span class="na">name=</span><span class="s">"android_kernel_xiaomi_violet"</span>
<span class="w"> </span><span class="na">remote=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span><span class="w"> </span><span class="nt">/></span>
<span class="w"> </span><span class="cm"><!-- Proprietary/Vendor blobs --></span>
<span class="w"> </span><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/xiaomi/violet"</span>
<span class="w"> </span><span class="na">name=</span><span class="s">"android_vendor_xiaomi_violet"</span>
<span class="w"> </span><span class="na">remote=</span><span class="s">"ubuntu-touch-xiaomi-violet"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"></manifest></span>
</pre></div>
<p>Fetching all the sources mentioned in the manifest:</p>
<div class="code"><pre class="code literal-block">./halium/devices/setup violet
</pre></div>
<p>Full output:</p>
<div class="code"><pre class="code literal-block">*****************************************
I: Configuring for device xiaomi_violet
*****************************************
Fetching projects: 100% (393/393), done.
hardware/qcom/audio-caf/apq8084: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8916: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8952: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8960: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8974: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8994: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8996: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/msm8998: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/sdm845: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio-caf/sm8150: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/audio/default: Shared project LineageOS/android_hardware_qcom_audio found, disabling pruning.
hardware/qcom/display: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/apq8084: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8916: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8952: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8960: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8974: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8994: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8996: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/msm8998: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/sdm845: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/display-caf/sm8150: Shared project LineageOS/android_hardware_qcom_display found, disabling pruning.
hardware/qcom/media: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/apq8084: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8916: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8952: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8960: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8974: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8994: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8996: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/msm8998: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/sdm845: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/qcom/media-caf/sm8150: Shared project LineageOS/android_hardware_qcom_media found, disabling pruning.
hardware/ril: Shared project LineageOS/android_hardware_ril found, disabling pruning.
hardware/ril-caf: Shared project LineageOS/android_hardware_ril found, disabling pruning.
hardware/qcom/wlan: Shared project LineageOS/android_hardware_qcom_wlan found, disabling pruning.
hardware/qcom/wlan-caf: Shared project LineageOS/android_hardware_qcom_wlan found, disabling pruning.
hardware/qcom/bt: Shared project LineageOS/android_hardware_qcom_bt found, disabling pruning.
hardware/qcom/bt-caf: Shared project LineageOS/android_hardware_qcom_bt found, disabling pruning.
Updating files: 100% (67031/67031), done.nel/testsUpdating files: 36% (24663/67031)
lineage/scripts/: discarding 1 commits
Updating files: 100% (1406/1406), done.ineageOS/android_vendor_qcom_opensource_thermal-engineUpdating files: 47% (666/1406)
Checking out projects: 100% (392/392), done.
*******************************************
I: Refreshing device vendor repository: device/xiaomi/violet
I: Processing proprietary blob file: device/xiaomi/violet/./proprietary-files.txt
I: Processing fstab file: device/xiaomi/violet/./rootdir/etc/fstab.qcom
I: Removing components relying on SettingsLib from: device/xiaomi/violet
*******************************************
</pre></div>
<h3>Starting the build</h3>
<p>Setting up the environment:</p>
<div class="code"><pre class="code literal-block">$ source build/envsetup.sh
including device/xiaomi/violet/vendorsetup.sh
including vendor/lineage/vendorsetup.sh
$
</pre></div>
<p>Running the breakfast command:</p>
<div class="code"><pre class="code literal-block">$ breakfast violet
including vendor/lineage/vendorsetup.sh
Trying dependencies-only mode on a non-existing device tree?
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=9
LINEAGE_VERSION=16.0-20210109-UNOFFICIAL-violet
TARGET_PRODUCT=lineage_violet
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm64
TARGET_ARCH_VARIANT=armv8-a
TARGET_CPU_VARIANT=kryo300
TARGET_2ND_ARCH=arm
TARGET_2ND_ARCH_VARIANT=armv8-a
TARGET_2ND_CPU_VARIANT=cortex-a75
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-5.4.0-58-generic-x86_64-Ubuntu-20.04.1-LTS
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=PQ3A.190801.002
OUT_DIR=/home/mardy/projects/port/halium/out
PRODUCT_SOONG_NAMESPACES= hardware/qcom/audio-caf/sm8150 hardware/qcom/display-caf/sm8150 hardware/qcom/media-caf/sm8150
============================================
</pre></div>
<h4>Building the kernel</h4>
<p>The configuration needs to be adapted for Halium. Locating the kernel config:</p>
<div class="code"><pre class="code literal-block">$ grep "TARGET_KERNEL_CONFIG" device/xiaomi/violet/BoardConfig.mk
TARGET_KERNEL_CONFIG := vendor/violet-perf_defconfig
$
</pre></div>
<p>Getting the Mer checker tool and running it:</p>
<div class="code"><pre class="code literal-block">git clone https://github.com/mer-hybris/mer-kernel-check
cd mer-kernel-check
./mer_verify_kernel_config ../kernel/xiaomi/violet/arch/arm64/configs/vendor/violet-perf_defconfig
</pre></div>
<p>Prints lots of warnings, starting with:</p>
<div class="code"><pre class="code literal-block">WARNING: kernel version missing, some reports maybe misleading
</pre></div>
<p>To help the tool, we need to let him know the kernel version. It can be seen at
the beginning of the kernel Makefile, located in
<code>../kernel/xiaomi/violet/Makefile</code>; in my case it was</p>
<div class="code"><pre class="code literal-block">VERSION = 4
PATCHLEVEL = 14
SUBLEVEL = 83
EXTRAVERSION =
</pre></div>
<p>So I edited the configuration file
<code>../kernel/xiaomi/violet/arch/arm64/configs/vendor/violet-perf_defconfig</code> and
added this line at the beginning:</p>
<div class="code"><pre class="code literal-block"># Version 4.14.83
</pre></div>
<p>then ran the checker tool again. This time the output was a long list of kernel
options that needed to be fixed, but as I went asking for some explanation in
the Telegram channel for Ubuntu Touch porters, I was told that I could/should
skip this test and instead use the <code>check-kernel-config</code> tool <a href="https://raw.githubusercontent.com/Halium/halium-boot/halium-9.0/check-kernel-config">from
Halium</a>.
I downloaded it, made it executable (<code>chmod +x check-kernel-config</code>) and ran
it:</p>
<div class="code"><pre class="code literal-block">./check-kernel-config kernel/xiaomi/violet/arch/arm64/configs/vendor/violet-perf_defconfig
[...lots of red and green lines...]
Config file checked, found 288 errors that I did not fix.
</pre></div>
<p>I ran it again with the <code>-w</code> option, and it reportedly fixed 287 errors. Weird,
does that mean that an error was still left? I ran the tool again (without
<code>-w</code>), and it reported 2 errors. Ran it once more in write mode, and if fixed
them. So, one might need to run it twice.</p>
<p>Next, added the line</p>
<div class="code"><pre class="code literal-block">BOARD_KERNEL_CMDLINE += console=tty0
</pre></div>
<p>in <code>device/xiaomi/violet/BoardConfig.mk</code>. One more thing that needs to be done
before starting the build is <a href="https://docs.ubports.com/en/latest/porting/halium_7-1/Building.html#fix-mount-points">fixing the mount
points</a>,
so I opened <code>device/xiaomi/violet/rootdir/etc/fstab.qcom</code> and changed the type
of the userdata partition from <code>f2fs</code> to <code>ext4</code>. There were no lines with the
<code>context</code> option, so that was the only change I needed to do.</p>
<p>At this point, while browsing throught the documentation, I found a link to
<a href="https://github.com/ubports/porting-notes/wiki/Halium-9">this page</a> which
contains some notes on Halium 9 porting, which are substantially different from
the official porting guide in halium.org (which I was already told in Telegram
to be obsolete in several points).</p>
<p>So, following the instructions from this new link I ran</p>
<div class="code"><pre class="code literal-block">hybris-patches/apply-patches.sh --mb
</pre></div>
<p>which completed successfully.</p>
<p>Then, continuing following the points from this page, I edited <code>device/xiaomi/violet/lineage_violet.mk</code>, commented out the lines</p>
<div class="code"><pre class="code literal-block"><span class="k">$(</span><span class="nv">call</span> <span class="nv">inherit-product</span>, <span class="k">$(</span><span class="nv">SRC_TARGET_DIR</span><span class="k">)</span>/<span class="nv">product</span>/<span class="nv">full_base_telephony.mk</span><span class="k">)</span>
<span class="c"># ...</span>
<span class="k">$(</span><span class="nv">call</span> <span class="nv">inherit-product</span>, <span class="nv">vendor</span>/<span class="nv">lineage</span>/<span class="nv">config</span>/<span class="nv">common_full_phone.mk</span><span class="k">)</span>
</pre></div>
<p>and replaced the first one with a similar line pointing to <code>halium.mk</code>. For the
record, a <code>find</code> revealed that the <code>SRC_TARGET_DIR</code> variable in my case was
<code>build/make/target/</code>. It contained also the <code>halium.mk</code> file, which was created
by the hybris patches before. As for removing the Java dependencies, I cound't
find any modules similar to those listed <a href="https://github.com/ubports-on-fxtec-pro1/proprietary_vendor_idealte/commit/65cfe7583aaa886709b6142d8b79c276a3028ad9">in this
commit</a>
in any of the makefiles in my source tree, so I just started the build:</p>
<div class="code"><pre class="code literal-block"><span class="nb">source</span><span class="w"> </span>build/envsetup.sh<span class="w"> </span><span class="o">&&</span><span class="w"> </span>breakfast<span class="w"> </span>violet
make<span class="w"> </span>halium-boot
</pre></div>
<p>This failed the first time with the compiler being killed (out of memory, most
likely), but it succeeded on the second run. There was a suspicious warning,
though:</p>
<div class="code"><pre class="code literal-block">drivers/input/touchscreen/Kconfig:1290:warning: multi-line strings not supported
</pre></div>
<p>And indeed my <code>kernel/xiaomi/violet/drivers/input/touchscreen/Kconfig</code> had this line:</p>
<div class="code"><pre class="code literal-block">source "drivers/input/touchscreen/ft8719_touch_f7b/Kconfig
</pre></div>
<p>(notice the unterminated string quote). I don't know if this had any impact on
the build, but just to be on the safe side I added the missing quote and
rebuilt.</p>
<h4>Building the system image</h4>
<p>Running</p>
<div class="code"><pre class="code literal-block">make systemimage
</pre></div>
<p>failed pretty soon:</p>
<div class="code"><pre class="code literal-block">ninja: error: '/home/mardy/projects/port/halium/out/soong/host/linux-x86/framework/turbine.jar', needed by '/home/mardy/projects/port/halium/out/soong/.intermediates/libcore/core-oj/android_common/turbine/core-oj.jar', missing and no known rule to make it
</pre></div>
<p>The error is due to all the Java-related stuff that I should have disabled but
couldn't find. So, I tried to have a look at the <a href="https://github.com/ubuntu-touch-lavender/android_device_xiaomi_lavender/commit/ad640e121d0dc75017f31d860332327f8acd5d36">changes made on another
xiaomi
device</a>
(<code>lavender</code>, the Redmi note 7, which might not be that different, I thought)
and started editing <code>device/xiaomi/violet/device.mk</code> and removing a couple of
Android packages. Eventually the build proceeded, just to stop at a python
error:</p>
<div class="code"><pre class="code literal-block"> File "build/make/tools/check_radio_versions.py", line 56
print "*** Error opening \"%s.sha1\"; can't verify %s" % (fn, key)
^
SyntaxError: invalid syntax
</pre></div>
<p>Yes, it's the python3 vs python2 issue, since in my system <code>python</code> is python
version 3. In order to fix it, I created a virtual environment:</p>
<div class="code"><pre class="code literal-block">virtualenv --python 2.7 ../python27 # adjust the path to your prefs
source ../python27/bin/activate
</pre></div>
<p>Remember that the second line must be run every time you'll need to setup the
Halium build environment (that is, every time you run <code>breakfast</code>).</p>
<p>The build then proceeded for several minutes, until it failed due to some unresolved symbols:</p>
<div class="code"><pre class="code literal-block">prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/aarch64-linux-android/bin/ld.gold: error: /home/mardy/projects/port/halium/out/target/product/violet/obj/STATIC_LIBRARIES/lib_driver_cmd_qcwcn_intermediates/lib_driver_cmd_qcwcn.a: member at 7694 is not an ELF object
external/wpa_supplicant_8/hostapd/src/drivers/driver_nl80211.c:7936: error: undefined reference to 'wpa_driver_set_p2p_ps'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_set_ap_wps_p2p_ie'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_get_p2p_noa'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_set_p2p_noa'
/home/mardy/projects/port/halium/out/target/product/violet/obj/EXECUTABLES/hostapd_intermediates/src/drivers/driver_nl80211.o:driver_nl80211.c:wpa_driver_nl80211_ops: error: undefined reference to 'wpa_driver_nl80211_driver_cmd'
</pre></div>
<p>As people told me in the Telegram channel, this driver is not used since "our
<code>wpa_supplicant</code> talks to kernel directly". OK, so I simply disabled the
driver, copying again from the <a href="https://github.com/ubuntu-touch-lavender/android_device_xiaomi_lavender/commit/ad640e121d0dc75017f31d860332327f8acd5d36"><code>lavender</code> Halium
changes</a>:
commented out the <code>BOARD_WLAN_DEVICE := qcwcn</code> line (and all lines referring
this variable) from <code>device/xiaomi/violet/BoardConfig.mk</code> and ran <code>make
systemimage</code> again. This time, surprisingly, it all worked.</p>
<h4>Try it out</h4>
<p>I first tried to flash the kernel only. I rebooted into fastboot, and on my PC ran this:</p>
<div class="code"><pre class="code literal-block">cout
fastboot<span class="w"> </span>flash<span class="w"> </span>boot<span class="w"> </span>halium-boot.img
</pre></div>
<p>I then rebooted my phone, but after showing the boot logo for a few seconds it
would jump to the fastboot screen. Indeed, flashing the previous kernel would
restore the normal boot, so there had to be something wrong with my own kernel.</p>
<p>While looking at what the problem could be, I noticed that in the <code>lavender</code>
port the author did not modify the lineageos kernel config file in place, but
instead created a new one and changed the <code>BoardConfig.mk</code> file to point to his
new copy. Since it sounded like a good idea, I did the same and created
<code>kernel/xiaomi/violet/arch/arm64/configs/vendor/violet_halium_defconfig</code> for
the Halium changes. And then the line in the board config file became</p>
<div class="code"><pre class="code literal-block">TARGET_KERNEL_CONFIG := vendor/violet_halium_defconfig
</pre></div>
<p>I then continued investigating the boot issue, and I was told that it might
have been due to an Android option, <code>skip_initramfs</code>, which is set by the
bootloader and causes our Halium boot to fail. The fix is to just disable this
option in the kernel, by editing <code>init/initramfs.c</code> and change the
<code>skip_initramfs_param</code> function to always set the <code>do_skip_initramfs</code> variable
to <code>0</code>, rather than to <code>1</code>. After doing this, the boot proceeded to show the
Ubuntu splash screen with the five dots being lit, but it didn't proceed from
there.</p>
<h5>Setting up the udev rules</h5>
<p>Even in this state, the device was detected by my host PC and these lines
appeared in the system log:</p>
<div class="code"><pre class="code literal-block">kernel: usb 1-3: new high-speed USB device number 26 using xhci_hcd
kernel: usb 1-3: New USB device found, idVendor=0fce, idProduct=7169, bcdDevice= 4.14
kernel: usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
kernel: usb 1-3: Product: Unknown
kernel: usb 1-3: Manufacturer: GNU/Linux Device
kernel: usb 1-3: SerialNumber: GNU/Linux Device on usb0 10.15.19.82
</pre></div>
<p>Indeed, I guess the reason I could do this without even flashing my systemimage
is because I had first flashed another UT system image from another porter. I'm
not sure if I'd had the same results with my own image. Anyway, the USB
networking was there, so I connected and ran the following commands to generate
the udev rules:</p>
<div class="code"><pre class="code literal-block">ssh<span class="w"> </span>phablet@10.15.19.82
<span class="c1"># used 0000 as password</span>
sudo<span class="w"> </span>-i
<span class="c1"># same password again</span>
<span class="nb">cd</span><span class="w"> </span>/home/phablet
<span class="nv">DEVICE</span><span class="o">=</span>violet
cat<span class="w"> </span>/var/lib/lxc/android/rootfs/ueventd*.rc<span class="w"> </span>/vendor/ueventd*.rc<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^/dev<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/^\/dev\///'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{printf "ACTION==\"add\", KERNEL==\"%s\", OWNER=\"%s\", GROUP=\"%s\", MODE=\"%s\"\n",$1,$3,$4,$2}'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/\r//'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>>70-<span class="nv">$DEVICE</span>.rules
</pre></div>
<p>I then copied (with <code>scp</code>) this file to my host PC, and I moved it to
<code>device/xiaomi/violet/ubuntu/70-violet.rules</code> (I had to create the <code>ubuntu</code>
directory first). Then I edited the <code>device/xiaomi/violet/device.mk</code> file and
added these lines at the end:</p>
<div class="code"><pre class="code literal-block"><span class="c">### Ubuntu Touch ###</span>
<span class="nv">PRODUCT_COPY_FILES</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="k">$(</span>LOCAL_PATH<span class="k">)</span>/ubuntu/70-violet.rules:system/halium/lib/udev/rules.d/70-android.rules
<span class="c">### End Ubuntu Touch ###</span>
</pre></div>
<p>It was now time to try my own system image. I rebooted my device into TWRP, I
cloned the <a href="https://gitlab.com/JBBgameich/halium-install">halium-install
repository</a> into my halium build
dir, downloaded <a href="https://ci.ubports.com/job/xenial-hybris-android9-rootfs-arm64/107/">a rootfs for
Halium9</a>,
and ran</p>
<div class="code"><pre class="code literal-block">./halium-install/halium-install<span class="w"> </span>-p<span class="w"> </span>ut<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>~/Downloads/ubuntu-touch-android9-arm64.tar.gz<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>out/target/product/violet/system.img
</pre></div>
<p>The first time this failed because the <code>simg2img</code> tool was not installed. The
second time it proceeded to create the image, asked me for a password for the
phablet user (gave "0000") and pushed the image onto the device, into the
<code>/data/</code> partition. I then rebooted.</p>
<h5>My first Ubuntu Touch image</h5>
<p>Upon reboot, the usual splash screen appeared, followed by several seconds of
black screen. It definitely didn't look right, but at least it proved that
<em>something</em> had been flashed. After some more seconds, to my big surprise, the
Ubuntu boot screen appeared, just with a smaller logo than how it used to be
before, which also confimed that my system image was being used — in fact, I
did not adjust the <code>GRID_UNIT_PX</code> variable before. Since this was an easy fix,
I chose to focus on that, rather than fix the boot issues (indeed, my device
did not move on from the Ubuntu boot screen). SSH was working.</p>
<p>I took the scaling.conf file from the <a href="https://github.com/ubuntu-touch-lavender/android_device_xiaomi_lavender/commit/e02875e15a09e96635d32aaf9427fcd057d7a9ae">lavender
changes</a>,
put it in <code>device/xiaomi/violet/ubuntu/</code> and added this line in the
<code>PRODUCT_COPY_FILES</code> in <code>device.mk</code>:</p>
<div class="code"><pre class="code literal-block">$(LOCAL_PATH)/ubuntu/scaling.conf:system/halium/etc/ubuntu-touch-session.d/android.conf
</pre></div>
<p>I initially used <code>system/ubuntu/etc/...</code> as the target destination for the
config file, like in the lavender commit, but this didn't work out for me. Then
I changed the path to start with <code>system/halium/</code>, like it's mentioned in the
ubports documentation, but it apparently had no effect either.</p>
<p>After a couple of days spent trying to understand why my files were not
appearing under <code>/etc</code>, it turned out that the device was not using my system
image at all: with <code>halium-install</code> I had my image installed in
<code>/userdata/system.img</code>, while the correct path for Halium 9 devices is
<code>/userdata/android-rootfs.img</code>. I was told that the option <code>-s</code> of
<code>halium-install</code> would do the trick. Instead of re-running the script, I took
the shortcut of renaming <code>/userdata/system.img</code> to
<code>/userdata/android-rootfs.img</code> and after rebooting I could see that the Ubuntu
logo was displayed at the correct size. And indeed my system image was being
used.</p>
<p>So I started to debug why unity8 didn't start. The logs terminated with this line:</p>
<div class="code"><pre class="code literal-block">terminate called after throwing an instance of 'std::runtime_error'
what(): org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying
initctl: Event failed
</pre></div>
<p>It means, that unity8 did not handle correctly the situation where another
D-Bus service crashed while handing a call from unity8. The problem now was how
to figure out which service it was. I ran <code>dbus-monitor</code> and restarted unity8
(<code>initctl start unity8</code>), then examined the dbus logs; I saw the point where
unity8 got disconnected from the bus, but before that point I didn't find any
failed D-Bus calls. So it had to be the system bus. I did exactly the same
steps, just this time after running <code>dbus-monitor --system</code> as root, I found
the place where unity8 got disconnected, and found this D-bus error shortly
before that:</p>
<div class="code"><pre class="code literal-block">method call time=1605676421.968667 sender=:1.306 -> destination=com.ubuntu.biometryd.Service serial=3 path=/default_device; interface=com.ubuntu.biometryd.Device; member=Identifier
method return time=1605676421.970222 sender=:1.283 -> destination=:1.306 serial=4 reply_serial=3
object path "/default_device/identifier"
...
method call time=1605676421.974218 sender=:1.283 -> destination=org.freedesktop.DBus serial=6 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=GetConnectionAppArmorSecurityContext
string ":1.306"
error time=1605676421.974278 sender=org.freedesktop.DBus -> destination=:1.283 error_name=org.freedesktop.DBus.Error.AppArmorSecurityContextUnknown reply_serial=6
string "Could not determine security context for ':1.306'"
...
signal time=1605676421.989074 sender=org.freedesktop.DBus -> destination=:1.283 serial=6 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameLost
string "com.ubuntu.biometryd.Service"
...
error time=1605676421.989302 sender=org.freedesktop.DBus -> destination=:1.306 error_name=org.freedesktop.DBus.Error.NoReply reply_serial=4
string "Message recipient disconnected from message bus without replying"
</pre></div>
<p>So, it looks like unity8 (<code>:1.306</code>) made a request to biometryd, who asked the
D-Bus daemon what was the AppArmor label of the caller, but since I didn't
backport the D-Bus mediation patches for AppArmor, the label could not be
resolved. <code>biometryd</code> decided to crash instead of properly handling the error,
and so did unity8.</p>
<p>So I backported the AppArmor patches. I took them from the <a href="https://kernel.ubuntu.com/git/jj/linux-apparmor-backports/log/?h=v4.15%2b-apparmor-backport-to-v4.14-presquash">Canonical kernel
repo</a>,
but they did not apply cleanly because they are meant to be applied on top of a
pristine 4.14 branch, whereas the Android kernel had already some AppArmor
security fixes backported from later releases. So I set and quickly inspected
the contents of each patch, and found out that a couple of them had already
been applied, and the last patch had to be applied as the first (yeah, it's
hard to explain, but it all depends on other patches that Android has
backported from newer kernels). Anyway, after rebuilding the kernel and
reflashing it, my phone could finally boot to unity8!</p>
<h3>Conclusion (of the first episode)</h3>
<p>The actual porting, as I've been told, starts here. What has been documented
here are only the very first steps of the bring-up; what awaits me now is to
make all the hardware subsystems work properly, and this, according to people
more experienced in porting, is the harder part.</p>
<p>So far, very few things work, to the point that it's faster to me to list the
things that <em>do work</em>; it's safe to assume that all what is not listed here is
not working:</p>
<ul>
<li>Mir, with graphics and input: Unity8 starts and is usable</li>
<li>Camera: can take photos; video recording start but an error appears when the
stop button is pressed</li>
<li>Flashlight</li>
<li>Fingerprint reader: surprisingly, this worked out of the box</li>
<li>Screen brightness (though it can be changed only manually)</li>
</ul>
<p>For all the rest, please keep an eye on this blog: I'll write, when I make some
progress!</p>Announcing CuteVNChttp://mardy.it/ia/blog/2020/12/announcing-cutevnc.html2020-12-13T16:38:48+03:002020-12-13T16:38:48+03:00Alberto Mardegan<p>Having a single screen connected to my PC, I was wondering if I could use my
nice <a href="https://en.wikipedia.org/wiki/BQ_Aquaris_M10">BQ M10 tablet</a> running
UBports as an additional screen. The idea was that if I could convince the X
server to create a new virtual display, and then run the x11vnc server on it, I
could then connect the tablet to it and enjoy my desktop on two screens. Yes,
the M10 has not such a big display, but it couls still be useful to host a
window with a mail or a chat client, so that I would not need to switch between
applications just to check if new emails or chat messages have arrived (and I
hate instant notifications, so I tend to disable them in every program that
allows me to).</p>
<p>Anyway, I digress. I found some articles, and specifically <a href="https://unix.stackexchange.com/questions/156412/vnc-server-as-a-virtual-x11-monitor-to-expand-screen">this
question</a>,
that show how one can add a new virtual screen and connect to it via VNC. I
tried it and it kind of worked, but there were some visual artifacts that
annoyed me (most likely due to the AMDGPU drivers), so I'll defer the creation
of a demo video till when I'll have figured out a better setup. But the main
point is that the idea, in principle, can work.</p>
<h2>CuteVNC</h2>
<p>So I sat down and started writing a VNC client for UBports, since it appears
that there wasn't one yet in the store. I use the past tense, because after a
week or so of work, <a href="https://open-store.io/app/it.mardy.lomiri-vnc">CuteVNC is
there</a>. It's still a very alpha
version, and does seem to get confused if the VNC server is using caching, but
overall it's working. Its impressive feature list includes:</p>
<ul>
<li>Can connect to a VNC server, and show the remote display</li>
<li>Can send mouse events</li>
<li>Windowed and fullscreen mode</li>
<li>Pinch to zoom</li>
</ul>
<p>There's a lot more that could be done, but for a first release this can be OK.</p>
<h2>Development of a UBports Qt/QML app with QBS</h2>
<p>If, like me, you are a fan of QBS, you might want to have a look at the <a href="https://gitlab.com/mardy/lomiri-vnc">source
code of CuteVNC</a> (yes, the project was
born as "Lomiri VNC", but since it's a generic QtQuick.Controls 2 application,
there was no need to tie it to the Lomiri environment); there you can find a
<a href="https://gitlab.com/mardy/lomiri-vnc/-/tree/master/qbs/modules/ubuntutouch">QBS module for UBports
applications</a>,
which can help in filling in the manifest file and deploying other Ubuntu Touch
specific files into their proper install location.</p>
<p>You can also see the <a href="https://gitlab.com/mardy/lomiri-vnc/-/blob/master/clickable.json">configuration
file</a> to make
<a href="https://clickable-ut.dev/en/stable/">clickable</a> build a QBS-based project. In
my mental task backlog I still have the idea of making clickable support QBS
out of the box, but the integration of the two technologies is so easy even
now, that I gave this task a quite low priority.</p>
<h2>A class for computing viewport transformations</h2>
<p>Last but not least, a developer might find useful the <a href="https://gitlab.com/mardy/lomiri-vnc/-/blob/master/src/scaler.h">generic Scaler
class</a> I wrote
to compute the viewport transformations: it's a simple, QtGui-only C++ class
which takes an input structure with the viewport information, and returns a
similar structure containing a <a href="https://doc.qt.io/qt-5/qtransform.html">QTransform
matrix</a> to map item coordinates to
coordinates in the source object coordinates (in my case, the source object is
the remote screen served by VNC) and some other useful parameters, such as the
scale, center offset, and the painted area rectangle.</p>
<p>The class has a 100% line and branch coverage in the <a href="https://gitlab.com/mardy/lomiri-vnc/-/blob/master/tests/tst_scaler.cpp">unit
tests</a>,
so I hope I can rely on it.</p><p>Having a single screen connected to my PC, I was wondering if I could use my
nice <a href="https://en.wikipedia.org/wiki/BQ_Aquaris_M10">BQ M10 tablet</a> running
UBports as an additional screen. The idea was that if I could convince the X
server to create a new virtual display, and then run the x11vnc server on it, I
could then connect the tablet to it and enjoy my desktop on two screens. Yes,
the M10 has not such a big display, but it couls still be useful to host a
window with a mail or a chat client, so that I would not need to switch between
applications just to check if new emails or chat messages have arrived (and I
hate instant notifications, so I tend to disable them in every program that
allows me to).</p>
<p>Anyway, I digress. I found some articles, and specifically <a href="https://unix.stackexchange.com/questions/156412/vnc-server-as-a-virtual-x11-monitor-to-expand-screen">this
question</a>,
that show how one can add a new virtual screen and connect to it via VNC. I
tried it and it kind of worked, but there were some visual artifacts that
annoyed me (most likely due to the AMDGPU drivers), so I'll defer the creation
of a demo video till when I'll have figured out a better setup. But the main
point is that the idea, in principle, can work.</p>
<h2>CuteVNC</h2>
<p>So I sat down and started writing a VNC client for UBports, since it appears
that there wasn't one yet in the store. I use the past tense, because after a
week or so of work, <a href="https://open-store.io/app/it.mardy.lomiri-vnc">CuteVNC is
there</a>. It's still a very alpha
version, and does seem to get confused if the VNC server is using caching, but
overall it's working. Its impressive feature list includes:</p>
<ul>
<li>Can connect to a VNC server, and show the remote display</li>
<li>Can send mouse events</li>
<li>Windowed and fullscreen mode</li>
<li>Pinch to zoom</li>
</ul>
<p>There's a lot more that could be done, but for a first release this can be OK.</p>
<h2>Development of a UBports Qt/QML app with QBS</h2>
<p>If, like me, you are a fan of QBS, you might want to have a look at the <a href="https://gitlab.com/mardy/lomiri-vnc">source
code of CuteVNC</a> (yes, the project was
born as "Lomiri VNC", but since it's a generic QtQuick.Controls 2 application,
there was no need to tie it to the Lomiri environment); there you can find a
<a href="https://gitlab.com/mardy/lomiri-vnc/-/tree/master/qbs/modules/ubuntutouch">QBS module for UBports
applications</a>,
which can help in filling in the manifest file and deploying other Ubuntu Touch
specific files into their proper install location.</p>
<p>You can also see the <a href="https://gitlab.com/mardy/lomiri-vnc/-/blob/master/clickable.json">configuration
file</a> to make
<a href="https://clickable-ut.dev/en/stable/">clickable</a> build a QBS-based project. In
my mental task backlog I still have the idea of making clickable support QBS
out of the box, but the integration of the two technologies is so easy even
now, that I gave this task a quite low priority.</p>
<h2>A class for computing viewport transformations</h2>
<p>Last but not least, a developer might find useful the <a href="https://gitlab.com/mardy/lomiri-vnc/-/blob/master/src/scaler.h">generic Scaler
class</a> I wrote
to compute the viewport transformations: it's a simple, QtGui-only C++ class
which takes an input structure with the viewport information, and returns a
similar structure containing a <a href="https://doc.qt.io/qt-5/qtransform.html">QTransform
matrix</a> to map item coordinates to
coordinates in the source object coordinates (in my case, the source object is
the remote screen served by VNC) and some other useful parameters, such as the
scale, center offset, and the painted area rectangle.</p>
<p>The class has a 100% line and branch coverage in the <a href="https://gitlab.com/mardy/lomiri-vnc/-/blob/master/tests/tst_scaler.cpp">unit
tests</a>,
so I hope I can rely on it.</p>A simple cross-compiler for the Raspberry Pihttp://mardy.it/ia/blog/2020/02/a-simple-cross-compiler-for-the-raspberry-pi.html2020-02-19T23:44:42+03:002020-02-19T23:44:42+03:00Alberto Mardegan<p>I've recently found some use for a first-generation Raspberry Pi (Pi 1 model
B) which had been lying in a drawer since many years. A few days ago I've
installed the Raspbian distribution in it, and was about to install
<a href="https://motion-project.github.io/">motion</a> on it, but I stopped as soon as I
noticed that <code>apt</code> was suggesting to bring in 1 GB worth of dependencies. Adding
<code>--no-install-recommends</code> reduced the proposal a bit, but it was still around
700 MB -- a bit too much for my taste. I figured out that the <code>motion</code> package
for Debian (and Raspbian) depends on MySQL, PostgreSQL, FFmpeg and what not;
so, I decided that I could probably just recompile it and disable all the stuff
I didn't need at <code>configure</code> time.</p>
<p>But I didn't want to install all the build dependencies and the cross-compiler in
my machine; containers exist for a reason, after all. So I had a look at the
<a href="https://github.com/ubports/crossbuilder/">crossbuilder</a> tool that we use in
UBports: this is a nice little shell program that uses an Ubuntu-based LXD
image to cross-compile a package after automatically fetching all its
dependencies, and installs it into an UBports device. It does some magic with
<code>adb</code> and stuff, but I thought that the basic functionality should work with
minor modifications on any Debian-based distribution.</p>
<p>And indeed, some hours later, I got <a href="https://github.com/mardy/crossbuilder/tree/raspbian">a
branch</a> where I can use
<code>crossbuilder</code> to build packages for the Raspberry Pi. Assuming that you have
LXD properly setup, the command</p>
<div class="code"><pre class="code literal-block">crossbuilder --raspbian source motion
</pre></div>
<p>will cause crossbuilder to create a container and download the Debian source
package for <code>motion</code>; at this point you can modify the source code as you see
fit, and rebuild it. I only changed the <code>debian/rules</code> file to add a few flags
such as <code>--without-mysql</code>, <code>--without-ffmpeg</code>, etc. And</p>
<div class="code"><pre class="code literal-block">crossbuilder --raspbian
</pre></div>
<p>is the command to run in order to perform the build. This will first download
all the dependencies (according to the <code>debian/control</code> file), build the
package, and create a <code>tar</code> archive containing all the generated <code>.deb</code> files.
This archive can then be copied into the target device and unpacked there.</p>
<p>Now, there's a small problem in that Raspbian claims to be <code>armhf</code>, while in
fact its floating-point processor is somehow not compliant with the armhf
architecture. So, you generally cannot use an <code>armhf</code> package from Debian or
Ubuntu on the Raspberry Pi. Given that I didn't have the time to prepare a
proper Raspbian image for LXD, I used the Debian distribution as a base
instead, and I chose to target the <code>armel</code> architecture: this might impose some
penalties on the performance (at least for floating-point code), but it seems
to work fine on the Raspberry Pi. Unfortunately, this means that you cannot
just install the generated packages o the Pi, as <code>dpkg</code> will complain about the
architecture mismatch (your package is <code>armel</code>, while the distro claims to be
<code>armhf</code>). But unpacking the debian package with</p>
<div class="code"><pre class="code literal-block">dpkg -x motion_<...>.deb tmp
</pre></div>
<p>will indeed give you package that you can use.</p>
<p>I will eventually get back to this and make the script work better with
Raspbian, but this is a quick start.</p><p>I've recently found some use for a first-generation Raspberry Pi (Pi 1 model
B) which had been lying in a drawer since many years. A few days ago I've
installed the Raspbian distribution in it, and was about to install
<a href="https://motion-project.github.io/">motion</a> on it, but I stopped as soon as I
noticed that <code>apt</code> was suggesting to bring in 1 GB worth of dependencies. Adding
<code>--no-install-recommends</code> reduced the proposal a bit, but it was still around
700 MB -- a bit too much for my taste. I figured out that the <code>motion</code> package
for Debian (and Raspbian) depends on MySQL, PostgreSQL, FFmpeg and what not;
so, I decided that I could probably just recompile it and disable all the stuff
I didn't need at <code>configure</code> time.</p>
<p>But I didn't want to install all the build dependencies and the cross-compiler in
my machine; containers exist for a reason, after all. So I had a look at the
<a href="https://github.com/ubports/crossbuilder/">crossbuilder</a> tool that we use in
UBports: this is a nice little shell program that uses an Ubuntu-based LXD
image to cross-compile a package after automatically fetching all its
dependencies, and installs it into an UBports device. It does some magic with
<code>adb</code> and stuff, but I thought that the basic functionality should work with
minor modifications on any Debian-based distribution.</p>
<p>And indeed, some hours later, I got <a href="https://github.com/mardy/crossbuilder/tree/raspbian">a
branch</a> where I can use
<code>crossbuilder</code> to build packages for the Raspberry Pi. Assuming that you have
LXD properly setup, the command</p>
<div class="code"><pre class="code literal-block">crossbuilder --raspbian source motion
</pre></div>
<p>will cause crossbuilder to create a container and download the Debian source
package for <code>motion</code>; at this point you can modify the source code as you see
fit, and rebuild it. I only changed the <code>debian/rules</code> file to add a few flags
such as <code>--without-mysql</code>, <code>--without-ffmpeg</code>, etc. And</p>
<div class="code"><pre class="code literal-block">crossbuilder --raspbian
</pre></div>
<p>is the command to run in order to perform the build. This will first download
all the dependencies (according to the <code>debian/control</code> file), build the
package, and create a <code>tar</code> archive containing all the generated <code>.deb</code> files.
This archive can then be copied into the target device and unpacked there.</p>
<p>Now, there's a small problem in that Raspbian claims to be <code>armhf</code>, while in
fact its floating-point processor is somehow not compliant with the armhf
architecture. So, you generally cannot use an <code>armhf</code> package from Debian or
Ubuntu on the Raspberry Pi. Given that I didn't have the time to prepare a
proper Raspbian image for LXD, I used the Debian distribution as a base
instead, and I chose to target the <code>armel</code> architecture: this might impose some
penalties on the performance (at least for floating-point code), but it seems
to work fine on the Raspberry Pi. Unfortunately, this means that you cannot
just install the generated packages o the Pi, as <code>dpkg</code> will complain about the
architecture mismatch (your package is <code>armel</code>, while the distro claims to be
<code>armhf</code>). But unpacking the debian package with</p>
<div class="code"><pre class="code literal-block">dpkg -x motion_<...>.deb tmp
</pre></div>
<p>will indeed give you package that you can use.</p>
<p>I will eventually get back to this and make the script work better with
Raspbian, but this is a quick start.</p>Using the latest QBS on older distributionshttp://mardy.it/ia/blog/2019/11/using-the-latest-qbs-on-older-distributions.html2019-11-17T21:48:08+03:002019-11-17T21:48:08+03:00Alberto Mardegan<p>A short announcement, probably meaningless to most people, but who knows: I've
created an Ubuntu <a href="https://launchpad.net/~mardy/+archive/ubuntu/qbs-on-lts">PPA with the latest
QBS</a>. The reason why
this might make some sense is that this PPA targets the older Ubuntu
distributions. It's currently built for 14.04 (Trusty, which is no longer
supported by Canonical), and I'll eventually upload the QBS package for 16.04,
too.</p>
<p>This package can be useful to people distributing applications in the <a href="https://appimage.org/">AppImage
format</a>, where one usually builds the application in one
older distribution in order to increase the chances of it being runnable in as
many Linux distributions as possible. A simpler way to obtain QBS on Trusty is
to install QtCreator, but that's not trivial to install in an automated way and
might not come with the latest QBS. Especially when building on Ubuntu, a
package from a PPA is much easier to install.</p>
<p>This QBS is built statically, and won't install any Qt libraries on your
system; this is good, because it allows you to use whatever Qt version you like
without any risk of conflicts.</p><p>A short announcement, probably meaningless to most people, but who knows: I've
created an Ubuntu <a href="https://launchpad.net/~mardy/+archive/ubuntu/qbs-on-lts">PPA with the latest
QBS</a>. The reason why
this might make some sense is that this PPA targets the older Ubuntu
distributions. It's currently built for 14.04 (Trusty, which is no longer
supported by Canonical), and I'll eventually upload the QBS package for 16.04,
too.</p>
<p>This package can be useful to people distributing applications in the <a href="https://appimage.org/">AppImage
format</a>, where one usually builds the application in one
older distribution in order to increase the chances of it being runnable in as
many Linux distributions as possible. A simpler way to obtain QBS on Trusty is
to install QtCreator, but that's not trivial to install in an automated way and
might not come with the latest QBS. Especially when building on Ubuntu, a
package from a PPA is much easier to install.</p>
<p>This QBS is built statically, and won't install any Qt libraries on your
system; this is good, because it allows you to use whatever Qt version you like
without any risk of conflicts.</p>Notes on porting the Samsung J3 to Halium + Ubportshttp://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html2019-07-04T18:47:27+03:002019-07-04T18:47:27+03:00Alberto Mardegan<p>A due premise: <strong>this is not a guide!</strong> While at the end I managed to build an
image for my phone, it doesn't even boot. So, this post should be taken just as
a collection of notes for myself, written to be able to reproduce the build and
keep track of those steps I might want to revisit later.</p>
<p>The reason why I'm publishing these notes in my blog is that I believe they can
still be useful for other people who should stumble into the same errors, and
also to give you an idea of how much fun (or lack thereof) porting is.</p>
<p>It's a very long post, but unless you are starting to port Halium or Ubports to
a new device, you can safely stop reading now.</p>
<p>I hope to write some better porting news in some future post. :-)</p>
<h2>Before we start: getting the info</h2>
<p>The first step is finding out a few informations about our device, and whether
a LineageOS port for our device exists (hadn't there been a LineageOS port, I
would have been out of luck: I certainly wouldn't have the time to do one
myself). From the Settings application we can find the device name, the model
number, the Android version and the kernel version:</p>
<ul>
<li>Device name: "Samsung Galaxy J3 (2016)"</li>
<li>Model: SM-J320F</li>
<li>Android version: 5.1.1</li>
<li>Kernel version: 3.10.65-[…]</li>
</ul>
<p>With this information we can look for a LineageOS port, either via a search
engine or directly in <a href="https://forum.xda-developers.com">the XDA forums</a>; in my
case the thread is <a href="https://forum.xda-developers.com/galaxy-j3-2016/development/rom-lineageos-14-1-samsung-galaxy-j3-t3667015">this
one</a>.
The post announcing the port should
contain links to the source code, besides some other information. We do
especially care about:</p>
<ul>
<li>the device codename: <code>j3xlte</code></li>
<li>the CM/LineageOS version: 14.1</li>
<li>links to <a href="https://github.com/djeman/android_device_samsung_j3xnlte/tree/cm-14.1">Device
Tree</a>,
<a href="https://github.com/djeman/android_kernel_samsung_sharkls/tree/cm-14.1">Kernel</a>,
<a name="local_manifest"></a><a href="https://github.com/djeman/android_device_samsung_j3xnlte/wiki/Local-manifest-CM-14.1">Local
Manifest</a>
(keep the local manifest handy, as you are going to need it soon)</li>
</ul>
<p>To be on the safe side, get a copy of everything. In case these are GitHub
repositories, forking them is enough.</p>
<h2>Getting root access on the device</h2>
<p>Another precondition for a successful port is the ability to be able to flash a
custom kernel and rootfs image. The existance of a LineageOS port is not a
guarantee that we can flash it: some devices are sold with a locked bootloader
in certain markets or when sold by certain mobile operators. So, before
spending time on the port, we should make sure that we can hack on our device
freely.</p>
<p>In order to get <code>adb</code> working, I had to enable "Developer mode", which is done
by hitting the "Build numer" item under the "Software info" page for 10 times
in a row. Then from the newly appeared "Developer options" menu I enabled "USB
debugging".</p>
<p>Being a total novice to Android phones, I had to go through several attempts
and internet searches, before being able to get root access on the device
(which I needed in order to read the <code>/etc/fstab</code> file, for example): by
default, <code>adb</code> only let me as normal user, and <code>su</code> didn't work.</p>
<p>In order to flash a rootkit on this phone, we need to use
<a href="https://gitlab.com/BenjaminDobell/Heimdall">Heimdall</a>, because on Samsung
devices this is what is used instead of <code>fastboot</code>. I just followed the
instructions in the <code>Linux/README</code> file and could easily build and install it.</p>
<p>We are going to flash <a href="https://twrp.me/">TWRP</a> and
<a href="http://supersu.com/">SuperSU</a>; a link to the former can be found in the XDA
<a href="https://forum.xda-developers.com/galaxy-j3-2016/development/rom-lineageos-14-1-samsung-galaxy-j3-t3667015">post</a>
about the LineageOS port for our device (under the "Download" button) and the
latter from its own homepage (in my case I had to download the "Recovery
flashable" <code>zip</code> file, and for some weird reason I got the 2.76 version, but I
bet that any newer version works as well). Once I got these files, I followed
these steps:</p>
<ul>
<li>Extract TWRP in my host machine: <code>cd /tmp && tar xvf
~/Downloads/sm-j320fgm_twrp_3.1.1_f2fs.tar.md5</code> (that leaves a <code>recovery.img</code>
file in my <code>/tmp</code>)</li>
<li>Upload SuperSU to the phone:</li>
</ul>
<div class="code"><pre class="code literal-block"><span class="w"> </span>adb<span class="w"> </span>push<span class="w"> </span>~/Downloads/UPDATE-SuperSU-v2.76-20160630161323.zip<span class="w"> </span>/sdcard/supersu.zip
</pre></div>
<ul>
<li>Power off the phone</li>
<li>Power it on by keeping the VOLUME DOWN and HOME keys pressed while
pressing the power button. The phone will be brought up into Odin mode.</li>
<li>Run</li>
</ul>
<div class="code"><pre class="code literal-block"><span class="w"> </span>sudo<span class="w"> </span>heimdall<span class="w"> </span>flash<span class="w"> </span>--RECOVERY<span class="w"> </span>/tmp/recovery.img
</pre></div>
<ul>
<li>
<p>Before the phone reboots, start keeping the VOLUME UP (<em>note: it's not the
same as before!</em>) and HOME keys, to make it reboot into recovery and not into
the ordinary image. If the phone boots into the ordinary image it will
overwrite the TWRP we just flashed, so, if that happens, you need to start
from scratch.</p>
</li>
<li>
<p>Find supersu (the zip file) and install it.</p>
</li>
</ul>
<p>If you reboot the phone and use <code>adb shell</code>, you'll be able to run <code>su</code> and get
root rights. At this point, I charged myself with hopes and got to the real
work.</p>
<h2>Let the Odyssey start</h2>
<p>I just followed the <a href="http://docs.halium.org/en/latest/porting/first-steps.html">"First steps"
instructions</a> step
by step, with no major surprises.</p>
<p>Once done with that, I moved to the <a href="http://docs.halium.org/en/latest/porting/get-sources.html">"Get sources"
page</a>. Here it was
not just a matter of following the instructions: assembling the manifest file
took me several iterations, so I had to come back to this step a few times.
Anyway, I followed the instructions for the <code>halium-7.1</code> version, given that my
device has a LineageOS 14.1 port. Then:</p>
<ul>
<li>I created a git branch in the <code>halium/devices/</code> tree, to keep track of my changes</li>
<li>I added the manifest <code>halium/devices/manifests/samsung_j3xlte.xml</code> (you can
see the final version in <a href="https://github.com/mardy/halium-devices/blob/samsung_j3xlte/manifests/samsung_j3xlte.xml">my
repository</a>,
but initially I was missing some important lines)</li>
<li>Run <code>JOBS=2 ./halium/devices/setup j3xlte</code></li>
<li>The makefile needed to be changed: the
<code>device/samsung/j3xlte/setup-makefiles.sh</code> script (from <a href="https://github.com/djeman/android_device_samsung_j3xnlte/tree/cm-14.1">the device
tree</a>)
was missing a <code>mkdir -p ../../../$OUTDIR</code> near the beginning.</li>
</ul>
<p>At this point I continued to the <a href="http://docs.halium.org/en/latest/porting/build-sources.html">"Build sources"
page</a>. Here starts
a long <em>trial and error</em> phase, and the longer part of this post. For the sake
of documentation, I will show you all the commands I ran, including failed
attempts (there are many of them), just to give you an idea of what problems
you might run into with your own port, and hopefully help in finding a
solution.</p>
<p>I first ran the <code>breakfast</code> command for my device:</p>
<div class="code"><pre class="code literal-block">$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?
</pre></div>
<p>It looks like the device/samsung/sharkls-common was not checked out properly. After re-doing with force-sync, it failed with other errors:</p>
<div class="code"><pre class="code literal-block">$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?
</pre></div>
<p>Indeed, I had nothing under <code>device/common</code>. So, I looked under
<code>halium/devices/manifests</code> to see if any other manifest was populating this
directory, and found that the <code>bq_krillin.xml</code> file had such a line. So, I
copied that line (the one adding the <code>lineageos/android_device_common</code> project)
into my manifest. After doing that, the <code>breakfast</code> command succeeded:</p>
<div class="code"><pre class="code literal-block">$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
Trying dependencies-only mode on a non-existing device tree?
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190619-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================
</pre></div>
<h3>Building the kernel</h3>
<p>According to the <a href="https://www.youtube.com/watch?v=nShXVDXM50A">livestream</a>, the
proper script to check for the kernel config is not the Mer one, but
<code>halium/halium-boot/check-kernel-config</code> (this is also documented in the
<a href="https://docs.ubports.com/en/latest/porting/building-halium-boot.html#edit-kernel-config">Ubports - Building Halium boot documentation
page</a>).</p>
<p>At this point I realized that I didn't have a kernel: an entry starting with
<code>kernel/</code> should always be in the manifest file. So I added the kernel line
taken from the LineageOS <a href="http://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html#local_manifest">local manifest</a> file.</p>
<p>I then looked for the config file, which I eventually found in
<code>arch/arm/configs/j3xlte_defconfig</code>. So I ran</p>
<div class="code"><pre class="code literal-block">./halium/halium-boot/check-kernel-config kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig -w
</pre></div>
<p>which reported no errors. However, in the command output I noticed this:</p>
<div class="code"><pre class="code literal-block"> CONFIG_TMPFS_POSIX_ACL is already set
CONFIG_DEFAULT_SECURITY is set, but to y "selinux" y not "apparmor".
Setting CONFIG_DEFAULT_SECURITY="apparmor" correctly
sed: -e expression #1, char 60: unterminated `s' command
Setting CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
</pre></div>
<p>When I checked the config file, I noticed that AppArmor was not really enabled.
So I changed it manually.</p>
<p>Next, the fstab. I had the Ubports and Halium guides both opened in my browser,
and I went for the <a href="https://docs.ubports.com/en/latest/porting/building-halium-boot.html#fix-mounts">"Fix
mounts"</a>
step from the Ubports guide. I used <code>find</code> to locate the fstab, and found two:</p>
<ul>
<li><a href="https://github.com/djeman/android_device_samsung_sharkls-common/blob/cm-14.1/recovery/root/fstab.sc8830"><code>device/samsung/sharkls-common/recovery/root/fstab.sc8830</code></a></li>
<li><a href="https://github.com/djeman/android_device_samsung_sharkls-common/blob/cm-14.1/rootdir/etc/fstab.sc8830"><code>device/samsung/sharkls-common/rootdir/etc/fstab.sc8830</code></a></li>
</ul>
<p>The recovery file seemed already fine to me, while the rootdir one had most
entry duplicated; ones with f2fs and xattrs, the other ones with ext4. I
commented out the f2fs ones and ran <code>mka halium-boot</code>:</p>
<div class="code"><pre class="code literal-block">[...]
/mnt/Lavoro/halium/out/build-aosp_arm.ninja is missing, regenerating...
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=
TARGET_PRODUCT=aosp_arm
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
============================================
Checking build tools versions...
find: ‘device/*/generic’: No such file or directory
find: ‘device/unknown’: No such file or directory
find: ‘device/android’: No such file or directory
find: ‘device/*/generic’: No such file or directory
find: ‘device/unknown’: No such file or directory
find: ‘device/android’: No such file or directory
halium/hybris-boot/Android.mk:69: ********************* /boot appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:70: ********************* /data appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:73: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-aosp_arm.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-aosp_arm.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
#### make failed to build some targets (11 seconds) ####
</pre></div>
<p>this seems to be documented in <a href="http://docs.halium.org/en/latest/porting/common-system-build-errors.html#hybris-boot-part-and-hybris-data-part">the "Common system build errors"
page</a>:
I had to rerun <code>source build/envsetup.sh</code> and <code>breakfast j3xlte</code>. After that, the <code>mka halium-boot</code> command failed with</p>
<div class="code"><pre class="code literal-block">device/samsung/sharkls-common/ims/sec_samsung/Android.mk:25: build/core/java_library.mk: No such file or directory
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
</pre></div>
<p>which looks like a progress. The issue about the missing <code>java_library.mk</code> has been raised in the <code>#halium</code> channel; possible solutions include:</p>
<ul>
<li>symlink from <code>static_java_library.mk</code></li>
<li>edit the <code>Android.mk</code> file</li>
</ul>
<p>I went for the former; this probably needs to be fixed in Halium.</p>
<p>Reran <code>mka halium-boot</code>, got some improvement: <code>fstab</code> is found:</p>
<div class="code"><pre class="code literal-block">halium/hybris-boot/Android.mk:70: ********************* /boot appears to live on
halium/hybris-boot/Android.mk:71: ********************* /data appears to live on /dev/block/platform/sdio_emmc/by-name/userdata
halium/hybris-boot/Android.mk:74: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
</pre></div>
<p>It looks like my <code>fstab</code> was not complete, but missing the <code>/boot</code> partition.
given that none of the <code>fstab</code> I located in my source tree had a line for the
boot partition, I checked out the Android's <code>fstab</code> from the phone, via <code>adb
shell</code>. There is no <code>fstab</code> in <code>/etc</code>, but two fstab files right in the root
directory of the filesystem: <code>fstab.sc8830</code> with exactly the same contents as
the recovery fstab I found before in
<code>device/samsung/sharkls-common/recovery/root/fstab.sc8830</code>, and a
<code>fstab.goldfish</code>:</p>
<div class="code"><pre class="code literal-block">127|root@j3xlte:/ # cat /fstab.goldfish
# Android fstab file.
#<src> <mnt_point> <type> <mnt_flags and options> <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
/dev/block/mtdblock0 /system ext4 ro,noatime,barrier=1 wait
/dev/block/mtdblock1 /data ext4 noatime,nosuid,nodev,barrier=1,nomblk_io_submit wait,check
/dev/block/mtdblock2 /cache ext4 noatime,nosuid,nodev wait,check
/devices/platform/goldfish_mmc.0 auto vfat defaults voldmanaged=sdcard:auto
</pre></div>
<p>While <code>mount</code> says:</p>
<div class="code"><pre class="code literal-block">~ # mount
rootfs on / type rootfs (rw,seclabel)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
proc on /proc type proc (rw,relatime,gid=3009,hidepid=2)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,seclabel,relatime)
pstore on /sys/fs/pstore type pstore (rw,seclabel,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
/dev/block/mmcblk0p27 on /data type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p27 on /sdcard type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p24 on /cache type ext4 (rw,seclabel,relatime,data=ordered)
</pre></div>
<p>None of the above helps in locating the boot partition. I then tried to look under <code>/dev/block</code>:</p>
<div class="code"><pre class="code literal-block">root@j3xlte:/dev/block/platform/sdio_emmc/by-name # ls -l
lrwxrwxrwx root root 2019-06-26 19:13 CACHE -> /dev/block/mmcblk0p24
lrwxrwxrwx root root 2019-06-26 19:13 FOTA_SIG -> /dev/block/mmcblk0p11
lrwxrwxrwx root root 2019-06-26 19:13 HIDDEN -> /dev/block/mmcblk0p26
lrwxrwxrwx root root 2019-06-26 19:13 KERNEL -> /dev/block/mmcblk0p20
lrwxrwxrwx root root 2019-06-26 19:13 PARAM -> /dev/block/mmcblk0p16
lrwxrwxrwx root root 2019-06-26 19:13 PERSDATA -> /dev/block/mmcblk0p23
lrwxrwxrwx root root 2019-06-26 19:13 PERSISTENT -> /dev/block/mmcblk0p22
lrwxrwxrwx root root 2019-06-26 19:13 RECOVERY -> /dev/block/mmcblk0p21
lrwxrwxrwx root root 2019-06-26 19:13 RESERVED2 -> /dev/block/mmcblk0p19
lrwxrwxrwx root root 2019-06-26 19:13 SBOOT -> /dev/block/mmcblk0p1
lrwxrwxrwx root root 2019-06-26 19:13 SBOOT2 -> /dev/block/mmcblk0p2
lrwxrwxrwx root root 2019-06-26 19:13 SYSTEM -> /dev/block/mmcblk0p25
lrwxrwxrwx root root 2019-06-26 19:13 efs -> /dev/block/mmcblk0p17
lrwxrwxrwx root root 2019-06-26 19:13 l_fixnv1 -> /dev/block/mmcblk0p3
lrwxrwxrwx root root 2019-06-26 19:13 l_fixnv2 -> /dev/block/mmcblk0p4
lrwxrwxrwx root root 2019-06-26 19:13 l_gdsp -> /dev/block/mmcblk0p9
lrwxrwxrwx root root 2019-06-26 19:13 l_ldsp -> /dev/block/mmcblk0p7
lrwxrwxrwx root root 2019-06-26 19:13 l_modem -> /dev/block/mmcblk0p8
lrwxrwxrwx root root 2019-06-26 19:13 l_runtimenv1 -> /dev/block/mmcblk0p12
lrwxrwxrwx root root 2019-06-26 19:13 l_runtimenv2 -> /dev/block/mmcblk0p13
lrwxrwxrwx root root 2019-06-26 19:13 l_warm -> /dev/block/mmcblk0p10
lrwxrwxrwx root root 2019-06-26 19:13 pm_sys -> /dev/block/mmcblk0p5
lrwxrwxrwx root root 2019-06-26 19:13 prodnv -> /dev/block/mmcblk0p18
lrwxrwxrwx root root 2019-06-26 19:13 rsvdfixnv1 -> /dev/block/mmcblk0p6
lrwxrwxrwx root root 2019-06-26 19:13 td_runtimenv1 -> /dev/block/mmcblk0p14
lrwxrwxrwx root root 2019-06-26 19:13 td_runtimenv2 -> /dev/block/mmcblk0p15
lrwxrwxrwx root root 2019-06-26 19:13 userdata -> /dev/block/mmcblk0p27
</pre></div>
<p>This, combined from the <a href="https://www.droidviews.com/need-know-samsung-pit-files/">PIT
file</a> which we can
extract with Heimdall from the "Odin mode", gives us a better understanding of
the device partitions:</p>
<div class="code"><pre class="code literal-block">$ sudo heimdall print-pit --no-reboot
Heimdall v1.4.2
Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/
This software is provided free of charge. Copying and redistribution is
encouraged.
If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/
Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...
Initialising protocol...
Protocol initialisation successful.
Beginning session...
Some devices may take up to 2 minutes to respond.
Please be patient!
Session begun.
Downloading device's PIT file...
PIT file download successful.
Entry Count: 31
Unknown 1: 1598902083
Unknown 2: 844251476
Unknown 3: 20563
Unknown 4: 17490
Unknown 5: 14136
Unknown 6: 13619
Unknown 7: 0
Unknown 8: 0
--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT
Flash Filename: spl.img
FOTA Filename:
--- Entry #1 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 90
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT2
Flash Filename: spl2.img
FOTA Filename:
--- Entry #2 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 70
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 1024
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PIT
Flash Filename: J3XLTE.pit
FOTA Filename:
--- Entry #3 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 71
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 2048
Partition Block Count: 6144
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: MD5HDR
Flash Filename: md5.img
FOTA Filename:
--- Entry #4 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 1
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 8192
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT
Flash Filename: sboot.bin
FOTA Filename:
--- Entry #5 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 2
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 12288
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT2
Flash Filename: sboot2.bin
FOTA Filename:
--- Entry #6 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 3
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 16384
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv1
Flash Filename: nvitem1.bin
FOTA Filename:
--- Entry #7 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 4
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 18432
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv2
Flash Filename: nvitem.bin
FOTA Filename:
--- Entry #8 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 5
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 20480
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: pm_sys
Flash Filename: PM_sharkl_arm7.bin
FOTA Filename:
--- Entry #9 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 6
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 22528
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: rsvdfixnv1
Flash Filename:
FOTA Filename:
--- Entry #10 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 7
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 24576
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_ldsp
Flash Filename: SPRDLTEDSP.img
FOTA Filename:
--- Entry #11 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 8
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 32768
Partition Block Count: 32768
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_modem
Flash Filename: SPRDCP.img
FOTA Filename:
--- Entry #12 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 9
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 65536
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_gdsp
Flash Filename: SPRDGDSP.img
FOTA Filename:
--- Entry #13 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 10
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 73728
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_warm
Flash Filename: SPRDWDSP.img
FOTA Filename:
--- Entry #14 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 11
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 81920
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: FOTA_SIG
Flash Filename:
FOTA Filename:
--- Entry #15 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 12
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 83968
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv1
Flash Filename:
FOTA Filename:
--- Entry #16 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 13
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 86016
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv2
Flash Filename:
FOTA Filename:
--- Entry #17 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 14
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 88064
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv1
Flash Filename:
FOTA Filename:
--- Entry #18 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 15
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 90112
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv2
Flash Filename:
FOTA Filename:
--- Entry #19 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 16
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 92160
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PARAM
Flash Filename: param.lfs
FOTA Filename:
--- Entry #20 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 17
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 96256
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: efs
Flash Filename: efs.img
FOTA Filename:
--- Entry #21 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 18
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 137216
Partition Block Count: 10240
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: prodnv
Flash Filename: prodnv.img
FOTA Filename:
--- Entry #22 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 19
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 147456
Partition Block Count: 12288
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RESERVED2
Flash Filename:
FOTA Filename:
--- Entry #23 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 20
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 159744
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: KERNEL
Flash Filename: boot.img
FOTA Filename:
--- Entry #24 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 21
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 200704
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RECOVERY
Flash Filename: recovery.img
FOTA Filename:
--- Entry #25 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 22
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 241664
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSISTENT
Flash Filename:
FOTA Filename:
--- Entry #26 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 23
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 243712
Partition Block Count: 18432
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSDATA
Flash Filename: persdata.img
FOTA Filename:
--- Entry #27 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 24
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 262144
Partition Block Count: 409600
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: CACHE
Flash Filename: cache.img
FOTA Filename:
--- Entry #28 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 25
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 671744
Partition Block Count: 4194304
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SYSTEM
Flash Filename: system.img
FOTA Filename:
--- Entry #29 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 26
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4866048
Partition Block Count: 81920
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: HIDDEN
Flash Filename: hidden.img
FOTA Filename:
--- Entry #30 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 27
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4947968
Partition Block Count: 0
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: userdata
Flash Filename: userdata.img
FOTA Filename: remained
Ending session...
Releasing device interface...
</pre></div>
<p>Looking at the name, size and flash filename of the partitions, we can
guess that the <code>BOOT</code> and <code>BOOT2</code> are about splash screens, and what we need to
use as our boot partition is in fact the <code>KERNEL</code> partition:
<code>/dev/block/mmcblk0p20</code> (note how the <code>Identifier</code> field from the PIT file
matches what we got from block devices in the <code>/dev</code> filesystem). So, we edit
our <code>fstab</code> file and add a line</p>
<div class="code"><pre class="code literal-block">/dev/block/platform/sdio_emmc/by-name/KERNEL /boot ext4 ro wait
</pre></div>
<p>From the IRC #halium channel, I understood that the <a href="http://docs.halium.org/en/latest/porting/build-sources.html#include-your-device-in-fixup-mountpoints">fixup mountpoints
step</a>
is needed. The block I added to the <code>halium/hybris-boot/fixup-mountpoints</code> script is this:</p>
<div class="code"><pre class="code literal-block"><span class="w"> </span><span class="s2">"j3xlte"</span><span class="o">)</span>
<span class="w"> </span>sed<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/SYSTEM mmcblk0p25 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/userdata mmcblk0p27 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/CACHE mmcblk0p24 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/efs mmcblk0p17 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/prodnv mmcblk0p18 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/KERNEL mmcblk0p20 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="w"> </span><span class="p">;;</span>
</pre></div>
<p>I then ran <code>mka halium-boot</code> again, and it finally started to build the kernel.
But it soon stopped:</p>
<div class="code"><pre class="code literal-block">/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c: In function 'subsys_cgroup_allow_attach':
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2138:37: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
if (current != task && cred->euid != tcred->uid &&
^
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2139:18: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
cred->euid != tcred->suid)
^
</pre></div>
<p>Luckily, this is one of the <a href="https://docs.halium.org/en/latest/porting/common-kernel-build-errors.html">known build
errors</a>,
so I didn't waste a lot of time on it: setting <code>CONFIG_USER_NS=n</code> in
<code>kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig</code> fixed it. Then the build
failed with</p>
<div class="code"><pre class="code literal-block"> LD init/built-in.o
mm/built-in.o: In function `set_AKSM_level':
/mnt/Lavoro/halium/kernel/samsung/sharkls/mm/ksm.c:1822: undefined reference to `get_minfree_high_value'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2
</pre></div>
<p>Looking at the code, I understood I could probably switch off this code path
with <code>CONFIG_ADAPTIVE_KSM=n</code>. After doing that, got another error:</p>
<div class="code"><pre class="code literal-block"> LD init/built-in.o
arch/arm/lib/lib.a(memcmpksm.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2
</pre></div>
<p>Disabling also <code>CONFIG_KSM_ASSEMBLY_MEMCMP</code> did the trick:</p>
<div class="code"><pre class="code literal-block"> CC /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.mod.o
LD [M] /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.ko
make[1]: Leaving directory '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ'
make: Leaving directory '/mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0'
[ 77% 7/9] Target dt image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
DTB combiner:
Input directory: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ/arch/arm/boot/dts/'
Output file: '/mnt/Lavoro/halium/out/target/product/j3xlte/dt.img'
Found file: sprd-scx35l_sharkls_j3xlte_rev03.dtb ... chipset: 9830, platform: 3, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev02.dtb ... chipset: 9830, platform: 2, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev06.dtb ... chipset: 9830, platform: 6, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev07.dtb ... chipset: 9830, platform: 7, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev04.dtb ... chipset: 9830, platform: 4, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev01.dtb ... chipset: 9830, platform: 1, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev05.dtb ... chipset: 9830, platform: 5, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev00.dtb ... chipset: 9830, platform: 0, rev: 131072
=> Found 8 unique DTB(s)
Generating master DTB... completed
Made DT image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
[100% 9/9] Install: /mnt/Lavoro/halium/out/target/product/j3xlte/halium-boot.img
make: Leaving directory '/mnt/Lavoro/halium'
#### make completed successfully (01:28 (mm:ss)) ####
</pre></div>
<h3>Building the Halium system image</h3>
<p>Continuing following the Halium documentation, the next step was building the
system image with the <code>mka systemimage</code> command. Of course, it failed:</p>
<div class="code"><pre class="code literal-block">make: Entering directory '/mnt/Lavoro/halium'
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190702-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-52-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================
Running kati to generate build-lineage_j3xlte.ninja...
No need to regenerate ninja file
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libcurl_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstrongswan_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
#### make failed to build some targets (2 seconds) ####
</pre></div>
<p>I found similar (though not exactly the same) errors in <code>#halium</code>: the solution
is to locate the <code>.mk</code> file causing the build of the package requiring the
missing file, and remove it. I ran a <code>repo grep libstrongswan</code>, and found that
it's mentioned in <code>device/samsung/sharkls-common/sharkls.mk</code>. I removed it from
there, but I got the same build error. Removing the whole block of
<code>PRODUCT_PACKAGES</code> led to some progress:</p>
<div class="code"><pre class="code literal-block">build/core/binary.mk:1253: vendor/sprd/modules/libcamera/Android.mk: libcamisp2.0: Unused source files: isp2.0/calibration/backup isp2.0/third_lib/alc_ip
build/core/binary.mk:1253: vendor/sprd/open-source/libs/vpu/mmf/openmax/libomxil-bellagio-0.9.3/Android.mk: omxregister-bellagio: Unused source files: src/omxregister.h
No private recovery resources for TARGET_DEVICE j3xlte
build/core/Makefile:34: warning: overriding commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
build/core/base_rules.mk:316: warning: ignoring old commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbt-utils_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/engpc_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>I then removed <code>engpc</code> from the same <code>sharkls.mk</code> file:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/vold_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>The last comment from <a href="https://github.com/Halium/halium-devices/pull/116">here</a>
suggests that <code>vold</code> might not be needed (though some people in <code>#halium</code>
claimed that it could be needed, and indeed in the Meizu Pro 5 by Canonical I
do have that program under <code>/system/bin/</code>). Anyway, I edited
<code>build/target/product/base.mk</code> and removed <code>vold</code> from the <code>PRODUCT_PACKAGES</code>
variable. I also read that it's recommended to run the make command as <code>LANG=C
mka systemimage</code>, since the outcome can be affected by the user locale. Next:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libfmjni_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengfm_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>This is mentioned in <code>vendor/sprd/wcn/platform/wcnd/Android.mk</code>, so let's remove it from there (<code>LOCAL_SHARED_LIBRARIES</code> variable):</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/at_distributor', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/at_distributor', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>This is brought in by <code>device/samsung/j3xlte/proprietary-files.txt</code>, so I
commented it out from there. This didn't seem to make any difference, and a
<code>mka clean</code> didn't help either. Turns out it's also mentioned in
<code>vendor/samsung/j3xlte/j3xlte-vendor-blobs.mk</code>, so I removed it from there too
(<code>PRODUCT_COPY_FILES</code> variable). Reading the comment block on top of this
makefile we understand that it's generated by the <code>setup-makefiles.sh</code> script
based on the contents of <code>proprietary-files.txt</code>. So, one option is to re-run
the <code>setup-makefiles.sh</code> script, but I preferred to take a no risk approach and
edit both files by hand. The next failure was about <code>mfgloader</code>, which is also
mentioned in the same file:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/mfgloader', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/mfgloader', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>Let's try to remove it as well, from the same two files:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/sprdSleepLog', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/sprdSleepLog', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>Next is <code>modemd</code> (exactly same logs). At this point I started suspecting that
this was not the right way to proceed, that that instead I was just missing all
the proprietary blobs. The <a href="https://github.com/djeman/android_device_samsung_j3xnlte/tree/cm-14.1">device
tree</a>
repository has an <code>extract-files.sh</code> which seems to get these binaries from a
device. After looking at its source, it looks like it must be run from the
<code>device/samsung/j3xlte</code> directory with the device connected and reachable via
<code>adb</code>. Before running it, I reverted my changes to the <code>proprietary-files.txt</code>
file, in order to retrieve all the needed blobs. Running the extraction tool
failed once:</p>
<div class="code"><pre class="code literal-block">./extract-files.sh
846 KB/s (92260 bytes in 0.106s)
108 KB/s (9664 bytes in 0.086s)
106 KB/s (9444 bytes in 0.086s)
279 KB/s (34092 bytes in 0.119s)
44 KB/s (1992 bytes in 0.043s)
115 KB/s (5296 bytes in 0.044s)
2 KB/s (265 bytes in 0.087s)
40 KB/s (3770 bytes in 0.090s)
25 KB/s (2247 bytes in 0.087s)
4 KB/s (427 bytes in 0.087s)
0 KB/s (70 bytes in 0.087s)
remote object '/system/bin/IPSecService' does not exist
</pre></div>
<p>but after removing the missing file from <code>proprietary-files.txt</code>, the
extraction completed successfully. However, one file was still not retrieved,
as this line from the output showed:</p>
<div class="code"><pre class="code literal-block">[...]
4779 KB/s (1386252 bytes in 0.283s)
405 KB/s (37928 bytes in 0.091s)
failed to copy '/system/vendor/firmware/vbc_eq' to '../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq': Permission denied
3015 KB/s (415875 bytes in 0.134s)
[...]
</pre></div>
<p>I checked the situation by connecting to the device with <code>adb shell</code>, and the
file was there but readable by root only. After copying the file to
<code>/data/user</code> and calling <code>chmod 0666 vbc_eq</code>, I could manually retrieve it with</p>
<div class="code"><pre class="code literal-block">adb<span class="w"> </span>pull<span class="w"> </span>/data/user/vbc_eq<span class="w"> </span>../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq
</pre></div>
<p>I preferred not to touch the permissions directly on the original file, not to
risk messing up with the system; and after retrieving the file, I removed my
copy from the device. The I ran</p>
<div class="code"><pre class="code literal-block"><span class="nv">JOBS</span><span class="o">=</span><span class="m">2</span><span class="w"> </span>./halium/devices/setup<span class="w"> </span>j3xlte
</pre></div>
<p>once again. That completed fine, and the next <code>mka systemimage</code> failed with
another error:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/init_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>I played with the
<a href="https://github.com/Halium/android_system_core">android_system_core</a>
repository, which is checked out in <code>system/core/</code>, to remove references to
<code>bootloader_message</code> from <code>init/builtins.cpp</code>, and I also removed it from the
<code>LOCAL_STATIC_LIBRARIES</code> variable in the <code>system/core/init/Android.mk</code> file.
Next:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libminui_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/healthd_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>This is listed in <code>build/target/product/core_base.mk</code> and
<code>build/target/product/embedded.mk</code> and needs to be removed from both (it was
<a href="https://github.com/Halium/android_system_core/commit/eab7224aeafc07f2892f2af6fc7c3cc4859a4edb#diff-4b4390397dd367143db924ef288a5d21">disabled in Halium's
init</a>,
so it's definitely not needed). It then failed with:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'bootable/recovery-twrp/etc/init.rc', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/ramdisk-recovery.cpio', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>The <code>recovery-twrp</code> project is listed in <code>device/samsung/sharkls-common/lineage.dependencies</code>, so let's try adding a line</p>
<div class="code"><pre class="code literal-block"><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"bootable/recovery-twrp"</span><span class="w"> </span><span class="na">name=</span><span class="s">"omnirom/android_bootable_recovery"</span><span class="w"> </span><span class="na">remote=</span><span class="s">"github"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"android-7.1"</span><span class="w"> </span><span class="nt">/></span>
</pre></div>
<p>to our manifest file and run <code>JOBS=2 ./halium/devices/setup j3xlte</code> once more. After that, <code>mka systemimage</code> gives another error:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/host/linux-x86/bin/imgdiff', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>Adding <code>bootable/recovery-twrp</code> to the <code>subdirs</code> variable in
<code>build/core/main.mk</code> seems to get us further. But before showing you the next
error, I need to stop for a second, because at this point I started suspecting
that the previous error about the <code>bootloader_message</code> library could go away
now that I started using the same recovery-twrp tree used by the LineageOS
porter. So I reverted my changes about the <code>bootloader_message</code> library and
indeed it looked like the previous error was gone forever. And it might be that
some of the other stuff I disabled above might be restored back; but I decided
to leave it for later (it's documented in this blog post, after all, so I can
go back to it at any time) and proceed with the build:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
[ 0% 125/16364] host C++: libadb <= system/core/adb/usb_linux.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I system/core/adb -I /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates -I /mnt/Lavoro/halium/out/host/linux-x86/gen/STATIC_LIBRARIES/libadb_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem tools/include -isystem /mnt/Lavoro/halium/out/host/linux-x86/obj/include -c -fno-exceptions -Wno-multichar -m64 -Wa,--noexecstack -fPIC -no-canonical-prefixes -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -O2 -g -fno-strict-aliasing -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 -fstack-protector-strong --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -Bprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin -target x86_64-linux-gnu -Wsign-promo -Wno-inconsistent-missing-override --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8 -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/x86_64-linux -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/backward -target x86_64-linux-gnu -Wall -Wextra -Werror -Wno-unused-parameter -Wno-missing-field-initializers -Wvla -DADB_REVISION='\"80811505f1ff-android\"' -fvisibility=hidden -DADB_HOST=1 -fPIC -std=c++14 -Wexit-time-destructors -D_USING_LIBCXX -std=gnu++14 -nostdinc++ -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d -o /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.o system/core/adb/usb_linux.cpp ) && (cp /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d >> /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; rm -f /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d )"
In file included from system/core/adb/usb_linux.cpp:25:
In file included from device/samsung/sharkls-common/include/linux/usb/ch9.h:35:
In file included from device/samsung/sharkls-common/include/uapi/linux/usb/ch9.h:36:
In file included from device/samsung/sharkls-common/include/linux/types.h:5:
In file included from device/samsung/sharkls-common/include/uapi/linux/types.h:13:
In file included from prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/posix_types.h:4:
device/samsung/sharkls-common/include/linux/stddef.h:11:2: error: expected identifier
false = 0,
^
prebuilts/clang/host/linux-x86/clang-2690385/bin/../lib64/clang/3.8.256229/include/stdbool.h:38:15: note: expanded from macro 'false'
#define false false
^
</pre></div>
<p>(see <a href="https://pastebin.com/PBwKbDFX">here</a> for the full log). After some
resultless internet search and asking around, I tried to look into the issue
myself, and noticed that in that failed command there's a mix of headers coming
from the userspace and from the kernel (which usually are not used together).
Since we are building userspace programs now, it seems a bit strange that there
are also kernel headers being used (those under
<code>device/samsung/sharkls-common</code>). Looking at <code>system/core/adb/usb_linux.cpp</code>, I
saw that it contains a line <code>#include <linux/usb/ch9.h></code> which causes the
inclusion of <code>device/samsung/sharkls-common/include/linux/usb/ch9.h</code>; but
maybe there are other <code>ch9.h</code> files in the build tree, provided by userspace
libraries and not by the kernel? Indeed, a <code>find</code> showed that I had also a
<code>./prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/usb/ch9.h</code>
file, which should probably be used instead of the kernel's one. So I copied
the failing command line (starting from what comes after <code>FAILED:</code>) into a
terminal and ran it, and could reproduce the issue; but if after removing the
first include directive (<code>-I device/samsung/sharkls-common/include</code>) the
command succeeded.</p>
<p>The line is added in <code>build/core/binary.mk</code>:</p>
<div class="code"><pre class="code literal-block">my_c_includes := $(TOPDIR)$(TARGET_SPECIFIC_HEADER_PATH) $(my_c_includes)
</pre></div>
<p>(I found it out by modifying the file, and adding random text strings around
the right-hand parts of these assignments, then running the build again and see
which random strings were surrounding the
<code>device/samsung/sharkls-common/include</code> text in the failing command line). So,
we need to find why <code>TARGET_SPECIFIC_HEADER_PATH</code> is set. I ran a <code>repo grep
TARGET_SPECIFIC_HEADER_PATH</code> and found that it's being set in
<code>device/samsung/sharkls-common/BoardConfigCommon.mk</code>. Browsing the various
directory trees, and double-checking the manifest file, I realized that the
<code>device/samsung/sharkls-common</code> should not contain the kernel source tree, yet
in my local disk it did (and all the files were untracked by git). It was
probably the result of one of my previous attempts of compiling the manifest
file, where some projects were checked out locally but I didn't clean them up
after updating the manifest. I therefore ran a <code>git clean -dfx</code> in that
directory, and the kernel source were gone. The next build continued for a
while, then failed:</p>
<div class="code"><pre class="code literal-block">[ 76% 12454/16364] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:10: fatal error: 'bt_types.h' file not found
#include "bt_types.h"
^
1 error generated.
</pre></div>
<p>Indeed the <code>bt_types.h</code> was nowhere to be found in my source tree. A search in GitHub reveals that this file is provided by <code>libncf-nci</code>, so it's probably related to NFC. The local manifest from LineageOS indeed had a few lines with NFC-related projects (at least, judging from their name):</p>
<div class="code"><pre class="code literal-block"><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/nxp-nfc/opensource/Nfc"</span><span class="w"> </span><span class="na">name=</span><span class="s">"LineageOS/android_vendor_nxp-nfc_opensource_Nfc"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/nxp-nfc/opensource/frameworks"</span><span class="w"> </span><span class="na">name=</span><span class="s">"LineageOS/android_vendor_nxp-nfc_opensource_frameworks"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/nxp-nfc/opensource/libnfc-nci"</span><span class="w"> </span><span class="na">name=</span><span class="s">"LineageOS/android_vendor_nxp-nfc_opensource_libnfc-nci"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
</pre></div>
<p>I initially excluded them from my manifest, given that my device doesn't have
NFC, but at this point I thought I'd better include them, just for the sake of
building the binaries (the alternative being removing the binaries required
these libraries, or hacking on them to remove the dependency; this is a
decision I might want to revisit later). So I added them to my manifest file
(adding the proper <code>remote</code> attribute!), re-ran the setup script which fetched
the newly-added repositories, but the build still failed with the same error,
as the include lines were not updated. However, adding
<code>vendor/nxp-nfc/opensource/libnfc-nci/src/include</code> to the
<code>TARGET_SPECIFIC_HEADER_PATH</code> variable in
<code>device/samsung/sharkls-common/BoardConfigCommon.mk</code> caused the build to fail
even earlier. I then tried to add it to the local project being built
(<code>engmode</code>), by changing the <code>LOCAL_C_INCLUDES</code> variable in
<code>vendor/sprd/open-source/apps/engmode/bt/Android.mk</code> and that brought me just a
little further:</p>
<div class="code"><pre class="code literal-block">[ 97% 1507/1552] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
In file included from vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:
vendor/nxp-nfc/opensource/libnfc-nci/src/include/bt_types.h:22:10: fatal error: 'data_types.h' file not found
#include "data_types.h"
^
1 error generated.
</pre></div>
<p>(note that in order to speed up the development I was no longer running <code>mka
systemimage</code>, but <code>mka libengbt</code>, which is the library affected by the
failure). I had three candidates:</p>
<div class="code"><pre class="code literal-block">vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/src/gki/ulinux/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/halimpl/bcm2079x/gki/ulinux/data_types.h
</pre></div>
<p>they didn't seem to be very different, so I picked the first one. I got then
another failure:</p>
<div class="code"><pre class="code literal-block">[ 35% 12/34] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/ -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:31:10: fatal error: 'btm_api.h' file not found
#include "btm_api.h"
^
1 error generated.
</pre></div>
<p>I search in the internet revealed that this is part of Bluedroid, which I
assume is the Android bluetooth stack; so I looked in the Halium manifest
directory for a repo line:</p>
<div class="code"><pre class="code literal-block">$ git grep blue
motorola_potter.xml: <project path="external/bluetooth/bluedroid" name="android_external_bluetooth_bluedroid" remote="los" revision="cm-12.1" />
</pre></div>
<p>This one seemed promising, so I added it to my manifest. I ran the setup script
again, and had a look at the newly downloaded
<code>external/bluetooth/bluedroid/stack/include/</code> directory: that not only included
the <code>btm_api.h</code> file, but also the <code>bt_types.h</code> which we were needing before.
So I went back and tried removing the two include lines related to the
<code>nxp-nfc</code> projects and replaced them with this one from bluedroid. I got the
same issue about the missing <code>data_types.h</code> file, so I added back the
<code>vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc</code> line. I then got an
error about a missing <code>tb_target.h</code> file, which is found in
<code>external/bluetooth/bluedroid/include/</code>, so I added this line as well. This let
me build <code>libengbt</code> successfully. The <code>systemimage</code> target instead failed
later:</p>
<div class="code"><pre class="code literal-block">[ 12% 502/3912] target thumb C++: audio.primary.sc8830 <= vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I external/tinyalsa/include -I external/expat/lib -I system/media/audio_utils/include -I system/media/audio_effects/include -I vendor/sprd/open-source/apps/engmode -I vendor/sprd/modules/audio/normal/vb_pga -I vendor/sprd/modules/audio/normal/record_process -I vendor/sprd/modules/audio/normal/include -I vendor/sprd/modules/audio/normal/DumpData -I vendor/sprd/modules/audio/normal/audiotest -I vendor/sprd/modules/audio/normal/record_nr -I vendor/sprd/modules/audio/normal/skd -I vendor/sprd/modules/audio/normal/resample_api -I vendor/sprd/modules/audio/normal/custom_mmi -I vendor/sprd/modules/resampler -I vendor/sprd/modules/audio/normal/libaudioril -I vendor/sprd/open-source/libs/libatci -I vendor/sprd/modules/audio/normal/vb_effect/v2 -I vendor/sprd/modules/audio/normal -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -fvisibility-inlines-hidden -Wsign-promo -Wno-inconsistent-missing-override -nostdlibinc -target arm-linux-androideabi -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -fno-rtti -D_POSIX_SOURCE -Wno-multichar -g -DAUDIO_SPIPE_TD -D_LPA_IRAM -DVOIP_DSP_PROCESS -DAUDIO_HAL_ANDROID_N_API -DFM_VERSION_IS_GOOGLE -fPIC -D_USING_LIBCXX -std=gnu++14 -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.o vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d )"
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:21:9: warning: 'LOG_TAG' macro redefined [-Wmacro-redefined]
#define LOG_TAG "SPRD_AUDIOTRACK"
^
system/core/include/log/log.h:65:9: note: previous definition is here
#define LOG_TAG NULL
^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:186:39: warning: unused parameter 'user' [-Wunused-parameter]
void playerCallback( int event, void* user, void *info )
^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:245:9: error: use of undeclared identifier 'AUDIO_DEVICE_OUT_FM_HEADSET'; did you mean 'AUDIO_DEVICE_OUT_WIRED_HEADSET'?
AUDIO_DEVICE_OUT_FM_HEADSET,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
AUDIO_DEVICE_OUT_WIRED_HEADSET
system/media/audio/include/system/audio.h:711:5: note: 'AUDIO_DEVICE_OUT_WIRED_HEADSET' declared here
AUDIO_DEVICE_OUT_WIRED_HEADSET = 0x4,
^
</pre></div>
<p>This is an issue reported
<a href="https://github.com/djeman/android_vendor_sprd/issues/1">here</a>, The link
mentioned there is for CyanogenMod 13, but I found one for LineageOS 14.1
<a href="https://github.com/djeman/android_device_samsung_sharkls-common/tree/cm-14.1/patches">here</a>.
So I ran</p>
<div class="code"><pre class="code literal-block">cd device/samsung/sharkls-common/patches/
./apply_sprd-diff.sh
</pre></div>
<p>This failed on a couple of patches, for which I reported an issue
<a href="https://github.com/djeman/android_device_samsung_sharkls-common/issues/1">here</a>,
though it's likely not important for us (as it mainly regards Java code). The
build proceeded:</p>
<div class="code"><pre class="code literal-block">[ 10% 436/4011] target thumb C++: libstagefright <= frameworks/av/media/libstagefright/CameraSource.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I ./frameworks/av/include/media/ -I ./frameworks/av/media/libavextensions -I ./frameworks/av/media/libstagefright/mpeg2ts -I ./frameworks/av/include/media/stagefright/timedtext -I ./frameworks/native/include/media/hardware -I ./frameworks/native/include/media/openmax -I ./external/flac/include -I ./external/tremolo -I ./external/libvpx/libwebm -I ./system/netd/include -I system/media/audio_utils/include -I hardware/qcom/media/sc8830/mm-core/inc -I ./external/stagefright-plugins/include -I frameworks/av/media/libstagefright -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libstagefright_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -fvisibility-inlines-hidden -Wsign-promo -Wno-inconsistent-missing-override -nostdlibinc -target arm-linux-androideabi -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -fno-rtti -Wno-multichar -Werror -Wno-error=deprecated-declarations -Wall -DENABLE_STAGEFRIGHT_EXPERIMENTS -DUSE_SPRD_COLORFORMAT -fPIC -D_USING_LIBCXX -fsanitize=unsigned-integer-overflow,signed-integer-overflow -fsanitize-trap=all -ftrap-function=abort -std=gnu++14 -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.o frameworks/av/media/libstagefright/CameraSource.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d )"
frameworks/av/media/libstagefright/CameraSource.cpp:134:16: error: use of undeclared identifier 'OMX_SPRD_COLOR_FormatYVU420SemiPlanar'; did you mean 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar'?
return OMX_SPRD_COLOR_FormatYVU420SemiPlanar;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OMX_QCOM_COLOR_FormatYVU420SemiPlanar
./frameworks/native/include/media/openmax/OMX_IVCommon.h:169:5: note: 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar' declared here
OMX_QCOM_COLOR_FormatYVU420SemiPlanar = 0x7FA30C00,
^
1 error generated.
</pre></div>
<p>I realized that there were more patches to be applied, and that the script stopped as soon as it stumbled into a failing patch. So, I took all the failing patches out of the way:</p>
<div class="code"><pre class="code literal-block">mv<span class="w"> </span>sprd-diff/frameworks_base.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/frameworks_opt.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/packages_apps.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/packages_services.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/system_bt.diff<span class="o">{</span>,.skip<span class="o">}</span><span class="w"> </span><span class="c1"># This looks potentially worth a revisit: should we have a system/bt dir?</span>
</pre></div>
<p>The next build failed here:</p>
<div class="code"><pre class="code literal-block">[ 81% 3506/4283] target thumb C: libbt-vendor <= vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I vendor/sprd/wcn/bt/libbt/include -I system/bt/hci/include -I vendor/sprd/wcn/bt/libbt/conf/sprd/marlin/include -I vendor/sprd/wcn/bt/libbt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libbt-vendor_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.o vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d )"
In file included from vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c:31:
vendor/sprd/wcn/bt/libbt/include/bt_vendor_sprd.h:33:10: fatal error: 'bt_vendor_lib.h' file not found
#include "bt_vendor_lib.h"
^
1 error generated.
</pre></div>
<p>I had this file in <code>external/bluetooth/bluedroid/hci/include/bt_vendor_lib.h</code>, and I was about to add this directory to the <code>vendor/sprd/wcn/bt/libbt/Android.mk</code> file when I realized that this file was already including <code>$(BDROID_DIR)/hci/include</code> and defined <code>BDROID_DIR := $(TOP_DIR)system/bt</code>. So I understood that I should have changed my manifest file to check out the bluedroid repo under <code>system/bt</code> (which also meant that I was wrong in excluding the <code>sprd-diff/system_bt.diff</code> patch before). I also had a quick look at other manifest files, and I saw that they were typically getting <code>system/bt</code> from the LineageOS <code>android_system_bt</code> repo. Therefore I went to modify my manifest file, changing the bluedroid line to</p>
<div class="code"><pre class="code literal-block"><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"system/bt"</span><span class="w"> </span><span class="na">name=</span><span class="s">"android_system_bt"</span><span class="w"> </span><span class="na">remote=</span><span class="s">"los"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
</pre></div>
<p>and removed the three <code>vendor/nxp-nfc</code> repositories I added a while before.
Then I manually applied the patch in
<code>device/samsung/sharkls-common/patches/sprd-diff/system_bt.diff</code> and I
revisited the <code>vendor/sprd/open-source/apps/engmode/bt/Android.mk</code> to remove
all the include lines I added before. The next build succeeded:</p>
<div class="code"><pre class="code literal-block">[...]
[ 99% 2250/2255] build /mnt/Lavoro/halium/out/target/product/j3xlte/obj/NOTICE.html
Combining NOTICE files into HTML
Combining NOTICE files into text
[ 99% 2254/2255] Target system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
BuildImage: in_dir = /mnt/Lavoro/halium/out/target/product/j3xlte/system, out_file = /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
fs type is not ext4
Running: mkuserimg.sh -s /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img ext4 system 2147483648 -D /mnt/Lavoro/halium/out/target/product/j3xlte/system -L system /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin
make_ext4fs -s -T -1 -S /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin -L system -l 2147483648 -a system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/system
Creating filesystem with parameters:
Size: 2147483648
Block size: 4096
Blocks per group: 32768
Inodes per group: 8192
Inode size: 256
Journal blocks: 8192
Label: system
Blocks: 524288
Block groups: 16
Reserved block group size: 127
Created filesystem with 1381/131072 inodes and 55467/524288 blocks
Running ['mkuserimg.sh', '-s', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img', 'ext4', 'system', '2147483648', '-D', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '-L', 'system', '/mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin'] command, exit code = 0
[ 99% 2254/2255] Construct recovery from boot
failed to reconstruct target deflate chunk 1 [(null)]; treating as normal
chunk 0: type 0 start 0 len 5582858
chunk 1: type 2 start 5582858 len 8611584
chunk 2: type 0 start 9205682 len 673886
Construct patches for 3 chunks...
patch 0 is 212 bytes (of 5582858)
patch 1 is 4311951 bytes (of 3622824)
patch 2 is 207 bytes (of 673886)
chunk 0: normal ( 0, 5582858) 212
chunk 1: deflate ( 5582858, 8574717) 4311951 (null)
chunk 2: normal ( 14157575, 674057) 207
[100% 2255/2255] Install system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/system.img
/mnt/Lavoro/halium/out/target/product/j3xlte/system.img+/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p maxsize=2192424960 blocksize=135168 total=197470210 reserve=22167552
make: Leaving directory '/mnt/Lavoro/halium'
#### make completed successfully (01:22 (mm:ss)) ####
</pre></div>
<p>Just to be on the safe side (to make sure I didn't have stale files from previous build attempts), I ran </p>
<div class="code"><pre class="code literal-block">mka<span class="w"> </span>clean
mka<span class="w"> </span>halium-boot
<span class="nv">LANG</span><span class="o">=</span>C<span class="w"> </span>mka<span class="w"> </span>systemimage
</pre></div>
<p>again, without encountering any errors. I also ran <code>mka hybris-boot</code> as per the
Halium documentation, as it might be a good idea to test the Halium reference
image first. <code>hybris-boot.img</code> was built successfully at the first try.</p>
<h3>Flashing and testing the Halium reference image</h3>
<p>Following the Halium documentation, I downloaded the Halium rootfs image:</p>
<div class="code"><pre class="code literal-block">mkdir<span class="w"> </span>rootfs
<span class="nb">cd</span><span class="w"> </span>rootfs
wget<span class="w"> </span><span class="s1">'http://bshah.in/halium/halium-rootfs-20170630-151006.tar.gz'</span>
</pre></div>
<p>A small section of the documentation is dedicated on <a href="http://docs.halium.org/en/latest/porting/install-build/reference-rootfs.html#install-hybris-boot-img-on-samsung-devices">how to install the boot
image for Samsung
devices</a>;
it closely matches the steps I took before to install TWRP, which is
comforting.</p>
<div class="code"><pre class="code literal-block">adb<span class="w"> </span>reboot<span class="w"> </span>download<span class="w"> </span><span class="c1"># much easier than the weird key combinations!</span>
sudo<span class="w"> </span>heimdall<span class="w"> </span>flash<span class="w"> </span>--BOOT<span class="w"> </span>out/target/product/j3xlte/hybris-boot.img
</pre></div>
<p>The last command ended with this output:</p>
<div class="code"><pre class="code literal-block">Uploading BOOT
100%
BOOT upload successful
Ending session...
ERROR: Failed to receive session end confirmation!
Releasing device interface...
</pre></div>
<p>and the device was stuck on the same screen. I then thought of using TWRP to
flash this device. I rebooted in download mode and installed TWRP as explained
near the beginning of this blog post, then I uploaded my boot image to the SD
card:</p>
<div class="code"><pre class="code literal-block">adb<span class="w"> </span>push<span class="w"> </span>out/target/product/j3xlte/hybris-boot.img<span class="w"> </span>/sdcard/
</pre></div>
<p>In TWRP, I did a wipe first: I wiped off the "System", "Data" and "cache"
partitions (you can choose which partitions to wipe under the "Advanced wipe"
option). Then I went back and pressed the "Install" button, then "Install
Image" and picked my <code>hybris-boot.img</code> file (and chose "Boot" as target, of
course). Make sure you don't reboot at this stage, or you'll lose <code>adb</code> access
and the rest of the commands will fail.</p>
<p>I then installed the <a href="https://gitlab.com/JBBgameich/halium-install/">halium-install</a> script:</p>
<div class="code"><pre class="code literal-block">git<span class="w"> </span>clone<span class="w"> </span>https://gitlab.com/JBBgameich/halium-install.git
</pre></div>
<p>I then downloaded the original Samsung ROM (see <a href="http://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html#stock">below</a> for the link)
and unzipped it in the <code>rootfs</code> directory I created before. That produced a
file named <code>J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5</code>, from which
I extracted the <code>system.img</code> and uploaded it to the device along with the
halium rootfs, as explained in the "Halium guide":</p>
<div class="code"><pre class="code literal-block">tar<span class="w"> </span>xvf<span class="w"> </span>J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5<span class="w"> </span>system.img
../halium-install/halium-install<span class="w"> </span>-p<span class="w"> </span>halium<span class="w"> </span>-v<span class="w"> </span>halium-rootfs-20170630-151006.tar.gz<span class="w"> </span>system.img
</pre></div>
<p>The last command complained about <code>simg2img</code> not being in <code>$PATH</code>, despite the
fact that it was. A <a href="https://gitlab.com/JBBgameich/halium-install/merge_requests/28/diffs">small
change</a>
to the script fixed this. Once that was fixed it successfully pushed the
files to the device, after asking me which password I wanted to set for the
root user (on the phone): I chose a familiar "0000".</p>
<p>After rebooting, however, the device didn't go beyond the boot screen (with a
red line warning about SELinux not being enabled). It just got stuck there, and
was not even detected via USB.</p>
<p>Therefore, it's time to end this post with a "To be continued" message, and
with a bonus section on how to restore the stock image below.</p>
<p><a name="stock"></a></p>
<h3>Restoring the Samsung stock ROM</h3>
<p>The stock ROM can be downloaded from
<a href="https://www.sammobile.com/samsung/galaxy-j3/firmware/SM-J320F/">sammobile.com</a>;
you can restore the original image with this commands:</p>
<div class="code"><pre class="code literal-block">unzip<span class="w"> </span>J320*.zip
tar<span class="w"> </span>xvf<span class="w"> </span>J320*.tar.md5
sudo<span class="w"> </span>heimdall<span class="w"> </span>flash<span class="w"> </span>--KERNEL<span class="w"> </span>boot.img<span class="w"> </span>--CACHE<span class="w"> </span>cache.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--HIDDEN<span class="w"> </span>hidden.img<span class="w"> </span>--l_fixnv2<span class="w"> </span>nvitem.bin<span class="w"> </span>--PARAM<span class="w"> </span>param.lfs<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--pm_sys<span class="w"> </span>PM_sharkl_arm7.bin<span class="w"> </span>--RECOVERY<span class="w"> </span>recovery.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--SBOOT<span class="w"> </span>sboot.bin<span class="w"> </span>--SBOOT2<span class="w"> </span>sboot2.bin<span class="w"> </span>--BOOT<span class="w"> </span>spl.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--l_modem<span class="w"> </span>SPRDCP.img<span class="w"> </span>--l_gdsp<span class="w"> </span>SPRDGDSP.img<span class="w"> </span>--l_ldsp<span class="w"> </span>SPRDLTEDSP.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--l_warm<span class="w"> </span>SPRDWDSP.img<span class="w"> </span>--SYSTEM<span class="w"> </span>system.img
</pre></div>
<p>If you were so crazy as to follow my steps, now running the command above is
probably the wisest thing you could do. :-)</p><p>A due premise: <strong>this is not a guide!</strong> While at the end I managed to build an
image for my phone, it doesn't even boot. So, this post should be taken just as
a collection of notes for myself, written to be able to reproduce the build and
keep track of those steps I might want to revisit later.</p>
<p>The reason why I'm publishing these notes in my blog is that I believe they can
still be useful for other people who should stumble into the same errors, and
also to give you an idea of how much fun (or lack thereof) porting is.</p>
<p>It's a very long post, but unless you are starting to port Halium or Ubports to
a new device, you can safely stop reading now.</p>
<p>I hope to write some better porting news in some future post. :-)</p>
<h2>Before we start: getting the info</h2>
<p>The first step is finding out a few informations about our device, and whether
a LineageOS port for our device exists (hadn't there been a LineageOS port, I
would have been out of luck: I certainly wouldn't have the time to do one
myself). From the Settings application we can find the device name, the model
number, the Android version and the kernel version:</p>
<ul>
<li>Device name: "Samsung Galaxy J3 (2016)"</li>
<li>Model: SM-J320F</li>
<li>Android version: 5.1.1</li>
<li>Kernel version: 3.10.65-[…]</li>
</ul>
<p>With this information we can look for a LineageOS port, either via a search
engine or directly in <a href="https://forum.xda-developers.com">the XDA forums</a>; in my
case the thread is <a href="https://forum.xda-developers.com/galaxy-j3-2016/development/rom-lineageos-14-1-samsung-galaxy-j3-t3667015">this
one</a>.
The post announcing the port should
contain links to the source code, besides some other information. We do
especially care about:</p>
<ul>
<li>the device codename: <code>j3xlte</code></li>
<li>the CM/LineageOS version: 14.1</li>
<li>links to <a href="https://github.com/djeman/android_device_samsung_j3xnlte/tree/cm-14.1">Device
Tree</a>,
<a href="https://github.com/djeman/android_kernel_samsung_sharkls/tree/cm-14.1">Kernel</a>,
<a name="local_manifest"></a><a href="https://github.com/djeman/android_device_samsung_j3xnlte/wiki/Local-manifest-CM-14.1">Local
Manifest</a>
(keep the local manifest handy, as you are going to need it soon)</li>
</ul>
<p>To be on the safe side, get a copy of everything. In case these are GitHub
repositories, forking them is enough.</p>
<h2>Getting root access on the device</h2>
<p>Another precondition for a successful port is the ability to be able to flash a
custom kernel and rootfs image. The existance of a LineageOS port is not a
guarantee that we can flash it: some devices are sold with a locked bootloader
in certain markets or when sold by certain mobile operators. So, before
spending time on the port, we should make sure that we can hack on our device
freely.</p>
<p>In order to get <code>adb</code> working, I had to enable "Developer mode", which is done
by hitting the "Build numer" item under the "Software info" page for 10 times
in a row. Then from the newly appeared "Developer options" menu I enabled "USB
debugging".</p>
<p>Being a total novice to Android phones, I had to go through several attempts
and internet searches, before being able to get root access on the device
(which I needed in order to read the <code>/etc/fstab</code> file, for example): by
default, <code>adb</code> only let me as normal user, and <code>su</code> didn't work.</p>
<p>In order to flash a rootkit on this phone, we need to use
<a href="https://gitlab.com/BenjaminDobell/Heimdall">Heimdall</a>, because on Samsung
devices this is what is used instead of <code>fastboot</code>. I just followed the
instructions in the <code>Linux/README</code> file and could easily build and install it.</p>
<p>We are going to flash <a href="https://twrp.me/">TWRP</a> and
<a href="http://supersu.com/">SuperSU</a>; a link to the former can be found in the XDA
<a href="https://forum.xda-developers.com/galaxy-j3-2016/development/rom-lineageos-14-1-samsung-galaxy-j3-t3667015">post</a>
about the LineageOS port for our device (under the "Download" button) and the
latter from its own homepage (in my case I had to download the "Recovery
flashable" <code>zip</code> file, and for some weird reason I got the 2.76 version, but I
bet that any newer version works as well). Once I got these files, I followed
these steps:</p>
<ul>
<li>Extract TWRP in my host machine: <code>cd /tmp && tar xvf
~/Downloads/sm-j320fgm_twrp_3.1.1_f2fs.tar.md5</code> (that leaves a <code>recovery.img</code>
file in my <code>/tmp</code>)</li>
<li>Upload SuperSU to the phone:</li>
</ul>
<div class="code"><pre class="code literal-block"><span class="w"> </span>adb<span class="w"> </span>push<span class="w"> </span>~/Downloads/UPDATE-SuperSU-v2.76-20160630161323.zip<span class="w"> </span>/sdcard/supersu.zip
</pre></div>
<ul>
<li>Power off the phone</li>
<li>Power it on by keeping the VOLUME DOWN and HOME keys pressed while
pressing the power button. The phone will be brought up into Odin mode.</li>
<li>Run</li>
</ul>
<div class="code"><pre class="code literal-block"><span class="w"> </span>sudo<span class="w"> </span>heimdall<span class="w"> </span>flash<span class="w"> </span>--RECOVERY<span class="w"> </span>/tmp/recovery.img
</pre></div>
<ul>
<li>
<p>Before the phone reboots, start keeping the VOLUME UP (<em>note: it's not the
same as before!</em>) and HOME keys, to make it reboot into recovery and not into
the ordinary image. If the phone boots into the ordinary image it will
overwrite the TWRP we just flashed, so, if that happens, you need to start
from scratch.</p>
</li>
<li>
<p>Find supersu (the zip file) and install it.</p>
</li>
</ul>
<p>If you reboot the phone and use <code>adb shell</code>, you'll be able to run <code>su</code> and get
root rights. At this point, I charged myself with hopes and got to the real
work.</p>
<h2>Let the Odyssey start</h2>
<p>I just followed the <a href="http://docs.halium.org/en/latest/porting/first-steps.html">"First steps"
instructions</a> step
by step, with no major surprises.</p>
<p>Once done with that, I moved to the <a href="http://docs.halium.org/en/latest/porting/get-sources.html">"Get sources"
page</a>. Here it was
not just a matter of following the instructions: assembling the manifest file
took me several iterations, so I had to come back to this step a few times.
Anyway, I followed the instructions for the <code>halium-7.1</code> version, given that my
device has a LineageOS 14.1 port. Then:</p>
<ul>
<li>I created a git branch in the <code>halium/devices/</code> tree, to keep track of my changes</li>
<li>I added the manifest <code>halium/devices/manifests/samsung_j3xlte.xml</code> (you can
see the final version in <a href="https://github.com/mardy/halium-devices/blob/samsung_j3xlte/manifests/samsung_j3xlte.xml">my
repository</a>,
but initially I was missing some important lines)</li>
<li>Run <code>JOBS=2 ./halium/devices/setup j3xlte</code></li>
<li>The makefile needed to be changed: the
<code>device/samsung/j3xlte/setup-makefiles.sh</code> script (from <a href="https://github.com/djeman/android_device_samsung_j3xnlte/tree/cm-14.1">the device
tree</a>)
was missing a <code>mkdir -p ../../../$OUTDIR</code> near the beginning.</li>
</ul>
<p>At this point I continued to the <a href="http://docs.halium.org/en/latest/porting/build-sources.html">"Build sources"
page</a>. Here starts
a long <em>trial and error</em> phase, and the longer part of this post. For the sake
of documentation, I will show you all the commands I ran, including failed
attempts (there are many of them), just to give you an idea of what problems
you might run into with your own port, and hopefully help in finding a
solution.</p>
<p>I first ran the <code>breakfast</code> command for my device:</p>
<div class="code"><pre class="code literal-block">$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist. Stop.
** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?
</pre></div>
<p>It looks like the device/samsung/sharkls-common was not checked out properly. After re-doing with force-sync, it failed with other errors:</p>
<div class="code"><pre class="code literal-block">$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist. Stop.
** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?
</pre></div>
<p>Indeed, I had nothing under <code>device/common</code>. So, I looked under
<code>halium/devices/manifests</code> to see if any other manifest was populating this
directory, and found that the <code>bq_krillin.xml</code> file had such a line. So, I
copied that line (the one adding the <code>lineageos/android_device_common</code> project)
into my manifest. After doing that, the <code>breakfast</code> command succeeded:</p>
<div class="code"><pre class="code literal-block">$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
Trying dependencies-only mode on a non-existing device tree?
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190619-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================
</pre></div>
<h3>Building the kernel</h3>
<p>According to the <a href="https://www.youtube.com/watch?v=nShXVDXM50A">livestream</a>, the
proper script to check for the kernel config is not the Mer one, but
<code>halium/halium-boot/check-kernel-config</code> (this is also documented in the
<a href="https://docs.ubports.com/en/latest/porting/building-halium-boot.html#edit-kernel-config">Ubports - Building Halium boot documentation
page</a>).</p>
<p>At this point I realized that I didn't have a kernel: an entry starting with
<code>kernel/</code> should always be in the manifest file. So I added the kernel line
taken from the LineageOS <a href="http://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html#local_manifest">local manifest</a> file.</p>
<p>I then looked for the config file, which I eventually found in
<code>arch/arm/configs/j3xlte_defconfig</code>. So I ran</p>
<div class="code"><pre class="code literal-block">./halium/halium-boot/check-kernel-config kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig -w
</pre></div>
<p>which reported no errors. However, in the command output I noticed this:</p>
<div class="code"><pre class="code literal-block"> CONFIG_TMPFS_POSIX_ACL is already set
CONFIG_DEFAULT_SECURITY is set, but to y "selinux" y not "apparmor".
Setting CONFIG_DEFAULT_SECURITY="apparmor" correctly
sed: -e expression #1, char 60: unterminated `s' command
Setting CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
</pre></div>
<p>When I checked the config file, I noticed that AppArmor was not really enabled.
So I changed it manually.</p>
<p>Next, the fstab. I had the Ubports and Halium guides both opened in my browser,
and I went for the <a href="https://docs.ubports.com/en/latest/porting/building-halium-boot.html#fix-mounts">"Fix
mounts"</a>
step from the Ubports guide. I used <code>find</code> to locate the fstab, and found two:</p>
<ul>
<li><a href="https://github.com/djeman/android_device_samsung_sharkls-common/blob/cm-14.1/recovery/root/fstab.sc8830"><code>device/samsung/sharkls-common/recovery/root/fstab.sc8830</code></a></li>
<li><a href="https://github.com/djeman/android_device_samsung_sharkls-common/blob/cm-14.1/rootdir/etc/fstab.sc8830"><code>device/samsung/sharkls-common/rootdir/etc/fstab.sc8830</code></a></li>
</ul>
<p>The recovery file seemed already fine to me, while the rootdir one had most
entry duplicated; ones with f2fs and xattrs, the other ones with ext4. I
commented out the f2fs ones and ran <code>mka halium-boot</code>:</p>
<div class="code"><pre class="code literal-block">[...]
/mnt/Lavoro/halium/out/build-aosp_arm.ninja is missing, regenerating...
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=
TARGET_PRODUCT=aosp_arm
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
============================================
Checking build tools versions...
find: ‘device/*/generic’: No such file or directory
find: ‘device/unknown’: No such file or directory
find: ‘device/android’: No such file or directory
find: ‘device/*/generic’: No such file or directory
find: ‘device/unknown’: No such file or directory
find: ‘device/android’: No such file or directory
halium/hybris-boot/Android.mk:69: ********************* /boot appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:70: ********************* /data appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:73: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-aosp_arm.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-aosp_arm.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
#### make failed to build some targets (11 seconds) ####
</pre></div>
<p>this seems to be documented in <a href="http://docs.halium.org/en/latest/porting/common-system-build-errors.html#hybris-boot-part-and-hybris-data-part">the "Common system build errors"
page</a>:
I had to rerun <code>source build/envsetup.sh</code> and <code>breakfast j3xlte</code>. After that, the <code>mka halium-boot</code> command failed with</p>
<div class="code"><pre class="code literal-block">device/samsung/sharkls-common/ims/sec_samsung/Android.mk:25: build/core/java_library.mk: No such file or directory
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
</pre></div>
<p>which looks like a progress. The issue about the missing <code>java_library.mk</code> has been raised in the <code>#halium</code> channel; possible solutions include:</p>
<ul>
<li>symlink from <code>static_java_library.mk</code></li>
<li>edit the <code>Android.mk</code> file</li>
</ul>
<p>I went for the former; this probably needs to be fixed in Halium.</p>
<p>Reran <code>mka halium-boot</code>, got some improvement: <code>fstab</code> is found:</p>
<div class="code"><pre class="code literal-block">halium/hybris-boot/Android.mk:70: ********************* /boot appears to live on
halium/hybris-boot/Android.mk:71: ********************* /data appears to live on /dev/block/platform/sdio_emmc/by-name/userdata
halium/hybris-boot/Android.mk:74: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
</pre></div>
<p>It looks like my <code>fstab</code> was not complete, but missing the <code>/boot</code> partition.
given that none of the <code>fstab</code> I located in my source tree had a line for the
boot partition, I checked out the Android's <code>fstab</code> from the phone, via <code>adb
shell</code>. There is no <code>fstab</code> in <code>/etc</code>, but two fstab files right in the root
directory of the filesystem: <code>fstab.sc8830</code> with exactly the same contents as
the recovery fstab I found before in
<code>device/samsung/sharkls-common/recovery/root/fstab.sc8830</code>, and a
<code>fstab.goldfish</code>:</p>
<div class="code"><pre class="code literal-block">127|root@j3xlte:/ # cat /fstab.goldfish
# Android fstab file.
#<src> <mnt_point> <type> <mnt_flags and options> <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
/dev/block/mtdblock0 /system ext4 ro,noatime,barrier=1 wait
/dev/block/mtdblock1 /data ext4 noatime,nosuid,nodev,barrier=1,nomblk_io_submit wait,check
/dev/block/mtdblock2 /cache ext4 noatime,nosuid,nodev wait,check
/devices/platform/goldfish_mmc.0 auto vfat defaults voldmanaged=sdcard:auto
</pre></div>
<p>While <code>mount</code> says:</p>
<div class="code"><pre class="code literal-block">~ # mount
rootfs on / type rootfs (rw,seclabel)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
proc on /proc type proc (rw,relatime,gid=3009,hidepid=2)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,seclabel,relatime)
pstore on /sys/fs/pstore type pstore (rw,seclabel,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
/dev/block/mmcblk0p27 on /data type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p27 on /sdcard type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p24 on /cache type ext4 (rw,seclabel,relatime,data=ordered)
</pre></div>
<p>None of the above helps in locating the boot partition. I then tried to look under <code>/dev/block</code>:</p>
<div class="code"><pre class="code literal-block">root@j3xlte:/dev/block/platform/sdio_emmc/by-name # ls -l
lrwxrwxrwx root root 2019-06-26 19:13 CACHE -> /dev/block/mmcblk0p24
lrwxrwxrwx root root 2019-06-26 19:13 FOTA_SIG -> /dev/block/mmcblk0p11
lrwxrwxrwx root root 2019-06-26 19:13 HIDDEN -> /dev/block/mmcblk0p26
lrwxrwxrwx root root 2019-06-26 19:13 KERNEL -> /dev/block/mmcblk0p20
lrwxrwxrwx root root 2019-06-26 19:13 PARAM -> /dev/block/mmcblk0p16
lrwxrwxrwx root root 2019-06-26 19:13 PERSDATA -> /dev/block/mmcblk0p23
lrwxrwxrwx root root 2019-06-26 19:13 PERSISTENT -> /dev/block/mmcblk0p22
lrwxrwxrwx root root 2019-06-26 19:13 RECOVERY -> /dev/block/mmcblk0p21
lrwxrwxrwx root root 2019-06-26 19:13 RESERVED2 -> /dev/block/mmcblk0p19
lrwxrwxrwx root root 2019-06-26 19:13 SBOOT -> /dev/block/mmcblk0p1
lrwxrwxrwx root root 2019-06-26 19:13 SBOOT2 -> /dev/block/mmcblk0p2
lrwxrwxrwx root root 2019-06-26 19:13 SYSTEM -> /dev/block/mmcblk0p25
lrwxrwxrwx root root 2019-06-26 19:13 efs -> /dev/block/mmcblk0p17
lrwxrwxrwx root root 2019-06-26 19:13 l_fixnv1 -> /dev/block/mmcblk0p3
lrwxrwxrwx root root 2019-06-26 19:13 l_fixnv2 -> /dev/block/mmcblk0p4
lrwxrwxrwx root root 2019-06-26 19:13 l_gdsp -> /dev/block/mmcblk0p9
lrwxrwxrwx root root 2019-06-26 19:13 l_ldsp -> /dev/block/mmcblk0p7
lrwxrwxrwx root root 2019-06-26 19:13 l_modem -> /dev/block/mmcblk0p8
lrwxrwxrwx root root 2019-06-26 19:13 l_runtimenv1 -> /dev/block/mmcblk0p12
lrwxrwxrwx root root 2019-06-26 19:13 l_runtimenv2 -> /dev/block/mmcblk0p13
lrwxrwxrwx root root 2019-06-26 19:13 l_warm -> /dev/block/mmcblk0p10
lrwxrwxrwx root root 2019-06-26 19:13 pm_sys -> /dev/block/mmcblk0p5
lrwxrwxrwx root root 2019-06-26 19:13 prodnv -> /dev/block/mmcblk0p18
lrwxrwxrwx root root 2019-06-26 19:13 rsvdfixnv1 -> /dev/block/mmcblk0p6
lrwxrwxrwx root root 2019-06-26 19:13 td_runtimenv1 -> /dev/block/mmcblk0p14
lrwxrwxrwx root root 2019-06-26 19:13 td_runtimenv2 -> /dev/block/mmcblk0p15
lrwxrwxrwx root root 2019-06-26 19:13 userdata -> /dev/block/mmcblk0p27
</pre></div>
<p>This, combined from the <a href="https://www.droidviews.com/need-know-samsung-pit-files/">PIT
file</a> which we can
extract with Heimdall from the "Odin mode", gives us a better understanding of
the device partitions:</p>
<div class="code"><pre class="code literal-block">$ sudo heimdall print-pit --no-reboot
Heimdall v1.4.2
Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/
This software is provided free of charge. Copying and redistribution is
encouraged.
If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/
Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...
Initialising protocol...
Protocol initialisation successful.
Beginning session...
Some devices may take up to 2 minutes to respond.
Please be patient!
Session begun.
Downloading device's PIT file...
PIT file download successful.
Entry Count: 31
Unknown 1: 1598902083
Unknown 2: 844251476
Unknown 3: 20563
Unknown 4: 17490
Unknown 5: 14136
Unknown 6: 13619
Unknown 7: 0
Unknown 8: 0
--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT
Flash Filename: spl.img
FOTA Filename:
--- Entry #1 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 90
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT2
Flash Filename: spl2.img
FOTA Filename:
--- Entry #2 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 70
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 1024
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PIT
Flash Filename: J3XLTE.pit
FOTA Filename:
--- Entry #3 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 71
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 2048
Partition Block Count: 6144
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: MD5HDR
Flash Filename: md5.img
FOTA Filename:
--- Entry #4 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 1
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 8192
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT
Flash Filename: sboot.bin
FOTA Filename:
--- Entry #5 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 2
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 12288
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT2
Flash Filename: sboot2.bin
FOTA Filename:
--- Entry #6 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 3
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 16384
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv1
Flash Filename: nvitem1.bin
FOTA Filename:
--- Entry #7 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 4
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 18432
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv2
Flash Filename: nvitem.bin
FOTA Filename:
--- Entry #8 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 5
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 20480
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: pm_sys
Flash Filename: PM_sharkl_arm7.bin
FOTA Filename:
--- Entry #9 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 6
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 22528
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: rsvdfixnv1
Flash Filename:
FOTA Filename:
--- Entry #10 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 7
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 24576
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_ldsp
Flash Filename: SPRDLTEDSP.img
FOTA Filename:
--- Entry #11 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 8
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 32768
Partition Block Count: 32768
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_modem
Flash Filename: SPRDCP.img
FOTA Filename:
--- Entry #12 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 9
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 65536
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_gdsp
Flash Filename: SPRDGDSP.img
FOTA Filename:
--- Entry #13 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 10
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 73728
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_warm
Flash Filename: SPRDWDSP.img
FOTA Filename:
--- Entry #14 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 11
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 81920
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: FOTA_SIG
Flash Filename:
FOTA Filename:
--- Entry #15 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 12
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 83968
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv1
Flash Filename:
FOTA Filename:
--- Entry #16 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 13
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 86016
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv2
Flash Filename:
FOTA Filename:
--- Entry #17 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 14
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 88064
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv1
Flash Filename:
FOTA Filename:
--- Entry #18 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 15
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 90112
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv2
Flash Filename:
FOTA Filename:
--- Entry #19 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 16
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 92160
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PARAM
Flash Filename: param.lfs
FOTA Filename:
--- Entry #20 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 17
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 96256
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: efs
Flash Filename: efs.img
FOTA Filename:
--- Entry #21 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 18
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 137216
Partition Block Count: 10240
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: prodnv
Flash Filename: prodnv.img
FOTA Filename:
--- Entry #22 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 19
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 147456
Partition Block Count: 12288
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RESERVED2
Flash Filename:
FOTA Filename:
--- Entry #23 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 20
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 159744
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: KERNEL
Flash Filename: boot.img
FOTA Filename:
--- Entry #24 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 21
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 200704
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RECOVERY
Flash Filename: recovery.img
FOTA Filename:
--- Entry #25 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 22
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 241664
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSISTENT
Flash Filename:
FOTA Filename:
--- Entry #26 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 23
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 243712
Partition Block Count: 18432
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSDATA
Flash Filename: persdata.img
FOTA Filename:
--- Entry #27 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 24
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 262144
Partition Block Count: 409600
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: CACHE
Flash Filename: cache.img
FOTA Filename:
--- Entry #28 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 25
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 671744
Partition Block Count: 4194304
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SYSTEM
Flash Filename: system.img
FOTA Filename:
--- Entry #29 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 26
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4866048
Partition Block Count: 81920
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: HIDDEN
Flash Filename: hidden.img
FOTA Filename:
--- Entry #30 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 27
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4947968
Partition Block Count: 0
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: userdata
Flash Filename: userdata.img
FOTA Filename: remained
Ending session...
Releasing device interface...
</pre></div>
<p>Looking at the name, size and flash filename of the partitions, we can
guess that the <code>BOOT</code> and <code>BOOT2</code> are about splash screens, and what we need to
use as our boot partition is in fact the <code>KERNEL</code> partition:
<code>/dev/block/mmcblk0p20</code> (note how the <code>Identifier</code> field from the PIT file
matches what we got from block devices in the <code>/dev</code> filesystem). So, we edit
our <code>fstab</code> file and add a line</p>
<div class="code"><pre class="code literal-block">/dev/block/platform/sdio_emmc/by-name/KERNEL /boot ext4 ro wait
</pre></div>
<p>From the IRC #halium channel, I understood that the <a href="http://docs.halium.org/en/latest/porting/build-sources.html#include-your-device-in-fixup-mountpoints">fixup mountpoints
step</a>
is needed. The block I added to the <code>halium/hybris-boot/fixup-mountpoints</code> script is this:</p>
<div class="code"><pre class="code literal-block"><span class="w"> </span><span class="s2">"j3xlte"</span><span class="o">)</span>
<span class="w"> </span>sed<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/SYSTEM mmcblk0p25 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/userdata mmcblk0p27 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/CACHE mmcblk0p24 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/efs mmcblk0p17 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/prodnv mmcblk0p18 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s block/platform/sdio_emmc/by-name/KERNEL mmcblk0p20 '</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
<span class="w"> </span><span class="p">;;</span>
</pre></div>
<p>I then ran <code>mka halium-boot</code> again, and it finally started to build the kernel.
But it soon stopped:</p>
<div class="code"><pre class="code literal-block">/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c: In function 'subsys_cgroup_allow_attach':
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2138:37: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
if (current != task && cred->euid != tcred->uid &&
^
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2139:18: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
cred->euid != tcred->suid)
^
</pre></div>
<p>Luckily, this is one of the <a href="https://docs.halium.org/en/latest/porting/common-kernel-build-errors.html">known build
errors</a>,
so I didn't waste a lot of time on it: setting <code>CONFIG_USER_NS=n</code> in
<code>kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig</code> fixed it. Then the build
failed with</p>
<div class="code"><pre class="code literal-block"> LD init/built-in.o
mm/built-in.o: In function `set_AKSM_level':
/mnt/Lavoro/halium/kernel/samsung/sharkls/mm/ksm.c:1822: undefined reference to `get_minfree_high_value'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2
</pre></div>
<p>Looking at the code, I understood I could probably switch off this code path
with <code>CONFIG_ADAPTIVE_KSM=n</code>. After doing that, got another error:</p>
<div class="code"><pre class="code literal-block"> LD init/built-in.o
arch/arm/lib/lib.a(memcmpksm.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2
</pre></div>
<p>Disabling also <code>CONFIG_KSM_ASSEMBLY_MEMCMP</code> did the trick:</p>
<div class="code"><pre class="code literal-block"> CC /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.mod.o
LD [M] /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.ko
make[1]: Leaving directory '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ'
make: Leaving directory '/mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0'
[ 77% 7/9] Target dt image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
DTB combiner:
Input directory: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ/arch/arm/boot/dts/'
Output file: '/mnt/Lavoro/halium/out/target/product/j3xlte/dt.img'
Found file: sprd-scx35l_sharkls_j3xlte_rev03.dtb ... chipset: 9830, platform: 3, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev02.dtb ... chipset: 9830, platform: 2, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev06.dtb ... chipset: 9830, platform: 6, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev07.dtb ... chipset: 9830, platform: 7, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev04.dtb ... chipset: 9830, platform: 4, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev01.dtb ... chipset: 9830, platform: 1, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev05.dtb ... chipset: 9830, platform: 5, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev00.dtb ... chipset: 9830, platform: 0, rev: 131072
=> Found 8 unique DTB(s)
Generating master DTB... completed
Made DT image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
[100% 9/9] Install: /mnt/Lavoro/halium/out/target/product/j3xlte/halium-boot.img
make: Leaving directory '/mnt/Lavoro/halium'
#### make completed successfully (01:28 (mm:ss)) ####
</pre></div>
<h3>Building the Halium system image</h3>
<p>Continuing following the Halium documentation, the next step was building the
system image with the <code>mka systemimage</code> command. Of course, it failed:</p>
<div class="code"><pre class="code literal-block">make: Entering directory '/mnt/Lavoro/halium'
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190702-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-52-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================
Running kati to generate build-lineage_j3xlte.ninja...
No need to regenerate ninja file
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libcurl_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstrongswan_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
make: Leaving directory '/mnt/Lavoro/halium'
#### make failed to build some targets (2 seconds) ####
</pre></div>
<p>I found similar (though not exactly the same) errors in <code>#halium</code>: the solution
is to locate the <code>.mk</code> file causing the build of the package requiring the
missing file, and remove it. I ran a <code>repo grep libstrongswan</code>, and found that
it's mentioned in <code>device/samsung/sharkls-common/sharkls.mk</code>. I removed it from
there, but I got the same build error. Removing the whole block of
<code>PRODUCT_PACKAGES</code> led to some progress:</p>
<div class="code"><pre class="code literal-block">build/core/binary.mk:1253: vendor/sprd/modules/libcamera/Android.mk: libcamisp2.0: Unused source files: isp2.0/calibration/backup isp2.0/third_lib/alc_ip
build/core/binary.mk:1253: vendor/sprd/open-source/libs/vpu/mmf/openmax/libomxil-bellagio-0.9.3/Android.mk: omxregister-bellagio: Unused source files: src/omxregister.h
No private recovery resources for TARGET_DEVICE j3xlte
build/core/Makefile:34: warning: overriding commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
build/core/base_rules.mk:316: warning: ignoring old commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbt-utils_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/engpc_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>I then removed <code>engpc</code> from the same <code>sharkls.mk</code> file:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/vold_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>The last comment from <a href="https://github.com/Halium/halium-devices/pull/116">here</a>
suggests that <code>vold</code> might not be needed (though some people in <code>#halium</code>
claimed that it could be needed, and indeed in the Meizu Pro 5 by Canonical I
do have that program under <code>/system/bin/</code>). Anyway, I edited
<code>build/target/product/base.mk</code> and removed <code>vold</code> from the <code>PRODUCT_PACKAGES</code>
variable. I also read that it's recommended to run the make command as <code>LANG=C
mka systemimage</code>, since the outcome can be affected by the user locale. Next:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libfmjni_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengfm_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>This is mentioned in <code>vendor/sprd/wcn/platform/wcnd/Android.mk</code>, so let's remove it from there (<code>LOCAL_SHARED_LIBRARIES</code> variable):</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/at_distributor', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/at_distributor', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>This is brought in by <code>device/samsung/j3xlte/proprietary-files.txt</code>, so I
commented it out from there. This didn't seem to make any difference, and a
<code>mka clean</code> didn't help either. Turns out it's also mentioned in
<code>vendor/samsung/j3xlte/j3xlte-vendor-blobs.mk</code>, so I removed it from there too
(<code>PRODUCT_COPY_FILES</code> variable). Reading the comment block on top of this
makefile we understand that it's generated by the <code>setup-makefiles.sh</code> script
based on the contents of <code>proprietary-files.txt</code>. So, one option is to re-run
the <code>setup-makefiles.sh</code> script, but I preferred to take a no risk approach and
edit both files by hand. The next failure was about <code>mfgloader</code>, which is also
mentioned in the same file:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/mfgloader', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/mfgloader', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>Let's try to remove it as well, from the same two files:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/sprdSleepLog', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/sprdSleepLog', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>Next is <code>modemd</code> (exactly same logs). At this point I started suspecting that
this was not the right way to proceed, that that instead I was just missing all
the proprietary blobs. The <a href="https://github.com/djeman/android_device_samsung_j3xnlte/tree/cm-14.1">device
tree</a>
repository has an <code>extract-files.sh</code> which seems to get these binaries from a
device. After looking at its source, it looks like it must be run from the
<code>device/samsung/j3xlte</code> directory with the device connected and reachable via
<code>adb</code>. Before running it, I reverted my changes to the <code>proprietary-files.txt</code>
file, in order to retrieve all the needed blobs. Running the extraction tool
failed once:</p>
<div class="code"><pre class="code literal-block">./extract-files.sh
846 KB/s (92260 bytes in 0.106s)
108 KB/s (9664 bytes in 0.086s)
106 KB/s (9444 bytes in 0.086s)
279 KB/s (34092 bytes in 0.119s)
44 KB/s (1992 bytes in 0.043s)
115 KB/s (5296 bytes in 0.044s)
2 KB/s (265 bytes in 0.087s)
40 KB/s (3770 bytes in 0.090s)
25 KB/s (2247 bytes in 0.087s)
4 KB/s (427 bytes in 0.087s)
0 KB/s (70 bytes in 0.087s)
remote object '/system/bin/IPSecService' does not exist
</pre></div>
<p>but after removing the missing file from <code>proprietary-files.txt</code>, the
extraction completed successfully. However, one file was still not retrieved,
as this line from the output showed:</p>
<div class="code"><pre class="code literal-block">[...]
4779 KB/s (1386252 bytes in 0.283s)
405 KB/s (37928 bytes in 0.091s)
failed to copy '/system/vendor/firmware/vbc_eq' to '../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq': Permission denied
3015 KB/s (415875 bytes in 0.134s)
[...]
</pre></div>
<p>I checked the situation by connecting to the device with <code>adb shell</code>, and the
file was there but readable by root only. After copying the file to
<code>/data/user</code> and calling <code>chmod 0666 vbc_eq</code>, I could manually retrieve it with</p>
<div class="code"><pre class="code literal-block">adb<span class="w"> </span>pull<span class="w"> </span>/data/user/vbc_eq<span class="w"> </span>../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq
</pre></div>
<p>I preferred not to touch the permissions directly on the original file, not to
risk messing up with the system; and after retrieving the file, I removed my
copy from the device. The I ran</p>
<div class="code"><pre class="code literal-block"><span class="nv">JOBS</span><span class="o">=</span><span class="m">2</span><span class="w"> </span>./halium/devices/setup<span class="w"> </span>j3xlte
</pre></div>
<p>once again. That completed fine, and the next <code>mka systemimage</code> failed with
another error:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/init_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>I played with the
<a href="https://github.com/Halium/android_system_core">android_system_core</a>
repository, which is checked out in <code>system/core/</code>, to remove references to
<code>bootloader_message</code> from <code>init/builtins.cpp</code>, and I also removed it from the
<code>LOCAL_STATIC_LIBRARIES</code> variable in the <code>system/core/init/Android.mk</code> file.
Next:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libminui_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/healthd_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>This is listed in <code>build/target/product/core_base.mk</code> and
<code>build/target/product/embedded.mk</code> and needs to be removed from both (it was
<a href="https://github.com/Halium/android_system_core/commit/eab7224aeafc07f2892f2af6fc7c3cc4859a4edb#diff-4b4390397dd367143db924ef288a5d21">disabled in Halium's
init</a>,
so it's definitely not needed). It then failed with:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'bootable/recovery-twrp/etc/init.rc', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/ramdisk-recovery.cpio', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>The <code>recovery-twrp</code> project is listed in <code>device/samsung/sharkls-common/lineage.dependencies</code>, so let's try adding a line</p>
<div class="code"><pre class="code literal-block"><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"bootable/recovery-twrp"</span><span class="w"> </span><span class="na">name=</span><span class="s">"omnirom/android_bootable_recovery"</span><span class="w"> </span><span class="na">remote=</span><span class="s">"github"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"android-7.1"</span><span class="w"> </span><span class="nt">/></span>
</pre></div>
<p>to our manifest file and run <code>JOBS=2 ./halium/devices/setup j3xlte</code> once more. After that, <code>mka systemimage</code> gives another error:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/host/linux-x86/bin/imgdiff', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
</pre></div>
<p>Adding <code>bootable/recovery-twrp</code> to the <code>subdirs</code> variable in
<code>build/core/main.mk</code> seems to get us further. But before showing you the next
error, I need to stop for a second, because at this point I started suspecting
that the previous error about the <code>bootloader_message</code> library could go away
now that I started using the same recovery-twrp tree used by the LineageOS
porter. So I reverted my changes about the <code>bootloader_message</code> library and
indeed it looked like the previous error was gone forever. And it might be that
some of the other stuff I disabled above might be restored back; but I decided
to leave it for later (it's documented in this blog post, after all, so I can
go back to it at any time) and proceed with the build:</p>
<div class="code"><pre class="code literal-block">Starting build with ninja
ninja: Entering directory `.'
[ 0% 125/16364] host C++: libadb <= system/core/adb/usb_linux.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I system/core/adb -I /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates -I /mnt/Lavoro/halium/out/host/linux-x86/gen/STATIC_LIBRARIES/libadb_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem tools/include -isystem /mnt/Lavoro/halium/out/host/linux-x86/obj/include -c -fno-exceptions -Wno-multichar -m64 -Wa,--noexecstack -fPIC -no-canonical-prefixes -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -O2 -g -fno-strict-aliasing -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 -fstack-protector-strong --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -Bprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin -target x86_64-linux-gnu -Wsign-promo -Wno-inconsistent-missing-override --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8 -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/x86_64-linux -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/backward -target x86_64-linux-gnu -Wall -Wextra -Werror -Wno-unused-parameter -Wno-missing-field-initializers -Wvla -DADB_REVISION='\"80811505f1ff-android\"' -fvisibility=hidden -DADB_HOST=1 -fPIC -std=c++14 -Wexit-time-destructors -D_USING_LIBCXX -std=gnu++14 -nostdinc++ -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d -o /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.o system/core/adb/usb_linux.cpp ) && (cp /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d >> /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; rm -f /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d )"
In file included from system/core/adb/usb_linux.cpp:25:
In file included from device/samsung/sharkls-common/include/linux/usb/ch9.h:35:
In file included from device/samsung/sharkls-common/include/uapi/linux/usb/ch9.h:36:
In file included from device/samsung/sharkls-common/include/linux/types.h:5:
In file included from device/samsung/sharkls-common/include/uapi/linux/types.h:13:
In file included from prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/posix_types.h:4:
device/samsung/sharkls-common/include/linux/stddef.h:11:2: error: expected identifier
false = 0,
^
prebuilts/clang/host/linux-x86/clang-2690385/bin/../lib64/clang/3.8.256229/include/stdbool.h:38:15: note: expanded from macro 'false'
#define false false
^
</pre></div>
<p>(see <a href="https://pastebin.com/PBwKbDFX">here</a> for the full log). After some
resultless internet search and asking around, I tried to look into the issue
myself, and noticed that in that failed command there's a mix of headers coming
from the userspace and from the kernel (which usually are not used together).
Since we are building userspace programs now, it seems a bit strange that there
are also kernel headers being used (those under
<code>device/samsung/sharkls-common</code>). Looking at <code>system/core/adb/usb_linux.cpp</code>, I
saw that it contains a line <code>#include <linux/usb/ch9.h></code> which causes the
inclusion of <code>device/samsung/sharkls-common/include/linux/usb/ch9.h</code>; but
maybe there are other <code>ch9.h</code> files in the build tree, provided by userspace
libraries and not by the kernel? Indeed, a <code>find</code> showed that I had also a
<code>./prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/usb/ch9.h</code>
file, which should probably be used instead of the kernel's one. So I copied
the failing command line (starting from what comes after <code>FAILED:</code>) into a
terminal and ran it, and could reproduce the issue; but if after removing the
first include directive (<code>-I device/samsung/sharkls-common/include</code>) the
command succeeded.</p>
<p>The line is added in <code>build/core/binary.mk</code>:</p>
<div class="code"><pre class="code literal-block">my_c_includes := $(TOPDIR)$(TARGET_SPECIFIC_HEADER_PATH) $(my_c_includes)
</pre></div>
<p>(I found it out by modifying the file, and adding random text strings around
the right-hand parts of these assignments, then running the build again and see
which random strings were surrounding the
<code>device/samsung/sharkls-common/include</code> text in the failing command line). So,
we need to find why <code>TARGET_SPECIFIC_HEADER_PATH</code> is set. I ran a <code>repo grep
TARGET_SPECIFIC_HEADER_PATH</code> and found that it's being set in
<code>device/samsung/sharkls-common/BoardConfigCommon.mk</code>. Browsing the various
directory trees, and double-checking the manifest file, I realized that the
<code>device/samsung/sharkls-common</code> should not contain the kernel source tree, yet
in my local disk it did (and all the files were untracked by git). It was
probably the result of one of my previous attempts of compiling the manifest
file, where some projects were checked out locally but I didn't clean them up
after updating the manifest. I therefore ran a <code>git clean -dfx</code> in that
directory, and the kernel source were gone. The next build continued for a
while, then failed:</p>
<div class="code"><pre class="code literal-block">[ 76% 12454/16364] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:10: fatal error: 'bt_types.h' file not found
#include "bt_types.h"
^
1 error generated.
</pre></div>
<p>Indeed the <code>bt_types.h</code> was nowhere to be found in my source tree. A search in GitHub reveals that this file is provided by <code>libncf-nci</code>, so it's probably related to NFC. The local manifest from LineageOS indeed had a few lines with NFC-related projects (at least, judging from their name):</p>
<div class="code"><pre class="code literal-block"><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/nxp-nfc/opensource/Nfc"</span><span class="w"> </span><span class="na">name=</span><span class="s">"LineageOS/android_vendor_nxp-nfc_opensource_Nfc"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/nxp-nfc/opensource/frameworks"</span><span class="w"> </span><span class="na">name=</span><span class="s">"LineageOS/android_vendor_nxp-nfc_opensource_frameworks"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"vendor/nxp-nfc/opensource/libnfc-nci"</span><span class="w"> </span><span class="na">name=</span><span class="s">"LineageOS/android_vendor_nxp-nfc_opensource_libnfc-nci"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
</pre></div>
<p>I initially excluded them from my manifest, given that my device doesn't have
NFC, but at this point I thought I'd better include them, just for the sake of
building the binaries (the alternative being removing the binaries required
these libraries, or hacking on them to remove the dependency; this is a
decision I might want to revisit later). So I added them to my manifest file
(adding the proper <code>remote</code> attribute!), re-ran the setup script which fetched
the newly-added repositories, but the build still failed with the same error,
as the include lines were not updated. However, adding
<code>vendor/nxp-nfc/opensource/libnfc-nci/src/include</code> to the
<code>TARGET_SPECIFIC_HEADER_PATH</code> variable in
<code>device/samsung/sharkls-common/BoardConfigCommon.mk</code> caused the build to fail
even earlier. I then tried to add it to the local project being built
(<code>engmode</code>), by changing the <code>LOCAL_C_INCLUDES</code> variable in
<code>vendor/sprd/open-source/apps/engmode/bt/Android.mk</code> and that brought me just a
little further:</p>
<div class="code"><pre class="code literal-block">[ 97% 1507/1552] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
In file included from vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:
vendor/nxp-nfc/opensource/libnfc-nci/src/include/bt_types.h:22:10: fatal error: 'data_types.h' file not found
#include "data_types.h"
^
1 error generated.
</pre></div>
<p>(note that in order to speed up the development I was no longer running <code>mka
systemimage</code>, but <code>mka libengbt</code>, which is the library affected by the
failure). I had three candidates:</p>
<div class="code"><pre class="code literal-block">vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/src/gki/ulinux/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/halimpl/bcm2079x/gki/ulinux/data_types.h
</pre></div>
<p>they didn't seem to be very different, so I picked the first one. I got then
another failure:</p>
<div class="code"><pre class="code literal-block">[ 35% 12/34] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/ -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:31:10: fatal error: 'btm_api.h' file not found
#include "btm_api.h"
^
1 error generated.
</pre></div>
<p>I search in the internet revealed that this is part of Bluedroid, which I
assume is the Android bluetooth stack; so I looked in the Halium manifest
directory for a repo line:</p>
<div class="code"><pre class="code literal-block">$ git grep blue
motorola_potter.xml: <project path="external/bluetooth/bluedroid" name="android_external_bluetooth_bluedroid" remote="los" revision="cm-12.1" />
</pre></div>
<p>This one seemed promising, so I added it to my manifest. I ran the setup script
again, and had a look at the newly downloaded
<code>external/bluetooth/bluedroid/stack/include/</code> directory: that not only included
the <code>btm_api.h</code> file, but also the <code>bt_types.h</code> which we were needing before.
So I went back and tried removing the two include lines related to the
<code>nxp-nfc</code> projects and replaced them with this one from bluedroid. I got the
same issue about the missing <code>data_types.h</code> file, so I added back the
<code>vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc</code> line. I then got an
error about a missing <code>tb_target.h</code> file, which is found in
<code>external/bluetooth/bluedroid/include/</code>, so I added this line as well. This let
me build <code>libengbt</code> successfully. The <code>systemimage</code> target instead failed
later:</p>
<div class="code"><pre class="code literal-block">[ 12% 502/3912] target thumb C++: audio.primary.sc8830 <= vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I external/tinyalsa/include -I external/expat/lib -I system/media/audio_utils/include -I system/media/audio_effects/include -I vendor/sprd/open-source/apps/engmode -I vendor/sprd/modules/audio/normal/vb_pga -I vendor/sprd/modules/audio/normal/record_process -I vendor/sprd/modules/audio/normal/include -I vendor/sprd/modules/audio/normal/DumpData -I vendor/sprd/modules/audio/normal/audiotest -I vendor/sprd/modules/audio/normal/record_nr -I vendor/sprd/modules/audio/normal/skd -I vendor/sprd/modules/audio/normal/resample_api -I vendor/sprd/modules/audio/normal/custom_mmi -I vendor/sprd/modules/resampler -I vendor/sprd/modules/audio/normal/libaudioril -I vendor/sprd/open-source/libs/libatci -I vendor/sprd/modules/audio/normal/vb_effect/v2 -I vendor/sprd/modules/audio/normal -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -fvisibility-inlines-hidden -Wsign-promo -Wno-inconsistent-missing-override -nostdlibinc -target arm-linux-androideabi -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -fno-rtti -D_POSIX_SOURCE -Wno-multichar -g -DAUDIO_SPIPE_TD -D_LPA_IRAM -DVOIP_DSP_PROCESS -DAUDIO_HAL_ANDROID_N_API -DFM_VERSION_IS_GOOGLE -fPIC -D_USING_LIBCXX -std=gnu++14 -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.o vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d )"
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:21:9: warning: 'LOG_TAG' macro redefined [-Wmacro-redefined]
#define LOG_TAG "SPRD_AUDIOTRACK"
^
system/core/include/log/log.h:65:9: note: previous definition is here
#define LOG_TAG NULL
^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:186:39: warning: unused parameter 'user' [-Wunused-parameter]
void playerCallback( int event, void* user, void *info )
^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:245:9: error: use of undeclared identifier 'AUDIO_DEVICE_OUT_FM_HEADSET'; did you mean 'AUDIO_DEVICE_OUT_WIRED_HEADSET'?
AUDIO_DEVICE_OUT_FM_HEADSET,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
AUDIO_DEVICE_OUT_WIRED_HEADSET
system/media/audio/include/system/audio.h:711:5: note: 'AUDIO_DEVICE_OUT_WIRED_HEADSET' declared here
AUDIO_DEVICE_OUT_WIRED_HEADSET = 0x4,
^
</pre></div>
<p>This is an issue reported
<a href="https://github.com/djeman/android_vendor_sprd/issues/1">here</a>, The link
mentioned there is for CyanogenMod 13, but I found one for LineageOS 14.1
<a href="https://github.com/djeman/android_device_samsung_sharkls-common/tree/cm-14.1/patches">here</a>.
So I ran</p>
<div class="code"><pre class="code literal-block">cd device/samsung/sharkls-common/patches/
./apply_sprd-diff.sh
</pre></div>
<p>This failed on a couple of patches, for which I reported an issue
<a href="https://github.com/djeman/android_device_samsung_sharkls-common/issues/1">here</a>,
though it's likely not important for us (as it mainly regards Java code). The
build proceeded:</p>
<div class="code"><pre class="code literal-block">[ 10% 436/4011] target thumb C++: libstagefright <= frameworks/av/media/libstagefright/CameraSource.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I ./frameworks/av/include/media/ -I ./frameworks/av/media/libavextensions -I ./frameworks/av/media/libstagefright/mpeg2ts -I ./frameworks/av/include/media/stagefright/timedtext -I ./frameworks/native/include/media/hardware -I ./frameworks/native/include/media/openmax -I ./external/flac/include -I ./external/tremolo -I ./external/libvpx/libwebm -I ./system/netd/include -I system/media/audio_utils/include -I hardware/qcom/media/sc8830/mm-core/inc -I ./external/stagefright-plugins/include -I frameworks/av/media/libstagefright -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libstagefright_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -fvisibility-inlines-hidden -Wsign-promo -Wno-inconsistent-missing-override -nostdlibinc -target arm-linux-androideabi -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -fno-rtti -Wno-multichar -Werror -Wno-error=deprecated-declarations -Wall -DENABLE_STAGEFRIGHT_EXPERIMENTS -DUSE_SPRD_COLORFORMAT -fPIC -D_USING_LIBCXX -fsanitize=unsigned-integer-overflow,signed-integer-overflow -fsanitize-trap=all -ftrap-function=abort -std=gnu++14 -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.o frameworks/av/media/libstagefright/CameraSource.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d )"
frameworks/av/media/libstagefright/CameraSource.cpp:134:16: error: use of undeclared identifier 'OMX_SPRD_COLOR_FormatYVU420SemiPlanar'; did you mean 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar'?
return OMX_SPRD_COLOR_FormatYVU420SemiPlanar;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OMX_QCOM_COLOR_FormatYVU420SemiPlanar
./frameworks/native/include/media/openmax/OMX_IVCommon.h:169:5: note: 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar' declared here
OMX_QCOM_COLOR_FormatYVU420SemiPlanar = 0x7FA30C00,
^
1 error generated.
</pre></div>
<p>I realized that there were more patches to be applied, and that the script stopped as soon as it stumbled into a failing patch. So, I took all the failing patches out of the way:</p>
<div class="code"><pre class="code literal-block">mv<span class="w"> </span>sprd-diff/frameworks_base.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/frameworks_opt.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/packages_apps.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/packages_services.diff<span class="o">{</span>,.skip<span class="o">}</span>
mv<span class="w"> </span>sprd-diff/system_bt.diff<span class="o">{</span>,.skip<span class="o">}</span><span class="w"> </span><span class="c1"># This looks potentially worth a revisit: should we have a system/bt dir?</span>
</pre></div>
<p>The next build failed here:</p>
<div class="code"><pre class="code literal-block">[ 81% 3506/4283] target thumb C: libbt-vendor <= vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I vendor/sprd/wcn/bt/libbt/include -I system/bt/hci/include -I vendor/sprd/wcn/bt/libbt/conf/sprd/marlin/include -I vendor/sprd/wcn/bt/libbt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libbt-vendor_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc -target arm-linux-androideabi -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -std=gnu99 -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -fPIC -D_USING_LIBCXX -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.o vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d )"
In file included from vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c:31:
vendor/sprd/wcn/bt/libbt/include/bt_vendor_sprd.h:33:10: fatal error: 'bt_vendor_lib.h' file not found
#include "bt_vendor_lib.h"
^
1 error generated.
</pre></div>
<p>I had this file in <code>external/bluetooth/bluedroid/hci/include/bt_vendor_lib.h</code>, and I was about to add this directory to the <code>vendor/sprd/wcn/bt/libbt/Android.mk</code> file when I realized that this file was already including <code>$(BDROID_DIR)/hci/include</code> and defined <code>BDROID_DIR := $(TOP_DIR)system/bt</code>. So I understood that I should have changed my manifest file to check out the bluedroid repo under <code>system/bt</code> (which also meant that I was wrong in excluding the <code>sprd-diff/system_bt.diff</code> patch before). I also had a quick look at other manifest files, and I saw that they were typically getting <code>system/bt</code> from the LineageOS <code>android_system_bt</code> repo. Therefore I went to modify my manifest file, changing the bluedroid line to</p>
<div class="code"><pre class="code literal-block"><span class="nt"><project</span><span class="w"> </span><span class="na">path=</span><span class="s">"system/bt"</span><span class="w"> </span><span class="na">name=</span><span class="s">"android_system_bt"</span><span class="w"> </span><span class="na">remote=</span><span class="s">"los"</span><span class="w"> </span><span class="na">revision=</span><span class="s">"cm-14.1"</span><span class="w"> </span><span class="nt">/></span>
</pre></div>
<p>and removed the three <code>vendor/nxp-nfc</code> repositories I added a while before.
Then I manually applied the patch in
<code>device/samsung/sharkls-common/patches/sprd-diff/system_bt.diff</code> and I
revisited the <code>vendor/sprd/open-source/apps/engmode/bt/Android.mk</code> to remove
all the include lines I added before. The next build succeeded:</p>
<div class="code"><pre class="code literal-block">[...]
[ 99% 2250/2255] build /mnt/Lavoro/halium/out/target/product/j3xlte/obj/NOTICE.html
Combining NOTICE files into HTML
Combining NOTICE files into text
[ 99% 2254/2255] Target system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
BuildImage: in_dir = /mnt/Lavoro/halium/out/target/product/j3xlte/system, out_file = /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
fs type is not ext4
Running: mkuserimg.sh -s /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img ext4 system 2147483648 -D /mnt/Lavoro/halium/out/target/product/j3xlte/system -L system /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin
make_ext4fs -s -T -1 -S /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin -L system -l 2147483648 -a system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/system
Creating filesystem with parameters:
Size: 2147483648
Block size: 4096
Blocks per group: 32768
Inodes per group: 8192
Inode size: 256
Journal blocks: 8192
Label: system
Blocks: 524288
Block groups: 16
Reserved block group size: 127
Created filesystem with 1381/131072 inodes and 55467/524288 blocks
Running ['mkuserimg.sh', '-s', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img', 'ext4', 'system', '2147483648', '-D', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '-L', 'system', '/mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin'] command, exit code = 0
[ 99% 2254/2255] Construct recovery from boot
failed to reconstruct target deflate chunk 1 [(null)]; treating as normal
chunk 0: type 0 start 0 len 5582858
chunk 1: type 2 start 5582858 len 8611584
chunk 2: type 0 start 9205682 len 673886
Construct patches for 3 chunks...
patch 0 is 212 bytes (of 5582858)
patch 1 is 4311951 bytes (of 3622824)
patch 2 is 207 bytes (of 673886)
chunk 0: normal ( 0, 5582858) 212
chunk 1: deflate ( 5582858, 8574717) 4311951 (null)
chunk 2: normal ( 14157575, 674057) 207
[100% 2255/2255] Install system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/system.img
/mnt/Lavoro/halium/out/target/product/j3xlte/system.img+/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p maxsize=2192424960 blocksize=135168 total=197470210 reserve=22167552
make: Leaving directory '/mnt/Lavoro/halium'
#### make completed successfully (01:22 (mm:ss)) ####
</pre></div>
<p>Just to be on the safe side (to make sure I didn't have stale files from previous build attempts), I ran </p>
<div class="code"><pre class="code literal-block">mka<span class="w"> </span>clean
mka<span class="w"> </span>halium-boot
<span class="nv">LANG</span><span class="o">=</span>C<span class="w"> </span>mka<span class="w"> </span>systemimage
</pre></div>
<p>again, without encountering any errors. I also ran <code>mka hybris-boot</code> as per the
Halium documentation, as it might be a good idea to test the Halium reference
image first. <code>hybris-boot.img</code> was built successfully at the first try.</p>
<h3>Flashing and testing the Halium reference image</h3>
<p>Following the Halium documentation, I downloaded the Halium rootfs image:</p>
<div class="code"><pre class="code literal-block">mkdir<span class="w"> </span>rootfs
<span class="nb">cd</span><span class="w"> </span>rootfs
wget<span class="w"> </span><span class="s1">'http://bshah.in/halium/halium-rootfs-20170630-151006.tar.gz'</span>
</pre></div>
<p>A small section of the documentation is dedicated on <a href="http://docs.halium.org/en/latest/porting/install-build/reference-rootfs.html#install-hybris-boot-img-on-samsung-devices">how to install the boot
image for Samsung
devices</a>;
it closely matches the steps I took before to install TWRP, which is
comforting.</p>
<div class="code"><pre class="code literal-block">adb<span class="w"> </span>reboot<span class="w"> </span>download<span class="w"> </span><span class="c1"># much easier than the weird key combinations!</span>
sudo<span class="w"> </span>heimdall<span class="w"> </span>flash<span class="w"> </span>--BOOT<span class="w"> </span>out/target/product/j3xlte/hybris-boot.img
</pre></div>
<p>The last command ended with this output:</p>
<div class="code"><pre class="code literal-block">Uploading BOOT
100%
BOOT upload successful
Ending session...
ERROR: Failed to receive session end confirmation!
Releasing device interface...
</pre></div>
<p>and the device was stuck on the same screen. I then thought of using TWRP to
flash this device. I rebooted in download mode and installed TWRP as explained
near the beginning of this blog post, then I uploaded my boot image to the SD
card:</p>
<div class="code"><pre class="code literal-block">adb<span class="w"> </span>push<span class="w"> </span>out/target/product/j3xlte/hybris-boot.img<span class="w"> </span>/sdcard/
</pre></div>
<p>In TWRP, I did a wipe first: I wiped off the "System", "Data" and "cache"
partitions (you can choose which partitions to wipe under the "Advanced wipe"
option). Then I went back and pressed the "Install" button, then "Install
Image" and picked my <code>hybris-boot.img</code> file (and chose "Boot" as target, of
course). Make sure you don't reboot at this stage, or you'll lose <code>adb</code> access
and the rest of the commands will fail.</p>
<p>I then installed the <a href="https://gitlab.com/JBBgameich/halium-install/">halium-install</a> script:</p>
<div class="code"><pre class="code literal-block">git<span class="w"> </span>clone<span class="w"> </span>https://gitlab.com/JBBgameich/halium-install.git
</pre></div>
<p>I then downloaded the original Samsung ROM (see <a href="http://mardy.it/ia/blog/2019/07/notes-on-porting-the-samsung-j3-to-halium-%2B-ubports.html#stock">below</a> for the link)
and unzipped it in the <code>rootfs</code> directory I created before. That produced a
file named <code>J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5</code>, from which
I extracted the <code>system.img</code> and uploaded it to the device along with the
halium rootfs, as explained in the "Halium guide":</p>
<div class="code"><pre class="code literal-block">tar<span class="w"> </span>xvf<span class="w"> </span>J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5<span class="w"> </span>system.img
../halium-install/halium-install<span class="w"> </span>-p<span class="w"> </span>halium<span class="w"> </span>-v<span class="w"> </span>halium-rootfs-20170630-151006.tar.gz<span class="w"> </span>system.img
</pre></div>
<p>The last command complained about <code>simg2img</code> not being in <code>$PATH</code>, despite the
fact that it was. A <a href="https://gitlab.com/JBBgameich/halium-install/merge_requests/28/diffs">small
change</a>
to the script fixed this. Once that was fixed it successfully pushed the
files to the device, after asking me which password I wanted to set for the
root user (on the phone): I chose a familiar "0000".</p>
<p>After rebooting, however, the device didn't go beyond the boot screen (with a
red line warning about SELinux not being enabled). It just got stuck there, and
was not even detected via USB.</p>
<p>Therefore, it's time to end this post with a "To be continued" message, and
with a bonus section on how to restore the stock image below.</p>
<p><a name="stock"></a></p>
<h3>Restoring the Samsung stock ROM</h3>
<p>The stock ROM can be downloaded from
<a href="https://www.sammobile.com/samsung/galaxy-j3/firmware/SM-J320F/">sammobile.com</a>;
you can restore the original image with this commands:</p>
<div class="code"><pre class="code literal-block">unzip<span class="w"> </span>J320*.zip
tar<span class="w"> </span>xvf<span class="w"> </span>J320*.tar.md5
sudo<span class="w"> </span>heimdall<span class="w"> </span>flash<span class="w"> </span>--KERNEL<span class="w"> </span>boot.img<span class="w"> </span>--CACHE<span class="w"> </span>cache.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--HIDDEN<span class="w"> </span>hidden.img<span class="w"> </span>--l_fixnv2<span class="w"> </span>nvitem.bin<span class="w"> </span>--PARAM<span class="w"> </span>param.lfs<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--pm_sys<span class="w"> </span>PM_sharkl_arm7.bin<span class="w"> </span>--RECOVERY<span class="w"> </span>recovery.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--SBOOT<span class="w"> </span>sboot.bin<span class="w"> </span>--SBOOT2<span class="w"> </span>sboot2.bin<span class="w"> </span>--BOOT<span class="w"> </span>spl.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--l_modem<span class="w"> </span>SPRDCP.img<span class="w"> </span>--l_gdsp<span class="w"> </span>SPRDGDSP.img<span class="w"> </span>--l_ldsp<span class="w"> </span>SPRDLTEDSP.img<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--l_warm<span class="w"> </span>SPRDWDSP.img<span class="w"> </span>--SYSTEM<span class="w"> </span>system.img
</pre></div>
<p>If you were so crazy as to follow my steps, now running the command above is
probably the wisest thing you could do. :-)</p>WIP: changing the backend for contacts in Ubportshttp://mardy.it/ia/blog/2019/06/wip-changing-the-backend-for-contacts-in-ubports.html2019-06-13T17:59:35+03:002019-06-13T17:59:35+03:00Alberto Mardegan<p>More than one year has passed since the <a href="https://list.ubports.com/pipermail/ubports-dev/2018-March/000005.html">initial
announcement</a>
of my plan to investigate using a different backend for contact storage. If you
want to get a better understanding of the plan, that mail is still a good read
-- not much has changed since them, planning wise.</p>
<p>The reason for this blog post is to give a small update on what has happened
since then, and as a start nothing can be better than a couple of screenshots:</p>
<div>
<figure style="float:left; margin-left: 3em; width: 340px">
<img src="http://mardy.it/archivos/imagines/blog/ubports-carddav-accounts.png" class="shadow-sm">
<figcaption>Adding CardDAV accounts in the Addressbook application</figcaption>
</figure>
<figure style="float:left; margin-left: 3em; width: 340px">
<img src="http://mardy.it/archivos/imagines/blog/ubports-carddav-contact.png" class="shadow-sm">
<figcaption>Aggregated contact details from multiple sources</figcaption>
</figure>
<div style="clear:both"></div>
</div>
<p>In other words, that means that contact synchonisation works, both with the new
CardDAV protocol (for which we'll have preconfigured setups for NextCloud and
OwnCloud accounts) and with Google Contacts, for which we are now using a
different engine. What you see in the second screenshot (although indeed it's
not obvious at all) is that the new qtcontacts-sqlite backend performs
automatic contact merging based on some simple heuristics, meaning that when
synchonising the same contact from multiple sources you should not happen to
find a multitude of semi-identical copies of the contact, but a single one
having all the aggregated details.</p>
<p>Before you get too excited, I have to say that this code is pre-alpha quality
and that it's not even available for testing yet. The next step is indeed to
setup CI so that the packaggqes get automatically built and published to a public
repository, at which point I'll probably issue another update here in my blog.</p>
<h3>The boring stuff</h3>
<p>And now some detail for those who might wonder why this feature is not ready
yet, or would like to get an idea on the time-frame for its completion.</p>
<p>Apart from a chronical lack of time from my part, the feature complexity is due
to the large number of components involved:</p>
<ul>
<li><a href="https://git.sailfishos.org/mer-core/qtcontacts-sqlite">qtcontacts-sqlite</a>:
the QtContacts backend we are migrating to. This is a backend for the
QtContacts API (used by our Addressbook application) which uses a SQLite
database as storage for your contacts.</li>
<li><a href="https://git.sailfishos.org/mer-core/buteo-sync-plugin-carddav">buteo-sync-plugin-carddav</a>:
the CardDAV plugin for Buteo (our synchronisation manager). This plugin is
loaded by Buteo and synchronises the contacts between a CardDAV remote source
and the <code>qtcontacts-sqlite</code> database.</li>
<li><a href="https://git.sailfishos.org/mer-core/buteo-sync-plugins-social">buteo-sync-plugins-social</a>:
a Buteo plugin which can synchronise contacts from a multitude of sources,
including Google, Facebook and Vk. At the moment we only care about Google,
but once this feature has landed we can easily extend it to work with the
other two as well.</li>
<li><a href="https://github.com/ubports/address-book-app">address-book-app</a>: this is our
well-known Contacts application. It needs some minor changes to adapt to the
<code>qtcontacts-sqlite</code> backend and to support the creation of new CardDAV,
NextCloud and OwnCloud accounts.</li>
<li><a href="https://github.com/ubports/qtpim-opensource-src-packaging">QtPim</a>: the
contacts and calendar API developed by the Qt project. Our Contacts
application is using the front-end side of this API, and the
<code>qtcontacts-sqlite</code> component implements the backend side. There are some
improvements proposed by Jolla, which we need to include in order to support
grouping contacts by their initials.</li>
</ul>
<p>The other tricky aspect is that the first three projects are maintained by
Jolla as part of the <a href="https://sailfishos.org/">Sailfish OS</a>, and while on one
side this means that we can share the development and maintenance burden with
Jolla, on the other side of the coin it means that we need to apply extra care
when submitting changes, in order not to step on each other's shoes.
Specifically, Sailfish OS is using a much older version of <code>QtPim</code> than Ubports
is, and the APIs between the two versions have changes in an incompatible
version, so that it's nearly impossible to have a single code base working with
both versions of <code>QtPim</code>. Luckily <code>git</code> supports branches, and Chris from Jolla
was kind enough to create a branch for us in their upstream repository where
I've proposed our changes (<a href="https://git.sailfishos.org/mer-core/qtcontacts-sqlite/merge_requests/30">and they are a
lot!</a>).</p>
<p>However, this is not as bad as it sounds, and the fact that I have a roughly
working version on my development device is a good sign that things are moving
forwards.</p><p>More than one year has passed since the <a href="https://list.ubports.com/pipermail/ubports-dev/2018-March/000005.html">initial
announcement</a>
of my plan to investigate using a different backend for contact storage. If you
want to get a better understanding of the plan, that mail is still a good read
-- not much has changed since them, planning wise.</p>
<p>The reason for this blog post is to give a small update on what has happened
since then, and as a start nothing can be better than a couple of screenshots:</p>
<div>
<figure style="float:left; margin-left: 3em; width: 340px">
<img src="http://mardy.it/archivos/imagines/blog/ubports-carddav-accounts.png" class="shadow-sm">
<figcaption>Adding CardDAV accounts in the Addressbook application</figcaption>
</figure>
<figure style="float:left; margin-left: 3em; width: 340px">
<img src="http://mardy.it/archivos/imagines/blog/ubports-carddav-contact.png" class="shadow-sm">
<figcaption>Aggregated contact details from multiple sources</figcaption>
</figure>
<div style="clear:both"></div>
</div>
<p>In other words, that means that contact synchonisation works, both with the new
CardDAV protocol (for which we'll have preconfigured setups for NextCloud and
OwnCloud accounts) and with Google Contacts, for which we are now using a
different engine. What you see in the second screenshot (although indeed it's
not obvious at all) is that the new qtcontacts-sqlite backend performs
automatic contact merging based on some simple heuristics, meaning that when
synchonising the same contact from multiple sources you should not happen to
find a multitude of semi-identical copies of the contact, but a single one
having all the aggregated details.</p>
<p>Before you get too excited, I have to say that this code is pre-alpha quality
and that it's not even available for testing yet. The next step is indeed to
setup CI so that the packaggqes get automatically built and published to a public
repository, at which point I'll probably issue another update here in my blog.</p>
<h3>The boring stuff</h3>
<p>And now some detail for those who might wonder why this feature is not ready
yet, or would like to get an idea on the time-frame for its completion.</p>
<p>Apart from a chronical lack of time from my part, the feature complexity is due
to the large number of components involved:</p>
<ul>
<li><a href="https://git.sailfishos.org/mer-core/qtcontacts-sqlite">qtcontacts-sqlite</a>:
the QtContacts backend we are migrating to. This is a backend for the
QtContacts API (used by our Addressbook application) which uses a SQLite
database as storage for your contacts.</li>
<li><a href="https://git.sailfishos.org/mer-core/buteo-sync-plugin-carddav">buteo-sync-plugin-carddav</a>:
the CardDAV plugin for Buteo (our synchronisation manager). This plugin is
loaded by Buteo and synchronises the contacts between a CardDAV remote source
and the <code>qtcontacts-sqlite</code> database.</li>
<li><a href="https://git.sailfishos.org/mer-core/buteo-sync-plugins-social">buteo-sync-plugins-social</a>:
a Buteo plugin which can synchronise contacts from a multitude of sources,
including Google, Facebook and Vk. At the moment we only care about Google,
but once this feature has landed we can easily extend it to work with the
other two as well.</li>
<li><a href="https://github.com/ubports/address-book-app">address-book-app</a>: this is our
well-known Contacts application. It needs some minor changes to adapt to the
<code>qtcontacts-sqlite</code> backend and to support the creation of new CardDAV,
NextCloud and OwnCloud accounts.</li>
<li><a href="https://github.com/ubports/qtpim-opensource-src-packaging">QtPim</a>: the
contacts and calendar API developed by the Qt project. Our Contacts
application is using the front-end side of this API, and the
<code>qtcontacts-sqlite</code> component implements the backend side. There are some
improvements proposed by Jolla, which we need to include in order to support
grouping contacts by their initials.</li>
</ul>
<p>The other tricky aspect is that the first three projects are maintained by
Jolla as part of the <a href="https://sailfishos.org/">Sailfish OS</a>, and while on one
side this means that we can share the development and maintenance burden with
Jolla, on the other side of the coin it means that we need to apply extra care
when submitting changes, in order not to step on each other's shoes.
Specifically, Sailfish OS is using a much older version of <code>QtPim</code> than Ubports
is, and the APIs between the two versions have changes in an incompatible
version, so that it's nearly impossible to have a single code base working with
both versions of <code>QtPim</code>. Luckily <code>git</code> supports branches, and Chris from Jolla
was kind enough to create a branch for us in their upstream repository where
I've proposed our changes (<a href="https://git.sailfishos.org/mer-core/qtcontacts-sqlite/merge_requests/30">and they are a
lot!</a>).</p>
<p>However, this is not as bad as it sounds, and the fact that I have a roughly
working version on my development device is a good sign that things are moving
forwards.</p>Ubports at the LinuxPiter conferencehttp://mardy.it/ia/blog/2019/03/ubports-at-the-linuxpiter-conference.html2019-03-13T19:07:23+03:002019-03-13T19:07:23+03:00Alberto Mardegan<p>Last November I was invited to talk at the <a class="reference external" href="https://linuxpiter.com/">LinuxPiter</a> conference. I held a presentation of the <a class="reference external" href="https://ubports.com/">Ubports
project</a>, to which I still contribute in my little
spare time.</p>
<p>The video recording from the conference has finally been published:</p>
<div class="youtube-video align-center">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/_gWTiVPahT0?rel=0&wmode=transparent" frameborder="0" allow="encrypted-media" allowfullscreen></iframe>
</div><p>(there's also a <a class="reference external" href="https://www.youtube.com/watch?v=pYiqOzZZg-w">version in Russian</a>)</p>
<p>There was not a big audience, to be honest, but those that were there expressed
a lot of interest in the project.</p><p>Last November I was invited to talk at the <a class="reference external" href="https://linuxpiter.com/">LinuxPiter</a> conference. I held a presentation of the <a class="reference external" href="https://ubports.com/">Ubports
project</a>, to which I still contribute in my little
spare time.</p>
<p>The video recording from the conference has finally been published:</p>
<div class="youtube-video align-center">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/_gWTiVPahT0?rel=0&wmode=transparent" frameborder="0" allow="encrypted-media" allowfullscreen></iframe>
</div><p>(there's also a <a class="reference external" href="https://www.youtube.com/watch?v=pYiqOzZZg-w">version in Russian</a>)</p>
<p>There was not a big audience, to be honest, but those that were there expressed
a lot of interest in the project.</p>