Testing Preempt RT on the i.MX8MM SoC in 15 minutes

NXP publishes its Linux kernel tree through a git repository on the CodeAurora site backed by the Linux Foundation. For efficiency I just add this repo as a remote repository to my Linux git repository:

dzu@krikkit:/opt/src/git/linux (master)$ git remote add qoriq-components https://source.codeaurora.org/external/qoriq/qoriq-components/linux
dzu@krikkit:/opt/src/git/linux (master)$ git remote update

With this remote repository in place, we can access the tags and branches from it. The tags we are interested in originate in the "Linux Factory" group inside NXP. The idea is that this group works on the Linux support for all of the NXP products. This merges the previously separate releases for e.g. QorIQ products and i.MX products:

dzu@krikkit:/opt/src/git/linux (master)$ git tag -l | grep lf-
lf-5.10.35-2.0.0
lf-5.10.35-rt-2.0.0
lf-5.10.52-2.1.0
lf-5.10.52-rt-2.1.0
lf-5.10.72-2.2.0
lf-5.10.72-rt-2.2.0
lf-5.10.y-1.0.0
lf-5.10.y-1.0.0-rt
lf-5.4.47-1.1.0
lf-5.4.y-1.0.0
dzu@krikkit:/opt/src/git/linux (master)$

Just like most of the SoC vendors nowadays, NXP targets the Long Term Support (LTS) Linux kernels. You can find up to date information on these kernels through the active kernel releases page on kernel.org.

Although at the time of writing (February 2022) the latest upstream LTS kernel is 5.15, NXP does not yet offer this version through the Linux Factory kernels, so we will use 5.10 for this exercise which will be supported until end of 2026. The "-rt" tags hint that starting with this version, NXP provides branches pre-patched with the corresponding Preempt RT patchset. So let's checkout the most recent tag into its own branch:

dzu@krikkit:/opt/src/git/linux (master)$ git checkout -b branch-lf-5.10.72-rt-2.2.0 lf-5.10.72-rt-2.2.0
Updating files: 100% (41134/41134), done.
Switched to a new branch 'branch-lf-5.10.72-rt-2.2.0'
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ 

To cross compile the kernel for the ARMv8 target, we need a suitable gcc cross compiler. There are different solutions to this but for the sake of simplicity, we will just use the cross compiler provided directly by Debian. Just install the gcc-aarch64-linux-gnu package and you are good to go. If you want to use another toolchain, i.e. the one produced by Yocto, this should of course also work just the same.

With this in place, we use the NXP provided i.MX kernel configuration as a starting point. In addition to patching the kernel, we also need to configure the kernel to be "fully pre-emptive" as "real-time" also has its drawbacks. By allowing the kernel to be fully pre-emptive, it will actually loose some performance for non real-time workloads. This is one of the reasons why the patch set is still not completely in the mainline Linux kernel.

So in order to enable CONFIG_PREEMPT_RT we enter the configuration GUI:

dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ export ARCH=arm64
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ export CROSS_COMPILE=aarch64-linux-gnu-
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ make imx_v8_defconfig
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ make menuconfig

In the GUI we head straight to "General Setup" and "Preemption Model". To our surprise we see just the "regular" options of non-RT enabled kernels:

So with our previous knowledge we know that we are looking for the PREEMPT_RT option (usually we leave off the "CONFIG_" prefix to save some space), but it is not shown in the interface. So even though the kernel itself is patched with the RT preempt patch set, we cannot enable the relevant kernel configuration. This tells us that the NXP team likely has not yet finalized the RT preempt support and does not yet provide a ready to use configuration. Considering that this is the first LTS version where the patch set is already included in the NXP branch this is kind of understandable. Things take time and I expect the situation to change in the next versions.

But for now we have to find out ourselves why we cannot select the option present in the sources. We could start by looking at all the relevant Kconfig files but this is a manual process and by looking at the sources we will only see an "offline" view of the data structures. For conditional constructs you have to mentally parse them and substitute the actual values for a given .config file to work out the actual truth values. It is much better to have a "live" view of the option set together with showing the conditions of options. Fortunately menuconfig can do that for us. It is contained within its "search" functionality.

So in order to make progress in our quest for a real time enabled kernel we search for the CONFIG_PREEMPT_RT symbol itself. Just type "/" followed by "PREEMPT_RT":

The "Depends on:" line tells us the option is hidden because of EXPERT currently being disabled and because of ARCH_SUPPORTS_RT also being false (n). EXPERT is an option in the same hierarchy and can easily be activated. So let's look at what is the matter with ARCH_SUPPORTS_RT:

ARM64 is enabled, so we need to dig down further on HAVE_POSIX_CPU_TIMERS_TASK_WORK:

Aha! So in this kernel this option is incompatible with the Kernel Virtual Machine (KVM) virtualization support. I would be surprised if this was a permanent situation but for Linux 5.10 on ARM64 we can either have KVM or RT preempt but not both.

So let's test our hypothesis by enabling EXPERT and disabling KVM. You can easily do this with the GUI but here is a quick hack to do it from the command line. Just amend .config with the wanted choices and on saving a configuration the next time from within menuconfig the config file will again be in a canonical format. You will get warnings on the console however when you do this - but they have not been captured in the screenshots here.

dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ make imx_v8_defconfig
#
# configuration written to .config
#
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ echo -e "CONFIG_EXPERT=y\nCONFIG_KVM=n" >> .config
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ make menuconfig

Finally, we can now select PREEMPT_RT, save the configuration and build the kernel.

dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$ make -j $(nproc) Image
arch/arm64/Makefile:33: LSE atomics not supported by binutils
arch/arm64/Makefile:44: Detected assembler with broken .inst; disassembly will be unreliable
  SYNC    include/config/auto.conf.cmd
  CC      scripts/mod/empty.o
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/modpost.o
  CC      scripts/mod/devicetable-offsets.s
  HOSTCC  scripts/mod/file2alias.o
  HOSTCC  scripts/mod/sumversion.o
  HOSTLD  scripts/mod/modpost
  CC      kernel/bounds.s
[...]
  AR      drivers/mxc/built-in.a
  AR      drivers/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  OBJCOPY arch/arm64/boot/Image
dzu@krikkit:/opt/src/git/linux (branch-lf-5.10.72-rt-2.2.0)$

To quickly test the kernel, I copied Image and imx8mm-evk.dtb into the directory served by my tftp server. As any root filesystem will work, I will use the Yocto image already contained on the SD card and only load the kernel and the dtb from my tftp server. To do this, I stop U-Boot, define a new command, set up the IP configuration for the tftp server and execute it:

u-boot=> setenv net_mmc 'tftp $fdt_addr $fdtfile; tftp $loadaddr $image; run mmcargs; booti $loadaddr - $fdt_addr'
u-boot=> setenv serverip 192.168.44.35
u-boot=> setenv ipaddr 192.168.44.119
u-boot=> saveenv
Saving Environment to MMC... Writing to MMC(1)... OK
u-boot=> run net_mmc
Using ethernet@30be0000 device
TFTP from server 192.168.44.35; our IP address is 192.168.44.119
Filename 'imx8mm-evk.dtb'.
Load address: 0x43000000
Loading: ####
         751 KiB/s
done
Bytes transferred = 46942 (b75e hex)
Using ethernet@30be0000 device
TFTP from server 192.168.44.35; our IP address is 192.168.44.119
Filename 'Image'.
Load address: 0x40480000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################

[...]

NXP i.MX Release Distro 5.10-hardknott imx8mmevk ttymxc1

imx8mmevk login: root
[   25.385526] audit: type=1006 audit(1645537297.109:4): pid=953 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=3 res=1
root@imx8mmevk:~# uname -a
Linux imx8mmevk 5.10.72-rt53-00346-g3e5f5dc7a6de #6 SMP PREEMPT_RT Sun Jan 30 15:52:04 CET 2022 aarch64 aarch64 aarch64 GNU/Linux
root@imx8mmevk:~# 

So indeed it is now possible to get an RT preempt kernel running on the i.MX8M-Mini in a minimum of time. But currently this still requires some deeper knowledge of how things work on the kernel level. As this is the first Linux LTS version with such support from NXP I expect the situation change for the better in the future. Currently I also do not see any Yocto recipe to compile such a kernel and I am sure that this will also be added sooner or later.

In order to find out if the kernel not only boots but also improves the latency behavior, we will have to run some test, but this will be the content of another post. So stay tuned!

Through the comment of XQ Zhou I was made aware that I mistakenly attributed the "lf-5.10.72-rt-2.2.0" tag to the imx/linux-imx repository on CodeAurora instead of the qoriq-components/linux repository it seems to originate from. Of course the whole idea of the "Linux Factory" is to unite those repositories but this is not quite true yet.