Skip to main content

Bmaptool - the forgotten joker

Comments

Source

Tags:

gnome-terminal-logo.png

The Yocto project has become a de-facto standard for Embedded GNU/Linux Systems and on many embedded systems the final image will be written to an SD card to be tested on the target board. Those images are meant to be written in one go and thus contain a partition table and multiple file systems. In preparing them, one usually chooses a suitable maximum size for the file system partitions and this will be the lower limit of the final image size no matter how much of this space is actually used. So without further tooling we will always be writing a lot of zeros not relevant for the system itself to the target medium.

Luckily enough there exists tooling for this use case and in this blog post I will show how easy it is to use bmaptool from the bmap-tools project to save time and wear on the SD card. On Debian systems installing the package is as easy as sudo apt install bmap-tools.

NXP Yocto

The i.MX Arm SoCs from NXP have their own i.MX Yocto Project Users Guide as Yocto is the delivery vehicle of the NXP BSPs. Chapter 6 "Image Deployment" tells us explicitly how to write an image to an SD card: 

$ bzcat <image_name>.wic.bz2 | sudo dd of=/dev/sd<partition> bs=1M conv=fsync

The problem with this approach is that it will be pretty inefficient as it will copy the image in its total size derived from maximum file system sizes. It will be much more efficient to only write the data required for the image. Luckily, Yocto gives us all that we need: 

dzu@buster:/media/dzu/hd01-ext4/nxp/imx-yocto-bsp/build-imx8mmevk-wayland$ ls tmp/deploy/images/imx8mmevk/*wic*
tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk-20220222110908.rootfs.wic.bmap
tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk-20220222110908.rootfs.wic.bz2
tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk.wic.bmap
tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk.wic.bz2
dzu@buster:/media/dzu/hd01-ext4/nxp/imx-yocto-bsp/build-imx8mmevk-wayland$

The ".wic.bz2" file is the compressed image that we want to write to the SD card and next to it we also see an ".wic.bmap" file required for bmaptool to do its magic, so let's take a look at tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk-20220222110908.rootfs.wic.bmap: 

<?xml version="1.0" ?>
<!-- This file contains the block map for an image file, which is basically
     a list of useful (mapped) block numbers in the image file. In other words,
     it lists only those blocks which contain data (boot sector, partition
     table, file-system metadata, files, directories, extents, etc). These
     blocks have to be copied to the target device. The other blocks do not
     contain any useful data and do not have to be copied to the target
     device.

     The block map an optimization which allows to copy or flash the image to
     the image quicker than copying of flashing the entire image. This is
     because with bmap less data is copied: <MappedBlocksCount> blocks instead
     of <BlocksCount> blocks.

     Besides the machine-readable data, this file contains useful commentaries
     which contain human-readable information like image size, percentage of
     mapped data, etc.

     The 'version' attribute is the block map file format version in the
     'major.minor' format. The version major number is increased whenever an
     incompatible block map format change is made. The minor number changes
     in case of minor backward-compatible changes. -->

<bmap version="2.0">
    <!-- Image size in bytes: 3.0 GiB -->
    <ImageSize> 3192814592 </ImageSize>

    <!-- Size of a block in bytes -->
    <BlockSize> 4096 </BlockSize>

    <!-- Count of blocks in the image file -->
    <BlocksCount> 779496 </BlocksCount>

    <!-- Count of mapped blocks: 1.8 GiB or 59.9%    -->
    <MappedBlocksCount> 466986 </MappedBlocksCount>

    <!-- Type of checksum used in this file -->
    <ChecksumType> sha256 </ChecksumType>

    <!-- The checksum of this bmap file. When it is calculated, the value of
         the checksum has be zero (all ASCII "0" symbols).  -->
    <BmapFileChecksum> c9f1ffc037706f24c282c8007fdb837e9956ac191817634930fcee4fbe0ce510 </BmapFileChecksum>

    <!-- The block map which consists of elements which may either be a
         range of blocks or a single block. The 'chksum' attribute
         (if present) is the checksum of this blocks range. -->
    <BlockMap>
        <Range chksum="5ce3cde21dcde76677e2c03fb9999be8df7b82e9fbb048c0f0dde3de8fc27cd7"> 0-2 </Range>
        <Range chksum="8ebc2bbc2e53639ecef923ac05730cd0170fec95c9b1fdd41fa4ca0c64e8adca"> 8-55 </Range>
        <Range chksum="79aa355184c96211f95818db87e645f0677027a2dd78bc080336e49f19029099"> 95-483 </Range>
        <Range chksum="1a5dbf3afe823cecbff877b44277a5acb6c1f63fbc878e3dfb8a106ffd668cb1"> 2048-9877 </Range>
        <Range chksum="b2b0cf74de7af9f4d3721a20e67a1a92fe8d1da6806afdea2e619a5e2af22dd2"> 24576-24960 </Range>
        <Range chksum="622b6c8b15eed657aa63c4a258c6aad2988a901cd725cc4b5cf5d026ff31175f"> 24962-24964 </Range>
        <Range chksum="5ae456faf38ff6a4d6b670a7b68ae0422e2c10f6dc1dfa99e3cdb3fbced668a5"> 24978-27658 </Range>
        <Range chksum="8efc9e13c546453f3093a34638f1fa67b232f046b1a32e8ba786f2bf023d38e0"> 40706-57345 </Range>
        <Range chksum="658fe23cb7d0f7909ca5f8ac68756c55ca3c11f996dd87f87784d607f212e78f"> 57714-122881 </Range>
        <Range chksum="b79669f24d678e453b0a57032f7d615c7947caf89e47d2dc8752d9eca0a3e30c"> 123250-188417 </Range>
        <Range chksum="c34cf38528c933540300abcd0a9b22f3db321129938d78ae9a9fb993c5944d47"> 188786-253953 </Range>
        <Range chksum="d3a73d6cf3f4b2f031cc1fdd0b0efd3055bb75998abacd9b804298251fc30b62"> 254322-319489 </Range>
        <Range chksum="d84cdf83090e40da52cbfa4c537f29fcb7b74c75a6dd60707a4d9e777afdde60"> 319858-498166 </Range>
        <Range chksum="6bc7270def6eecf599ad36e968327179d8a4d10cfc0b30d6e82dd2214d9524a6"> 548864 </Range>
        <Range chksum="18b2bf341618df2c2ad5be7b3cd53d4700d900d5b94d8342c8aa753e320bb521"> 548871 </Range>
        <Range chksum="572a601695d8dd96558fa3e69f82d9d4396abb0a6b0e48cdfec7b1d5011e58cf"> 779472-779495 </Range>
    </BlockMap>
</bmap>

Already the contents show us that the (uncompressed) image size will be 3 GiB, but only 1.8 GiB are actually used (or mapped). So we can save around 40% using bmaptool instead of the bare uncompressed image.

Using bmaptool

The tool itself presumes that it can find bmap files by replacing the last suffix of a filename by "bmap", so in our case we can simply call bmaptool on the *.wic.bz2 file: 

dzu@krikkit:/media/dzu/hd01-ext4/nxp/imx-yocto-bsp/build-imx8mmevk-wayland$ time sudo bmaptool copy tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk.wic.bz2 /dev/sde
bmaptool: info: discovered bmap file 'tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk.wic.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 779496 blocks of size 4096 (3.0 GiB), mapped 466986 blocks (1.8 GiB or 59.9%)
bmaptool: info: copying image 'imx-image-multimedia-imx8mmevk.wic.bz2' to block device '/dev/sde' using bmap file 'imx-image-multimedia-imx8mmevk.wic.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sde'
bmaptool: info: copying time: 8m 28.0s, copying speed 3.6 MiB/sec

real	8m28,378s
user	0m0,036s
sys	0m0,081s
dzu@krikkit:/media/dzu/hd01-ext4/nxp/imx-yocto-bsp/build-imx8mmevk-wayland$

It is not visible in the transcript above, but bmaptool also provides you with a nice progress indicator. Let's compare this with the execution time of the command suggested by the NXP documentation: 

dzu@krikkit:/media/dzu/hd01-ext4/nxp/imx-yocto-bsp/build-imx8mmevk-wayland$ time ( bzcat tmp/deploy/images/imx8mmevk/imx-image-multimedia-imx8mmevk.wic.bz2 | sudo dd of=/dev/sde bs=1M conv=fsync )
0+622821 Datensätze ein
0+622821 Datensätze aus
3192814592 Bytes (3,2 GB, 3,0 GiB) kopiert, 1276,08 s, 2,5 MB/s

real	21m16,094s
user	0m57,899s
sys	0m1,494s
dzu@krikkit:/media/dzu/hd01-ext4/nxp/imx-yocto-bsp/build-imx8mmevk-wayland$

So in total bmaptool saved us nearly 13 minutes in total! As a bonus we do get a progress indicator and an additional safety net as the ranges are protected by checksums. So there is really no reason at all not to use it.

Comments

Comments powered by Disqus