Build Toolchain Using Crosstool-NG

Today, we’re going to build our own toolchain using crosstool-NG. There are so many cases that we want to build a particular toolchain for a specific embedded system development board. One MOST important reason for that is probably because the specific development board has limited resource to build upon, therefore, the time-consumption to build some software on the development board is VERY inefficient. In the other way around, cross compiling on the host PC is MORE efficient with respective to the time-consumption.

In this blog, for simplicity, we take Raspberry Pi 3B as our demo development board, for which we are building the cross compiler. Some references can be found at:

1. Installation

How to install crosstool-NG are thoroughly summarized at its official website. In my case, I had it installed under folder /opt/compilers/crosstool-ng. Let’s take a look:

1
2
3
4
5
6
longervision-GT72-6QE% pwd
/opt/compilers/crosstool-ng
longervision-GT72-6QE% ls
bin libexec share
longervision-GT72-6QE% ls bin
ct-ng

And make sure /opt/compilers/crosstool-ng/bin is under environment variable PATH.

2. Configuration

Under any directory that you want to save your .config file, we can configure our target cross compiler.

1
longervision-GT72-6QE% ct-ng menuconfig

crosstool-NG menuconfig

According to elinux: RPi Linaro GCC Compilation, we need to do the following selections:

  • Paths and misc options:

    • Try features marked as EXPERIMENTAL: ticked
    • Prefix directory: input the full path where you want to save the built toolchains
    • Number of parallel jobs: 4. Sorry that I’ve NO idea if this is the number of cores on Raspberry Pi 3B, but Raspberry Pi 3B does have 4 cores.
  • Target options:

    • Target Architecture: arm
    • Default instruction set mode: arm
    • Use EABI: ticked
    • append hf to the tuple (EXPERIMENTAL): ticked
    • Endianness: Little endian
    • Bitness: 32-bit
    • Emit assembly for CPU: cortex-a53
    • Use specific FPU: vfp
    • Floating point: hardware (FPU)
  • Toolchain options:

    • Tuple’s vendor string: rpi
  • Operating System

    😝
    1
    2
    ➜  ~ uname -a
    Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux
  • Binary utilities:

    • Binary format: ELF
    • Version of binutils: 2.31.1
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    ➜  ~ apt show binutils
    Package: binutils
    Version: 2.31.1-16+rpi1
    Priority: optional
    Section: devel
    Maintainer: Matthias Klose <doko@debian.org>
    Installed-Size: 95.2 kB
    Provides: binutils-gold, elf-binutils
    Depends: binutils-common (= 2.31.1-16+rpi1), libbinutils (= 2.31.1-16+rpi1), binutils-arm-linux-gnueabihf (= 2.31.1-16+rpi1)
    Suggests: binutils-doc (>= 2.31.1-16+rpi1)
    Conflicts: binutils-multiarch (<< 2.27-8), modutils (<< 2.4.19-1)
    Homepage: https://www.gnu.org/software/binutils/
    Download-Size: 56.9 kB
    APT-Manual-Installed: no
    APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
    Description: GNU assembler, linker and binary utilities
    The programs in this package are used to assemble, link and manipulate
    binary and object files. They may be used in conjunction with a compiler
    and various libraries to build programs.
  • C-library:

    • C library: glibc
    • Version of glibc: 2.28
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    ➜  ~ apt show libc-bin
    Package: libc-bin
    Version: 2.28-10+rpi1
    Priority: required
    Essential: yes
    Section: libs
    Source: glibc
    Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
    Installed-Size: 3,015 kB
    Depends: libc6 (>> 2.28), libc6 (<< 2.29)
    Recommends: manpages
    Homepage: https://www.gnu.org/software/libc/libc.html
    Download-Size: 657 kB
    APT-Manual-Installed: yes
    APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
    Description: GNU C Library: Binaries
    This package contains utility programs related to the GNU C Library.
    .
    * catchsegv: catch segmentation faults in programs
    * getconf: query system configuration variables
    * getent: get entries from administrative databases
    * iconv, iconvconfig: convert between character encodings
    * ldd, ldconfig: print/configure shared library dependencies
    * locale, localedef: show/generate locale definitions
    * tzselect, zdump, zic: select/dump/compile time zones
  • C compiler:

    • Show gcc versions from: GNU
    • Version of gcc: 8.3.0
    • gcc extra config: --with-float=hard
    • Link libstdc++ statically into the gcc binary: tick
    • C++: tick
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    ➜  ~ apt show gcc
    Package: gcc
    Version: 4:8.3.0-1+rpi2
    Priority: optional
    Section: devel
    Source: gcc-defaults (1.181+rpi2)
    Maintainer: Debian GCC Maintainers <debian-gcc@lists.debian.org>
    Installed-Size: 46.1 kB
    Provides: c-compiler, gcc-arm-linux-gnueabihf (= 4:8.3.0-1+rpi2)
    Depends: cpp (= 4:8.3.0-1+rpi2), gcc-8 (>= 8.3.0-1~)
    Recommends: libc6-dev | libc-dev
    Suggests: gcc-multilib, make, manpages-dev, autoconf, automake, libtool, flex, bison, gdb, gcc-doc
    Conflicts: gcc-doc (<< 1:2.95.3)
    Download-Size: 5,200 B
    APT-Manual-Installed: no
    APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
    Description: GNU C compiler
    This is the GNU C compiler, a fairly portable optimizing compiler for C.
    .
    This is a dependency package providing the default GNU C compiler.

After we save the configuration to file .config, we Exit the crosstool-NG configuration dialog.

3. Build

Please remember to:

  • unset LD_LIBRARY_PATH before building. Otherwise, you’ll meet some ERROR messages.
  • mkdir ~/src before building. Otherwise, whenever you tried to rerun ct-ng build, you’ll have to download ALL required packages from scratch.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
longervision-GT72-6QE% ct-ng build
[INFO ] Performing some trivial sanity checks
[WARN ] Number of open files 1024 may not be sufficient to build the toolchain; increasing to 2048
[INFO ] Build started 20190906.014128
[INFO ] Building environment variables
[EXTRA] Preparing working directories
[EXTRA] Installing user-supplied crosstool-NG configuration
[EXTRA] =================================================================
[EXTRA] Dumping internal crosstool-NG configuration
[EXTRA] Building a toolchain for:
[EXTRA] build = x86_64-pc-linux-gnu
[EXTRA] host = x86_64-pc-linux-gnu
[EXTRA] target = arm-rpi-linux-gnueabihf
[EXTRA] Dumping internal crosstool-NG configuration: done in 0.07s (at 00:01)
[INFO ] =================================================================
[INFO ] Retrieving needed toolchain components' tarballs
[EXTRA] Retrieving 'ncurses-6.1'
[EXTRA] Verifying SHA512 checksum for 'ncurses-6.1.tar.gz'
[EXTRA] Saving 'ncurses-6.1.tar.gz' to local storage
[EXTRA] Retrieving 'libiconv-1.15'
[EXTRA] Verifying SHA512 checksum for 'libiconv-1.15.tar.gz'
[EXTRA] Saving 'libiconv-1.15.tar.gz' to local storage
[EXTRA] Retrieving 'gettext-0.19.8.1'
[EXTRA] Verifying SHA512 checksum for 'gettext-0.19.8.1.tar.xz'
[EXTRA] Saving 'gettext-0.19.8.1.tar.xz' to local storage
[EXTRA] Retrieving 'binutils-2.31.1'
[EXTRA] Verifying SHA512 checksum for 'binutils-2.31.1.tar.xz'
[EXTRA] Saving 'binutils-2.31.1.tar.xz' to local storage
[EXTRA] Retrieving 'gcc-8.3.0'
[EXTRA] Verifying SHA512 checksum for 'gcc-8.3.0.tar.xz'
[EXTRA] Saving 'gcc-8.3.0.tar.xz' to local storage
[EXTRA] Retrieving 'glibc-2.28'
[EXTRA] Verifying SHA512 checksum for 'glibc-2.28.tar.xz'
[EXTRA] Saving 'glibc-2.28.tar.xz' to local storage
[INFO ] Retrieving needed toolchain components' tarballs: done in 130.35s (at 02:12)
[INFO ] =================================================================
[INFO ] Extracting and patching toolchain components
[EXTRA] Extracting linux-4.20.8
[EXTRA] Patching linux-4.20.8
[EXTRA] Extracting zlib-1.2.11
[EXTRA] Patching zlib-1.2.11
[EXTRA] Extracting gmp-6.1.2
[EXTRA] Patching gmp-6.1.2
[EXTRA] Extracting mpfr-4.0.2
[EXTRA] Patching mpfr-4.0.2
[EXTRA] Extracting isl-0.20
[EXTRA] Patching isl-0.20
[EXTRA] Extracting mpc-1.1.0
[EXTRA] Patching mpc-1.1.0
[EXTRA] Extracting ncurses-6.1
[EXTRA] Patching ncurses-6.1
[EXTRA] Extracting libiconv-1.15
[EXTRA] Patching libiconv-1.15
[EXTRA] Extracting gettext-0.19.8.1
[EXTRA] Patching gettext-0.19.8.1
[EXTRA] Extracting binutils-2.31.1
[EXTRA] Patching binutils-2.31.1
[EXTRA] Extracting gcc-8.3.0
[EXTRA] Patching gcc-8.3.0
[EXTRA] Extracting glibc-2.28
[EXTRA] Patching glibc-2.28
[INFO ] Extracting and patching toolchain components: done in 33.39s (at 02:45)
[INFO ] =================================================================
[INFO ] Installing ncurses for build
[EXTRA] Configuring ncurses
[EXTRA] Building ncurses
[EXTRA] Installing ncurses
[INFO ] Installing ncurses for build: done in 14.62s (at 03:00)
[INFO ] =================================================================
[INFO ] Installing zlib for host
[EXTRA] Configuring zlib
[EXTRA] Building zlib
[EXTRA] Installing zlib
[INFO ] Installing zlib for host: done in 1.12s (at 03:01)
[INFO ] =================================================================
[INFO ] Installing GMP for host
[EXTRA] Configuring GMP
[EXTRA] Building GMP
[EXTRA] Installing GMP
[INFO ] Installing GMP for host: done in 30.12s (at 03:31)
[INFO ] =================================================================
[INFO ] Installing MPFR for host
[EXTRA] Configuring MPFR
[EXTRA] Building MPFR
[EXTRA] Installing MPFR
[INFO ] Installing MPFR for host: done in 17.35s (at 03:48)
[INFO ] =================================================================
[INFO ] Installing ISL for host
[EXTRA] Configuring ISL
[EXTRA] Building ISL
[EXTRA] Installing ISL
[INFO ] Installing ISL for host: done in 17.73s (at 04:06)
[INFO ] =================================================================
[INFO ] Installing MPC for host
[EXTRA] Configuring MPC
[EXTRA] Building MPC
[EXTRA] Installing MPC
[INFO ] Installing MPC for host: done in 5.47s (at 04:12)
[INFO ] =================================================================
[INFO ] Installing ncurses for host
[EXTRA] Configuring ncurses
[EXTRA] Building ncurses
[EXTRA] Installing ncurses
[INFO ] Installing ncurses for host: done in 13.64s (at 04:25)
[INFO ] =================================================================
[INFO ] Installing libiconv for host
[EXTRA] Skipping (included in GNU C library)
[INFO ] Installing libiconv for host: done in 0.01s (at 04:25)
[INFO ] =================================================================
[INFO ] Installing gettext for host
[EXTRA] Skipping (included in GNU C library)
[INFO ] Installing gettext for host: done in 0.01s (at 04:25)
[INFO ] =================================================================
[INFO ] Installing binutils for host
[EXTRA] Configuring binutils
[EXTRA] Building binutils
[EXTRA] Installing binutils
[INFO ] Installing binutils for host: done in 46.89s (at 05:12)
[INFO ] =================================================================
[INFO ] Installing pass-1 core C gcc compiler
[EXTRA] Configuring core C gcc compiler
[EXTRA] Building gcc
[EXTRA] Installing gcc
[EXTRA] Housekeeping for core gcc compiler
[EXTRA] '' --> lib (gcc) lib (os)
[INFO ] Installing pass-1 core C gcc compiler: done in 285.36s (at 09:58)
[INFO ] =================================================================
[INFO ] Installing kernel headers
[EXTRA] Installing kernel headers
[EXTRA] Checking installed headers
[INFO ] Installing kernel headers: done in 6.47s (at 10:04)
[INFO ] =================================================================
[INFO ] Installing C library headers & start files
[INFO ] =================================================================
[INFO ] Building for multilib 1/1: ''
[EXTRA] Configuring C library
[EXTRA] Installing C library headers
[EXTRA] Installing C library start files
[INFO ] Building for multilib 1/1: '': done in 7.65s (at 10:12)
[INFO ] Installing C library headers & start files: done in 7.72s (at 10:12)
[INFO ] =================================================================
[INFO ] Installing pass-2 core C gcc compiler
[EXTRA] Configuring core C gcc compiler
[EXTRA] Building gcc
[EXTRA] Installing gcc
[EXTRA] Housekeeping for core gcc compiler
[EXTRA] '' --> lib (gcc) lib (os)
[INFO ] Installing pass-2 core C gcc compiler: done in 354.87s (at 16:07)
[INFO ] =================================================================
[INFO ] Installing C library
[INFO ] =================================================================
[INFO ] Building for multilib 1/1: ''
[EXTRA] Cleaning up start files
[EXTRA] Configuring C library
[EXTRA] Building C library
[EXTRA] Configuring C library
[EXTRA] Building C library
[EXTRA] Installing C library
[INFO ] Building for multilib 1/1: '': done in 184.69s (at 19:12)
[INFO ] Installing C library: done in 184.78s (at 19:12)
[INFO ] =================================================================
[INFO ] Installing final gcc compiler
[EXTRA] Configuring final gcc compiler
[EXTRA] Building final gcc compiler
[EXTRA] Installing final gcc compiler
[EXTRA] Housekeeping for final gcc compiler
[EXTRA] '' --> lib (gcc) lib (os)
[INFO ] Installing final gcc compiler: done in 339.13s (at 24:51)
[INFO ] =================================================================
[INFO ] Finalizing the toolchain's directory
[INFO ] Stripping all toolchain executables
[EXTRA] Installing the populate helper
[EXTRA] Installing a cross-ldd helper
[EXTRA] Creating toolchain aliases
[EXTRA] Removing installed documentation
[EXTRA] Collect license information from: ~/Crosscompile/.build/arm-rpi-linux-gnueabihf/src
[EXTRA] Put the license information to: ~/....../RPi/share/licenses
[INFO ] Finalizing the toolchain's directory: done in 1.80s (at 24:52)
[INFO ] Build completed at 20190906.020619
[INFO ] (elapsed: 24:51.86)
[INFO ] Finishing installation (may take a few seconds)...
[24:52] / % longervision-GT72-6QE%

This process may take a while. I’m going to sleep tonight. Continue tomorrow…

Alright, let’s continue today. Glad to know it’s built successfully.

😄

Let’s FIRST take a look at what’s under the current folder.

1
2
longervision-GT72-6QE% ls
build.log rpi.config

And then, let’s take a look at what’s built under the specified destination folder.

1
2
3
4
5
6
7
8
9
10
longervision-GT72-6QE% ls ~/....../CrossCompile/RPi
arm-rpi-linux-gnueabihf bin build.log.bz2 include lib libexec share
longervision-GT72-6QE% ls ~/....../CrossCompile/RPi/bin
arm-rpi-linux-gnueabihf-addr2line arm-rpi-linux-gnueabihf-cpp arm-rpi-linux-gnueabihf-gcc-ar arm-rpi-linux-gnueabihf-gprof arm-rpi-linux-gnueabihf-objdump arm-rpi-linux-gnueabihf-strip
arm-rpi-linux-gnueabihf-ar arm-rpi-linux-gnueabihf-ct-ng.config arm-rpi-linux-gnueabihf-gcc-nm arm-rpi-linux-gnueabihf-ld arm-rpi-linux-gnueabihf-populate
arm-rpi-linux-gnueabihf-as arm-rpi-linux-gnueabihf-elfedit arm-rpi-linux-gnueabihf-gcc-ranlib arm-rpi-linux-gnueabihf-ld.bfd arm-rpi-linux-gnueabihf-ranlib
arm-rpi-linux-gnueabihf-c++ arm-rpi-linux-gnueabihf-g++ arm-rpi-linux-gnueabihf-gcov arm-rpi-linux-gnueabihf-ldd arm-rpi-linux-gnueabihf-readelf
arm-rpi-linux-gnueabihf-cc arm-rpi-linux-gnueabihf-gcc arm-rpi-linux-gnueabihf-gcov-dump arm-rpi-linux-gnueabihf-nm arm-rpi-linux-gnueabihf-size
arm-rpi-linux-gnueabihf-c++filt arm-rpi-linux-gnueabihf-gcc-8.3.0 arm-rpi-linux-gnueabihf-gcov-tool arm-rpi-linux-gnueabihf-objcopy arm-rpi-linux-gnueabihf-strings
longervision-GT72-6QE%

Finally, let’s take a look at the version of our built cross compilers for Raspberry Pi 3B.

1
2
3
4
5
6
7
8
9
10
11
12
13
longervision-GT72-6QE% arm-rpi-linux-gnueabihf-gcc --version
zsh: command not found: arm-rpi-linux-gnueabihf-gcc
longervision-GT72-6QE% ./arm-rpi-linux-gnueabihf-gcc --version
arm-rpi-linux-gnueabihf-gcc (crosstool-NG 1.24.0.6-afaf7b9) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

longervision-GT72-6QE% ./arm-rpi-linux-gnueabihf-g++ --version
arm-rpi-linux-gnueabihf-g++ (crosstool-NG 1.24.0.6-afaf7b9) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Additional issue: It seems current crosstool-NG does NOT officially support Python. Please refer to my issue at crosstool-NG.

4. Compile/Build a Package with Generated Toolchain

For most of the packages nowadays, they are

  • either supported by make: ./configure -> make -> make install
  • or supported by CMake: mkdir build -> cd build -> ccmake ../ -> make -> make install

How to use our generated toolchain to compile/build our target packages?

Today, we’re going to take the package flann as an example, which are to be built with CMake.

4.1 CMakeLists.txt Modification

4.1.1 CMake Toolchains - Cross Compiling for Linux

By following CMake Toolchains - Cross Compiling for Linux, we FIRST modify flann CMakeLists.txt a bit by adding the following lines before project(flann).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cmake_minimum_required(VERSION 2.6)

if(COMMAND cmake_policy)
cmake_policy(SET CMP0003 NEW)
endif(COMMAND cmake_policy)

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(CMAKE_SYSROOT ....../arm-rpi-linux-gnueabihf/sysroot)
set(CMAKE_STAGING_PREFIX ....../stage)

set(tools /opt/compilers/RPi)
set(CMAKE_C_COMPILER ${tools}/bin/arm-rpi-linux-gnueabihf-gcc)
set(CMAKE_CXX_COMPILER ${tools}/bin/arm-rpi-linux-gnueabihf-g++)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

set(CMAKE_TRY_COMPILE_TARGET_TYPE "STATIC_LIBRARY")

project(flann)
string(TOLOWER ${PROJECT_NAME} PROJECT_NAME_LOWER)
  • CMAKE_SYSROOT specify the sysroot directory which emulate your target environment, here, Raspberry Pi 3B
  • CMAKE_STAGING_PREFIX is where the STAGING results store, for the reason that final results may require to be built in multiple stages. You may refer to Linux From Scratch for further background knowledge about that.
  • tools actually specify the building tool directory. Making sure all generated cross compiling tools are under folder ${tools}/bin.

4.1.2 Ignore hdf5

In addition, for the emulated Raspberry Pi 3B sysroot, hdf5 is NOT supported. Therefore, let’s simply comment out the following line in flann CMakeLists.txt.

1
#find_hdf5()

4.2 Cross Compile

Now, let’s start cross-compiling flann.

1
2
3
longervision-GT72-6QE% mkdir build
longervision-GT72-6QE% cd build
longervision-GT72-6QE% ccmake ../

and press c and then t, you’ll see the cross compiling toolchains have been automatically configured as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
BUILD_CUDA_LIB                  *OFF
BUILD_C_BINDINGS *ON
BUILD_DOC *OFF
BUILD_EXAMPLES *OFF
BUILD_MATLAB_BINDINGS *OFF
BUILD_PYTHON_BINDINGS *ON
BUILD_TESTS *OFF
CMAKE_AR */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ar
CMAKE_BUILD_TYPE *RelWithDebInfo
CMAKE_COLOR_MAKEFILE *ON
CMAKE_CXX_COMPILER_AR */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-gcc-ar
CMAKE_CXX_COMPILER_RANLIB */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-gcc-ranlib
CMAKE_CXX_FLAGS *-I/usr/include
CMAKE_CXX_FLAGS_DEBUG *-g
CMAKE_CXX_FLAGS_MINSIZEREL *-Os -DNDEBUG
CMAKE_CXX_FLAGS_RELEASE *-O3 -DNDEBUG
CMAKE_CXX_FLAGS_RELWITHDEBINFO *-O2 -g -DNDEBUG
CMAKE_C_COMPILER_AR */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-gcc-ar
CMAKE_C_COMPILER_RANLIB */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-gcc-ranlib
CMAKE_C_FLAGS *-I/usr/include
CMAKE_C_FLAGS_DEBUG *-g
CMAKE_C_FLAGS_MINSIZEREL *-Os -DNDEBUG
CMAKE_C_FLAGS_RELEASE *-O3 -DNDEBUG
CMAKE_C_FLAGS_RELWITHDEBINFO *-O2 -g -DNDEBUG
CMAKE_EXE_LINKER_FLAGS *
CMAKE_EXE_LINKER_FLAGS_DEBUG *
CMAKE_EXE_LINKER_FLAGS_MINSIZE *
CMAKE_EXE_LINKER_FLAGS_RELEASE *
CMAKE_EXE_LINKER_FLAGS_RELWITH *
CMAKE_EXPORT_COMPILE_COMMANDS *OFF
CMAKE_INSTALL_PREFIX */usr/local
CMAKE_LINKER */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ld
CMAKE_MAKE_PROGRAM */usr/bin/make
CMAKE_MODULE_LINKER_FLAGS *
CMAKE_MODULE_LINKER_FLAGS_DEBU *
CMAKE_MODULE_LINKER_FLAGS_MINS *
CMAKE_MODULE_LINKER_FLAGS_RELE *
CMAKE_MODULE_LINKER_FLAGS_RELW *
CMAKE_NM */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-nm
CMAKE_OBJCOPY */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-objcopy
CMAKE_OBJDUMP */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-objdump
CMAKE_RANLIB */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ranlib
CMAKE_SHARED_LINKER_FLAGS *
CMAKE_SHARED_LINKER_FLAGS_DEBU *
CMAKE_SHARED_LINKER_FLAGS_MINS *
CMAKE_SHARED_LINKER_FLAGS_RELE *
CMAKE_SHARED_LINKER_FLAGS_RELW *
CMAKE_SKIP_INSTALL_RPATH *OFF
CMAKE_SKIP_RPATH *OFF
CMAKE_STATIC_LINKER_FLAGS *
CMAKE_STATIC_LINKER_FLAGS_DEBU *
CMAKE_STATIC_LINKER_FLAGS_MINS *
CMAKE_STATIC_LINKER_FLAGS_RELE *
CMAKE_STATIC_LINKER_FLAGS_RELW *
CMAKE_STRIP */opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-strip
CMAKE_VERBOSE_MAKEFILE *ON
CPACK_SOURCE_RPM *OFF
CPACK_SOURCE_TBZ2 *ON
CPACK_SOURCE_TGZ *ON
CPACK_SOURCE_TXZ *ON
CPACK_SOURCE_TZ *ON
CPACK_SOURCE_ZIP *OFF
DPKG_PROGRAM */usr/bin/dpkg
NSIS_PROGRAM */usr/bin/makensis
NVCC_COMPILER_BINDIR *
PACKAGE_MAKER_PROGRAM *PACKAGE_MAKER_PROGRAM-NOTFOUND
PKG_CONFIG_EXECUTABLE */usr/bin/pkg-config
PYTHON_EXECUTABLE */usr/bin/python
RPM_PROGRAM *RPM_PROGRAM-NOTFOUND
USE_MPI *OFF
USE_OPENMP *OFF

The LAST step before make is to modify some of the parameters accordingly, in my case:

  • BUILD_PYTHON_BINDINGS: ON -> OFF
  • CMAKE_CXX_FLAGS: -I/usr/include (for lz4.h from my host Ubuntu 19.04)
  • CMAKE_C_FLAGS: -I/usr/include (for lz4.h from my host Ubuntu 19.04)
  • CMAKE_VERBOSE_MAKEFILE: OFF -> ON
  • CMAKE_INSTALL_PREFIX: /usr/local/ -> under which directory you want to install, which can be IGNORED for now

Now, press c, you’ll see:

1
2
CMake Warning at CMakeLists.txt:99 (message):
hdf5 library not found, some tests will not be run

A warining of missing hdf5 is clearly reasonable and acceptable. Then press g.

Finally, it’s the time for us to cross build flann.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
longervision-GT72-6QE% make -j8
/usr/bin/cmake -S....../flann -B....../flann/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start ....../flann/build/CMakeFiles ....../flann/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '....../flann/build'
make -f src/cpp/CMakeFiles/flann_s.dir/build.make src/cpp/CMakeFiles/flann_s.dir/depend
make -f src/cpp/CMakeFiles/flann_cpp_s.dir/build.make src/cpp/CMakeFiles/flann_cpp_s.dir/depend
make[2]: Entering directory '....../flann/build'
cd ....../flann/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" ....../flann ....../flann/src/cpp ....../flann/build ....../flann/build/src/cpp ....../flann/build/src/cpp/CMakeFiles/flann_s.dir/DependInfo.cmake --color=
make[2]: Entering directory '....../flann/build'
cd ....../flann/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" ....../flann ....../flann/src/cpp ....../flann/build ....../flann/build/src/cpp ....../flann/build/src/cpp/CMakeFiles/flann_cpp_s.dir/DependInfo.cmake --color=
Scanning dependencies of target flann_cpp_s
make[2]: Leaving directory '....../flann/build'
make -f src/cpp/CMakeFiles/flann_s.dir/build.make src/cpp/CMakeFiles/flann_s.dir/build
make[2]: Entering directory '....../flann/build'
make[2]: Leaving directory '....../flann/build'
make -f src/cpp/CMakeFiles/flann_cpp_s.dir/build.make src/cpp/CMakeFiles/flann_cpp_s.dir/build
make[2]: Entering directory '....../flann/build'
[ 12%] Building CXX object src/cpp/CMakeFiles/flann_s.dir/flann/flann.cpp.o
cd ....../flann/build/src/cpp && /opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-g++ --sysroot=....../RPi/arm-rpi-linux-gnueabihf/sysroot -DFLANN_STATIC -D_FLANN_VERSION=1.9.1 -I....../flann/src/cpp -I/usr/include -O2 -g -DNDEBUG -Wall -Wno-unknown-pragmas -Wno-unused-function -fPIC -o CMakeFiles/flann_s.dir/flann/flann.cpp.o -c ....../flann/src/cpp/flann/flann.cpp
[ 25%] Building CXX object src/cpp/CMakeFiles/flann_cpp_s.dir/flann/flann_cpp.cpp.o
cd ....../flann/build/src/cpp && /opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-g++ --sysroot=....../RPi/arm-rpi-linux-gnueabihf/sysroot -DFLANN_STATIC -DFLANN_USE_CUDA -D_FLANN_VERSION=1.9.1 -I....../flann/src/cpp -I/usr/include -O2 -g -DNDEBUG -Wall -Wno-unknown-pragmas -Wno-unused-function -fPIC -o CMakeFiles/flann_cpp_s.dir/flann/flann_cpp.cpp.o -c ....../flann/src/cpp/flann/flann_cpp.cpp
In file included from ....../flann/src/cpp/flann/util/params.h:33,
from ....../flann/src/cpp/flann/flann.hpp:42,
from ....../flann/src/cpp/flann/flann.h:603,
from ....../flann/src/cpp/flann/flann.cpp:31:
....../flann/src/cpp/flann/util/any.h: In member function 'virtual void flann::anyimpl::big_any_policy<T>::static_delete(void**)':
....../flann/src/cpp/flann/util/any.h:81:9: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
if (* x) delete (* reinterpret_cast<T**>(x)); *x = NULL;
^~
....../flann/src/cpp/flann/util/any.h:81:55: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
if (* x) delete (* reinterpret_cast<T**>(x)); *x = NULL;
^
In file included from ....../flann/src/cpp/flann/util/params.h:33,
from ....../flann/src/cpp/flann/flann.hpp:42,
from ....../flann/src/cpp/flann/flann_cpp.cpp:30:
....../flann/src/cpp/flann/util/any.h: In member function 'virtual void flann::anyimpl::big_any_policy<T>::static_delete(void**)':
....../flann/src/cpp/flann/util/any.h:81:9: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
if (* x) delete (* reinterpret_cast<T**>(x)); *x = NULL;
^~
....../flann/src/cpp/flann/util/any.h:81:55: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
if (* x) delete (* reinterpret_cast<T**>(x)); *x = NULL;
^
[ 37%] Linking CXX static library ../../lib/libflann_cpp_s.a
cd ....../flann/build/src/cpp && /usr/bin/cmake -P CMakeFiles/flann_cpp_s.dir/cmake_clean_target.cmake
cd ....../flann/build/src/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/flann_cpp_s.dir/link.txt --verbose=1
/opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ar qc ../../lib/libflann_cpp_s.a CMakeFiles/flann_cpp_s.dir/flann/flann_cpp.cpp.o
/opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ranlib ../../lib/libflann_cpp_s.a
make[2]: Leaving directory '....../flann/build'
[ 37%] Built target flann_cpp_s
make -f src/cpp/CMakeFiles/flann_cpp.dir/build.make src/cpp/CMakeFiles/flann_cpp.dir/depend
make[2]: Entering directory '....../flann/build'
cd ....../flann/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" ....../flann ....../flann/src/cpp ....../flann/build ....../flann/build/src/cpp ....../flann/build/src/cpp/CMakeFiles/flann_cpp.dir/DependInfo.cmake --color=
Scanning dependencies of target flann_cpp
make[2]: Leaving directory '....../flann/build'
make -f src/cpp/CMakeFiles/flann_cpp.dir/build.make src/cpp/CMakeFiles/flann_cpp.dir/build
make[2]: Entering directory '....../flann/build'
[ 50%] Building CXX object src/cpp/CMakeFiles/flann_cpp.dir/empty.cpp.o
cd ....../flann/build/src/cpp && /opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-g++ --sysroot=....../RPi/arm-rpi-linux-gnueabihf/sysroot -DFLANN_EXPORTS -D_FLANN_VERSION=1.9.1 -I....../flann/src/cpp -I/usr/include -O2 -g -DNDEBUG -fPIC -Wall -Wno-unknown-pragmas -Wno-unused-function -o CMakeFiles/flann_cpp.dir/empty.cpp.o -c ....../flann/src/cpp/empty.cpp
[ 62%] Linking CXX shared library ../../lib/libflann_cpp.so
cd ....../flann/build/src/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/flann_cpp.dir/link.txt --verbose=1
/opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-g++ --sysroot=....../RPi/arm-rpi-linux-gnueabihf/sysroot -fPIC -I/usr/include -O2 -g -DNDEBUG -shared -Wl,-soname,libflann_cpp.so.1.9 -o ../../lib/libflann_cpp.so.1.9.1 CMakeFiles/flann_cpp.dir/empty.cpp.o -Wl,-whole-archive ../../lib/libflann_cpp_s.a -Wl,-no-whole-archive
cd ....../flann/build/src/cpp && /usr/bin/cmake -E cmake_symlink_library ../../lib/libflann_cpp.so.1.9.1 ../../lib/libflann_cpp.so.1.9 ../../lib/libflann_cpp.so
make[2]: Leaving directory '....../flann/build'
[ 62%] Built target flann_cpp
In file included from ....../flann/src/cpp/flann/algorithms/all_indices.h:35,
from ....../flann/src/cpp/flann/flann.hpp:45,
from ....../flann/src/cpp/flann/flann.h:603,
from ....../flann/src/cpp/flann/flann.cpp:31:

VERY Important:

  • If you build flann from source on a Raspberry Pi 3B, your system is going to hang HERE, possibly due to lack of memory…
  • Raspberry Pi 3B has 4 cores ONLY, but now you can use MORE cores on your host PC, which is clearly one advantage of using cross compiling.

Finally, you’ll see flann has been successfully cross compiled as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
....../flann/src/cpp/flann/algorithms/autotuned_index.h: In member function 'void flann::AutotunedIndex<Distance>::optimizeKMeans(std::vector<flann::AutotunedIndex<Distance>::CostData>&) [with Distance = flann::L2<double>]':
....../flann/src/cpp/flann/algorithms/autotuned_index.h:379:31: note: parameter passing for argument of type 'flann::L2<double>' changed in GCC 7.1
KMeansIndex<Distance> kmeans(sampledDataset_, cost.params, distance_);
^~~~~~
[ 75%] Linking CXX static library ../../lib/libflann_s.a
cd ....../flann/build/src/cpp && /usr/bin/cmake -P CMakeFiles/flann_s.dir/cmake_clean_target.cmake
cd ....../flann/build/src/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/flann_s.dir/link.txt --verbose=1
/opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ar qc ../../lib/libflann_s.a CMakeFiles/flann_s.dir/flann/flann.cpp.o
/opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-ranlib ../../lib/libflann_s.a
make[2]: Leaving directory '....../flann/build'
[ 75%] Built target flann_s
make -f src/cpp/CMakeFiles/flann.dir/build.make src/cpp/CMakeFiles/flann.dir/depend
make[2]: Entering directory '....../flann/build'
cd ....../flann/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" ....../flann ....../flann/src/cpp ....../flann/build ....../flann/build/src/cpp ....../flann/build/src/cpp/CMakeFiles/flann.dir/DependInfo.cmake --color=
Scanning dependencies of target flann
make[2]: Leaving directory '....../flann/build'
make -f src/cpp/CMakeFiles/flann.dir/build.make src/cpp/CMakeFiles/flann.dir/build
make[2]: Entering directory '....../flann/build'
[ 87%] Building CXX object src/cpp/CMakeFiles/flann.dir/empty.cpp.o
cd ....../flann/build/src/cpp && /opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-g++ --sysroot=....../RPi/arm-rpi-linux-gnueabihf/sysroot -DFLANN_EXPORTS -D_FLANN_VERSION=1.9.1 -I....../flann/src/cpp -I/usr/include -O2 -g -DNDEBUG -fPIC -Wall -Wno-unknown-pragmas -Wno-unused-function -o CMakeFiles/flann.dir/empty.cpp.o -c ....../flann/src/cpp/empty.cpp
[100%] Linking CXX shared library ../../lib/libflann.so
cd ....../flann/build/src/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/flann.dir/link.txt --verbose=1
/opt/compilers/RPi/bin/arm-rpi-linux-gnueabihf-g++ --sysroot=....../RPi/arm-rpi-linux-gnueabihf/sysroot -fPIC -I/usr/include -O2 -g -DNDEBUG -shared -Wl,-soname,libflann.so.1.9 -o ../../lib/libflann.so.1.9.1 CMakeFiles/flann.dir/empty.cpp.o -Wl,-whole-archive ../../lib/libflann_s.a -Wl,-no-whole-archive -Wl,--push-state,--no-as-needed -Wl,--pop-state
cd ....../flann/build/src/cpp && /usr/bin/cmake -E cmake_symlink_library ../../lib/libflann.so.1.9.1 ../../lib/libflann.so.1.9 ../../lib/libflann.so
make[2]: Leaving directory '....../flann/build'
[100%] Built target flann
make[1]: Leaving directory '....../flann/build'
/usr/bin/cmake -E cmake_progress_start ....../flann/build/CMakeFiles 0

You can now:

  • make install to install the built/generated libraries installed under CMAKE_INSTALL_PREFIX
  • copy and paste the built/generated libraries onto Raspberry Pi 3B and use it directly.

BTW: do NOT forget to install the header files.

5. Additional Issues

Multilib/multiarch seems to be problematic nowadays. Please pay attention to Multilib/multiarch. Some related issues are enumuated as the end of this blog.

Alright, that’s all for today. Let me go to bed. Good bye…

😪 😴