Zepyhr: reproducible project setup

This blog post demonstrates how to set up a Zephyr project in a reproducible
manner. Additionally, it provides some Makefile tricks and best practices for
using this powerful tool effectively.

While you can set up a Zephyr project manually, following the Getting Started
Guide
, a reproducible and automatic approach has several advantages. Firstly,
any changes made to the project will be automatically documented in GIT.
Furthermore, it is easier to move the project onto CI/CD servers or into Docker
containers.

(Ab)use of Makefiles

The entire setup is primarily managed by a Makefile. Despite the fact that
Zephyr utilizes CMake and Ninja, Makefiles offer a more convenient way to
consolidate numerous shell commands into a single location. You can consider
this Makefile as a repository of knowledge or as a mechanism for ensuring
replicability.

The full Makefile is accessible as
https://github.com/holgerschurig/zephyr-multi-board/blob/main/Makefile.zephyr_init

Basic project setup

Make sure you have all dependencies installed

The project setup happens automatically as soon as you try to build for one of
the support boards, e.g. with “make nucleo”.

How?

Let’s look at the first makefile part:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
UID := $(shell id -u)

debs .west/stamp.debs:
ifeq ($(UID),0)
	apt install -y --no-install-recommends \
		build-essential \
		ccache \
		cmake \
		device-tree-compiler \
		dfu-util \
		doxygen \
		file \
		g++-multilib \
		gcc \
		gcc-arm-none-eabi \
		gcc-multilib \
		gdb-multiarch \
		git \
		gperf \
		graphviz \
		libmagic1 \
		libnewlib-arm-none-eabi \
		libsdl2-dev \
		make \
		ninja-build \
		openocd \
		plantuml \
		python3-cbor \
		python3-click \
		python3-cryptography \
		python3-dev \
		python3-intelhex \
		python3-pip \
		python3-setuptools \
		python3-tk \
		python3-venv \
		python3-wheel \
		quilt \
		wget \
		xz-utils \
		zip
else
	sudo $(MAKE) --no-print-directory debs
	mkdir -p .west
	touch .west/stamp.debs
endif

In this section, we employ a trick using the Makefile to detect the user ID of
the current user in line 1. Line 4 is used to verify if the
Makefile is running as a user or root. If it’s running as root, we can utilize
apt” in line 5 to install all necessary dependencies.

If we’re non-root, we use “sudo” in line 43 to become root and execute the
“debs” Makefile target again. The “--no-print-directory” command-line argument
is employed to remove visual clutter from the output.

Lastly, as a normal user, we create the directory “.west” if it doesn’t exist
("-p") and place a stamp file inside it. The “make init” command checks the
existence of the stamp, preventing unnecessary re-execution of this part if it
already exists. In contrast, “make debs” does not check for the stamp and
always runs “apt”. This can be used if you want to install additional Debian
packages in an existing project setup.

Setting up a python virtual environment

Zephyr requires a tool named “west” that is written in Python and is installed
using “pip3”. Along with several Python modules. To prevent these modules from
conflicting with those installed by Debian (or Ubuntu), we need to create a
virtual environment.

Execution: either “make init” or, as a single step, “make debs”.

How?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
PWD := $(shell pwd)

.PHONY:: venv
venv:: .west/stamp.debs
ifeq ("$(wildcard .venv/bin/activate)","")
	python3 -m venv $(PWD)/.venv
endif
ifeq ("$(VIRTUAL_ENV)", "")
	@echo ""
	@echo "... ideally by sourcing all environments: source .env"
	@echo ""
	@exit 1
endif

help::
	@echo "   venv               create and check Python3 virtual environment"

In line 5, we verify if the environment already exists. While Make’s
dependency checking can be used for this purpose, it would check not only for
file existence, but also for the timestamp. In this case, this is undesirable.

If the environment does not exist, we use the Python “venv” module in line
6 to create one. While we could source “.venv/bin/activate” to activate
this within Make, unfortunately, it has to be done outside of Make. Instead, we
ask to source “.env” so that we can also set up the required Zephyr
environment variables.

Pro tip: On my development PCs, I have a shell function “pro” that
automatically changes into a project directory and sources “.env” if it
exists. It looks like this:

pro ()
{
    cd ~/src/$1 2> /dev/null || cd ~/d/$1 2> /dev/null || cd /usr/src/$1;
    test -f .env && . .env
}

So now I can do “pro cool-zephyr-project” and my environment is automatically
setup.

(This shell function assumes that you have your projects in your home directory
below the “d” (like development) or “src” directories. Adjust as needed.)

Install the “west” tool

Now that we have a virtual environent, we can install the “west” tool.

Execution: either “make init” or, as a single step, “make west”.

How?

.PHONY:: west
west .west/config:
	@type west >/dev/null || pip3 install west pyelftools
	mkdir -p .west
	/bin/echo -e "[manifest]\npath = zephyr\nfile = west.yml\n[zephyr]\nbase = zephyr" >.west/config

Actually this does 3 steps:

  • install west if it isn’t yet available
  • install pyelftools (needed on Debian Bookworm, as the distro provided ones are
    too old)
  • configure Zephyr via “.west/config

Note that other parts of the Makefile can depend on west being installed by
simply depending on this “.west/config” file.

Install Zephyr

Now we require the source of Zephyr. On some projects, you may want to use the
current development version, while on others, you may wish to pin yourself to a
specific version. Additionally, you might have local patches for Zephyr that you
don’t want to publish upstream and that you want to apply automatically. This
step accomplishes all of this!

Execution: either “make init” or, as a single step, “make zephyr”.

How?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#ZEPHYR_VERSION=zephyr-v3.5.0-3531-g6564e8b756

.PHONY:: zephyr
zephyr zephyr/.git/HEAD:
	git clone https://github.com/zephyrproject-rtos/zephyr.git
ifneq ("$(ZEPHYR_VERSION)", "")
	cd zephyr; git checkout -b my $(ZEPHYR_VERSION)
endif
ifneq ("$(wildcard patches-zepyhr/series)","")
	ln -s ../patches-zephyr zephyr/patches
	cd zephyr; quilt push -a
endif

Again, other makefile parts can depend on Zepyhr being installed by simply
depending on “zephyr/.git/HEAD”.

The first step is a typical “git clone”. If you don’t care about Zephyr’s
commit history (e.g., you don’t want to run things like “git log” or “git blame”), you can also add “--depth 1”. This reduces the size of the cloned
zephyr/” directory.

Specific version: you can uncommend and modify ZEPHYR_VERSION in line
1 to your liking. This will pin Zephyr to the specified version. This
is done by creating a branch “my” in line 6.

BTW, the value of ZEPHYR_VERSION is the output of “git describe --tags”.

Try “v3.6.0” for example.

Background: when should you start to lock Zephyr? This depends on your
circumstances. When a project is still in EVT phase, I tend to follow Zephyr
closely, e.g. use development version so it. “ZEPYHR_VERSION” would be
uncommented then. But then the projects enters DVT phase, or even MP phase, I’ll
certainly lock Zephyr to a well-known version.

Local patches: in one of my projects, I have patches that will probably never
be accepted by upstream Zephyr. I could put them directly into Zephyr, in my own
branch … but I prefer to have them in my own GIT project. So I use the
quilt” tool to manage a stack of patches.

The existence of quilt patches is checked in line 9 and, if they
exist, line 11 rolls them in.

Final note: It’s worth mentioning that due to version pinning and local
patches, we intentionally don’t use “west init” in this step.

Install needed Zephyr modules, e.g. HALs from the µC vendor

Some (actually almost all) of the SOCs that Zephyr supports need HALs (hardware
abstraction layers) provided by the chip vendor. If they don’t exist, we cannot
compile at all. So let’s install them!

Execution: either “make init” or, as a single step, “make modules”.

How?

.PHONY:: modules
help::
	@echo "   modules            install Zeyphr modules (e.g. STM32 and ESP32 HAL, CMSIS ...)"

.PHONY:: module_cmsis
modules module_cmsis modules/hal/cmsis/.git/HEAD:: .west/config
	mkdir -p modules
	west update cmsis
	touch --no-create modules/hal/cmsis/.git/HEAD
help::
	@echo "     module_cmsis     update only CMSIS"
ifneq ("$(wildcard modules/hal/cmsis/.git/HEAD)","")
update::
	west update cmsis
endif

The section with “cmsis” is copied again, but then with “hal_espressif”, “hal_st” and “hal_stm32”.

And any board target should now simply depend on these modules it needs. For
example, “native” (using Zephyr’s native_sim) doesn’t need anything. “nucleo”
needs cmsis, hal_st and hal_stm32. And “esp32c3” needs only hal_espressif. So
they should just declare the relevant “.../.git/HEAD” files as their
dependency.

Theoretically one could pin the modules also to specific version, like in the
step above. I however noticed that they are quite stable and this was never
needed. And also I need to have something to assign to you as homework, didn’t I
????

Getting help

If you look at the actual Makefile, you’ll notice that I ommited a whole lot of lines like

help::
	@echo "   modules            install Zeyphr modules (e.g. ST and STM32 HAL, CMSIS ...)"

from above. They aren’t strictly necessary, but nice. They allow you to run “make help” and
see all the common makefile targets meant for users. Like so:

(.venv) holger@holger:~/src/multi-board-zephyr$ make -f Makefile.zephyr_init help
debs                  only install debian packages
venv                  create and check Python3 virtual environment
west                  install and configure the 'west' tool
update                update 'west' and downloaded modules
zephyr                clone Zephyr
modules               install Zeyphr modules (e.g. STM32 and ESP32 HAL, CMSIS ...)
   module_cmsis       update only CMSIS
   module_espressif   update only ESPRESSIF HAL (ESP-32)
   module_st          update only ST HAL
   module_stm32       update only STM32 HAL

All of the above

All of the above targets from “Makefile.zephyr_init” are only there for
debugging. So that you can execute each of them by itself. Normally you just ask
the Makefile to compile for your selected boards after you cloned this
repository. It will then download the needed things for this all by itself.

Using this makefile in your project

You can simply add your own clauses at the end of this Makefile … your you can
include it from a main Makefile. This is demonstrated in the Github project
https://github.com/holgerschurig/zephyr-multi-board/:

Main “Makefile

PWD := $(shell pwd)
UID := $(shell id -u)

.PHONY:: all
all::


# Include common boilerplate Makefile to get Zephyr up on running
include Makefile.zephyr_init

# ... many more lines ...

First at the top we set two environment variables that we often use, PWD
(working directory) and UID (user id). You can then later just use them via
“$(PWD)” — note that Make want’s round brances here, not curly braces like
Bash.

Then I set a default target, to be executed if you just run “make” without specifying
a target by yourself.

The double colon here needs to be used for all targets that are defined more
than once in a Makefile. As you see, here the target is empty. It’s fleshed out
in much more complexity below, but this is beyond this blog post.

Also note the “.PHONY:: all” line. It helps Make to understand that “make
or “make all” isn’t supposed to actually create file called “all”. This
helps it’s dependency resolvement engine, and is good style. My makefile uses
.PHONY::” liberally, for each pseudo-target (shell script snippet) basically.

Finally, we use Make’s “include” clause to include our boilerplate Makefile.

You could also run the Boilerplate makefile itself, with “make -f Makefile.zephyr_init”, e.g. for debugging purposes.