Building FreeBSD in meta mode

This doc describes the process of converting the FreeBSD build to use bmake in meta mode.

Note: while I normally talk of meta mode and using dirdeps.mk together, either can be used without the other.

The initial commit of bmake went into FreeBSD head in October 2012. FreeBSD 10 is the first release to use bmake by default. The meta mode infrastructure went into FreeBSD head in June 2015.

To avoid? confusion I will use fmake when referring to the older FreeBSD make and bmake regardless of which is installed as make.

Why meta mode

An obvious question, with a very simple answer: to improve the build.

There are several aspects to this:

more reliable update builds

Provided filemon is being used, the .meta files created during a build, allow bmake much greater visibility into what happens during the creation of a target. This allows it to be more thorough when deciding if a target is out-of-date.

captured command line

Even without filemon, the .meta file captures the expanded command line used last time, and by default any change to that command line causes the target to be out-of-date.

The comparison of command lines can be controlled at multiple levels; per build, makefile, target or even line within a target script. The last is perhaps the most useful.

Any command that that references the special variable .OODATE cannot be usefully compared and so isn't. This can be leveraged by using .OODATE in such a way that it does not affect the command: ${.OODATE:M.no.cmp.} will expand to nothing (change the match string if .no.cmp. might actually appear ;-)

captured output

The .meta files capture the output from the commands that generate a target.

This allows a much cleaner - and easier to read build log, and even for people who refuse to log the output of builds, the vital clues are captured when something goes wrong.

.ERROR

The .ERROR target is run when bmake hits an error, and if it was while generating a target, .ERROR_TARGET is set to the name of the target and .ERROR_META_FILE will be set to the name of the meta file for it. Thus we can capture a copy of the relevant .meta file in a well known location.

This then becomes a simple and reliable means of spotting how and why a build failed. I have scripts that will parse that .meta file, and identify the commit that broken the build.

tree dependencies

When filemon is used, the .meta files capture lots of useful data, which can be used in many ways, but one of the most obvious is learning the dependencies of one directory on others.

simplify the build

This is one of the key benefits. This is mostly a benefit of using dirdeps.mk, though many individual makefiles can be simplified by use of meta mode.

The use of meta mode also allows easy automation of the data needed by dirdeps.mk, so we usually just refer to this as the meta mode build. The dirdeps.mk or meta mode build is very easy to understand.

Look at the top-level makefiles of any BSD system today, and try to understand exactly what they do. It can be a challenge indeed. Ignoring comments and blank lines, FreeBSD 10's top-level Makefile and Makefile.inc1 have more than 1800 lines.

The pre-meta-mode Junos build had over 5000 lines in its top-level makefiles.

By contrast, Makefile, pkgs/Makefile and pkgs/Makefile.inc which drive the Junos meta mode build were until recently less than 130 lines.

Of course I am not counting more than 10k autogenerated Makefile.depend* files which make it possible, but;

a/ they are autogenerated and each by itself is easy to understand

b/ they are not only used for top-level builds

In fact in this model, the top-level build is little different from any other.

Not to mention that top-level builds can often be dispensed with.

We use the example below of make -j8 -C bin/cat in a clean tree.

Bmake

The key makefile dirdeps.mk requires all the functionality in bmake, so the first step was building FreeBSD with bmake.

meta mode

Being able to do buildworld is a useful stepping stone, but isn't very interesting.

Being able to make -j8 -C bin/cat in a freshly checked out tree, and have it correctly build all the libs etc. in one pass, is more like it. Which is where meta mode or more correctly dirdeps.mk comes in. The output below shows the directories checked - only things needed by cat get built:

Checking /b/sjg/work/FreeBSD/projects-bmake/src/targets/pseudo/stage for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/gnu/lib/libssp/libssp_nonshared for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/xlocale for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/rpc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/rpcsvc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/csu/amd64 for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/libcompiler_rt for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/libc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/gnu/lib/libgcc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat for amd64 ...

Perhaps also interesting is that we can get the build to show us the complete tree dependency graph from any starting point. For example:

$ build-graph -C bin/cat
bin/cat.amd64: gnu/lib/libgcc.amd64 gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/xlocale.amd64 lib/csu/amd64.amd64 lib/libc.amd64 lib/libcompiler_rt.amd64 targets/pseudo/stage.amd64
gnu/lib/libgcc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/xlocale.amd64 lib/csu/amd64.amd64 lib/libc.amd64 targets/pseudo/stage.amd64
gnu/lib/libssp/libssp_nonshared.amd64: targets/pseudo/stage.amd64
include.amd64: gnu/lib/libssp/libssp_nonshared.amd64 targets/pseudo/stage.amd64
include/rpc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 targets/pseudo/stage.amd64
include/rpcsvc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 targets/pseudo/stage.amd64
include/xlocale.amd64: gnu/lib/libssp/libssp_nonshared.amd64 targets/pseudo/stage.amd64
lib/csu/amd64.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 targets/pseudo/stage.amd64
lib/libc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/rpc.amd64 include/rpcsvc.amd64 lib/csu/amd64.amd64 lib/libcompiler_rt.amd64 targets/pseudo/stage.amd64
lib/libcompiler_rt.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 targets/pseudo/stage.amd64
$

The build-graph script just leverages the debug information that dirdeps.mk can output as it examines the Makefile.depend* files.

Even better; using -DWITH_DIRDEPS_CACHE recent dirdeps.mk saves all its work in ${DIRDEPS_CACHE} which defaults to ${.OBJDIR}/dirdeps.cache with a suffix added if non-default target is being built. For example for bin/cat it starts:

# Autogenerated - do NOT edit!

BUILD_DIRDEPS=no

.include <dirdeps.mk>

# bin/cat.i386
dirdeps: \
        ${SRCTOP}/bin/cat.i386 \
        ${SRCTOP}/cddl/usr.bin/ctfconvert.host \
        ${SRCTOP}/cddl/usr.bin/ctfmerge.host \
        ${SRCTOP}/include.i386 \
        ${SRCTOP}/include/xlocale.i386 \
        ${SRCTOP}/lib/csu/i386.i386 \
        ${SRCTOP}/lib/libc.i386 \
        ${SRCTOP}/targets/pseudo/stage.i386

Which means that the dirdeps target requires all of those built, but that does not define the order.

${SRCTOP}/bin/cat.i386: _DIRDEP_USE
${SRCTOP}/cddl/usr.bin/ctfconvert.host: _DIRDEP_USE
${SRCTOP}/cddl/usr.bin/ctfmerge.host: _DIRDEP_USE
${SRCTOP}/include.i386: _DIRDEP_USE
${SRCTOP}/include/xlocale.i386: _DIRDEP_USE
${SRCTOP}/lib/csu/i386.i386: _DIRDEP_USE
${SRCTOP}/lib/libc.i386: _DIRDEP_USE
${SRCTOP}/targets/pseudo/stage.i386: _DIRDEP_USE

As noted in dirdeps.mk the _DIRDEP_USE is what actually causes things to be built. We then see blocks like:

${SRCTOP}/bin/cat.i386: \
        ${SRCTOP}/cddl/usr.bin/ctfconvert.host \
        ${SRCTOP}/cddl/usr.bin/ctfmerge.host


${SRCTOP}/bin/cat.i386: \
        ${SRCTOP}/include.i386 \
        ${SRCTOP}/include/xlocale.i386 \
        ${SRCTOP}/lib/csu/i386.i386 \
        ${SRCTOP}/lib/libc.i386 \
        ${SRCTOP}/targets/pseudo/stage.i386

which express the things which need to be built before ${SRCTOP}/bin/cat.i386, this sort of info repeats:

# lib/libc.i386
dirdeps: \
        ${SRCTOP}/cddl/usr.bin/ctfconvert.host \
        ${SRCTOP}/cddl/usr.bin/ctfmerge.host \
        ${SRCTOP}/gnu/lib/libssp/libssp_nonshared.i386 \
        ${SRCTOP}/include.i386 \
        ${SRCTOP}/include/rpc.i386 \
        ${SRCTOP}/include/rpcsvc.i386 \
        ${SRCTOP}/lib/csu/i386.i386 \
        ${SRCTOP}/lib/libc_nonshared.i386 \
        ${SRCTOP}/targets/pseudo/stage.i386 \
        ${SRCTOP}/usr.bin/rpcgen.host \
        ${SRCTOP}/usr.bin/yacc.host

${SRCTOP}/gnu/lib/libssp/libssp_nonshared.i386: _DIRDEP_USE
${SRCTOP}/include/rpc.i386: _DIRDEP_USE
${SRCTOP}/include/rpcsvc.i386: _DIRDEP_USE
${SRCTOP}/lib/libc_nonshared.i386: _DIRDEP_USE
${SRCTOP}/usr.bin/rpcgen.host: _DIRDEP_USE
${SRCTOP}/usr.bin/yacc.host: _DIRDEP_USE

${SRCTOP}/lib/libc.i386: \
        ${SRCTOP}/cddl/usr.bin/ctfconvert.host \
        ${SRCTOP}/cddl/usr.bin/ctfmerge.host \
        ${SRCTOP}/usr.bin/rpcgen.host \
        ${SRCTOP}/usr.bin/yacc.host


${SRCTOP}/lib/libc.i386: \
        ${SRCTOP}/gnu/lib/libssp/libssp_nonshared.i386 \
        ${SRCTOP}/include.i386 \
        ${SRCTOP}/include/rpc.i386 \
        ${SRCTOP}/include/rpcsvc.i386 \
        ${SRCTOP}/lib/csu/i386.i386 \
        ${SRCTOP}/lib/libc_nonshared.i386 \
        ${SRCTOP}/targets/pseudo/stage.i386

finally we have:

.info ${.newline}${TRACER}Makefiles read: total=71 depend=15 dirdeps=17

The end result is a generated makefile that expresses the graph needed to build bin/cat.

So long as none of the``Makefile.depend*`` files involved change, that cache can be reused to speed up the build, but in many cases the information it provides is even more important.

tree dependencies

Everyone is familiar with doing make depend to capture the dependencies that a directory has on the files that it uses, to help ensure that re-running make will do the right thing.

What we need is the same level of information for the tree as a whole. As detailed in Building BSD with meta mode I've implemented that a couple of ways in the Junos build, and meta mode is the current method used.

meta files

In meta mode, for most targets bmake creates a .meta file to capture information like the expanded command used, any command output (useful for debugging), and a record of all the successful system calls that are interesting to make:

# Meta data file /var/obj/projects-bmake/amd64/bin/cat/cat.o.meta
CMD cc -O2 -pipe  -nostdinc -isystem /var/obj/projects-bmake/stage/amd64/usr/include -isystem /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2 -std=gnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -c /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
CMD ctfconvert -L VERSION cat.o
CWD /var/obj/projects-bmake/amd64/bin/cat
TARGET cat.o
-- command output --
-- filemon acquired metadata --
# filemon version 4
# Target pid 42504
# Start 1363631731.803698
V 4
F 42504 42529
E 42529 /bin/sh
R 42529 /var/run/ld-elf.so.hints
R 42529 /lib/libedit.so.7
R 42529 /lib/libncurses.so.8
R 42529 /lib/libc.so.7
S 42529 .
S 42529 /var/obj/projects-bmake/amd64/bin/cat
S 42529 /usr/bin/cc
F 42529 42530
E 42530 /usr/bin/cc
S 42530 /usr/bin/cc
S 42530 /usr/bin/cc
F 42530 42533
E 42533 /usr/bin/cc
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2
R 42533 cat.o-24d10fa0
W 42533 cat.o-24d10fa0
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/cdefs.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_null.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/types.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/endian.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/endian.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_pthreadtypes.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_stdint.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/select.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sigset.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timeval.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/syslimits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/stat.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/time.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/socket.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_iovec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sockaddr_storage.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/un.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/errno.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/runetype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/err.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/fcntl.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stddef.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdio.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdlib.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/strings.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/unistd.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/unistd.h
M 42533 '/var/obj/projects-bmake/amd64/bin/cat/cat.o-24d10fa0' 'cat.o'
X 42533 0
X 42530 0
S 42529 /usr/bin/ctfconvert
F 42529 42541
E 42541 /usr/bin/ctfconvert
R 42541 /var/run/ld-elf.so.hints
R 42541 /lib/libctf.so.2
R 42541 /usr/lib/libdwarf.so.3
R 42541 /usr/lib/libelf.so.1
R 42541 /lib/libz.so.6
R 42541 /lib/libthr.so.3
R 42541 /lib/libc.so.7
R 42541 cat.o
R 42541 cat.o
R 42541 cat.o.ctf
W 42541 cat.o.ctf
M 42541 'cat.o.ctf' 'cat.o'
X 42541 0
X 42529 0
# Stop 1363631732.322849
# Bye bye

Note: in the NetBSD version of filemon we don't bother recording the stat calls (S).

From this we can derive not only the equivalent data of make depend, but we can derive tree based dependencies.

By that I mean that any file read from an object directory which is not .OBJDIR represents a directory which needs to be built before .CURDIR, it's that simple.

Of course having some consistency in the relationship between object dirs and src dirs helps. Even when that isn't possible though we can work around it.

In the example above, a number of headers are read from /var/obj/projects-bmake/stage/amd64/usr/include/. This is where we stage headers as we build the tree.

${STAGE_OBJTOP}/usr/include

As we build libs and other things which have headers to be installed, they get staged into this directory, a long with a .dirdep file to say who put the file there. For example:

/var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
/var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h.dirdep

the .dirdep file contains:

include/xlocale.amd64

which says it was put there by the makefile in include/xlocale while building for the machine amd64.

Thus as we read the .meta file and find a file read, we can look for the same path with .dirdep appended to find the directory we should add to our dependencies. If no .dirdep file exists, it is presumed we can derive the src location from the objdir.

${STAGE_OBJTOP}/include (deprecated)

The makefile in src/include/ does not fit the normal usage pattern, and fixing it at this stage isn't an attractive option. So rather than use our normal staging method, we let it install into a directory which only it populates, and for each directory thus created we add a .dirdep file.

We now have a stage-install.sh wrapper script which allows running install and then deposits the necessary .dirdep files. This allows all headers to be staged to ${STAGE_OBJTOP}/usr/include.

filemon

This is a clone device used to monitor the successful system calls (that are interesting to make) performed by a process and its descendants.

This is how bmake captures the data in the meta files For each job, bmake opens /dev/filemon and gives it the path to a temp file. After it forks, the child gives its pid to filemon. When the child exits bmake reads the syscall trace from the temp file.

meta2deps

Originally a shell script meta2deps.sh was used to extract the useful information from .meta files. It works fine. I should say worked fine - it hasn't had any attention recently.

If Python is available though we will use meta2deps.py which produces the same result, but much more efficiently (5-10 times faster) and can gather additional data without overhead.

Makefile.depend

After processing the .meta files for bin/cat we end up with the following in ${.MAKE.DEPENDFILE}:

# Autogenerated - do NOT edit!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        gnu/lib/libgcc \
        include \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
        lib/libcompiler_rt \

.include <dirdeps.mk>

.if ${DEP_RELDIR} == ${_DEP_RELDIR}
# local dependencies - needed for -jN in clean tree
.endif

which shows a number of interesting things.

SRCTOP

Defines the top of the src tree. This along with OBJTOP and OBJROOT (eg. OBJTOP=${OBJROOT}${MACHINE}) allow for being able to consistently refer to things in the src and object trees.

RELDIR

Being able to trim ${SRCTOP} from the start of ${.CURDIR} provides a useful relative location within the src tree. This forms the basis of DIRDEPS.

The .dirdep files mentioned above are created by simply doing:

.dirdep:
        @echo ${RELDIR}.${MACHINE} > $@

.PARSEDIR/.PARSEFILE

NetBSD make (bmake is just a portable version of it), defines .PARSEDIR as the directory from which the current .PARSEFILE is being read. This is extremely useful to us. As is the :tA modifier:

_PARSEDIR= ${.PARSEDIR:tA}

or the absolute path to .PARSEDIR.

As you can see in the example above we can use ${_PARSEDIR} and ${SRCTOP} to derive the RELDIR of the makefile being read.

We can also use .PARSEDIR as a clue that the makefile is being run by bmake rather than fmake:

.if defined(.PARSEDIR)
# we are bmake ...
.endif

DEP_RELDIR

Allows dirdeps.mk to keep track of the RELDIR it is gathering/computing dependencies for, and it gets automatically updated as each Makefile.depend file is read.

DEP_MACHINE

In the example above, there are no machine qualified entries in DIRDEPS, if there were, as in targets/pseuod/kernel/Makefile.depend for example:

DIRDEPS = \
        include \
        include/xlocale \
        usr.sbin/config.host \

the last entry indicates that usr.sbin/config must be built for the pseudo machine host. When dirdeps.mk is reading usr.sbin/config/Makefile.depend it will have set DEP_MACHINE=host which can influence filtering and other things as well as ensuring the correct build dependencies are created.

dirdeps.mk

This is a complex makefile, and I recommend against touching it.

This is what makes everything work, and thus the complexity pays off; because everything else (especially the bits people might need to touch) can be quite simple.

bootstrapping meta mode

The easy way to bootstrap meta mode, is to use the old build with top-level makefiles running in non-meta mode, but leaf directories running in meta mode. Thus we do not rely on meta mode for any of the sequencing (that only happens if the 0th bmake is running in meta mode), but we can generate the Makefile.depend files as a consequence of building.

That was easy with the Junos, build because it hasn't used anything but the leaf directories for at least a decade.

After fiddling for a bit trying to leverage buildworld to do this, I gave up and tried the brute force approach:

find * -name Makefile > mfile.list
egrep -v '^(contrib|crypto)/|dist/' mfile.list |
sed 's,\(.*\)/Makefile,   \1 \\,' > dirdep.list

and edit that into a simple the-lot/Makefile.depend file. By starting a build using that, bmake would try to visit every directory in the tree with a makefile. Not very smart. Especially since there are a number of directories in the tree which do not actually build any more.

buildworld -DWITH_META_MODE

Currently we can do buildworld -DWITH_META_MODE to create meta files in a build that otherwise behaves like normal buildworld. These meta files are very useful for comparing against those in the meta mode build - especially when after a sync, things seem to be broken.

We can probably leverage these meta files for bootstrapping too though that is basically done already.

bootstrapping new content

Once the bulk of the tree can build in meta mode, dealing with new dependencies is not that big a deal.

Eg. after last sync, usr.bin/kdump needs libcapsicum and libnv:

mk -C lib/libcapsicum
mk -C lib/libnv
mk -C usr.bin/kdump

is all that is needed - of course there will now be Makefile.depend files in the new libs which need to be added to SVN.

targets/pseudo/

The targets/Makefile which we use as a top-level makefile looks for subdirs under targets/ and targets/pseudo/ that match the target to be built.

Subdirs of targets/pseudo/ are not expected to build anything themselves, just be place holders for Makefile.depend files (which are invariably manually maintained). Thus their makefiles do nothing - except to say not to update Makefile.depend.

From our bootstrapping exercise above we got:

targets/pseudo/bin/Makefile.depend
targets/pseudo/cddl/Makefile.depend
targets/pseudo/clang/Makefile.depend
targets/pseudo/games/Makefile.depend
targets/pseudo/gnu/Makefile.depend
targets/pseudo/include/Makefile.depend
targets/pseudo/kerberos5/Makefile.depend
targets/pseudo/lib/Makefile.depend
targets/pseudo/libexec/Makefile.depend
targets/pseudo/misc/Makefile.depend
targets/pseudo/sbin/Makefile.depend
targets/pseudo/secure/Makefile.depend
targets/pseudo/share/Makefile.depend
targets/pseudo/usr.bin/Makefile.depend
targets/pseudo/usr.sbin/Makefile.depend

which we simply list in targets/pseudo/userland/Makefile.depend:

# This file is not autogenerated - take care!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        targets/pseudo/bin \
        targets/pseudo/cddl \
        targets/pseudo/games \
        targets/pseudo/gnu \
        targets/pseudo/include \
        targets/pseudo/kerberos5 \
        targets/pseudo/lib \
        targets/pseudo/libexec \
        targets/pseudo/sbin \
        targets/pseudo/secure \
        targets/pseudo/share \
        targets/pseudo/usr.bin \
        targets/pseudo/usr.sbin \

.include <dirdeps.mk>

which is in turn referenced by targets/pseudo/the-lot/Makefile.depend:

# This file is not autogenerated - take care!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        targets/pseudo/kernel \
        targets/pseudo/toolchain \
        targets/pseudo/toolchain.host \
        targets/pseudo/userland \


.include <dirdeps.mk>

The compilers are built via toolchain for both the host and target machines.

For the embedded world we don't normally build compilers at all.

machine dependent dirdeps

The DIRDEPS.amd64 lines from our bootstrap output, lead us to run a separate extraction process for each machine type:

getMdirdeps() {
    for mf in "$@"
    do
        case "$mf" in
        *targets/*|*~) continue;;
        esac
        d=${mf%/*}
        m=${mf##*.}
        case "$m" in
        ""|inc|orig|rej|bak|old) continue;;
        esac
        MACHINE=$m ${MAKE:-make} -C $d -f bsd.arch.inc.mk gen-dirdeps 2>&1
    done
}

which we can then add to targets/pseudo/sbin/Makefile.depend etc:

DIRDEPS.amd64= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol
DIRDEPS.arm= sbin/bsdlabel sbin/fdisk
DIRDEPS.i386= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol sbin/sconfig
DIRDEPS.ia64= sbin/mca
DIRDEPS.mips= sbin/bsdlabel sbin/fdisk
DIRDEPS.pc98= sbin/bsdlabel sbin/fdisk_pc98 sbin/sconfig
DIRDEPS.sparc64= sbin/bsdlabel sbin/sunlabel

DIRDEPS+= ${DIRDEPS.${MACHINE}:U}

.include <dirdeps.mk>

Note that any directory listed in DIRDEPS.amd64 should be removed from the generic DIRDEPS - bsd.subdir.mk would have entered them and caused them to list themselves.

Also note, that the DIRDEPS.amd64 lines output by bsd.arch.inc.mk are not always leaf dirs. For instance in targets/pseudo/usr.sbin/Makefile.depend we originally see:

DIRDEPS = \
        usr.sbin/acpi/acpiconf \
        usr.sbin/acpi/acpidb \
        usr.sbin/acpi/acpidump \
        usr.sbin/acpi/iasl \

but we see:

DIRDEPS.amd64=  usr.sbin/acpi

which means we need to move usr.sbin/acpi/* from DIRDEPS to DIRDEPS.amd64 and any other that references usr.sbin/acpi.

package manifests

Of course all this is moot once you have a set of manifests or mtree specs used to build the install packages/media. These simply list the applications and shared libs needed, and from those we learn the dependencies we need.

In the Junos build we produce packages that contain isofs images. These are built in targets/* and unlike the targets/pseudo/*, auto updated dependencies based on what went into the package.

targets/pseudo/kernel

The exception to the rule is targets/pseudo/kernel which actually builds a kernel in the FreeBSD way (GENERIC by default).

In the Junos build we need to build multiple kernels for multiple machines and want to automatically capture dependencies for them all. So there is a src directory that corresponds to each kernel.

In keeping with the traditional FreeBSD kernel build though, this makefile just does:

# $FreeBSD: projects/bmake/targets/pseudo/kernel/Makefile 248288 2013-03-14 22:04:25Z sjg $

# Build the kernel ${KERNCONF}
KERNCONF?= ${KERNEL:UGENERIC}

TARGET?= ${MACHINE}
# keep this compatible with peoples expectations...
KERN_OBJDIR= ${OBJTOP}/sys/compile/${KERNCONF}
KERN_CONFDIR= ${SRCTOP}/sys/${TARGET}/conf

CONFIG= ${STAGE_HOST_OBJTOP}/usr/sbin/config

${KERNCONF}.config: .MAKE .META
        mkdir -p ${KERN_OBJDIR:H}
        (cd ${KERN_CONFDIR} && \
        ${CONFIG} ${CONFIGARGS} -d ${KERN_OBJDIR} ${KERNCONF})
        (cd ${KERN_OBJDIR} && ${.MAKE} depend)
        @touch $@

# we need to pass curdirOk=yes to meta mode, since we want .meta files
# in ${KERN_OBJDIR}
${KERNCONF}.build: .MAKE ${KERNCONF}.config
        (cd ${KERN_OBJDIR} && META_MODE="${.MAKE.MODE} curdirOk=yes" ${.MAKE})

.if ${.MAKE.LEVEL} > 0
all: ${KERNCONF}.build
.endif

UPDATE_DEPENDFILE= no

.include <bsd.prog.mk>

circular dependencies

Our goal is to be able to build the tree in a single pass. That is; visiting dirs in the correct order - once, and have the default (all) target do all that is required, such as building objects (libs, progs) installing (staging) headers, libs even progs if -DWITH_STAGING_PROG, and finally updating any dependencies.

This requires that there be no circular dependencies within the tree. Currently in head, lib/libc depends on headers from both lib/libutil and lib/msun, and since libc.so needs to be scanned when linking shared libs, we have a potential circular dependency.

In the case of lib/libutil this is trivial to avoid, since it has only two headers and they are both public, so adding:

CFLAGS+= -I${.CURDIR:H}/libutil

to lib/libc/Makefile is fine.

Things are not quite as neat for lib/msun which not only has headers in a machine specific subdir, but also has math.h in a directory which contains both public and private headers. The following in lib/libc/Makefile avoids the dependency:

MSUN_ARCH_SUBDIR != ${MAKE} -B -C ${.CURDIR:H}/msun -V ARCH_SUBDIR
# unfortunately msun/src contains both private and public headers
CFLAGS+= -I${.CURDIR:H}/msun/${MSUN_ARCH_SUBDIR} -I${.CURDIR:H}/msun/src

but is not as clean as the previous case.

Of course lib/msun/Makefile does something similar to grab headers from libc:

# Location of fpmath.h and _fpmath.h
LIBCDIR=        ${.CURDIR}/../libc
.if exists(${LIBCDIR}/${MACHINE_ARCH})
LIBC_ARCH=${MACHINE_ARCH}
.else
LIBC_ARCH=${MACHINE_CPUARCH}
.endif
CFLAGS+=        -I${.CURDIR}/src -I${LIBCDIR}/include \
        -I${LIBCDIR}/${LIBC_ARCH}

Keeping public headers separate from private headers helps ensure that the above sort of dance is harmless.

Another example is lib/libproc and lib/librtld_db each of which includes a header staged by the other - resulting in a circular dependency. The fix is simple:

Index: lib/libproc/Makefile
===================================================================
--- lib/libproc/Makefile        (revision 242545)
+++ lib/libproc/Makefile        (working copy)
@@ -14,6 +14,8 @@
 INCS=  libproc.h

 CFLAGS+=       -I${.CURDIR}
+# avoid cyclic dependency
+CFLAGS+=       -I${.CURDIR:H}/librtld_db

 .if ${MK_LIBCPLUSPLUS} != "no"
 LDADD+=                -lcxxrt

MACHINE specific depend files

Having previously built the tree for i386, and now building for amd64 we can detect directories where a single Makefile.depend is probably not ideal:

Index: usr.bin/truss/Makefile.depend
===================================================================
--- usr.bin/truss/Makefile.depend       (revision 242503)
+++ usr.bin/truss/Makefile.depend       (working copy)
@@ -8,7 +8,6 @@
        gnu/lib/libgcc \
        include \
        include/arpa \
-       include/rpc \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
@@ -18,10 +17,12 @@

 .if ${DEP_RELDIR} == ${_DEP_RELDIR}
 # local dependencies - needed for -jN in clean tree
-i386-fbsd.o: syscalls.h
-i386-fbsd.po: syscalls.h
-i386-linux.o: linux_syscalls.h
-i386-linux.po: linux_syscalls.h
+amd64-fbsd.o: syscalls.h
+amd64-fbsd.po: syscalls.h
+amd64-fbsd32.o: freebsd32_syscalls.h
+amd64-fbsd32.po: freebsd32_syscalls.h
+amd64-linux32.o: linux32_syscalls.h
+amd64-linux32.po: linux32_syscalls.h
 ioctl.o: ioctl.c
 ioctl.po: ioctl.c
 .endif

There are very few of these cases (so far), where machine specific local dependencies need to be captured. The simplest solution is for these to create Makefile.depend.${MACHINE}.

The vast majority of the tree seems to be fine with a simple Makefile.depend.

We configure .MAKE.DEPENDFILE_PREFERENCE such that once a Makefile.depend.${MACHINE} exists, it will be used.

We set .MAKE.DEPENDFILE_DEFAULT to the plain Makefile.depend.

Once a directory contains a machine specific depend file, we will automatically follow suit when building for other machines.

Staging conflicts

The latest version of meta.stage.mk throws an error when it detects that a different directory has already staged the file that it wants to.

Most of the cases found so far are simple - and easily fixed. For example lib/ncurses/ncursesw/Makefile includes lib/ncurses/ncurses/Makefile and thus attempts to install the same headers in the same location. This is easily fixed:

Index: lib/ncurses/ncurses/Makefile
===================================================================
--- lib/ncurses/ncurses/Makefile        (revision 242503)
+++ lib/ncurses/ncurses/Makefile        (working copy)
@@ -304,6 +304,7 @@
 SYMLINKS+=     libncurses${LIB_SUFFIX}_p.a
 ${LIBDIR}/libtinfo${LIB_SUFFIX}_p.a
 .endif

+.if ${.CURDIR:T} == "ncurses"
 DOCSDIR=       ${SHAREDIR}/doc/ncurses
 DOCS=          ncurses-intro.html hackguide.html

@@ -311,6 +312,7 @@
 .PATH: ${NCURSES_DIR}/doc/html
 FILESGROUPS=   DOCS
 .endif
+.endif

 # Generated source
 .ORDER: names.c codes.c

beforeinstall

When building WITH_STAGING_PROG we want to stage pretty much everything. There are a number of makefiles however that rely on the target beforeinstall to prepare files that we would want to stage.

Rather than re-write all these makefiles, we leverage beforeinstall and set DESTDIR=${STAGE_OBJTOP} which is very similar to what we did for src/include/.

targets/pseudo/stage

Many of those beforeinstall targets rely on mtree having been run in DESTDIR. The makefile in targets/pseudo/stage ensures that this is done, and is inserted as a dependency on every directory except itself.

bootstrap-tools

Anyone remember porting gcc in the early days? It was a 3 stage dance.

  1. build stage1-gcc with cc
  2. build stage2-gcc with stage1-gcc
  3. build gcc with stage2-gcc

now we could do even that without breaking our single pass through the tree goal - by having separate makefiles for the three stages.

But since the whole toolchain topic needs is own overhaul, for now I'm happy to largely punt it, and even leverage the bootstrap targets from src/Makefile.inc1.

Stable dependencies

Being able to build in meta mode is cool, but a conversion cannot be considered complete while there is dependency churn. By that I mean certain Makefile.depend files may be observed to change without obvious cause. This is invariably due to bugs in the makefiles.

The good news is that such churn does not necessarily affect the build results, which is handy if certain makefile bugs cannot be fixed during the transition phase.

So far I have seen little evidence of this in FreeBSD.

Changing behavior based on clean/update build

A very common cause is a makefile using .if exists(something) and changing its behavior as a result. Thus the result from a clean tree build can differ from an update build. This should be fixed.

Use of SRCTOP, OBJTOP, OBJROOT etc, can greatly help canonicalize references to locations within the tree which can eliminate many such issues.

Conflicting makefiles

Whenever two (or more) makefiles try to do the same thing, eg include/arpa/Makefile and lib/libtelnet/Makefile both installing usr/include/arpa/telnet.h. In this case meta.stage.mk will throw an error so the issue is quickly resolved.

Other cases can be more subtle. In general it is dangerous for any makefile to stomp on anything outside of its .OBJDIR.

Foreign builds

If part of the build is done with say gmake, our ability to reliably capture what it does is limited to clean tree builds. There is explicit mechanism in meta.autodep.mk to cope with this.

Current state of FreeBSD

Update: as of June 2017, thanks to lots of work from Bryan Drewery, building WITH_META_MODE may soon be enabled by default.

For WITH_DIRDEPS_BUILD instead of simply using ${MACHINE} to distinguish targets we use TARGET_SPEC_VARS concatenated with , eg. ${MACHINE},${MACHINE_ARCH}

Environment

I should note that I generally run make via a wrapper script (mk) which first reads a file $SB/.sandbox-env (where SB is typically the parent dir to src/) to condition the environment. I have dozens of active trees for Junos, NetBSD and FreeBSD and multiple branches and this allows me use the same user interface for all.

For the DIRDEPS_BUILD the key environment items are:

export SRCTOP=$SB/src
export OBJROOT=/var/obj/projects-bmake/
export OBJTOP="$OBJROOT\${MACHINE}"
export MAKESYSPATH=$SRCTOP/share/mk
export MAKEOBJDIR='${.CURDIR:S,${SRCTOP},${OBJTOP},}'
export HOST_TARGET=freebsd10-amd64
export HOST_OBJTOP="$OBJROOT$HOST_TARGET"

Not all are strictly necessary as *sys.mk can set them, but putting such logic in the makefiles constrains choices.

Note that MAKEOBJDIR value is single quoted, this defers expansion until bmake sees it.

HOST_TARGET is set by the wrapper script, based on uname output of the host machine. For example freebsd10-amd64 or freebsd7-i386. When building for the pseudo machine host we use ${HOST_OBJTOP} rather than ${OBJTOP}. This allows us to avoid trouble when the same tree is mounted via NFS and built from incompatible machines.

I have local.sys.mk set a number of options:

WITH_INSTALL_AS_USER= yes
WITH_AUTO_OBJ= yes
WITH_META_MODE= yes
WITH_STAGING= yes
WITH_STAGING_PROG= yes

WITH_AUTO_OBJ is important, since it causes the equivalent of make obj to be done automatically and early, so that as the makefile is read .OBJDIR and .CURDIR have their correct values which matters in the gcc build where we see:

.PATH: ../cc_tools

which is supposed to add ${.OBJDIR}/../cc_tools to .PATH, but if ${.OBJDIR} does not exist at the time that line is read, the right thing will not happen.

Getting started

I need to document this because I keep forgetting to rebuild the toolchains after sync'ing from head.

With the environment initialized, and WITH_META_MODE=yes, after new checkout or sync from head I do:

mk-host -j8 bootstrap-tools

Note mk-host is just a link to the wraper script mk and causes it to set MACHINE=host. If you don't want to use mk, then something like:

env MACHINE=host make -j8 bootstrap-tools

should achieve the same result.

After that:

mk-host -j8 toolchain

and now we should be able to build anything we like:

mk -C bin/cat -j8

mk -j8 kernel KERNCONF=GENERIC

mk -j8 userland

Note mk will set MACHINE to $DEFAULT_MACHINE if set, or the host's native value.


Author:sjg@crufty.net
Revision:$Id: freebsd-meta-mode.txt,v 1e4c828042fb 2020-09-12 21:27:16Z sjg $