← Back to home

Makefile notes

Just some useful notes on Makefiles.

Here’s some working notes on how I keep Makefiles. This isn’t so much a tutorial as it is a set of patterns that make sense to me for monorepo projects, or even just large projects.

There’s a subtle difference between variables and prerequisite expansion. It expands variables in the order it finds them, targets and prerequisites immediately, but rules are deferred. So you can’t do things like using variables as prerequisite names, or grouping variable dependencies into variables, unless you want to explicitly define orders of those dependencies.

A better explanation of this is https://blog.summercat.com/on-gnu-make-prerequisites-expansion.html.

Difference between := and = is that the first one is normal, and the second is recursive, evaluating each use.

See https://stackoverflow.com/questions/4879592/whats-the-difference-between-and-in-makefile.

Turns out .PHONY targets aren’t super great, because they run every time. See https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html for more info. The fix is to just not use them, use the real thing,

app_source := $(wildcard src/**.*)
target/app.bin: $(app_source)
	@echo "This is where you build." > target/app.bin
build-app: target/app.bin
    @# intentionally blank, proxy for prerequisite.

I’ve learned a huge amount from https://makefiletutorial.com, and it’s the first place I look when I inevitably can’t remember the difference between $? and $^.

There’s a certain amount of work that goes into maintaining a makefile. If it starts to get out of control, then it’s a sign that you have too many people touching it, or too many projects, and you might need to restructure them.

Dropping into bash is messy but possible, and ultimately super useful. At times, it’s just straight up easier to write bash scripts for a given library, and then call that with Make instead. For example, grabbing source files for a target using find, grep, or ls is a bit cleaner than using GNU-Make’s wildcard and adjacent functions.

Or it’s just cleaner to chain build commands together inside a shell script using &&, eliminating the need to get too clever.

I’m a fan of defining custom reusable shell scripts inside variables, and then calling them. For example, when I build a target I like to get some useful information about it, including timing. For this I use:

# Print metadata in table-like format for a target.
# $1 = source to hash
# $2 = target to hash
# $3, optional = start of build time in unix epoch ms
# $4, optional = end of build time in unix epoch ms
METADATA=zsh -c '\
paste -d"\n" \
<(echo time   \\t$$(echo $$(( 0$$4 - 0$$3 ))ms)) \
<(echo source \\t$$(find ./$$1 -type f -exec md5sum {} + | sort | md5sum | cut -d" " -f1-1)) \
<(echo target \\t$$(find ./$$2 -type f -exec md5sum {} + | sort | md5sum | cut -d" " -f1-1)) \
<(du -ha target/$$1  | sort -k 2) \
  | column -t \
  | sed "s/^/    /" \
\
' METADATA

# Example of use.
prototype_source := src/prototype.go
prototype_target := target/prototype.bin
$(prototype_target): $(prototype_source)
	@echo "Building prototype..." && \
      start=${NOW} && \
      @go build src/prototype.go && \
      end=${NOW} && \
      ${METADATA} ${prototype_source} ${prototype_target} $${start} $${end}

So then I get something like this out:

Building prototype...
    time    89ms
    source  c19ab4871ba2b018fe1ac1182a0110db
    target  229b50c6bdf4a49aaf53019161cb6b17
    72K     target/prototype.bin

I’m also a fan of using shell scripts as templates, and then just including them in our rule.

# Get unix epoch ms. Mostly use this as a "template" inside a
# rule, rather than executing it when building the rules,
# because we want the real wall clock time when we call it.
# Doing this with Node because macOS's `date` is not the same
# as linux's `date`. (See more info below.)
NOW=`node -e 'process.stdout.write((+(new Date())).toString())'`

# Example of use.
tool_source := src/tool.go
target/tool.bin: $(tool_source)
	@echo "Building tool..." && \
      start=${NOW} && \
      @go build src/tool.go && \
      end=${NOW} && \
      echo "start  $${start}" && \
      echo "end    $${end}"

Should output something like this.

Building tool...
start  1666894388209
end    1666894389646

You could also use perl to do the epoch timestamp. perl -MTime::HiRes=time -E 'say int(time * 1000)'

I like to use my Makefile as a definition for environmental variables, so I know what they are, and how they’re used. Some builds need to have env variables included in them, but bound at build time, like a runtime config shipped with a JS app. To achieve that I use a shell sed include/export configuration to bind them based on what environment we’re targeting (eg: production, development). Then we can include out .env files in our build pipeline.

# ==============================================================
# env variables definitions
# ==============================================================
ENV ?= development
# Documentation here about OAuth client ID.
GOOGLE_OAUTH_CLIENT_ID := "SECRET"
# Documentation here about OAuth client secret.
GOOGLE_OAUTH_CLIENT_SECRET := "SECRET"

# ==============================================================
# environmental includes
# ==============================================================
include .env.$(ENV)
export $(shell sed 's/=.*//' .env.$(ENV))

I like to get programmatic with my -all rules. It’s easier to build all targets with a similar pattern by searching the Makefile itself, rather than listing them manually. Here’s a rough example, assuming we have multiple targets following the naming pattern build-{name}.

all_targets := $(shell grep "^build-" Makefile \
    | awk -F: '{print $$1}' \
    | sed '/build-all/d' \
    | uniq)
all_targets_blank_target := target/all.txt
$(all_targets_blank_target):
	@echo "Building all..." \
	    $(foreach f,$(all_targets), \
	      "\n    $(subst build-,,$(f))" $(__EXEC))
	@date > target/all.txt
build-all: $(all_targets_blank_target) $(all_targets)
# intentionally blank, proxy for prerequisite.

It’s a little rough, because it produces target/all.txt, but that’s just to be sure all targets have some output.

The wildcard function is nice, but often times I want to use a recursive wildcard. John Graham-Cumming has a good post about using recursive wildcards at https://blog.jgc.org/2011/07/gnu-make-recursive-wildcard-function.html. It’s simple enough, but it requires indicating a directory first. And anyway, I often want to just recursively include all files. For this it’s just easier to use a shell alias to find.

find=$(shell find $1 -type f)
# ...
source_files := $(call find,src)

Some links:

makefile | code
2022-10-27