Dependency ordering error for multiuser work
The Makefile below should create (multiple) output directories and generate output in those directories, starting with input in the directory above. So dirn exists on login and dirn / file.foo exists. The assembly should create a file dirn / out / file.bar.
This Makefile runs as a single job when run (note that it creates the two required source directories and files in $(shell)
). This appears to work because it makedirs
is the first / leftmost precondition for all
. However, this doesn't work for creating multiple jobs (i.e. make -j4
/ whatever).
Any ideas on how to fix the dependencies to ensure that the output directories are created before they are executed?
EDIT
I should have made it clear that I had tried various pre-order-only solutions, but I could not do that and ensured that the target was actually rebuilt (the order-only point tends to prevent rebuilding rather than enforcing the ordering dependency). If you have an OO solution, check it out! Thank.
# expected output:
# made directories
# copying dir1/out/../file.foo to dir1/out/file.bar
# copying dir2/out/../file.foo to dir2/out/file.bar
# created all output files
# done
$(shell mkdir dir1 >& /dev/null; touch dir1/file.foo; \
mkdir dir2 >& /dev/null; touch dir2/file.foo)
OUTDIRS = dir1/out dir2/out
OUTPUTS = dir1/out/file.bar dir2/out/file.bar
.DEFAULT_GOAL := all
.PHONY: makedirs $(OUTDIRS)
.SUFFIXES: .foo .bar
%.bar : ../%.foo
@echo "copying $< to $@"
@cp $< $@
all : makedirs outputs
@echo "done"
outputs : $(OUTPUTS)
@echo "created all output files"
makedirs : $(OUTDIRS)
@mkdir -p $(OUTDIRS)
@echo "made directories"
clean :
@rm -rf dir1 dir2
source to share
make $(OUTPUTS)
have a dependency only by order of the directories themselves:
$(OUTDIRS) :
mkdir -p $@
$(OUTPUTS) : | $(OUTDIRS)
This will ensure that directories are created before $(OUTPUTS)
, but will not trigger recovery exits if the directories are newer than the targets (which is important, since the directories are timestamped every time a file is added to it ...).
Note: you can also add mkdir -p
to your output recipe that will create a directory if it doesn't already exist every time you run the output rule, but I prefer the above method.
Note2: in your existing makefile you can also just add the line: $(OUTPUTS): makedirs
which will cause the rule makedirs
to run before any of the outputs are built, but again, I preferred the solution above :)
---- EDIT: -----
Something weird, then what version of make are you using? I just ran the following, pay attention to sleep 1
when creating directories, which means if there was a concurrency issue it would definitely hit:
$(shell mkdir dir1 >& /dev/null; touch dir1/file.foo; \
mkdir dir2 >& /dev/null; touch dir2/file.foo)
OUTDIRS = dir1/out dir2/out
OUTPUTS = dir1/out/file.bar dir2/out/file.bar
.DEFAULT_GOAL := all
$(OUTPUTS) : | $(OUTDIRS)
$(OUTDIRS) :
@echo "making $@"
sleep 1
mkdir -p $@
@echo "done making $@"
%.bar : ../%.foo
@echo "copying $< to $@"
@cp $< $@
all : outputs
@echo "done $@"
outputs : $(OUTPUTS)
@echo "created all output files"
clean :
@rm -rf dir1 dir2
And the result was:
~/sandbox/tmp20> make -j -f Makefile2
making dir1/out
making dir2/out
sleep 1
sleep 1
mkdir -p dir1/out
mkdir -p dir2/out
done making dir1/out
done making dir2/out
copying dir1/out/../file.foo to dir1/out/file.bar
copying dir2/out/../file.foo to dir2/out/file.bar
created all output files
done all
Note that it creates dir1/out
and dir2/out
and does not run template rules at the same time until both run out. I have also confirmed that the solution mentioned in note 2 also works (at least on my machine ...).
When creating template rules, you can specify dependencies outside of the template so that you can:
foo.o: foo.h
%.o: %.c
recipe here...
which will rebuild foo.o
if foo.h
newer or try to build foo.h
if it doesn't exist before being created foo.o
.
source to share