janware/jw-pkg
Watch
1
Code Pull requests Activity Actions
jw-pkg/make/cache.mk

44 lines
1.4 KiB
Makefile
Raw Normal View History

make: Further improve build time This commit sees several improvements to the build performance: - Introduce cache.mk, which creates makefiles caching often used variables, per tree and per project. - Define more variables with := enclosed in condistions, instead of defining them with ?=, because the RHS of ?= is expanded deferredly. - Add more definitions for executables. - Move some more specialized definitions out into specialized makefiles, notably htdocs.mk and tmpl.mk Signed-off-by: Jan Lindemann <jan@janware.com>
2019-02-28 13:22:12 +00:00
# ----- define these variables
# CACHED_FILES = $(TOPDIR)/VERSION
# CACHED_VARS = VERSION
cache.mk: Include from topdir.mk, not make.mk During a topdir "make all", caching variables is currently not the first thing that happens. Instead, variables are cached as soon as a project recurses into the make subdirectory. That was necessary, because some makefiles were regenerated in the make subdirectory by autoconf, potentially contributing variables that needed to be cached. As of now, autoconf is long gone and this is no longer true. And for some variables, the two step process becomes involved, notably for PYTHONPATH, which coding agents would like to look at from the topdir very early on. This commit moves the .project-cache.mk creation to topdir.mk, and .projects-cache.mk creation to jw-pkg/Makefile to address that. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-07-09 07:44:35 +02:00
.PHONY: cache clean.cache
CACHED_FILES ?= $(VERSION_FILE) $(wildcard $(TOPDIR)/make/project.conf)
CACHED_VARS ?= PROJECT PREREQ_BUILD PREREQ_RUN PREREQ VERSION HEX_VERSION \
FULL_NAME WHICH PYTHON ECHO TEST BROWSER SED RM PWD ID CUT TR \
AWK GETENT XARGS FIND PRINTF PLATFORM_INFO OS_NAME_VERSION \
CAT BIN_INSTALL SUDO \
JW_PKG_LD_LIBRARY_PATH JW_PKG_EXE_PATH JW_PKG_PYTHON_PATH
make: Further improve build time This commit sees several improvements to the build performance: - Introduce cache.mk, which creates makefiles caching often used variables, per tree and per project. - Define more variables with := enclosed in condistions, instead of defining them with ?=, because the RHS of ?= is expanded deferredly. - Add more definitions for executables. - Move some more specialized definitions out into specialized makefiles, notably htdocs.mk and tmpl.mk Signed-off-by: Jan Lindemann <jan@janware.com>
2019-02-28 13:22:12 +00:00
make/[Makefile|*.mk): Improve variable caching This commit aims at improving speed by using better caching. - Makefile, cache.mk: Split .cache.mk up To allow caching of runtime path variables which are project-specific, split .cache.mk up in .cache-project.mk and .cache-projects.mk - ldlibpath.mk: Cache ldlibpath, exepath and pythonpath Place the output of $(call proj_query ldlibpath), $(call proj_query, exepath) and $(call proj_query pythonpath) in JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH respectively, and cache the variables in make/.project-cache.mk. - cache.mk: Use = instead of := Recursively expanded variables are nearly as fast as := variables if the assigned value is a fixed string. And sometimes it's not, rightly so, because variables get assigned below, as with JW_PKG_XXX for instance. - cache.mk: Use $(TOPDIR) as variable values Replace absolute references to project's topdir by $(TOPDIR) with sed. As soon as the project queries produce absolute paths, they will be transformed into relative paths which allow the code base to be moved to a different location and still remain functional without a rebuild. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-01-22 14:12:58 +01:00
CACHED_VARS_ONLY_PROJECT ?= \
PROJECT \
topdir.mk: Move PREREQ definition into defs.mk At present, the PREREQ-variable is effectively only used to detect if prerequiste packages haven't run "make all" before make is run in a given package. Also, it's only useful in $(TOPDIR). This commit splits the variable up into PREREQ_BUILD and PREREQ_RUN, and makes the variables available in every Makefile of a package by placing them in defs.mk instead of topdir.mk. This also fixes a problem that PREREQ was cached before being filled, hence empty. Which effectively wasn't much of a problem, because it was basically unused, but still. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-06-17 14:08:58 +02:00
PREREQ_BUILD \
PREREQ_RUN \
make/[Makefile|*.mk): Improve variable caching This commit aims at improving speed by using better caching. - Makefile, cache.mk: Split .cache.mk up To allow caching of runtime path variables which are project-specific, split .cache.mk up in .cache-project.mk and .cache-projects.mk - ldlibpath.mk: Cache ldlibpath, exepath and pythonpath Place the output of $(call proj_query ldlibpath), $(call proj_query, exepath) and $(call proj_query pythonpath) in JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH respectively, and cache the variables in make/.project-cache.mk. - cache.mk: Use = instead of := Recursively expanded variables are nearly as fast as := variables if the assigned value is a fixed string. And sometimes it's not, rightly so, because variables get assigned below, as with JW_PKG_XXX for instance. - cache.mk: Use $(TOPDIR) as variable values Replace absolute references to project's topdir by $(TOPDIR) with sed. As soon as the project queries produce absolute paths, they will be transformed into relative paths which allow the code base to be moved to a different location and still remain functional without a rebuild. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-01-22 14:12:58 +01:00
PREREQ \
VERSION \
HEX_VERSION \
FULL_NAME \
JW_PKG_LD_LIBRARY_PATH \
JW_PKG_EXE_PATH \
JW_PKG_PYTHON_PATH
cache.mk: Include from topdir.mk, not make.mk During a topdir "make all", caching variables is currently not the first thing that happens. Instead, variables are cached as soon as a project recurses into the make subdirectory. That was necessary, because some makefiles were regenerated in the make subdirectory by autoconf, potentially contributing variables that needed to be cached. As of now, autoconf is long gone and this is no longer true. And for some variables, the two step process becomes involved, notably for PYTHONPATH, which coding agents would like to look at from the topdir very early on. This commit moves the .project-cache.mk creation to topdir.mk, and .projects-cache.mk creation to jw-pkg/Makefile to address that. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-07-09 07:44:35 +02:00
# We need this to cache JW_PKG_PYTHON_PATH
include $(JWBDIR)/make/py-path.mk
Further improve build time: Try to avoid calling projects.py This commit tries to remove the necessity to call projects.py from $(TOPDIR) to speed up recursive builds over all projects yet again. This is a major undertaking. There are two variables which are filled py projects.py in $(TOPDIR): PREREQ and PREREQ_DIRS. Sadly, the latter is a path relative to $(TOPDIR)/make, so this is kind of pointless. Unless the cache is maintained in $(TOPDIR), a thing I tried to avoid. So this commit is only able to cache $(PREREQ), not $(PREREQ_DIRS), which still is a hassle. Introduced defs-dirs.mk for that, to make it accessible to make.mk, and modified all the other parts of the machinery, too. Signed-off-by: Jan Lindemann <jan@janware.com>
2019-02-28 14:48:57 +00:00
all: cache
cache.mk: Include from topdir.mk, not make.mk During a topdir "make all", caching variables is currently not the first thing that happens. Instead, variables are cached as soon as a project recurses into the make subdirectory. That was necessary, because some makefiles were regenerated in the make subdirectory by autoconf, potentially contributing variables that needed to be cached. As of now, autoconf is long gone and this is no longer true. And for some variables, the two step process becomes involved, notably for PYTHONPATH, which coding agents would like to look at from the topdir very early on. This commit moves the .project-cache.mk creation to topdir.mk, and .projects-cache.mk creation to jw-pkg/Makefile to address that. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-07-09 07:44:35 +02:00
clean: clean.cache
clean.cache:
make/[Makefile|*.mk): Improve variable caching This commit aims at improving speed by using better caching. - Makefile, cache.mk: Split .cache.mk up To allow caching of runtime path variables which are project-specific, split .cache.mk up in .cache-project.mk and .cache-projects.mk - ldlibpath.mk: Cache ldlibpath, exepath and pythonpath Place the output of $(call proj_query ldlibpath), $(call proj_query, exepath) and $(call proj_query pythonpath) in JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH respectively, and cache the variables in make/.project-cache.mk. - cache.mk: Use = instead of := Recursively expanded variables are nearly as fast as := variables if the assigned value is a fixed string. And sometimes it's not, rightly so, because variables get assigned below, as with JW_PKG_XXX for instance. - cache.mk: Use $(TOPDIR) as variable values Replace absolute references to project's topdir by $(TOPDIR) with sed. As soon as the project queries produce absolute paths, they will be transformed into relative paths which allow the code base to be moved to a different location and still remain functional without a rebuild. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-01-22 14:12:58 +01:00
rm -f $(CACHE_PROJECT_MK)
cache: $(CACHE_PROJECT_MK)
$(CACHE_PROJECT_MK): $(CACHED_FILES) $(JWBDIR)/make/cache.mk
cache.mk: Support JW_PKG_NO_CACHE cache.mk generates .project-cache.mk, and this commit supports disabling the definitions in the generated cache by setting JW_PKG_NO_CACHE=true. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-06-17 14:10:12 +02:00
@echo 'ifneq ($$(JW_PKG_NO_CACHE),true)' | tee $@.tmp
cache.mk: Don't overwrite defined variables Check with ifndef if a variable is defined before hard defining it in cache.mk. This gives more flexibility when overriding variables with local.mk. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-05-05 16:54:18 +02:00
@echo $(foreach v,$(CACHED_VARS),$v = $(value $(v)) EOL) | \
$(SED) 's/\s\+EOL/\n/g;' | \
$(SED) 's/\s*\(\S\+\)\s*=\s*\(.*\)\s*/ifndef \1\n \1 = \2\nendif\n/g;' | \
make/[Makefile|*.mk): Improve variable caching This commit aims at improving speed by using better caching. - Makefile, cache.mk: Split .cache.mk up To allow caching of runtime path variables which are project-specific, split .cache.mk up in .cache-project.mk and .cache-projects.mk - ldlibpath.mk: Cache ldlibpath, exepath and pythonpath Place the output of $(call proj_query ldlibpath), $(call proj_query, exepath) and $(call proj_query pythonpath) in JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH respectively, and cache the variables in make/.project-cache.mk. - cache.mk: Use = instead of := Recursively expanded variables are nearly as fast as := variables if the assigned value is a fixed string. And sometimes it's not, rightly so, because variables get assigned below, as with JW_PKG_XXX for instance. - cache.mk: Use $(TOPDIR) as variable values Replace absolute references to project's topdir by $(TOPDIR) with sed. As soon as the project queries produce absolute paths, they will be transformed into relative paths which allow the code base to be moved to a different location and still remain functional without a rebuild. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-01-22 14:12:58 +01:00
$(SED) 's|$(realpath $(TOPDIR))|$$(TOPDIR)|g' | \
$(SED) 's|$(realpath $(PROJECTS_DIR))|$$(PROJECTS_DIR)|g' | \
$(GREP) . | \
cache.mk: Support JW_PKG_NO_CACHE cache.mk generates .project-cache.mk, and this commit supports disabling the definitions in the generated cache by setting JW_PKG_NO_CACHE=true. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-06-17 14:10:12 +02:00
tee -a $@.tmp
@echo 'endif # ifneq ($$(JW_PKG_NO_CACHE),true)' | tee -a $@.tmp
make/[Makefile|*.mk): Improve variable caching This commit aims at improving speed by using better caching. - Makefile, cache.mk: Split .cache.mk up To allow caching of runtime path variables which are project-specific, split .cache.mk up in .cache-project.mk and .cache-projects.mk - ldlibpath.mk: Cache ldlibpath, exepath and pythonpath Place the output of $(call proj_query ldlibpath), $(call proj_query, exepath) and $(call proj_query pythonpath) in JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH respectively, and cache the variables in make/.project-cache.mk. - cache.mk: Use = instead of := Recursively expanded variables are nearly as fast as := variables if the assigned value is a fixed string. And sometimes it's not, rightly so, because variables get assigned below, as with JW_PKG_XXX for instance. - cache.mk: Use $(TOPDIR) as variable values Replace absolute references to project's topdir by $(TOPDIR) with sed. As soon as the project queries produce absolute paths, they will be transformed into relative paths which allow the code base to be moved to a different location and still remain functional without a rebuild. Signed-off-by: Jan Lindemann <jan@janware.com>
2026-01-22 14:12:58 +01:00
mv $@.tmp $@
Copy permalink