pkg-dist.mk queries the package relations with
--dont-expand-version-macros, and build_pkg() expanded the remaining
VERSION and VERSION-REVISION macros with a local sed-based
expand_version_macros(). That substitutes the version of the project being
built for every macro, instead of the version of the dependency the macro
refers to.
Drop --dont-expand-version-macros from the proj_query calls so the macros
resolve to the dependency's own version, and remove expand_version_macros()
from build_pkg(), which passes the PKG_* variables to rpmbuild as they are
queried.
Signed-off-by: Jan Lindemann <jan@janware.com>
The existence of CONFIG_SUBDIR hijacks INSTALL_CFGDIR to a subdirectory (by
default /etc/opt/<package>/$(CONFIG_SUBDIR)). Then std_install_rules apply,
and (the bent) INSTALL_CFGDIR is installed, but its parent directory, the
original $(INSTALL_CFGDIR), is not a prerequisite of install anylonger, and
has no rule anymore, hence log-install is never run for it. Instead it's
implicitly created by log-install -D $(INSTALL_CFGDIR).
This commit gives /etc/opt/<package> std-install rule and variables back as
CFGTOPDIR variants, and inserts it early into the install target's
prerequisite list.
Signed-off-by: Jan Lindemann <jan@janware.com>
During "make install", the packaging machinery should run "$(LOG_INSTALL)
-D /opt/<pkg>" but runs "mkdir -p /opt/<pkg>" instead. As a consequence,
the created directory is not owned by any of the created packages. This
commit fixes that by introducing INSTALL_PREFIXDIR and installing it like
all other directories.
Signed-off-by: Jan Lindemann <jan@janware.com>
Automatically include symbols of the form
MySymbol: TypeAlias = Something # export
in generated __init__.py.
Signed-off-by: Jan Lindemann <jan@janware.com>
For in-tree tests, the jw package is split across several projects, and
each directory containing a py-ns-dir.mk contributes its subtree. So far,
it's up to that directory how the contribution is handled. Usually an
__init__.py with pkgutil.extend_path() is present in version control to
glue these parts together. This commit makes py-ns-dir.mk handle the
contribution method centrally by default unless PY_UPDATE_INIT_PY is set to
false.
As of this commit, this is the case by default, i.e. PY_UPDATE_INIT_PY is
set to false in py-ns-dir.mk, maintaining the current behaviour.
Downstream projects or modules can decide to have __init__.py centrally
maintained. For this, they need to remove __init__.py from version control
and set PY_UPDATE_INIT_PY to true in the respective Makefile.
PY_UPDATE_INIT_PY = false is also set explicitly in src/python/jw/Makefile,
because that file should never be generated. Bootstrapping the entire
workspace hinges on it.
Pending better testing, most notably of consistent in-tree testing
functionality, PY_UPDATE_INIT_PY = true might become a global default in
the future.
Signed-off-by: Jan Lindemann <jan@janware.com>
Renovate the build-order and build-order-% targets as follows:
- Add BUILD_ORDER_DEP_FLAVOURS and default to all currently supported
build flavours: run,build,test,release. This adds convenience for
packages declaring differentiated types of dependencies, not only the
default build dependency, but still want any project it uses somehow
be be built before them.
- Don't print the recipe's command when running "make build-order", it
gets in the way if the caller wants to parse the result, so spare him
filtering it.
- Only log messages with priority warning or higher. Messages go to stderr
anyway to keep them from throwing wrenches into consuming parsers, but
they are likely confusingly visible without context if they leak to the
calling program's console.
Signed-off-by: Jan Lindemann <jan@janware.com>
SUBDIRS combines $(ORDERED_SUBDIRS) with the subdirectories found on disk.
An $(ORDERED_SUBDIRS) entry that does not exist in the tree is still passed
to the recursive make loop, which then errors out over the missing
directory.
Filter the ordered subdirectories through the wildcard of their Makefile
paths, keeping only the ones that are actually there.
Signed-off-by: Jan Lindemann <jan@janware.com>
LOCAL_SYSTEMD in defs.mk only collects the classic unit types, not
quadlet types - .container, .pod, .network, .volume, .image, .kube.
They're all systemd units that are installed under the system's systemd
directory and read by the podman-systemd-generator at boot.
Add all six quadlet extensions to the wildcard.
Signed-off-by: Jan Lindemann <jan@janware.com>
The meaning of the many PY_XXX variables isn't easy to tell apart, so
start renaming some of them.
PY_ALL_PY and PY_PYC consistently are now named PY_LOCAL|INSTALL_PY and
PY_LOCAL|INSTALL_PYC, depending on whether or not they are intended to
be installed.
The old PY_INSTALL_PY was renamed to PY_DO_INSTALLL_PY, a move that
would be better applied to all boolean variables. I won't try with this
commit, however, because a lot of downstream packages depend on
PY_UPDATE_INIT_PY and PY_INSTALL_INIT_PY, i.e. without "_DO".
Signed-off-by: Jan Lindemann <jan@janware.com>
py.clean unconditionally runs scm.sh clean -f __init__.py, which
deletes untracked __init__.py files in the source directory. In a
package with PY_UPDATE_INIT_PY=false the build never generates
__init__.py, as the generation rule in py-mod.mk is gated on the
same variable.
The __init__.py files in question are hand-written sources which are
always tracked in SCM, and for those, "scm.sh clean" becomes a no-op, so
with the current code, no real problem occurs, but explicitly protecting
__init__.py from deletion if PY_UPDATE_INIT_PY is false is certainly
more obvious.
Move the __init__.py cleanup inside the PY_UPDATE_INIT_PY=true
conditional so that py.clean only runs removal commands on files the
build actually generates.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
PY_ALL_PY is derived from PY_SRC_PY, which wildcards *.py in the module
directory, so a checked-in __init__.py enters the install list
unconditionally. PY_INSTALL_INIT_PY only gates the later append of
__init__.py, which covers files generated at build time. Setting it to
false thus had no effect on an existing __init__.py, which was still
installed together with its .pyc.
Filter __init__.py out of PY_ALL_PY. When PY_INSTALL_INIT_PY is true,
the existing append adds it back, so generated and checked-in
__init__.py files are installed as before. When it is false, an
existing __init__.py is now excluded from PY_ALL_PY and therefore also
from PY_INSTALLED_PY and PY_PYC.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
Add empty target "test" to keep the build from erroring out over make
test in documentation directories.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a development tree, $(INSTALL) falls back to plain install(1), which
refuses dead symlinks, so unprivileged "make install" fails outright.
Packaging is unaffected, because the spec exports INSTALL_LOG, which
already routes $(INSTALL) through "pkg.sh log-install"
Route the remaining $(INSTALL) defaults through $(LOG_INSTALL_SH) as
well to address the failures: $(SUDO) jw.pkg log-install -p for
DEVELOPMENT=false, plain jw.pkg log-install otherwise. Move -f $(PKG_FORMAT)
out of $(LOG_INSTALL_SH) and into the two logging call sites, so the
no-log paths don't carry a package format they never use.
Signed-off-by: Jan Lindemann <jan@janware.com>
Running "make clean" in some directories can break "make test" in
others. That notably happens by running clean somewhere in or below
$(TOPDIR)/src/python, which removes __init__.py files potentially needed
by unit and integration tests in $(TOPDIR)/test.
This commit makes the failure easily visible by introducing the notion
of "testabiltity" - if the tree is not testable, "make test" logs
exactly that in a clear error message and gives up.
Makefiles which need to check "testability" can use it as a
prerequisite. This commit does that with py-run.mk and jw-py-test.mk.
Testability is asserted by running "make", "make all" or "make test"
from $(TOPDIR). A successful toplevel build coincides with testability,
and leaves a $(TOPDIR)/dirs-all.done behind, which is why that file is
the perfect testability marker.
It is automatically created by a toplevel build and cleared by using the
"invalidate-testability" prerequisite. This commit makes target "clean"
depend on it in py-mod.mk. To be extended to other use cases / makefile
snippets as needed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't start a fresh project with version 1.0.0-0-dev. By default no
project can honestly claim the level of maturity suggested by 1.0.0-0
with its first commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
The diff-all and diff targets diff all projects in the workspace
without filtering.
Add a target "diff-projects". It sets PGIT_SH_PROJECTS to the list of
projects from build-order, limiting the diff to the dependency
closure around $(PROJECTS).
Signed-off-by: Jan Lindemann <jan@janware.com>
If /usr/bin/isort is found, run it during "make format" to get a
defined way the imports are sorted. tool.isort in pyproject.toml is
updated to match the other fixers.
Commit the fallout of this change. Running the other fixers alone
doesn't change the formatting, so this should be safe.
Signed-off-by: Jan Lindemann <jan@janware.com>
Some uses of $(JW_PKG_PY) need to ensure their own topdir format,
which results in two options on jw-pkg.py's command line, with the
second overwriting the first. This works but is a minor uglyness.
Since the default built into jw-pkg works for all other cases, remove
--topdir-format from the JW_PKG_PY variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
BUILD_PKG_CONFIG installs files across project directories into
$(JWBDIR) because of this:
BUILD_PKG_CONFIGDIR = $(JWBDIR)/pkgconfig
The concept is currently not used by any downstream project anymore,
hence it lacks sufficient testing at this point. Moreover, it breaks
building repos which are sandboxed to their own root directory,
because it tries to install the generated .pc file back into
$(JWBDIR). The concept itself isn't entirely off, so leave the code
in, disabled, with a comment.
Signed-off-by: Jan Lindemann <jan@janware.com>
If $(CACHE_PROJECT_MK) AKA .cache-project.mk already exists but needs
to be remade because of dependencies, it is included, its content
read, then the content cached again to the same file. That way stale
variables can leak into the refreshed cache.
This commit fixes the situation by removing it, then invoking make
again to remake it, thereby clearing the variable database and
forcing make to fill the variables afresh.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for the targets "check" and "check-post" to
projects-dir.mk to make them usable from inside the projects
directory.
Note that "check-pre" is intentionally left out: Running "make check"
in a project before its prerequisite projects have built their
__init__.py files will fail due to broken import resolution.
Signed-off-by: Jan Lindemann <jan@janware.com>
BUILD_MAKEDIR is a variable which exists for consistency's sake. On
the other hand, nor jw-pkg nor any downstream package ever copies
makefile snippets during build time. The rule introduced by
BUILD_MAKEDIR is quite costly in terms of performance and makes
caching harder to understand. This commit disables the variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add target py-path to py-path.mk. At this point, it's mostly
introduced for documentation purposes, giving people and machines a
defined, easily accessible and omni-present way to determine how
Python imports should be resolved.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add target py-check-bad-patterns, which currently looks for leftover
breakpoints and orphaned "export" comments on the closing line of a
function prototype.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add py-check.mk and use it from py-topdir.mk and py-rules.mk. This
removes redundant check definitions, making sure that that checks run
from a repo's subdirectory match the checks run from its toplevel
directory.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-defs.mk uses the variables ECHO and SED defined in defs.mk. The
complexity this introduces doesn't justify the reduced redundancy,
though, so this commit defines them redundantly in py-defs.mk and to
allow inclusion without prior defs.mk.
Signed-off-by: Jan Lindemann <jan@janware.com>
For backwards compatibility, ldlibpath.mk kept py-path.mk included.
The projects depending on that have been fixed by now and this is no
longer needed. So, move it to the py-defs.mk where it belongs and has
narrower scope.
Signed-off-by: Jan Lindemann <jan@janware.com>
local/src is a janware-specific path, remove it from App and defs.mk.
It's still in pkg.sh as a safety measure. Will have to go, too, but
is kept in for now until further audit.
Signed-off-by: Jan Lindemann <jan@janware.com>
.mypy_cache, .ruff_cache and .pytest_cache are not consistently
cleaned in all subdirectories. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
During a topdir "make all", caching variables is currently not the
first thing that happens. Instead, variables are cached as soon as a
project recurses into the make subdirectory. That was necessary,
because some makefiles were regenerated in the make subdirectory by
autoconf, potentially contributing variables that needed to be
cached.
As of now, autoconf is long gone and this is no longer true. And for
some variables, the two step process becomes involved, notably for
PYTHONPATH, which coding agents would like to look at from the topdir
very early on.
This commit moves the .project-cache.mk creation to topdir.mk, and
.projects-cache.mk creation to jw-pkg/Makefile to address that.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-topdir.mk has this:
ifndef PY_CHECK_ROOTS
PY_CHECK_ROOTS += ...
endif
That is too involved: Either PY_CHECK_ROOTS is defined, then nothing
is appended, or it's undefined, then a simple "=" would be just fine.
Use that instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
If run from $(TOPDIR), "make clean all" runs fine, because it
recurses twice into $(TOPDIR)/make, once for every target. If invoked
directly from $(TOPDIR)/make, it can break in two different ways:
If the cache files don't exist, "make clean all" in $(TOPDIR)/make
tries to create them too early as implicit make target. This leaves
variables empty which should have a value.
If the cache files do exist, "make clean all" in $(TOPDIR)/make
includes them, cleans them, and re-creates them from the same
variables just read from cache. Undesirable for cache purging.
Signed-off-by: Jan Lindemann <jan@janware.com>
Generate a standard conftest.py, mostly for customizing pytest's
bombastic test header, which otherwise lets the more informative make
output look too pale in comparison.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a dedicated py-test.mk for running tests with pytest instead of
python.
The commit introduces a new variable PYTHON_RUNNER, which is then
used by py-run.mk to do it's usual thing. Running pytest is similar
enough to running Python programs to keep the rest of the machinery
and prevent redundancy from creeping in.
Signed-off-by: Jan Lindemann <jan@janware.com>
Every directory should have a test target, and the test target should
depend on all so that all files are generated that might be needed
for testing. This commit fixes some missing targets, and adds some
missing dependencies.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add include file to provide some definitions for a first integration
test suite. It provides the shared make variables, most notably
TEST_CMD_LINE as handy default for running jw-pkg.py.
Signed-off-by: Jan Lindemann <jan@janware.com>
Recursive make fails in scripts/usr-bin when driven through jw-pkg.
Add empty test targets to remedy that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the target sh-syntax-check, which triggers bash syntax linting
with shellcheck This commit stops short of making target all depend
on it, because the fallout is impressive.
Signed-off-by: Jan Lindemann <jan@janware.com>
To make EXE_SH useful for automated shell syntax checking, remove all
non POSIX shell / bash scripts from that variable, and place the
non-shell scripts into EXE_SCRIPTS.
Signed-off-by: Jan Lindemann <jan@janware.com>
Modifying JW_PKG_XXX_PATH in pre-local.mk is fragile.
Amending PREREQ_RUN in pre-local.mk works, but only with
$(JW_PKG_NO_CACHE) == true, or if "undefine JW_PKG_XXX_PATH" is also
added in pre-local.mk. Otherwise JW_PKG_XXX_PATH will not be
recomputed, because it's already defined from the cache.
Introduce the new variable PREREQ_RUN_ADD to solve that. If it's
defined, it automatically invalidates the JW_PKG_XXX_PATH variables
and sets them up for recalculation in py-path.mk / ldlibpath.mk.
Signed-off-by: Jan Lindemann <jan@janware.com>
Reverse inclusion order of .cache-project.mk and cache-projects.mk:
Definitions in .cache-project.mk should win over cache-projects.mk,
because it's the more specialized include file, and the way the
definitions in both files are structured, the later doesn't overwrite
the earlier.
Signed-off-by: Jan Lindemann <jan@janware.com>
The clean-dirs target does not only clean the repos present in
PROJECTS, but all repos it finds to be dirty, and clean-all-dirs does
the opposite. I suppose that was an oversight, swap their recipes.
Moreover, cleaning all directories goes about its business in an
overly complicated and unecessarily time-consuming way, fix that,
too.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-path.mk is the place to host PYTHONPATH and MYPYPATH definitions,
so move them there from py-defs.mk for consistency. Also, remove some
dead code.
Signed-off-by: Jan Lindemann <jan@janware.com>
