Some uses of $(JW_PKG_PY) need to ensure their own topdir format,
which results in two options on jw-pkg.py's command line, with the
second overwriting the first. This works but is a minor uglyness.
Since the default built into jw-pkg works for all other cases, remove
--topdir-format from the JW_PKG_PY variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
BUILD_PKG_CONFIG installs files across project directories into
$(JWBDIR) because of this:
BUILD_PKG_CONFIGDIR = $(JWBDIR)/pkgconfig
The concept is currently not used by any downstream project anymore,
hence it lacks sufficient testing at this point. Moreover, it breaks
building repos which are sandboxed to their own root directory,
because it tries to install the generated .pc file back into
$(JWBDIR). The concept itself isn't entirely off, so leave the code
in, disabled, with a comment.
Signed-off-by: Jan Lindemann <jan@janware.com>
If $(CACHE_PROJECT_MK) AKA .cache-project.mk already exists but needs
to be remade because of dependencies, it is included, its content
read, then the content cached again to the same file. That way stale
variables can leak into the refreshed cache.
This commit fixes the situation by removing it, then invoking make
again to remake it, thereby clearing the variable database and
forcing make to fill the variables afresh.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for the targets "check" and "check-post" to
projects-dir.mk to make them usable from inside the projects
directory.
Note that "check-pre" is intentionally left out: Running "make check"
in a project before its prerequisite projects have built their
__init__.py files will fail due to broken import resolution.
Signed-off-by: Jan Lindemann <jan@janware.com>
BUILD_MAKEDIR is a variable which exists for consistency's sake. On
the other hand, nor jw-pkg nor any downstream package ever copies
makefile snippets during build time. The rule introduced by
BUILD_MAKEDIR is quite costly in terms of performance and makes
caching harder to understand. This commit disables the variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add target py-path to py-path.mk. At this point, it's mostly
introduced for documentation purposes, giving people and machines a
defined, easily accessible and omni-present way to determine how
Python imports should be resolved.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add target py-check-bad-patterns, which currently looks for leftover
breakpoints and orphaned "export" comments on the closing line of a
function prototype.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add py-check.mk and use it from py-topdir.mk and py-rules.mk. This
removes redundant check definitions, making sure that that checks run
from a repo's subdirectory match the checks run from its toplevel
directory.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-defs.mk uses the variables ECHO and SED defined in defs.mk. The
complexity this introduces doesn't justify the reduced redundancy,
though, so this commit defines them redundantly in py-defs.mk and to
allow inclusion without prior defs.mk.
Signed-off-by: Jan Lindemann <jan@janware.com>
For backwards compatibility, ldlibpath.mk kept py-path.mk included.
The projects depending on that have been fixed by now and this is no
longer needed. So, move it to the py-defs.mk where it belongs and has
narrower scope.
Signed-off-by: Jan Lindemann <jan@janware.com>
local/src is a janware-specific path, remove it from App and defs.mk.
It's still in pkg.sh as a safety measure. Will have to go, too, but
is kept in for now until further audit.
Signed-off-by: Jan Lindemann <jan@janware.com>
.mypy_cache, .ruff_cache and .pytest_cache are not consistently
cleaned in all subdirectories. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
During a topdir "make all", caching variables is currently not the
first thing that happens. Instead, variables are cached as soon as a
project recurses into the make subdirectory. That was necessary,
because some makefiles were regenerated in the make subdirectory by
autoconf, potentially contributing variables that needed to be
cached.
As of now, autoconf is long gone and this is no longer true. And for
some variables, the two step process becomes involved, notably for
PYTHONPATH, which coding agents would like to look at from the topdir
very early on.
This commit moves the .project-cache.mk creation to topdir.mk, and
.projects-cache.mk creation to jw-pkg/Makefile to address that.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-topdir.mk has this:
ifndef PY_CHECK_ROOTS
PY_CHECK_ROOTS += ...
endif
That is too involved: Either PY_CHECK_ROOTS is defined, then nothing
is appended, or it's undefined, then a simple "=" would be just fine.
Use that instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
If run from $(TOPDIR), "make clean all" runs fine, because it
recurses twice into $(TOPDIR)/make, once for every target. If invoked
directly from $(TOPDIR)/make, it can break in two different ways:
If the cache files don't exist, "make clean all" in $(TOPDIR)/make
tries to create them too early as implicit make target. This leaves
variables empty which should have a value.
If the cache files do exist, "make clean all" in $(TOPDIR)/make
includes them, cleans them, and re-creates them from the same
variables just read from cache. Undesirable for cache purging.
Signed-off-by: Jan Lindemann <jan@janware.com>
Generate a standard conftest.py, mostly for customizing pytest's
bombastic test header, which otherwise lets the more informative make
output look too pale in comparison.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a dedicated py-test.mk for running tests with pytest instead of
python.
The commit introduces a new variable PYTHON_RUNNER, which is then
used by py-run.mk to do it's usual thing. Running pytest is similar
enough to running Python programs to keep the rest of the machinery
and prevent redundancy from creeping in.
Signed-off-by: Jan Lindemann <jan@janware.com>
Every directory should have a test target, and the test target should
depend on all so that all files are generated that might be needed
for testing. This commit fixes some missing targets, and adds some
missing dependencies.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add include file to provide some definitions for a first integration
test suite. It provides the shared make variables, most notably
TEST_CMD_LINE as handy default for running jw-pkg.py.
Signed-off-by: Jan Lindemann <jan@janware.com>
Recursive make fails in scripts/usr-bin when driven through jw-pkg.
Add empty test targets to remedy that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the target sh-syntax-check, which triggers bash syntax linting
with shellcheck This commit stops short of making target all depend
on it, because the fallout is impressive.
Signed-off-by: Jan Lindemann <jan@janware.com>
To make EXE_SH useful for automated shell syntax checking, remove all
non POSIX shell / bash scripts from that variable, and place the
non-shell scripts into EXE_SCRIPTS.
Signed-off-by: Jan Lindemann <jan@janware.com>
Modifying JW_PKG_XXX_PATH in pre-local.mk is fragile.
Amending PREREQ_RUN in pre-local.mk works, but only with
$(JW_PKG_NO_CACHE) == true, or if "undefine JW_PKG_XXX_PATH" is also
added in pre-local.mk. Otherwise JW_PKG_XXX_PATH will not be
recomputed, because it's already defined from the cache.
Introduce the new variable PREREQ_RUN_ADD to solve that. If it's
defined, it automatically invalidates the JW_PKG_XXX_PATH variables
and sets them up for recalculation in py-path.mk / ldlibpath.mk.
Signed-off-by: Jan Lindemann <jan@janware.com>
Reverse inclusion order of .cache-project.mk and cache-projects.mk:
Definitions in .cache-project.mk should win over cache-projects.mk,
because it's the more specialized include file, and the way the
definitions in both files are structured, the later doesn't overwrite
the earlier.
Signed-off-by: Jan Lindemann <jan@janware.com>
The clean-dirs target does not only clean the repos present in
PROJECTS, but all repos it finds to be dirty, and clean-all-dirs does
the opposite. I suppose that was an oversight, swap their recipes.
Moreover, cleaning all directories goes about its business in an
overly complicated and unecessarily time-consuming way, fix that,
too.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-path.mk is the place to host PYTHONPATH and MYPYPATH definitions,
so move them there from py-defs.mk for consistency. Also, remove some
dead code.
Signed-off-by: Jan Lindemann <jan@janware.com>
The following variables contain colons as path-separators:
- JW_PKG_PYTHON_PATH
- JW_PKG_EXE_PATH
- JW_PKG_LD_LIBRARY_PATH
This commit makes them use spaces instead, so they can be more
easily amended by Makefiles using them. Also define them in a more
uniform way, and use the newly introduced PREREQ_RUN variable to fill
them, which in turn can also be appended to before that.
Signed-off-by: Jan Lindemann <jan@janware.com>
At present, the PREREQ-variable is effectively only used to detect if
prerequiste packages haven't run "make all" before make is run in a
given package. Also, it's only useful in $(TOPDIR). This commit
splits the variable up into PREREQ_BUILD and PREREQ_RUN, and makes
the variables available in every Makefile of a package by placing
them in defs.mk instead of topdir.mk.
This also fixes a problem that PREREQ was cached before being filled,
hence empty. Which effectively wasn't much of a problem, because it
was basically unused, but still.
Signed-off-by: Jan Lindemann <jan@janware.com>
cache.mk generates .project-cache.mk, and this commit supports
disabling the definitions in the generated cache by setting
JW_PKG_NO_CACHE=true.
Signed-off-by: Jan Lindemann <jan@janware.com>
make/Makefile is responsible to generate $(TOPDIR)/cache-projects.mk.
The variables are taken from .cache-project.mk, with some variables
intentionally omitted, but their ifndef / endif blocks remain in
place. Not harmful but ugly. Make sed range-delete the left-over
blocks entirely.
Signed-off-by: Jan Lindemann <jan@janware.com>
Aside from PYTHONPATH, ldlibpath.mk runs jw-pkg.py for determining
other paths, too, which is often unneeded and can impact performance.
Split the PYTHONPATH detection into a dedicated py-path.mk, and
include it from ldlibpath.mk, so it can be used instead where needed.
Signed-off-by: Jan Lindemann <jan@janware.com>
If make echo-py's output is accidentally subjected to shell
expansion, it can yield surprising results. Protect it from that
happening.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for PY_INIT_SUBMODULES to py-mod.mk. If it is defined in
a Makefile including py-mod.mk, the listed submodules will be added
to __init__.py and thus included in the list of things that can be
imported from a module.
This commit also adds support for --submodules to python-tools.sh for
that to happen.
Signed-off-by: Jan Lindemann <jan@janware.com>
Symbols in the __all__ list of a module marked with "# export"
arent't currently added to __init__.py. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Letting python-tools.sh rewrite symbols is more robust than rewriting
an entire __init__.py with PY_INIT_FILTER in the including Makefile.
The latter can break in non-obvious ways if python-tools.sh changes
__init__.py's format.
Make python-tools.sh support --symbol-filter to remedy that. The
option takes an sed script which should expect a string of two
non-whitespace tokens: The module from which the symbol is imported,
and the name of the symbol in that module. It's output will then be
used as the symbol to be exported from __init__.py.
Also, support the PY_SYMBOL_FILTER variable in py-mod.mk. If it's
defined, it is used for --symbol-filter.
Signed-off-by: Jan Lindemann <jan@janware.com>
Make use of the newly introduced --prefix option to the pythonpath
command, and generate what's subseqently used to fill in mypy_path in
pyproject.toml.
By decoupling it from PYTHONPATH, this commit makes the creation of
mypy_path less involved and easier to understand. It also obviates
the need replace the relatively heavy ldlibpath.mk by the relatively
lightweight projects.mk, thereby enhancing performance.
Signed-off-by: Jan Lindemann <jan@janware.com>
pyproject.toml is currently copied unchanged from conf/topdir to the
toplevel directory. Set up machinery in py-topdir.mk to render it
from a template in conf/templates instead, replacing {mypypath} in
the process.
Signed-off-by: Jan Lindemann <jan@janware.com>
ruff tries to recursivley use every config file it finds and stumbles
over a template:
/usr/bin/ruff check --select TC,FA --fix --unsafe-fixes .
ruff failed
Cause: Failed to parse /home/jan/local/src/jw.dev/proj/jw-pkg/conf/templates/pyproject.toml
Cause: TOML parse error at line 3, column 3
|
3 | {mypypath}
| ^
invalid key-value pair, expected key
Limiting it to the toplevel pyproject.toml by explicitly specifying
--config fixes the behaviour, so that's what this commit does.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add $(TOPDIR)/conf/templates as a location for templates, i.e. input
files to the CmdCreateFile template rendering command.
Signed-off-by: Jan Lindemann <jan@janware.com>
Replace variable PY_SRC_ROOT by PY_CHECK_ROOTS. The name PY_SRC_ROOT
was a bad choice, given that it isn't immediately obvious that it a)
can contain multiple root locations to be checked, and that it b)
specifically concerns static type checking.
As of this commit, it's possible to limit the type checking scope
with PY_CHECK_ROOTS as in
PY_CHECK_ROOTS="src/python/jw/pkg/CmdBase.py src/python/jw/lib" \
make check
Signed-off-by: Jan Lindemann <jan@janware.com>
Running pyright in a minimal docker container gives this error:
$ pyright
/usr/bin/npm-default: No such file or directory
Traceback (most recent call last):
File "/usr/bin/pyright-3.13", line 6, in <module>
sys.exit(entrypoint())
~~~~~~~~~~^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 31, in entrypoint
sys.exit(main(sys.argv[1:]))
~~~~^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 18, in main
return run(*args, **kwargs).returncode
~~~^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 22, in run
pkg_dir = install_pyright(args, quiet=None)
File "/usr/lib/python3.13/site-packages/pyright/_utils.py", line 69, in install_pyright
node.run(
~~~~~~~~^
'npm',
^^^^^^
...<5 lines>...
stderr=subprocess.PIPE if silent else sys.stderr,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/lib/python3.13/site-packages/pyright/node.py", line 144, in run
subprocess.run(node_args, **kwargs),
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/npm', 'install', \
'pyright@1.1.409']' returned non-zero exit status 255.
This means that on openSUSE, python3-pyright tries to pull in
packages from the NPM registry. This increases the CI supply chain
attack surface inacceptably, so remove pyright from the release
prerequisites. That should be enough to remove it from the
prerequisites of target check as well and allow it to succeed.
The pyright check machinery itself remains useful, so keep it in
place for developers who install python3-pyright manually.
Signed-off-by: Jan Lindemann <jan@janware.com>
By the time projects-dir.mk is used during onboarding, it's already
cloned, and so is jw-pkg in all its glory. So better use a
ssh-wrapper.sh directly under jw-pkg's version control instead of
plainly generating one with echo some-script-logic > ssh-wrapper.sh.
This has the main benefit of allowing a more elaborate script. The
one added by this commit removes "-l user" from remotes which have a
standard-user@gitserver form, typically because they differentiate
users via their SSH pubkeys only, and which would deny access if both
-l user and standard-user@ were specified.
ssh-wrapper.sh still needs to be a target which is updated by a
recipe, because the version found in jw-pkg can't be trusted to be
executable during bootstrapping, because "make all" has not run, yet.
Signed-off-by: Jan Lindemann <jan@janware.com>
The git-get-pub does not have the same effect as the other git-get-%
targets, and this commit makes it.
The other git-get-% targets run pgit.sh, which rebases the current
branch onto the fetched branch, and git-get-pub doesn't. Since devops
merges contributor forges fast-forward without a merge-commit, the
pub remote's master needs to be the last to be rebased on, because
otherwise it will not allow to force-push the result.
As soon as multiple forges with protected master branches contribute,
fast-forward merging of the master branch will need to be abolished
anyway, and the release machinery will need an overhaul.
Signed-off-by: Jan Lindemann <jan@janware.com>
