Symbols in the __all__ list of a module marked with "# export"
arent't currently added to __init__.py. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Letting python-tools.sh rewrite symbols is more robust than rewriting
an entire __init__.py with PY_INIT_FILTER in the including Makefile.
The latter can break in non-obvious ways if python-tools.sh changes
__init__.py's format.
Make python-tools.sh support --symbol-filter to remedy that. The
option takes an sed script which should expect a string of two
non-whitespace tokens: The module from which the symbol is imported,
and the name of the symbol in that module. It's output will then be
used as the symbol to be exported from __init__.py.
Also, support the PY_SYMBOL_FILTER variable in py-mod.mk. If it's
defined, it is used for --symbol-filter.
Signed-off-by: Jan Lindemann <jan@janware.com>
Make use of the newly introduced --prefix option to the pythonpath
command, and generate what's subseqently used to fill in mypy_path in
pyproject.toml.
By decoupling it from PYTHONPATH, this commit makes the creation of
mypy_path less involved and easier to understand. It also obviates
the need replace the relatively heavy ldlibpath.mk by the relatively
lightweight projects.mk, thereby enhancing performance.
Signed-off-by: Jan Lindemann <jan@janware.com>
pyproject.toml is currently copied unchanged from conf/topdir to the
toplevel directory. Set up machinery in py-topdir.mk to render it
from a template in conf/templates instead, replacing {mypypath} in
the process.
Signed-off-by: Jan Lindemann <jan@janware.com>
ruff tries to recursivley use every config file it finds and stumbles
over a template:
/usr/bin/ruff check --select TC,FA --fix --unsafe-fixes .
ruff failed
Cause: Failed to parse /home/jan/local/src/jw.dev/proj/jw-pkg/conf/templates/pyproject.toml
Cause: TOML parse error at line 3, column 3
|
3 | {mypypath}
| ^
invalid key-value pair, expected key
Limiting it to the toplevel pyproject.toml by explicitly specifying
--config fixes the behaviour, so that's what this commit does.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add $(TOPDIR)/conf/templates as a location for templates, i.e. input
files to the CmdCreateFile template rendering command.
Signed-off-by: Jan Lindemann <jan@janware.com>
Replace variable PY_SRC_ROOT by PY_CHECK_ROOTS. The name PY_SRC_ROOT
was a bad choice, given that it isn't immediately obvious that it a)
can contain multiple root locations to be checked, and that it b)
specifically concerns static type checking.
As of this commit, it's possible to limit the type checking scope
with PY_CHECK_ROOTS as in
PY_CHECK_ROOTS="src/python/jw/pkg/CmdBase.py src/python/jw/lib" \
make check
Signed-off-by: Jan Lindemann <jan@janware.com>
Running pyright in a minimal docker container gives this error:
$ pyright
/usr/bin/npm-default: No such file or directory
Traceback (most recent call last):
File "/usr/bin/pyright-3.13", line 6, in <module>
sys.exit(entrypoint())
~~~~~~~~~~^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 31, in entrypoint
sys.exit(main(sys.argv[1:]))
~~~~^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 18, in main
return run(*args, **kwargs).returncode
~~~^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/pyright/cli.py", line 22, in run
pkg_dir = install_pyright(args, quiet=None)
File "/usr/lib/python3.13/site-packages/pyright/_utils.py", line 69, in install_pyright
node.run(
~~~~~~~~^
'npm',
^^^^^^
...<5 lines>...
stderr=subprocess.PIPE if silent else sys.stderr,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/lib/python3.13/site-packages/pyright/node.py", line 144, in run
subprocess.run(node_args, **kwargs),
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 577, in run
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/npm', 'install', \
'pyright@1.1.409']' returned non-zero exit status 255.
This means that on openSUSE, python3-pyright tries to pull in
packages from the NPM registry. This increases the CI supply chain
attack surface inacceptably, so remove pyright from the release
prerequisites. That should be enough to remove it from the
prerequisites of target check as well and allow it to succeed.
The pyright check machinery itself remains useful, so keep it in
place for developers who install python3-pyright manually.
Signed-off-by: Jan Lindemann <jan@janware.com>
By the time projects-dir.mk is used during onboarding, it's already
cloned, and so is jw-pkg in all its glory. So better use a
ssh-wrapper.sh directly under jw-pkg's version control instead of
plainly generating one with echo some-script-logic > ssh-wrapper.sh.
This has the main benefit of allowing a more elaborate script. The
one added by this commit removes "-l user" from remotes which have a
standard-user@gitserver form, typically because they differentiate
users via their SSH pubkeys only, and which would deny access if both
-l user and standard-user@ were specified.
ssh-wrapper.sh still needs to be a target which is updated by a
recipe, because the version found in jw-pkg can't be trusted to be
executable during bootstrapping, because "make all" has not run, yet.
Signed-off-by: Jan Lindemann <jan@janware.com>
The git-get-pub does not have the same effect as the other git-get-%
targets, and this commit makes it.
The other git-get-% targets run pgit.sh, which rebases the current
branch onto the fetched branch, and git-get-pub doesn't. Since devops
merges contributor forges fast-forward without a merge-commit, the
pub remote's master needs to be the last to be rebased on, because
otherwise it will not allow to force-push the result.
As soon as multiple forges with protected master branches contribute,
fast-forward merging of the master branch will need to be abolished
anyway, and the release machinery will need an overhaul.
Signed-off-by: Jan Lindemann <jan@janware.com>
pkg.requires.os.release = python3-pyright breaks CI on Kali Linux. It
is present in the janware repos, but using those would cross a line:
jw-pkg must be buildable from the base repositories alone, so don't make
pyright mandatory for Debian, because that pulls it in for Kali, too.
Ironically, the Debian repo provides it. Which makes it obvious that
we will need another entry in the os cascade for Debian proper to
allow pulling in such packages on Debian.
Signed-off-by: Jan Lindemann <jan@janware.com>
Be prepared to not have working pyright. This is necessary, because
the next commit will remove it for Debian.
Signed-off-by: Jan Lindemann <jan@janware.com>
Target all should create all necessary files in topdir. Currently
they're only needed for static file checks, but they might well be
prerequistes for the build to succeed in the future, so make target
all depend on topdir.
Also, place target all before the block of includes, so that the
execution order is defined in topdir.mk rather than the included
snippets.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add two new targets, basically
py-check-annotation-imports:
ruff check --select TC,FA --fix --unsafe-fixes .
py-format-annotation-imports:
ruff format --select TC,FA --fix --unsafe-fixes .
They basically import statements merely used for annotation
only during type checking runs:
if TYPE_CHECKING:
import AirFrobnicator from frobnication
Signed-off-by: Jan Lindemann <jan@janware.com>
"make check" fails on Kali Linux, because the machinery lacks pieces:
- Add mypy, yapf3, python3-pyright to project.conf for Debian. The
pyright package is not in upstream Kali, but is now supplied by
the jw-foss repo.
- ifdef ruff out from "make check" in py-topdir.mk, because it's
too much work to get that to work from packages on Kali Linux for
now.
project.conf, py-topdir.mk: Make Kali work
Signed-off-by: Jan Lindemann <jan@janware.com>
This file can be included from toplevel Makefiles of projects which are part
of a jw-pkg project tree but don't want to provide necessary targets
themselves. The targets in this makefile simply do nothing. Feel free to add
functionality to whatever target seems useful to you.
Signed-off-by: Jan Lindemann <jan@janware.com>
make topdir doesn't reliably regenerate pyrightconfig.json because of
.SECONDEXPANSION. Adding a second dollar symbol $$(TD_GENERATE_FILES)
solves the problem.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-topdir.mk: Use ruff and yapf
- Use pyright for the target py-check-syntax
- Generate a $(TOPDIR)/pyrightconfig.json for that
- Add pyrightconfig-base.json because it's used by
pyrightconfig.json
- Add python3-pyright to pkg.requires.release, anticipating the use
of the py-syntax-check target by CI
Signed-off-by: Jan Lindemann <jan@janware.com>
- Use ruff and yapf for the targets py-check-syntax, py-format and
py-check-format.
- Add a pyproject.toml for those. It also includes configuration
for isort, albeit not being directly used in the linter targets.
- Make .gitignore igore that in newly created projects.
- Add ruff, yapf and isort to pkg.requires.release, anticipating
their use by CI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a file to be included from every toplevel Makefile of every repo
that owns Python code. It defines the following targets:
py-check: py-check-syntax py-check-format
py-check-syntax:
py-check-format:
py-format:
Currently, only mypy is run from py-check-syntax, but the targets are
meant as entry point for further linter jobs in the future.
There's also
py-format-assignments:
which is meant to add spaces around "=" in multi-line assignment
blocks, but, pending future experiments, I thing that target is going
to be removed again. Left in for now.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add py-ns-dir.mk. It's first intended use is inside the
$(TOPDIR)/src/python/jw directories of each package. These are to be
treated specially, because they contribute to the same namespace: jw.
This is true when installed, which means that no __init__.py should
be packaged from that directory, see the guide linked below.
And it's also true when not installed, in which case there has to be
a path-extending __init__.py in the directory. Normally, it should
work without __init__.py in that case, as well, and it does for
running Python code with PYTHONPATH pointing to the respective
package roots. However, pyright doesn't seem to pull in code
scattered over multiple locations without an old-style namespace
path-extender, so we're likely going to add (but not install) an
__init__.py there.
Not sure if it's going to be auto-generated py py-ns-dir.mk or
committed to version control, we'll see with more testing.
Here's where I have my info from, aside from experiments.
https://packaging.python.org/en/latest/guides/packaging-namespace-packages/
Signed-off-by: Jan Lindemann <jan@janware.com>
If $(wildcard py.typed) is found in a python module directory,
install it. py.typed should be used by every repo that declares
properly typed code, for jw-pkg that would be at the jw.pkg node, i.e
installed to <site-dir>/jw/jw-pkg/py.typed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add generic machinery to dynamically create files in $(TOPDIR). The
need arises because version controlled configuration files for
linters are going to be introduced.
For that, this commit introduces a variable $(TD_GENERATE_FILES),
which target all depends on, and which topdir.clean removes.
It defaults to another variable also introduced by this commit,
$(TD_COPY_FILES), which in turn defaults to $(TOPDIR)/conf/topdir.
This commit also adds support for JW_PKG_TOPDIR_COPY_PATH. It
supports a PATH-style syntax, which allows pointing to multiple
directories to be checked for source files. If they exist, they will
be appended to the files found in $(TOPDIR)/conf/topdir after
copying. Defining arbitray files to copy is not supported before
security implications during CI runs are better understood.
Having the copy prerequisites work comes at the cost of having to add
.SECONDEXPANSION. Since it's limited to the toplevel Makefile, I
suppose that's acceptable.
Signed-off-by: Jan Lindemann <jan@janware.com>
The __init__.py files as gnerated by python-tools.sh contain multiple
issues, fix them:
- Make the machinery fail if the same type name is imported from
different modules
- Support relative imports from .Module import Module instead of
having to use the entire module path as import source
- Import types explicitly re-exported with "as":
from .Module import Module as Module
Otherwise ruff will regard the type as "imported but not used"
- Add "# ruff: noqa: E501" near the top. The import lines can get
long and are beyond manual control (except for renaming the
modules themselves, that is). This can cause ruff to fail, so get
it to accept long lines in __init__.py. The style violation
doesn't make much of a difference in generated code, anyway,
because nobody reads that. Plus what's happening in the code
isn't rocket science, so good style wouldn't help much with
understanding, either.
This promptly digs up two symbol name conflicts lib.pm.dpkg and
lib.pm.rpm. Fix them along with this commit to keep it from breaking the build.
Signed-off-by: Jan Lindemann <jan@janware.com>
- Apply some style fixup to topdir.mk. Notably beautify indentation
and move the target area down below the area with the include
statements.
- Add the following targets:
format:
check-syntax:
check-format:
check: check-syntax check-format
These targets currently don't do anything and are meant as hooks
for individual linters to add dependencies.
- Make .PHONY depend on all phony non-pattern targets
- Make all phony pattern-targets depend on FORCE
Signed-off-by: Jan Lindemann <jan@janware.com>
Make pkg-release-reinstall depend on get-pub to do the obvious: Have
the changes from Forgejo pull requests integrated by CI, too.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove JW_PKG_EXTRA_SSH_OPTS before git pull in the context of
get-pub / git-get-pub because it contains -l username, which collides
with Forge-style git@<forge> remote URIs.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add get-pub for top directory / projects directory level. This is a
janware specific target needed for CI. It integrates the current
master branch from the new Forgejo-based Git repos.
That said, this will likely need to go along with other release
machinery. Packaging is good, but releasing over a bunch of
directories is an intricate process, as-is only usable by janware
itself, and doesn't need to be part of jw-pkg.
Signed-off-by: Jan Lindemann <jan@janware.com>
If a project is not initialized via make pkg-init-%, it doesn't
contain a VERSION file. When CI tries to build and package such a
project, it auto-creates a VERSION file, but a broken one: The
revision isn't properly seperated by a dash but by a dot, which makes
CI give up while parsing it.
Signed-off-by: Jan Lindemann <jan@janware.com>
janware upstream development server moved from git.janware.com to
devgit.janware.com. This commit follows the move with pretty much a
simple
s/git.janware.com/devgit.janware.com/
over jw-pkg. It found 14 matches, that's pretty bad.
FIXME: Reduce the redundancy, or, better, replace the
devgit.janware.com goodies by a more generally useful concept
altogether.
Signed-off-by: Jan Lindemann <jan@janware.com>
To make it easier to force-push feature branches, allow that, albeit
under --force-with-lease protection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a new target git-show-ahead-of-master, which does what
git-show-pushable did up to now: List all repos where the currently
checked-out branch is ahead of origin/master.
git-show-pushable now shows repos that have their checked-out branch
ahead of their respective upstream branches. Which is truer to what
the target's name suggests.
Signed-off-by: Jan Lindemann <jan@janware.com>
Check with ifndef if a variable is defined before hard defining it in
cache.mk. This gives more flexibility when overriding variables with
local.mk.
Signed-off-by: Jan Lindemann <jan@janware.com>
cleandirs.mk doesn't descend into subdirectories, because
defs-dirs.mk is not included, and, hence FIND_SUBDIRS is undefined.
Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename command "distro" to "pkg" together with "info", its last
remaining subcommand. "distro" is often used in the sense of "Linux
distribution", which would be too narrow for the targets jw-pkg could
theoretically support.
Signed-off-by: Jan Lindemann <jan@janware.com>
With the exception of the "info" subcommand, nearly all of distro's
subcommands deal with package managing, so push them into their own
command category.
Signed-off-by: Jan Lindemann <jan@janware.com>
pgit.sh get is not invoked with --create-remote-user-repos by
default. For the janware Git servers, this makes CI fail over new
repos: Repos from the maintainers are pulled, but before
pkg-rebuild-reinstall, repos that the devops user has no remote repo
for are purged again and are subseqently missing during build.
Passing --create-remote-user-repos to all pgit.sh get invocations
should fix the problem, so do that.
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdCanonicalizeRemotes / canonicalize-remotes and the respective
target in topdir.mk remove the /srv/git portion from all remotes'
URLs pointing to git.janware.com.
Signed-off-by: Jan Lindemann <jan@janware.com>
Installation type SCRIPT has beed disabled long ago because of its
overlap with EXE. Remove the dead code around it.
Signed-off-by: Jan Lindemann <jan@janware.com>
/usr/bin/file <candidate> | grep text is used to detect if a file is
a text file or not. Replace that with grep -I., because that adds
some files left out by /usr/bin/file, notably systemd service files.
Signed-off-by: Jan Lindemann <jan@janware.com>
PROJECTS_DIR_REMOTE_BASE and JANWARE_USER is determined by jw-pkg.py
get-auth-info. Interactivity hurts here, though, turn it off.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename git-show-pushable-master-branches to git-show-pushable,
because, the target's recipe doesn't show pushable master branches,
but all commits in the working directories current branch not present
in origin/master.
Signed-off-by: Jan Lindemann <jan@janware.com>
This reverts commit 88e81 and effectively re-enables persistent SSH
connections to speed up builds.
commit 88e8197ed7
Author: Jan Lindemann <jan@janware.com>
Date: Thu Apr 20 20:00:46 2017 +0000
projects-dir.mk: Partly disable persistent SSH
Don't use persistent SSH-connections any more for "pkg-" targets,
because, sadly, this hangs after uploading a package.
This commit puts some safeguards against hanging SSH into place,
namely setting default SSH timeouts down, SSH keepalive, setting SSH
BatchMode to yes.
Signed-off-by: Jan Lindemann <jan@janware.com>
Use DEP_PROJECTS instead of BUILD_PROJECTS everywhere. DEP_PROJECTS
considers more projects, and for all targets it is relevant to, it's
desirable to have them operate on the maximum blast radius.
Signed-off-by: Jan Lindemann <jan@janware.com>
make git-show-pushable-master-branches misses projects which should
better be pulled in as dependencies. It searches $(BUILLD_PROJECTS)
for repositories with pushable commits, that variable leaves some
out, and DEP_PROJECTS has them. It make use of the renovated
pkg-requires command.
This move should be extended to other uses of BUILD_PROJECTS as well,
after giving it some test runs. And the legacy and redundant command
prereq should be removed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Log to stderr and add some ASCII-art around the output. Also, add a
--porcelain option to allow more stable output parsing. Subsequently,
use that option in make targets parsing the output, notably make diff
and make git-show-xxx.
Signed-off-by: Jan Lindemann <jan@janware.com>
