App.__get_project_refs() recurses into the package dependency graph, but
the recursion end conditions only work well if all required packages are
present. Whether a required package is installed or not is decided upon a
wrong condition, however - existence of the project's project directory,
which may or may not be misleading for both the -devel and the -run
requirements. This can lead to various unwanted outcomes: Missing packages
happily inserted into the recursion buffer, process termination instead of
recursion stop, path lookup errors instead of a clearer "unmet dependency"
message (No project path found for module xyz, Failed to find directory of
project foo).
This commit cleanly detects if -run or -devel are installed and makes the
walk stop or raise under the appropriate conditions:
1. -run missing but required, raises unmet dependency
2. -devel missing but required, raises unmet dependency
3. -run present, no -devel required, stop recursing
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The installed state of a project is judged by the proofs of installation,
which differ per subpackage: make/project.conf in the dev tree or /opt
proves the -devel package or a buildable checkout, a VERSION file in the
project directory or /usr/share/doc/packages the -run package.
Add is_installed(), which answers the proof question for a given subpackage
by searching the per-subpackage locations.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
The iteration order contains every project the dependency walk resolved,
including projects that are only installed as -devel packages below /opt.
Running the target in such a project ran make in a read-only, source-less
directory and failed the run.
Resolve each module against the workspace root only, and skip the target
with a notice when there is no buildable project directory. Factor the skip
notice into log_skip() so the platform-exclusion path shares it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
find_dir() and its helpers resolve a project name against the workspace
root and /opt. The build iteration needs to resolve a module against the
workspace only, to tell buildable projects apart from installed ones.
Add a projs_roots parameter to find_dir(), __find_dir() and __proj_dir()
that restricts the search to the given roots. With the parameter omitted
the previous behavior is unchanged.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
pkg-dist.mk queries the package relations with
--dont-expand-version-macros, and build_pkg() expanded the remaining
VERSION and VERSION-REVISION macros with a local sed-based
expand_version_macros(). That substitutes the version of the project being
built for every macro, instead of the version of the dependency the macro
refers to.
Drop --dont-expand-version-macros from the proj_query calls so the macros
resolve to the dependency's own version, and remove expand_version_macros()
from build_pkg(), which passes the PKG_* variables to rpmbuild as they are
queried.
Signed-off-by: Jan Lindemann <jan@janware.com>
The lib.version module carries no documentation: the Syntax, Component and
Boundary types, the Version and Dependency classes and their public methods
are bare, and the compatibility tiers the next_* stepping methods implement
are not documented anywhere.
Add docstrings: a line each for the base.py types, the compatibility
tier table in the Version class, and a short description for every
public method, with the three next_* methods citing the tier each one
steps to.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
- The pkg_relations() function is a horribly bad read. It contains a
mind-bending amount of interwoven case distinctions that are not clearly
reflected in the participating variable and function names. Much of it is
version handling. That's intricate by nature, but much of it has now been
implemented in the lib.version module, so moving the logic there makes
the function a good deal more readable than before. That's most of what
this commit does.
- Use version syntax macros from lib.version instead of
pkg_relations-defined macros, and fix the fallout in BaseCmdPkgRelations
and CmdCreateFile.
- Use lib.version also as a central place for dep string parsing from App.
Signed-off-by: Jan Lindemann <jan@janware.com>
pkg_relations() parses dependency specifications such as 'foo-devel >=
1.2.3-4' with inline regular expressions and mixes the parsing with version
macro expansion and rendering in three different syntaxes.
This commit adds the lib.version module that owns the parsing of a single
specification: the base and full name and the version boundary with its
operator. Rendering happens in constraint_str() in SEM_VER, DEBIAN or
NAMES_ONLY syntax, with untemplated, include_revision, as_range,
no_subpackages and quote options. The VERSION, VERSION-REVISION and
REVISION macros are resolved through a version lookup callback,
lib.version.Version.parse_deps_spec() splits comma-separated specification
strings into Version objects, and Version breaks a version down into its
major, minor, micro and revision parts.
Unit tests cover parsing, macro resolution, rendering in all syntaxes and
the range expansion.
The module is designed to be integrated into the status quo and
intentionally does not address a couple of further TODOs. Notably
version.Syntax.SEM_VER is intended to replace
pkg_relations.VersionSyntax.SemVer, but neither are really semantic
versioning according to spec. They are close but more targeted toward RPM
and Debian package versioning. And the naming and semantics differ from
SemVer.
Naming of the four components (major and minor version) is identical,
semantic meaning aside, there's no disagreement between real SemVer,
Debian, RPM and jw-pkg. The third and fourth component deviate:
SemVer Debian RPM jw-pkg lib.version
3 PATCH RELEASE MICRO
4 PRERELEASE Revision Release REVISION REVISION
TODOs: RELEASE over the rest of jw-pkg needs to be adjusted by a later
commit. Syntax.SEM_VER should also be renamed, to Syntax.JW_PKG maybe,
because, as said, it's not SemVer and will probably never be - SemVer
doesn't provide the compatibility guarantees that the jw-pkg versioning
scheme offers. To be documented.
Signed-off-by: Jan Lindemann <jan@janware.com>
The existence of CONFIG_SUBDIR hijacks INSTALL_CFGDIR to a subdirectory (by
default /etc/opt/<package>/$(CONFIG_SUBDIR)). Then std_install_rules apply,
and (the bent) INSTALL_CFGDIR is installed, but its parent directory, the
original $(INSTALL_CFGDIR), is not a prerequisite of install anylonger, and
has no rule anymore, hence log-install is never run for it. Instead it's
implicitly created by log-install -D $(INSTALL_CFGDIR).
This commit gives /etc/opt/<package> std-install rule and variables back as
CFGTOPDIR variants, and inserts it early into the install target's
prerequisite list.
Signed-off-by: Jan Lindemann <jan@janware.com>
During "make install", the packaging machinery should run "$(LOG_INSTALL)
-D /opt/<pkg>" but runs "mkdir -p /opt/<pkg>" instead. As a consequence,
the created directory is not owned by any of the created packages. This
commit fixes that by introducing INSTALL_PREFIXDIR and installing it like
all other directories.
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commit reads __abstractmethods__ via a getattr() with an empty
frozenset fallback, because in mypy 2.3.1 the scanned classes are typed as
type[object], which does not declare the attribute.
Add is_abc_class() as a TypeGuard, and skip the classes it rejects with a
debug line: LoadTypes now yields only ABC-derived classes. The guard
narrows to the classes that carry the attribute, so that the debug line can
now read __abstractmethods__ directly.
The command loaders are unaffected: every class they load is derived from
AbstractCmd, and hence from ABC. For loads that rely on the name filter
alone, plain classes are now skipped instead of being yielded.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
LoadTypes._classes() reads the __abstractmethods__ attribute of each class
that inspect.getmembers() returns for its debug output. mypy 2.2.0 allows
this, mypy 2.3.1 doesn't: It now types those classes as type[object]
instead of Any. The attribute itself is only declared on the ABCMeta
metaclass, so the direct access fails the type check.
Read the attribute through getattr() with an 'unknown' fallback string, and
annotate the variable explicitly. The runtime behavior is unchanged, since
the attribute exists whenever inspect.isabstract() is true, and the call
satisfies the checker.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: user.email <jan@janware.com>
_connect_kwargs(hide_secrets = True) is used to log the connection
parameters when a connection fails, without leaking the password. The
filtered dictionary is built before the password is replaced with
'<hidden>', and the replacement is applied to the local kwargs
dictionary afterwards, after the filtered copy has already been made.
The dictionary that ends up in the log therefore still contains the
real password.
Hide the password before building the filtered dictionary.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_put() catches a failure of the remote command sequence and logs
"Failed to get <path> from <root>", a phrase copied over from the
get() path. This path, however, pushes content, so the message
describes the wrong operation.
Log a put-oriented message instead.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_is_dir() logs a DEBUG message when _stat() is not implemented and it
must guess from the trailing slash whether a path is a directory. The
second part of that message is a plain string rather than an f-string,
so '{path}' is logged verbatim instead of the path.
Make it an f-string so the path is interpolated.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_put() writes the content into a temporary file with tee and, when
atomic is set, moves it to the target path with a final mv. The loop
over the command list sets tmp_file to None after each command, on the
assumption stated in the comment that the file has been moved at that
point - which is only true for the last command. When a chown, chmod,
or mv after the tee fails, the finally block finds tmp_file is None
and leaves the temporary file behind.
Reset tmp_file only after all commands have completed, so that the
finally block erases the temporary file whenever a step fails.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_run_ssh() waits for the remote process's exit status before reading
stdout and stderr. When a command produces more output than the
channel's flow-control window can hold, the server stops sending, the
remote process blocks on its write and never exits, and
recv_exit_status() blocks forever.
Drain stdout and stderr to EOF first - the channels close when the
process exits, so reading them also implies completion - and only
then query the exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_run_ssh() writes cmd_input to the remote stdin channel but never
closes the write side. The channel stays open until the client
process exits, so a remote command that reads stdin (cat, a login
shell, ...) never sees EOF and blocks forever - including for
cmd_input = None, which is supposed to mean non-interactive with
stdin from /dev/null.
Call shutdown_write() on the channel after writing the input, or
immediately when there is none, so the remote command gets EOF.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
__client() connects using only the URI's hostname and username. The
URI's port is ignored, so ssh://host:2222/... ends up connecting to
port 22, and a password carried in the URI is never passed to
paramiko, so URI-based password authentication cannot work. The Exec
and AsyncSSH clients both honor the port and the password.
Pass the port and the password to connect(). The port argument is
omitted entirely when the URI carries no port, because
getaddrinfo() would interpret a None port as service port 0; without
the argument, paramiko falls back to its default of 22.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
__new_with_path() joins base and path with exactly one '/', assuming a
trailing '/' in base is a path separator. That assumption breaks for
URIs with empty authority, where scheme_plus_authority ends in '://'
(e.g. 'file:///tmp/x'): new_replace_path('/etc/hosts') drops the
leading slash of the path and produces 'file://etc/hosts', which
parses with hostname 'etc' and path '/hosts' instead of path
'/etc/hosts'.
Treat a base ending in '://' separately and keep a leading slash in
the path, adding one if it is missing.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
Assembling a scheme-bearing form always writes '://' after the scheme, even
if the input has no authority part. For a schemeless relative path such as
'../../local/path', .full then becomes 'file://../../local/path', which
re-parses with '..' as the host and '/../local/path' as the path, both
entirely broken.
Fix __assemble() to write a bare ':' instead of '://' when the input has no
host and a relative path, so the assembled form re-parses back to the same
relative path: full of '../../local/path' is now 'file:../../local/path'.
Absolute paths and forms with an authority are unchanged.
The assembled 'file:../../local/path' form is a valid URI under the RFC
3986 generic grammar (scheme with a rootless path), while RFC 8089's file
URI ABNF admits only an empty or an absolute path. Uri thus trades RFC 8089
conformance for RFC 3986 compatibility: a relative path survives the
full-and-reparse cycle unchanged.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
Automatically include symbols of the form
MySymbol: TypeAlias = Something # export
in generated __init__.py.
Signed-off-by: Jan Lindemann <jan@janware.com>
For in-tree tests, the jw package is split across several projects, and
each directory containing a py-ns-dir.mk contributes its subtree. So far,
it's up to that directory how the contribution is handled. Usually an
__init__.py with pkgutil.extend_path() is present in version control to
glue these parts together. This commit makes py-ns-dir.mk handle the
contribution method centrally by default unless PY_UPDATE_INIT_PY is set to
false.
As of this commit, this is the case by default, i.e. PY_UPDATE_INIT_PY is
set to false in py-ns-dir.mk, maintaining the current behaviour.
Downstream projects or modules can decide to have __init__.py centrally
maintained. For this, they need to remove __init__.py from version control
and set PY_UPDATE_INIT_PY to true in the respective Makefile.
PY_UPDATE_INIT_PY = false is also set explicitly in src/python/jw/Makefile,
because that file should never be generated. Bootstrapping the entire
workspace hinges on it.
Pending better testing, most notably of consistent in-tree testing
functionality, PY_UPDATE_INIT_PY = true might become a global default in
the future.
Signed-off-by: Jan Lindemann <jan@janware.com>
There's a random string "some-random-string-to-id-this-makefile" in
projects-dir-minimial.mk intended to check if the Makefile in the
projects-directory is actually the final link target. It's re-used in
pgit.sh to find the projects toplevel directory, but since it doesn't need
to be an actual projects-dir-minimal.mk copy, another case should also be
taken into consideration: Check if projects-dir.mk is included.
Signed-off-by: Jan Lindemann <jan@janware.com>
Renovate the build-order and build-order-% targets as follows:
- Add BUILD_ORDER_DEP_FLAVOURS and default to all currently supported
build flavours: run,build,test,release. This adds convenience for
packages declaring differentiated types of dependencies, not only the
default build dependency, but still want any project it uses somehow
be be built before them.
- Don't print the recipe's command when running "make build-order", it
gets in the way if the caller wants to parse the result, so spare him
filtering it.
- Only log messages with priority warning or higher. Messages go to stderr
anyway to keep them from throwing wrenches into consuming parsers, but
they are likely confusingly visible without context if they leak to the
calling program's console.
Signed-off-by: Jan Lindemann <jan@janware.com>
SUBDIRS combines $(ORDERED_SUBDIRS) with the subdirectories found on disk.
An $(ORDERED_SUBDIRS) entry that does not exist in the tree is still passed
to the recursive make loop, which then errors out over the missing
directory.
Filter the ordered subdirectories through the wildcard of their Makefile
paths, keeping only the ones that are actually there.
Signed-off-by: Jan Lindemann <jan@janware.com>
LOCAL_SYSTEMD in defs.mk only collects the classic unit types, not
quadlet types - .container, .pod, .network, .volume, .image, .kube.
They're all systemd units that are installed under the system's systemd
directory and read by the podman-systemd-generator at boot.
Add all six quadlet extensions to the wildcard.
Signed-off-by: Jan Lindemann <jan@janware.com>
The meaning of the many PY_XXX variables isn't easy to tell apart, so
start renaming some of them.
PY_ALL_PY and PY_PYC consistently are now named PY_LOCAL|INSTALL_PY and
PY_LOCAL|INSTALL_PYC, depending on whether or not they are intended to
be installed.
The old PY_INSTALL_PY was renamed to PY_DO_INSTALLL_PY, a move that
would be better applied to all boolean variables. I won't try with this
commit, however, because a lot of downstream packages depend on
PY_UPDATE_INIT_PY and PY_INSTALL_INIT_PY, i.e. without "_DO".
Signed-off-by: Jan Lindemann <jan@janware.com>
py.clean unconditionally runs scm.sh clean -f __init__.py, which
deletes untracked __init__.py files in the source directory. In a
package with PY_UPDATE_INIT_PY=false the build never generates
__init__.py, as the generation rule in py-mod.mk is gated on the
same variable.
The __init__.py files in question are hand-written sources which are
always tracked in SCM, and for those, "scm.sh clean" becomes a no-op, so
with the current code, no real problem occurs, but explicitly protecting
__init__.py from deletion if PY_UPDATE_INIT_PY is false is certainly
more obvious.
Move the __init__.py cleanup inside the PY_UPDATE_INIT_PY=true
conditional so that py.clean only runs removal commands on files the
build actually generates.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Add a parameter "base_types" to detect_modules(), defaulting to None. If
it is not None, a module is only exported if its same-named object
inherits from one of the given types. Modules without a same-named class
are skipped instead of raising AttributeError, which covers helper
modules.
The return annotation becomes Sequence[str] instead of list[str] to
remove mutability for easier type checking.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecApp is a ready-made base class for applications that operate
through an ExecContext: it adds the --interactive, --verbose and
--target options, exposes interactive, verbose and exec_context
properties, and closes the exec context when the async context
manager exits.
The code for that has lived in jw.pkg.App code before, which now
inherits from ExecApp.
A fix along the way: __aexit__() closes the exec context and then chains
to super().__aexit__(), so App.close() runs when the async context
manager exits. Before the change, exiting the async context left the app
unclosed; close() ran only on the run() path. Add a unit test that
builds an ExecApp with a root command and asserts that close() runs on
context exit and that the exec options are registered.
The exec options are now registered before App's own options,
which moves them up in the rendered --help output. Update the
golden file of the help integration test to match.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
