jw-base verify pipes the output of rpm -qV for all janware packages
into a grep for md5 mismatches and missing files. The same capability,
distribution-independent, belongs in the Distro abstraction so that tools
like jw-nettools can check the integrity of the packages they install on a
fleet of hosts.
- Add Distro.verify(), which verifies the given package names, or all
packages selected by the package filter when none are given, and returns
the flagged files
- Add Distro._verify() as the distribution hook
- Add FlaggedFile, the record for a flagged file: package, file, and the
package manager's verify status line - not distribution independent as of
today, but it's a start
- pm/rpm.py: parse_verify_output() parses the output of `rpm -qV` and
reports md5 mismatches and missing files, the latter reported with the
literal word "missing" in the flag field; size, mode, mtime, user, and
group mismatches are noise on dev machines and are ignored.
verify_packages() runs `rpm -qV` per package and collects the findings
- pm/dpkg.py: parse_verify_output() parses the output of `dpkg -V` and
reports md5sum (P) mismatches and missing files (M); verify_packages()
runs `dpkg -V` per package
- The SUSE, Red Hat, and Debian Distro classes implement _verify() with the
respective pm helpers; the Arch Distro raises NotImplementedError, like
its other package manager hooks
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
Distro.pkg_files() takes a package name and returns a list of files in that
package. This commit generalizes that - pass None to it and get the list of
all packaged files in the system, and pass a list of strings to retrieve
all files belonging to the named packages.
The same arguments are propagated to _pkg_files(), which in turn propagates
them to list_files().
This allows faster generation of file lists than before. Having to retrieve
a list of matching packages and query each package for its contained files
can be collapsed into one or two backend calls.
Signed-off-by: Jan Lindemann <jan@janware.com>
File attributes of the compiled output of "jw-pkg secrets
compile-templates" can be specified via the # conf line atop a .jw-tmpl
file, the parsing of that line is buggy, however.
Its mode is parsed as a decimal number unless it carries a leading zero, so
"mode=640" yields 640 decimal, 1200 octal, and the output file ends up
without usable permissions. The same decimal parse applies to a conf line atop
the .jw-secret of a target, whose attributes are honored the same way, and
the --mode option of compile-templates passes its string value through to
Attrs, whose mode is typed int, so a target without a conf line of its own
crashes in _format_metadata().
Other than that, the # conf line itself should be removed from the compiled
output, but isn't.
This commit makes the conf line handling correct:
- _parse_attributes() parses the mode as octal, with or without a leading
zero
- _compile_one_template_file() drops the conf line from the compiled
content
- the --mode option is converted to an octal int by octal_mode() at the
argument boundary
The integration test writes its conf line modes without leading zeros,
expects the conf line gone from the compiled output, takes one target's
mode from a conf line atop its .jw-secret, and passes a --mode that the
conf lines override.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
rewrite() renames the members of a tar blob, but the TarInfo objects
carried over from the input still hold the pax "path" supplement that was
recorded for the original name. Re-serialization (tf_out.addfile()) uses
the pax supplement with higher precedence than what's in the base header's
name filed, so for member names exceeding the 100 character ustar limit the
rewrite silently has no effect and the original name is written back.
Drop the stale "path" supplement after renaming a member. tf_out.addfile()
then builds a rewritten, up-to-date supplement from the name field if it
needs one.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The -m option of posix copy is optional, and its argparse default is None,
but _run() converts it unconditionally and crashes:
mode = int(args.mode, 0)
so every copy without -m died with
TypeError: int() can't convert non-string with explicit base
before the copy even started. Convert the mode only when it is given, and
pass None otherwise, which copy() already treats as "no mode change".
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
_raise_stat_error() maps the message of a failed remote stat to an
exception. A stat below a file answers "not a directory", which falls
through to the generic EIO: the path is simply absent, and callers see a
re-raised error instead of the FileNotFoundError the "no such file" branch
already gives. Map it to FileNotFoundError, so existence checks see
absence.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg secrets list-secrets spams the console without --interactive=false
because it thinks it should query packages and files in interactive mode.
The underlying bug is that run_cmd() derives the interactive flag of a
freshly created Local context from cmd_input, but it never forwards
cmd_input to ExecContext.run(). Since run() defaults to
InputMode.OptInteractive, an explicit cmd_input is silently ignored: a byte
string is not fed to the command's stdin, InputMode.Auto is downgraded to
non-interactive, and with a caller-provided ExecContext the requested mode
is lost in all cases.
Forward cmd_input in kwargs so the requested input mode reaches run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
mkdir() passes its path to _mkdir() without chrooting it, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Signed-off-by: Jan Lindemann <jan@janware.com>
The [build] exclude list names platforms (OS cascade entries) on which a
project's build is excluded, not projects. pkg_relations() implements
--skip-excluded by walking the seeds' exclude values as if they were
project specs, so a value such as "debian" is visited as a project, the
installed-package check fails with "Unmet dependency on debian: the -run
package of project debian is not installed", and `make
pkg-install-testbuild-deps` breaks.
Implement the flag's documented semantics using
App.is_excluded_from_build(): an excluded seed is neither considered nor
output, an excluded module in the dependency closure contributes no
dependencies, and excluded modules are not output as jw dependencies. This
matches what required-os-pkg already does.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
get-auth-info() reads the origin URL of the jw-pkg repository and reports
its username and password. Uri.username and Uri.password are None for URLs
without a credentials part, but _run() stored them in the result anyway, so
--username --only-values printed the literal string "None" for such URLs.
The bootstrap Makefile derives JANWARE_USER from exactly that output on its
second parse, after the jw-pkg directory has been cloned from an anonymous
https origin whose URL has no user part. JANWARE_USER thus became the
string "None", pgit.sh is invoked with --login None and --refspec
None:current-branch:current-branch, and queries the Forgejo API for the
repositories of a non-existent "None" org, so every anonymous bootstrap
fails with "Failed to enumerate repositories".
Make _run() only record values that are not None, so missing credentials
produce no output instead of the string "None".
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The OS package list of required-os-pkg, like the build closure, is derived
from the [pkg.requires.jw] section only, hard-coded in _run(). Add a
--pkg-relations option to required-os-pkg, mirroring the one of CmdBuild.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The relations the build closure of projects build follows are hard-coded in
read_deps(): [pkg.requires.jw] only. A workspace that also wants to build
the projects its seeds recommend in [pkg.recommends.jw] has no way to say
so.
Add a --pkg-relations option that lists the relations between jw packages
to take into consideration for the build, comma or space separated,
defaulting to requires.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
get_project_refs() deduplicates the walk results by scanning the result
list for each appended element, so the cost grows quadratically with the
number of projects, which is noticeable when walking a dev tree of 320
projects.
Track the already-seen projects in a set, and append an element only when
it is new. The resulting list is unchanged, and the membership test is O(1)
instead of O(n).
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The list of OS packages printed by required-os-pkg is not reproducible: the
packages are collected in a set, and the set's iteration order depends on
the hash seed of the Python process, so two runs of the same invocation can
print the same packages in different orders.
Print the sorted list instead. The order of a package list is meaningless
to the package manager, and the output is now stable across runs and hash
seeds.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The build order printed by projects build is not reproducible:
calculate_order() picks the next module by iterating all_deps, a set, so
the order among modules that are ready at the same time depends on the hash
seed of the Python process, and two runs of the same invocation can print
different orders.
Select the next module from the sorted set instead. The order stays
topologically valid, and is now identical across runs and hash seeds. Sort
the unresolvable-dependency error message the same way, so that it is
stable, too.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
pkg_relations() walks the [pkg.<rel>.jw] edges with its own queue,
duplicating the walk that App.walk_project_deps() extracted from
App.__get_project_refs() in the previous commit. Port it onto the shared
walk: for each flavour, the walk computes the closure of the seed packages
(with the installed-package check off, the ignore set as exclude, and
recursion per the recursive flag), and the function then emits the
constraint strings of the edges of the visited projects.
The relation sets are unchanged. The order of recursive output follows the
walk (postorder) instead of the old breadth-first order; non-recursive
callers, such as the spec generation and the PREREQ_* queries, print
byte-identical output. The dont_expand_version_macros condition now tests
membership in the visited set instead of the old queue, which only matters
when the flag is passed manually.
Verified over all projects of the dev tree: pkg-requires, pkg-recommends,
pkg-conflicts and the skip-excluded, names-only and debian variants are
byte-identical for the non-recursive forms and set-identical for the
recursive ones.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
App.get_project_refs() uses __get_project_refs() to walk the [pkg.*.jw]
sections and pkg_relations() walks the same edges with its own
implementation. In an attempt to unify the walks, this commit tries to make
the first walk implementation so generally useful that it can be used by
the second.
- Make __get_project_refs() public as walk_project_deps(). Name changed
because it sounds less awkward
- Add three parameters needed from the other call site:
- check_installed applies the installed-package check that skips the
dependencies of an installed run package and fails on an unmet
dependency
- exclude names specs that are neither traversed nor appended
- recurse = False stops the walk at the starting spec
- Wrap the per-project dependency reading into App.read_dep_edges(),
because the other call site does the same thing
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
get_project_refs() takes a list of sections to look for references in. To
now, that has never been exercised, i.e. only one section had ever been
passed in. With the introduction of pkg.recommends, that could change now,
and it has a subtle bug.
Say, a caller wants to retrieve all requires and recommends of package A
with this graph:
A recommends B, requires nothing
B requires C
then A, B, C should be returned, but the current implementation only
returns A and B. The cause is that every section is recursed into in
isolation, and the union of all results is returned. Since A doesn't
require anything, the requires-branch never reaches B, and C is missed.
The fix is to make __get_project_refs(), the recursing method, take a list
of sections instead of one section, and decide if it recurses deeper based
on all of its entries at every depth.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The framework instantiates command classes in two places: the top-level
commands in App and the lazily materialized subcommands in
Cmd.add_subcommands(). Both pass the parent parameter positionally.
Change that to passing parent by keyword at both call sites. Every command
class names its first __init__() parameter parent, so the keyword form
binds the same slot for all of them regardless of parameter order, and it
allows a command class to declare parent as keyword-only.
Signed-off-by: Jan Lindemann <jan@janware.com>
The spec generator only supports Requires, Conflicts and Provides from
the [pkg.requires.*], [pkg.conflicts.*] and [pkg.provides.*] sections of
make/project.conf. A package that works without but is better with another
package, like jw-amavis-run without jw-clamav-run, must hard-require it, so
the dependency is pulled in on every install and cannot be removed without
dragging the depending package along.
This commit adds Recommends support to the spec generation chain:
- CmdPkgRecommends: new pkg-recommends command that reads the
[pkg.recommends.<flavour>] sections, same base as pkg-requires
- pkg-dist.mk: compute PKG_RECOMMENDS_RUN and pass it to pkg.sh via a new
-E option
- pkg.sh: accept -E and forward it to the mkspec wrapper
- mkspec-wrapper.sh: accept -E and export RECOMMENDS_RUN
- create-mkspec.sh: emit a Recommends: line when RECOMMENDS_<subpkg> is set
- create-mkdebian.sh: emit a Recommends: field in the debian control file
Projects without a [pkg.recommends.*] section generate unchanged specs:
every new line is guarded by a non-empty test. zypper follows Recommends by
default, so existing OpenSUSE installations are unaffected; a recommended
package can be removed without dragging the depending package along.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
A workspace project directory that has a Makefile but no package proofs -
no project.conf, no VERSION - is buildable with a plain make, yet the build
fails on it: calculate_order() walks the requested modules, and the walk in
App.__get_project_refs() raises an unmet dependency, because the
installation state of such a project is unprovable. The walk has to stay
strict: its proofs are package proofs, and loosening App.is_installed()
would weaken the unmet dependency detection for every get_project_refs()
caller.
This commit therefore keeps the tolerance in the build: before calculating
the order, run() partitions the requested modules. A module that is not
installed in any flavour but resolves to a workspace directory with a
Makefile is appended to the build order as a dependency-less leaf and built
with a plain make; modules without any directory at all still raise the
unmet dependency error. A jw dependency declared in a project.conf on such
a project still raises, which is correct: it is a declared dependency on a
package that does not exist.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Use Version.next() in __version_boundaries, which steps the last existing
part, so '= 1.0' spans '>= 1.0, < 1.1'.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Range expansion needs a bound that steps the last existing part of a
version, add that.
next() increments the last part, whatever it is: next of '1' is '2', of
'1.0' is '1.1', of '1.2.3' is '1.2.4', of '1.2.3-45' is '1.2.3-46'.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
A multi-boundary constraint is rendered as 'foo >= 1.2.3-45 foo < 1.2.4',
which is invalid: the RPM spec template writes the Requires: line from it
verbatim, and RPM entries are comma-separated, so the second clause is
swallowed into one bad entry. Debian's format_depends() normalizes runs of
whitespace to commas, so it tolerates the space join, but the comma is the
only delimiter that is correct for both.
Join the boundary clauses with ', ' instead of a space.
Also:
- Drop the stray trailing comma in the version_boundaries() call.
- Fix the untemplated docstring, which describes the opposite of the code:
untemplated = True (the default) resolves the macros, untemplated = False
keeps them as written.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Full version dependency specs ("= 1.2.3-4" or "= VERSION-REVISION") don't
need expansion, they pin the wanted version with an = fine exactly.
__version_boundary has that the wrong way around, fix that.
Also fix the is_full and version_boundaries() docstrings, which still
describe the old behavior.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__resolved_id() returns the full version if only VERSION was specified, fix
that.
Also: raise Version.Error instead of a bare Exception.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Running make format with the new isort setting rewrites the two wrapped
import blocks:
- The argparse import is split one-per-line, since the logical line exceeds
88 columns
- A trailing comma is added to both imports, which makes yapf keep the
split layout instead of re-joining it
The result is a fixed point that both isort and yapf accept, so make format
and make check agree.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__init_askpass() embeds the password verbatim in a generated bash script,
with a newline included inside the double quotes of the echo -n. The
askpass program therefore always appends a newline to the password, and
any password containing quotes or shell metacharacters either breaks the
script or injects commands into it.
Embed the password as base64 and decode it with printf piped into base64
-d, so the script is safe for any password and prints the password exactly,
byte for byte.
Make __del__() idempotent: it deletes the environment variables and the
script file, so a second call, e.g. an explicit one followed by garbage
collection, raises KeyError and FileNotFoundError.
Add a unit test that executes the generated script and compares its output
with the password byte for byte.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
expand_macros() looks up each %{name} via getattr with '-' turned into '_',
but macro_names advertises two names that don't resolve: 'os-release' has
no corresponding attribute, and 'os-cascade' resolves to the list property
os_cascade, which str.replace() rejects. Both macros are listed in the
platform info help output, yet always fail when used in a format string.
The space-joined cascade string is the cascade property, and 'cascade' is
the macro name the make recipes and scripts already use, together with the
--format default in cmds/platform/CmdInfo.py.
Before the App.distro_* properties moved to Distro, the advertised macro
names were derived from the distro_* attribute names, so the advertised
macro was %{cascade}; the hand-written macro_names replaced it with the
list attribute's name.
- Add the missing os_release attribute as an alias of os_release_str
- Advertise the 'cascade' macro in macro_names instead of 'os-cascade',
leaving os_cascade as the purely programmatic list
- Update the platform info help expected output accordingly
- Add unit tests for the os-release field parsers, the derived properties,
and the expansion of every advertised macro
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
__cleanup_requires() replaces every run of whitespace with ", " before
re-pairing the version constraints, so input that is already
comma-separated, e.g. "jw-core >= 1.0, jw-base", comes out with a double
comma, "jw-core >= 1.0,, jw-base". And the Requires line is appended
without a trailing newline, so a following Requires.private line runs
straight into it.
Split the input on commas and whitespace, treating the version constraint
operators as delimiters that are re-paired with the preceding name and the
following version, so that comma- and space-separated input alike comes out
as a clean ", "-joined list. Add the missing newline after the Requires
line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
List values in a rendered template are joined with a fixed separator of
",\n" (comma plus newline), and there is no way to change that. Templates
sometimes want a different separator, e.g. a plain newline or a single-line
comma-separated list.
Add a --field-separator option that controls how list values are joined
in the rendered output. %n expands to a newline, and the default remains
",\n" for both formats. render_tmpl() applies the separator to the rendered
template, while render_pyright() rejects any other value: the built-in
pyrightconfig.json template is a fixed JSON document, and no alternative
separator is supported for it.
Update the help output test expectation accordingly.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The --field option accepts arbitrary KEY=VALUE pairs, inserting them into
the rendered template output, but there is no way to declare which keys
a template actually supports. Passing a key the template does not use goes
unnoticed, and a template cannot offer optional fields that drop out of
the output when not supplied.
Add a --field-keys option, a comma-separated list of the keys a template
accepts. Passing a key via --field that is not in that list is an error,
and keys from the list that are not passed are added with an empty value,
so the respective field renders as empty and effectively drops out of the
output.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The possible values of --format are defined by an Enum which is never
really used as such. Derive them from class introspection instead, i.e.
offer all formats that have a corresponding render_<format>() name.
Also, turn the option into real argparse-backed choices, and make the
argument mandatory, because that reflects the reality of the implementation
- there is no default.
Signed-off-by: Jan Lindemann <jan@janware.com>
create-file and create-pkg-config both print the rendered template with a
newline appended over the original template, owed to using print() instead
of sys.stdout.write(). Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
format_list_dict() substitutes {key} markers with str.replace(), which also
matches the marker inside ${key} and {{key}}, mangling text meant to
survive verbatim, e.g. the ${prefix} variable reference of a pkg-config
file. The built-in pkg-config template works around that with doubled
braces, which produces {/usr}-style output instead.
Replace the markers with a regular expression that skips markers preceded
by '$' or '{', and let the pkg-config template use the ${prefix} and
${exec_prefix} references pkg-config expects.
Extend the unit tests with marker escaping, value escaping, quoting,
indentation, and the rendering of the built-in templates.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
is_list_dict() accepts a string value as a list-dict value, and
render_values_to_list_dict() passes such values through unchanged.
merge_values() then merges them with list extension, and [] += 'xyz'
appends the individual characters, so every string value ends up as a list
of its characters. Normalize string values to single-element lists in
render_values_to_list_dict(), so that merge_values() appends whole values.
jw-pkg projects create-pkg-config is hit by this, because all of its values
are strings: the generated file contains one line per character. The code
path was just never exercised lately.
Add unit tests for the value layout guards, the normalization, and the
merging of the three supported layouts.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
--skip-excluded removed excluded modules from deps with
deps.remove() while iterating over deps. Removing an element
shifts the remaining elements to the left, so the loop skipped
the entry following each removed one, and consecutively
excluded modules survived the filter and leaked into the
output.
Build the filtered list with a comprehension instead.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
The spec split pattern ([=><]+) tokenizes only the =, > and < characters,
so a PEP 440 operator like ~= or != leaves its ~ or ! glued to the package
name: Dependency('pkg~=1.0') parses to base name 'pkg~' with operator '='.
App.__get_project_refs() carries the same pattern inline, so the same specs
corrupt the module name used for the -devel subpackage check. The split
also tolerates operator strings the language does not support, most visibly
==, which parses and renders but raises NotImplementedError only when
expansion is requested.
The spec language supports exactly =, <, <=, > and >=, and boundary
expansion implements all of them. Extend the split pattern with ~ and ! so
that foreign operators tokenize as operator strings, and reject every
operator outside the supported set in Dependency.__parsed_spec() with
Dependency.Error, naming the supported operators. Route
App.__get_project_refs() through Dependency for the name and module parts
instead of the second inline split, so the validation lives in one place.
The catch all in Dependency.__version_boundaries() stays as a backstop
against drift between the allow list and the expansion cases.
The tests drop == from the accepted operator loop, drop the now-unreachable
not-expandable case, and assert that ~=, !=, ~, ==, ===, << and >> are
rejected at parse time with a message naming the operator.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
App.__get_project_refs() recurses into the package dependency graph, but
the recursion end conditions only work well if all required packages are
present. Whether a required package is installed or not is decided upon a
wrong condition, however - existence of the project's project directory,
which may or may not be misleading for both the -devel and the -run
requirements. This can lead to various unwanted outcomes: Missing packages
happily inserted into the recursion buffer, process termination instead of
recursion stop, path lookup errors instead of a clearer "unmet dependency"
message (No project path found for module xyz, Failed to find directory of
project foo).
This commit cleanly detects if -run or -devel are installed and makes the
walk stop or raise under the appropriate conditions:
1. -run missing but required, raises unmet dependency
2. -devel missing but required, raises unmet dependency
3. -run present, no -devel required, stop recursing
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The installed state of a project is judged by the proofs of installation,
which differ per subpackage: make/project.conf in the dev tree or /opt
proves the -devel package or a buildable checkout, a VERSION file in the
project directory or /usr/share/doc/packages the -run package.
Add is_installed(), which answers the proof question for a given subpackage
by searching the per-subpackage locations.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
The iteration order contains every project the dependency walk resolved,
including projects that are only installed as -devel packages below /opt.
Running the target in such a project ran make in a read-only, source-less
directory and failed the run.
Resolve each module against the workspace root only, and skip the target
with a notice when there is no buildable project directory. Factor the skip
notice into log_skip() so the platform-exclusion path shares it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
find_dir() and its helpers resolve a project name against the workspace
root and /opt. The build iteration needs to resolve a module against the
workspace only, to tell buildable projects apart from installed ones.
Add a projs_roots parameter to find_dir(), __find_dir() and __proj_dir()
that restricts the search to the given roots. With the parameter omitted
the previous behavior is unchanged.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
The lib.version module carries no documentation: the Syntax, Component and
Boundary types, the Version and Dependency classes and their public methods
are bare, and the compatibility tiers the next_* stepping methods implement
are not documented anywhere.
Add docstrings: a line each for the base.py types, the compatibility
tier table in the Version class, and a short description for every
public method, with the three next_* methods citing the tier each one
steps to.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
- The pkg_relations() function is a horribly bad read. It contains a
mind-bending amount of interwoven case distinctions that are not clearly
reflected in the participating variable and function names. Much of it is
version handling. That's intricate by nature, but much of it has now been
implemented in the lib.version module, so moving the logic there makes
the function a good deal more readable than before. That's most of what
this commit does.
- Use version syntax macros from lib.version instead of
pkg_relations-defined macros, and fix the fallout in BaseCmdPkgRelations
and CmdCreateFile.
- Use lib.version also as a central place for dep string parsing from App.
Signed-off-by: Jan Lindemann <jan@janware.com>
pkg_relations() parses dependency specifications such as 'foo-devel >=
1.2.3-4' with inline regular expressions and mixes the parsing with version
macro expansion and rendering in three different syntaxes.
This commit adds the lib.version module that owns the parsing of a single
specification: the base and full name and the version boundary with its
operator. Rendering happens in constraint_str() in SEM_VER, DEBIAN or
NAMES_ONLY syntax, with untemplated, include_revision, as_range,
no_subpackages and quote options. The VERSION, VERSION-REVISION and
REVISION macros are resolved through a version lookup callback,
lib.version.Version.parse_deps_spec() splits comma-separated specification
strings into Version objects, and Version breaks a version down into its
major, minor, micro and revision parts.
Unit tests cover parsing, macro resolution, rendering in all syntaxes and
the range expansion.
The module is designed to be integrated into the status quo and
intentionally does not address a couple of further TODOs. Notably
version.Syntax.SEM_VER is intended to replace
pkg_relations.VersionSyntax.SemVer, but neither are really semantic
versioning according to spec. They are close but more targeted toward RPM
and Debian package versioning. And the naming and semantics differ from
SemVer.
Naming of the four components (major and minor version) is identical,
semantic meaning aside, there's no disagreement between real SemVer,
Debian, RPM and jw-pkg. The third and fourth component deviate:
SemVer Debian RPM jw-pkg lib.version
3 PATCH RELEASE MICRO
4 PRERELEASE Revision Release REVISION REVISION
TODOs: RELEASE over the rest of jw-pkg needs to be adjusted by a later
commit. Syntax.SEM_VER should also be renamed, to Syntax.JW_PKG maybe,
because, as said, it's not SemVer and will probably never be - SemVer
doesn't provide the compatibility guarantees that the jw-pkg versioning
scheme offers. To be documented.
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commit reads __abstractmethods__ via a getattr() with an empty
frozenset fallback, because in mypy 2.3.1 the scanned classes are typed as
type[object], which does not declare the attribute.
Add is_abc_class() as a TypeGuard, and skip the classes it rejects with a
debug line: LoadTypes now yields only ABC-derived classes. The guard
narrows to the classes that carry the attribute, so that the debug line can
now read __abstractmethods__ directly.
The command loaders are unaffected: every class they load is derived from
AbstractCmd, and hence from ABC. For loads that rely on the name filter
alone, plain classes are now skipped instead of being yielded.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
