The unresolved-macro check scans the substituted content for __MACRO__
names, and a commented-out reference fails the compile like a live one. A
disabled config section keeps its macros in the comments as documentation,
and the matching store keys are retired along with it, so compiling the
template fails on lines that no longer take effect.
Skip comment lines when scanning for unresolved macros. The integration
test gains a target whose template carries a commented-out macro without a
store entry.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
_compile_one_template_file() re-parses the template's own conf line and,
when present, discards the attrs that compile_template_file() merged from
the command-line defaults, the template, the .jw-secret-file, and the
.jw-secret. The merge already orders the sources so that later non-empty
fields override earlier ones; discarding it inverts the order and lets the
template's line override the one the operator placed in the .jw-secret.
Strip the template's conf line from the content and apply the merged attrs
instead, so the .jw-secret conf line takes the last word again, as the
compiler did before the rewrite. The integration test gains a target whose
template and secret both carry a conf line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit adds the Vault store and the vault:// dispatch in FileContext.
A Vault KV v2 server maps nicely into jw-pkg's secret handling abstraction
as just another secret store backend. Environment variables identify the
storage details, analogous to the way it's handled in the SOPS backend.
- JW_SECRETS_VAULT_ADDR names the server, a unix socket address or an http
URL; the standard VAULT_ADDR is never read; JW_SECRETS_VAULT_MOUNT the
mount path, default secret; the token is the standard VAULT_TOKEN
- The transport is one curl per API call, through the Local context: a
unix:// server is reached through its socket, an http:// server through
its port
- A node's type is derived at stat time: an entry, a blob, a key below an
entry, a branch if the path has children, otherwise a miss
- A blob's content is base64 under a single key of its KV secret, the
marker being a path component, so a KV secret that is not a blob is
always an entry
The test drives a dev server fixture from the workspace: stat/readdir/get
on the fixture, the default fallback, the storage marker, and the copy
forms - single key, whole entry, recursive entry, deploy to the on-host
layout, and a host subtree.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
This commit adds a new class inheriting from secret_store.Base: Sops. The
currently available classes allow to store secrets offline in a
GPG-encrypted tarball, but those are not versioned. This commit allows to
use a SOPS / AGE-encrypted Git repository as storage backend instead.
The additions in detail:
- Sops: the store implementation; the repository location comes from
JW_SECRETS_SOPS_REPO, get() on an entry returns the decrypted serialized
content, and a typed put (FT_SECRET) re-encrypts the entry wholesale
- The age key comes from JW_SECRETS_SOPS_KEY_PATH only: the house variable
is the sole input to the store, and SOPS_AGE_KEY_FILE is set on the sops
subprocess per invocation, never read from the process environment;
SopsKeyError is raised when the variable is not set
- FileContext.create(): the sops:// scheme
- The store's unit tests, covering the node types, the deploy and fetch
round trip, the whole-entry get and put, the key resolution, and the
default fallback
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The TarGpg store is read-only: put() and unlink() raise, so the store can
be copied from, but not into, and a copy that lands entries and blobs below
a host prefix has no tarball target.
This matches the way jw-pkg secrets has worked for a long time. Since
Stores are now FileContext-backed, a write path can be cleanly implemented,
and that's what this commit does.
A put rebuilds the tarball: the existing members minus the removed, plus
the written one; the result is signed with the recipient from
JW_SECRETS_TARGPG_RECIPIENT and written back to the tarball location, which
must be a local path. A missing tarball is an empty store, so a put creates
it.
The put dispatch follows the other stores: a typed entry put replaces the
entry with the given serialized content, a key put merges into its entry,
creating it if it does not exist yet, and an untyped put below an entry
lands a key there, otherwise a blob. unlink() removes the node's member.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
"jw-pkg secrets install" copies secret material from a GPG-encrypted tar
archive to dotenv .jw-secret files on a target host. This commit refactors
the underlying code and adds hooks for the inverse copy direction, and for
more secret sources and targets.
The peers between which secrets are copied are represented by the new
abstract secret_store.Base class, and the existing tar blob and host
storages, inheriting from that class, are TarGpg and KvFile. Moreover,
secret_store.Base is inheriting from FileContext, so all of jw-pkg's
file-like operations are available as hooks.
Secrets inside instances of two file contexts are addressable as
(targpg|kvfile)://target.host.com/path/to/file.conf.jw-secret. The on-host
KvFile will look either locally or via SSH for that file, and TarGpg will
look inside a tar archive specified in the JW_SECRETS_TARGPG_TAR
environment variable.
Uris with these schemas can be passed to FileContext.create(), which will
return a chrooted file context of the correct type.
KvFile.stat("/path/to/file.conf.jw-secret").ftype is FT_SECRET which
inherits from FT_COMPOUND - which means that readdir() works on such
animals. Not entirely sure if that's a good idea, but it feels pretty
generic. Let's see where that takes us.
DistroContext.install() after this refactor reads its source through the
store interface and writes the landing through the KvFile store.
Both secret stores are exercised through the existing unit test of jw-pkg
secrets install, and unit tests for both stores are also added by this
commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a write-through read cache in front of a file context.
stat() and get() results are cached: stat per (path, follow_symlinks), get
per (path, wd, throw). Successful stats and FileNotFoundErrors are cached;
other errors are re-probed, so a transient transport failure does not
poison the cache. Writes pass through to the wrapped context and drop the
cache entries of the affected paths. The existence and directory checks are
rerouted through the cached stat, so they pay the cache instead of the
wrapped context's own fast path.
warm() fills the stat cache of a set of paths, as file_exists() would stat
them, without following symlinks. If the wrapped context is an exec
context, one host command stats all of the paths; otherwise, one stat per
path runs.
The GNU stat format of ExecContext is lifted to the module constant
STAT_GNU_FORMAT, and the field parser is exported as parse_stat_fields(),
so warm() fills entries that are indistinguishable from a real stat().
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a file context that sits in front of another one and delegates every
primitive call to it.
The wrapper adopts the identity of the wrapped context: uri, root, chroot
flag, and the chroot path composition are the wrapped context's, so a path
composed by the wrapper composes identically in the wrapped context. Every
primitive is forwarded to the wrapped context's primitives, which receive
the already-composed path; open() and close() are forwarded to the wrapped
context's public methods, so the open count is kept in one place.
The point is to give derived contexts a stable base: a caching context, or
any other filter, overrides the primitives it cares about and inherits the
rest, instead of re-deriving the chroot bookkeeping of every method.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() can now land a single object at its source path, but the copy
CLI command exposes no way to ask for it.
Add a --parents flag forwarded to copy()'s parents argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The walk needlessly descends into every directory node: A compound node is
gettable as a whole, so copy it over and don't recurse deeper.
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() lands a source below the destination root, taking at most the
source's basename, so there is no way to land it at its source path, as cp
--parents does.
Add a parents flag that requests just that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Using a FileContext, notably with its chroot flag set, is useful as a
tree source in copy(), so allow it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The file type framework gives mkdir() and put() an ftype argument, but
copy() calls them untyped, so the source's node types are discarded at the
landing: a destination that applies its own node types receives only the
defaults.
This commit threads the source's types through.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Refactor copy(), making the recursive logic more coherent: Every node the
walk creates sits below a node that already exists, so a plain per-node
mkdir() suffices and the chain- creating __mkdir_p() is dropped: the
landing root's parent must exist now, as in cp -r.
Signed-off-by: Jan Lindemann <jan@janware.com>
Give FileContext a read-only property to access self.__chroot, so a caller
holding a context can predict whether the addresses it passes in are
relative to the URI path or absolute.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Analogous to the is_dir() method, add an is_gettable() method: The last
commit introduces FT_COMPOUND, a file type which is a hybrid between
directory and regular file. is_dir() succeeds on it, so is_gettable()
symmetrically provides a test find out if get() also works on a file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Introduce FileType, a class consisting of a name and a parent property,
designed to make the information FileContext.stat() provides about a
path-addressable object richer.
A FileType property goes as ftype into the StatResult structure. Along with
it, this commit adds a central registry (FileTypes) where the type hierarchy
is kept, and additional parameters to put() and mkdir() to allow creating
typed FileContext objects.
Builtin types are:
- FT_REG: A regular file, the default thing put() creates
- FT_DIR: A directory, the default thing mkdir() creates
- FT_TOP: The parent of the two
- FT_COMPOUND: An FT_REG returning child entries from readdir()
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Add the "packages verify" command: it runs Distro.verify() with the app's
package filter and prints one line per flagged file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
jw-base verify pipes the output of rpm -qV for all janware packages
into a grep for md5 mismatches and missing files. The same capability,
distribution-independent, belongs in the Distro abstraction so that tools
like jw-nettools can check the integrity of the packages they install on a
fleet of hosts.
- Add Distro.verify(), which verifies the given package names, or all
packages selected by the package filter when none are given, and returns
the flagged files
- Add Distro._verify() as the distribution hook
- Add FlaggedFile, the record for a flagged file: package, file, and the
package manager's verify status line - not distribution independent as of
today, but it's a start
- pm/rpm.py: parse_verify_output() parses the output of `rpm -qV` and
reports md5 mismatches and missing files, the latter reported with the
literal word "missing" in the flag field; size, mode, mtime, user, and
group mismatches are noise on dev machines and are ignored.
verify_packages() runs `rpm -qV` per package and collects the findings
- pm/dpkg.py: parse_verify_output() parses the output of `dpkg -V` and
reports md5sum (P) mismatches and missing files (M); verify_packages()
runs `dpkg -V` per package
- The SUSE, Red Hat, and Debian Distro classes implement _verify() with the
respective pm helpers; the Arch Distro raises NotImplementedError, like
its other package manager hooks
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
Distro.pkg_files() takes a package name and returns a list of files in that
package. This commit generalizes that - pass None to it and get the list of
all packaged files in the system, and pass a list of strings to retrieve
all files belonging to the named packages.
The same arguments are propagated to _pkg_files(), which in turn propagates
them to list_files().
This allows faster generation of file lists than before. Having to retrieve
a list of matching packages and query each package for its contained files
can be collapsed into one or two backend calls.
Signed-off-by: Jan Lindemann <jan@janware.com>
File attributes of the compiled output of "jw-pkg secrets
compile-templates" can be specified via the # conf line atop a .jw-tmpl
file, the parsing of that line is buggy, however.
Its mode is parsed as a decimal number unless it carries a leading zero, so
"mode=640" yields 640 decimal, 1200 octal, and the output file ends up
without usable permissions. The same decimal parse applies to a conf line atop
the .jw-secret of a target, whose attributes are honored the same way, and
the --mode option of compile-templates passes its string value through to
Attrs, whose mode is typed int, so a target without a conf line of its own
crashes in _format_metadata().
Other than that, the # conf line itself should be removed from the compiled
output, but isn't.
This commit makes the conf line handling correct:
- _parse_attributes() parses the mode as octal, with or without a leading
zero
- _compile_one_template_file() drops the conf line from the compiled
content
- the --mode option is converted to an octal int by octal_mode() at the
argument boundary
The integration test writes its conf line modes without leading zeros,
expects the conf line gone from the compiled output, takes one target's
mode from a conf line atop its .jw-secret, and passes a --mode that the
conf lines override.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
rewrite() renames the members of a tar blob, but the TarInfo objects
carried over from the input still hold the pax "path" supplement that was
recorded for the original name. Re-serialization (tf_out.addfile()) uses
the pax supplement with higher precedence than what's in the base header's
name filed, so for member names exceeding the 100 character ustar limit the
rewrite silently has no effect and the original name is written back.
Drop the stale "path" supplement after renaming a member. tf_out.addfile()
then builds a rewritten, up-to-date supplement from the name field if it
needs one.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The -m option of posix copy is optional, and its argparse default is None,
but _run() converts it unconditionally and crashes:
mode = int(args.mode, 0)
so every copy without -m died with
TypeError: int() can't convert non-string with explicit base
before the copy even started. Convert the mode only when it is given, and
pass None otherwise, which copy() already treats as "no mode change".
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
_raise_stat_error() maps the message of a failed remote stat to an
exception. A stat below a file answers "not a directory", which falls
through to the generic EIO: the path is simply absent, and callers see a
re-raised error instead of the FileNotFoundError the "no such file" branch
already gives. Map it to FileNotFoundError, so existence checks see
absence.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg secrets list-secrets spams the console without --interactive=false
because it thinks it should query packages and files in interactive mode.
The underlying bug is that run_cmd() derives the interactive flag of a
freshly created Local context from cmd_input, but it never forwards
cmd_input to ExecContext.run(). Since run() defaults to
InputMode.OptInteractive, an explicit cmd_input is silently ignored: a byte
string is not fed to the command's stdin, InputMode.Auto is downgraded to
non-interactive, and with a caller-provided ExecContext the requested mode
is lost in all cases.
Forward cmd_input in kwargs so the requested input mode reaches run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
mkdir() passes its path to _mkdir() without chrooting it, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Signed-off-by: Jan Lindemann <jan@janware.com>
The [build] exclude list names platforms (OS cascade entries) on which a
project's build is excluded, not projects. pkg_relations() implements
--skip-excluded by walking the seeds' exclude values as if they were
project specs, so a value such as "debian" is visited as a project, the
installed-package check fails with "Unmet dependency on debian: the -run
package of project debian is not installed", and `make
pkg-install-testbuild-deps` breaks.
Implement the flag's documented semantics using
App.is_excluded_from_build(): an excluded seed is neither considered nor
output, an excluded module in the dependency closure contributes no
dependencies, and excluded modules are not output as jw dependencies. This
matches what required-os-pkg already does.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
get-auth-info() reads the origin URL of the jw-pkg repository and reports
its username and password. Uri.username and Uri.password are None for URLs
without a credentials part, but _run() stored them in the result anyway, so
--username --only-values printed the literal string "None" for such URLs.
The bootstrap Makefile derives JANWARE_USER from exactly that output on its
second parse, after the jw-pkg directory has been cloned from an anonymous
https origin whose URL has no user part. JANWARE_USER thus became the
string "None", pgit.sh is invoked with --login None and --refspec
None:current-branch:current-branch, and queries the Forgejo API for the
repositories of a non-existent "None" org, so every anonymous bootstrap
fails with "Failed to enumerate repositories".
Make _run() only record values that are not None, so missing credentials
produce no output instead of the string "None".
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The OS package list of required-os-pkg, like the build closure, is derived
from the [pkg.requires.jw] section only, hard-coded in _run(). Add a
--pkg-relations option to required-os-pkg, mirroring the one of CmdBuild.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The relations the build closure of projects build follows are hard-coded in
read_deps(): [pkg.requires.jw] only. A workspace that also wants to build
the projects its seeds recommend in [pkg.recommends.jw] has no way to say
so.
Add a --pkg-relations option that lists the relations between jw packages
to take into consideration for the build, comma or space separated,
defaulting to requires.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
get_project_refs() deduplicates the walk results by scanning the result
list for each appended element, so the cost grows quadratically with the
number of projects, which is noticeable when walking a dev tree of 320
projects.
Track the already-seen projects in a set, and append an element only when
it is new. The resulting list is unchanged, and the membership test is O(1)
instead of O(n).
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The list of OS packages printed by required-os-pkg is not reproducible: the
packages are collected in a set, and the set's iteration order depends on
the hash seed of the Python process, so two runs of the same invocation can
print the same packages in different orders.
Print the sorted list instead. The order of a package list is meaningless
to the package manager, and the output is now stable across runs and hash
seeds.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The build order printed by projects build is not reproducible:
calculate_order() picks the next module by iterating all_deps, a set, so
the order among modules that are ready at the same time depends on the hash
seed of the Python process, and two runs of the same invocation can print
different orders.
Select the next module from the sorted set instead. The order stays
topologically valid, and is now identical across runs and hash seeds. Sort
the unresolvable-dependency error message the same way, so that it is
stable, too.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
pkg_relations() walks the [pkg.<rel>.jw] edges with its own queue,
duplicating the walk that App.walk_project_deps() extracted from
App.__get_project_refs() in the previous commit. Port it onto the shared
walk: for each flavour, the walk computes the closure of the seed packages
(with the installed-package check off, the ignore set as exclude, and
recursion per the recursive flag), and the function then emits the
constraint strings of the edges of the visited projects.
The relation sets are unchanged. The order of recursive output follows the
walk (postorder) instead of the old breadth-first order; non-recursive
callers, such as the spec generation and the PREREQ_* queries, print
byte-identical output. The dont_expand_version_macros condition now tests
membership in the visited set instead of the old queue, which only matters
when the flag is passed manually.
Verified over all projects of the dev tree: pkg-requires, pkg-recommends,
pkg-conflicts and the skip-excluded, names-only and debian variants are
byte-identical for the non-recursive forms and set-identical for the
recursive ones.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
App.get_project_refs() uses __get_project_refs() to walk the [pkg.*.jw]
sections and pkg_relations() walks the same edges with its own
implementation. In an attempt to unify the walks, this commit tries to make
the first walk implementation so generally useful that it can be used by
the second.
- Make __get_project_refs() public as walk_project_deps(). Name changed
because it sounds less awkward
- Add three parameters needed from the other call site:
- check_installed applies the installed-package check that skips the
dependencies of an installed run package and fails on an unmet
dependency
- exclude names specs that are neither traversed nor appended
- recurse = False stops the walk at the starting spec
- Wrap the per-project dependency reading into App.read_dep_edges(),
because the other call site does the same thing
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
get_project_refs() takes a list of sections to look for references in. To
now, that has never been exercised, i.e. only one section had ever been
passed in. With the introduction of pkg.recommends, that could change now,
and it has a subtle bug.
Say, a caller wants to retrieve all requires and recommends of package A
with this graph:
A recommends B, requires nothing
B requires C
then A, B, C should be returned, but the current implementation only
returns A and B. The cause is that every section is recursed into in
isolation, and the union of all results is returned. Since A doesn't
require anything, the requires-branch never reaches B, and C is missed.
The fix is to make __get_project_refs(), the recursing method, take a list
of sections instead of one section, and decide if it recurses deeper based
on all of its entries at every depth.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The framework instantiates command classes in two places: the top-level
commands in App and the lazily materialized subcommands in
Cmd.add_subcommands(). Both pass the parent parameter positionally.
Change that to passing parent by keyword at both call sites. Every command
class names its first __init__() parameter parent, so the keyword form
binds the same slot for all of them regardless of parameter order, and it
allows a command class to declare parent as keyword-only.
Signed-off-by: Jan Lindemann <jan@janware.com>
The spec generator only supports Requires, Conflicts and Provides from
the [pkg.requires.*], [pkg.conflicts.*] and [pkg.provides.*] sections of
make/project.conf. A package that works without but is better with another
package, like jw-amavis-run without jw-clamav-run, must hard-require it, so
the dependency is pulled in on every install and cannot be removed without
dragging the depending package along.
This commit adds Recommends support to the spec generation chain:
- CmdPkgRecommends: new pkg-recommends command that reads the
[pkg.recommends.<flavour>] sections, same base as pkg-requires
- pkg-dist.mk: compute PKG_RECOMMENDS_RUN and pass it to pkg.sh via a new
-E option
- pkg.sh: accept -E and forward it to the mkspec wrapper
- mkspec-wrapper.sh: accept -E and export RECOMMENDS_RUN
- create-mkspec.sh: emit a Recommends: line when RECOMMENDS_<subpkg> is set
- create-mkdebian.sh: emit a Recommends: field in the debian control file
Projects without a [pkg.recommends.*] section generate unchanged specs:
every new line is guarded by a non-empty test. zypper follows Recommends by
default, so existing OpenSUSE installations are unaffected; a recommended
package can be removed without dragging the depending package along.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
A workspace project directory that has a Makefile but no package proofs -
no project.conf, no VERSION - is buildable with a plain make, yet the build
fails on it: calculate_order() walks the requested modules, and the walk in
App.__get_project_refs() raises an unmet dependency, because the
installation state of such a project is unprovable. The walk has to stay
strict: its proofs are package proofs, and loosening App.is_installed()
would weaken the unmet dependency detection for every get_project_refs()
caller.
This commit therefore keeps the tolerance in the build: before calculating
the order, run() partitions the requested modules. A module that is not
installed in any flavour but resolves to a workspace directory with a
Makefile is appended to the build order as a dependency-less leaf and built
with a plain make; modules without any directory at all still raise the
unmet dependency error. A jw dependency declared in a project.conf on such
a project still raises, which is correct: it is a declared dependency on a
package that does not exist.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Use Version.next() in __version_boundaries, which steps the last existing
part, so '= 1.0' spans '>= 1.0, < 1.1'.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Range expansion needs a bound that steps the last existing part of a
version, add that.
next() increments the last part, whatever it is: next of '1' is '2', of
'1.0' is '1.1', of '1.2.3' is '1.2.4', of '1.2.3-45' is '1.2.3-46'.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
A multi-boundary constraint is rendered as 'foo >= 1.2.3-45 foo < 1.2.4',
which is invalid: the RPM spec template writes the Requires: line from it
verbatim, and RPM entries are comma-separated, so the second clause is
swallowed into one bad entry. Debian's format_depends() normalizes runs of
whitespace to commas, so it tolerates the space join, but the comma is the
only delimiter that is correct for both.
Join the boundary clauses with ', ' instead of a space.
Also:
- Drop the stray trailing comma in the version_boundaries() call.
- Fix the untemplated docstring, which describes the opposite of the code:
untemplated = True (the default) resolves the macros, untemplated = False
keeps them as written.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Full version dependency specs ("= 1.2.3-4" or "= VERSION-REVISION") don't
need expansion, they pin the wanted version with an = fine exactly.
__version_boundary has that the wrong way around, fix that.
Also fix the is_full and version_boundaries() docstrings, which still
describe the old behavior.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__resolved_id() returns the full version if only VERSION was specified, fix
that.
Also: raise Version.Error instead of a bare Exception.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Running make format with the new isort setting rewrites the two wrapped
import blocks:
- The argparse import is split one-per-line, since the logical line exceeds
88 columns
- A trailing comma is added to both imports, which makes yapf keep the
split layout instead of re-joining it
The result is a fixed point that both isort and yapf accept, so make format
and make check agree.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__init_askpass() embeds the password verbatim in a generated bash script,
with a newline included inside the double quotes of the echo -n. The
askpass program therefore always appends a newline to the password, and
any password containing quotes or shell metacharacters either breaks the
script or injects commands into it.
Embed the password as base64 and decode it with printf piped into base64
-d, so the script is safe for any password and prints the password exactly,
byte for byte.
Make __del__() idempotent: it deletes the environment variables and the
script file, so a second call, e.g. an explicit one followed by garbage
collection, raises KeyError and FileNotFoundError.
Add a unit test that executes the generated script and compares its output
with the password byte for byte.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
expand_macros() looks up each %{name} via getattr with '-' turned into '_',
but macro_names advertises two names that don't resolve: 'os-release' has
no corresponding attribute, and 'os-cascade' resolves to the list property
os_cascade, which str.replace() rejects. Both macros are listed in the
platform info help output, yet always fail when used in a format string.
The space-joined cascade string is the cascade property, and 'cascade' is
the macro name the make recipes and scripts already use, together with the
--format default in cmds/platform/CmdInfo.py.
Before the App.distro_* properties moved to Distro, the advertised macro
names were derived from the distro_* attribute names, so the advertised
macro was %{cascade}; the hand-written macro_names replaced it with the
list attribute's name.
- Add the missing os_release attribute as an alias of os_release_str
- Advertise the 'cascade' macro in macro_names instead of 'os-cascade',
leaving os_cascade as the purely programmatic list
- Update the platform info help expected output accordingly
- Add unit tests for the os-release field parsers, the derived properties,
and the expansion of every advertised macro
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
__cleanup_requires() replaces every run of whitespace with ", " before
re-pairing the version constraints, so input that is already
comma-separated, e.g. "jw-core >= 1.0, jw-base", comes out with a double
comma, "jw-core >= 1.0,, jw-base". And the Requires line is appended
without a trailing newline, so a following Requires.private line runs
straight into it.
Split the input on commas and whitespace, treating the version constraint
operators as delimiters that are re-paired with the preceding name and the
following version, so that comma- and space-separated input alike comes out
as a clean ", "-joined list. Add the missing newline after the Requires
line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
