_compile_one_template_file() re-parses the template's own conf line and,
when present, discards the attrs that compile_template_file() merged from
the command-line defaults, the template, the .jw-secret-file, and the
.jw-secret. The merge already orders the sources so that later non-empty
fields override earlier ones; discarding it inverts the order and lets the
template's line override the one the operator placed in the .jw-secret.
Strip the template's conf line from the content and apply the merged attrs
instead, so the .jw-secret conf line takes the last word again, as the
compiler did before the rewrite. The integration test gains a target whose
template and secret both carry a conf line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
"jw-pkg secrets install" copies secret material from a GPG-encrypted tar
archive to dotenv .jw-secret files on a target host. This commit refactors
the underlying code and adds hooks for the inverse copy direction, and for
more secret sources and targets.
The peers between which secrets are copied are represented by the new
abstract secret_store.Base class, and the existing tar blob and host
storages, inheriting from that class, are TarGpg and KvFile. Moreover,
secret_store.Base is inheriting from FileContext, so all of jw-pkg's
file-like operations are available as hooks.
Secrets inside instances of two file contexts are addressable as
(targpg|kvfile)://target.host.com/path/to/file.conf.jw-secret. The on-host
KvFile will look either locally or via SSH for that file, and TarGpg will
look inside a tar archive specified in the JW_SECRETS_TARGPG_TAR
environment variable.
Uris with these schemas can be passed to FileContext.create(), which will
return a chrooted file context of the correct type.
KvFile.stat("/path/to/file.conf.jw-secret").ftype is FT_SECRET which
inherits from FT_COMPOUND - which means that readdir() works on such
animals. Not entirely sure if that's a good idea, but it feels pretty
generic. Let's see where that takes us.
DistroContext.install() after this refactor reads its source through the
store interface and writes the landing through the KvFile store.
Both secret stores are exercised through the existing unit test of jw-pkg
secrets install, and unit tests for both stores are also added by this
commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() can now land a single object at its source path, but the copy
CLI command exposes no way to ask for it.
Add a --parents flag forwarded to copy()'s parents argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Add the "packages verify" command: it runs Distro.verify() with the app's
package filter and prints one line per flagged file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
Distro.pkg_files() takes a package name and returns a list of files in that
package. This commit generalizes that - pass None to it and get the list of
all packaged files in the system, and pass a list of strings to retrieve
all files belonging to the named packages.
The same arguments are propagated to _pkg_files(), which in turn propagates
them to list_files().
This allows faster generation of file lists than before. Having to retrieve
a list of matching packages and query each package for its contained files
can be collapsed into one or two backend calls.
Signed-off-by: Jan Lindemann <jan@janware.com>
The mypy module names of the test tree derive from the full path, anchored
by the top-level test/__init__.py, but a file below a path component that
is not a valid identifier - the dashes in jw-pkg - falls back to the bare
file name, so two same-named test.py drivers collide (the duplicate module
error that drove the runme.py and main.py renames).
Rename the directory a sane Python name - jw_pkg, that's the better fix.
Then name the driver test.py like all the others for consistency.
Signed-off-by: Jan Lindemann <jan@janware.com>
File attributes of the compiled output of "jw-pkg secrets
compile-templates" can be specified via the # conf line atop a .jw-tmpl
file, the parsing of that line is buggy, however.
Its mode is parsed as a decimal number unless it carries a leading zero, so
"mode=640" yields 640 decimal, 1200 octal, and the output file ends up
without usable permissions. The same decimal parse applies to a conf line atop
the .jw-secret of a target, whose attributes are honored the same way, and
the --mode option of compile-templates passes its string value through to
Attrs, whose mode is typed int, so a target without a conf line of its own
crashes in _format_metadata().
Other than that, the # conf line itself should be removed from the compiled
output, but isn't.
This commit makes the conf line handling correct:
- _parse_attributes() parses the mode as octal, with or without a leading
zero
- _compile_one_template_file() drops the conf line from the compiled
content
- the --mode option is converted to an octal int by octal_mode() at the
argument boundary
The integration test writes its conf line modes without leading zeros,
expects the conf line gone from the compiled output, takes one target's
mode from a conf line atop its .jw-secret, and passes a --mode that the
conf lines override.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Drive the real "jw-pkg secrets" CLI in-process against a GPG-encrypted
tarball in the canonical layout: reverse-domain host prefixes plus a
default tree, host wins. The distro is stubbed only in its package file
list; argument parsing, DistroContext, install, template compilation and
cleanup run as production code.
The fixture package carries three templates: one secret present in both
host and default trees (the host entry must win), one present only in the
default tree, and one missing from the tarball entirely (install must not
extract it). A tar member that is not a secret of the package must not be
extracted either.
Trees whose FileContext supports the targpg:// scheme address the tar
source through that scheme, with the tarball location in the environment,
and additionally exercise the rooted-URI rejection and the install from
the on-host file tree; earlier trees install from the tarball URI
directly.
The Makefile's SECRETS_TEST_BASE variable pins the fixture to a persistent
directory that survives the test run for inspection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Add an integration test for pkg-requires --skip-excluded with a fixture
project tree: excl-never lists the platform name "centos" in [build]
exclude and is not excluded from the build on the test platforms, excl-all
is excluded from the build everywhere, and dep-target is a plain dependency
target.
The test verifies that a platform name in [build] exclude is not treated as
a project, that an excluded jw dependency is not reported with
--skip-excluded, that an excluded seed is neither considered nor output,
and that without the flag the excluded projects are reported with their
dependencies.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The OS package list of required-os-pkg, like the build closure, is derived
from the [pkg.requires.jw] section only, hard-coded in _run(). Add a
--pkg-relations option to required-os-pkg, mirroring the one of CmdBuild.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The relations the build closure of projects build follows are hard-coded in
read_deps(): [pkg.requires.jw] only. A workspace that also wants to build
the projects its seeds recommend in [pkg.recommends.jw] has no way to say
so.
Add a --pkg-relations option that lists the relations between jw packages
to take into consideration for the build, comma or space separated,
defaulting to requires.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The spec generator only supports Requires, Conflicts and Provides from
the [pkg.requires.*], [pkg.conflicts.*] and [pkg.provides.*] sections of
make/project.conf. A package that works without but is better with another
package, like jw-amavis-run without jw-clamav-run, must hard-require it, so
the dependency is pulled in on every install and cannot be removed without
dragging the depending package along.
This commit adds Recommends support to the spec generation chain:
- CmdPkgRecommends: new pkg-recommends command that reads the
[pkg.recommends.<flavour>] sections, same base as pkg-requires
- pkg-dist.mk: compute PKG_RECOMMENDS_RUN and pass it to pkg.sh via a new
-E option
- pkg.sh: accept -E and forward it to the mkspec wrapper
- mkspec-wrapper.sh: accept -E and export RECOMMENDS_RUN
- create-mkspec.sh: emit a Recommends: line when RECOMMENDS_<subpkg> is set
- create-mkdebian.sh: emit a Recommends: field in the debian control file
Projects without a [pkg.recommends.*] section generate unchanged specs:
every new line is guarded by a non-empty test. zypper follows Recommends by
default, so existing OpenSUSE installations are unaffected; a recommended
package can be removed without dragging the depending package along.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
expand_macros() looks up each %{name} via getattr with '-' turned into '_',
but macro_names advertises two names that don't resolve: 'os-release' has
no corresponding attribute, and 'os-cascade' resolves to the list property
os_cascade, which str.replace() rejects. Both macros are listed in the
platform info help output, yet always fail when used in a format string.
The space-joined cascade string is the cascade property, and 'cascade' is
the macro name the make recipes and scripts already use, together with the
--format default in cmds/platform/CmdInfo.py.
Before the App.distro_* properties moved to Distro, the advertised macro
names were derived from the distro_* attribute names, so the advertised
macro was %{cascade}; the hand-written macro_names replaced it with the
list attribute's name.
- Add the missing os_release attribute as an alias of os_release_str
- Advertise the 'cascade' macro in macro_names instead of 'os-cascade',
leaving os_cascade as the purely programmatic list
- Update the platform info help expected output accordingly
- Add unit tests for the os-release field parsers, the derived properties,
and the expansion of every advertised macro
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
create-pkg-config's output is not covered by any test, even though the
template and the Requires cleanup produce exact text that pkg-config
consumes, so regressions in the generated file went unnoticed for a long
time.
Add an integration test that runs create-pkg-config with a full option set,
with space- and comma-mixed Requires input, and with a
Requires.private-only input, and compares the output against a reference
file.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdCreateFile renders a file from project metadata, yet nothing runs it end
to end. The "tmpl" format substitutes --field values into template markers,
and the "pyright" format computes extra paths from the jw run dependencies
of a module. Both formats are untested, and so is the handling of missing
templates, unknown --field keys, and malformed command-line arguments.
Add an integration test under test/integration/jw-pkg/projects that drives
the command through the real CLI and diffs the rendered output against
a reference. It covers single- and multi-value field substitution, the
default and custom field separators including the %n newline escape, value
quoting, and --field-keys filling in keys the caller omitted. The pyright
case uses a relative topdir so the computed project paths stay relocatable.
The error cases check that the command rejects a missing template name, an
unknown template file, a --field key outside --field-keys, an unsupported
pyright separator, and invalid --format and --field arguments with the
expected message and exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
List values in a rendered template are joined with a fixed separator of
",\n" (comma plus newline), and there is no way to change that. Templates
sometimes want a different separator, e.g. a plain newline or a single-line
comma-separated list.
Add a --field-separator option that controls how list values are joined
in the rendered output. %n expands to a newline, and the default remains
",\n" for both formats. render_tmpl() applies the separator to the rendered
template, while render_pyright() rejects any other value: the built-in
pyrightconfig.json template is a fixed JSON document, and no alternative
separator is supported for it.
Update the help output test expectation accordingly.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The --field option accepts arbitrary KEY=VALUE pairs, inserting them into
the rendered template output, but there is no way to declare which keys
a template actually supports. Passing a key the template does not use goes
unnoticed, and a template cannot offer optional fields that drop out of
the output when not supplied.
Add a --field-keys option, a comma-separated list of the keys a template
accepts. Passing a key via --field that is not in that list is an error,
and keys from the list that are not passed are added with an empty value,
so the respective field renders as empty and effectively drops out of the
output.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The possible values of --format are defined by an Enum which is never
really used as such. Derive them from class introspection instead, i.e.
offer all formats that have a corresponding render_<format>() name.
Also, turn the option into real argparse-backed choices, and make the
argument mandatory, because that reflects the reality of the implementation
- there is no default.
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecApp is a ready-made base class for applications that operate
through an ExecContext: it adds the --interactive, --verbose and
--target options, exposes interactive, verbose and exec_context
properties, and closes the exec context when the async context
manager exits.
The code for that has lived in jw.pkg.App code before, which now
inherits from ExecApp.
A fix along the way: __aexit__() closes the exec context and then chains
to super().__aexit__(), so App.close() runs when the async context
manager exits. Before the change, exiting the async context left the app
unclosed; close() ran only on the run() path. Add a unit test that
builds an ExecApp with a root command and asserts that close() runs on
context exit and that the exec options are registered.
The exec options are now registered before App's own options,
which moves them up in the rendered --help output. Update the
golden file of the help integration test to match.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
The __init__() method reads default log configuration directly
from environment variables, making it impossible for subclasses to
change defaults.
Extract the defaults into _default_log_flags(), _default_log_level(),
_default_log_file(), and _default_show_backtrace() methods, then call
them from __init__(). Subclasses can now override these methods to
customize defaults without needing to override the entire __init__()
method.
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdBuild supports the targets all, clean and pkg-*. It uses builtin
dependency flavours matching those targets to resolve build order. To
make it more flexible in general, and allow it to support more
targets, e.g. "check" and "test", this commit adds a --dep-flavours
option.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add tests for jw-pkg.py calls run during makefile caching. Broken
caching can compromise the build without causing it to fail entirely,
i.e. in non-obvious ways.
Signed-off-by: Jan Lindemann <jan@janware.com>
local/src is a janware-specific path, remove it from App and defs.mk.
It's still in pkg.sh as a safety measure. Will have to go, too, but
is kept in for now until further audit.
Signed-off-by: Jan Lindemann <jan@janware.com>
After a pipeline change, CI now runs "make all" in a repo's root,
which uncovers two problems:
1. The help integration test only succeeded as long CI didn't run
"make all" before "make test". That way, the checked out
repository lacked the generated __init__.py files needed for some
modular subcommands to be fully loaded, and hence, the test should
have failed. The entire machinery only worked because the
subcommands in question are not not essential to building jw-pkg
itself: "secrets" and "posix". So, this commit adapts the help
integration test to the new reality.
2. Regarding python-tools.sh: Commit 55060486 satisfies yapf in some
places of the source code, but in others not anylonger. So patch
python-tools.sh's newline handling again.
While not thematically similar, both fixes get baked into one commit
to satisfy the requirement that every single commit needs to pass
"make clean all check test" individually.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a test/integration subdirectory as a location for integration
tests. The first tests that are added are unproblematic in that they
don't need superuser privileges to run:
- help: Recursively check if jw-pkg.py's help messages are unchanged
- packages ls: Check if the bash package exists and contains
/usr/bin/bash
/usr/bin/bashbug
Let's see how that fares in CI.
Signed-off-by: Jan Lindemann <jan@janware.com>
