The install methods of Distro inconsistently resolve dependencies or don't
depending on the descendant's implementation. Pin that more unambigously by
adding a new parameter install_deps, defaulting to True, to the following
methods of lib.Distro:
_install_local_files()
_install_urls()
_install_urls_and_names()
install()
The implementations in lib.distros.* make use of the new
lib.pm.*.install_packages() method introduced by the previous commit.
Add a unit test directory per Distro descendant.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Currently, jw-pkg install is sometimes left to the high-level package
manager, which is allowed to install dependencies along, and that's
sometimes unwanted, but there's no backend in the low-level package manager
module lib.pm to allow specifying that. This commit adds it:
lib.pm.*.install_packages().
The functions install a list of local package files using the low-level
package managers rpm / dpkg, bypassing the high-level package managers yum
/ apt / zypper.
pm.rpm runs rpm -U and pm.dpkg runs dpkg -i. Every file is classified
against the installed state of its package, queried by name: a file
for a package that is not installed is installed fresh, a file with
another version installed is upgraded, and a file whose version is already
installed is reinstalled when force is set and skipped otherwise. On rpm,
plain -U refuses a package that is already installed, and --reinstall
aborts the transaction when a file of the incoming package differs from
the installed one, so the upgrades run before the reinstalls and the
same-version files need force.
With only_update, a file for a package that is not installed raises before
anything is installed.
The dpkg route is gated on an apt simulation, because dpkg -i leaves a
half-configured package when a dependency is missing: the simulation runs
first and the install is refused when it would install or remove anything
but the given files' own packages.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a version field to the Package class and make query_packages() fill it.
It's very much distribution and package dependent what shape that version
has, so from Package's perspective, it's just a str-typed member.
Further improvements and fixes:
- Make query_packages() accept throw, defaulting to the previous behaviour,
so a caller that wants to be prepared for a missing package / an empty
result can pass throw = False.
- Make dpkg's _run() and run_dpkg_query() forward ec and the remaining
keyword arguments on the sudo path, which dropped them before.
Add unit tests for the file type tree and its StatResult predicates.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Tonight's CI upgraded dput from 1.2.4 to 1.2.6 on Kali Linux and
subsequently failed:
xzcat /tmp/20261027-010000-rollout-failed.log.xz | grep -ai -C 1 checksum
Trying to upload package to janware-debian
Checksum doesn't match for dist/src/jw-pkg_1.0.0-281.dsc
make[1]: *** [make/pkg-dist.mk:288: pkg-release-reinstall] Error 1
The dput config generated for debian uploads sets hash = sha, while the
Files: section of a .changes file only carries MD5 checksums. dput verifies
each listed file by comparing the configured hash of the file against
that MD5 column, so a sha verification can never match and the upload is
rejected at the first file, the .dsc.
Set the option to md5, the only hash the .changes Files: section carries
and dput's built-in default.
Not sure, yet, if this is a fix or a workaround of a dput bug.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The postinst generator looks up the install log line of each file with a
regex grep on the file path, and the dots of the path match any character.
When the grep returns several lines, milk_install_log_spec_attr() passes
the extra lines through, so the emitted chown and chmod commands split
across lines, dpkg -i reports errors for the fragments, and the owner and
mode of the file are not applied.
Match the end of the log line against the file path literally and take the
first match. The install log line ends in the file path, but may carry a
build prefix before it, so the literal comparison of the tail of the line
selects exactly the line of the file, for plain and %config entries alike.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
argcomplete 3.7 declares BaseCompleter.__call__() with an Iterable return,
and mypy rejects the None-returning override in NoopCompleter. Return an
empty list instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
The unresolved-macro check scans the substituted content for __MACRO__
names, and a commented-out reference fails the compile like a live one. A
disabled config section keeps its macros in the comments as documentation,
and the matching store keys are retired along with it, so compiling the
template fails on lines that no longer take effect.
Skip comment lines when scanning for unresolved macros. The integration
test gains a target whose template carries a commented-out macro without a
store entry.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
_compile_one_template_file() re-parses the template's own conf line and,
when present, discards the attrs that compile_template_file() merged from
the command-line defaults, the template, the .jw-secret-file, and the
.jw-secret. The merge already orders the sources so that later non-empty
fields override earlier ones; discarding it inverts the order and lets the
template's line override the one the operator placed in the .jw-secret.
Strip the template's conf line from the content and apply the merged attrs
instead, so the .jw-secret conf line takes the last word again, as the
compiler did before the rewrite. The integration test gains a target whose
template and secret both carry a conf line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.4.0
Signed-off-by: Jan Lindemann <jan@janware.com>
f70b14e92 ("defs.mk: Disable BUILD_PKG_CONFIG") disabled the
BUILD_PKG_CONFIG copy target, but the dir_build_PKG_CONFIG.done rule
still runs `mkdir -p $(BUILD_PKG_CONFIGDIR)`, and that directory is
keyed off DEVELOPMENT. A dev-version downstream package built against an
installed, read-only jw-pkg therefore resolves BUILD_PKG_CONFIGDIR to
$(JWBDIR)/pkgconfig and fails the build with "mkdir: Permission denied".
Default BUILD_PKG_CONFIGDIR to project-local $(TOPDIR)/make regardless of
DEVELOPMENT, and leave it overridable so that a self-contained build over
multiple projects can point it at some shared $(JWBDIR)/pkgconfig
directory at the user's discretion.
Signed-off-by: Jan Lindemann <jan@janware.com>
defs.mk derives WHICH from the external which binary, first from a
hardcoded /usr/bin/which path and then with a `which which` fallback, while
the probes for time, ruff, pyright, and python3 call which directly.
Systems that do not ship the which package, such as minimal containers, are
left to the shell's PATH search.
Define WHICH once in common.mk as the POSIX command -v builtin and drop the
old definition from defs.mk. Route the remaining direct which calls through
$(WHICH):
- projects-dir.mk: TIME probe
- py-check.mk: PY_CHECK_RUFF and PY_CHECK_PYRIGHT probes
- py-version.mk: PYTHON lookup
command -v runs in the /bin/sh that $(shell) already uses, so the build no
longer depends on the which package. The builtin names that command -v
reports for echo, test, pwd, and printf are safe, because every use of
those variables runs through the shell, and the bash time keyword keeps the
-p output format identical to the GNU time binary.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.3.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Using --skip-excluded in pkg-install-*'s recipes breaks CI, and here's how:
During a CI run, a package is expected to be testable with only jw-pkg and
the package itself installed in the test environment. "testable" here means
that it's possible to install the necessary dependencies with make
pkg-install-* commands before building. That condition is inherently always
met by packages which have only jw-pkg as its only dependency. But it's not
true for packages with intermediate packages in their dependency tree. For
for those packages it's not advisable to pass --skip-excluded to the
dependency detection, because that will cause the resolver to try and find
the intermediate's project.conf to read its [build].excluded section.
Which fails because that intermediate is neither installed nor checked out.
So, until there's another mechansim in place which allows excluding
uninstalled packages (caller-supplied knowledge, reading remote
repositories, package-manager-like package metadata cache), drop
--skip-excluded from the pkg-install-* targets' recipes.
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit adds the Vault store and the vault:// dispatch in FileContext.
A Vault KV v2 server maps nicely into jw-pkg's secret handling abstraction
as just another secret store backend. Environment variables identify the
storage details, analogous to the way it's handled in the SOPS backend.
- JW_SECRETS_VAULT_ADDR names the server, a unix socket address or an http
URL; the standard VAULT_ADDR is never read; JW_SECRETS_VAULT_MOUNT the
mount path, default secret; the token is the standard VAULT_TOKEN
- The transport is one curl per API call, through the Local context: a
unix:// server is reached through its socket, an http:// server through
its port
- A node's type is derived at stat time: an entry, a blob, a key below an
entry, a branch if the path has children, otherwise a miss
- A blob's content is base64 under a single key of its KV secret, the
marker being a path component, so a KV secret that is not a blob is
always an entry
The test drives a dev server fixture from the workspace: stat/readdir/get
on the fixture, the default fallback, the storage marker, and the copy
forms - single key, whole entry, recursive entry, deploy to the on-host
layout, and a host subtree.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
This commit adds a new class inheriting from secret_store.Base: Sops. The
currently available classes allow to store secrets offline in a
GPG-encrypted tarball, but those are not versioned. This commit allows to
use a SOPS / AGE-encrypted Git repository as storage backend instead.
The additions in detail:
- Sops: the store implementation; the repository location comes from
JW_SECRETS_SOPS_REPO, get() on an entry returns the decrypted serialized
content, and a typed put (FT_SECRET) re-encrypts the entry wholesale
- The age key comes from JW_SECRETS_SOPS_KEY_PATH only: the house variable
is the sole input to the store, and SOPS_AGE_KEY_FILE is set on the sops
subprocess per invocation, never read from the process environment;
SopsKeyError is raised when the variable is not set
- FileContext.create(): the sops:// scheme
- The store's unit tests, covering the node types, the deploy and fetch
round trip, the whole-entry get and put, the key resolution, and the
default fallback
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The TarGpg store is read-only: put() and unlink() raise, so the store can
be copied from, but not into, and a copy that lands entries and blobs below
a host prefix has no tarball target.
This matches the way jw-pkg secrets has worked for a long time. Since
Stores are now FileContext-backed, a write path can be cleanly implemented,
and that's what this commit does.
A put rebuilds the tarball: the existing members minus the removed, plus
the written one; the result is signed with the recipient from
JW_SECRETS_TARGPG_RECIPIENT and written back to the tarball location, which
must be a local path. A missing tarball is an empty store, so a put creates
it.
The put dispatch follows the other stores: a typed entry put replaces the
entry with the given serialized content, a key put merges into its entry,
creating it if it does not exist yet, and an untyped put below an entry
lands a key there, otherwise a blob. unlink() removes the node's member.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
"jw-pkg secrets install" copies secret material from a GPG-encrypted tar
archive to dotenv .jw-secret files on a target host. This commit refactors
the underlying code and adds hooks for the inverse copy direction, and for
more secret sources and targets.
The peers between which secrets are copied are represented by the new
abstract secret_store.Base class, and the existing tar blob and host
storages, inheriting from that class, are TarGpg and KvFile. Moreover,
secret_store.Base is inheriting from FileContext, so all of jw-pkg's
file-like operations are available as hooks.
Secrets inside instances of two file contexts are addressable as
(targpg|kvfile)://target.host.com/path/to/file.conf.jw-secret. The on-host
KvFile will look either locally or via SSH for that file, and TarGpg will
look inside a tar archive specified in the JW_SECRETS_TARGPG_TAR
environment variable.
Uris with these schemas can be passed to FileContext.create(), which will
return a chrooted file context of the correct type.
KvFile.stat("/path/to/file.conf.jw-secret").ftype is FT_SECRET which
inherits from FT_COMPOUND - which means that readdir() works on such
animals. Not entirely sure if that's a good idea, but it feels pretty
generic. Let's see where that takes us.
DistroContext.install() after this refactor reads its source through the
store interface and writes the landing through the KvFile store.
Both secret stores are exercised through the existing unit test of jw-pkg
secrets install, and unit tests for both stores are also added by this
commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a write-through read cache in front of a file context.
stat() and get() results are cached: stat per (path, follow_symlinks), get
per (path, wd, throw). Successful stats and FileNotFoundErrors are cached;
other errors are re-probed, so a transient transport failure does not
poison the cache. Writes pass through to the wrapped context and drop the
cache entries of the affected paths. The existence and directory checks are
rerouted through the cached stat, so they pay the cache instead of the
wrapped context's own fast path.
warm() fills the stat cache of a set of paths, as file_exists() would stat
them, without following symlinks. If the wrapped context is an exec
context, one host command stats all of the paths; otherwise, one stat per
path runs.
The GNU stat format of ExecContext is lifted to the module constant
STAT_GNU_FORMAT, and the field parser is exported as parse_stat_fields(),
so warm() fills entries that are indistinguishable from a real stat().
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a file context that sits in front of another one and delegates every
primitive call to it.
The wrapper adopts the identity of the wrapped context: uri, root, chroot
flag, and the chroot path composition are the wrapped context's, so a path
composed by the wrapper composes identically in the wrapped context. Every
primitive is forwarded to the wrapped context's primitives, which receive
the already-composed path; open() and close() are forwarded to the wrapped
context's public methods, so the open count is kept in one place.
The point is to give derived contexts a stable base: a caching context, or
any other filter, overrides the primitives it cares about and inherits the
rest, instead of re-deriving the chroot bookkeeping of every method.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() can now land a single object at its source path, but the copy
CLI command exposes no way to ask for it.
Add a --parents flag forwarded to copy()'s parents argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The walk needlessly descends into every directory node: A compound node is
gettable as a whole, so copy it over and don't recurse deeper.
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() lands a source below the destination root, taking at most the
source's basename, so there is no way to land it at its source path, as cp
--parents does.
Add a parents flag that requests just that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Using a FileContext, notably with its chroot flag set, is useful as a
tree source in copy(), so allow it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The file type framework gives mkdir() and put() an ftype argument, but
copy() calls them untyped, so the source's node types are discarded at the
landing: a destination that applies its own node types receives only the
defaults.
This commit threads the source's types through.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Refactor copy(), making the recursive logic more coherent: Every node the
walk creates sits below a node that already exists, so a plain per-node
mkdir() suffices and the chain- creating __mkdir_p() is dropped: the
landing root's parent must exist now, as in cp -r.
Signed-off-by: Jan Lindemann <jan@janware.com>
Give FileContext a read-only property to access self.__chroot, so a caller
holding a context can predict whether the addresses it passes in are
relative to the URI path or absolute.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Analogous to the is_dir() method, add an is_gettable() method: The last
commit introduces FT_COMPOUND, a file type which is a hybrid between
directory and regular file. is_dir() succeeds on it, so is_gettable()
symmetrically provides a test find out if get() also works on a file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Introduce FileType, a class consisting of a name and a parent property,
designed to make the information FileContext.stat() provides about a
path-addressable object richer.
A FileType property goes as ftype into the StatResult structure. Along with
it, this commit adds a central registry (FileTypes) where the type hierarchy
is kept, and additional parameters to put() and mkdir() to allow creating
typed FileContext objects.
Builtin types are:
- FT_REG: A regular file, the default thing put() creates
- FT_DIR: A directory, the default thing mkdir() creates
- FT_TOP: The parent of the two
- FT_COMPOUND: An FT_REG returning child entries from readdir()
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Add the "packages verify" command: it runs Distro.verify() with the app's
package filter and prints one line per flagged file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
jw-base verify pipes the output of rpm -qV for all janware packages
into a grep for md5 mismatches and missing files. The same capability,
distribution-independent, belongs in the Distro abstraction so that tools
like jw-nettools can check the integrity of the packages they install on a
fleet of hosts.
- Add Distro.verify(), which verifies the given package names, or all
packages selected by the package filter when none are given, and returns
the flagged files
- Add Distro._verify() as the distribution hook
- Add FlaggedFile, the record for a flagged file: package, file, and the
package manager's verify status line - not distribution independent as of
today, but it's a start
- pm/rpm.py: parse_verify_output() parses the output of `rpm -qV` and
reports md5 mismatches and missing files, the latter reported with the
literal word "missing" in the flag field; size, mode, mtime, user, and
group mismatches are noise on dev machines and are ignored.
verify_packages() runs `rpm -qV` per package and collects the findings
- pm/dpkg.py: parse_verify_output() parses the output of `dpkg -V` and
reports md5sum (P) mismatches and missing files (M); verify_packages()
runs `dpkg -V` per package
- The SUSE, Red Hat, and Debian Distro classes implement _verify() with the
respective pm helpers; the Arch Distro raises NotImplementedError, like
its other package manager hooks
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
