This commit adds the Vault store and the vault:// dispatch in FileContext.
A Vault KV v2 server maps nicely into jw-pkg's secret handling abstraction
as just another secret store backend. Environment variables identify the
storage details, analogous to the way it's handled in the SOPS backend.
- JW_SECRETS_VAULT_ADDR names the server, a unix socket address or an http
URL; the standard VAULT_ADDR is never read; JW_SECRETS_VAULT_MOUNT the
mount path, default secret; the token is the standard VAULT_TOKEN
- The transport is one curl per API call, through the Local context: a
unix:// server is reached through its socket, an http:// server through
its port
- A node's type is derived at stat time: an entry, a blob, a key below an
entry, a branch if the path has children, otherwise a miss
- A blob's content is base64 under a single key of its KV secret, the
marker being a path component, so a KV secret that is not a blob is
always an entry
The test drives a dev server fixture from the workspace: stat/readdir/get
on the fixture, the default fallback, the storage marker, and the copy
forms - single key, whole entry, recursive entry, deploy to the on-host
layout, and a host subtree.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
This commit adds a new class inheriting from secret_store.Base: Sops. The
currently available classes allow to store secrets offline in a
GPG-encrypted tarball, but those are not versioned. This commit allows to
use a SOPS / AGE-encrypted Git repository as storage backend instead.
The additions in detail:
- Sops: the store implementation; the repository location comes from
JW_SECRETS_SOPS_REPO, get() on an entry returns the decrypted serialized
content, and a typed put (FT_SECRET) re-encrypts the entry wholesale
- The age key comes from JW_SECRETS_SOPS_KEY_PATH only: the house variable
is the sole input to the store, and SOPS_AGE_KEY_FILE is set on the sops
subprocess per invocation, never read from the process environment;
SopsKeyError is raised when the variable is not set
- FileContext.create(): the sops:// scheme
- The store's unit tests, covering the node types, the deploy and fetch
round trip, the whole-entry get and put, the key resolution, and the
default fallback
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The TarGpg store is read-only: put() and unlink() raise, so the store can
be copied from, but not into, and a copy that lands entries and blobs below
a host prefix has no tarball target.
This matches the way jw-pkg secrets has worked for a long time. Since
Stores are now FileContext-backed, a write path can be cleanly implemented,
and that's what this commit does.
A put rebuilds the tarball: the existing members minus the removed, plus
the written one; the result is signed with the recipient from
JW_SECRETS_TARGPG_RECIPIENT and written back to the tarball location, which
must be a local path. A missing tarball is an empty store, so a put creates
it.
The put dispatch follows the other stores: a typed entry put replaces the
entry with the given serialized content, a key put merges into its entry,
creating it if it does not exist yet, and an untyped put below an entry
lands a key there, otherwise a blob. unlink() removes the node's member.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
"jw-pkg secrets install" copies secret material from a GPG-encrypted tar
archive to dotenv .jw-secret files on a target host. This commit refactors
the underlying code and adds hooks for the inverse copy direction, and for
more secret sources and targets.
The peers between which secrets are copied are represented by the new
abstract secret_store.Base class, and the existing tar blob and host
storages, inheriting from that class, are TarGpg and KvFile. Moreover,
secret_store.Base is inheriting from FileContext, so all of jw-pkg's
file-like operations are available as hooks.
Secrets inside instances of two file contexts are addressable as
(targpg|kvfile)://target.host.com/path/to/file.conf.jw-secret. The on-host
KvFile will look either locally or via SSH for that file, and TarGpg will
look inside a tar archive specified in the JW_SECRETS_TARGPG_TAR
environment variable.
Uris with these schemas can be passed to FileContext.create(), which will
return a chrooted file context of the correct type.
KvFile.stat("/path/to/file.conf.jw-secret").ftype is FT_SECRET which
inherits from FT_COMPOUND - which means that readdir() works on such
animals. Not entirely sure if that's a good idea, but it feels pretty
generic. Let's see where that takes us.
DistroContext.install() after this refactor reads its source through the
store interface and writes the landing through the KvFile store.
Both secret stores are exercised through the existing unit test of jw-pkg
secrets install, and unit tests for both stores are also added by this
commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a write-through read cache in front of a file context.
stat() and get() results are cached: stat per (path, follow_symlinks), get
per (path, wd, throw). Successful stats and FileNotFoundErrors are cached;
other errors are re-probed, so a transient transport failure does not
poison the cache. Writes pass through to the wrapped context and drop the
cache entries of the affected paths. The existence and directory checks are
rerouted through the cached stat, so they pay the cache instead of the
wrapped context's own fast path.
warm() fills the stat cache of a set of paths, as file_exists() would stat
them, without following symlinks. If the wrapped context is an exec
context, one host command stats all of the paths; otherwise, one stat per
path runs.
The GNU stat format of ExecContext is lifted to the module constant
STAT_GNU_FORMAT, and the field parser is exported as parse_stat_fields(),
so warm() fills entries that are indistinguishable from a real stat().
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a file context that sits in front of another one and delegates every
primitive call to it.
The wrapper adopts the identity of the wrapped context: uri, root, chroot
flag, and the chroot path composition are the wrapped context's, so a path
composed by the wrapper composes identically in the wrapped context. Every
primitive is forwarded to the wrapped context's primitives, which receive
the already-composed path; open() and close() are forwarded to the wrapped
context's public methods, so the open count is kept in one place.
The point is to give derived contexts a stable base: a caching context, or
any other filter, overrides the primitives it cares about and inherits the
rest, instead of re-deriving the chroot bookkeeping of every method.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() can now land a single object at its source path, but the copy
CLI command exposes no way to ask for it.
Add a --parents flag forwarded to copy()'s parents argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The walk needlessly descends into every directory node: A compound node is
gettable as a whole, so copy it over and don't recurse deeper.
Signed-off-by: Jan Lindemann <jan@janware.com>
copy() lands a source below the destination root, taking at most the
source's basename, so there is no way to land it at its source path, as cp
--parents does.
Add a parents flag that requests just that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Using a FileContext, notably with its chroot flag set, is useful as a
tree source in copy(), so allow it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The file type framework gives mkdir() and put() an ftype argument, but
copy() calls them untyped, so the source's node types are discarded at the
landing: a destination that applies its own node types receives only the
defaults.
This commit threads the source's types through.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Refactor copy(), making the recursive logic more coherent: Every node the
walk creates sits below a node that already exists, so a plain per-node
mkdir() suffices and the chain- creating __mkdir_p() is dropped: the
landing root's parent must exist now, as in cp -r.
Signed-off-by: Jan Lindemann <jan@janware.com>
Give FileContext a read-only property to access self.__chroot, so a caller
holding a context can predict whether the addresses it passes in are
relative to the URI path or absolute.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Analogous to the is_dir() method, add an is_gettable() method: The last
commit introduces FT_COMPOUND, a file type which is a hybrid between
directory and regular file. is_dir() succeeds on it, so is_gettable()
symmetrically provides a test find out if get() also works on a file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Introduce FileType, a class consisting of a name and a parent property,
designed to make the information FileContext.stat() provides about a
path-addressable object richer.
A FileType property goes as ftype into the StatResult structure. Along with
it, this commit adds a central registry (FileTypes) where the type hierarchy
is kept, and additional parameters to put() and mkdir() to allow creating
typed FileContext objects.
Builtin types are:
- FT_REG: A regular file, the default thing put() creates
- FT_DIR: A directory, the default thing mkdir() creates
- FT_TOP: The parent of the two
- FT_COMPOUND: An FT_REG returning child entries from readdir()
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Add the "packages verify" command: it runs Distro.verify() with the app's
package filter and prints one line per flagged file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
jw-base verify pipes the output of rpm -qV for all janware packages
into a grep for md5 mismatches and missing files. The same capability,
distribution-independent, belongs in the Distro abstraction so that tools
like jw-nettools can check the integrity of the packages they install on a
fleet of hosts.
- Add Distro.verify(), which verifies the given package names, or all
packages selected by the package filter when none are given, and returns
the flagged files
- Add Distro._verify() as the distribution hook
- Add FlaggedFile, the record for a flagged file: package, file, and the
package manager's verify status line - not distribution independent as of
today, but it's a start
- pm/rpm.py: parse_verify_output() parses the output of `rpm -qV` and
reports md5 mismatches and missing files, the latter reported with the
literal word "missing" in the flag field; size, mode, mtime, user, and
group mismatches are noise on dev machines and are ignored.
verify_packages() runs `rpm -qV` per package and collects the findings
- pm/dpkg.py: parse_verify_output() parses the output of `dpkg -V` and
reports md5sum (P) mismatches and missing files (M); verify_packages()
runs `dpkg -V` per package
- The SUSE, Red Hat, and Debian Distro classes implement _verify() with the
respective pm helpers; the Arch Distro raises NotImplementedError, like
its other package manager hooks
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
Distro.pkg_files() takes a package name and returns a list of files in that
package. This commit generalizes that - pass None to it and get the list of
all packaged files in the system, and pass a list of strings to retrieve
all files belonging to the named packages.
The same arguments are propagated to _pkg_files(), which in turn propagates
them to list_files().
This allows faster generation of file lists than before. Having to retrieve
a list of matching packages and query each package for its contained files
can be collapsed into one or two backend calls.
Signed-off-by: Jan Lindemann <jan@janware.com>
The mypy module names of the test tree derive from the full path, anchored
by the top-level test/__init__.py, but a file below a path component that
is not a valid identifier - the dashes in jw-pkg - falls back to the bare
file name, so two same-named test.py drivers collide (the duplicate module
error that drove the runme.py and main.py renames).
Rename the directory a sane Python name - jw_pkg, that's the better fix.
Then name the driver test.py like all the others for consistency.
Signed-off-by: Jan Lindemann <jan@janware.com>
PYTHON_RUNNER is defined in py-run.mk but is more generally useful, e.g.
for hand-rolled test Makefiles. It's also a cheap definition. Lift it into
py-defs.mk instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The PYTHONPATH export in py-path.mk is built by substituting the spaces of
the space-separated JW_PKG_PYTHON_PATH list with colons, using "space" from
defs.mk. Which may not be included if somebody wants to include a bare
py-defs.mk, so define it in platform.mk. Not happening in jw-pkg, but nice
to have.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-path.mk invokes $(JW_PKG_PY) to compute JW_PKG_PYTHON_PATH, but does not
include the file that defines the variable. Not a problem from inside
jw-pkg, but nice to have if you want to include std-targets.mk and
py-defs.mk without anything else, so include it.
Also drop a dead debug line in the py-path target.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
INSTALL_LOG's default value is composed from a variable that's never set,
$ID. That used to work until the same user builds multiple packages
concurrently on the same host. Use $PROJECT instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
File attributes of the compiled output of "jw-pkg secrets
compile-templates" can be specified via the # conf line atop a .jw-tmpl
file, the parsing of that line is buggy, however.
Its mode is parsed as a decimal number unless it carries a leading zero, so
"mode=640" yields 640 decimal, 1200 octal, and the output file ends up
without usable permissions. The same decimal parse applies to a conf line atop
the .jw-secret of a target, whose attributes are honored the same way, and
the --mode option of compile-templates passes its string value through to
Attrs, whose mode is typed int, so a target without a conf line of its own
crashes in _format_metadata().
Other than that, the # conf line itself should be removed from the compiled
output, but isn't.
This commit makes the conf line handling correct:
- _parse_attributes() parses the mode as octal, with or without a leading
zero
- _compile_one_template_file() drops the conf line from the compiled
content
- the --mode option is converted to an octal int by octal_mode() at the
argument boundary
The integration test writes its conf line modes without leading zeros,
expects the conf line gone from the compiled output, takes one target's
mode from a conf line atop its .jw-secret, and passes a --mode that the
conf lines override.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Drive the real "jw-pkg secrets" CLI in-process against a GPG-encrypted
tarball in the canonical layout: reverse-domain host prefixes plus a
default tree, host wins. The distro is stubbed only in its package file
list; argument parsing, DistroContext, install, template compilation and
cleanup run as production code.
The fixture package carries three templates: one secret present in both
host and default trees (the host entry must win), one present only in the
default tree, and one missing from the tarball entirely (install must not
extract it). A tar member that is not a secret of the package must not be
extracted either.
Trees whose FileContext supports the targpg:// scheme address the tar
source through that scheme, with the tarball location in the environment,
and additionally exercise the rooted-URI rejection and the install from
the on-host file tree; earlier trees install from the tarball URI
directly.
The Makefile's SECRETS_TEST_BASE variable pins the fixture to a persistent
directory that survives the test run for inspection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
The !-exceptions in the generated .gitignore are computed by a grep pattern
that hardcodes the project name, while the top ignore line comes from
GITIGNORE_PATTERNS_TOP. A project that overrides GITIGNORE_PATTERNS_TOP -
jw-pkg's Makefile sets it to empty - still gets the hardcoded exceptions,
and patterns a project adds there get no exceptions at all, so the two can
drift apart.
Derive the exception pattern from the same variable:
- new GITIGNORE_PATTERNS_TOP_INV joins the patterns of
GITIGNORE_PATTERNS_TOP with |, each wrapped in word boundaries, plus the
__init__.py pattern
- GITIGNORE_PATTERNS_END greps the tracked files for that alternation with
grep -E instead of a hardcoded BRE, so the dot in __init__.py is literal
instead of matching any character
With the default GITIGNORE_PATTERNS_TOP the generated .gitignore is
byte-for-byte identical to before.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
rewrite() renames the members of a tar blob, but the TarInfo objects
carried over from the input still hold the pax "path" supplement that was
recorded for the original name. Re-serialization (tf_out.addfile()) uses
the pax supplement with higher precedence than what's in the base header's
name filed, so for member names exceeding the 100 character ustar limit the
rewrite silently has no effect and the original name is written back.
Drop the stale "path" supplement after renaming a member. tf_out.addfile()
then builds a rewritten, up-to-date supplement from the name field if it
needs one.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The -m option of posix copy is optional, and its argparse default is None,
but _run() converts it unconditionally and crashes:
mode = int(args.mode, 0)
so every copy without -m died with
TypeError: int() can't convert non-string with explicit base
before the copy even started. Convert the mode only when it is given, and
pass None otherwise, which copy() already treats as "no mode change".
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
_raise_stat_error() maps the message of a failed remote stat to an
exception. A stat below a file answers "not a directory", which falls
through to the generic EIO: the path is simply absent, and callers see a
re-raised error instead of the FileNotFoundError the "no such file" branch
already gives. Map it to FileNotFoundError, so existence checks see
absence.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg secrets list-secrets spams the console without --interactive=false
because it thinks it should query packages and files in interactive mode.
The underlying bug is that run_cmd() derives the interactive flag of a
freshly created Local context from cmd_input, but it never forwards
cmd_input to ExecContext.run(). Since run() defaults to
InputMode.OptInteractive, an explicit cmd_input is silently ignored: a byte
string is not fed to the command's stdin, InputMode.Auto is downgraded to
non-interactive, and with a caller-provided ExecContext the requested mode
is lost in all cases.
Forward cmd_input in kwargs so the requested input mode reaches run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
mkdir() passes its path to _mkdir() without chrooting it, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Signed-off-by: Jan Lindemann <jan@janware.com>
Target git-import adds everything it finds in the current directory. If
it's already a Git project, that fails over git commit -m "First commit".
Since it's a prerequisite of target git-init-remote, it breaks that target,
too for non-empty git repos, which isn't helpful when trying to run make
git-init-remote to publish an an already existing project.
So only do something meaningful in git-import if the project it's run in
doesn't already contain files.
Signed-off-by: Jan Lindemann <jan@janware.com>
