This commit adds the Vault store and the vault:// dispatch in FileContext.
A Vault KV v2 server maps nicely into jw-pkg's secret handling abstraction
as just another secret store backend. Environment variables identify the
storage details, analogous to the way it's handled in the SOPS backend.
- JW_SECRETS_VAULT_ADDR names the server, a unix socket address or an http
URL; the standard VAULT_ADDR is never read; JW_SECRETS_VAULT_MOUNT the
mount path, default secret; the token is the standard VAULT_TOKEN
- The transport is one curl per API call, through the Local context: a
unix:// server is reached through its socket, an http:// server through
its port
- A node's type is derived at stat time: an entry, a blob, a key below an
entry, a branch if the path has children, otherwise a miss
- A blob's content is base64 under a single key of its KV secret, the
marker being a path component, so a KV secret that is not a blob is
always an entry
The test drives a dev server fixture from the workspace: stat/readdir/get
on the fixture, the default fallback, the storage marker, and the copy
forms - single key, whole entry, recursive entry, deploy to the on-host
layout, and a host subtree.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
This commit adds a new class inheriting from secret_store.Base: Sops. The
currently available classes allow to store secrets offline in a
GPG-encrypted tarball, but those are not versioned. This commit allows to
use a SOPS / AGE-encrypted Git repository as storage backend instead.
The additions in detail:
- Sops: the store implementation; the repository location comes from
JW_SECRETS_SOPS_REPO, get() on an entry returns the decrypted serialized
content, and a typed put (FT_SECRET) re-encrypts the entry wholesale
- The age key comes from JW_SECRETS_SOPS_KEY_PATH only: the house variable
is the sole input to the store, and SOPS_AGE_KEY_FILE is set on the sops
subprocess per invocation, never read from the process environment;
SopsKeyError is raised when the variable is not set
- FileContext.create(): the sops:// scheme
- The store's unit tests, covering the node types, the deploy and fetch
round trip, the whole-entry get and put, the key resolution, and the
default fallback
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The TarGpg store is read-only: put() and unlink() raise, so the store can
be copied from, but not into, and a copy that lands entries and blobs below
a host prefix has no tarball target.
This matches the way jw-pkg secrets has worked for a long time. Since
Stores are now FileContext-backed, a write path can be cleanly implemented,
and that's what this commit does.
A put rebuilds the tarball: the existing members minus the removed, plus
the written one; the result is signed with the recipient from
JW_SECRETS_TARGPG_RECIPIENT and written back to the tarball location, which
must be a local path. A missing tarball is an empty store, so a put creates
it.
The put dispatch follows the other stores: a typed entry put replaces the
entry with the given serialized content, a key put merges into its entry,
creating it if it does not exist yet, and an untyped put below an entry
lands a key there, otherwise a blob. unlink() removes the node's member.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
"jw-pkg secrets install" copies secret material from a GPG-encrypted tar
archive to dotenv .jw-secret files on a target host. This commit refactors
the underlying code and adds hooks for the inverse copy direction, and for
more secret sources and targets.
The peers between which secrets are copied are represented by the new
abstract secret_store.Base class, and the existing tar blob and host
storages, inheriting from that class, are TarGpg and KvFile. Moreover,
secret_store.Base is inheriting from FileContext, so all of jw-pkg's
file-like operations are available as hooks.
Secrets inside instances of two file contexts are addressable as
(targpg|kvfile)://target.host.com/path/to/file.conf.jw-secret. The on-host
KvFile will look either locally or via SSH for that file, and TarGpg will
look inside a tar archive specified in the JW_SECRETS_TARGPG_TAR
environment variable.
Uris with these schemas can be passed to FileContext.create(), which will
return a chrooted file context of the correct type.
KvFile.stat("/path/to/file.conf.jw-secret").ftype is FT_SECRET which
inherits from FT_COMPOUND - which means that readdir() works on such
animals. Not entirely sure if that's a good idea, but it feels pretty
generic. Let's see where that takes us.
DistroContext.install() after this refactor reads its source through the
store interface and writes the landing through the KvFile store.
Both secret stores are exercised through the existing unit test of jw-pkg
secrets install, and unit tests for both stores are also added by this
commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__init_askpass() embeds the password verbatim in a generated bash script,
with a newline included inside the double quotes of the echo -n. The
askpass program therefore always appends a newline to the password, and
any password containing quotes or shell metacharacters either breaks the
script or injects commands into it.
Embed the password as base64 and decode it with printf piped into base64
-d, so the script is safe for any password and prints the password exactly,
byte for byte.
Make __del__() idempotent: it deletes the environment variables and the
script file, so a second call, e.g. an explicit one followed by garbage
collection, raises KeyError and FileNotFoundError.
Add a unit test that executes the generated script and compares its output
with the password byte for byte.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
