Introduce FileType, a class consisting of a name and a parent property,
designed to make the information FileContext.stat() provides about a
path-addressable object richer.
A FileType property goes as ftype into the StatResult structure. Along with
it, this commit adds a central registry (FileTypes) where the type hierarchy
is kept, and additional parameters to put() and mkdir() to allow creating
typed FileContext objects.
Builtin types are:
- FT_REG: A regular file, the default thing put() creates
- FT_DIR: A directory, the default thing mkdir() creates
- FT_TOP: The parent of the two
- FT_COMPOUND: An FT_REG returning child entries from readdir()
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.1
Add the "packages verify" command: it runs Distro.verify() with the app's
package filter and prints one line per flagged file.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
jw-base verify pipes the output of rpm -qV for all janware packages
into a grep for md5 mismatches and missing files. The same capability,
distribution-independent, belongs in the Distro abstraction so that tools
like jw-nettools can check the integrity of the packages they install on a
fleet of hosts.
- Add Distro.verify(), which verifies the given package names, or all
packages selected by the package filter when none are given, and returns
the flagged files
- Add Distro._verify() as the distribution hook
- Add FlaggedFile, the record for a flagged file: package, file, and the
package manager's verify status line - not distribution independent as of
today, but it's a start
- pm/rpm.py: parse_verify_output() parses the output of `rpm -qV` and
reports md5 mismatches and missing files, the latter reported with the
literal word "missing" in the flag field; size, mode, mtime, user, and
group mismatches are noise on dev machines and are ignored.
verify_packages() runs `rpm -qV` per package and collects the findings
- pm/dpkg.py: parse_verify_output() parses the output of `dpkg -V` and
reports md5sum (P) mismatches and missing files (M); verify_packages()
runs `dpkg -V` per package
- The SUSE, Red Hat, and Debian Distro classes implement _verify() with the
respective pm helpers; the Arch Distro raises NotImplementedError, like
its other package manager hooks
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.2.0
Distro.pkg_files() takes a package name and returns a list of files in that
package. This commit generalizes that - pass None to it and get the list of
all packaged files in the system, and pass a list of strings to retrieve
all files belonging to the named packages.
The same arguments are propagated to _pkg_files(), which in turn propagates
them to list_files().
This allows faster generation of file lists than before. Having to retrieve
a list of matching packages and query each package for its contained files
can be collapsed into one or two backend calls.
Signed-off-by: Jan Lindemann <jan@janware.com>
The mypy module names of the test tree derive from the full path, anchored
by the top-level test/__init__.py, but a file below a path component that
is not a valid identifier - the dashes in jw-pkg - falls back to the bare
file name, so two same-named test.py drivers collide (the duplicate module
error that drove the runme.py and main.py renames).
Rename the directory a sane Python name - jw_pkg, that's the better fix.
Then name the driver test.py like all the others for consistency.
Signed-off-by: Jan Lindemann <jan@janware.com>
PYTHON_RUNNER is defined in py-run.mk but is more generally useful, e.g.
for hand-rolled test Makefiles. It's also a cheap definition. Lift it into
py-defs.mk instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The PYTHONPATH export in py-path.mk is built by substituting the spaces of
the space-separated JW_PKG_PYTHON_PATH list with colons, using "space" from
defs.mk. Which may not be included if somebody wants to include a bare
py-defs.mk, so define it in platform.mk. Not happening in jw-pkg, but nice
to have.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-path.mk invokes $(JW_PKG_PY) to compute JW_PKG_PYTHON_PATH, but does not
include the file that defines the variable. Not a problem from inside
jw-pkg, but nice to have if you want to include std-targets.mk and
py-defs.mk without anything else, so include it.
Also drop a dead debug line in the py-path target.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
INSTALL_LOG's default value is composed from a variable that's never set,
$ID. That used to work until the same user builds multiple packages
concurrently on the same host. Use $PROJECT instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
File attributes of the compiled output of "jw-pkg secrets
compile-templates" can be specified via the # conf line atop a .jw-tmpl
file, the parsing of that line is buggy, however.
Its mode is parsed as a decimal number unless it carries a leading zero, so
"mode=640" yields 640 decimal, 1200 octal, and the output file ends up
without usable permissions. The same decimal parse applies to a conf line atop
the .jw-secret of a target, whose attributes are honored the same way, and
the --mode option of compile-templates passes its string value through to
Attrs, whose mode is typed int, so a target without a conf line of its own
crashes in _format_metadata().
Other than that, the # conf line itself should be removed from the compiled
output, but isn't.
This commit makes the conf line handling correct:
- _parse_attributes() parses the mode as octal, with or without a leading
zero
- _compile_one_template_file() drops the conf line from the compiled
content
- the --mode option is converted to an octal int by octal_mode() at the
argument boundary
The integration test writes its conf line modes without leading zeros,
expects the conf line gone from the compiled output, takes one target's
mode from a conf line atop its .jw-secret, and passes a --mode that the
conf lines override.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Drive the real "jw-pkg secrets" CLI in-process against a GPG-encrypted
tarball in the canonical layout: reverse-domain host prefixes plus a
default tree, host wins. The distro is stubbed only in its package file
list; argument parsing, DistroContext, install, template compilation and
cleanup run as production code.
The fixture package carries three templates: one secret present in both
host and default trees (the host entry must win), one present only in the
default tree, and one missing from the tarball entirely (install must not
extract it). A tar member that is not a secret of the package must not be
extracted either.
Trees whose FileContext supports the targpg:// scheme address the tar
source through that scheme, with the tarball location in the environment,
and additionally exercise the rooted-URI rejection and the install from
the on-host file tree; earlier trees install from the tarball URI
directly.
The Makefile's SECRETS_TEST_BASE variable pins the fixture to a persistent
directory that survives the test run for inspection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
The !-exceptions in the generated .gitignore are computed by a grep pattern
that hardcodes the project name, while the top ignore line comes from
GITIGNORE_PATTERNS_TOP. A project that overrides GITIGNORE_PATTERNS_TOP -
jw-pkg's Makefile sets it to empty - still gets the hardcoded exceptions,
and patterns a project adds there get no exceptions at all, so the two can
drift apart.
Derive the exception pattern from the same variable:
- new GITIGNORE_PATTERNS_TOP_INV joins the patterns of
GITIGNORE_PATTERNS_TOP with |, each wrapped in word boundaries, plus the
__init__.py pattern
- GITIGNORE_PATTERNS_END greps the tracked files for that alternation with
grep -E instead of a hardcoded BRE, so the dot in __init__.py is literal
instead of matching any character
With the default GITIGNORE_PATTERNS_TOP the generated .gitignore is
byte-for-byte identical to before.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
rewrite() renames the members of a tar blob, but the TarInfo objects
carried over from the input still hold the pax "path" supplement that was
recorded for the original name. Re-serialization (tf_out.addfile()) uses
the pax supplement with higher precedence than what's in the base header's
name filed, so for member names exceeding the 100 character ustar limit the
rewrite silently has no effect and the original name is written back.
Drop the stale "path" supplement after renaming a member. tf_out.addfile()
then builds a rewritten, up-to-date supplement from the name field if it
needs one.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The -m option of posix copy is optional, and its argparse default is None,
but _run() converts it unconditionally and crashes:
mode = int(args.mode, 0)
so every copy without -m died with
TypeError: int() can't convert non-string with explicit base
before the copy even started. Convert the mode only when it is given, and
pass None otherwise, which copy() already treats as "no mode change".
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
_raise_stat_error() maps the message of a failed remote stat to an
exception. A stat below a file answers "not a directory", which falls
through to the generic EIO: the path is simply absent, and callers see a
re-raised error instead of the FileNotFoundError the "no such file" branch
already gives. Map it to FileNotFoundError, so existence checks see
absence.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg secrets list-secrets spams the console without --interactive=false
because it thinks it should query packages and files in interactive mode.
The underlying bug is that run_cmd() derives the interactive flag of a
freshly created Local context from cmd_input, but it never forwards
cmd_input to ExecContext.run(). Since run() defaults to
InputMode.OptInteractive, an explicit cmd_input is silently ignored: a byte
string is not fed to the command's stdin, InputMode.Auto is downgraded to
non-interactive, and with a caller-provided ExecContext the requested mode
is lost in all cases.
Forward cmd_input in kwargs so the requested input mode reaches run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
mkdir() passes its path to _mkdir() without chrooting it, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Signed-off-by: Jan Lindemann <jan@janware.com>
Target git-import adds everything it finds in the current directory. If
it's already a Git project, that fails over git commit -m "First commit".
Since it's a prerequisite of target git-init-remote, it breaks that target,
too for non-empty git repos, which isn't helpful when trying to run make
git-init-remote to publish an an already existing project.
So only do something meaningful in git-import if the project it's run in
doesn't already contain files.
Signed-off-by: Jan Lindemann <jan@janware.com>
projects_dir is searched by looking for a directory named .git up the
direcctory lineage. That does't work for worktrees, which are regular files
pointing back to worktree git dir within the the real git directory.
Broaden the check from "test -d .git" to "test -e .git".
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commit renamed the dist/pckg directory to dist/pkg but kept
the pckg spelling in the variables that reference it.
Rename the variables to PKG:
- DIST_PCKG_DIR to DIST_PKG_DIR (make/pkg-dist.mk, pkg.sh)
- PCKG_TAR to PKG_TAR, PCKG_ROOT to PKG_ROOT and DIST_PCKG_RPM to
DIST_PKG_RPM (make/pkg-dist.mk)
None of the other 320 janware projects use these variables.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The directory holding the built packages uses the pckg abbreviation while
every other package-related name in the build system uses pkg (PKG_PROJECT,
PKG_FORMAT, PKG_ARCH).
Rename dist/pckg to dist/pkg, in the DIST_PCKG_DIR variable and in the
recipes that hardcode the path.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
pkg.sh hardcodes $HOME/rpmbuild as the rpmbuild topdir while make computes
the same location separately as PKG_DIR_BUILD, and a sandboxed shell cannot
build at all: rpmbuild writes its scratch trees and results under the
topdir, outside the project.
- Default PKG_DIR_BUILD to $(TOPDIR)/dist/build for non-root builds and
pass it to pkg.sh as the new -T option (root keeps /usr/src/packages;
local.mk can still override)
- Build with --define _topdir pointing at that directory, resolved to an
absolute path, so the scratch tree, RPMS and SRPMS stay in the project
tree where make clean drops them
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Remove INSTALL_PCKG_DEPS_DIR and friends from jw-pkg. It was introduced
with 2a4a95 in 2009 and has not seen much use since. By now it has been
made obsolete by declaring dependencies in project.conf. One source of
truth is enough.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add an integration test for pkg-requires --skip-excluded with a fixture
project tree: excl-never lists the platform name "centos" in [build]
exclude and is not excluded from the build on the test platforms, excl-all
is excluded from the build everywhere, and dep-target is a plain dependency
target.
The test verifies that a platform name in [build] exclude is not treated as
a project, that an excluded jw dependency is not reported with
--skip-excluded, that an excluded seed is neither considered nor output,
and that without the flag the excluded projects are reported with their
dependencies.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The [build] exclude list names platforms (OS cascade entries) on which a
project's build is excluded, not projects. pkg_relations() implements
--skip-excluded by walking the seeds' exclude values as if they were
project specs, so a value such as "debian" is visited as a project, the
installed-package check fails with "Unmet dependency on debian: the -run
package of project debian is not installed", and `make
pkg-install-testbuild-deps` breaks.
Implement the flag's documented semantics using
App.is_excluded_from_build(): an excluded seed is neither considered nor
output, an excluded module in the dependency closure contributes no
dependencies, and excluded modules are not output as jw dependencies. This
matches what required-os-pkg already does.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
platform.mk needs JWB_SCRIPT_DIR before it checks if TARGET_TUPLE is empty,
otherwise downstream projects including platform.mk without defs.mk error
out (as does jw-bind-zones-ldap-priv via projects.mk:
/usr/bin/python3.13: can't open file '/jw-pkg.py': [Errno 2] No such file or directory
../jw-pkg/make/platform.mk:172: *** TARGET_TUPLE is empty. Schluss..
The fix is to include common.mk earlier in the stack in platform.mk.
defs.mk then includes it via projects.mk, so remove direct inclusion from
there.
Signed-off-by: Jan Lindemann <jan@janware.com>
