The mypy module names of the test tree derive from the full path, anchored
by the top-level test/__init__.py, but a file below a path component that
is not a valid identifier - the dashes in jw-pkg - falls back to the bare
file name, so two same-named test.py drivers collide (the duplicate module
error that drove the runme.py and main.py renames).
Rename the directory a sane Python name - jw_pkg, that's the better fix.
Then name the driver test.py like all the others for consistency.
Signed-off-by: Jan Lindemann <jan@janware.com>
PYTHON_RUNNER is defined in py-run.mk but is more generally useful, e.g.
for hand-rolled test Makefiles. It's also a cheap definition. Lift it into
py-defs.mk instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
The PYTHONPATH export in py-path.mk is built by substituting the spaces of
the space-separated JW_PKG_PYTHON_PATH list with colons, using "space" from
defs.mk. Which may not be included if somebody wants to include a bare
py-defs.mk, so define it in platform.mk. Not happening in jw-pkg, but nice
to have.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-path.mk invokes $(JW_PKG_PY) to compute JW_PKG_PYTHON_PATH, but does not
include the file that defines the variable. Not a problem from inside
jw-pkg, but nice to have if you want to include std-targets.mk and
py-defs.mk without anything else, so include it.
Also drop a dead debug line in the py-path target.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
INSTALL_LOG's default value is composed from a variable that's never set,
$ID. That used to work until the same user builds multiple packages
concurrently on the same host. Use $PROJECT instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
File attributes of the compiled output of "jw-pkg secrets
compile-templates" can be specified via the # conf line atop a .jw-tmpl
file, the parsing of that line is buggy, however.
Its mode is parsed as a decimal number unless it carries a leading zero, so
"mode=640" yields 640 decimal, 1200 octal, and the output file ends up
without usable permissions. The same decimal parse applies to a conf line atop
the .jw-secret of a target, whose attributes are honored the same way, and
the --mode option of compile-templates passes its string value through to
Attrs, whose mode is typed int, so a target without a conf line of its own
crashes in _format_metadata().
Other than that, the # conf line itself should be removed from the compiled
output, but isn't.
This commit makes the conf line handling correct:
- _parse_attributes() parses the mode as octal, with or without a leading
zero
- _compile_one_template_file() drops the conf line from the compiled
content
- the --mode option is converted to an octal int by octal_mode() at the
argument boundary
The integration test writes its conf line modes without leading zeros,
expects the conf line gone from the compiled output, takes one target's
mode from a conf line atop its .jw-secret, and passes a --mode that the
conf lines override.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.1.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Drive the real "jw-pkg secrets" CLI in-process against a GPG-encrypted
tarball in the canonical layout: reverse-domain host prefixes plus a
default tree, host wins. The distro is stubbed only in its package file
list; argument parsing, DistroContext, install, template compilation and
cleanup run as production code.
The fixture package carries three templates: one secret present in both
host and default trees (the host entry must win), one present only in the
default tree, and one missing from the tarball entirely (install must not
extract it). A tar member that is not a secret of the package must not be
extracted either.
Trees whose FileContext supports the targpg:// scheme address the tar
source through that scheme, with the tarball location in the environment,
and additionally exercise the rooted-URI rejection and the install from
the on-host file tree; earlier trees install from the tarball URI
directly.
The Makefile's SECRETS_TEST_BASE variable pins the fixture to a persistent
directory that survives the test run for inspection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
The !-exceptions in the generated .gitignore are computed by a grep pattern
that hardcodes the project name, while the top ignore line comes from
GITIGNORE_PATTERNS_TOP. A project that overrides GITIGNORE_PATTERNS_TOP -
jw-pkg's Makefile sets it to empty - still gets the hardcoded exceptions,
and patterns a project adds there get no exceptions at all, so the two can
drift apart.
Derive the exception pattern from the same variable:
- new GITIGNORE_PATTERNS_TOP_INV joins the patterns of
GITIGNORE_PATTERNS_TOP with |, each wrapped in word boundaries, plus the
__init__.py pattern
- GITIGNORE_PATTERNS_END greps the tracked files for that alternation with
grep -E instead of a hardcoded BRE, so the dot in __init__.py is literal
instead of matching any character
With the default GITIGNORE_PATTERNS_TOP the generated .gitignore is
byte-for-byte identical to before.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
rewrite() renames the members of a tar blob, but the TarInfo objects
carried over from the input still hold the pax "path" supplement that was
recorded for the original name. Re-serialization (tf_out.addfile()) uses
the pax supplement with higher precedence than what's in the base header's
name filed, so for member names exceeding the 100 character ustar limit the
rewrite silently has no effect and the original name is written back.
Drop the stale "path" supplement after renaming a member. tf_out.addfile()
then builds a rewritten, up-to-date supplement from the name field if it
needs one.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v1.0.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The -m option of posix copy is optional, and its argparse default is None,
but _run() converts it unconditionally and crashes:
mode = int(args.mode, 0)
so every copy without -m died with
TypeError: int() can't convert non-string with explicit base
before the copy even started. Convert the mode only when it is given, and
pass None otherwise, which copy() already treats as "no mode change".
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
_raise_stat_error() maps the message of a failed remote stat to an
exception. A stat below a file answers "not a directory", which falls
through to the generic EIO: the path is simply absent, and callers see a
re-raised error instead of the FileNotFoundError the "no such file" branch
already gives. Map it to FileNotFoundError, so existence checks see
absence.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg secrets list-secrets spams the console without --interactive=false
because it thinks it should query packages and files in interactive mode.
The underlying bug is that run_cmd() derives the interactive flag of a
freshly created Local context from cmd_input, but it never forwards
cmd_input to ExecContext.run(). Since run() defaults to
InputMode.OptInteractive, an explicit cmd_input is silently ignored: a byte
string is not fed to the command's stdin, InputMode.Auto is downgraded to
non-interactive, and with a caller-provided ExecContext the requested mode
is lost in all cases.
Forward cmd_input in kwargs so the requested input mode reaches run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
mkdir() passes its path to _mkdir() without chrooting it, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Signed-off-by: Jan Lindemann <jan@janware.com>
Target git-import adds everything it finds in the current directory. If
it's already a Git project, that fails over git commit -m "First commit".
Since it's a prerequisite of target git-init-remote, it breaks that target,
too for non-empty git repos, which isn't helpful when trying to run make
git-init-remote to publish an an already existing project.
So only do something meaningful in git-import if the project it's run in
doesn't already contain files.
Signed-off-by: Jan Lindemann <jan@janware.com>
projects_dir is searched by looking for a directory named .git up the
direcctory lineage. That does't work for worktrees, which are regular files
pointing back to worktree git dir within the the real git directory.
Broaden the check from "test -d .git" to "test -e .git".
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commit renamed the dist/pckg directory to dist/pkg but kept
the pckg spelling in the variables that reference it.
Rename the variables to PKG:
- DIST_PCKG_DIR to DIST_PKG_DIR (make/pkg-dist.mk, pkg.sh)
- PCKG_TAR to PKG_TAR, PCKG_ROOT to PKG_ROOT and DIST_PCKG_RPM to
DIST_PKG_RPM (make/pkg-dist.mk)
None of the other 320 janware projects use these variables.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The directory holding the built packages uses the pckg abbreviation while
every other package-related name in the build system uses pkg (PKG_PROJECT,
PKG_FORMAT, PKG_ARCH).
Rename dist/pckg to dist/pkg, in the DIST_PCKG_DIR variable and in the
recipes that hardcode the path.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
pkg.sh hardcodes $HOME/rpmbuild as the rpmbuild topdir while make computes
the same location separately as PKG_DIR_BUILD, and a sandboxed shell cannot
build at all: rpmbuild writes its scratch trees and results under the
topdir, outside the project.
- Default PKG_DIR_BUILD to $(TOPDIR)/dist/build for non-root builds and
pass it to pkg.sh as the new -T option (root keeps /usr/src/packages;
local.mk can still override)
- Build with --define _topdir pointing at that directory, resolved to an
absolute path, so the scratch tree, RPMS and SRPMS stay in the project
tree where make clean drops them
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Remove INSTALL_PCKG_DEPS_DIR and friends from jw-pkg. It was introduced
with 2a4a95 in 2009 and has not seen much use since. By now it has been
made obsolete by declaring dependencies in project.conf. One source of
truth is enough.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add an integration test for pkg-requires --skip-excluded with a fixture
project tree: excl-never lists the platform name "centos" in [build]
exclude and is not excluded from the build on the test platforms, excl-all
is excluded from the build everywhere, and dep-target is a plain dependency
target.
The test verifies that a platform name in [build] exclude is not treated as
a project, that an excluded jw dependency is not reported with
--skip-excluded, that an excluded seed is neither considered nor output,
and that without the flag the excluded projects are reported with their
dependencies.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The [build] exclude list names platforms (OS cascade entries) on which a
project's build is excluded, not projects. pkg_relations() implements
--skip-excluded by walking the seeds' exclude values as if they were
project specs, so a value such as "debian" is visited as a project, the
installed-package check fails with "Unmet dependency on debian: the -run
package of project debian is not installed", and `make
pkg-install-testbuild-deps` breaks.
Implement the flag's documented semantics using
App.is_excluded_from_build(): an excluded seed is neither considered nor
output, an excluded module in the dependency closure contributes no
dependencies, and excluded modules are not output as jw dependencies. This
matches what required-os-pkg already does.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
platform.mk needs JWB_SCRIPT_DIR before it checks if TARGET_TUPLE is empty,
otherwise downstream projects including platform.mk without defs.mk error
out (as does jw-bind-zones-ldap-priv via projects.mk:
/usr/bin/python3.13: can't open file '/jw-pkg.py': [Errno 2] No such file or directory
../jw-pkg/make/platform.mk:172: *** TARGET_TUPLE is empty. Schluss..
The fix is to include common.mk earlier in the stack in platform.mk.
defs.mk then includes it via projects.mk, so remove direct inclusion from
there.
Signed-off-by: Jan Lindemann <jan@janware.com>
This is the first in a series of attempts to clean up jw-pkg's makefile
fragments make/*.mk. It collects redundantly defined workspace and project
variables into one shared common.mk.
Variables needed in both the workspace directory and the project
directories are defined separately in the two contexts: Q in defs.mk and
projects-dir-minimal.mk, INTERACTIVE in topdir.mk and projects-dir.mk,
BROWSER and JWB_SCRIPT_DIR in defs.mk/platform.mk and projects-dir.mk, and
LIST_VCS_FILES and LIST_VCS_FILES_SH wrapping the same scm.sh ls-files call
under different names. PKG_MANAGER is defined in topdir.mk and
projects-dir.mk, and JW_PKG_PY_PROJECTS is a prefix only used to build two
other variables in projects-dir.mk.
This commit adds make/common.mk, included from defs.mk (and hence from
every project directory) and from projects-dir.mk (the workspace), and
collects the shared variables there:
- Q, INTERACTIVE, BROWSER, JWB_SCRIPT_DIR: one ?= definition each
- LIST_VCS_FILES: one definition; LIST_VCS_FILES_SH is dropped and the
list-files target uses the shared name
- PKG_MANAGER moves to projects+project.mk, the fragment shared by the
workspace and the topdir
- projects-dir-minimal.mk keeps its own Q, because the bootstrap recipes
run before common.mk is parsed
- JW_PKG_PY_PROJECTS is inlined into JW_PKG_PY_BUILD and
JW_PKG_PY_REQUIRED_OS_PKG
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
get-auth-info() reads the origin URL of the jw-pkg repository and reports
its username and password. Uri.username and Uri.password are None for URLs
without a credentials part, but _run() stored them in the result anyway, so
--username --only-values printed the literal string "None" for such URLs.
The bootstrap Makefile derives JANWARE_USER from exactly that output on its
second parse, after the jw-pkg directory has been cloned from an anonymous
https origin whose URL has no user part. JANWARE_USER thus became the
string "None", pgit.sh is invoked with --login None and --refspec
None:current-branch:current-branch, and queries the Forgejo API for the
repositories of a non-existent "None" org, so every anonymous bootstrap
fails with "Failed to enumerate repositories".
Make _run() only record values that are not None, so missing credentials
produce no output instead of the string "None".
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add recommended projects into the build closure. Would be nice if they got
built along. Let's see if nightly survives it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add and use JW_PKG_PY_REQUIRED_OS_PKG. The variable and its use effectively
ties more targets to act on the same package set as the make all, influenced
by the contents of $(PKG_RELATIONS_BUILD):
echo-build-deps
echo-install-deps
echo-release-deps
pkg-install-build-deps
pkg-install-release-deps
Signed-off-by: Jan Lindemann <jan@janware.com>
DEP_PROJECTS is computed with pkg-requires --recursive, which follows
[pkg.requires.jw] only. That set does not match what the build actually
builds, and it omits seed projects that no other project of the closure
depends on.
Derive DEP_PROJECTS from projects build --build-order with the same
dependency flavours as make all, the very command the build-order make
target and the workspace mount closure (mount-projects-dir.sh) use.
DEP_PROJECTS is now exactly the build closure, so the file listings, the
text-files cache and the git helpers of a workspace cover the same project
set as make all.
Introduce the PKG_RELATIONS_BUILD variable, defaulting to requires, and
pass it through JW_PKG_PY_BUILD to every build invocation of the workspace,
so the relations of the build closure are a workspace property.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The OS package list of required-os-pkg, like the build closure, is derived
from the [pkg.requires.jw] section only, hard-coded in _run(). Add a
--pkg-relations option to required-os-pkg, mirroring the one of CmdBuild.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The relations the build closure of projects build follows are hard-coded in
read_deps(): [pkg.requires.jw] only. A workspace that also wants to build
the projects its seeds recommend in [pkg.recommends.jw] has no way to say
so.
Add a --pkg-relations option that lists the relations between jw packages
to take into consideration for the build, comma or space separated,
defaulting to requires.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
