The -m option of posix copy is optional, and its argparse default is None,
but _run() converts it unconditionally and crashes:
mode = int(args.mode, 0)
so every copy without -m died with
TypeError: int() can't convert non-string with explicit base
before the copy even started. Convert the mode only when it is given, and
pass None otherwise, which copy() already treats as "no mode change".
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
_raise_stat_error() maps the message of a failed remote stat to an
exception. A stat below a file answers "not a directory", which falls
through to the generic EIO: the path is simply absent, and callers see a
re-raised error instead of the FileNotFoundError the "no such file" branch
already gives. Map it to FileNotFoundError, so existence checks see
absence.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg secrets list-secrets spams the console without --interactive=false
because it thinks it should query packages and files in interactive mode.
The underlying bug is that run_cmd() derives the interactive flag of a
freshly created Local context from cmd_input, but it never forwards
cmd_input to ExecContext.run(). Since run() defaults to
InputMode.OptInteractive, an explicit cmd_input is silently ignored: a byte
string is not fed to the command's stdin, InputMode.Auto is downgraded to
non-interactive, and with a caller-provided ExecContext the requested mode
is lost in all cases.
Forward cmd_input in kwargs so the requested input mode reaches run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.99.2
copy() copies a single file: it fetches the source root as one content and
puts it at the destination, so addressing a directory source fails, and
there is no way to mirror a tree between file contexts.
Add a recursive mode:
- a directory source is mirrored below the destination root, into it if the
root is an existing directory, as it if it does not exist yet
- entries are enumerated with readdir(), destination directories are
created level by level, and every leaf is put with the requested owner,
group, and mode
- a file source is unaffected, and the destination parent must exist, as
before
posix copy exposes the mode as -r/--recursive; the help test reference
picks up the new option.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
mkdir() passes its path to _mkdir() without chrooting it, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Signed-off-by: Jan Lindemann <jan@janware.com>
Target git-import adds everything it finds in the current directory. If
it's already a Git project, that fails over git commit -m "First commit".
Since it's a prerequisite of target git-init-remote, it breaks that target,
too for non-empty git repos, which isn't helpful when trying to run make
git-init-remote to publish an an already existing project.
So only do something meaningful in git-import if the project it's run in
doesn't already contain files.
Signed-off-by: Jan Lindemann <jan@janware.com>
projects_dir is searched by looking for a directory named .git up the
direcctory lineage. That does't work for worktrees, which are regular files
pointing back to worktree git dir within the the real git directory.
Broaden the check from "test -d .git" to "test -e .git".
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commit renamed the dist/pckg directory to dist/pkg but kept
the pckg spelling in the variables that reference it.
Rename the variables to PKG:
- DIST_PCKG_DIR to DIST_PKG_DIR (make/pkg-dist.mk, pkg.sh)
- PCKG_TAR to PKG_TAR, PCKG_ROOT to PKG_ROOT and DIST_PCKG_RPM to
DIST_PKG_RPM (make/pkg-dist.mk)
None of the other 320 janware projects use these variables.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The directory holding the built packages uses the pckg abbreviation while
every other package-related name in the build system uses pkg (PKG_PROJECT,
PKG_FORMAT, PKG_ARCH).
Rename dist/pckg to dist/pkg, in the DIST_PCKG_DIR variable and in the
recipes that hardcode the path.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
pkg.sh hardcodes $HOME/rpmbuild as the rpmbuild topdir while make computes
the same location separately as PKG_DIR_BUILD, and a sandboxed shell cannot
build at all: rpmbuild writes its scratch trees and results under the
topdir, outside the project.
- Default PKG_DIR_BUILD to $(TOPDIR)/dist/build for non-root builds and
pass it to pkg.sh as the new -T option (root keeps /usr/src/packages;
local.mk can still override)
- Build with --define _topdir pointing at that directory, resolved to an
absolute path, so the scratch tree, RPMS and SRPMS stay in the project
tree where make clean drops them
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Remove INSTALL_PCKG_DEPS_DIR and friends from jw-pkg. It was introduced
with 2a4a95 in 2009 and has not seen much use since. By now it has been
made obsolete by declaring dependencies in project.conf. One source of
truth is enough.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add an integration test for pkg-requires --skip-excluded with a fixture
project tree: excl-never lists the platform name "centos" in [build]
exclude and is not excluded from the build on the test platforms, excl-all
is excluded from the build everywhere, and dep-target is a plain dependency
target.
The test verifies that a platform name in [build] exclude is not treated as
a project, that an excluded jw dependency is not reported with
--skip-excluded, that an excluded seed is neither considered nor output,
and that without the flag the excluded projects are reported with their
dependencies.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
The [build] exclude list names platforms (OS cascade entries) on which a
project's build is excluded, not projects. pkg_relations() implements
--skip-excluded by walking the seeds' exclude values as if they were
project specs, so a value such as "debian" is visited as a project, the
installed-package check fails with "Unmet dependency on debian: the -run
package of project debian is not installed", and `make
pkg-install-testbuild-deps` breaks.
Implement the flag's documented semantics using
App.is_excluded_from_build(): an excluded seed is neither considered nor
output, an excluded module in the dependency closure contributes no
dependencies, and excluded modules are not output as jw dependencies. This
matches what required-os-pkg already does.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
platform.mk needs JWB_SCRIPT_DIR before it checks if TARGET_TUPLE is empty,
otherwise downstream projects including platform.mk without defs.mk error
out (as does jw-bind-zones-ldap-priv via projects.mk:
/usr/bin/python3.13: can't open file '/jw-pkg.py': [Errno 2] No such file or directory
../jw-pkg/make/platform.mk:172: *** TARGET_TUPLE is empty. Schluss..
The fix is to include common.mk earlier in the stack in platform.mk.
defs.mk then includes it via projects.mk, so remove direct inclusion from
there.
Signed-off-by: Jan Lindemann <jan@janware.com>
This is the first in a series of attempts to clean up jw-pkg's makefile
fragments make/*.mk. It collects redundantly defined workspace and project
variables into one shared common.mk.
Variables needed in both the workspace directory and the project
directories are defined separately in the two contexts: Q in defs.mk and
projects-dir-minimal.mk, INTERACTIVE in topdir.mk and projects-dir.mk,
BROWSER and JWB_SCRIPT_DIR in defs.mk/platform.mk and projects-dir.mk, and
LIST_VCS_FILES and LIST_VCS_FILES_SH wrapping the same scm.sh ls-files call
under different names. PKG_MANAGER is defined in topdir.mk and
projects-dir.mk, and JW_PKG_PY_PROJECTS is a prefix only used to build two
other variables in projects-dir.mk.
This commit adds make/common.mk, included from defs.mk (and hence from
every project directory) and from projects-dir.mk (the workspace), and
collects the shared variables there:
- Q, INTERACTIVE, BROWSER, JWB_SCRIPT_DIR: one ?= definition each
- LIST_VCS_FILES: one definition; LIST_VCS_FILES_SH is dropped and the
list-files target uses the shared name
- PKG_MANAGER moves to projects+project.mk, the fragment shared by the
workspace and the topdir
- projects-dir-minimal.mk keeps its own Q, because the bootstrap recipes
run before common.mk is parsed
- JW_PKG_PY_PROJECTS is inlined into JW_PKG_PY_BUILD and
JW_PKG_PY_REQUIRED_OS_PKG
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
get-auth-info() reads the origin URL of the jw-pkg repository and reports
its username and password. Uri.username and Uri.password are None for URLs
without a credentials part, but _run() stored them in the result anyway, so
--username --only-values printed the literal string "None" for such URLs.
The bootstrap Makefile derives JANWARE_USER from exactly that output on its
second parse, after the jw-pkg directory has been cloned from an anonymous
https origin whose URL has no user part. JANWARE_USER thus became the
string "None", pgit.sh is invoked with --login None and --refspec
None:current-branch:current-branch, and queries the Forgejo API for the
repositories of a non-existent "None" org, so every anonymous bootstrap
fails with "Failed to enumerate repositories".
Make _run() only record values that are not None, so missing credentials
produce no output instead of the string "None".
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add recommended projects into the build closure. Would be nice if they got
built along. Let's see if nightly survives it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add and use JW_PKG_PY_REQUIRED_OS_PKG. The variable and its use effectively
ties more targets to act on the same package set as the make all, influenced
by the contents of $(PKG_RELATIONS_BUILD):
echo-build-deps
echo-install-deps
echo-release-deps
pkg-install-build-deps
pkg-install-release-deps
Signed-off-by: Jan Lindemann <jan@janware.com>
DEP_PROJECTS is computed with pkg-requires --recursive, which follows
[pkg.requires.jw] only. That set does not match what the build actually
builds, and it omits seed projects that no other project of the closure
depends on.
Derive DEP_PROJECTS from projects build --build-order with the same
dependency flavours as make all, the very command the build-order make
target and the workspace mount closure (mount-projects-dir.sh) use.
DEP_PROJECTS is now exactly the build closure, so the file listings, the
text-files cache and the git helpers of a workspace cover the same project
set as make all.
Introduce the PKG_RELATIONS_BUILD variable, defaulting to requires, and
pass it through JW_PKG_PY_BUILD to every build invocation of the workspace,
so the relations of the build closure are a workspace property.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The OS package list of required-os-pkg, like the build closure, is derived
from the [pkg.requires.jw] section only, hard-coded in _run(). Add a
--pkg-relations option to required-os-pkg, mirroring the one of CmdBuild.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The relations the build closure of projects build follows are hard-coded in
read_deps(): [pkg.requires.jw] only. A workspace that also wants to build
the projects its seeds recommend in [pkg.recommends.jw] has no way to say
so.
Add a --pkg-relations option that lists the relations between jw packages
to take into consideration for the build, comma or space separated,
defaulting to requires.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
get_project_refs() deduplicates the walk results by scanning the result
list for each appended element, so the cost grows quadratically with the
number of projects, which is noticeable when walking a dev tree of 320
projects.
Track the already-seen projects in a set, and append an element only when
it is new. The resulting list is unchanged, and the membership test is O(1)
instead of O(n).
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The list of OS packages printed by required-os-pkg is not reproducible: the
packages are collected in a set, and the set's iteration order depends on
the hash seed of the Python process, so two runs of the same invocation can
print the same packages in different orders.
Print the sorted list instead. The order of a package list is meaningless
to the package manager, and the output is now stable across runs and hash
seeds.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The build order printed by projects build is not reproducible:
calculate_order() picks the next module by iterating all_deps, a set, so
the order among modules that are ready at the same time depends on the hash
seed of the Python process, and two runs of the same invocation can print
different orders.
Select the next module from the sorted set instead. The order stays
topologically valid, and is now identical across runs and hash seeds. Sort
the unresolvable-dependency error message the same way, so that it is
stable, too.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
pkg_relations() walks the [pkg.<rel>.jw] edges with its own queue,
duplicating the walk that App.walk_project_deps() extracted from
App.__get_project_refs() in the previous commit. Port it onto the shared
walk: for each flavour, the walk computes the closure of the seed packages
(with the installed-package check off, the ignore set as exclude, and
recursion per the recursive flag), and the function then emits the
constraint strings of the edges of the visited projects.
The relation sets are unchanged. The order of recursive output follows the
walk (postorder) instead of the old breadth-first order; non-recursive
callers, such as the spec generation and the PREREQ_* queries, print
byte-identical output. The dont_expand_version_macros condition now tests
membership in the visited set instead of the old queue, which only matters
when the flag is passed manually.
Verified over all projects of the dev tree: pkg-requires, pkg-recommends,
pkg-conflicts and the skip-excluded, names-only and debian variants are
byte-identical for the non-recursive forms and set-identical for the
recursive ones.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
App.get_project_refs() uses __get_project_refs() to walk the [pkg.*.jw]
sections and pkg_relations() walks the same edges with its own
implementation. In an attempt to unify the walks, this commit tries to make
the first walk implementation so generally useful that it can be used by
the second.
- Make __get_project_refs() public as walk_project_deps(). Name changed
because it sounds less awkward
- Add three parameters needed from the other call site:
- check_installed applies the installed-package check that skips the
dependencies of an installed run package and fails on an unmet
dependency
- exclude names specs that are neither traversed nor appended
- recurse = False stops the walk at the starting spec
- Wrap the per-project dependency reading into App.read_dep_edges(),
because the other call site does the same thing
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
get_project_refs() takes a list of sections to look for references in. To
now, that has never been exercised, i.e. only one section had ever been
passed in. With the introduction of pkg.recommends, that could change now,
and it has a subtle bug.
Say, a caller wants to retrieve all requires and recommends of package A
with this graph:
A recommends B, requires nothing
B requires C
then A, B, C should be returned, but the current implementation only
returns A and B. The cause is that every section is recursed into in
isolation, and the union of all results is returned. Since A doesn't
require anything, the requires-branch never reaches B, and C is missed.
The fix is to make __get_project_refs(), the recursing method, take a list
of sections instead of one section, and decide if it recurses deeper based
on all of its entries at every depth.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The commit message of 29abf14a6 clearly states
As of this commit, this is the case by default, i.e. PY_UPDATE_INIT_PY is
set to false in py-ns-dir.mk, maintaining the current behaviour.
The commit's diff then proceeds to set PY_UPDATE_INIT_PY to true. Which is
a clear-cut case of my brain doing something different from what my fingers
do.
Correcting the commit. Some downstream projects have already consumed this
blunder. Not quite sure where that leaves us, but we need to start
somewhere with fixing, so I decide to make the code match the documented
behaviour, and wait for the fallout to fix.
Signed-off-by: Jan Lindemann <jan@janware.com>
The framework instantiates command classes in two places: the top-level
commands in App and the lazily materialized subcommands in
Cmd.add_subcommands(). Both pass the parent parameter positionally.
Change that to passing parent by keyword at both call sites. Every command
class names its first __init__() parameter parent, so the keyword form
binds the same slot for all of them regardless of parameter order, and it
allows a command class to declare parent as keyword-only.
Signed-off-by: Jan Lindemann <jan@janware.com>
The top-level all, test, check, check-pre, and check-post targets invoke
the build tool without --dep-flavours, so it falls back to the auto default
and only resolves dependencies of the flavour "build". The build-order-%
target, by contrast, already passes the run,build,test,release set via a
single BUILD_ORDER_DEP_FLAVOURS variable. Planning and building therefore
rely on different dependency flavours, and none of the targets can be tuned
individually.
Introduce one flavour variable per target group, all of them defaulting to
the run,build,test,release set:
- BUILD_DEP_FLAVOURS for all
- BUILD_DEP_FLAVOURS_LINT for check, check-pre, and check-post
- BUILD_DEP_FLAVOURS_TEST for test
- BUILD_DEP_FLAVOURS_ORDER, renamed from BUILD_ORDER_DEP_FLAVOURS, used by
build-order-%
Split the combined all test check check-pre check-post rule into three
rules so that each group passes its own --dep-flavours to the build tool.
all, test, and the check targets now also resolve run, test, and release
dependencies, as build-order already does, and each target's flavour set
can be overridden independently.
Note that they all default the whole shebang: run,build,test,release. And
this opens up the door to dependency cycles that don't exist with the
subset of projects currently used. It's still better to ask for all and
then cut down as needed instead of starting small.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The spec generator only supports Requires, Conflicts and Provides from
the [pkg.requires.*], [pkg.conflicts.*] and [pkg.provides.*] sections of
make/project.conf. A package that works without but is better with another
package, like jw-amavis-run without jw-clamav-run, must hard-require it, so
the dependency is pulled in on every install and cannot be removed without
dragging the depending package along.
This commit adds Recommends support to the spec generation chain:
- CmdPkgRecommends: new pkg-recommends command that reads the
[pkg.recommends.<flavour>] sections, same base as pkg-requires
- pkg-dist.mk: compute PKG_RECOMMENDS_RUN and pass it to pkg.sh via a new
-E option
- pkg.sh: accept -E and forward it to the mkspec wrapper
- mkspec-wrapper.sh: accept -E and export RECOMMENDS_RUN
- create-mkspec.sh: emit a Recommends: line when RECOMMENDS_<subpkg> is set
- create-mkdebian.sh: emit a Recommends: field in the debian control file
Projects without a [pkg.recommends.*] section generate unchanged specs:
every new line is guarded by a non-empty test. zypper follows Recommends by
default, so existing OpenSUSE installations are unaffected; a recommended
package can be removed without dragging the depending package along.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
