This is the first in a series of attempts to clean up jw-pkg's makefile
fragments make/*.mk. It collects redundantly defined workspace and project
variables into one shared common.mk.
Variables needed in both the workspace directory and the project
directories are defined separately in the two contexts: Q in defs.mk and
projects-dir-minimal.mk, INTERACTIVE in topdir.mk and projects-dir.mk,
BROWSER and JWB_SCRIPT_DIR in defs.mk/platform.mk and projects-dir.mk, and
LIST_VCS_FILES and LIST_VCS_FILES_SH wrapping the same scm.sh ls-files call
under different names. PKG_MANAGER is defined in topdir.mk and
projects-dir.mk, and JW_PKG_PY_PROJECTS is a prefix only used to build two
other variables in projects-dir.mk.
This commit adds make/common.mk, included from defs.mk (and hence from
every project directory) and from projects-dir.mk (the workspace), and
collects the shared variables there:
- Q, INTERACTIVE, BROWSER, JWB_SCRIPT_DIR: one ?= definition each
- LIST_VCS_FILES: one definition; LIST_VCS_FILES_SH is dropped and the
list-files target uses the shared name
- PKG_MANAGER moves to projects+project.mk, the fragment shared by the
workspace and the topdir
- projects-dir-minimal.mk keeps its own Q, because the bootstrap recipes
run before common.mk is parsed
- JW_PKG_PY_PROJECTS is inlined into JW_PKG_PY_BUILD and
JW_PKG_PY_REQUIRED_OS_PKG
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
get-auth-info() reads the origin URL of the jw-pkg repository and reports
its username and password. Uri.username and Uri.password are None for URLs
without a credentials part, but _run() stored them in the result anyway, so
--username --only-values printed the literal string "None" for such URLs.
The bootstrap Makefile derives JANWARE_USER from exactly that output on its
second parse, after the jw-pkg directory has been cloned from an anonymous
https origin whose URL has no user part. JANWARE_USER thus became the
string "None", pgit.sh is invoked with --login None and --refspec
None:current-branch:current-branch, and queries the Forgejo API for the
repositories of a non-existent "None" org, so every anonymous bootstrap
fails with "Failed to enumerate repositories".
Make _run() only record values that are not None, so missing credentials
produce no output instead of the string "None".
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Signed-off-by: Jan Lindemann <jan@janware.com>
Add recommended projects into the build closure. Would be nice if they got
built along. Let's see if nightly survives it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add and use JW_PKG_PY_REQUIRED_OS_PKG. The variable and its use effectively
ties more targets to act on the same package set as the make all, influenced
by the contents of $(PKG_RELATIONS_BUILD):
echo-build-deps
echo-install-deps
echo-release-deps
pkg-install-build-deps
pkg-install-release-deps
Signed-off-by: Jan Lindemann <jan@janware.com>
DEP_PROJECTS is computed with pkg-requires --recursive, which follows
[pkg.requires.jw] only. That set does not match what the build actually
builds, and it omits seed projects that no other project of the closure
depends on.
Derive DEP_PROJECTS from projects build --build-order with the same
dependency flavours as make all, the very command the build-order make
target and the workspace mount closure (mount-projects-dir.sh) use.
DEP_PROJECTS is now exactly the build closure, so the file listings, the
text-files cache and the git helpers of a workspace cover the same project
set as make all.
Introduce the PKG_RELATIONS_BUILD variable, defaulting to requires, and
pass it through JW_PKG_PY_BUILD to every build invocation of the workspace,
so the relations of the build closure are a workspace property.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The OS package list of required-os-pkg, like the build closure, is derived
from the [pkg.requires.jw] section only, hard-coded in _run(). Add a
--pkg-relations option to required-os-pkg, mirroring the one of CmdBuild.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The relations the build closure of projects build follows are hard-coded in
read_deps(): [pkg.requires.jw] only. A workspace that also wants to build
the projects its seeds recommend in [pkg.recommends.jw] has no way to say
so.
Add a --pkg-relations option that lists the relations between jw packages
to take into consideration for the build, comma or space separated,
defaulting to requires.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
get_project_refs() deduplicates the walk results by scanning the result
list for each appended element, so the cost grows quadratically with the
number of projects, which is noticeable when walking a dev tree of 320
projects.
Track the already-seen projects in a set, and append an element only when
it is new. The resulting list is unchanged, and the membership test is O(1)
instead of O(n).
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The list of OS packages printed by required-os-pkg is not reproducible: the
packages are collected in a set, and the set's iteration order depends on
the hash seed of the Python process, so two runs of the same invocation can
print the same packages in different orders.
Print the sorted list instead. The order of a package list is meaningless
to the package manager, and the output is now stable across runs and hash
seeds.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The build order printed by projects build is not reproducible:
calculate_order() picks the next module by iterating all_deps, a set, so
the order among modules that are ready at the same time depends on the hash
seed of the Python process, and two runs of the same invocation can print
different orders.
Select the next module from the sorted set instead. The order stays
topologically valid, and is now identical across runs and hash seeds. Sort
the unresolvable-dependency error message the same way, so that it is
stable, too.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
pkg_relations() walks the [pkg.<rel>.jw] edges with its own queue,
duplicating the walk that App.walk_project_deps() extracted from
App.__get_project_refs() in the previous commit. Port it onto the shared
walk: for each flavour, the walk computes the closure of the seed packages
(with the installed-package check off, the ignore set as exclude, and
recursion per the recursive flag), and the function then emits the
constraint strings of the edges of the visited projects.
The relation sets are unchanged. The order of recursive output follows the
walk (postorder) instead of the old breadth-first order; non-recursive
callers, such as the spec generation and the PREREQ_* queries, print
byte-identical output. The dont_expand_version_macros condition now tests
membership in the visited set instead of the old queue, which only matters
when the flag is passed manually.
Verified over all projects of the dev tree: pkg-requires, pkg-recommends,
pkg-conflicts and the skip-excluded, names-only and debian variants are
byte-identical for the non-recursive forms and set-identical for the
recursive ones.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
App.get_project_refs() uses __get_project_refs() to walk the [pkg.*.jw]
sections and pkg_relations() walks the same edges with its own
implementation. In an attempt to unify the walks, this commit tries to make
the first walk implementation so generally useful that it can be used by
the second.
- Make __get_project_refs() public as walk_project_deps(). Name changed
because it sounds less awkward
- Add three parameters needed from the other call site:
- check_installed applies the installed-package check that skips the
dependencies of an installed run package and fails on an unmet
dependency
- exclude names specs that are neither traversed nor appended
- recurse = False stops the walk at the starting spec
- Wrap the per-project dependency reading into App.read_dep_edges(),
because the other call site does the same thing
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
Signed-off-by: Jan Lindemann <jan@janware.com>
get_project_refs() takes a list of sections to look for references in. To
now, that has never been exercised, i.e. only one section had ever been
passed in. With the introduction of pkg.recommends, that could change now,
and it has a subtle bug.
Say, a caller wants to retrieve all requires and recommends of package A
with this graph:
A recommends B, requires nothing
B requires C
then A, B, C should be returned, but the current implementation only
returns A and B. The cause is that every section is recursed into in
isolation, and the union of all results is returned. Since A doesn't
require anything, the requires-branch never reaches B, and C is missed.
The fix is to make __get_project_refs(), the recursing method, take a list
of sections instead of one section, and decide if it recurses deeper based
on all of its entries at every depth.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The commit message of 29abf14a6 clearly states
As of this commit, this is the case by default, i.e. PY_UPDATE_INIT_PY is
set to false in py-ns-dir.mk, maintaining the current behaviour.
The commit's diff then proceeds to set PY_UPDATE_INIT_PY to true. Which is
a clear-cut case of my brain doing something different from what my fingers
do.
Correcting the commit. Some downstream projects have already consumed this
blunder. Not quite sure where that leaves us, but we need to start
somewhere with fixing, so I decide to make the code match the documented
behaviour, and wait for the fallout to fix.
Signed-off-by: Jan Lindemann <jan@janware.com>
The framework instantiates command classes in two places: the top-level
commands in App and the lazily materialized subcommands in
Cmd.add_subcommands(). Both pass the parent parameter positionally.
Change that to passing parent by keyword at both call sites. Every command
class names its first __init__() parameter parent, so the keyword form
binds the same slot for all of them regardless of parameter order, and it
allows a command class to declare parent as keyword-only.
Signed-off-by: Jan Lindemann <jan@janware.com>
The top-level all, test, check, check-pre, and check-post targets invoke
the build tool without --dep-flavours, so it falls back to the auto default
and only resolves dependencies of the flavour "build". The build-order-%
target, by contrast, already passes the run,build,test,release set via a
single BUILD_ORDER_DEP_FLAVOURS variable. Planning and building therefore
rely on different dependency flavours, and none of the targets can be tuned
individually.
Introduce one flavour variable per target group, all of them defaulting to
the run,build,test,release set:
- BUILD_DEP_FLAVOURS for all
- BUILD_DEP_FLAVOURS_LINT for check, check-pre, and check-post
- BUILD_DEP_FLAVOURS_TEST for test
- BUILD_DEP_FLAVOURS_ORDER, renamed from BUILD_ORDER_DEP_FLAVOURS, used by
build-order-%
Split the combined all test check check-pre check-post rule into three
rules so that each group passes its own --dep-flavours to the build tool.
all, test, and the check targets now also resolve run, test, and release
dependencies, as build-order already does, and each target's flavour set
can be overridden independently.
Note that they all default the whole shebang: run,build,test,release. And
this opens up the door to dependency cycles that don't exist with the
subset of projects currently used. It's still better to ask for all and
then cut down as needed instead of starting small.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The spec generator only supports Requires, Conflicts and Provides from
the [pkg.requires.*], [pkg.conflicts.*] and [pkg.provides.*] sections of
make/project.conf. A package that works without but is better with another
package, like jw-amavis-run without jw-clamav-run, must hard-require it, so
the dependency is pulled in on every install and cannot be removed without
dragging the depending package along.
This commit adds Recommends support to the spec generation chain:
- CmdPkgRecommends: new pkg-recommends command that reads the
[pkg.recommends.<flavour>] sections, same base as pkg-requires
- pkg-dist.mk: compute PKG_RECOMMENDS_RUN and pass it to pkg.sh via a new
-E option
- pkg.sh: accept -E and forward it to the mkspec wrapper
- mkspec-wrapper.sh: accept -E and export RECOMMENDS_RUN
- create-mkspec.sh: emit a Recommends: line when RECOMMENDS_<subpkg> is set
- create-mkdebian.sh: emit a Recommends: field in the debian control file
Projects without a [pkg.recommends.*] section generate unchanged specs:
every new line is guarded by a non-empty test. zypper follows Recommends by
default, so existing OpenSUSE installations are unaffected; a recommended
package can be removed without dragging the depending package along.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
A workspace project directory that has a Makefile but no package proofs -
no project.conf, no VERSION - is buildable with a plain make, yet the build
fails on it: calculate_order() walks the requested modules, and the walk in
App.__get_project_refs() raises an unmet dependency, because the
installation state of such a project is unprovable. The walk has to stay
strict: its proofs are package proofs, and loosening App.is_installed()
would weaken the unmet dependency detection for every get_project_refs()
caller.
This commit therefore keeps the tolerance in the build: before calculating
the order, run() partitions the requested modules. A module that is not
installed in any flavour but resolves to a workspace directory with a
Makefile is appended to the build order as a dependency-less leaf and built
with a plain make; modules without any directory at all still raise the
unmet dependency error. A jw dependency declared in a project.conf on such
a project still raises, which is correct: it is a declared dependency on a
package that does not exist.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The "Remove everything non-essential for make clean all" commit (bc883deed)
dropped disabled.mk, judged non-essential from the jw-pkg tree's own
perspective. Legacy projects still include it, though: their Makefiles use
it to replace a directory build with no-op targets that report the target
as disabled. Without the file, make fails in such projects in any workspace
that mounts a current jw-pkg.
Re-add it verbatim. The post-purge re-adds dummy.mk and dummy-topdir.mk
cover the current-style stubs; disabled.mk covers the legacy includes.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a run-success.mk which provides a $(RUN_SUCCESS_CREATE_MARKER)
variable, and include it from run.mk and py-run.mk. If it's not overridden,
it logs the current commit to a marker file if the run succeeds. The marker
file is ignored from Git and cleaned in the context of the distclean
target.
The machinery can be customized by variables:
RUN_SUCCESS_MARKER is the name of the marker file
RUN_SUCCESS_RECORD_CMD is the generator for its content
For example, putting this into local.mk or jw-pkg/make/local.mk will record
the hashes of every repository in the entire workspace:
define RUN_SUCCESS_RECORD_CMD
make git-log-1
make git-log-tree-1
endef
Signed-off-by: Jan Lindemann <jan@janware.com>
The git targets of the workspace Makefile cover push, diff, and status, but
not history: inspecting commits means running git log by hand in a project
directory, and no target shows the history of all projects of a tree at
once.
Add git-log targets at both levels; the target suffix is passed to git log
as-is, so make git-log-5 asks for the last five commits.
- git-log-% in projects-dir.mk runs pgit.sh log over every project of the
tree, one section per project
- git-log-% in rules.mk runs git log -$* in the current project directory
- git-log-tree-% in rules.mk runs git-log-$* in $(PROJECTS_DIR), giving the
whole-tree view from inside a project; it prints a notice if PROJECTS_DIR
is missing
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The source hash is an md5sum over all versioned files in the working tree.
md5sum follows symlinks, so a project with links to files outside the tree
may produce different hashes depending on files outside the repo.
Compute the hash over the VCS file ids instead. Add a -s option to scm.sh
ls-files that lists each versioned file together with its content id. The
listing is sorted by path rather than by id, so an entry keeps its position
when its content changes.
For Git repos, git ls-tree -r HEAD reports the blob of a symlink's target
string and the pinned commit of a submodule, so the resulting hash depends
only on the committed tree, never on the working tree or on the machine.
calculate_hash() and the HASH variable in make/pkg-dist.mk feed the id
listing, minus the release metadata files, directly to md5sum.
calculate_hash() fails loudly when the pipeline fails, with its callers
refusing to proceed on an empty result.
The scheme change invalidates the stored HASH values, so every project
gets one more release the first time it runs against the new code, and the
hashes are stable again afterwards.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
In CVS, the -f option keeps directory entries out of scm.sh ls-file's
listing; in Git, it's a silent no-op. Notably symlinks are not excluded
despite -f.
Implement -f for git with a mode filter on git ls-files -s output: only
100644 and 100755 entries are listed, so the option means the same thing in
both code paths.
For Git, this actually means a behaviour change which needs to be fixed:
The dist archive targets tar-files and cpio-files in make/list-files.mk and
the scm_files() helper in scripts/pkg.sh then may no longer pass it to get
unchanged behaviour.
The targets list-files and list-text-files keep it, so the text-files cache
now matches its documented regular-files-only membership.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Use Version.next() in __version_boundaries, which steps the last existing
part, so '= 1.0' spans '>= 1.0, < 1.1'.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Range expansion needs a bound that steps the last existing part of a
version, add that.
next() increments the last part, whatever it is: next of '1' is '2', of
'1.0' is '1.1', of '1.2.3' is '1.2.4', of '1.2.3-45' is '1.2.3-46'.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
A multi-boundary constraint is rendered as 'foo >= 1.2.3-45 foo < 1.2.4',
which is invalid: the RPM spec template writes the Requires: line from it
verbatim, and RPM entries are comma-separated, so the second clause is
swallowed into one bad entry. Debian's format_depends() normalizes runs of
whitespace to commas, so it tolerates the space join, but the comma is the
only delimiter that is correct for both.
Join the boundary clauses with ', ' instead of a space.
Also:
- Drop the stray trailing comma in the version_boundaries() call.
- Fix the untemplated docstring, which describes the opposite of the code:
untemplated = True (the default) resolves the macros, untemplated = False
keeps them as written.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Full version dependency specs ("= 1.2.3-4" or "= VERSION-REVISION") don't
need expansion, they pin the wanted version with an = fine exactly.
__version_boundary has that the wrong way around, fix that.
Also fix the is_full and version_boundaries() docstrings, which still
describe the old behavior.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__resolved_id() returns the full version if only VERSION was specified, fix
that.
Also: raise Version.Error instead of a bare Exception.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Running make format with the new isort setting rewrites the two wrapped
import blocks:
- The argparse import is split one-per-line, since the logical line exceeds
88 columns
- A trailing comma is added to both imports, which makes yapf keep the
split layout instead of re-joining it
The result is a fixed point that both isort and yapf accept, so make format
and make check agree.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The isort configuration leaves include_trailing_comma at its default, false,
so isort strips the trailing commas from wrapped imports. However, yapf's
split_arguments_when_comma_terminated setting depends on those commas:
without them, yapf re-joins the one-per-line import layout that isort just
produced, and the two formatters disagree on files such as
src/python/jw/pkg/lib/App.py.
Set include_trailing_comma so that isort keeps trailing commas and yapf
honors them, making both tools converge on the same layout.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
conf/templates/gitignore contains duplicates (local.mk) and typos
(-test-out.txt). Fix them and run make gitignore again.
Signed-off-by: Jan Lindemann <jan@janware.com>
TarIo._match() and _filter_tar_file() are pure logic without test coverage.
Add unit tests for the exact path matching, the unfiltered round trip,
filtered extraction with a matched list, and a filter without hits.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
PackageFilterString is pure regex logic without test coverage.
Add unit tests for the url=~ filter, packages without a url, whitespace
around the operator, and the rejection of unsupported filter definitions.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
ProcPipeline and the run() helper are pure logic without test coverage.
Add unit tests for the identity filter, chained filter execution order, the
append() layouts, and the chain handling of run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
parse_spec_str(), parse_specs_str(), order_tags(), and __repr__ are pure
text processing without test coverage, shared by the dpkg and rpm package
manager backends.
Add unit tests for valid and invalid spec strings, multi-line input with
and without a trailing newline, tag ordering with default values, and the
repr layout.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
