Add a run-success.mk which provides a $(RUN_SUCCESS_CREATE_MARKER)
variable, and include it from run.mk and py-run.mk. If it's not overridden,
it logs the current commit to a marker file if the run succeeds. The marker
file is ignored from Git and cleaned in the context of the distclean
target.
The machinery can be customized by variables:
RUN_SUCCESS_MARKER is the name of the marker file
RUN_SUCCESS_RECORD_CMD is the generator for its content
For example, putting this into local.mk or jw-pkg/make/local.mk will record
the hashes of every repository in the entire workspace:
define RUN_SUCCESS_RECORD_CMD
make git-log-1
make git-log-tree-1
endef
Signed-off-by: Jan Lindemann <jan@janware.com>
The git targets of the workspace Makefile cover push, diff, and status, but
not history: inspecting commits means running git log by hand in a project
directory, and no target shows the history of all projects of a tree at
once.
Add git-log targets at both levels; the target suffix is passed to git log
as-is, so make git-log-5 asks for the last five commits.
- git-log-% in projects-dir.mk runs pgit.sh log over every project of the
tree, one section per project
- git-log-% in rules.mk runs git log -$* in the current project directory
- git-log-tree-% in rules.mk runs git-log-$* in $(PROJECTS_DIR), giving the
whole-tree view from inside a project; it prints a notice if PROJECTS_DIR
is missing
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The source hash is an md5sum over all versioned files in the working tree.
md5sum follows symlinks, so a project with links to files outside the tree
may produce different hashes depending on files outside the repo.
Compute the hash over the VCS file ids instead. Add a -s option to scm.sh
ls-files that lists each versioned file together with its content id. The
listing is sorted by path rather than by id, so an entry keeps its position
when its content changes.
For Git repos, git ls-tree -r HEAD reports the blob of a symlink's target
string and the pinned commit of a submodule, so the resulting hash depends
only on the committed tree, never on the working tree or on the machine.
calculate_hash() and the HASH variable in make/pkg-dist.mk feed the id
listing, minus the release metadata files, directly to md5sum.
calculate_hash() fails loudly when the pipeline fails, with its callers
refusing to proceed on an empty result.
The scheme change invalidates the stored HASH values, so every project
gets one more release the first time it runs against the new code, and the
hashes are stable again afterwards.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
In CVS, the -f option keeps directory entries out of scm.sh ls-file's
listing; in Git, it's a silent no-op. Notably symlinks are not excluded
despite -f.
Implement -f for git with a mode filter on git ls-files -s output: only
100644 and 100755 entries are listed, so the option means the same thing in
both code paths.
For Git, this actually means a behaviour change which needs to be fixed:
The dist archive targets tar-files and cpio-files in make/list-files.mk and
the scm_files() helper in scripts/pkg.sh then may no longer pass it to get
unchanged behaviour.
The targets list-files and list-text-files keep it, so the text-files cache
now matches its documented regular-files-only membership.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Use Version.next() in __version_boundaries, which steps the last existing
part, so '= 1.0' spans '>= 1.0, < 1.1'.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Range expansion needs a bound that steps the last existing part of a
version, add that.
next() increments the last part, whatever it is: next of '1' is '2', of
'1.0' is '1.1', of '1.2.3' is '1.2.4', of '1.2.3-45' is '1.2.3-46'.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
A multi-boundary constraint is rendered as 'foo >= 1.2.3-45 foo < 1.2.4',
which is invalid: the RPM spec template writes the Requires: line from it
verbatim, and RPM entries are comma-separated, so the second clause is
swallowed into one bad entry. Debian's format_depends() normalizes runs of
whitespace to commas, so it tolerates the space join, but the comma is the
only delimiter that is correct for both.
Join the boundary clauses with ', ' instead of a space.
Also:
- Drop the stray trailing comma in the version_boundaries() call.
- Fix the untemplated docstring, which describes the opposite of the code:
untemplated = True (the default) resolves the macros, untemplated = False
keeps them as written.
Tests written by AI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Full version dependency specs ("= 1.2.3-4" or "= VERSION-REVISION") don't
need expansion, they pin the wanted version with an = fine exactly.
__version_boundary has that the wrong way around, fix that.
Also fix the is_full and version_boundaries() docstrings, which still
describe the old behavior.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
__resolved_id() returns the full version if only VERSION was specified, fix
that.
Also: raise Version.Error instead of a bare Exception.
Tests written by AI.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Running make format with the new isort setting rewrites the two wrapped
import blocks:
- The argparse import is split one-per-line, since the logical line exceeds
88 columns
- A trailing comma is added to both imports, which makes yapf keep the
split layout instead of re-joining it
The result is a fixed point that both isort and yapf accept, so make format
and make check agree.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The isort configuration leaves include_trailing_comma at its default, false,
so isort strips the trailing commas from wrapped imports. However, yapf's
split_arguments_when_comma_terminated setting depends on those commas:
without them, yapf re-joins the one-per-line import layout that isort just
produced, and the two formatters disagree on files such as
src/python/jw/pkg/lib/App.py.
Set include_trailing_comma so that isort keeps trailing commas and yapf
honors them, making both tools converge on the same layout.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
conf/templates/gitignore contains duplicates (local.mk) and typos
(-test-out.txt). Fix them and run make gitignore again.
Signed-off-by: Jan Lindemann <jan@janware.com>
TarIo._match() and _filter_tar_file() are pure logic without test coverage.
Add unit tests for the exact path matching, the unfiltered round trip,
filtered extraction with a matched list, and a filter without hits.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
PackageFilterString is pure regex logic without test coverage.
Add unit tests for the url=~ filter, packages without a url, whitespace
around the operator, and the rejection of unsupported filter definitions.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
ProcPipeline and the run() helper are pure logic without test coverage.
Add unit tests for the identity filter, chained filter execution order, the
append() layouts, and the chain handling of run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
parse_spec_str(), parse_specs_str(), order_tags(), and __repr__ are pure
text processing without test coverage, shared by the dpkg and rpm package
manager backends.
Add unit tests for valid and invalid spec strings, multi-line input with
and without a trailing newline, tag ordering with default values, and the
repr layout.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
__init_askpass() embeds the password verbatim in a generated bash script,
with a newline included inside the double quotes of the echo -n. The
askpass program therefore always appends a newline to the password, and
any password containing quotes or shell metacharacters either breaks the
script or injects commands into it.
Embed the password as base64 and decode it with printf piped into base64
-d, so the script is safe for any password and prints the password exactly,
byte for byte.
Make __del__() idempotent: it deletes the environment variables and the
script file, so a second call, e.g. an explicit one followed by garbage
collection, raises KeyError and FileNotFoundError.
Add a unit test that executes the generated script and compares its output
with the password byte for byte.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
expand_macros() looks up each %{name} via getattr with '-' turned into '_',
but macro_names advertises two names that don't resolve: 'os-release' has
no corresponding attribute, and 'os-cascade' resolves to the list property
os_cascade, which str.replace() rejects. Both macros are listed in the
platform info help output, yet always fail when used in a format string.
The space-joined cascade string is the cascade property, and 'cascade' is
the macro name the make recipes and scripts already use, together with the
--format default in cmds/platform/CmdInfo.py.
Before the App.distro_* properties moved to Distro, the advertised macro
names were derived from the distro_* attribute names, so the advertised
macro was %{cascade}; the hand-written macro_names replaced it with the
list attribute's name.
- Add the missing os_release attribute as an alias of os_release_str
- Advertise the 'cascade' macro in macro_names instead of 'os-cascade',
leaving os_cascade as the purely programmatic list
- Update the platform info help expected output accordingly
- Add unit tests for the os-release field parsers, the derived properties,
and the expansion of every advertised macro
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
JWB_SCRIPT_DIR bypasses $(JWBDIR) for its scripts candidate, re-assembling
./$(JWBDIR_NAME)/scripts from the bare name. That works, but is redundant
resolution machinery which should be removed per se.
Resolve the scripts directory from $(JWBDIR) instead. Aside from
simplifying, that fixes the resolution when projects-dir.mk is included
from Makefiles other than projects-dir-miminimal.mk, and complements work
to commit f3f0ef10ca which also aims at allowing to run make from a
bind-mounted dependency closure of repositories.
Signed-off-by: Jan Lindemann <jan@janware.com>
Update .gitignore by running "make gitignore. Add a couple of jw-pkg-
specific build-artifacts to GITIGNORE_PATTERNS_END before:
.cache-projects.mk
make/ssh-wrapper.sh
Also cancel the automatic inclusion of file paths containing jw-pkg with
an empty GITIGNORE_PATTERNS_TOP.
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg-create-project.sh contains its own .gitignore template, which is
redundant to the new gitignore mechanism. Replace it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a target gitignore to topdir.mk, which blasts jw-pkg's default idea of
how .gitignore should look like into $(TOPDIR)/.gitignore. It's not a
prerequisite of any other target, because it's not meant to run
automatically. The idea is to keep some central maintenance of what should
be in there from jw-pkg's perspective, make it convenvient to update, but
leave the final decision to the commiter of any project using the
machinery.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add variables to shorten typical invocations of "jw-pkg project update
--format tmpl", and use them in py-topdir.mk
JW_PKG_TMPL_SEARCH_PATH
JW_PKG_TMPL_UPDATE
Signed-off-by: Jan Lindemann <jan@janware.com>
create-pkg-config's output is not covered by any test, even though the
template and the Requires cleanup produce exact text that pkg-config
consumes, so regressions in the generated file went unnoticed for a long
time.
Add an integration test that runs create-pkg-config with a full option set,
with space- and comma-mixed Requires input, and with a
Requires.private-only input, and compares the output against a reference
file.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
__cleanup_requires() replaces every run of whitespace with ", " before
re-pairing the version constraints, so input that is already
comma-separated, e.g. "jw-core >= 1.0, jw-base", comes out with a double
comma, "jw-core >= 1.0,, jw-base". And the Requires line is appended
without a trailing newline, so a following Requires.private line runs
straight into it.
Split the input on commas and whitespace, treating the version constraint
operators as delimiters that are re-paired with the preceding name and the
following version, so that comma- and space-separated input alike comes out
as a clean ", "-joined list. Add the missing newline after the Requires
line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdCreateFile renders a file from project metadata, yet nothing runs it end
to end. The "tmpl" format substitutes --field values into template markers,
and the "pyright" format computes extra paths from the jw run dependencies
of a module. Both formats are untested, and so is the handling of missing
templates, unknown --field keys, and malformed command-line arguments.
Add an integration test under test/integration/jw-pkg/projects that drives
the command through the real CLI and diffs the rendered output against
a reference. It covers single- and multi-value field substitution, the
default and custom field separators including the %n newline escape, value
quoting, and --field-keys filling in keys the caller omitted. The pyright
case uses a relative topdir so the computed project paths stay relocatable.
The error cases check that the command rejects a missing template name, an
unknown template file, a --field key outside --field-keys, an unsupported
pyright separator, and invalid --format and --field arguments with the
expected message and exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
List values in a rendered template are joined with a fixed separator of
",\n" (comma plus newline), and there is no way to change that. Templates
sometimes want a different separator, e.g. a plain newline or a single-line
comma-separated list.
Add a --field-separator option that controls how list values are joined
in the rendered output. %n expands to a newline, and the default remains
",\n" for both formats. render_tmpl() applies the separator to the rendered
template, while render_pyright() rejects any other value: the built-in
pyrightconfig.json template is a fixed JSON document, and no alternative
separator is supported for it.
Update the help output test expectation accordingly.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The --field option accepts arbitrary KEY=VALUE pairs, inserting them into
the rendered template output, but there is no way to declare which keys
a template actually supports. Passing a key the template does not use goes
unnoticed, and a template cannot offer optional fields that drop out of
the output when not supplied.
Add a --field-keys option, a comma-separated list of the keys a template
accepts. Passing a key via --field that is not in that list is an error,
and keys from the list that are not passed are added with an empty value,
so the respective field renders as empty and effectively drops out of the
output.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The possible values of --format are defined by an Enum which is never
really used as such. Derive them from class introspection instead, i.e.
offer all formats that have a corresponding render_<format>() name.
Also, turn the option into real argparse-backed choices, and make the
argument mandatory, because that reflects the reality of the implementation
- there is no default.
Signed-off-by: Jan Lindemann <jan@janware.com>
create-file and create-pkg-config both print the rendered template with a
newline appended over the original template, owed to using print() instead
of sys.stdout.write(). Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
format_list_dict() substitutes {key} markers with str.replace(), which also
matches the marker inside ${key} and {{key}}, mangling text meant to
survive verbatim, e.g. the ${prefix} variable reference of a pkg-config
file. The built-in pkg-config template works around that with doubled
braces, which produces {/usr}-style output instead.
Replace the markers with a regular expression that skips markers preceded
by '$' or '{', and let the pkg-config template use the ${prefix} and
${exec_prefix} references pkg-config expects.
Extend the unit tests with marker escaping, value escaping, quoting,
indentation, and the rendering of the built-in templates.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
is_list_dict() accepts a string value as a list-dict value, and
render_values_to_list_dict() passes such values through unchanged.
merge_values() then merges them with list extension, and [] += 'xyz'
appends the individual characters, so every string value ends up as a list
of its characters. Normalize string values to single-element lists in
render_values_to_list_dict(), so that merge_values() appends whole values.
jw-pkg projects create-pkg-config is hit by this, because all of its values
are strings: the generated file contains one line per character. The code
path was just never exercised lately.
Add unit tests for the value layout guards, the normalization, and the
merging of the three supported layouts.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
--skip-excluded removed excluded modules from deps with
deps.remove() while iterating over deps. Removing an element
shifts the remaining elements to the left, so the loop skipped
the entry following each removed one, and consecutively
excluded modules survived the filter and leaked into the
output.
Build the filtered list with a comprehension instead.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
PY_CHECK_ROOTS defaults to the src and tools directories, so the ruff,
mypy, pyright, and yapf checks and the bad-pattern scan silently skip test
trees. Python tests below test/ therefore escape linting, type checking,
and formatting enforcement, even though py-check.mk itself defaults to the
whole directory when no roots are given at all.
Add $(TOPDIR)/test to the default PY_CHECK_ROOTS after the build is
through, so that the test tree receives the same checks and formatting as
src and tools. The previous commit prepared this repo's unit tests for the
strict type check and the formatter, so the extended roots pass cleanly
here.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
create-conftest-py generates the conftest.py that pytest test directories
pick up at build time, and make clean removes it. The generated file lacks
type annotations, though: pytest_configure() and the nested patched() take
untyped parameters, and the replacement of .pytest_sessionstart() on the
class is rejected by strict mypy.
Emit a type-annotated version instead: add a future annotations import,
type both functions, move the types that are only used in annotations
behind a TYPE_CHECKING guard, and silence the method replacement with a
targeted type ignore.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The unit test files below test/unit/python/jw/pkg/lib/ define Cmd and App
subclasses with unannotated methods: __init__(), add_arguments(), _run()
and close() lack signatures, the made class attribute lacks a type, and the
_cleanup() and exit_context() helpers are untyped, so a strict mypy run
over the test tree fails on them. Likewise, the ExecApp and version tests
carry list formatting that yapf rejects.
Annotate the test classes following the library conventions: parent is App
| Cmd, the parser is ArgumentParser, and args is Namespace, and mark every
overridden method with @override. Guard the imports that are only used for
annotations behind TYPE_CHECKING, and add the future annotations import so
they stay out of the runtime path. Then reformat the test tree with yapf.
That folds the opts list of the ExecApp test and re-indents the rejected
operator list of the version test.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
