__init_askpass() embeds the password verbatim in a generated bash script,
with a newline included inside the double quotes of the echo -n. The
askpass program therefore always appends a newline to the password, and
any password containing quotes or shell metacharacters either breaks the
script or injects commands into it.
Embed the password as base64 and decode it with printf piped into base64
-d, so the script is safe for any password and prints the password exactly,
byte for byte.
Make __del__() idempotent: it deletes the environment variables and the
script file, so a second call, e.g. an explicit one followed by garbage
collection, raises KeyError and FileNotFoundError.
Add a unit test that executes the generated script and compares its output
with the password byte for byte.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
expand_macros() looks up each %{name} via getattr with '-' turned into '_',
but macro_names advertises two names that don't resolve: 'os-release' has
no corresponding attribute, and 'os-cascade' resolves to the list property
os_cascade, which str.replace() rejects. Both macros are listed in the
platform info help output, yet always fail when used in a format string.
The space-joined cascade string is the cascade property, and 'cascade' is
the macro name the make recipes and scripts already use, together with the
--format default in cmds/platform/CmdInfo.py.
Before the App.distro_* properties moved to Distro, the advertised macro
names were derived from the distro_* attribute names, so the advertised
macro was %{cascade}; the hand-written macro_names replaced it with the
list attribute's name.
- Add the missing os_release attribute as an alias of os_release_str
- Advertise the 'cascade' macro in macro_names instead of 'os-cascade',
leaving os_cascade as the purely programmatic list
- Update the platform info help expected output accordingly
- Add unit tests for the os-release field parsers, the derived properties,
and the expansion of every advertised macro
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
create-pkg-config's output is not covered by any test, even though the
template and the Requires cleanup produce exact text that pkg-config
consumes, so regressions in the generated file went unnoticed for a long
time.
Add an integration test that runs create-pkg-config with a full option set,
with space- and comma-mixed Requires input, and with a
Requires.private-only input, and compares the output against a reference
file.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
__cleanup_requires() replaces every run of whitespace with ", " before
re-pairing the version constraints, so input that is already
comma-separated, e.g. "jw-core >= 1.0, jw-base", comes out with a double
comma, "jw-core >= 1.0,, jw-base". And the Requires line is appended
without a trailing newline, so a following Requires.private line runs
straight into it.
Split the input on commas and whitespace, treating the version constraint
operators as delimiters that are re-paired with the preceding name and the
following version, so that comma- and space-separated input alike comes out
as a clean ", "-joined list. Add the missing newline after the Requires
line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdCreateFile renders a file from project metadata, yet nothing runs it end
to end. The "tmpl" format substitutes --field values into template markers,
and the "pyright" format computes extra paths from the jw run dependencies
of a module. Both formats are untested, and so is the handling of missing
templates, unknown --field keys, and malformed command-line arguments.
Add an integration test under test/integration/jw-pkg/projects that drives
the command through the real CLI and diffs the rendered output against
a reference. It covers single- and multi-value field substitution, the
default and custom field separators including the %n newline escape, value
quoting, and --field-keys filling in keys the caller omitted. The pyright
case uses a relative topdir so the computed project paths stay relocatable.
The error cases check that the command rejects a missing template name, an
unknown template file, a --field key outside --field-keys, an unsupported
pyright separator, and invalid --format and --field arguments with the
expected message and exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
List values in a rendered template are joined with a fixed separator of
",\n" (comma plus newline), and there is no way to change that. Templates
sometimes want a different separator, e.g. a plain newline or a single-line
comma-separated list.
Add a --field-separator option that controls how list values are joined
in the rendered output. %n expands to a newline, and the default remains
",\n" for both formats. render_tmpl() applies the separator to the rendered
template, while render_pyright() rejects any other value: the built-in
pyrightconfig.json template is a fixed JSON document, and no alternative
separator is supported for it.
Update the help output test expectation accordingly.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The --field option accepts arbitrary KEY=VALUE pairs, inserting them into
the rendered template output, but there is no way to declare which keys
a template actually supports. Passing a key the template does not use goes
unnoticed, and a template cannot offer optional fields that drop out of
the output when not supplied.
Add a --field-keys option, a comma-separated list of the keys a template
accepts. Passing a key via --field that is not in that list is an error,
and keys from the list that are not passed are added with an empty value,
so the respective field renders as empty and effectively drops out of the
output.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The possible values of --format are defined by an Enum which is never
really used as such. Derive them from class introspection instead, i.e.
offer all formats that have a corresponding render_<format>() name.
Also, turn the option into real argparse-backed choices, and make the
argument mandatory, because that reflects the reality of the implementation
- there is no default.
Signed-off-by: Jan Lindemann <jan@janware.com>
format_list_dict() substitutes {key} markers with str.replace(), which also
matches the marker inside ${key} and {{key}}, mangling text meant to
survive verbatim, e.g. the ${prefix} variable reference of a pkg-config
file. The built-in pkg-config template works around that with doubled
braces, which produces {/usr}-style output instead.
Replace the markers with a regular expression that skips markers preceded
by '$' or '{', and let the pkg-config template use the ${prefix} and
${exec_prefix} references pkg-config expects.
Extend the unit tests with marker escaping, value escaping, quoting,
indentation, and the rendering of the built-in templates.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
is_list_dict() accepts a string value as a list-dict value, and
render_values_to_list_dict() passes such values through unchanged.
merge_values() then merges them with list extension, and [] += 'xyz'
appends the individual characters, so every string value ends up as a list
of its characters. Normalize string values to single-element lists in
render_values_to_list_dict(), so that merge_values() appends whole values.
jw-pkg projects create-pkg-config is hit by this, because all of its values
are strings: the generated file contains one line per character. The code
path was just never exercised lately.
Add unit tests for the value layout guards, the normalization, and the
merging of the three supported layouts.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
The unit test files below test/unit/python/jw/pkg/lib/ define Cmd and App
subclasses with unannotated methods: __init__(), add_arguments(), _run()
and close() lack signatures, the made class attribute lacks a type, and the
_cleanup() and exit_context() helpers are untyped, so a strict mypy run
over the test tree fails on them. Likewise, the ExecApp and version tests
carry list formatting that yapf rejects.
Annotate the test classes following the library conventions: parent is App
| Cmd, the parser is ArgumentParser, and args is Namespace, and mark every
overridden method with @override. Guard the imports that are only used for
annotations behind TYPE_CHECKING, and add the future annotations import so
they stay out of the runtime path. Then reformat the test tree with yapf.
That folds the opts list of the ExecApp test and re-indents the rejected
operator list of the version test.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The spec split pattern ([=><]+) tokenizes only the =, > and < characters,
so a PEP 440 operator like ~= or != leaves its ~ or ! glued to the package
name: Dependency('pkg~=1.0') parses to base name 'pkg~' with operator '='.
App.__get_project_refs() carries the same pattern inline, so the same specs
corrupt the module name used for the -devel subpackage check. The split
also tolerates operator strings the language does not support, most visibly
==, which parses and renders but raises NotImplementedError only when
expansion is requested.
The spec language supports exactly =, <, <=, > and >=, and boundary
expansion implements all of them. Extend the split pattern with ~ and ! so
that foreign operators tokenize as operator strings, and reject every
operator outside the supported set in Dependency.__parsed_spec() with
Dependency.Error, naming the supported operators. Route
App.__get_project_refs() through Dependency for the name and module parts
instead of the second inline split, so the validation lives in one place.
The catch all in Dependency.__version_boundaries() stays as a backstop
against drift between the allow list and the expansion cases.
The tests drop == from the accepted operator loop, drop the now-unreachable
not-expandable case, and assert that ~=, !=, ~, ==, ===, << and >> are
rejected at parse time with a message naming the operator.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
pkg_relations() parses dependency specifications such as 'foo-devel >=
1.2.3-4' with inline regular expressions and mixes the parsing with version
macro expansion and rendering in three different syntaxes.
This commit adds the lib.version module that owns the parsing of a single
specification: the base and full name and the version boundary with its
operator. Rendering happens in constraint_str() in SEM_VER, DEBIAN or
NAMES_ONLY syntax, with untemplated, include_revision, as_range,
no_subpackages and quote options. The VERSION, VERSION-REVISION and
REVISION macros are resolved through a version lookup callback,
lib.version.Version.parse_deps_spec() splits comma-separated specification
strings into Version objects, and Version breaks a version down into its
major, minor, micro and revision parts.
Unit tests cover parsing, macro resolution, rendering in all syntaxes and
the range expansion.
The module is designed to be integrated into the status quo and
intentionally does not address a couple of further TODOs. Notably
version.Syntax.SEM_VER is intended to replace
pkg_relations.VersionSyntax.SemVer, but neither are really semantic
versioning according to spec. They are close but more targeted toward RPM
and Debian package versioning. And the naming and semantics differ from
SemVer.
Naming of the four components (major and minor version) is identical,
semantic meaning aside, there's no disagreement between real SemVer,
Debian, RPM and jw-pkg. The third and fourth component deviate:
SemVer Debian RPM jw-pkg lib.version
3 PATCH RELEASE MICRO
4 PRERELEASE Revision Release REVISION REVISION
TODOs: RELEASE over the rest of jw-pkg needs to be adjusted by a later
commit. Syntax.SEM_VER should also be renamed, to Syntax.JW_PKG maybe,
because, as said, it's not SemVer and will probably never be - SemVer
doesn't provide the compatibility guarantees that the jw-pkg versioning
scheme offers. To be documented.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assembling a scheme-bearing form always writes '://' after the scheme, even
if the input has no authority part. For a schemeless relative path such as
'../../local/path', .full then becomes 'file://../../local/path', which
re-parses with '..' as the host and '/../local/path' as the path, both
entirely broken.
Fix __assemble() to write a bare ':' instead of '://' when the input has no
host and a relative path, so the assembled form re-parses back to the same
relative path: full of '../../local/path' is now 'file:../../local/path'.
Absolute paths and forms with an authority are unchanged.
The assembled 'file:../../local/path' form is a valid URI under the RFC
3986 generic grammar (scheme with a rootless path), while RFC 8089's file
URI ABNF admits only an empty or an absolute path. Uri thus trades RFC 8089
conformance for RFC 3986 compatibility: a relative path survives the
full-and-reparse cycle unchanged.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecApp is a ready-made base class for applications that operate
through an ExecContext: it adds the --interactive, --verbose and
--target options, exposes interactive, verbose and exec_context
properties, and closes the exec context when the async context
manager exits.
The code for that has lived in jw.pkg.App code before, which now
inherits from ExecApp.
A fix along the way: __aexit__() closes the exec context and then chains
to super().__aexit__(), so App.close() runs when the async context
manager exits. Before the change, exiting the async context left the app
unclosed; close() ran only on the run() path. Add a unit test that
builds an ExecApp with a root command and asserts that close() runs on
context exit and that the exec options are registered.
The exec options are now registered before App's own options,
which moves them up in the rendered --help output. Update the
golden file of the help integration test to match.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
The application's top-level behavior is defined by overriding
App._add_arguments() and App._run(). The lightweight run-and-options
unit, Cmd, can already be mounted at any node of the command tree, but
the root is reserved for the application itself. An application that
wants to host a plain command at the top level therefore has to
subclass App and carry its full lifecycle implementation.
Add a root parameter to App.__init__(). When it is given a command
class, App instantiates it and uses it as the top level: the command's
options are registered on the top-level parser, it becomes the parent
of the top-level subcommands, and App._run() delegates the run to it.
The command's children are wired as the top-level subcommands, so the
same Cmd can now occupy the root node. When root is not given, the
previous auto-discovery behavior is preserved unchanged.
Keep the top-level subcommand heading as plain "Available subcommands"
whether it is hosted by the application or by a root command, while
nested command levels continue to qualify the heading with the parent
name. Add a unit test that mounts a root command hosting a child and
checks option registration, dispatch, setup and teardown, and
resolution of the application through the parent chain.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
The __init__() method reads default log configuration directly
from environment variables, making it impossible for subclasses to
change defaults.
Extract the defaults into _default_log_flags(), _default_log_level(),
_default_log_file(), and _default_show_backtrace() methods, then call
them from __init__(). Subclasses can now override these methods to
customize defaults without needing to override the entire __init__()
method.
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdBuild supports the targets all, clean and pkg-*. It uses builtin
dependency flavours matching those targets to resolve build order. To
make it more flexible in general, and allow it to support more
targets, e.g. "check" and "test", this commit adds a --dep-flavours
option.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add tests for jw-pkg.py calls run during makefile caching. Broken
caching can compromise the build without causing it to fail entirely,
i.e. in non-obvious ways.
Signed-off-by: Jan Lindemann <jan@janware.com>
local/src is a janware-specific path, remove it from App and defs.mk.
It's still in pkg.sh as a safety measure. Will have to go, too, but
is kept in for now until further audit.
Signed-off-by: Jan Lindemann <jan@janware.com>
Mypy scans both src/python/ and test/ under the same jw namespace,
and finds three test files named test.py. Without a package root at
the test/ level, mypy resolves all of them to the bare module name
test, and rejects them as duplicates.
Add __init__.py at test/ marks it as a package root, so each test.py
resolves to a unique full path, e.g.
test.unit.python.jw.pkg.lib.Result.test
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL with pi.dev v0.80.2
Signed-off-by: Jan Lindemann <jan@janware.com>
Yapf complains about spacing inconsistencies around assignment operators
in function calls, fix that.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL with pi.dev v0.80.2
Signed-off-by: Jan Lindemann <jan@janware.com>
The _load() helper in test/unit/python/jw/pkg/lib/ProjectConf/test.py
uses tempfile.NamedTemporaryFile(..., delete=False) but never removes
the created files. Each test call leaked a .conf file in /tmp.
Track all created paths in a _tmpfiles list and call _cleanup() at the
end of the test to unlink them.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL with pi.dev v0.80.2
Signed-off-by: Jan Lindemann <jan@janware.com>
The scheme property returns 'file://' for local paths instead of
'file', causing scheme_plus_authority to produce 'file://://' with a
triple slash.
Also fix the test to expect 'file' instead of 'file://'.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL with pi.dev v0.80.2
Signed-off-by: Jan Lindemann <jan@janware.com>
Add unit tests for the new ProjectConf module covering:
- Basic string and list value retrieval (get_str(), get_str_or_none(),
get_list(), get_list_or_none())
- Quoted values with preserved spaces and comment delimiters
- Inline comments outside quotes
- Comma-separated lists with quoted commas
- Line continuations
- Multiple sections
- Error cases: empty key, missing key/section, malformed sections,
unfinished continuations, unclosed quotes
- Error class is a subclass of ValueError
Also include a Makefile for running tests via `make test`.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a unit test for the Result class covering:
- stdout/stderr property access with various status values
- None output handling and exception behavior
- Encoding and strip_output property setters
- cmd and wd setters
- summary, summarize, and __repr__ behavior
- matches_error pattern matching
Signed-off-by: Jan Lindemann <jan@janware.com>
Move the unit test hierarchy to below test/unit/python/jw/pkg.
Nesting the subdirectories so deeply might be overly careful, but it
may as well be not - maybe in the future installable test packages
are going to be generated and need simple ways to install without
stepping onto each other's toes. If not, it's easier to cut two
directory components out than having to reorganize possibly
incoherent paths grown over multiple packages.
Signed-off-by: Jan Lindemann <jan@janware.com>
The Uri class provides URL parsing and manipulation utilities used
throughout jw-pkg. Add a unit test covering URL parsing, credential
handling, path manipulation, and safe string formatting.
Signed-off-by: Jan Lindemann <jan@janware.com>
After a pipeline change, CI now runs "make all" in a repo's root,
which uncovers two problems:
1. The help integration test only succeeded as long CI didn't run
"make all" before "make test". That way, the checked out
repository lacked the generated __init__.py files needed for some
modular subcommands to be fully loaded, and hence, the test should
have failed. The entire machinery only worked because the
subcommands in question are not not essential to building jw-pkg
itself: "secrets" and "posix". So, this commit adapts the help
integration test to the new reality.
2. Regarding python-tools.sh: Commit 55060486 satisfies yapf in some
places of the source code, but in others not anylonger. So patch
python-tools.sh's newline handling again.
While not thematically similar, both fixes get baked into one commit
to satisfy the requirement that every single commit needs to pass
"make clean all check test" individually.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a test/integration subdirectory as a location for integration
tests. The first tests that are added are unproblematic in that they
don't need superuser privileges to run:
- help: Recursively check if jw-pkg.py's help messages are unchanged
- packages ls: Check if the bash package exists and contains
/usr/bin/bash
/usr/bin/bashbug
Let's see how that fares in CI.
Signed-off-by: Jan Lindemann <jan@janware.com>
