jw-pkg-create-project.sh contains its own .gitignore template, which is
redundant to the new gitignore mechanism. Replace it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a target gitignore to topdir.mk, which blasts jw-pkg's default idea of
how .gitignore should look like into $(TOPDIR)/.gitignore. It's not a
prerequisite of any other target, because it's not meant to run
automatically. The idea is to keep some central maintenance of what should
be in there from jw-pkg's perspective, make it convenvient to update, but
leave the final decision to the commiter of any project using the
machinery.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add variables to shorten typical invocations of "jw-pkg project update
--format tmpl", and use them in py-topdir.mk
JW_PKG_TMPL_SEARCH_PATH
JW_PKG_TMPL_UPDATE
Signed-off-by: Jan Lindemann <jan@janware.com>
create-pkg-config's output is not covered by any test, even though the
template and the Requires cleanup produce exact text that pkg-config
consumes, so regressions in the generated file went unnoticed for a long
time.
Add an integration test that runs create-pkg-config with a full option set,
with space- and comma-mixed Requires input, and with a
Requires.private-only input, and compares the output against a reference
file.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
__cleanup_requires() replaces every run of whitespace with ", " before
re-pairing the version constraints, so input that is already
comma-separated, e.g. "jw-core >= 1.0, jw-base", comes out with a double
comma, "jw-core >= 1.0,, jw-base". And the Requires line is appended
without a trailing newline, so a following Requires.private line runs
straight into it.
Split the input on commas and whitespace, treating the version constraint
operators as delimiters that are re-paired with the preceding name and the
following version, so that comma- and space-separated input alike comes out
as a clean ", "-joined list. Add the missing newline after the Requires
line.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdCreateFile renders a file from project metadata, yet nothing runs it end
to end. The "tmpl" format substitutes --field values into template markers,
and the "pyright" format computes extra paths from the jw run dependencies
of a module. Both formats are untested, and so is the handling of missing
templates, unknown --field keys, and malformed command-line arguments.
Add an integration test under test/integration/jw-pkg/projects that drives
the command through the real CLI and diffs the rendered output against
a reference. It covers single- and multi-value field substitution, the
default and custom field separators including the %n newline escape, value
quoting, and --field-keys filling in keys the caller omitted. The pyright
case uses a relative topdir so the computed project paths stay relocatable.
The error cases check that the command rejects a missing template name, an
unknown template file, a --field key outside --field-keys, an unsupported
pyright separator, and invalid --format and --field arguments with the
expected message and exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
List values in a rendered template are joined with a fixed separator of
",\n" (comma plus newline), and there is no way to change that. Templates
sometimes want a different separator, e.g. a plain newline or a single-line
comma-separated list.
Add a --field-separator option that controls how list values are joined
in the rendered output. %n expands to a newline, and the default remains
",\n" for both formats. render_tmpl() applies the separator to the rendered
template, while render_pyright() rejects any other value: the built-in
pyrightconfig.json template is a fixed JSON document, and no alternative
separator is supported for it.
Update the help output test expectation accordingly.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The --field option accepts arbitrary KEY=VALUE pairs, inserting them into
the rendered template output, but there is no way to declare which keys
a template actually supports. Passing a key the template does not use goes
unnoticed, and a template cannot offer optional fields that drop out of
the output when not supplied.
Add a --field-keys option, a comma-separated list of the keys a template
accepts. Passing a key via --field that is not in that list is an error,
and keys from the list that are not passed are added with an empty value,
so the respective field renders as empty and effectively drops out of the
output.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The possible values of --format are defined by an Enum which is never
really used as such. Derive them from class introspection instead, i.e.
offer all formats that have a corresponding render_<format>() name.
Also, turn the option into real argparse-backed choices, and make the
argument mandatory, because that reflects the reality of the implementation
- there is no default.
Signed-off-by: Jan Lindemann <jan@janware.com>
create-file and create-pkg-config both print the rendered template with a
newline appended over the original template, owed to using print() instead
of sys.stdout.write(). Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
format_list_dict() substitutes {key} markers with str.replace(), which also
matches the marker inside ${key} and {{key}}, mangling text meant to
survive verbatim, e.g. the ${prefix} variable reference of a pkg-config
file. The built-in pkg-config template works around that with doubled
braces, which produces {/usr}-style output instead.
Replace the markers with a regular expression that skips markers preceded
by '$' or '{', and let the pkg-config template use the ${prefix} and
${exec_prefix} references pkg-config expects.
Extend the unit tests with marker escaping, value escaping, quoting,
indentation, and the rendering of the built-in templates.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
is_list_dict() accepts a string value as a list-dict value, and
render_values_to_list_dict() passes such values through unchanged.
merge_values() then merges them with list extension, and [] += 'xyz'
appends the individual characters, so every string value ends up as a list
of its characters. Normalize string values to single-element lists in
render_values_to_list_dict(), so that merge_values() appends whole values.
jw-pkg projects create-pkg-config is hit by this, because all of its values
are strings: the generated file contains one line per character. The code
path was just never exercised lately.
Add unit tests for the value layout guards, the normalization, and the
merging of the three supported layouts.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
--skip-excluded removed excluded modules from deps with
deps.remove() while iterating over deps. Removing an element
shifts the remaining elements to the left, so the loop skipped
the entry following each removed one, and consecutively
excluded modules survived the filter and leaked into the
output.
Build the filtered list with a comprehension instead.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
PY_CHECK_ROOTS defaults to the src and tools directories, so the ruff,
mypy, pyright, and yapf checks and the bad-pattern scan silently skip test
trees. Python tests below test/ therefore escape linting, type checking,
and formatting enforcement, even though py-check.mk itself defaults to the
whole directory when no roots are given at all.
Add $(TOPDIR)/test to the default PY_CHECK_ROOTS after the build is
through, so that the test tree receives the same checks and formatting as
src and tools. The previous commit prepared this repo's unit tests for the
strict type check and the formatter, so the extended roots pass cleanly
here.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
create-conftest-py generates the conftest.py that pytest test directories
pick up at build time, and make clean removes it. The generated file lacks
type annotations, though: pytest_configure() and the nested patched() take
untyped parameters, and the replacement of .pytest_sessionstart() on the
class is rejected by strict mypy.
Emit a type-annotated version instead: add a future annotations import,
type both functions, move the types that are only used in annotations
behind a TYPE_CHECKING guard, and silence the method replacement with a
targeted type ignore.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
The unit test files below test/unit/python/jw/pkg/lib/ define Cmd and App
subclasses with unannotated methods: __init__(), add_arguments(), _run()
and close() lack signatures, the made class attribute lacks a type, and the
_cleanup() and exit_context() helpers are untyped, so a strict mypy run
over the test tree fails on them. Likewise, the ExecApp and version tests
carry list formatting that yapf rejects.
Annotate the test classes following the library conventions: parent is App
| Cmd, the parser is ArgumentParser, and args is Namespace, and mark every
overridden method with @override. Guard the imports that are only used for
annotations behind TYPE_CHECKING, and add the future annotations import so
they stay out of the runtime path. Then reformat the test tree with yapf.
That folds the opts list of the ExecApp test and re-indents the rejected
operator list of the version test.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The spec split pattern ([=><]+) tokenizes only the =, > and < characters,
so a PEP 440 operator like ~= or != leaves its ~ or ! glued to the package
name: Dependency('pkg~=1.0') parses to base name 'pkg~' with operator '='.
App.__get_project_refs() carries the same pattern inline, so the same specs
corrupt the module name used for the -devel subpackage check. The split
also tolerates operator strings the language does not support, most visibly
==, which parses and renders but raises NotImplementedError only when
expansion is requested.
The spec language supports exactly =, <, <=, > and >=, and boundary
expansion implements all of them. Extend the split pattern with ~ and ! so
that foreign operators tokenize as operator strings, and reject every
operator outside the supported set in Dependency.__parsed_spec() with
Dependency.Error, naming the supported operators. Route
App.__get_project_refs() through Dependency for the name and module parts
instead of the second inline split, so the validation lives in one place.
The catch all in Dependency.__version_boundaries() stays as a backstop
against drift between the allow list and the expansion cases.
The tests drop == from the accepted operator loop, drop the now-unreachable
not-expandable case, and assert that ~=, !=, ~, ==, ===, << and >> are
rejected at parse time with a message naming the operator.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
App.__get_project_refs() recurses into the package dependency graph, but
the recursion end conditions only work well if all required packages are
present. Whether a required package is installed or not is decided upon a
wrong condition, however - existence of the project's project directory,
which may or may not be misleading for both the -devel and the -run
requirements. This can lead to various unwanted outcomes: Missing packages
happily inserted into the recursion buffer, process termination instead of
recursion stop, path lookup errors instead of a clearer "unmet dependency"
message (No project path found for module xyz, Failed to find directory of
project foo).
This commit cleanly detects if -run or -devel are installed and makes the
walk stop or raise under the appropriate conditions:
1. -run missing but required, raises unmet dependency
2. -devel missing but required, raises unmet dependency
3. -run present, no -devel required, stop recursing
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The installed state of a project is judged by the proofs of installation,
which differ per subpackage: make/project.conf in the dev tree or /opt
proves the -devel package or a buildable checkout, a VERSION file in the
project directory or /usr/share/doc/packages the -run package.
Add is_installed(), which answers the proof question for a given subpackage
by searching the per-subpackage locations.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
The iteration order contains every project the dependency walk resolved,
including projects that are only installed as -devel packages below /opt.
Running the target in such a project ran make in a read-only, source-less
directory and failed the run.
Resolve each module against the workspace root only, and skip the target
with a notice when there is no buildable project directory. Factor the skip
notice into log_skip() so the platform-exclusion path shares it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
find_dir() and its helpers resolve a project name against the workspace
root and /opt. The build iteration needs to resolve a module against the
workspace only, to tell buildable projects apart from installed ones.
Add a projs_roots parameter to find_dir(), __find_dir() and __proj_dir()
that restricts the search to the given roots. With the parameter omitted
the previous behavior is unchanged.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
pkg-dist.mk queries the package relations with
--dont-expand-version-macros, and build_pkg() expanded the remaining
VERSION and VERSION-REVISION macros with a local sed-based
expand_version_macros(). That substitutes the version of the project being
built for every macro, instead of the version of the dependency the macro
refers to.
Drop --dont-expand-version-macros from the proj_query calls so the macros
resolve to the dependency's own version, and remove expand_version_macros()
from build_pkg(), which passes the PKG_* variables to rpmbuild as they are
queried.
Signed-off-by: Jan Lindemann <jan@janware.com>
The lib.version module carries no documentation: the Syntax, Component and
Boundary types, the Version and Dependency classes and their public methods
are bare, and the compatibility tiers the next_* stepping methods implement
are not documented anywhere.
Add docstrings: a line each for the base.py types, the compatibility
tier table in the Version class, and a short description for every
public method, with the three next_* methods citing the tier each one
steps to.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
- The pkg_relations() function is a horribly bad read. It contains a
mind-bending amount of interwoven case distinctions that are not clearly
reflected in the participating variable and function names. Much of it is
version handling. That's intricate by nature, but much of it has now been
implemented in the lib.version module, so moving the logic there makes
the function a good deal more readable than before. That's most of what
this commit does.
- Use version syntax macros from lib.version instead of
pkg_relations-defined macros, and fix the fallout in BaseCmdPkgRelations
and CmdCreateFile.
- Use lib.version also as a central place for dep string parsing from App.
Signed-off-by: Jan Lindemann <jan@janware.com>
pkg_relations() parses dependency specifications such as 'foo-devel >=
1.2.3-4' with inline regular expressions and mixes the parsing with version
macro expansion and rendering in three different syntaxes.
This commit adds the lib.version module that owns the parsing of a single
specification: the base and full name and the version boundary with its
operator. Rendering happens in constraint_str() in SEM_VER, DEBIAN or
NAMES_ONLY syntax, with untemplated, include_revision, as_range,
no_subpackages and quote options. The VERSION, VERSION-REVISION and
REVISION macros are resolved through a version lookup callback,
lib.version.Version.parse_deps_spec() splits comma-separated specification
strings into Version objects, and Version breaks a version down into its
major, minor, micro and revision parts.
Unit tests cover parsing, macro resolution, rendering in all syntaxes and
the range expansion.
The module is designed to be integrated into the status quo and
intentionally does not address a couple of further TODOs. Notably
version.Syntax.SEM_VER is intended to replace
pkg_relations.VersionSyntax.SemVer, but neither are really semantic
versioning according to spec. They are close but more targeted toward RPM
and Debian package versioning. And the naming and semantics differ from
SemVer.
Naming of the four components (major and minor version) is identical,
semantic meaning aside, there's no disagreement between real SemVer,
Debian, RPM and jw-pkg. The third and fourth component deviate:
SemVer Debian RPM jw-pkg lib.version
3 PATCH RELEASE MICRO
4 PRERELEASE Revision Release REVISION REVISION
TODOs: RELEASE over the rest of jw-pkg needs to be adjusted by a later
commit. Syntax.SEM_VER should also be renamed, to Syntax.JW_PKG maybe,
because, as said, it's not SemVer and will probably never be - SemVer
doesn't provide the compatibility guarantees that the jw-pkg versioning
scheme offers. To be documented.
Signed-off-by: Jan Lindemann <jan@janware.com>
The existence of CONFIG_SUBDIR hijacks INSTALL_CFGDIR to a subdirectory (by
default /etc/opt/<package>/$(CONFIG_SUBDIR)). Then std_install_rules apply,
and (the bent) INSTALL_CFGDIR is installed, but its parent directory, the
original $(INSTALL_CFGDIR), is not a prerequisite of install anylonger, and
has no rule anymore, hence log-install is never run for it. Instead it's
implicitly created by log-install -D $(INSTALL_CFGDIR).
This commit gives /etc/opt/<package> std-install rule and variables back as
CFGTOPDIR variants, and inserts it early into the install target's
prerequisite list.
Signed-off-by: Jan Lindemann <jan@janware.com>
During "make install", the packaging machinery should run "$(LOG_INSTALL)
-D /opt/<pkg>" but runs "mkdir -p /opt/<pkg>" instead. As a consequence,
the created directory is not owned by any of the created packages. This
commit fixes that by introducing INSTALL_PREFIXDIR and installing it like
all other directories.
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commit reads __abstractmethods__ via a getattr() with an empty
frozenset fallback, because in mypy 2.3.1 the scanned classes are typed as
type[object], which does not declare the attribute.
Add is_abc_class() as a TypeGuard, and skip the classes it rejects with a
debug line: LoadTypes now yields only ABC-derived classes. The guard
narrows to the classes that carry the attribute, so that the debug line can
now read __abstractmethods__ directly.
The command loaders are unaffected: every class they load is derived from
AbstractCmd, and hence from ABC. For loads that rely on the name filter
alone, plain classes are now skipped instead of being yielded.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
LoadTypes._classes() reads the __abstractmethods__ attribute of each class
that inspect.getmembers() returns for its debug output. mypy 2.2.0 allows
this, mypy 2.3.1 doesn't: It now types those classes as type[object]
instead of Any. The attribute itself is only declared on the ABCMeta
metaclass, so the direct access fails the type check.
Read the attribute through getattr() with an 'unknown' fallback string, and
annotate the variable explicitly. The runtime behavior is unchanged, since
the attribute exists whenever inspect.isabstract() is true, and the call
satisfies the checker.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: user.email <jan@janware.com>
_connect_kwargs(hide_secrets = True) is used to log the connection
parameters when a connection fails, without leaking the password. The
filtered dictionary is built before the password is replaced with
'<hidden>', and the replacement is applied to the local kwargs
dictionary afterwards, after the filtered copy has already been made.
The dictionary that ends up in the log therefore still contains the
real password.
Hide the password before building the filtered dictionary.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_put() catches a failure of the remote command sequence and logs
"Failed to get <path> from <root>", a phrase copied over from the
get() path. This path, however, pushes content, so the message
describes the wrong operation.
Log a put-oriented message instead.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_is_dir() logs a DEBUG message when _stat() is not implemented and it
must guess from the trailing slash whether a path is a directory. The
second part of that message is a plain string rather than an f-string,
so '{path}' is logged verbatim instead of the path.
Make it an f-string so the path is interpolated.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_put() writes the content into a temporary file with tee and, when
atomic is set, moves it to the target path with a final mv. The loop
over the command list sets tmp_file to None after each command, on the
assumption stated in the comment that the file has been moved at that
point - which is only true for the last command. When a chown, chmod,
or mv after the tee fails, the finally block finds tmp_file is None
and leaves the temporary file behind.
Reset tmp_file only after all commands have completed, so that the
finally block erases the temporary file whenever a step fails.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_run_ssh() waits for the remote process's exit status before reading
stdout and stderr. When a command produces more output than the
channel's flow-control window can hold, the server stops sending, the
remote process blocks on its write and never exits, and
recv_exit_status() blocks forever.
Drain stdout and stderr to EOF first - the channels close when the
process exits, so reading them also implies completion - and only
then query the exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
