Commit 4e347683 removed a tralining newline after imports. Yapf
doesn't like that, so revert that change.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a test/integration subdirectory as a location for integration
tests. The first tests that are added are unproblematic in that they
don't need superuser privileges to run:
- help: Recursively check if jw-pkg.py's help messages are unchanged
- packages ls: Check if the bash package exists and contains
/usr/bin/bash
/usr/bin/bashbug
Let's see how that fares in CI.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add include file to provide some definitions for a first integration
test suite. It provides the shared make variables, most notably
TEST_CMD_LINE as handy default for running jw-pkg.py.
Signed-off-by: Jan Lindemann <jan@janware.com>
Recursive make fails in scripts/usr-bin when driven through jw-pkg.
Add empty test targets to remedy that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Relabel the toplevel command CmdPkg from "pkg" to "packages", because
it rolls off the tounge much more nicely. Keep "pkg" as an alias for
compatibilty.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a property aliases to AbstractCmd in prepeparation for commands
to bear multiple names / abbreviations / aliases.
The App.add_cmds_to_parser() function uses parse_known_args() to
determine which subcommand was invoked, then conditionally registers
nested subcommands. The lookup dictionary (scs) contains only
canonical names, not aliases, so add them too, otherwise using the
alias instead of the canonical name causes the lookup to fail and
nested subcommands to never be registered.
Fix: Register each alias in scs pointing to the same SubCommand
object, and deduplicate with id(sc) when iterating in all=True mode
to avoid infinite recursion on help output.
Signed-off-by: Jan Lindemann <jan@janware.com>
Make python-tools.sh generate empty __all__ as [] on one line instead
of multi-line format, and remove trailing blank line.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the target sh-syntax-check, which triggers bash syntax linting
with shellcheck This commit stops short of making target all depend
on it, because the fallout is impressive.
Signed-off-by: Jan Lindemann <jan@janware.com>
To make EXE_SH useful for automated shell syntax checking, remove all
non POSIX shell / bash scripts from that variable, and place the
non-shell scripts into EXE_SCRIPTS.
Signed-off-by: Jan Lindemann <jan@janware.com>
Modifying JW_PKG_XXX_PATH in pre-local.mk is fragile.
Amending PREREQ_RUN in pre-local.mk works, but only with
$(JW_PKG_NO_CACHE) == true, or if "undefine JW_PKG_XXX_PATH" is also
added in pre-local.mk. Otherwise JW_PKG_XXX_PATH will not be
recomputed, because it's already defined from the cache.
Introduce the new variable PREREQ_RUN_ADD to solve that. If it's
defined, it automatically invalidates the JW_PKG_XXX_PATH variables
and sets them up for recalculation in py-path.mk / ldlibpath.mk.
Signed-off-by: Jan Lindemann <jan@janware.com>
Reverse inclusion order of .cache-project.mk and cache-projects.mk:
Definitions in .cache-project.mk should win over cache-projects.mk,
because it's the more specialized include file, and the way the
definitions in both files are structured, the later doesn't overwrite
the earlier.
Signed-off-by: Jan Lindemann <jan@janware.com>
The clean-dirs target does not only clean the repos present in
PROJECTS, but all repos it finds to be dirty, and clean-all-dirs does
the opposite. I suppose that was an oversight, swap their recipes.
Moreover, cleaning all directories goes about its business in an
overly complicated and unecessarily time-consuming way, fix that,
too.
Signed-off-by: Jan Lindemann <jan@janware.com>
py-path.mk is the place to host PYTHONPATH and MYPYPATH definitions,
so move them there from py-defs.mk for consistency. Also, remove some
dead code.
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit adds more tweaks to shell command output in order to make
it nicer. The biggest patch is in Result.__summarize(), which makes
it more versatile, and allows removal of some code in SSHClient.
App sees some independent, minor result format beautification.
Signed-off-by: Jan Lindemann <jan@janware.com>
The following variables contain colons as path-separators:
- JW_PKG_PYTHON_PATH
- JW_PKG_EXE_PATH
- JW_PKG_LD_LIBRARY_PATH
This commit makes them use spaces instead, so they can be more
easily amended by Makefiles using them. Also define them in a more
uniform way, and use the newly introduced PREREQ_RUN variable to fill
them, which in turn can also be appended to before that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Support a --delimiter option to the ldlibpath and exepath commands.
Notable use case are the JW_PKG_XXX_PATH variables, which should use
spaces instead of colons.
TODO: Merging those two command modules with BaseCmdPkgRelations
would have made introducing this redundancy unnecessary, check if
that's a possibility.
Signed-off-by: Jan Lindemann <jan@janware.com>
Some options to the pkg-xxx commands, like flavour, --subsections and
--ignore understand a comma as delimiter if multiple option values
are specified. The comma character is not very friendly to use in
$(call ...) macros, though, so support spaces and pipe characters as
well.
Signed-off-by: Jan Lindemann <jan@janware.com>
At present, the PREREQ-variable is effectively only used to detect if
prerequiste packages haven't run "make all" before make is run in a
given package. Also, it's only useful in $(TOPDIR). This commit
splits the variable up into PREREQ_BUILD and PREREQ_RUN, and makes
the variables available in every Makefile of a package by placing
them in defs.mk instead of topdir.mk.
This also fixes a problem that PREREQ was cached before being filled,
hence empty. Which effectively wasn't much of a problem, because it
was basically unused, but still.
Signed-off-by: Jan Lindemann <jan@janware.com>
cache.mk generates .project-cache.mk, and this commit supports
disabling the definitions in the generated cache by setting
JW_PKG_NO_CACHE=true.
Signed-off-by: Jan Lindemann <jan@janware.com>
make/Makefile is responsible to generate $(TOPDIR)/cache-projects.mk.
The variables are taken from .cache-project.mk, with some variables
intentionally omitted, but their ifndef / endif blocks remain in
place. Not harmful but ugly. Make sed range-delete the left-over
blocks entirely.
Signed-off-by: Jan Lindemann <jan@janware.com>
Aside from PYTHONPATH, ldlibpath.mk runs jw-pkg.py for determining
other paths, too, which is often unneeded and can impact performance.
Split the PYTHONPATH detection into a dedicated py-path.mk, and
include it from ldlibpath.mk, so it can be used instead where needed.
Signed-off-by: Jan Lindemann <jan@janware.com>
If make echo-py's output is accidentally subjected to shell
expansion, it can yield surprising results. Protect it from that
happening.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for PY_INIT_SUBMODULES to py-mod.mk. If it is defined in
a Makefile including py-mod.mk, the listed submodules will be added
to __init__.py and thus included in the list of things that can be
imported from a module.
This commit also adds support for --submodules to python-tools.sh for
that to happen.
Signed-off-by: Jan Lindemann <jan@janware.com>
Symbols in the __all__ list of a module marked with "# export"
arent't currently added to __init__.py. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
The global --topdir-format make:XXX option to jw-pkg is half-baked at
best, and __find_dir() ignores it entirely. Make __find_dir() return
some Makefile-syntax-formatted output if the option is present. Not
used anywhere, currently, and, hence, badly tested, but still better
than the situation before.
Signed-off-by: Jan Lindemann <jan@janware.com>
Overriding the _run() method entirely in App subclasses is currently
only possible if the application supports a subcommand structure.
Make it possible to use it as an abstraction for a single-command
application.
Signed-off-by: Jan Lindemann <jan@janware.com>
"jw-pkg secrets compile-templates" errors out over checking the
contents of inexistent secrets, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
The .strip property of class Result defaults to True, and its name
isn't very clear. Rename it to .strip_output, and default it to False
to avoid surprising contents for unsuspecting callers.
Signed-off-by: Jan Lindemann <jan@janware.com>
The command line jw-pkg.py is run with is logged with level "debug",
and reconstructed with ' '.join(sys.argv). Use pretty_cmd() instead,
this adds quotes around spaces.
Signed-off-by: Jan Lindemann <jan@janware.com>
Commands executed by ExecContext and its derived classes don't
populate the "cmd" parameter of "Result"'s constructor. Fixing that
makes for nicer error messages.
Signed-off-by: Jan Lindemann <jan@janware.com>
Use the (mostly) call-compatible log implementation from jw-python.
This buys us much nicer logs with color and source-code positon
annotations.
Signed-off-by: Jan Lindemann <jan@janware.com>
In LoadTypes' constructor, allow the type_filter parameter to be of
type Sequence[type[Any]] instead of list[type[T]]. a) Sequence is
more generic than list, and b) with T instead of Any, trying to
instantiate with an abstract class has mypy complain:
# E: Only concrete class can be given where "type[MyClass]" is
expected [type-abstract]
- type_filter: list[type[T]] = [],
+ type_filter: Sequence[type[Any]] | None = None,
Not that this makes mypy complain that it needs an annotation at the
places where LoadTypes is used.
Signed-off-by: Jan Lindemann <jan@janware.com>
OpenSUSE leaves installing local packages to the default
implementation in lib.Distro._install_local_files(), which passes the
package path to the package manager, i.e. zypper in OpenSUSE's case.
That has advantages, namely automatic installation of dependencies,
but also disadvantages, namely the attempt to install dependencies
even if the package manager is disfunctional, possibly because an
installed package containing installation sources is broken.
That could lead to a deadlock when trying to install a fixed package.
I see two ways out: Support an additional flag to jw-pkg's install
command which selects whether or not dependencies shall be resolved
along, or just use rpm directly for all local install attempts.
The latter is the less fancy way to handle this, so as a first step
make it the default by overriding suse.Distro._install_local_files().
Signed-off-by: Jan Lindemann <jan@janware.com>
Define default parameter values for Result's constructor, namely None
for exit status, stdout and stderr.
Instantiating a Result object without parameters signifies "this
object doesn't contain data from a real process's exit event". Up to
now, similar meaning has been hand-crafted by ExecContext's run() and
friends by using an error exit status (1) to make sure it wasn't
mistaken for success. This commit formalizes that into the Result
structure itself, but uses None instead for the exit status.
Controlling default values in Result itself also means that the
Result class gets better awareness of what it contains, and its log
messages and stdin / stdout can be more fitting:
- If a real process failed, make stdout return at least b''
- If a real process succeeded, make stdout return at least b''
Returning something from .stdout on success fixes a real bug: An
attempt to access what "rpm -U somepackage.rpm" returns, namely
nothing, raises a bogus exception, because stdout is None.
Signed-off-by: Jan Lindemann <jan@janware.com>
