Move the unit test hierarchy to below test/unit/python/jw/pkg. Nesting the subdirectories so deeply might be overly careful, but it may as well be not - maybe in the future installable test packages are going to be generated and need simple ways to install without stepping onto each other's toes. If not, it's easier to cut two directory components out than having to reorganize possibly incoherent paths grown over multiple packages.
The Uri class provides URL parsing and manipulation utilities used throughout jw-pkg. Add a unit test covering URL parsing, credential handling, path manipulation, and safe string formatting.
After a pipeline change, CI now runs "make all" in a repo's root, which uncovers two problems:
1. The help integration test only succeeded as long CI didn't run "make all" before "make test". That way, the checked out repository lacked the generated __init__.py files needed for some modular subcommands to be fully loaded, and hence, the test should have failed. The entire machinery only worked because the subcommands in question are not not essential to building jw-pkg itself: "secrets" and "posix". So, this commit adapts the help integration test to the new reality.
2. Regarding python-tools.sh: Commit 55060486 satisfies yapf in some places of the source code, but in others not anylonger. So patch python-tools.sh's newline handling again.
While not thematically similar, both fixes get baked into one commit to satisfy the requirement that every single commit needs to pass "make clean all check test" individually.
Add a test/integration subdirectory as a location for integration tests. The first tests that are added are unproblematic in that they don't need superuser privileges to run:
- help: Recursively check if jw-pkg.py's help messages are unchanged
- packages ls: Check if the bash package exists and contains
Add include file to provide some definitions for a first integration test suite. It provides the shared make variables, most notably TEST_CMD_LINE as handy default for running jw-pkg.py.
Relabel the toplevel command CmdPkg from "pkg" to "packages", because it rolls off the tounge much more nicely. Keep "pkg" as an alias for compatibilty.
Add a property aliases to AbstractCmd in prepeparation for commands to bear multiple names / abbreviations / aliases.
The App.add_cmds_to_parser() function uses parse_known_args() to determine which subcommand was invoked, then conditionally registers nested subcommands. The lookup dictionary (scs) contains only canonical names, not aliases, so add them too, otherwise using the alias instead of the canonical name causes the lookup to fail and nested subcommands to never be registered.
Fix: Register each alias in scs pointing to the same SubCommand object, and deduplicate with id(sc) when iterating in all=True mode to avoid infinite recursion on help output.
Add the target sh-syntax-check, which triggers bash syntax linting with shellcheck This commit stops short of making target all depend on it, because the fallout is impressive.
To make EXE_SH useful for automated shell syntax checking, remove all non POSIX shell / bash scripts from that variable, and place the non-shell scripts into EXE_SCRIPTS.
Modifying JW_PKG_XXX_PATH in pre-local.mk is fragile.
Amending PREREQ_RUN in pre-local.mk works, but only with $(JW_PKG_NO_CACHE) == true, or if "undefine JW_PKG_XXX_PATH" is also added in pre-local.mk. Otherwise JW_PKG_XXX_PATH will not be recomputed, because it's already defined from the cache.
Introduce the new variable PREREQ_RUN_ADD to solve that. If it's defined, it automatically invalidates the JW_PKG_XXX_PATH variables and sets them up for recalculation in py-path.mk / ldlibpath.mk.
Reverse inclusion order of .cache-project.mk and cache-projects.mk: Definitions in .cache-project.mk should win over cache-projects.mk, because it's the more specialized include file, and the way the definitions in both files are structured, the later doesn't overwrite the earlier.
The clean-dirs target does not only clean the repos present in PROJECTS, but all repos it finds to be dirty, and clean-all-dirs does the opposite. I suppose that was an oversight, swap their recipes.
Moreover, cleaning all directories goes about its business in an overly complicated and unecessarily time-consuming way, fix that, too.
This commit adds more tweaks to shell command output in order to make it nicer. The biggest patch is in Result.__summarize(), which makes it more versatile, and allows removal of some code in SSHClient.
App sees some independent, minor result format beautification.
This commit makes them use spaces instead, so they can be more easily amended by Makefiles using them. Also define them in a more uniform way, and use the newly introduced PREREQ_RUN variable to fill them, which in turn can also be appended to before that.
Support a --delimiter option to the ldlibpath and exepath commands. Notable use case are the JW_PKG_XXX_PATH variables, which should use spaces instead of colons.
TODO: Merging those two command modules with BaseCmdPkgRelations would have made introducing this redundancy unnecessary, check if that's a possibility.
Some options to the pkg-xxx commands, like flavour, --subsections and --ignore understand a comma as delimiter if multiple option values are specified. The comma character is not very friendly to use in $(call ...) macros, though, so support spaces and pipe characters as well.
At present, the PREREQ-variable is effectively only used to detect if prerequiste packages haven't run "make all" before make is run in a given package. Also, it's only useful in $(TOPDIR). This commit splits the variable up into PREREQ_BUILD and PREREQ_RUN, and makes the variables available in every Makefile of a package by placing them in defs.mk instead of topdir.mk.
This also fixes a problem that PREREQ was cached before being filled, hence empty. Which effectively wasn't much of a problem, because it was basically unused, but still.
make/Makefile is responsible to generate $(TOPDIR)/cache-projects.mk. The variables are taken from .cache-project.mk, with some variables intentionally omitted, but their ifndef / endif blocks remain in place. Not harmful but ugly. Make sed range-delete the left-over blocks entirely.
Aside from PYTHONPATH, ldlibpath.mk runs jw-pkg.py for determining other paths, too, which is often unneeded and can impact performance. Split the PYTHONPATH detection into a dedicated py-path.mk, and include it from ldlibpath.mk, so it can be used instead where needed.
Add support for PY_INIT_SUBMODULES to py-mod.mk. If it is defined in a Makefile including py-mod.mk, the listed submodules will be added to __init__.py and thus included in the list of things that can be imported from a module.
This commit also adds support for --submodules to python-tools.sh for that to happen.
The global --topdir-format make:XXX option to jw-pkg is half-baked at best, and __find_dir() ignores it entirely. Make __find_dir() return some Makefile-syntax-formatted output if the option is present. Not used anywhere, currently, and, hence, badly tested, but still better than the situation before.
Overriding the _run() method entirely in App subclasses is currently only possible if the application supports a subcommand structure. Make it possible to use it as an abstraction for a single-command application.
The .strip property of class Result defaults to True, and its name isn't very clear. Rename it to .strip_output, and default it to False to avoid surprising contents for unsuspecting callers.
The command line jw-pkg.py is run with is logged with level "debug", and reconstructed with ' '.join(sys.argv). Use pretty_cmd() instead, this adds quotes around spaces.
Commands executed by ExecContext and its derived classes don't populate the "cmd" parameter of "Result"'s constructor. Fixing that makes for nicer error messages.
