Add a new target git-show-ahead-of-master, which does what git-show-pushable did up to now: List all repos where the currently checked-out branch is ahead of origin/master.
git-show-pushable now shows repos that have their checked-out branch ahead of their respective upstream branches. Which is truer to what the target's name suggests.
Rename command "distro" to "pkg" together with "info", its last remaining subcommand. "distro" is often used in the sense of "Linux distribution", which would be too narrow for the targets jw-pkg could theoretically support.
With the exception of the "info" subcommand, nearly all of distro's subcommands deal with package managing, so push them into their own command category.
pgit.sh get is not invoked with --create-remote-user-repos by default. For the janware Git servers, this makes CI fail over new repos: Repos from the maintainers are pulled, but before pkg-rebuild-reinstall, repos that the devops user has no remote repo for are purged again and are subseqently missing during build.
Passing --create-remote-user-repos to all pgit.sh get invocations should fix the problem, so do that.
/usr/bin/file <candidate> | grep text is used to detect if a file is a text file or not. Replace that with grep -I., because that adds some files left out by /usr/bin/file, notably systemd service files.
Rename git-show-pushable-master-branches to git-show-pushable, because, the target's recipe doesn't show pushable master branches, but all commits in the working directories current branch not present in origin/master.
Don't use persistent SSH-connections any more for "pkg-" targets,
because, sadly, this hangs after uploading a package.
This commit puts some safeguards against hanging SSH into place, namely setting default SSH timeouts down, SSH keepalive, setting SSH BatchMode to yes.
Use DEP_PROJECTS instead of BUILD_PROJECTS everywhere. DEP_PROJECTS considers more projects, and for all targets it is relevant to, it's desirable to have them operate on the maximum blast radius.
make git-show-pushable-master-branches misses projects which should better be pulled in as dependencies. It searches $(BUILLD_PROJECTS) for repositories with pushable commits, that variable leaves some out, and DEP_PROJECTS has them. It make use of the renovated pkg-requires command.
This move should be extended to other uses of BUILD_PROJECTS as well, after giving it some test runs. And the legacy and redundant command prereq should be removed.
Log to stderr and add some ASCII-art around the output. Also, add a --porcelain option to allow more stable output parsing. Subsequently, use that option in make targets parsing the output, notably make diff and make git-show-xxx.
In a push to eventually merge the classes, somewhat align the command-line API of CmdRequiredOsPkg to the one of BaseCmdPkgRelations by using dependency flavours as mandatory, first argument.
Move the dependencies listed in BASE_PKGS from projcts-dir.mk and topdir.mk into project.conf.
Due to various hen-and-egg problems on a minimal system, in some situations these packages can't be installed from project.conf. The same is true with BASE_PKGS, however, so remove it, at least that does away with some redundancy.
Don't prefix JW_PKG_PY_PROJECTS with time -p. A timing summary shows up in too many places unexpectedly, e.g. in the context of the target update-text-files. Add back later as more concrete demand comes up.
The code below lib.distro, as left behind by the previous commit, is geared towards being directly used as a command-line API. This commit introduces the abstract base class Distro, a proxy for distribution-specific interactions. The proxy abstracts distro specifics into an API with proper method prototypes, not argparse.Namespace contents, and can thus be more easily driven by arbitrary code.
The Distro class is initialized with a member variable of type ExecContext, another new class introduced by this commit. It is designed to abstract the communication channel to the distribution instance. Currently only one specialization exists, Local, which interacts with the distribution and root file system it is running in, but is planned to be subclassed to support interaction via SSH, serial, chroot, or chains thereof.
"clone" in the Git sense means to copy a remote project over from scratch. pgit.sh clone has come from that, but has since evolved into something different, a mixture of clone, pull and fetch, so find a different name. "get" seems generic enough and doesn't clash with a Git meaning. Adapt variable names accordingly across the project.
invocations. Those invocations typically happen in the context of pkg-%install, so add that target, specializing the pkg-% target.
The problem this solves is that /etc/profile is currently read only once before bootstrapping all software on a pristine system is started. This might lead to the situation that package A has installed environment variable definitions into /etc/profile.d, package B needs them for building, but never gets to read them.
Multiple variables are redundantly defined both for a project and for the multiple-projects toplevel directory. Add a place to maintain them centrally, and add PGIT_SH as a first variable.
The target pkg-delete-ours, invoked from the projects directory, should wipe all packages from the system which have been created and installed via jw-pkg.
Currently they are selected via url =~ janware. This is a default string which can be overridden by redefining JANWARE_PACKAGE_FILTER. This might not be the most generic name, but is kind of consistent and will be matched once all variables get renamed to a more generic naming scheme.
This currently does not get all packages: Some are not labeled with URLs matching "janware", because jw-pkg is only used as a convenient way to package other people's open source projects.
There's pkg-manager-refresh already, so by adding pkg-manager-dup the distribution can be upgraded by distribution agnostic targets only through the Makefile. This might come in handy for CI, so add it.
--quote puts double quotation marks around the listed dependencies, protecting version requirements (>= 1.0) and parenthesis "perl(GD)" from the shell.
Retire pkg-manager.sh and replace it by the cleaner "jw-pkg.sh distro" command, essentially providing the same functionality and nearly the same command-line interface.
400 LOC more. That's what the move from a shell script to the more maintainable Python versions costs. Still a good idea, and the enhanced extensibility might pay off in terms of LOC with other shell scripts in the future.
Reorganize the Python module structure. Placing the command classes under jw.cmds.projects instead of jw.build.cmds will allow to add a nested command structure, with the current commands, being mostly related to building software, found below a "projects" toplevel command.
Other conceivable commands could be "package" for packaging, or "distro" for commands wrapping the distribution's package manager.
Kali Linux' default installation doesn't have /usr/bin/time which brings out a but: $(TIME) doesn't expand to nothing but to -p, which fails miserably, of course. Fix that.
PGIT_SH gets added --remote-base, but too late to make it into the non-recursive variable PGIT_SH_CLONE. This leads to --remote-base lacking from the clone invocation, and anonymous Git over HTTP failing because it tries to clone via SSH. Fix that.
make git-show-pushable-master-branches output too litte for two reasons: 1. grep -q returns zero also if no matches are found, and 2. PROJECTS doesn't contain all relevant projects. BUILD_PROJECTS is more meaningful.
--create-remote-user-repos had been disabled in 4053451bfd on the grounds that it's hard to test and possibly superflous. It actually is not superfluous, as devops builds show, and that's a valid test-case, so re-enable it.
jw-build doesn't stop at building software, packaging it afterwards is also a core feature, so this commit gives the package a better name.
The commit replaces strings s/jw-build/jw-pkg/ in text files and file names. Fallout to the functionality is fixed, variable names are left as they are, though. To be adjusted by later commits.
In the attempt to move away from communicating options via environment variables from one part of jw-build software to another, replace PGIT_CLONE_FROM_USER with the clearer --refspec option. Which is also more versatile.
