Add support for the targets "check" and "check-post" to
projects-dir.mk to make them usable from inside the projects
directory.
Note that "check-pre" is intentionally left out: Running "make check"
in a project before its prerequisite projects have built their
__init__.py files will fail due to broken import resolution.
Signed-off-by: Jan Lindemann <jan@janware.com>
The clean-dirs target does not only clean the repos present in
PROJECTS, but all repos it finds to be dirty, and clean-all-dirs does
the opposite. I suppose that was an oversight, swap their recipes.
Moreover, cleaning all directories goes about its business in an
overly complicated and unecessarily time-consuming way, fix that,
too.
Signed-off-by: Jan Lindemann <jan@janware.com>
By the time projects-dir.mk is used during onboarding, it's already
cloned, and so is jw-pkg in all its glory. So better use a
ssh-wrapper.sh directly under jw-pkg's version control instead of
plainly generating one with echo some-script-logic > ssh-wrapper.sh.
This has the main benefit of allowing a more elaborate script. The
one added by this commit removes "-l user" from remotes which have a
standard-user@gitserver form, typically because they differentiate
users via their SSH pubkeys only, and which would deny access if both
-l user and standard-user@ were specified.
ssh-wrapper.sh still needs to be a target which is updated by a
recipe, because the version found in jw-pkg can't be trusted to be
executable during bootstrapping, because "make all" has not run, yet.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add get-pub for top directory / projects directory level. This is a
janware specific target needed for CI. It integrates the current
master branch from the new Forgejo-based Git repos.
That said, this will likely need to go along with other release
machinery. Packaging is good, but releasing over a bunch of
directories is an intricate process, as-is only usable by janware
itself, and doesn't need to be part of jw-pkg.
Signed-off-by: Jan Lindemann <jan@janware.com>
janware upstream development server moved from git.janware.com to
devgit.janware.com. This commit follows the move with pretty much a
simple
s/git.janware.com/devgit.janware.com/
over jw-pkg. It found 14 matches, that's pretty bad.
FIXME: Reduce the redundancy, or, better, replace the
devgit.janware.com goodies by a more generally useful concept
altogether.
Signed-off-by: Jan Lindemann <jan@janware.com>
To make it easier to force-push feature branches, allow that, albeit
under --force-with-lease protection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a new target git-show-ahead-of-master, which does what
git-show-pushable did up to now: List all repos where the currently
checked-out branch is ahead of origin/master.
git-show-pushable now shows repos that have their checked-out branch
ahead of their respective upstream branches. Which is truer to what
the target's name suggests.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename command "distro" to "pkg" together with "info", its last
remaining subcommand. "distro" is often used in the sense of "Linux
distribution", which would be too narrow for the targets jw-pkg could
theoretically support.
Signed-off-by: Jan Lindemann <jan@janware.com>
With the exception of the "info" subcommand, nearly all of distro's
subcommands deal with package managing, so push them into their own
command category.
Signed-off-by: Jan Lindemann <jan@janware.com>
pgit.sh get is not invoked with --create-remote-user-repos by
default. For the janware Git servers, this makes CI fail over new
repos: Repos from the maintainers are pulled, but before
pkg-rebuild-reinstall, repos that the devops user has no remote repo
for are purged again and are subseqently missing during build.
Passing --create-remote-user-repos to all pgit.sh get invocations
should fix the problem, so do that.
Signed-off-by: Jan Lindemann <jan@janware.com>
/usr/bin/file <candidate> | grep text is used to detect if a file is
a text file or not. Replace that with grep -I., because that adds
some files left out by /usr/bin/file, notably systemd service files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename git-show-pushable-master-branches to git-show-pushable,
because, the target's recipe doesn't show pushable master branches,
but all commits in the working directories current branch not present
in origin/master.
Signed-off-by: Jan Lindemann <jan@janware.com>
This reverts commit 88e81 and effectively re-enables persistent SSH
connections to speed up builds.
commit 88e8197ed7
Author: Jan Lindemann <jan@janware.com>
Date: Thu Apr 20 20:00:46 2017 +0000
projects-dir.mk: Partly disable persistent SSH
Don't use persistent SSH-connections any more for "pkg-" targets,
because, sadly, this hangs after uploading a package.
This commit puts some safeguards against hanging SSH into place,
namely setting default SSH timeouts down, SSH keepalive, setting SSH
BatchMode to yes.
Signed-off-by: Jan Lindemann <jan@janware.com>
Use DEP_PROJECTS instead of BUILD_PROJECTS everywhere. DEP_PROJECTS
considers more projects, and for all targets it is relevant to, it's
desirable to have them operate on the maximum blast radius.
Signed-off-by: Jan Lindemann <jan@janware.com>
make git-show-pushable-master-branches misses projects which should
better be pulled in as dependencies. It searches $(BUILLD_PROJECTS)
for repositories with pushable commits, that variable leaves some
out, and DEP_PROJECTS has them. It make use of the renovated
pkg-requires command.
This move should be extended to other uses of BUILD_PROJECTS as well,
after giving it some test runs. And the legacy and redundant command
prereq should be removed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Log to stderr and add some ASCII-art around the output. Also, add a
--porcelain option to allow more stable output parsing. Subsequently,
use that option in make targets parsing the output, notably make diff
and make git-show-xxx.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a push to eventually merge the classes, somewhat align the
command-line API of CmdRequiredOsPkg to the one of
BaseCmdPkgRelations by using dependency flavours as mandatory, first
argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Move the dependencies listed in BASE_PKGS from projcts-dir.mk and
topdir.mk into project.conf.
Due to various hen-and-egg problems on a minimal system, in some
situations these packages can't be installed from project.conf. The
same is true with BASE_PKGS, however, so remove it, at least that
does away with some redundancy.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't prefix JW_PKG_PY_PROJECTS with time -p. A timing summary shows
up in too many places unexpectedly, e.g. in the context of the target
update-text-files. Add back later as more concrete demand comes up.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove echo realpath PROJECTS_MAKEFILE_NAME from the list-files
recipe. Misguided on more than one level.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove the now obsolete get-os.sh from jw-pkg. Use "jw-pkg.py distro
info" to get the information it used to provide.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a VCS-independent rule rule get-%, currently an alias for
git-get-%, with an empty recipe to keep GNU Make satisfied.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't use the master branch as target when getting code from
somebody, merge into the branch currently checked out.
Signed-off-by: Jan Lindemann <jan@janware.com>
"clone" in the Git sense means to copy a remote project over from
scratch. pgit.sh clone has come from that, but has since evolved into
something different, a mixture of clone, pull and fetch, so find a
different name. "get" seems generic enough and doesn't clash with a
Git meaning. Adapt variable names accordingly across the project.
Signed-off-by: Jan Lindemann <jan@janware.com>
To be explicit about what's happening during make fetch, default the
fetch refspec to $(CLONE_FROM_USER):current-branch:current-branch.
Signed-off-by: Jan Lindemann <jan@janware.com>
Purging the environment of pkg-release-reinstall did a tad too much:
Add SSH_AUTH_SOCK back to allow SSH agent-based authentication to go
through.
Signed-off-by: Jan Lindemann <jan@janware.com>
--source-profile=replace should be passed to all
jw-pkg.py projects build pkg-*install
invocations. Those invocations typically happen in the context of
pkg-%install, so add that target, specializing the pkg-% target.
The problem this solves is that /etc/profile is currently read only
once before bootstrapping all software on a pristine system is
started. This might lead to the situation that package A has
installed environment variable definitions into /etc/profile.d,
package B needs them for building, but never gets to read them.
Signed-off-by: Jan Lindemann <jan@janware.com>
Multiple variables are redundantly defined both for a project and for
the multiple-projects toplevel directory. Add a place to maintain
them centrally, and add PGIT_SH as a first variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
The target pkg-delete-ours, invoked from the projects directory,
should wipe all packages from the system which have been created and
installed via jw-pkg.
Currently they are selected via url =~ janware. This is a default
string which can be overridden by redefining JANWARE_PACKAGE_FILTER.
This might not be the most generic name, but is kind of consistent
and will be matched once all variables get renamed to a more generic
naming scheme.
This currently does not get all packages: Some are not labeled with
URLs matching "janware", because jw-pkg is only used as a convenient
way to package other people's open source projects.
Signed-off-by: Jan Lindemann <jan@janware.com>
There's pkg-manager-refresh already, so by adding pkg-manager-dup the
distribution can be upgraded by distribution agnostic targets only
through the Makefile. This might come in handy for CI, so add it.
Signed-off-by: Jan Lindemann <jan@janware.com>
--quote puts double quotation marks around the listed dependencies,
protecting version requirements (>= 1.0) and parenthesis "perl(GD)"
from the shell.
Signed-off-by: Jan Lindemann <jan@janware.com>
Retire pkg-manager.sh and replace it by the cleaner "jw-pkg.sh
distro" command, essentially providing the same functionality and
nearly the same command-line interface.
Not-so-fun-fact:
jw-pkg > git diff --stat jw-devops/master
...
71 files changed, 732 insertions(+), 340 deletions(-)
400 LOC more. That's what the move from a shell script to the more
maintainable Python versions costs. Still a good idea, and the
enhanced extensibility might pay off in terms of LOC with other shell
scripts in the future.
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-projects.py is now a multi-call executable, with "projects" being
just one of its subcommands. Rename it to jw-pkg.py to reflect that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Reorganize the Python module structure. Placing the command classes
under jw.cmds.projects instead of jw.build.cmds will allow to add a
nested command structure, with the current commands, being mostly
related to building software, found below a "projects" toplevel
command.
Other conceivable commands could be "package" for packaging, or
"distro" for commands wrapping the distribution's package manager.
Signed-off-by: Jan Lindemann <jan@janware.com>
Define Q ?= @, and replace @<command> in recipes by $(Q)<command>.
Meant to be overridden from the environment for debugging as in
Q= make
Signed-off-by: Jan Lindemann <jan@janware.com>
Kali Linux' default installation doesn't have /usr/bin/time which
brings out a but: $(TIME) doesn't expand to nothing but to -p, which
fails miserably, of course. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
PGIT_SH gets added --remote-base, but too late to make it into the
non-recursive variable PGIT_SH_CLONE. This leads to --remote-base
lacking from the clone invocation, and anonymous Git over HTTP
failing because it tries to clone via SSH. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
make git-show-pushable-master-branches output too litte for two
reasons: 1. grep -q returns zero also if no matches are found, and 2.
PROJECTS doesn't contain all relevant projects. BUILD_PROJECTS is
more meaningful.
Signed-off-by: Jan Lindemann <jan@janware.com>
--create-remote-user-repos had been disabled in
4053451bfd on the grounds that it's
hard to test and possibly superflous. It actually is not superfluous,
as devops builds show, and that's a valid test-case, so re-enable it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Support option --vcs. CVS is retired, but worked well as a test case
for mixing multiple version-control systems in one tree.
purge-stale-projects.sh is still pretty ugly and will have to go, but
its API might still serve as a working template.
Signed-off-by: Jan Lindemann <jan@janware.com>
