Rename git-show-pushable-master-branches to git-show-pushable,
because, the target's recipe doesn't show pushable master branches,
but all commits in the working directories current branch not present
in origin/master.
Signed-off-by: Jan Lindemann <jan@janware.com>
This reverts commit 88e81 and effectively re-enables persistent SSH
connections to speed up builds.
commit 88e8197ed7
Author: Jan Lindemann <jan@janware.com>
Date: Thu Apr 20 20:00:46 2017 +0000
projects-dir.mk: Partly disable persistent SSH
Don't use persistent SSH-connections any more for "pkg-" targets,
because, sadly, this hangs after uploading a package.
This commit puts some safeguards against hanging SSH into place,
namely setting default SSH timeouts down, SSH keepalive, setting SSH
BatchMode to yes.
Signed-off-by: Jan Lindemann <jan@janware.com>
Use DEP_PROJECTS instead of BUILD_PROJECTS everywhere. DEP_PROJECTS
considers more projects, and for all targets it is relevant to, it's
desirable to have them operate on the maximum blast radius.
Signed-off-by: Jan Lindemann <jan@janware.com>
make git-show-pushable-master-branches misses projects which should
better be pulled in as dependencies. It searches $(BUILLD_PROJECTS)
for repositories with pushable commits, that variable leaves some
out, and DEP_PROJECTS has them. It make use of the renovated
pkg-requires command.
This move should be extended to other uses of BUILD_PROJECTS as well,
after giving it some test runs. And the legacy and redundant command
prereq should be removed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Log to stderr and add some ASCII-art around the output. Also, add a
--porcelain option to allow more stable output parsing. Subsequently,
use that option in make targets parsing the output, notably make diff
and make git-show-xxx.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a push to eventually merge the classes, somewhat align the
command-line API of CmdRequiredOsPkg to the one of
BaseCmdPkgRelations by using dependency flavours as mandatory, first
argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Move the dependencies listed in BASE_PKGS from projcts-dir.mk and
topdir.mk into project.conf.
Due to various hen-and-egg problems on a minimal system, in some
situations these packages can't be installed from project.conf. The
same is true with BASE_PKGS, however, so remove it, at least that
does away with some redundancy.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't prefix JW_PKG_PY_PROJECTS with time -p. A timing summary shows
up in too many places unexpectedly, e.g. in the context of the target
update-text-files. Add back later as more concrete demand comes up.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove echo realpath PROJECTS_MAKEFILE_NAME from the list-files
recipe. Misguided on more than one level.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove the now obsolete get-os.sh from jw-pkg. Use "jw-pkg.py distro
info" to get the information it used to provide.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a VCS-independent rule rule get-%, currently an alias for
git-get-%, with an empty recipe to keep GNU Make satisfied.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't use the master branch as target when getting code from
somebody, merge into the branch currently checked out.
Signed-off-by: Jan Lindemann <jan@janware.com>
"clone" in the Git sense means to copy a remote project over from
scratch. pgit.sh clone has come from that, but has since evolved into
something different, a mixture of clone, pull and fetch, so find a
different name. "get" seems generic enough and doesn't clash with a
Git meaning. Adapt variable names accordingly across the project.
Signed-off-by: Jan Lindemann <jan@janware.com>
To be explicit about what's happening during make fetch, default the
fetch refspec to $(CLONE_FROM_USER):current-branch:current-branch.
Signed-off-by: Jan Lindemann <jan@janware.com>
Purging the environment of pkg-release-reinstall did a tad too much:
Add SSH_AUTH_SOCK back to allow SSH agent-based authentication to go
through.
Signed-off-by: Jan Lindemann <jan@janware.com>
--source-profile=replace should be passed to all
jw-pkg.py projects build pkg-*install
invocations. Those invocations typically happen in the context of
pkg-%install, so add that target, specializing the pkg-% target.
The problem this solves is that /etc/profile is currently read only
once before bootstrapping all software on a pristine system is
started. This might lead to the situation that package A has
installed environment variable definitions into /etc/profile.d,
package B needs them for building, but never gets to read them.
Signed-off-by: Jan Lindemann <jan@janware.com>
Multiple variables are redundantly defined both for a project and for
the multiple-projects toplevel directory. Add a place to maintain
them centrally, and add PGIT_SH as a first variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
The target pkg-delete-ours, invoked from the projects directory,
should wipe all packages from the system which have been created and
installed via jw-pkg.
Currently they are selected via url =~ janware. This is a default
string which can be overridden by redefining JANWARE_PACKAGE_FILTER.
This might not be the most generic name, but is kind of consistent
and will be matched once all variables get renamed to a more generic
naming scheme.
This currently does not get all packages: Some are not labeled with
URLs matching "janware", because jw-pkg is only used as a convenient
way to package other people's open source projects.
Signed-off-by: Jan Lindemann <jan@janware.com>
There's pkg-manager-refresh already, so by adding pkg-manager-dup the
distribution can be upgraded by distribution agnostic targets only
through the Makefile. This might come in handy for CI, so add it.
Signed-off-by: Jan Lindemann <jan@janware.com>
--quote puts double quotation marks around the listed dependencies,
protecting version requirements (>= 1.0) and parenthesis "perl(GD)"
from the shell.
Signed-off-by: Jan Lindemann <jan@janware.com>
Retire pkg-manager.sh and replace it by the cleaner "jw-pkg.sh
distro" command, essentially providing the same functionality and
nearly the same command-line interface.
Not-so-fun-fact:
jw-pkg > git diff --stat jw-devops/master
...
71 files changed, 732 insertions(+), 340 deletions(-)
400 LOC more. That's what the move from a shell script to the more
maintainable Python versions costs. Still a good idea, and the
enhanced extensibility might pay off in terms of LOC with other shell
scripts in the future.
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-projects.py is now a multi-call executable, with "projects" being
just one of its subcommands. Rename it to jw-pkg.py to reflect that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Reorganize the Python module structure. Placing the command classes
under jw.cmds.projects instead of jw.build.cmds will allow to add a
nested command structure, with the current commands, being mostly
related to building software, found below a "projects" toplevel
command.
Other conceivable commands could be "package" for packaging, or
"distro" for commands wrapping the distribution's package manager.
Signed-off-by: Jan Lindemann <jan@janware.com>
Define Q ?= @, and replace @<command> in recipes by $(Q)<command>.
Meant to be overridden from the environment for debugging as in
Q= make
Signed-off-by: Jan Lindemann <jan@janware.com>
Kali Linux' default installation doesn't have /usr/bin/time which
brings out a but: $(TIME) doesn't expand to nothing but to -p, which
fails miserably, of course. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
PGIT_SH gets added --remote-base, but too late to make it into the
non-recursive variable PGIT_SH_CLONE. This leads to --remote-base
lacking from the clone invocation, and anonymous Git over HTTP
failing because it tries to clone via SSH. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
make git-show-pushable-master-branches output too litte for two
reasons: 1. grep -q returns zero also if no matches are found, and 2.
PROJECTS doesn't contain all relevant projects. BUILD_PROJECTS is
more meaningful.
Signed-off-by: Jan Lindemann <jan@janware.com>
--create-remote-user-repos had been disabled in
4053451bfd on the grounds that it's
hard to test and possibly superflous. It actually is not superfluous,
as devops builds show, and that's a valid test-case, so re-enable it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Support option --vcs. CVS is retired, but worked well as a test case
for mixing multiple version-control systems in one tree.
purge-stale-projects.sh is still pretty ugly and will have to go, but
its API might still serve as a working template.
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-build doesn't stop at building software, packaging it afterwards
is also a core feature, so this commit gives the package a better
name.
The commit replaces strings s/jw-build/jw-pkg/ in text files and file
names. Fallout to the functionality is fixed, variable names are left
as they are, though. To be adjusted by later commits.
Signed-off-by: Jan Lindemann <jan@janware.com>
In the move away from environment variables, replace JANWARE_USER
support in pgit.sh by the --login option.
Signed-off-by: Jan Lindemann <jan@janware.com>
In the attempt to move away from communicating options via
environment variables from one part of jw-build software to another,
replace PGIT_CLONE_FROM_USER with the clearer --refspec option. Which
is also more versatile.
Signed-off-by: Jan Lindemann <jan@janware.com>
Creating user repositories is hard to test. Not sure if the concept
will survive the current workflow changes, disable it for now.
Signed-off-by: Jan Lindemann <jan@janware.com>
JW_BUILD_SSH_EXTRA_OPTS contains a -l $(JANWARE_USER), which make ssh
break if $(JANWARE_USER) is empty. Fix that case.
Signed-off-by: Jan Lindemann <jan@janware.com>
The GIT_ASKPASS environment variable can point to a relative path,
and since pgit.sh currently does a chdir, it needs to be made an
absolute path.
Signed-off-by: Jan Lindemann <jan@janware.com>
Currently, the primary discriminating criterion on how to handle a
set of remote repositories is whether or not JANWARE_USER is defined.
The canonical way to do that is PROJECTS_DIR_REMOTE_BASE, though, so
go from that definition.
Signed-off-by: Jan Lindemann <jan@janware.com>
Group variables related with pgit.sh (PGIT_SH and PGIT_SH_CLONE)
closer together. Define CLONE_FROM_USER early on.
Signed-off-by: Jan Lindemann <jan@janware.com>
The usage comments heading projects-dir-minimal.mk and
projects-dir.mk state that for cloning all repositories, JANWARE_USER
needs to be defined. That restriction is now gone, so reflect that in
the comment.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the variable PROJECTS_DIR_REMOTE_BASE, defaulting to
ssh://git.janware.com/srv/git if REMOTE_USER is defined, and to
https://janware.com/code in case it isn't.
Signed-off-by: Jan Lindemann <jan@janware.com>
