This is the first in a series of attempts to clean up jw-pkg's makefile
fragments make/*.mk. It collects redundantly defined workspace and project
variables into one shared common.mk.
Variables needed in both the workspace directory and the project
directories are defined separately in the two contexts: Q in defs.mk and
projects-dir-minimal.mk, INTERACTIVE in topdir.mk and projects-dir.mk,
BROWSER and JWB_SCRIPT_DIR in defs.mk/platform.mk and projects-dir.mk, and
LIST_VCS_FILES and LIST_VCS_FILES_SH wrapping the same scm.sh ls-files call
under different names. PKG_MANAGER is defined in topdir.mk and
projects-dir.mk, and JW_PKG_PY_PROJECTS is a prefix only used to build two
other variables in projects-dir.mk.
This commit adds make/common.mk, included from defs.mk (and hence from
every project directory) and from projects-dir.mk (the workspace), and
collects the shared variables there:
- Q, INTERACTIVE, BROWSER, JWB_SCRIPT_DIR: one ?= definition each
- LIST_VCS_FILES: one definition; LIST_VCS_FILES_SH is dropped and the
list-files target uses the shared name
- PKG_MANAGER moves to projects+project.mk, the fragment shared by the
workspace and the topdir
- projects-dir-minimal.mk keeps its own Q, because the bootstrap recipes
run before common.mk is parsed
- JW_PKG_PY_PROJECTS is inlined into JW_PKG_PY_BUILD and
JW_PKG_PY_REQUIRED_OS_PKG
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.0
Add recommended projects into the build closure. Would be nice if they got
built along. Let's see if nightly survives it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add and use JW_PKG_PY_REQUIRED_OS_PKG. The variable and its use effectively
ties more targets to act on the same package set as the make all, influenced
by the contents of $(PKG_RELATIONS_BUILD):
echo-build-deps
echo-install-deps
echo-release-deps
pkg-install-build-deps
pkg-install-release-deps
Signed-off-by: Jan Lindemann <jan@janware.com>
DEP_PROJECTS is computed with pkg-requires --recursive, which follows
[pkg.requires.jw] only. That set does not match what the build actually
builds, and it omits seed projects that no other project of the closure
depends on.
Derive DEP_PROJECTS from projects build --build-order with the same
dependency flavours as make all, the very command the build-order make
target and the workspace mount closure (mount-projects-dir.sh) use.
DEP_PROJECTS is now exactly the build closure, so the file listings, the
text-files cache and the git helpers of a workspace cover the same project
set as make all.
Introduce the PKG_RELATIONS_BUILD variable, defaulting to requires, and
pass it through JW_PKG_PY_BUILD to every build invocation of the workspace,
so the relations of the build closure are a workspace property.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.86.1
The top-level all, test, check, check-pre, and check-post targets invoke
the build tool without --dep-flavours, so it falls back to the auto default
and only resolves dependencies of the flavour "build". The build-order-%
target, by contrast, already passes the run,build,test,release set via a
single BUILD_ORDER_DEP_FLAVOURS variable. Planning and building therefore
rely on different dependency flavours, and none of the targets can be tuned
individually.
Introduce one flavour variable per target group, all of them defaulting to
the run,build,test,release set:
- BUILD_DEP_FLAVOURS for all
- BUILD_DEP_FLAVOURS_LINT for check, check-pre, and check-post
- BUILD_DEP_FLAVOURS_TEST for test
- BUILD_DEP_FLAVOURS_ORDER, renamed from BUILD_ORDER_DEP_FLAVOURS, used by
build-order-%
Split the combined all test check check-pre check-post rule into three
rules so that each group passes its own --dep-flavours to the build tool.
all, test, and the check targets now also resolve run, test, and release
dependencies, as build-order already does, and each target's flavour set
can be overridden independently.
Note that they all default the whole shebang: run,build,test,release. And
this opens up the door to dependency cycles that don't exist with the
subset of projects currently used. It's still better to ask for all and
then cut down as needed instead of starting small.
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
The git targets of the workspace Makefile cover push, diff, and status, but
not history: inspecting commits means running git log by hand in a project
directory, and no target shows the history of all projects of a tree at
once.
Add git-log targets at both levels; the target suffix is passed to git log
as-is, so make git-log-5 asks for the last five commits.
- git-log-% in projects-dir.mk runs pgit.sh log over every project of the
tree, one section per project
- git-log-% in rules.mk runs git log -$* in the current project directory
- git-log-tree-% in rules.mk runs git-log-$* in $(PROJECTS_DIR), giving the
whole-tree view from inside a project; it prints a notice if PROJECTS_DIR
is missing
Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
JWB_SCRIPT_DIR bypasses $(JWBDIR) for its scripts candidate, re-assembling
./$(JWBDIR_NAME)/scripts from the bare name. That works, but is redundant
resolution machinery which should be removed per se.
Resolve the scripts directory from $(JWBDIR) instead. Aside from
simplifying, that fixes the resolution when projects-dir.mk is included
from Makefiles other than projects-dir-miminimal.mk, and complements work
to commit f3f0ef10ca which also aims at allowing to run make from a
bind-mounted dependency closure of repositories.
Signed-off-by: Jan Lindemann <jan@janware.com>
Renovate the build-order and build-order-% targets as follows:
- Add BUILD_ORDER_DEP_FLAVOURS and default to all currently supported
build flavours: run,build,test,release. This adds convenience for
packages declaring differentiated types of dependencies, not only the
default build dependency, but still want any project it uses somehow
be be built before them.
- Don't print the recipe's command when running "make build-order", it
gets in the way if the caller wants to parse the result, so spare him
filtering it.
- Only log messages with priority warning or higher. Messages go to stderr
anyway to keep them from throwing wrenches into consuming parsers, but
they are likely confusingly visible without context if they leak to the
calling program's console.
Signed-off-by: Jan Lindemann <jan@janware.com>
The diff-all and diff targets diff all projects in the workspace
without filtering.
Add a target "diff-projects". It sets PGIT_SH_PROJECTS to the list of
projects from build-order, limiting the diff to the dependency
closure around $(PROJECTS).
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for the targets "check" and "check-post" to
projects-dir.mk to make them usable from inside the projects
directory.
Note that "check-pre" is intentionally left out: Running "make check"
in a project before its prerequisite projects have built their
__init__.py files will fail due to broken import resolution.
Signed-off-by: Jan Lindemann <jan@janware.com>
The clean-dirs target does not only clean the repos present in
PROJECTS, but all repos it finds to be dirty, and clean-all-dirs does
the opposite. I suppose that was an oversight, swap their recipes.
Moreover, cleaning all directories goes about its business in an
overly complicated and unecessarily time-consuming way, fix that,
too.
Signed-off-by: Jan Lindemann <jan@janware.com>
By the time projects-dir.mk is used during onboarding, it's already
cloned, and so is jw-pkg in all its glory. So better use a
ssh-wrapper.sh directly under jw-pkg's version control instead of
plainly generating one with echo some-script-logic > ssh-wrapper.sh.
This has the main benefit of allowing a more elaborate script. The
one added by this commit removes "-l user" from remotes which have a
standard-user@gitserver form, typically because they differentiate
users via their SSH pubkeys only, and which would deny access if both
-l user and standard-user@ were specified.
ssh-wrapper.sh still needs to be a target which is updated by a
recipe, because the version found in jw-pkg can't be trusted to be
executable during bootstrapping, because "make all" has not run, yet.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add get-pub for top directory / projects directory level. This is a
janware specific target needed for CI. It integrates the current
master branch from the new Forgejo-based Git repos.
That said, this will likely need to go along with other release
machinery. Packaging is good, but releasing over a bunch of
directories is an intricate process, as-is only usable by janware
itself, and doesn't need to be part of jw-pkg.
Signed-off-by: Jan Lindemann <jan@janware.com>
janware upstream development server moved from git.janware.com to
devgit.janware.com. This commit follows the move with pretty much a
simple
s/git.janware.com/devgit.janware.com/
over jw-pkg. It found 14 matches, that's pretty bad.
FIXME: Reduce the redundancy, or, better, replace the
devgit.janware.com goodies by a more generally useful concept
altogether.
Signed-off-by: Jan Lindemann <jan@janware.com>
To make it easier to force-push feature branches, allow that, albeit
under --force-with-lease protection.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a new target git-show-ahead-of-master, which does what
git-show-pushable did up to now: List all repos where the currently
checked-out branch is ahead of origin/master.
git-show-pushable now shows repos that have their checked-out branch
ahead of their respective upstream branches. Which is truer to what
the target's name suggests.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename command "distro" to "pkg" together with "info", its last
remaining subcommand. "distro" is often used in the sense of "Linux
distribution", which would be too narrow for the targets jw-pkg could
theoretically support.
Signed-off-by: Jan Lindemann <jan@janware.com>
With the exception of the "info" subcommand, nearly all of distro's
subcommands deal with package managing, so push them into their own
command category.
Signed-off-by: Jan Lindemann <jan@janware.com>
pgit.sh get is not invoked with --create-remote-user-repos by
default. For the janware Git servers, this makes CI fail over new
repos: Repos from the maintainers are pulled, but before
pkg-rebuild-reinstall, repos that the devops user has no remote repo
for are purged again and are subseqently missing during build.
Passing --create-remote-user-repos to all pgit.sh get invocations
should fix the problem, so do that.
Signed-off-by: Jan Lindemann <jan@janware.com>
/usr/bin/file <candidate> | grep text is used to detect if a file is
a text file or not. Replace that with grep -I., because that adds
some files left out by /usr/bin/file, notably systemd service files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename git-show-pushable-master-branches to git-show-pushable,
because, the target's recipe doesn't show pushable master branches,
but all commits in the working directories current branch not present
in origin/master.
Signed-off-by: Jan Lindemann <jan@janware.com>
This reverts commit 88e81 and effectively re-enables persistent SSH
connections to speed up builds.
commit 88e8197ed7
Author: Jan Lindemann <jan@janware.com>
Date: Thu Apr 20 20:00:46 2017 +0000
projects-dir.mk: Partly disable persistent SSH
Don't use persistent SSH-connections any more for "pkg-" targets,
because, sadly, this hangs after uploading a package.
This commit puts some safeguards against hanging SSH into place,
namely setting default SSH timeouts down, SSH keepalive, setting SSH
BatchMode to yes.
Signed-off-by: Jan Lindemann <jan@janware.com>
Use DEP_PROJECTS instead of BUILD_PROJECTS everywhere. DEP_PROJECTS
considers more projects, and for all targets it is relevant to, it's
desirable to have them operate on the maximum blast radius.
Signed-off-by: Jan Lindemann <jan@janware.com>
make git-show-pushable-master-branches misses projects which should
better be pulled in as dependencies. It searches $(BUILLD_PROJECTS)
for repositories with pushable commits, that variable leaves some
out, and DEP_PROJECTS has them. It make use of the renovated
pkg-requires command.
This move should be extended to other uses of BUILD_PROJECTS as well,
after giving it some test runs. And the legacy and redundant command
prereq should be removed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Log to stderr and add some ASCII-art around the output. Also, add a
--porcelain option to allow more stable output parsing. Subsequently,
use that option in make targets parsing the output, notably make diff
and make git-show-xxx.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a push to eventually merge the classes, somewhat align the
command-line API of CmdRequiredOsPkg to the one of
BaseCmdPkgRelations by using dependency flavours as mandatory, first
argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
Move the dependencies listed in BASE_PKGS from projcts-dir.mk and
topdir.mk into project.conf.
Due to various hen-and-egg problems on a minimal system, in some
situations these packages can't be installed from project.conf. The
same is true with BASE_PKGS, however, so remove it, at least that
does away with some redundancy.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't prefix JW_PKG_PY_PROJECTS with time -p. A timing summary shows
up in too many places unexpectedly, e.g. in the context of the target
update-text-files. Add back later as more concrete demand comes up.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove echo realpath PROJECTS_MAKEFILE_NAME from the list-files
recipe. Misguided on more than one level.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove the now obsolete get-os.sh from jw-pkg. Use "jw-pkg.py distro
info" to get the information it used to provide.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a VCS-independent rule rule get-%, currently an alias for
git-get-%, with an empty recipe to keep GNU Make satisfied.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't use the master branch as target when getting code from
somebody, merge into the branch currently checked out.
Signed-off-by: Jan Lindemann <jan@janware.com>
"clone" in the Git sense means to copy a remote project over from
scratch. pgit.sh clone has come from that, but has since evolved into
something different, a mixture of clone, pull and fetch, so find a
different name. "get" seems generic enough and doesn't clash with a
Git meaning. Adapt variable names accordingly across the project.
Signed-off-by: Jan Lindemann <jan@janware.com>
To be explicit about what's happening during make fetch, default the
fetch refspec to $(CLONE_FROM_USER):current-branch:current-branch.
Signed-off-by: Jan Lindemann <jan@janware.com>
Purging the environment of pkg-release-reinstall did a tad too much:
Add SSH_AUTH_SOCK back to allow SSH agent-based authentication to go
through.
Signed-off-by: Jan Lindemann <jan@janware.com>
--source-profile=replace should be passed to all
jw-pkg.py projects build pkg-*install
invocations. Those invocations typically happen in the context of
pkg-%install, so add that target, specializing the pkg-% target.
The problem this solves is that /etc/profile is currently read only
once before bootstrapping all software on a pristine system is
started. This might lead to the situation that package A has
installed environment variable definitions into /etc/profile.d,
package B needs them for building, but never gets to read them.
Signed-off-by: Jan Lindemann <jan@janware.com>
Multiple variables are redundantly defined both for a project and for
the multiple-projects toplevel directory. Add a place to maintain
them centrally, and add PGIT_SH as a first variable.
Signed-off-by: Jan Lindemann <jan@janware.com>
The target pkg-delete-ours, invoked from the projects directory,
should wipe all packages from the system which have been created and
installed via jw-pkg.
Currently they are selected via url =~ janware. This is a default
string which can be overridden by redefining JANWARE_PACKAGE_FILTER.
This might not be the most generic name, but is kind of consistent
and will be matched once all variables get renamed to a more generic
naming scheme.
This currently does not get all packages: Some are not labeled with
URLs matching "janware", because jw-pkg is only used as a convenient
way to package other people's open source projects.
Signed-off-by: Jan Lindemann <jan@janware.com>
There's pkg-manager-refresh already, so by adding pkg-manager-dup the
distribution can be upgraded by distribution agnostic targets only
through the Makefile. This might come in handy for CI, so add it.
Signed-off-by: Jan Lindemann <jan@janware.com>
--quote puts double quotation marks around the listed dependencies,
protecting version requirements (>= 1.0) and parenthesis "perl(GD)"
from the shell.
Signed-off-by: Jan Lindemann <jan@janware.com>
Retire pkg-manager.sh and replace it by the cleaner "jw-pkg.sh
distro" command, essentially providing the same functionality and
nearly the same command-line interface.
Not-so-fun-fact:
jw-pkg > git diff --stat jw-devops/master
...
71 files changed, 732 insertions(+), 340 deletions(-)
400 LOC more. That's what the move from a shell script to the more
maintainable Python versions costs. Still a good idea, and the
enhanced extensibility might pay off in terms of LOC with other shell
scripts in the future.
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-projects.py is now a multi-call executable, with "projects" being
just one of its subcommands. Rename it to jw-pkg.py to reflect that.
Signed-off-by: Jan Lindemann <jan@janware.com>
