"clone" in the Git sense means to copy a remote project over from scratch. pgit.sh clone has come from that, but has since evolved into something different, a mixture of clone, pull and fetch, so find a different name. "get" seems generic enough and doesn't clash with a Git meaning. Adapt variable names accordingly across the project.
git-pull-<username> doesn't use pgit.sh if username == login. pgit.sh should handle that case fine now, so remove the distinction from topdir.mk and make it in one place, i.e. pgit.sh. This has the additional advantage that pull as done by pgit.sh conveniently uses --autostash.
sudo is certainly not needed for the run package (which in itself is hardly useful at all), so move that dependency into the devel package. Same for gawk. /opt/jw-pkg/bin/get-os.sh depends on it, but I don't see where else but in a -devel context that would matter. And if it breaks something, it is going to be an easy fix without awk.
invocations. Those invocations typically happen in the context of pkg-%install, so add that target, specializing the pkg-% target.
The problem this solves is that /etc/profile is currently read only once before bootstrapping all software on a pristine system is started. This might lead to the situation that package A has installed environment variable definitions into /etc/profile.d, package B needs them for building, but never gets to read them.
To complement git-pull-maintainer with something more generic, also suitable for other SCMs, add the target pull-maintainer and make pkg-release-reinstall depend on it. Currently only visible in the context of pkg-% targets, scope might be expanded if need be.
Use pgit.sh to for the git-pull-% target. This should make git-pull-maintainer work. To limit the blast radius for now, only use it if the source user differs from the invoking user.
Multiple variables are redundantly defined both for a project and for the multiple-projects toplevel directory. Add a place to maintain them centrally, and add PGIT_SH as a first variable.
Add an init target. Use it if you want to tell the Makefile: _Just_ initalize the build machinery and nothing else, don't pull and build everything else you can. Not strictly necessary, most of the time pulling everything is what's wanted, and that does the init anyway.
The target pkg-delete-ours, invoked from the projects directory, should wipe all packages from the system which have been created and installed via jw-pkg.
Currently they are selected via url =~ janware. This is a default string which can be overridden by redefining JANWARE_PACKAGE_FILTER. This might not be the most generic name, but is kind of consistent and will be matched once all variables get renamed to a more generic naming scheme.
This currently does not get all packages: Some are not labeled with URLs matching "janware", because jw-pkg is only used as a convenient way to package other people's open source projects.
If VERSION_FILE is not found, trying to include makefile snippets triggers the attempt to regenerate it. This happened for installed Makefiles of jw-docker-images: It defines TOPDIR to /opt/jw-docker-images, projects.mk looks there, but the version file is installed under /usr/share/doc/packages/jw-docker-images/VERSION.
Allow VERSION_FILE to be overridden including code to keep that from happening.
make complains for plugins that soandso.dll has not been remade. The problem is that it tries to remake all targets of a multi-target. Split that up into two rules.
.cache-projects.mk is not installed / packaged, which makes builds against an installed jw-pkg considerably slower. Change that, at the risk of making the installed jw-pkg-devel less versatile. This commit installs a cache file cache-projects.mk, renamed from .cache-projects.mk, because there's no justification for hiding an installed makefile. At least I can't think of one.
There's pkg-manager-refresh already, so by adding pkg-manager-dup the distribution can be upgraded by distribution agnostic targets only through the Makefile. This might come in handy for CI, so add it.
--quote puts double quotation marks around the listed dependencies, protecting version requirements (>= 1.0) and parenthesis "perl(GD)" from the shell.
Retire pkg-manager.sh and replace it by the cleaner "jw-pkg.sh distro" command, essentially providing the same functionality and nearly the same command-line interface.
400 LOC more. That's what the move from a shell script to the more maintainable Python versions costs. Still a good idea, and the enhanced extensibility might pay off in terms of LOC with other shell scripts in the future.
As per info make, it turns out that ifndef SOME_VAR is true for SOME_VAR defined to an empty value. This is unusable for caching, so replace it with ifeq ($(origin SOME_VAR),undefined).
Add support for --topdir-format. The option supports several different values, affecting the console output of App wherever it knows that the output contains a reference to the projects' toplevel directory.
- "unaltered" will have it print the toplevel directory in the same
format as passed to the commandline
- "absolute" will try to resolve it to an absolute path before
printing
- make:XXX will return the make-varible $(XXX) instead
To implement this, the proj_dir() member function is turned into the private member function __proj_dir(), and a new member function find_dir() is supplied, with two additional parameters: search_subdirs and search_absdirs, which will try to find an existing directory relative to the toplevel directory of the given module, or in the search_absdirs list, respectively.
Command modules in cmds.projects have been updated to use the new function.
Reorganize the Python module structure. Placing the command classes under jw.cmds.projects instead of jw.build.cmds will allow to add a nested command structure, with the current commands, being mostly related to building software, found below a "projects" toplevel command.
Other conceivable commands could be "package" for packaging, or "distro" for commands wrapping the distribution's package manager.
Remove PY_PREREQ_BUILD and PY_PREREQ_BUILD_DIRS from py-defs.mk: Apparently they're not used anywhere, and are costly in terms of directory startup time.
This commit aims at improving speed by using better caching.
- Makefile, cache.mk: Split .cache.mk up
To allow caching of runtime path variables which are
project-specific, split .cache.mk up in .cache-project.mk and
.cache-projects.mk
- ldlibpath.mk: Cache ldlibpath, exepath and pythonpath
Place the output of $(call proj_query ldlibpath), $(call
proj_query, exepath) and $(call proj_query pythonpath) in
JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH
respectively, and cache the variables in make/.project-cache.mk.
- cache.mk: Use = instead of :=
Recursively expanded variables are nearly as fast as := variables
if the assigned value is a fixed string. And sometimes it's not,
rightly so, because variables get assigned below, as with
JW_PKG_XXX for instance.
- cache.mk: Use $(TOPDIR) as variable values
Replace absolute references to project's topdir by $(TOPDIR) with
sed. As soon as the project queries produce absolute paths, they
will be transformed into relative paths which allow the code base
to be moved to a different location and still remain functional
without a rebuild.
Remove unused code, and code which actually does something: CACHED_VARS looks as if it's sufficently and more centrally defined in make.mk, don't override that.
Add std-targets.mk, meant to be included mostly to make sure all mandatory targets are there. On clean and distclean, it also removes stuff that should not be there anymore after clean.
Change $(check_scm_sync) from "make git pull" to "make git-pull-maintainer", which most notably should delegate devops builds to the maintainer defined in project.conf.
There's little system in the pkg-install-xxx targets, add one more to increase the confusion. It's needed to install all packages needed to do a standalone build against the packages installed into the system via package manager. That said, the respective jw-projects.sh commands need broader refactoring, as well as the pkg-install-xxx target naming.
Make jw-projects.py list-repos support a local directory as base URL of all git repositories, notably used by PROJECTS_DIR_REMOTE_BASE, which can now point to a local directory.
Targets defined by projects-dir.mk are not available before it is included, but make makes up its mind about what targets are available after parsing the included makefiles, so remove that redundancy.
On the other hand, a dependency alone is not enough for make to understand that an included makefile has been remade, it needs a rule, so add a dummy-rule body. In this case only echoing that the include file has been provided.
Kali Linux' default installation doesn't have /usr/bin/time which brings out a but: $(TIME) doesn't expand to nothing but to -p, which fails miserably, of course. Fix that.
PGIT_SH gets added --remote-base, but too late to make it into the non-recursive variable PGIT_SH_CLONE. This leads to --remote-base lacking from the clone invocation, and anonymous Git over HTTP failing because it tries to clone via SSH. Fix that.
make git-show-pushable-master-branches output too litte for two reasons: 1. grep -q returns zero also if no matches are found, and 2. PROJECTS doesn't contain all relevant projects. BUILD_PROJECTS is more meaningful.
For a project to supply templates, it needs to advertise their location. For this, the tmpl_dir make variable is added to projects.mk. If other-project wants to get hold of some-project's templates, it can do, e.g.:
Templates (i.e. text files ending as .tmpl) are not part of jw-pkg anylonger, but controlling the way they are installed is beneficial to other packages, so add tmpl.mk back.
That said, the variable names will need some tweaking to avoid collisions. Postponed.
