.cache-projects.mk is not installed / packaged, which makes builds against an installed jw-pkg considerably slower. Change that, at the risk of making the installed jw-pkg-devel less versatile. This commit installs a cache file cache-projects.mk, renamed from .cache-projects.mk, because there's no justification for hiding an installed makefile. At least I can't think of one.
There's pkg-manager-refresh already, so by adding pkg-manager-dup the distribution can be upgraded by distribution agnostic targets only through the Makefile. This might come in handy for CI, so add it.
--quote puts double quotation marks around the listed dependencies, protecting version requirements (>= 1.0) and parenthesis "perl(GD)" from the shell.
Retire pkg-manager.sh and replace it by the cleaner "jw-pkg.sh distro" command, essentially providing the same functionality and nearly the same command-line interface.
400 LOC more. That's what the move from a shell script to the more maintainable Python versions costs. Still a good idea, and the enhanced extensibility might pay off in terms of LOC with other shell scripts in the future.
As per info make, it turns out that ifndef SOME_VAR is true for SOME_VAR defined to an empty value. This is unusable for caching, so replace it with ifeq ($(origin SOME_VAR),undefined).
Add support for --topdir-format. The option supports several different values, affecting the console output of App wherever it knows that the output contains a reference to the projects' toplevel directory.
- "unaltered" will have it print the toplevel directory in the same
format as passed to the commandline
- "absolute" will try to resolve it to an absolute path before
printing
- make:XXX will return the make-varible $(XXX) instead
To implement this, the proj_dir() member function is turned into the private member function __proj_dir(), and a new member function find_dir() is supplied, with two additional parameters: search_subdirs and search_absdirs, which will try to find an existing directory relative to the toplevel directory of the given module, or in the search_absdirs list, respectively.
Command modules in cmds.projects have been updated to use the new function.
Reorganize the Python module structure. Placing the command classes under jw.cmds.projects instead of jw.build.cmds will allow to add a nested command structure, with the current commands, being mostly related to building software, found below a "projects" toplevel command.
Other conceivable commands could be "package" for packaging, or "distro" for commands wrapping the distribution's package manager.
Remove PY_PREREQ_BUILD and PY_PREREQ_BUILD_DIRS from py-defs.mk: Apparently they're not used anywhere, and are costly in terms of directory startup time.
This commit aims at improving speed by using better caching.
- Makefile, cache.mk: Split .cache.mk up
To allow caching of runtime path variables which are
project-specific, split .cache.mk up in .cache-project.mk and
.cache-projects.mk
- ldlibpath.mk: Cache ldlibpath, exepath and pythonpath
Place the output of $(call proj_query ldlibpath), $(call
proj_query, exepath) and $(call proj_query pythonpath) in
JW_PKG_LD_LIBRARY_PATH, JW_PKG_EXE_PATH, and JW_PKG_PYTHON_PATH
respectively, and cache the variables in make/.project-cache.mk.
- cache.mk: Use = instead of :=
Recursively expanded variables are nearly as fast as := variables
if the assigned value is a fixed string. And sometimes it's not,
rightly so, because variables get assigned below, as with
JW_PKG_XXX for instance.
- cache.mk: Use $(TOPDIR) as variable values
Replace absolute references to project's topdir by $(TOPDIR) with
sed. As soon as the project queries produce absolute paths, they
will be transformed into relative paths which allow the code base
to be moved to a different location and still remain functional
without a rebuild.
Remove unused code, and code which actually does something: CACHED_VARS looks as if it's sufficently and more centrally defined in make.mk, don't override that.
Add std-targets.mk, meant to be included mostly to make sure all mandatory targets are there. On clean and distclean, it also removes stuff that should not be there anymore after clean.
Change $(check_scm_sync) from "make git pull" to "make git-pull-maintainer", which most notably should delegate devops builds to the maintainer defined in project.conf.
There's little system in the pkg-install-xxx targets, add one more to increase the confusion. It's needed to install all packages needed to do a standalone build against the packages installed into the system via package manager. That said, the respective jw-projects.sh commands need broader refactoring, as well as the pkg-install-xxx target naming.
Make jw-projects.py list-repos support a local directory as base URL of all git repositories, notably used by PROJECTS_DIR_REMOTE_BASE, which can now point to a local directory.
Targets defined by projects-dir.mk are not available before it is included, but make makes up its mind about what targets are available after parsing the included makefiles, so remove that redundancy.
On the other hand, a dependency alone is not enough for make to understand that an included makefile has been remade, it needs a rule, so add a dummy-rule body. In this case only echoing that the include file has been provided.
Kali Linux' default installation doesn't have /usr/bin/time which brings out a but: $(TIME) doesn't expand to nothing but to -p, which fails miserably, of course. Fix that.
PGIT_SH gets added --remote-base, but too late to make it into the non-recursive variable PGIT_SH_CLONE. This leads to --remote-base lacking from the clone invocation, and anonymous Git over HTTP failing because it tries to clone via SSH. Fix that.
make git-show-pushable-master-branches output too litte for two reasons: 1. grep -q returns zero also if no matches are found, and 2. PROJECTS doesn't contain all relevant projects. BUILD_PROJECTS is more meaningful.
For a project to supply templates, it needs to advertise their location. For this, the tmpl_dir make variable is added to projects.mk. If other-project wants to get hold of some-project's templates, it can do, e.g.:
Templates (i.e. text files ending as .tmpl) are not part of jw-pkg anylonger, but controlling the way they are installed is beneficial to other packages, so add tmpl.mk back.
That said, the variable names will need some tweaking to avoid collisions. Postponed.
To avoid name collisions, rename svg.mk to the more specialized svg-to-pixmap.mk, because that's what it does. To the same end, rename $(SVG) to $(PIXMAP_TO_SVG_SRC_SVG).
--create-remote-user-repos had been disabled in 4053451bfd on the grounds that it's hard to test and possibly superflous. It actually is not superfluous, as devops builds show, and that's a valid test-case, so re-enable it.
Running "make install" from an arbitrary source directory currently by default either installs to a user-accessible ENV_PREFIX, or, if DEVELOPMENT is set to false, tries to install into the system's root filesystem, but fails over permission errors. This was by design: To now, I considered trying the latter ill-conceived, because installing without package manager control bears the risk of leaving unversioned files in the system.
Actually, thinking again, during development this looks like a valid use case: Having run pkg-rebuild-reinstall before, installing from a source directory will leave a trace in the package manager's hash check output, will be handled during the next clean install, and might be a useful shortcut for trying things in the root file system.
JS_MINIFY_FILTER_IN can be defined to nothing, in which case minifying breaks, so don't minify if there's no filter. As an additional benifit, defining it to the empty string in local.mk allows to use Vim's quickfix window for syntax errors, because there's no intermediate file created.
tailwind.mk is meant to generate a CSS file with tailwind classes from configuration files named *.css.tw or *.css.tw.tmpl. The latter flavour understands some make-style variables, as of now only $(TOPDIR).
Define DATA_DIR, the directory where read-only, non-executable and non-configurable resources should be stored. And define JSON_DIR as $(DATA_DIR)/json.
PACKAGE_VCS_FILES defaults to false. Defining it to true before including rpmdist.mk includes the version-control metadata files in the source packages.
jw-build doesn't stop at building software, packaging it afterwards is also a core feature, so this commit gives the package a better name.
The commit replaces strings s/jw-build/jw-pkg/ in text files and file names. Fallout to the functionality is fixed, variable names are left as they are, though. To be adjusted by later commits.
git-pull-% pulls whatever $(GIT_MAIN_BRANCH) happens to be from the remote jw-% into the current branch, with --rebase and --autostash.
git-pull-maintainer does the same with <maintainer>. <maintainer> is figured out from the configuration in projects.conf. If it's the invoking user, origin is used.
In the attempt to move away from communicating options via environment variables from one part of jw-build software to another, replace PGIT_CLONE_FROM_USER with the clearer --refspec option. Which is also more versatile.
Don't unconditionally add proactiveAuth=basic to Git's config during clone, but only if cloning happens after authentication.
This saves unauthenticated users funny password prompts. On the other hand, this makes a server setting persistent which could be changed on the server.
URL =~ /api/ (or so) => 401, followed by Basic Auth URL !=~ /api/ (or so) => Redirect or free access, depending on resource
Currently all resources, including API, are accessible by either basic auth or a Cookie, but basic auth needs to be present in the first request, which throws off some clients (notably Git without proactiveAuth=basic).
Currently, the primary discriminating criterion on how to handle a set of remote repositories is whether or not JANWARE_USER is defined. The canonical way to do that is PROJECTS_DIR_REMOTE_BASE, though, so go from that definition.
