JW-Build

Features

The JW-Build package is basically a bunch of scripts and GNU makefile snippets for organizing software builds and releases.

You can install it to its standard location /opt/jw-build, or put it anywhere you want in your file system, typically next to your own projects in the same directory. To organize their respective builds, you can then include JW-Build's makefile snippets from your own projects' makefiles, like so:

include $(JWBDIR)/make/cpp.mk

where JWBDIR needs to point to JW-Build's installation directory. In this example, the snippet cpp.mk would by default take all C++ files it finds in the directory from where its included, compile them, and and add them to a shared project library. It would also take all header files and copy them to a project include directory. js.mk would by default minify all JavaSript it finds, java.mk jar up .java files into classes and jar-files, and so on. JW-Build also handles installation and packaging of these files, to customizable locations with standardish defaults.

JW-Build is small, its tarball is about 200K. It's small enough to be shipped with the source code of your project, if you choose to do so. And it's small enough to be self-documenting. Well, okay, somewhat self-documenting. You have to know GNU Makefile syntax to understand what it does, and dig into its somtimes arcane code, ideally with a working example. You can install it with your distribution's package manager, or you can keep it within your code versioning system, alongside your own code. It's also designed to be the lightest possible touch on any given source code package, in terms of code you need to add to a given package that should be built with it, and also in terms of software packages needed to be installed on your machine. This way, it's easily added - and it's also easy to replace, should you choose to do so at some point. You will then have all your settings like file system path definitions and compiler flags in well-defined places already.

JW-Build runs a recursive make, so, with a few exceptions such as submodules, you will need a makefile in every directory with source code. Most, if not all of these makefiles can be three-liners. This buys you convenience, minimal, clean, humanly digestable code, and a structured way to customize builds involving multiple packages - as centrally or as locally as you want. You can override any of JW-Build's default variables - for many packages, for an entire package, or for any subdirectory of a given package, at your option. You can write your own snippets and reuse them in multiple places. You can keep overrides in your versioning system or add them where needed in local.mk-files, which only your machine knows about. Or you can use environment variables, of course. JW-Build intentionally avoids makefile code generation as seen with CMake or GNU Autotools. This keeps the code small and readable for easy debugging. Okay, for relatively easy debugging. To achieve this, JW-Build has to detect a couple of things in every directory it enters, but it uses various caching mechanisms to keep builds still reasonably fast.

JW-Build has makefile snippets for building libraries and executables, snippets that output code compiled from C/C++, Python, Java, JavaScript, LaTeX, SWIG and other input formats, and it's easily extendible to support any given programming language or task. It's in use at janware for managing sub-builds of Maven, Ant, CMake and others, and for packaging the results. It provides targets to flash binaries onto microcontrollers, produce Debian, RPM and IPK packages, install them locally or remotely, or feed them into DevOps pipelines, taking note of released versions within GIT, SVN or CVS. It detects whether or not a package needs a new release because its source code changed. Or because a package it depends on has changed incompatibly. JW-Build has targets for collaboration over structured sets of remote Git repositories. It supports a simple configuration file per package for specifying package metadata, e.g. its dependencies, license, description, pre- and postinstall scriptlets, and so on. It has a SAT-solver built in, for building multiple packages in the right order, based on that metadata. With the same metadata, it can also automatically generate BitBake recipes and run Yocto-builds incorporating your software. Packages which are not organized with JW-Build can also be integrated into the build. It generates runtime, development and source code package variants. It supports cross compilation with MinGW and the GNU toolchain. It's tested on Debian, Ubuntu, OpenSUSE, Fedora / CentOS / RHEL, and many Unices.

JW-Build is designed to be friendly to both developers and integrators. Developers can cd into any given directory, edit the source code, run make, and expect the resulting binaries to work and be immediately testable, for a workflow that lets you focus on coding in your target language. For integrators on the other hand, a hotfix on a server or an embedded host can be as simple as

TARGET_HOST=myserver.acme.com make pkg-remote-install

And of course, it can build, package and release itself. Without being installed, which is a Good Thing (TM).

Basic Usage

For janware builds

See https://janware.com/wiki/pub/sw/build/ for documentation on how to get and use JW-Build within the janware build infrastructure.

Standalone

If you want to use it standalone, OTOH, do the following to get a minimal working example:

First, add a make subdirectory to the toplevel directory of your package, containing two files:

1. proj.mk, containing a definition of JWBDIR, pointing to the JW-Build installation directory, e.g. like so:

    JWBDIR ?= $(firstword $(wildcard $(addsuffix /jw-build,$(TOPDIR)/.. /opt)))
   

   TOPDIR points to, you guessed it, the toplevel directory of your package. You will most likely define it yourself in every makefile that uses proj.mk, see below. The right-hand side of the equation is GNU Make gibberish for: "Look for a directory named jw-build next to my package, and then below /opt, in that order." Assuming that's where the packages are in your file system.

   Things to note:

   • All directory paths can be relative, which is nice if you want to organize multiple packages in a fixed tree layout, but want them to work wherever you place the tree.
   • It doesn't have to be $(TOPDIR)/make. You can choose any other subdirectory of your package, but for now let's just assume it's $(TOPDIR)/make, the default location.

2. project.conf, containing

    [description]
    A frobnicator library
   

Then, add files named Makefile to the directories of your project, containing, e.g., in a C++ directory:

TOPDIR = ../..

include $(TOPDIR)/make/proj.mk
include $(JWBDIR)/make/cpp.mk

Done. Well, in principle. Other notable snippets are topdir.mk for the toplevel directory, dirs.mk for other directories with subdirectories, lib.mk for the directory containing the package's main library, which defaults to $(TOPDIR)/lib, bin.mk for $(TOPDIR)/bin, include.mk for $(TOPDIR)/include, and make.mk for $(TOPDIR)/make. You should add them in the same manner. Again, these are just default locations. Once you've added those makefiles, running make will do - something. Try and see what happens. Every snippet supports at least the targets all, install, clean and distclean. The target echo-makefiles shows you all included snippets, cat-makefiles concatenates them. Hitting TAB should show you all targets supported in a particular directory.

Good luck!

Advanced Usage, Custom Makefiles, Tips and Tricks, Things to Note

• JW-Build will use any Makefile, makefile or GNUmakefile it finds in the toplevel directory of your package. It will cd into the toplevel directory and run make during build runs, and subsequently make install during installation runs. It will also run make clean if invoked from outside. make distclean doesn't need to be supported, but comes in handy, if you want your packages to be wiped extra-clean.

  JW-Build will not run make install as the root user, and will therefore expect you to use $(INSTALL) instead of /usr/bin/install, to which it is command line compatible. pseudo support is underway and will allow to use plain /usr/bin/install, but is not yet implemented.

• If you want GNU Make to do what it's designed for for your custom task, i.e. do the right thing based on file modification times, do the following, for example:

    TOPDIR                 = ../..
  
    MY_CREATION_INPUT      = $(wildcard *.tmpl)
    MY_CREATION            = $(patsubst %.tmpl,%,$(MY_CREATION_INPUT))
    MY_INSTALL_DIR         = $(ENV_PREFIX)/var/lib/my-creation
    MY_INSTALLED_CREATION  = $(addprefix $(MY_INSTALL_DIR)/,$(MY_CREATION))
  
    include $(TOPDIR)/make/proj.mk
    include $(JWBDIR)/make/defs.mk
  
    all: $(MY_CREATION)
    install: $(MY_INSTALLED_CREATION)
    clean: clean.my-creation
    distclean: clean
  
    %: %.tmpl
            bash ./my-special-stuff-creator.sh $< > $@.tmp
            mv $@.tmp $@
    clean.my-creation:
            rm -f $(MY_CREATION) *.tmp
    $(MY_INSTALL_DIR):
        $(INSTALL) -o root -g root -m 755 -d $@
    $(MY_INSTALL_DIR)/%: % | $(MY_INSTALL_DIR)
            $(INSTALL) -o root -g root -m 644 $< $@
  

  Things to note:

  • If the example seems too verbose for you, consider putting the code into $(TOPDIR)/make/my-creation.mk (for example) and include that.
  • defs.mk doesn't provide any targets, but only defines central variables. It's implicitly included with most snippets, but if you want to define your own tasks, you can use it directly.
  • The MY_XXX variables have arbitrary names and are not evaluated by JW-Build.
  • $(ENV_PREFIX) will automatically point to something below your home directory if invoked from a package which has .git, .svn or CVS in its toplevel directory, and to the proper target root during package creation.
  • Using the intermediate $@.tmp is a good idea in case my-special-stuff-creator.sh can fail.
  • See info make for how pattern rules work (%), what the | pipe - symbol is for and all that jazz. If you don't know already.

• If you want to add arbitrary tasks to the build run, consider the following makefile:

  TOPDIR = ../..
  
  include $(TOPDIR)/make/proj.mk
  include $(JWBDIR)/make/defs.mk
  
  all: my-task.done
  clean: clean.my-task
  
  my-task.done:
          cd $(SUBMOD_SRC_DIR) && mvn -DskipTests --settings ../maven-settings.xml
          touch $@
  clean.my-task:
          rm -rf $(addprefix $(SUBMOD_SRC_DIR)/,dependency-reduced-pom.xml externs/target target) $(SOURCE_BASE)
          rm -f *.done