- Rename variable dep and deps to val and vals, respectively,
because that's more what they are values of key-value pairs. In
some cases that can represent dependencies, in some case other
things.
- Make a scope case distinction a little clearer by mentioning all
possible cases in a match / case block
Allow find_dir() to return None in case it couldn't find a directory, that's a legal outcome. Add a boolean parameter "throw" to support throwing an exception if the existence needs to be asserted.
It would probably be nicer for the type checkers to split this up into a throwing and non-throwing function. Postponed.
App.is_excluded_from_build() uses the wrong function entirely to query the [build.exclude] section of project.conf (App.get_project_refs() instead of App.get_value()). This has obviously never worked. It rose to prominence because commit 6db73873 introduced App.__proj_dir(), which now raises an Exception if passed garbage, which in turn surfaces as
Exception: No project path found for module "debian"
Use the correct function for that: App.get_value().
The previous commits have put rules for linting and formatting via ruff, yapf, mypy and pyright into place. They are checked with the make check target, and this commit adds the fixes for the target to succeed.
It does some refactoring where type checking dug up dirty bits, and also adds lots of churn in the Python code. To a good deal, that's owed to mere formatting changes. It would have been better to seperate those from syntax and refactoring fixes into multiple commits, so that the interesting changes don't drown in the formatting nose. However, that would have been a lot of additional work only to be thrown away by later commits, hence this commit has a big diff in one piece. The size of the diff is regrettable but hopefully a one-off: What it buys is automatic format checking for CI and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
- ruff check .
- mypy .
- pyright
- Format check:
- yapf --diff --recursive .
The refactoring includes:
- Turn the Result class into a more elaborate object, capable of
doing more heavy lifting around stderr and stdout decoding,
summarizing outcome, and matching error strings.
Aside from fixing broken type checks, this also removes lots of
boilerplate calling code which is currently used for handling
possible call outcome scenarios. Trying to access an inexistent,
decoded string should raise a meaningful exception by itself now,
which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
- Add the AbstractCmd class above Cmd. This is necessary because
the checker rightfully complains it can't instantiate a Cmd
instance where constructor arguments were needed. They never
were, but the type used at the instantiating code's location in
jw.pkg.App so claims.
- Lots of sub- and sub-subcommands are derived from the base
class of the invoking command. That provides some properties
shared across the ancestor hierarchy of a command, but is
semantically unsound. Fix that by introducing jw.pkg.BaseCmd
class as a place to provide basic helpers shared across all
commands used in a jw.pkg.App's context, and derive all command
classes from that afresh. The parent command is still reachable
via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
The line was needed by Python 2 which is not supported anylonger.
For Python 3, the default encoding is UTF-8, anyway.
- Allow to run "make py-format" without having it produce any
changes. It's basically "yapf --in-place --recursive ." with some
code style settings, see conf/topdir/pyproject.toml. The settings
may be debatable. I've had custom tweaks in place on that target,
too, but then again, IDEs would have more hassle to integrate
that.
- Introduce a 88 character line length limit
- One import per line, reshuffle them semantically, see
[tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
if TYPE_CHECKING
- Spaces around assignments accounts for much churn. Having having
no spaces in inline parameter list assignments and default
parameter values would arguably be more compact where it's
useful. On the other hand, I have not found a code formatter
which allows spaces around assignments in parameter lists broken
into one per line and that's often better than a wall of text.
- Add two spaces before # export, as this seems to be mandated by
PEP-8
--uri is unnecessarily generic in that it could mean the URI of anything. --target makes it clearer that operations are to be exectuted on that target.
Add global --pkg-filter argument, defaulting to JW_DEFAULT_PKG_FILTER. If it's specified, instantiate a PackageFilterString from it, and initialize App's Distro instance with it.
Commit a19679fec reverted the first attempt to make AsyncSSH reuse one connection during an instance lifetime. That failed because a lot of distribution-specific properties were filled in a new event loop thread started by AsyncRunner, and AsyncSSH didn't like that.
This commit is the first part of the solution: Move those properties from the App class to the Distro class, and load the Distro class in an async loader. As soon as it's instantiated, it can provide all its properties without cluttering the code with async keywords.
App.distro_info() accepts and returns str instances, interpret anything passed as fmt parameter which is not a str as iterable, and return lists of expanded strings in that case.
For now, move the definiions of Result, Input and InputMode from ExecContext into lib.base. Having to import them from the ExecContect module is too heavy-handed for those simple types.
Python's platform.system() outputs 'Linux', and to use it is tempting. Sadly, that's wrong, because it reflects the host's idea of the target system, not the execution context's, so replace it with straight 'linux' if the distro is known, or, failing that, the output of uname -s.
Even with --interactive=[true|auto], there's no point in trying to read /etc/os-release interactively, so don't do that. Most notably, this commit keeps the property method from spilling /etc/os-release's content over the terminal.
ExecContext's .sudo() omits many of run()'s parameters, and this commit adds them. To avoid redundancy around repeating and massaging the long parameter list of both functions and their return values, it also adds some deeper changes:
- Make run(), _run(), sudo() and _sudo() always return instances of
Result. Before it was allowed to return a triplet of stdout,
stderr, and exit status.
- Have ExecContext stay out of the business of decoding the result
entirely. Result provides a convenience method .decode()
operating on stdout and stderr and leaves the decision to the
caller.
This entails miniscule adaptations in calling code, namely in
App.os_release, util.get_profile_env() and CmdListRepos._run().
- Wrap the _run() and _sudo() callbacks in a context manager object
of type CallContext to avoid code duplication.
- Consistently name the first argument to run(), _run(), sudo() and
_sudo() "cmd", not "args". The latter suggests that the caller is
omitting the executable, which is not the case.
Don't open and parse /etc/os-release with Python built-in functions. Spawn "cat /etc/os-release" as a subprocess and capture the output for parsing instead. The obvious advantage is that this also works with a remote shell.
DistroBase's option --id is now redundant to the new global option --distro-id in the App class, so remove --id. The only added value DistroBase then brings to the table is its .distro property, which can be provided by App just fine at this point, given that App has all it needs to construct a Distro object, so add .distro to App and remove the entire DistroBase class.
For convenience, also make App.distro available as a newly added cmds.Cmd.distro property. This also obviates the need for the distro-related properties in the .distro.Cmd class, remove all that.
Add the --verbose global option, which is made available as the App.verbose property.
Some functions still take a verbose parameter, but the type of these parameters is converted from bool to bool|None. The idea is that, if they are None, their verbosity falls back to the global default.
The code below lib.distro, as left behind by the previous commit, is geared towards being directly used as a command-line API. This commit introduces the abstract base class Distro, a proxy for distribution-specific interactions. The proxy abstracts distro specifics into an API with proper method prototypes, not argparse.Namespace contents, and can thus be more easily driven by arbitrary code.
The Distro class is initialized with a member variable of type ExecContext, another new class introduced by this commit. It is designed to abstract the communication channel to the distribution instance. Currently only one specialization exists, Local, which interacts with the distribution and root file system it is running in, but is planned to be subclassed to support interaction via SSH, serial, chroot, or chains thereof.
get-os.sh returned "suse" for SuSE-like distros, and that seems more appropriate since SLES is not OpenSUSE but should share and ID with other SuSE variants.
Add more fields to the OS cascade returned by App.os_cascade, based on the ID field in /etc/os-release. This includes some new ones, prefixed by pkg-, revealing which package format is used.
Major - but not yet sufficient - code beautification starting from jw.pkg.App.
- Make more methods private
- Rename methods to be more self-explanatory
- Same for method arguments, notably clean up some inconsistent
uses of "module" vs "project"
- Add more type hints
ResultCache is a home-grown result cache. The @lru_cache decorator, now available in Python 3, accomplishes the same thing, so try to ditch ResultCache for it.
Sadly, this doesn't entirely work as of now, because it uses hash() to hash the arguments, which won't work for the two list-type arguments to add_modules_from_project_txt() (buf and visited).
Add support for --topdir-format. The option supports several different values, affecting the console output of App wherever it knows that the output contains a reference to the projects' toplevel directory.
- "unaltered" will have it print the toplevel directory in the same
format as passed to the commandline
- "absolute" will try to resolve it to an absolute path before
printing
- make:XXX will return the make-varible $(XXX) instead
To implement this, the proj_dir() member function is turned into the private member function __proj_dir(), and a new member function find_dir() is supplied, with two additional parameters: search_subdirs and search_absdirs, which will try to find an existing directory relative to the toplevel directory of the given module, or in the search_absdirs list, respectively.
Command modules in cmds.projects have been updated to use the new function.
Reorganize the Python module structure. Placing the command classes under jw.cmds.projects instead of jw.build.cmds will allow to add a nested command structure, with the current commands, being mostly related to building software, found below a "projects" toplevel command.
Other conceivable commands could be "package" for packaging, or "distro" for commands wrapping the distribution's package manager.
Make App.proj_dir() return an absolute path. This looks like a good idea, because some of the $(call proj_query xxx) paths end up being relative, because they get proj_dir()'s idea of a directory prepended. This prohibits caching them in $(TOPDIR)/make/.cache.mk for make benefit glorious nation of performance.
Derive jw.pkg.App from jw.pkg.lib.App. App.run() dissolves as follows:
- Its sub-command invocation logic is left to the base class - parser.add_arguments() are moved into self._add_arguments() - So is handling of early-parsed arguments - async def _run() is reimplemented to set some member variables
For a project to supply templates, it needs to advertise their location. For this, the tmpl_dir make variable is added to projects.mk. If other-project wants to get hold of some-project's templates, it can do, e.g.:
