ExecContext's .sudo() omits many of run()'s parameters, and this commit adds them. To avoid redundancy around repeating and massaging the long parameter list of both functions and their return values, it also adds some deeper changes:
- Make run(), _run(), sudo() and _sudo() always return instances of
Result. Before it was allowed to return a triplet of stdout,
stderr, and exit status.
- Have ExecContext stay out of the business of decoding the result
entirely. Result provides a convenience method .decode()
operating on stdout and stderr and leaves the decision to the
caller.
This entails miniscule adaptations in calling code, namely in
App.os_release, util.get_profile_env() and CmdListRepos._run().
- Wrap the _run() and _sudo() callbacks in a context manager object
of type CallContext to avoid code duplication.
- Consistently name the first argument to run(), _run(), sudo() and
_sudo() "cmd", not "args". The latter suggests that the caller is
omitting the executable, which is not the case.
Don't open and parse /etc/os-release with Python built-in functions. Spawn "cat /etc/os-release" as a subprocess and capture the output for parsing instead. The obvious advantage is that this also works with a remote shell.
Use the AsyncRunner class introduced in the previous commit to add a call_async() method, allowing to run async functions from sync functions by spawning an extra event loop.
Add class AsyncRunner. This is a wrapper around the ceremony needed to spawn an extra event loop in a synchronous function which wants to call an async function.
Guido van Rossum considers it bad design that such a function exists in the first place. While that may be true in the long run also for jw-pkg, at this point I'm unwilling to flag every lazyly initialized App property as async. It's not clear, yet, which will be async and which not, and I dread the churn. So I will accept this as a minimally invasive helper for now. When the API has stabilized, a design without it may be better.
Remove the key_filename parameter from the call to Paramiko's connect(). It's user-dependent, and the current DevOps implementation relies on having a SSH_AUTH_SOCK in the environment, anyway.
Take a positional uri argument to the constructor of ExecContext, forcing SSHClient to follow suit. The latter was instantiated with a hostname as only argument up to now, which still works as a special case of an uri.
Align the prototype of SSHClient.run_cmd() to ExecContext.run(). This is a push towards making the SSHClient code an ExceContext, too. Some arguments still log a warning or outright raise NotImplementedError.
CmdInfo._expand_macros() raises a custom exception during exception handling. Replace that by logging some details and raising the original exception to keep the stack trace readable.
In a push to eventually merge the classes, somewhat align the command-line API of CmdRequiredOsPkg to the one of BaseCmdPkgRelations by using dependency flavours as mandatory, first argument.
To support the pkg-install-testbuild-deps target, a selector is needed listing all prerequisites to be installed except the project under test. --hide-self should be useful for that.
Add a function pkg_relations_list(), doing pretty much the same as pkg_relations(), but taking individual arguments instead of an argparse.Namespace args argument, in order to provide the functionality to derived classes.
In commands taking lists of packages, namely install, delete and pkg_files, don't bother asking the backend. Uniformly log a warning and return successfully.
Add support for --syntax to BaseCmdPkgRelations.pkg_relations(), and default to 'semver', i.e. the current state of affairs. If that's changed to 'debian', relations declared in project.conf as
should include all packages required by flavour devel, because during the release process, -devel and -run packages are both installed, and installing the -devel package is only possible if its dependencies are installed.
Add support for the -o (--owner) -g (--group) -m (--mode) options. They allow to specify a default for compiling templates, but _don't_ override what's in the #conf: specification line in .jw-tmpl or .jw-secret files.
Support option --all to jw-pkg.py secrets list-compilation-output and list-secrets (CmdListCompilationOutput & CmdSecrets). This allows them to also report non-existent files.
jw-pkg.py secrets [sub-command] [packages] is a set of utility commands designed to manage configuration files containing secrets.
To keep secrets from leaking via version control or packages, a _template_ should be packaged for every sensitive configuration file. Then, during post-install, configuration files can be generated from packaged templates via
Not specifying any packages will compile or remove all templates on the system.
To identify which files to consider and generate or remove, the compilation scans <package> for files ending in .jw-tmpl. For each match, e.g.
/path/to/some.conf.jw-tmpl
it will read key-value pairs from
/path/to/some.conf.jw-secret
and generate
/path/to/some.conf
from it, replacing all keys by their respective values. The file attributes of the generated file can be determined by the first line: of some.conf.jw-tmpl or some.conf.jw-secret:
# conf: owner=mysql; group=mysql; mode=0640
There are other commands for managing all secrets on the system at once, see jw-pkg.py secrets --help:
compile-templates Compile package template files
list-compilation-output
List package compilation output files
list-secrets List package secret files
list-templates List package template files
rm-compilation-output
Remove package compilation output files
DistroBase's option --id is now redundant to the new global option --distro-id in the App class, so remove --id. The only added value DistroBase then brings to the table is its .distro property, which can be provided by App just fine at this point, given that App has all it needs to construct a Distro object, so add .distro to App and remove the entire DistroBase class.
For convenience, also make App.distro available as a newly added cmds.Cmd.distro property. This also obviates the need for the distro-related properties in the .distro.Cmd class, remove all that.
Add the --verbose global option, which is made available as the App.verbose property.
Some functions still take a verbose parameter, but the type of these parameters is converted from bool to bool|None. The idea is that, if they are None, their verbosity falls back to the global default.
