Add class AsyncRunner. This is a wrapper around the ceremony needed
to spawn an extra event loop in a synchronous function which wants to
call an async function.
Guido van Rossum considers it bad design that such a function exists
in the first place. While that may be true in the long run also for
jw-pkg, at this point I'm unwilling to flag every lazyly initialized
App property as async. It's not clear, yet, which will be async and
which not, and I dread the churn. So I will accept this as a
minimally invasive helper for now. When the API has stabilized, a
design without it may be better.
Signed-off-by: Jan Lindemann <jan@janware.com>
Expose App's __eloop member containing the application's main event
loop to allow outside async event loop trickery.
Signed-off-by: Jan Lindemann <jan@janware.com>
Take implementation burden from the derived classes _run() callback
by moving the respective code into the run() wrapper methods of the
base class.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove the key_filename parameter from the call to Paramiko's
connect(). It's user-dependent, and the current DevOps implementation
relies on having a SSH_AUTH_SOCK in the environment, anyway.
Signed-off-by: janware DevOps <devops@janware.com>
Add a wrapper around urlparse() and Paramiko's connect() function, in
order to log some more info in case an exception is thrown.
Signed-off-by: Jan Lindemann <jan@janware.com>
Take a positional uri argument to the constructor of ExecContext,
forcing SSHClient to follow suit. The latter was instantiated with a
hostname as only argument up to now, which still works as a special
case of an uri.
Signed-off-by: Jan Lindemann <jan@janware.com>
Use SSHClient as an ExecContext, i.e. use the .run() method instead
of .run_cmd(). Also, let SSHClient decide which implementation to
use.
Signed-off-by: Jan Lindemann <jan@janware.com>
Code beautification chore: Move the public methods of SSHClients to
the bottom of the class to be consistent with other classes.
Signed-off-by: Jan Lindemann <jan@janware.com>
Align the prototype of SSHClient.run_cmd() to ExecContext.run(). This
is a push towards making the SSHClient code an ExceContext, too. Some
arguments still log a warning or outright raise NotImplementedError.
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdInfo._expand_macros() raises a custom exception during exception
handling. Replace that by logging some details and raising the
original exception to keep the stack trace readable.
Signed-off-by: Jan Lindemann <jan@janware.com>
Make SSHClient accept a list of strings for the cmd argument to align
with the other run_cmd() functions in jw-pkg.
Signed-off-by: Jan Lindemann <jan@janware.com>
Make all backend package manager prototypes have the same arguments:
yum(self, args: list[str], verbose: bool=True, sudo: bool=True)
This also implies having them behave equally verbose, unless
otherwise specified by the caller. This changes the default for
Debian.
Signed-off-by: Jan Lindemann <jan@janware.com>
pkg_relations_list() has an intricate case distinction around
expand_semver_revision_range, clean that up.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a push to eventually merge the classes, somewhat align the
command-line API of CmdRequiredOsPkg to the one of
BaseCmdPkgRelations by using dependency flavours as mandatory, first
argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
To support the pkg-install-testbuild-deps target, a selector is
needed listing all prerequisites to be installed except the project
under test. --hide-self should be useful for that.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a push to eventually merge class CmdRequiredOsPkg into this class,
add the --skip-excluded option required by it.
Signed-off-by: Jan Lindemann <jan@janware.com>
In a push to eventually merge class CmdRequiredOsPkg into this class,
add the --quote option required by it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a function pkg_relations_list(), doing pretty much the same as
pkg_relations(), but taking individual arguments instead of an
argparse.Namespace args argument, in order to provide the
functionality to derived classes.
Signed-off-by: Jan Lindemann <jan@janware.com>
In commands taking lists of packages, namely install, delete and
pkg_files, don't bother asking the backend. Uniformly log a warning
and return successfully.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for --syntax to BaseCmdPkgRelations.pkg_relations(), and
default to 'semver', i.e. the current state of affairs. If that's
changed to 'debian', relations declared in project.conf as
pkg.requires.os.devel = jw-pkg-devel > 1.2.3
will be output as
jw-pkg-devel >> 1.2.3
which is what Debian expects.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for --expand-semver-revision-range to
cmds.projects.BaseCmdPkgRelations.pkg_relations(). The options turns
a dependency in projects.conf
pkg.requires.os.devel = jw-pkg-devel = VERSION
into
jw-pkg-devel >= 1.2.3, jw-pkg-devel < 1.2.4
Note that this will break as soon as a real range is specified in
projects.conf. To be fixed later, this commit is at least less
breakage than before.
Signed-off-by: Jan Lindemann <jan@janware.com>
"getting value xxx for project" is a prominent debug log message but
ugly. Beautify to e.g.:
Lookup jw-fail2ban -> jw-pkg / version
Meaning project "jw-fail2ban" looks up the value for key "version" in
project "jw-pkg".
Signed-off-by: Jan Lindemann <jan@janware.com>
The output of
jw.pkg.py projects required-os-pkg --flavours release
should include all packages required by flavour devel, because during
the release process, -devel and -run packages are both installed, and
installing the -devel package is only possible if its dependencies
are installed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't mention "error" in log message for exit codes > 0 from spawned
processes, because sometimes they don't mean an error.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for the -o (--owner) -g (--group) -m (--mode) options.
They allow to specify a default for compiling templates, but _don't_
override what's in the #conf: specification line in .jw-tmpl or
.jw-secret files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Support option --all to jw-pkg.py secrets list-compilation-output and
list-secrets (CmdListCompilationOutput & CmdSecrets). This allows
them to also report non-existent files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Fix errors dug up by testing on Kali Linux 2026.1:
- Nested class Cmds.Attrs is constructed without scope
- "replace" falls back to empty list, not empty dictionary
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg.py secrets [sub-command] [packages] is a set of utility
commands designed to manage configuration files containing secrets.
To keep secrets from leaking via version control or packages, a
_template_ should be packaged for every sensitive configuration file.
Then, during post-install, configuration files can be generated from
packaged templates via
jw-pkg.py secrets compile-templates <package> <package> ...
During post-uninstall
jw-pkg.py secrets rm-compilation-output <package> <package> ...
removes them.
Not specifying any packages will compile or remove all templates on
the system.
To identify which files to consider and generate or remove, the
compilation scans <package> for files ending in .jw-tmpl. For each
match, e.g.
/path/to/some.conf.jw-tmpl
it will read key-value pairs from
/path/to/some.conf.jw-secret
and generate
/path/to/some.conf
from it, replacing all keys by their respective values. The file
attributes of the generated file can be determined by the first line:
of some.conf.jw-tmpl or some.conf.jw-secret:
# conf: owner=mysql; group=mysql; mode=0640
There are other commands for managing all secrets on the system at
once, see jw-pkg.py secrets --help:
compile-templates Compile package template files
list-compilation-output
List package compilation output files
list-secrets List package secret files
list-templates List package template files
rm-compilation-output
Remove package compilation output files
Signed-off-by: Jan Lindemann <jan@janware.com>
DistroBase's option --id is now redundant to the new global option
--distro-id in the App class, so remove --id. The only added value
DistroBase then brings to the table is its .distro property, which
can be provided by App just fine at this point, given that App has
all it needs to construct a Distro object, so add .distro to App and
remove the entire DistroBase class.
For convenience, also make App.distro available as a newly added
cmds.Cmd.distro property. This also obviates the need for the
distro-related properties in the .distro.Cmd class, remove all that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the --verbose global option, which is made available as the
App.verbose property.
Some functions still take a verbose parameter, but the type of these
parameters is converted from bool to bool|None. The idea is that, if
they are None, their verbosity falls back to the global default.
Signed-off-by: Jan Lindemann <jan@janware.com>
Functions in lib.pm (i.e. run_dpkg(), run_rpm() and friends) also get
an ExecContext-type parameter. Use them in lib/distros/*/Distro.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove defaults from protected callback function parameters. They
have to be decided by the base class's public API.
Signed-off-by: Jan Lindemann <jan@janware.com>
Allow to specify the ExecContext in a call to run_cmd(). This
effectively makes run_cmd() an thin wrapper around ExecContext.run(),
which is what's going to be used in the future. The wrapper is for
backwards-compatibility.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the global options --distro-id and --interactive, and expose them
as properties App.distro_id and App.interactive, respectively.
Signed-off-by: Jan Lindemann <jan@janware.com>
Functions abstracting the distribution are not only needed in the
context of the distro subcommand, but also by other code, so make the
bulk of the code abstracting the distribution available in some place
more universally useful than below cmds.distro.
This commit leaves the source files mostly unchanged. They are only
patched to fix import paths, so that functionality is preserved.
Refactoring the code from command-line API to library API will be
done by the next commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
