"getting value xxx for project" is a prominent debug log message but
ugly. Beautify to e.g.:
Lookup jw-fail2ban -> jw-pkg / version
Meaning project "jw-fail2ban" looks up the value for key "version" in
project "jw-pkg".
Signed-off-by: Jan Lindemann <jan@janware.com>
The output of
jw.pkg.py projects required-os-pkg --flavours release
should include all packages required by flavour devel, because during
the release process, -devel and -run packages are both installed, and
installing the -devel package is only possible if its dependencies
are installed.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't mention "error" in log message for exit codes > 0 from spawned
processes, because sometimes they don't mean an error.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add support for the -o (--owner) -g (--group) -m (--mode) options.
They allow to specify a default for compiling templates, but _don't_
override what's in the #conf: specification line in .jw-tmpl or
.jw-secret files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Support option --all to jw-pkg.py secrets list-compilation-output and
list-secrets (CmdListCompilationOutput & CmdSecrets). This allows
them to also report non-existent files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Fix errors dug up by testing on Kali Linux 2026.1:
- Nested class Cmds.Attrs is constructed without scope
- "replace" falls back to empty list, not empty dictionary
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg.py secrets [sub-command] [packages] is a set of utility
commands designed to manage configuration files containing secrets.
To keep secrets from leaking via version control or packages, a
_template_ should be packaged for every sensitive configuration file.
Then, during post-install, configuration files can be generated from
packaged templates via
jw-pkg.py secrets compile-templates <package> <package> ...
During post-uninstall
jw-pkg.py secrets rm-compilation-output <package> <package> ...
removes them.
Not specifying any packages will compile or remove all templates on
the system.
To identify which files to consider and generate or remove, the
compilation scans <package> for files ending in .jw-tmpl. For each
match, e.g.
/path/to/some.conf.jw-tmpl
it will read key-value pairs from
/path/to/some.conf.jw-secret
and generate
/path/to/some.conf
from it, replacing all keys by their respective values. The file
attributes of the generated file can be determined by the first line:
of some.conf.jw-tmpl or some.conf.jw-secret:
# conf: owner=mysql; group=mysql; mode=0640
There are other commands for managing all secrets on the system at
once, see jw-pkg.py secrets --help:
compile-templates Compile package template files
list-compilation-output
List package compilation output files
list-secrets List package secret files
list-templates List package template files
rm-compilation-output
Remove package compilation output files
Signed-off-by: Jan Lindemann <jan@janware.com>
DistroBase's option --id is now redundant to the new global option
--distro-id in the App class, so remove --id. The only added value
DistroBase then brings to the table is its .distro property, which
can be provided by App just fine at this point, given that App has
all it needs to construct a Distro object, so add .distro to App and
remove the entire DistroBase class.
For convenience, also make App.distro available as a newly added
cmds.Cmd.distro property. This also obviates the need for the
distro-related properties in the .distro.Cmd class, remove all that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the --verbose global option, which is made available as the
App.verbose property.
Some functions still take a verbose parameter, but the type of these
parameters is converted from bool to bool|None. The idea is that, if
they are None, their verbosity falls back to the global default.
Signed-off-by: Jan Lindemann <jan@janware.com>
Functions in lib.pm (i.e. run_dpkg(), run_rpm() and friends) also get
an ExecContext-type parameter. Use them in lib/distros/*/Distro.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove defaults from protected callback function parameters. They
have to be decided by the base class's public API.
Signed-off-by: Jan Lindemann <jan@janware.com>
Allow to specify the ExecContext in a call to run_cmd(). This
effectively makes run_cmd() an thin wrapper around ExecContext.run(),
which is what's going to be used in the future. The wrapper is for
backwards-compatibility.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the global options --distro-id and --interactive, and expose them
as properties App.distro_id and App.interactive, respectively.
Signed-off-by: Jan Lindemann <jan@janware.com>
Functions abstracting the distribution are not only needed in the
context of the distro subcommand, but also by other code, so make the
bulk of the code abstracting the distribution available in some place
more universally useful than below cmds.distro.
This commit leaves the source files mostly unchanged. They are only
patched to fix import paths, so that functionality is preserved.
Refactoring the code from command-line API to library API will be
done by the next commit.
Signed-off-by: Jan Lindemann <jan@janware.com>
Move most of CmdDistro into the new class DistroBase, meant to serve
as base class for distro-aware commands other than "distro".
Signed-off-by: Jan Lindemann <jan@janware.com>
jw-pkg distro dup got hung in a chroot environment. strace shows that
write(2) into a pipe is the hanging syscall, with the write buffer
hinting at zypper dup output.
I strongly suspect that run_cmd() tries to write stdout into the pipe
which read_stream() fails to empty. So, make read_stream() more
resilient by using read(4096) instead of readline(), which I suspect
to be prone to hang on overlong lines.
Signed-off-by: Jan Lindemann <jan@janware.com>
--fix-broken is added to apt-get options in non-interactive mode, but
seems to work only with apt-get install, not with apt-get update.
Don't add it at all for now.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add utility method .dpkg() to the backend.debian.Utils class, saving
the need to import it, saving the need to import it.
Signed-off-by: Jan Lindemann <jan@janware.com>
apt_get(), like all other functions and methods spawning processes
should take a list instead of a starred *args array. Implement that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename the "packages" argument to "names" to be a little more
consistent with the pkg subcommand argument nomenclature.
Signed-off-by: Jan Lindemann <jan@janware.com>
run_cmd() with cmd_input == mode:interactive and verbose == true
logs output too often. First, __log() is called, then pty.spawn()
writes everything it reads from the PTY master to the terminal.
The fix it to not call __log() from _read() for the PTY reader.
Signed-off-by: Jan Lindemann <jan@janware.com>
Support distro pkg meta <package names ...>, returning meta
information for the specified package names.
Signed-off-by: Jan Lindemann <jan@janware.com>
As with cmds.distro.lib.rpm, the dpkg module is a place for Python
abstractions of the Debian package manager tools dpkg and friends.
Signed-off-by: Jan Lindemann <jan@janware.com>
Replace all_installed_packages() by query_packages(). The function
takes an optional list of packages to be queried. If it's empty, a
list of all installed packages are returned.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add more methods, to make Package more useful:
- __repr__()
- Class-mMethods to help with parsing of strings gathered from the
OS-tools' output:
parse_spec_str()
parse_specs_str()
order_tags()
- Add field maintainer
Signed-off-by: Jan Lindemann <jan@janware.com>
Make all pkg commands take a "names" argument (plural) instead of
"name", and make it a non-option-argument, to be passed to the
subcomand instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add --download-only to the options of jw-pkg.py distro dup, which
makes the command only download packages from the repositories
without installing them.
Signed-off-by: Jan Lindemann <jan@janware.com>
Passing --only-update should keep "jw-pkg.py distro install" from
installing packages that are not already installed on the the system.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the command distro.CmdRebootRequired, adding support for "distro
reboot-required". The command exits with status code 1 if a reboot is
required and 0 otherwise.
Signed-off-by: Jan Lindemann <jan@janware.com>
If a Cmd-classes's _run() method returns an integer between 0 and
255, use that as the program's exit status.
Signed-off-by: Jan Lindemann <jan@janware.com>
Most run_xxx() return stdout and stderr. There's no way, really, for
the caller to get hold of the exit code of the spawned executable. It
can pass throw=true, catch, and assume a non-zero exit status. But
that's not semantically clean, since the spawned function can well be
a test function which is expected to return a non-zero status code,
and the caller might be interested in what code that was, exactly.
The clearest way to solve this is to return the exit code as well.
This commit does that.
Signed-off-by: Jan Lindemann <jan@janware.com>
get-os.sh returned "suse" for SuSE-like distros, and that seems more
appropriate since SLES is not OpenSUSE but should share and ID with
other SuSE variants.
Signed-off-by: Jan Lindemann <jan@janware.com>
App.distro_id used to return "opensuse-tumbleweed", analogous to
what's in ID@/etc/os-release, but now returns "opensuse", and the
"tumbleweed" goes into "codename". That matches more what Debian-like
distributions do, but it confuses _backend_path. Adapt it to map the
new distro_id correctly.
Signed-off-by: Jan Lindemann <jan@janware.com>
