The fd.close() call inside the `with open(...) as fd:` block is
redundant because the file handle is managed by the context manager.
Remove it.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Signed-off-by: Jan Lindemann <jan@janware.com>
Replace the hardcoded /opt in __proj_dir() with a configurable ___opt_root
member, consistent with how __projs_root is configured.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Signed-off-by: Jan Lindemann <jan@janware.com>
Mutable default arguments (list) cause unexpected shared state
between calls. Use None as default and initialise to [] inside the
function body.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Signed-off-by: Jan Lindemann <jan@janware.com>
get_project_refs() always returns a list[str], never None. The `if
deps is None: continue` check in __read_dep_graph() is dead code,
remove it.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Signed-off-by: Jan Lindemann <jan@janware.com>
The __flip_dep_graph(graph) call sits inside the while loop and
performs redundant graph flipping on every iteration. Hoist it
outside to compute once and reuse the result.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove App.get_section() which parses raw file sections by scanning for
section headers and accumulating lines. This method is no longer needed
since ProjectConf now handles all config file parsing.
Signed-off-by: Jan Lindemann <jan@janware.com>
Refactor App to use the new ProjectConf module for parsing
make/project.conf files.
This commit
- removes the inline ad-hoc read_value() method and its nested
helper functions, replacing them with ProjectConf's
get_str_or_none() API.
- introduces two cached helper methods (__read_project_conf() and
__get_project_conf()) to read and cache ProjectConf instances per
project.
- updates get_value() to delegate to ProjectConf and moves the
proj_dir lookup to only run in the version branch where it is
needed.
- updates the topdir init to use __read_project_conf() with a
FileNotFoundError catch for optional config files.
Signed-off-by: Jan Lindemann <jan@janware.com>
Replace the @functools.lru_cache(maxsize=None) decorator with
@functools.cache throughout App. functools.cache is a shorthand for
functools.lru_cache(maxsize=None) introduced in Python 3.9 and is
more concise and readable with identical behaviour.
Signed-off-by: Jan Lindemann <jan@janware.com>
The global --topdir-format make:XXX option to jw-pkg is half-baked at
best, and __find_dir() ignores it entirely. Make __find_dir() return
some Makefile-syntax-formatted output if the option is present. Not
used anywhere, currently, and, hence, badly tested, but still better
than the situation before.
Signed-off-by: Jan Lindemann <jan@janware.com>
The global --topdir-format option governs how a project's root
directory is represented in paths output by various queries.
"absolute" means as absolute path, "unaltered" means verbatim as
specified via --topdir, make:xyz means replaced by the string $(xyz),
for later expansion in a makefile variable.
This commit adds another variant: "relative" yields the shortest
possible output format of the output path in question relative to
--topdir, with "shortest possible" in this context meaning
canonicalized and leading "./" stripped.
Signed-off-by: Jan Lindemann <jan@janware.com>
App.get_projects_refs() is a versatile tool, but what it does isn't
obvious. Use the simpler method .get_value() instead for
get_libname(), and return None if a project doesn't provide a
linkable library.
This is similar to fix aadcdfb5f.
Signed-off-by: Jan Lindemann <jan@janware.com>
In __get_project_refs():
- Rename variable dep and deps to val and vals, respectively,
because that's more what they are values of key-value pairs. In
some cases that can represent dependencies, in some case other
things.
- Make a scope case distinction a little clearer by mentioning all
possible cases in a match / case block
Signed-off-by: Jan Lindemann <jan@janware.com>
Allow find_dir() to return None in case it couldn't find a directory,
that's a legal outcome. Add a boolean parameter "throw" to support
throwing an exception if the existence needs to be asserted.
It would probably be nicer for the type checkers to split this up
into a throwing and non-throwing function. Postponed.
Signed-off-by: Jan Lindemann <jan@janware.com>
App.is_excluded_from_build() uses the wrong function entirely to
query the [build.exclude] section of project.conf
(App.get_project_refs() instead of App.get_value()). This has
obviously never worked. It rose to prominence because commit 6db73873
introduced App.__proj_dir(), which now raises an Exception if passed
garbage, which in turn surfaces as
Exception: No project path found for module "debian"
Use the correct function for that: App.get_value().
Signed-off-by: Jan Lindemann <jan@janware.com>
Add new ruff rules and fix their fallout:
future-annotations = true
select = [
"TC", # type-checking import placement rules
"FA", # future annotations rules
]
This comprises:
- Streamline imports and exports in cmds.xxx.Cmd
- Import base class as "Base"
- Export types Cmd and Parent via __all__
- Move all types imported only for annotation below TYPE_CHECKING
- Use "from __future__ import annotations" all over the place
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commits have put rules for linting and formatting via
ruff, yapf, mypy and pyright into place. They are checked with the
make check target, and this commit adds the fixes for the target to
succeed.
It does some refactoring where type checking dug up dirty bits, and
also adds lots of churn in the Python code. To a good deal, that's
owed to mere formatting changes. It would have been better to
seperate those from syntax and refactoring fixes into multiple
commits, so that the interesting changes don't drown in the
formatting nose. However, that would have been a lot of additional
work only to be thrown away by later commits, hence this commit has a
big diff in one piece. The size of the diff is regrettable but
hopefully a one-off: What it buys is automatic format checking for CI
and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
- ruff check .
- mypy .
- pyright
- Format check:
- yapf --diff --recursive .
The refactoring includes:
- Turn the Result class into a more elaborate object, capable of
doing more heavy lifting around stderr and stdout decoding,
summarizing outcome, and matching error strings.
Aside from fixing broken type checks, this also removes lots of
boilerplate calling code which is currently used for handling
possible call outcome scenarios. Trying to access an inexistent,
decoded string should raise a meaningful exception by itself now,
which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
- Add the AbstractCmd class above Cmd. This is necessary because
the checker rightfully complains it can't instantiate a Cmd
instance where constructor arguments were needed. They never
were, but the type used at the instantiating code's location in
jw.pkg.App so claims.
- Lots of sub- and sub-subcommands are derived from the base
class of the invoking command. That provides some properties
shared across the ancestor hierarchy of a command, but is
semantically unsound. Fix that by introducing jw.pkg.BaseCmd
class as a place to provide basic helpers shared across all
commands used in a jw.pkg.App's context, and derive all command
classes from that afresh. The parent command is still reachable
via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor
tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
The line was needed by Python 2 which is not supported anylonger.
For Python 3, the default encoding is UTF-8, anyway.
- Allow to run "make py-format" without having it produce any
changes. It's basically "yapf --in-place --recursive ." with some
code style settings, see conf/topdir/pyproject.toml. The settings
may be debatable. I've had custom tweaks in place on that target,
too, but then again, IDEs would have more hassle to integrate
that.
- Introduce a 88 character line length limit
- One import per line, reshuffle them semantically, see
[tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
if TYPE_CHECKING
- Spaces around assignments accounts for much churn. Having having
no spaces in inline parameter list assignments and default
parameter values would arguably be more compact where it's
useful. On the other hand, I have not found a code formatter
which allows spaces around assignments in parameter lists broken
into one per line and that's often better than a wall of text.
- Add two spaces before # export, as this seems to be mandated by
PEP-8
- Use single quotes by default
Signed-off-by: Jan Lindemann <jan@janware.com>
--uri is unnecessarily generic in that it could mean the URI of
anything. --target makes it clearer that operations are to be
exectuted on that target.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add global --pkg-filter argument, defaulting to
JW_DEFAULT_PKG_FILTER. If it's specified, instantiate a
PackageFilterString from it, and initialize App's Distro instance
with it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Commit a19679fec reverted the first attempt to make AsyncSSH reuse
one connection during an instance lifetime. That failed because a lot
of distribution-specific properties were filled in a new event loop
thread started by AsyncRunner, and AsyncSSH didn't like that.
This commit is the first part of the solution: Move those properties
from the App class to the Distro class, and load the Distro class
in an async loader. As soon as it's instantiated, it can provide all
its properties without cluttering the code with async keywords.
Signed-off-by: Jan Lindemann <jan@janware.com>
App.distro_info() accepts and returns str instances, interpret
anything passed as fmt parameter which is not a str as iterable, and
return lists of expanded strings in that case.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a method distro_info() to App, essentially providing CmdInfo's
macro-expansion of platform information to other interested code.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add lib.base to provide basic definitions.
For now, move the definiions of Result, Input and InputMode from
ExecContext into lib.base. Having to import them from the ExecContect
module is too heavy-handed for those simple types.
Signed-off-by: Jan Lindemann <jan@janware.com>
The property App.__os_release uses _run(['cat', '/etc/os-release']),
use ExecContext.get() instead as the default way to fetch content.
Signed-off-by: Jan Lindemann <jan@janware.com>
Enclose sections / keys taken from project.conf in [square.brackets],
hinting at what they are supposed to mean.
Signed-off-by: Jan Lindemann <jan@janware.com>
Python's platform.system() outputs 'Linux', and to use it is
tempting. Sadly, that's wrong, because it reflects the host's idea of
the target system, not the execution context's, so replace it with
straight 'linux' if the distro is known, or, failing that, the output
of uname -s.
Signed-off-by: Jan Lindemann <jan@janware.com>
Even with --interactive=[true|auto], there's no point in trying to
read /etc/os-release interactively, so don't do that. Most notably,
this commit keeps the property method from spilling /etc/os-release's
content over the terminal.
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecContext's .sudo() omits many of run()'s parameters, and this
commit adds them. To avoid redundancy around repeating and massaging
the long parameter list of both functions and their return values, it
also adds some deeper changes:
- Make run(), _run(), sudo() and _sudo() always return instances of
Result. Before it was allowed to return a triplet of stdout,
stderr, and exit status.
- Have ExecContext stay out of the business of decoding the result
entirely. Result provides a convenience method .decode()
operating on stdout and stderr and leaves the decision to the
caller.
This entails miniscule adaptations in calling code, namely in
App.os_release, util.get_profile_env() and CmdListRepos._run().
- Wrap the _run() and _sudo() callbacks in a context manager object
of type CallContext to avoid code duplication.
- Consistently name the first argument to run(), _run(), sudo() and
_sudo() "cmd", not "args". The latter suggests that the caller is
omitting the executable, which is not the case.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't open and parse /etc/os-release with Python built-in functions.
Spawn "cat /etc/os-release" as a subprocess and capture the output
for parsing instead. The obvious advantage is that this also works
with a remote shell.
Signed-off-by: Jan Lindemann <jan@janware.com>
"getting value xxx for project" is a prominent debug log message but
ugly. Beautify to e.g.:
Lookup jw-fail2ban -> jw-pkg / version
Meaning project "jw-fail2ban" looks up the value for key "version" in
project "jw-pkg".
Signed-off-by: Jan Lindemann <jan@janware.com>
DistroBase's option --id is now redundant to the new global option
--distro-id in the App class, so remove --id. The only added value
DistroBase then brings to the table is its .distro property, which
can be provided by App just fine at this point, given that App has
all it needs to construct a Distro object, so add .distro to App and
remove the entire DistroBase class.
For convenience, also make App.distro available as a newly added
cmds.Cmd.distro property. This also obviates the need for the
distro-related properties in the .distro.Cmd class, remove all that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the --verbose global option, which is made available as the
App.verbose property.
Some functions still take a verbose parameter, but the type of these
parameters is converted from bool to bool|None. The idea is that, if
they are None, their verbosity falls back to the global default.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the global options --distro-id and --interactive, and expose them
as properties App.distro_id and App.interactive, respectively.
Signed-off-by: Jan Lindemann <jan@janware.com>
get-os.sh returned "suse" for SuSE-like distros, and that seems more
appropriate since SLES is not OpenSUSE but should share and ID with
other SuSE variants.
Signed-off-by: Jan Lindemann <jan@janware.com>
In addition to the existing propert distro_id, add the properties
distro_codename, distro_name, distro_cascade and distro_gnu_triplet.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add more fields to the OS cascade returned by App.os_cascade, based
on the ID field in /etc/os-release. This includes some new ones,
prefixed by pkg-, revealing which package format is used.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add @property App.distro_id, returning the ID field of
/etc/os-release. It is supposed to be the most specific part of the
OS cascade.
Signed-off-by: Jan Lindemann <jan@janware.com>
Major - but not yet sufficient - code beautification starting from
jw.pkg.App.
- Make more methods private
- Rename methods to be more self-explanatory
- Same for method arguments, notably clean up some inconsistent
uses of "module" vs "project"
- Add more type hints
Fix API breakage in the command modules.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the Enum "Scope" to denote the scope argument of
jw.pkg.App.get_modules_from_project_txt(), because it explains itself
better than an integer.
Signed-off-by: Jan Lindemann <jan@janware.com>
ResultCache is a home-grown result cache. The @lru_cache decorator,
now available in Python 3, accomplishes the same thing, so try to
ditch ResultCache for it.
Sadly, this doesn't entirely work as of now, because it uses hash()
to hash the arguments, which won't work for the two list-type
arguments to add_modules_from_project_txt() (buf and visited).
Signed-off-by: Jan Lindemann <jan@janware.com>
