FileContext offers get(), put(), unlink(), mkdir(), and stat(), but no way
to list the entries of a directory, so consumers that need the names of the
entries under a path in a context have no portable means to obtain them.
Add readdir(), returning Iterable[str], following the _readdir() ->
readdir() pattern used by the other operations: the public method chroots
the path via _chroot() and delegates to the _readdir() hook, which raises
NotImplementedError by default.
Implement _readdir() in the context families:
- Local._readdir(): os.listdir()
- ExecContext._readdir(): runs ls -1A and splits the output into one entry
per line; inherited by SSHClient and the other command-based contexts
- Curl._readdir(): raises OSError with errno.ENOSYS
Add unit tests covering Local (listing, hidden entries, missing directory)
and Curl (ENOSYS).
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.87.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Add type annotations and casts to functions that were returning Any
where a specific type was declared, satisfying the new warn_return_any
mypy rule.
Fixes:
- log.py: get_caller_pos return type via cast
- AsyncRunner.py: cast T for fut.result()
- util.py: cast for getattr result, str() for args.username
- FileContext.py: verbose_default bool annotation
- SSHClient.py: cast SSHClient for dynamic import
- lib/App.py: cast ArgumentParser, add return types to inner funcs
- pm/rpm.py, dpkg.py: cast Iterable[Package]
- App.py: cast for self.args.func(), add return types to inner funcs
- BaseCmdPkgRelations.py: cast str for args.delimiter
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev 0.81.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Add missing type annotations to functions that are called from typed
contexts, satisfying the new disallow_untyped_calls mypy rule.
Fixes:
- Local.py: __log() with typed parameters
- lib/App.py: _add_arguments(), add_cmd_to_parser(), add_cmds_to_parser()
- pm/rpm.py, dpkg.py: meta_map() return type
- Exec.py: __init_askpass() return type
- App.py: strip_module_from_spec(), __get_project_refs_cached(),
ResultCache.__init__ and run(), _add_arguments()
- Added Collection type for truthy-iterable compliance
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev 0.81.1
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit adds @override decorators to approximately 300 methods
across 76 files that inherit from base classes such as AbstractCmd,
FileContext, ExecContext, Distro, SSHClient, and others.
The decorator ensures the type checker can verify that overridden
methods have compatible signatures and prevents accidental shadowing
of inherited methods without intent.
Files modified include command classes, library modules, distro
implementations, and SSH client implementations.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL with pi.dev v
Signed-off-by: Jan Lindemann <jan@janware.com>
Commands executed by ExecContext and its derived classes don't
populate the "cmd" parameter of "Result"'s constructor. Fixing that
makes for nicer error messages.
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commits have put rules for linting and formatting via
ruff, yapf, mypy and pyright into place. They are checked with the
make check target, and this commit adds the fixes for the target to
succeed.
It does some refactoring where type checking dug up dirty bits, and
also adds lots of churn in the Python code. To a good deal, that's
owed to mere formatting changes. It would have been better to
seperate those from syntax and refactoring fixes into multiple
commits, so that the interesting changes don't drown in the
formatting nose. However, that would have been a lot of additional
work only to be thrown away by later commits, hence this commit has a
big diff in one piece. The size of the diff is regrettable but
hopefully a one-off: What it buys is automatic format checking for CI
and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
- ruff check .
- mypy .
- pyright
- Format check:
- yapf --diff --recursive .
The refactoring includes:
- Turn the Result class into a more elaborate object, capable of
doing more heavy lifting around stderr and stdout decoding,
summarizing outcome, and matching error strings.
Aside from fixing broken type checks, this also removes lots of
boilerplate calling code which is currently used for handling
possible call outcome scenarios. Trying to access an inexistent,
decoded string should raise a meaningful exception by itself now,
which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
- Add the AbstractCmd class above Cmd. This is necessary because
the checker rightfully complains it can't instantiate a Cmd
instance where constructor arguments were needed. They never
were, but the type used at the instantiating code's location in
jw.pkg.App so claims.
- Lots of sub- and sub-subcommands are derived from the base
class of the invoking command. That provides some properties
shared across the ancestor hierarchy of a command, but is
semantically unsound. Fix that by introducing jw.pkg.BaseCmd
class as a place to provide basic helpers shared across all
commands used in a jw.pkg.App's context, and derive all command
classes from that afresh. The parent command is still reachable
via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor
tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
The line was needed by Python 2 which is not supported anylonger.
For Python 3, the default encoding is UTF-8, anyway.
- Allow to run "make py-format" without having it produce any
changes. It's basically "yapf --in-place --recursive ." with some
code style settings, see conf/topdir/pyproject.toml. The settings
may be debatable. I've had custom tweaks in place on that target,
too, but then again, IDEs would have more hassle to integrate
that.
- Introduce a 88 character line length limit
- One import per line, reshuffle them semantically, see
[tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
if TYPE_CHECKING
- Spaces around assignments accounts for much churn. Having having
no spaces in inline parameter list assignments and default
parameter values would arguably be more compact where it's
useful. On the other hand, I have not found a code formatter
which allows spaces around assignments in parameter lists broken
into one per line and that's often better than a wall of text.
- Add two spaces before # export, as this seems to be mandated by
PEP-8
- Use single quotes by default
Signed-off-by: Jan Lindemann <jan@janware.com>
Make FileContext.is_dir() usable:
- Add follow_symlinks parameter meant to do the obvious
- Fix missing await
Signed-off-by: Jan Lindemann <jan@janware.com>
Add .mkdir() to the API which should do the expected, and implement
it in ExecContext and Local specializations.
Signed-off-by: Jan Lindemann <jan@janware.com>
The name of the env parameter to ExecContext.run() and .sudo() is not
descriptive enough for which environment is supposed to be modified
and how, so rename and split it up as follows:
- .run(): env -> mod_env
- .sudo(): env -> mod_env_sudo and mod_env_cmd
The parameters have the following meaning:
- "mod_env*" means that the environment is modified, not replaced
- "mod_env" and "mod_env_cmd" modify the environment "cmd" runs in
- "mod_env_sudo" modifies the environment sudo runs in
Fix the fallout of the API change all over jw-pkg.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the property .username, backed by the protected _username()
callback. It should return the user run()'s cmd parameter is executed
as.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the following methods, meant to do the obvious:
unlink(self, path: str) -> None
erase(self, path: str) -> None
rename(self, src: str, dst: str) -> None
mktemp(self, tmpl: str, directory: bool=False) -> None
chown(self, path: str, owner: str|None=None, group: str|None=None) -> None
chmod(self, path: str, mode: int) -> None
stat(self, path: str, follow_symlinks: bool=True) -> StatResult
file_exists(self, path: str) -> bool
is_dir(self, path: str) -> bool
All methods are async and call their protected counterpart, which is
designed to be overridden. If possible, default implementations do
something meaningful, if not, they just raise plain
NotImplementedError.
Signed-off-by: Jan Lindemann <jan@janware.com>
An env argument environment passed to Local._run() entirely replaces
the environment. Make it modify the enviroment instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add lib.base to provide basic definitions.
For now, move the definiions of Result, Input and InputMode from
ExecContext into lib.base. Having to import them from the ExecContect
module is too heavy-handed for those simple types.
Signed-off-by: Jan Lindemann <jan@janware.com>
The Input instance passed as cmd_input to ExecContext.run() and
.sudo() currently may be of type str. Allow to pass bytes, too.
At the same time, disallow None to be passed as cmd_input. Force the
caller to be more explicit how it wants input to be handled, notably
with respect to interactivity.
Along the way fix a bug: Content in cmd_input should result in
CallContext.interactive == False but doesn't. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Since commit 02697af5, ExecContext.run() returns bytes for stdout and
stderr and fixes that in calling code. The thing it did not fix was
the code calling run_cmd(), which also made return bytes. This commit
catches up on that.
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecContext's .sudo() omits many of run()'s parameters, and this
commit adds them. To avoid redundancy around repeating and massaging
the long parameter list of both functions and their return values, it
also adds some deeper changes:
- Make run(), _run(), sudo() and _sudo() always return instances of
Result. Before it was allowed to return a triplet of stdout,
stderr, and exit status.
- Have ExecContext stay out of the business of decoding the result
entirely. Result provides a convenience method .decode()
operating on stdout and stderr and leaves the decision to the
caller.
This entails miniscule adaptations in calling code, namely in
App.os_release, util.get_profile_env() and CmdListRepos._run().
- Wrap the _run() and _sudo() callbacks in a context manager object
of type CallContext to avoid code duplication.
- Consistently name the first argument to run(), _run(), sudo() and
_sudo() "cmd", not "args". The latter suggests that the caller is
omitting the executable, which is not the case.
Signed-off-by: Jan Lindemann <jan@janware.com>
Take implementation burden from the derived classes _run() callback
by moving the respective code into the run() wrapper methods of the
base class.
Signed-off-by: Jan Lindemann <jan@janware.com>
Take a positional uri argument to the constructor of ExecContext,
forcing SSHClient to follow suit. The latter was instantiated with a
hostname as only argument up to now, which still works as a special
case of an uri.
Signed-off-by: Jan Lindemann <jan@janware.com>
Don't mention "error" in log message for exit codes > 0 from spawned
processes, because sometimes they don't mean an error.
Signed-off-by: Jan Lindemann <jan@janware.com>
Remove defaults from protected callback function parameters. They
have to be decided by the base class's public API.
Signed-off-by: Jan Lindemann <jan@janware.com>
Allow to specify the ExecContext in a call to run_cmd(). This
effectively makes run_cmd() an thin wrapper around ExecContext.run(),
which is what's going to be used in the future. The wrapper is for
backwards-compatibility.
Signed-off-by: Jan Lindemann <jan@janware.com>
The code below lib.distro, as left behind by the previous commit, is
geared towards being directly used as a command-line API. This commit
introduces the abstract base class Distro, a proxy for
distribution-specific interactions. The proxy abstracts distro
specifics into an API with proper method prototypes, not
argparse.Namespace contents, and can thus be more easily driven by
arbitrary code.
The Distro class is initialized with a member variable of type
ExecContext, another new class introduced by this commit. It is
designed to abstract the communication channel to the distribution
instance. Currently only one specialization exists, Local, which
interacts with the distribution and root file system it is running
in, but is planned to be subclassed to support interaction via SSH,
serial, chroot, or chains thereof.
Signed-off-by: Jan Lindemann <jan@janware.com>
