__init_askpass() generates an askpass script whose echo statement
embeds a newline inside the quoted password: echo -n "PASSWORD
". Bash happily spans the quote over the newline, and the script
therefore prints the password followed by a trailing newline.
run_askpass() returns that output verbatim, so the password passed
on carries a newline and authentication fails.
Write the script so it prints the password and nothing else.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_run_ssh() waits for the remote process's exit status before reading
stdout and stderr. When a command produces more output than the
channel's flow-control window can hold, the server stops sending, the
remote process blocks on its write and never exits, and
recv_exit_status() blocks forever.
Drain stdout and stderr to EOF first - the channels close when the
process exits, so reading them also implies completion - and only
then query the exit status.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_run_ssh() writes cmd_input to the remote stdin channel but never
closes the write side. The channel stays open until the client
process exits, so a remote command that reads stdin (cat, a login
shell, ...) never sees EOF and blocks forever - including for
cmd_input = None, which is supposed to mean non-interactive with
stdin from /dev/null.
Call shutdown_write() on the channel after writing the input, or
immediately when there is none, so the remote command gets EOF.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
__client() connects using only the URI's hostname and username. The
URI's port is ignored, so ssh://host:2222/... ends up connecting to
port 22, and a password carried in the URI is never passed to
paramiko, so URI-based password authentication cannot work. The Exec
and AsyncSSH clients both honor the port and the password.
Pass the port and the password to connect(). The port argument is
omitted entirely when the URI carries no port, because
getaddrinfo() would interpret a None port as service port 0; without
the argument, paramiko falls back to its default of 22.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
_connect_kwargs(hide_secrets = True) is used to log the connection
parameters when a connection fails, without leaking the password. The
filtered dictionary is built before the password is replaced with
'<hidden>', and the replacement is applied to the local kwargs
dictionary afterwards, after the filtered copy has already been made.
The dictionary that ends up in the log therefore still contains the
real password.
Hide the password before building the filtered dictionary.
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.84.2
Signed-off-by: Jan Lindemann <jan@janware.com>
The diff-all and diff targets diff all projects in the workspace
without filtering.
Add a target "diff-projects". It sets PGIT_SH_PROJECTS to the list of
projects from build-order, limiting the diff to the dependency
closure around $(PROJECTS).
Signed-off-by: Jan Lindemann <jan@janware.com>
If /usr/bin/isort is found, run it during "make format" to get a
defined way the imports are sorted. tool.isort in pyproject.toml is
updated to match the other fixers.
Commit the fallout of this change. Running the other fixers alone
doesn't change the formatting, so this should be safe.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add type annotations and casts to functions that were returning Any
where a specific type was declared, satisfying the new warn_return_any
mypy rule.
Fixes:
- log.py: get_caller_pos return type via cast
- AsyncRunner.py: cast T for fut.result()
- util.py: cast for getattr result, str() for args.username
- FileContext.py: verbose_default bool annotation
- SSHClient.py: cast SSHClient for dynamic import
- lib/App.py: cast ArgumentParser, add return types to inner funcs
- pm/rpm.py, dpkg.py: cast Iterable[Package]
- App.py: cast for self.args.func(), add return types to inner funcs
- BaseCmdPkgRelations.py: cast str for args.delimiter
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev 0.81.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Add missing type annotations to functions that are called from typed
contexts, satisfying the new disallow_untyped_calls mypy rule.
Fixes:
- Local.py: __log() with typed parameters
- lib/App.py: _add_arguments(), add_cmd_to_parser(), add_cmds_to_parser()
- pm/rpm.py, dpkg.py: meta_map() return type
- Exec.py: __init_askpass() return type
- App.py: strip_module_from_spec(), __get_project_refs_cached(),
ResultCache.__init__ and run(), _add_arguments()
- Added Collection type for truthy-iterable compliance
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev 0.81.1
Signed-off-by: Jan Lindemann <jan@janware.com>
Some '# type: ignore' comments are needed because they complain about missing
but optional third-party packages: argcomplete, paramiko, asyncssh. The next
commit will enable warn_unused_ignores, and since nor mypy nor pyright have a
way of knowing that this is a tolerable lack of packages, this commit teaches
them in advance.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add explicit type arguments to all generic type annotations that were
previously bare, satisfying the new disallow_any_generics mypy rule.
Fixes:
- Distro.py: Iterable[str] for expand_macros fmt parameter
- Cmd.py: Types[Any] for add_subcommands cmds parameter
- AsyncSSH.py: dict[str, Any] for _connect_kwargs return type
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL and pi.dev 0.81.1
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit adds @override decorators to approximately 300 methods
across 76 files that inherit from base classes such as AbstractCmd,
FileContext, ExecContext, Distro, SSHClient, and others.
The decorator ensures the type checker can verify that overridden
methods have compatible signatures and prevents accidental shadowing
of inherited methods without intent.
Files modified include command classes, library modules, distro
implementations, and SSH client implementations.
Assisted-by: unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_NL with pi.dev v
Signed-off-by: Jan Lindemann <jan@janware.com>
A couple of "# export" markers after function prototypes have been
pushed along with the closing parenthesis onto the wrong line by the
code formatter, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
Tolera$te missing paramiko imports. jw-pkg is designed to
work with what it finds and use plain /usr/bin/ssh if need
be.
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit adds more tweaks to shell command output in order to make
it nicer. The biggest patch is in Result.__summarize(), which makes
it more versatile, and allows removal of some code in SSHClient.
App sees some independent, minor result format beautification.
Signed-off-by: Jan Lindemann <jan@janware.com>
Commands executed by ExecContext and its derived classes don't
populate the "cmd" parameter of "Result"'s constructor. Fixing that
makes for nicer error messages.
Signed-off-by: Jan Lindemann <jan@janware.com>
Accept if AsyncSSH is missing. The package would be nice to have,
i.e. a good candidate for a "recommends" section, but until there's
support for that, better be able to do without and fall back to
command-line ssh.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add new ruff rules and fix their fallout:
future-annotations = true
select = [
"TC", # type-checking import placement rules
"FA", # future annotations rules
]
This comprises:
- Streamline imports and exports in cmds.xxx.Cmd
- Import base class as "Base"
- Export types Cmd and Parent via __all__
- Move all types imported only for annotation below TYPE_CHECKING
- Use "from __future__ import annotations" all over the place
Signed-off-by: Jan Lindemann <jan@janware.com>
The previous commits have put rules for linting and formatting via
ruff, yapf, mypy and pyright into place. They are checked with the
make check target, and this commit adds the fixes for the target to
succeed.
It does some refactoring where type checking dug up dirty bits, and
also adds lots of churn in the Python code. To a good deal, that's
owed to mere formatting changes. It would have been better to
seperate those from syntax and refactoring fixes into multiple
commits, so that the interesting changes don't drown in the
formatting nose. However, that would have been a lot of additional
work only to be thrown away by later commits, hence this commit has a
big diff in one piece. The size of the diff is regrettable but
hopefully a one-off: What it buys is automatic format checking for CI
and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
- ruff check .
- mypy .
- pyright
- Format check:
- yapf --diff --recursive .
The refactoring includes:
- Turn the Result class into a more elaborate object, capable of
doing more heavy lifting around stderr and stdout decoding,
summarizing outcome, and matching error strings.
Aside from fixing broken type checks, this also removes lots of
boilerplate calling code which is currently used for handling
possible call outcome scenarios. Trying to access an inexistent,
decoded string should raise a meaningful exception by itself now,
which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
- Add the AbstractCmd class above Cmd. This is necessary because
the checker rightfully complains it can't instantiate a Cmd
instance where constructor arguments were needed. They never
were, but the type used at the instantiating code's location in
jw.pkg.App so claims.
- Lots of sub- and sub-subcommands are derived from the base
class of the invoking command. That provides some properties
shared across the ancestor hierarchy of a command, but is
semantically unsound. Fix that by introducing jw.pkg.BaseCmd
class as a place to provide basic helpers shared across all
commands used in a jw.pkg.App's context, and derive all command
classes from that afresh. The parent command is still reachable
via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor
tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
The line was needed by Python 2 which is not supported anylonger.
For Python 3, the default encoding is UTF-8, anyway.
- Allow to run "make py-format" without having it produce any
changes. It's basically "yapf --in-place --recursive ." with some
code style settings, see conf/topdir/pyproject.toml. The settings
may be debatable. I've had custom tweaks in place on that target,
too, but then again, IDEs would have more hassle to integrate
that.
- Introduce a 88 character line length limit
- One import per line, reshuffle them semantically, see
[tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
if TYPE_CHECKING
- Spaces around assignments accounts for much churn. Having having
no spaces in inline parameter list assignments and default
parameter values would arguably be more compact where it's
useful. On the other hand, I have not found a code formatter
which allows spaces around assignments in parameter lists broken
into one per line and that's often better than a wall of text.
- Add two spaces before # export, as this seems to be mandated by
PEP-8
- Use single quotes by default
Signed-off-by: Jan Lindemann <jan@janware.com>
Make FileContext.is_dir() usable:
- Add follow_symlinks parameter meant to do the obvious
- Fix missing await
Signed-off-by: Jan Lindemann <jan@janware.com>
Add an async open() method which should allow to do what __init__()
couldn't, because it's not async, and to match the already existing
.close(). It's called by __aenter__() __aexit__() if the FileContext
is instantiated as context manager, or at will when the user finds it
a good idea.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add .mkdir() to the API which should do the expected, and implement
it in ExecContext and Local specializations.
Signed-off-by: Jan Lindemann <jan@janware.com>
- Remove _run_on_conn() because it doesn't add any value
- Add verbose try-except block around connect()
- Add try-except block around failing close
- Prefix private member variables with "__"
Signed-off-by: Jan Lindemann <jan@janware.com>
Commit a19679fec reverted the first attempt to make AsyncSSH reuse
one connection during an instance lifetime. That failed because a lot
of distribution-specific properties were filled in a new event loop
thread started by AsyncRunner, and AsyncSSH didn't like that.
The last commit provided the needed properties as members of the
Distro class. This commit is the second part of the solution: Keep
one connection around as a class member and reuse it on every _run()
invocation.
Signed-off-by: Jan Lindemann <jan@janware.com>
Apply some style changes:
- Replace double by single quotes for consistency
- Add spaces around equal signs in long parameter lists
Signed-off-by: Jan Lindemann <jan@janware.com>
The name of the env parameter to ExecContext.run() and .sudo() is not
descriptive enough for which environment is supposed to be modified
and how, so rename and split it up as follows:
- .run(): env -> mod_env
- .sudo(): env -> mod_env_sudo and mod_env_cmd
The parameters have the following meaning:
- "mod_env*" means that the environment is modified, not replaced
- "mod_env" and "mod_env_cmd" modify the environment "cmd" runs in
- "mod_env_sudo" modifies the environment sudo runs in
Fix the fallout of the API change all over jw-pkg.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the property .username, backed by the protected _username()
callback. It should return the user run()'s cmd parameter is executed
as.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the following methods, meant to do the obvious:
unlink(self, path: str) -> None
erase(self, path: str) -> None
rename(self, src: str, dst: str) -> None
mktemp(self, tmpl: str, directory: bool=False) -> None
chown(self, path: str, owner: str|None=None, group: str|None=None) -> None
chmod(self, path: str, mode: int) -> None
stat(self, path: str, follow_symlinks: bool=True) -> StatResult
file_exists(self, path: str) -> bool
is_dir(self, path: str) -> bool
All methods are async and call their protected counterpart, which is
designed to be overridden. If possible, default implementations do
something meaningful, if not, they just raise plain
NotImplementedError.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add the parameter "atomic" to put() / _put(). If instructs the
implementation to take extra precautions to make sure the operation
either succeeds or fails entirely, i.e. doesn't leave a broken target
file behind.
Signed-off-by: Jan Lindemann <jan@janware.com>
Rename class FileTransfer to FileContext because that's the better
name. It's the base class of ExecContext and also a context.
Signed-off-by: Jan Lindemann <jan@janware.com>
An env argument environment passed to Local._run() entirely replaces
the environment. Make it modify the enviroment instead.
Signed-off-by: Jan Lindemann <jan@janware.com>
AsyncSSH's implementation already supports modifying the execution
environment via env, so declare it to the base class with Caps.Env.
Signed-off-by: Jan Lindemann <jan@janware.com>
Exec's _run_ssh() ignores its "interactive" parameter and uses the
instances' default instead, fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
cmd_input is passed as None to _run(), which is legal, but then used
in a call to cmd_run(), which is a public API and, hence, illegal.
InputMode.NonInteractive should be used instead, do that.
Signed-off-by: Jan Lindemann <jan@janware.com>
run_cmd() is a thin layer over the public ExecContext API, which
falls back to using a Local instance if not other ExecContext is
specified explicitly. Both the default Local context as the
subsequent call to run() should have the same idea about
interactivity, so allowing to specify it in two parameters
("interactive" and "cmd_input") is a bad idea. Remove "interactive".
Signed-off-by: Jan Lindemann <jan@janware.com>
Allow to configure via the environment which class ssh_client()
picks. Can currently be exec, asyncssh, paramiko or a comma-separated
search list. The list will be tried through until a class is found
that can be instantiated.
Signed-off-by: Jan Lindemann <jan@janware.com>
cmd_input is passed as None to _run(), which is legal, but then used
in a call to cmd_run(), which is a public API and, hence, illegal.
Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
This reverts commit 04fef1e67a.
Reusing AsyncSSH's connection is fine and fast, but only if it's not
combined with the AsyncRunner. See commit 67e51cf0 why it was
introduced in the first place, along with a reasoning why it may be a
bad idea. Looks like we're now reaping what we sowed.
The current plan to get this to fly is to sprinkle async / await all
over the code paths to App.os_release(). That is a lot of churn, so
postpone and revert for now to keep CI working.
File "~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/ec/ssh/AsyncSSH.py", line 463, in _run_ssh
return await self._run_on_conn(
^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
File "~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/ec/ssh/AsyncSSH.py", line 403, in _run_on_conn
proc = await conn.create_process(
^^^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
File "/usr/lib/python3.13/site-packages/asyncssh/connection.py", line 4492, in create_process
chan, process = await self.create_session(
^^^^^^^^^^^^^^^^^^^^^^^^^^
SSHClientProcess, *args, **kwargs) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/connection.py", line 4385, in create_session
session = await chan.create(session_factory, command, subsystem,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
bool(self._agent_forward_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/channel.py", line 1149, in create
packet = await self._open(b'session')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/channel.py", line 717, in _open
return await self._open_waiter
^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Task <Task pending name='Task-1' coro=<App.__run() running at ~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/App.py:137> cb=[_run_until_complete_cb() at /usr/lib64/python3.13/asyncio/base_events.py:181]> got Future <Future pending> attached to a different loop
Signed-off-by: Jan Lindemann <jan@janware.com>
Add class Curl as the first pure FileTransfer class without _run()
/ _sudo(). It doesn't use any PycURL / libcurl-like binding, but runs
the curl binary in a subprocess. That looks the most portable still.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add lib.base to provide basic definitions.
For now, move the definiions of Result, Input and InputMode from
ExecContext into lib.base. Having to import them from the ExecContect
module is too heavy-handed for those simple types.
Signed-off-by: Jan Lindemann <jan@janware.com>
With CmdCopy as test case and ExecContext.close() in place, we can
actually implement connection reuse, so do it for AsyncSSH.
Signed-off-by: Jan Lindemann <jan@janware.com>
