App.distro_info() accepts and returns str instances, interpret anything passed as fmt parameter which is not a str as iterable, and return lists of expanded strings in that case.
Reusing AsyncSSH's connection is fine and fast, but only if it's not combined with the AsyncRunner. See commit 67e51cf0 why it was introduced in the first place, along with a reasoning why it may be a bad idea. Looks like we're now reaping what we sowed.
The current plan to get this to fly is to sprinkle async / await all over the code paths to App.os_release(). That is a lot of churn, so postpone and revert for now to keep CI working.
File "~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/ec/ssh/AsyncSSH.py", line 463, in _run_ssh
return await self._run_on_conn(
^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
File "~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/ec/ssh/AsyncSSH.py", line 403, in _run_on_conn
proc = await conn.create_process(
^^^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
File "/usr/lib/python3.13/site-packages/asyncssh/connection.py", line 4492, in create_process
chan, process = await self.create_session(
^^^^^^^^^^^^^^^^^^^^^^^^^^
SSHClientProcess, *args, **kwargs) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/connection.py", line 4385, in create_session
session = await chan.create(session_factory, command, subsystem,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
bool(self._agent_forward_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/channel.py", line 1149, in create
packet = await self._open(b'session')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/channel.py", line 717, in _open
return await self._open_waiter
^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Task <Task pending name='Task-1' coro=<App.__run() running at ~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/App.py:137> cb=[_run_until_complete_cb() at /usr/lib64/python3.13/asyncio/base_events.py:181]> got Future <Future pending> attached to a different loop
As of now, install() passes the "names" parameter on to _install(), which is expected to pass the list of package names on to the package manager for having it handle package download and installation. This commit makes it easier for Distro instances to support directly installing packages via an URL instead by providing a few callback methods to be overridden, in case the package manager doesn't handle package URLs the same way as package names.
Add class Curl as the first pure FileTransfer class without _run() / _sudo(). It doesn't use any PycURL / libcurl-like binding, but runs the curl binary in a subprocess. That looks the most portable still.
ExecContext has get() / _get() and put() / _put(), which make a fine API for a file transfer class. A class supporting file transfer should not, however, be forced to implement _run() and _sudo(), so place this functionality in a new class FileTransfer, and derive ExecContext from it.
For now, move the definiions of Result, Input and InputMode from ExecContext into lib.base. Having to import them from the ExecContect module is too heavy-handed for those simple types.
Add CmdCopy, designed to copy data from the filesystem to another location in the filesystem. Not necessarily local file systems, the URLs can be all URLs supported by ExecContext.run().
Add a new group of commands - "posix". The current command categories are not a good fit for that: "projects" is for CI, "distro" is distribution-specific for CD, and secrets is for handling secrets specifically. Introduce the more general command group "posix", a class of commands not POSIX compliant in the exposed API, but primarily using POSIX utilities as workhorse.
Add wrapper methods get() and put(), plus their wrapped methods _get() and _put(). The wrapped methods have default implementations, using POSIX utilities on the target machine over _run().
ExecContext.create() relies on properly formed URLs with a schema for deciding which backend gets created. Create a Local instance if an URL doesn't have schema.
The Input instance passed as cmd_input to ExecContext.run() and .sudo() currently may be of type str. Allow to pass bytes, too.
At the same time, disallow None to be passed as cmd_input. Force the caller to be more explicit how it wants input to be handled, notably with respect to interactivity.
Along the way fix a bug: Content in cmd_input should result in CallContext.interactive == False but doesn't. Fix that.
To have a pattern in lib.ExecContext and avoid future churn: If a
public wrapper calls a protected method, define the protected
method above the respective wrapper.
- sudo(): Make cmd_input default equal to run(): InputMode.OptInteractive
- CallContext: Expose parameters throw, wd, cmd as properties for
later use
CmdCanonicalizeRemotes / canonicalize-remotes and the respective target in topdir.mk remove the /srv/git portion from all remotes' URLs pointing to git.janware.com.
Without --backtrace, the outmost try-catch block logs exceptions plainly as their text. If it catches a key error, the exception text only consists of the key itself, which can be easily mistaken for a normal program output, so prefix it with a "Failed:".
To be able to use secret handling code from other modules, move the bulk of it from the "secrets"-command centric implementation in cmds.secrets.Cmd into a new module cmds.secrets.lib.util.
_run_ssh() of ssh.Exec doesn't pass throw=False to run_cmd(), which makes it throw exceptions, and effectively strips the caller of any chance to get hold of stdout and stderr. Pass throw=False and let run() decide according the the caller-provided throw parameter whether or not a problem should propagate up as exception or return value.
ssh_client() tries a predefined order of client class implementations until it finds a workable candidate. For testing all, it's desirable to be able to target the exact class. Add a "type" parameter to achieve that.
I'm aware that type is also a function. But the semantics look so compelling to me that I'm using the variable name anyway.
Naively join()ing a command list to be executed remotely via SSH also quotes shell operators which doesn't work, of course. Work around that. The workaround will not always work but covers lots of cases.
Instantiating a SSHClient-derived class with an invalid or missing uri parameter is accepted and fails later down the road. Raise an Exception early on to make the error log more comprehensible.
The SSHClient classes Paramiko and Exec are exported via # export. This is a bad idea, because if Paramiko is not installed, none of the other's can be instantiated either: On the attempt to load them, __init__.py is loaded first and fails. SSHClient.ssh_client() knows what to do, no need to auto-import them into the lib.ec.ssh module.
/usr/bin/file <candidate> | grep text is used to detect if a file is a text file or not. Replace that with grep -I., because that adds some files left out by /usr/bin/file, notably systemd service files.
jw-pkg is copied into $(TOPDIR)/bin during build, that's wrong. Write a rule precisely targeted at installing /usr/bin/jw-pkg, and cut all the scripts.mk machinery.
Also, make jw-pkg a relative link to avoid the respective RPM warning.
run_curl() has no clear API of whether or not the return values should be decoded. It has parse_json, which should imply decoding, but there's no way to specify that explicitly. Moreover, when it tries to decode, it decodes on the coroutine returned from run_cmd(), not the awaited coroutine return value.
Add a decode parameter, defaulting to False, change the parse_json parameter's default from True to False, and fix the run_cmd() return value evaluation.
