cmd_input is passed as None to _run(), which is legal, but then used
in a call to cmd_run(), which is a public API and, hence, illegal.
Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
App.distro_info() accepts and returns str instances, interpret
anything passed as fmt parameter which is not a str as iterable, and
return lists of expanded strings in that case.
Signed-off-by: Jan Lindemann <jan@janware.com>
This reverts commit 04fef1e67a.
Reusing AsyncSSH's connection is fine and fast, but only if it's not
combined with the AsyncRunner. See commit 67e51cf0 why it was
introduced in the first place, along with a reasoning why it may be a
bad idea. Looks like we're now reaping what we sowed.
The current plan to get this to fly is to sprinkle async / await all
over the code paths to App.os_release(). That is a lot of churn, so
postpone and revert for now to keep CI working.
File "~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/ec/ssh/AsyncSSH.py", line 463, in _run_ssh
return await self._run_on_conn(
^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
File "~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/ec/ssh/AsyncSSH.py", line 403, in _run_on_conn
proc = await conn.create_process(
^^^^^^^^^^^^^^^^^^^^^^^^^^
...<7 lines>...
)
^
File "/usr/lib/python3.13/site-packages/asyncssh/connection.py", line 4492, in create_process
chan, process = await self.create_session(
^^^^^^^^^^^^^^^^^^^^^^^^^^
SSHClientProcess, *args, **kwargs) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/connection.py", line 4385, in create_session
session = await chan.create(session_factory, command, subsystem,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
bool(self._agent_forward_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/channel.py", line 1149, in create
packet = await self._open(b'session')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/asyncssh/channel.py", line 717, in _open
return await self._open_waiter
^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Task <Task pending name='Task-1' coro=<App.__run() running at ~/local/src/jw.dev/proj/jw-pkg/scripts/jw/pkg/lib/App.py:137> cb=[_run_until_complete_cb() at /usr/lib64/python3.13/asyncio/base_events.py:181]> got Future <Future pending> attached to a different loop
Signed-off-by: Jan Lindemann <jan@janware.com>
Expand platform macros (as in %{codename}) in the src and dst command
arguments. Expansion can be turned off by -F --fixed-strings.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a method distro_info() to App, essentially providing CmdInfo's
macro-expansion of platform information to other interested code.
Signed-off-by: Jan Lindemann <jan@janware.com>
As of now, install() passes the "names" parameter on to _install(),
which is expected to pass the list of package names on to the package
manager for having it handle package download and installation. This
commit makes it easier for Distro instances to support directly
installing packages via an URL instead by providing a few callback
methods to be overridden, in case the package manager doesn't handle
package URLs the same way as package names.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add class Curl as the first pure FileTransfer class without _run()
/ _sudo(). It doesn't use any PycURL / libcurl-like binding, but runs
the curl binary in a subprocess. That looks the most portable still.
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecContext has get() / _get() and put() / _put(), which make a fine
API for a file transfer class. A class supporting file transfer
should not, however, be forced to implement _run() and _sudo(), so
place this functionality in a new class FileTransfer, and derive
ExecContext from it.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add lib.base to provide basic definitions.
For now, move the definiions of Result, Input and InputMode from
ExecContext into lib.base. Having to import them from the ExecContect
module is too heavy-handed for those simple types.
Signed-off-by: Jan Lindemann <jan@janware.com>
With CmdCopy as test case and ExecContext.close() in place, we can
actually implement connection reuse, so do it for AsyncSSH.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add CmdCopy, designed to copy data from the filesystem to another
location in the filesystem. Not necessarily local file systems, the
URLs can be all URLs supported by ExecContext.run().
Signed-off-by: Jan Lindemann <jan@janware.com>
Add a new group of commands - "posix". The current command categories
are not a good fit for that: "projects" is for CI, "distro" is
distribution-specific for CD, and secrets is for handling secrets
specifically. Introduce the more general command group "posix", a
class of commands not POSIX compliant in the exposed API, but
primarily using POSIX utilities as workhorse.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add copy(src_uri, dst_uri), instatiating two ExecContext instances,
and doing the obvious with them - copying from src_uri to dst_uri.
Signed-off-by: Jan Lindemann <jan@janware.com>
The property App.__os_release uses _run(['cat', '/etc/os-release']),
use ExecContext.get() instead as the default way to fetch content.
Signed-off-by: Jan Lindemann <jan@janware.com>
App currently has no hook to close async resources. Call it as
context manager, so that __aexit__() gets invoked if
run_sub_commands() exits.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add ExecContext.close() as a hook to clean up async resources living
longer than an ExecContext method call.
Also, implement __aenter__() and __aexit__(), to allow using
ExecContext as context manager. close() is invoked it goes out of
scope.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add wrapper methods get() and put(), plus their wrapped methods
_get() and _put(). The wrapped methods have default implementations,
using POSIX utilities on the target machine over _run().
Signed-off-by: Jan Lindemann <jan@janware.com>
ExecContext.create() relies on properly formed URLs with a schema for
deciding which backend gets created. Create a Local instance if an
URL doesn't have schema.
Signed-off-by: Jan Lindemann <jan@janware.com>
The Input instance passed as cmd_input to ExecContext.run() and
.sudo() currently may be of type str. Allow to pass bytes, too.
At the same time, disallow None to be passed as cmd_input. Force the
caller to be more explicit how it wants input to be handled, notably
with respect to interactivity.
Along the way fix a bug: Content in cmd_input should result in
CallContext.interactive == False but doesn't. Fix that.
Signed-off-by: Jan Lindemann <jan@janware.com>
- Move _sudo() above sudo()
To have a pattern in lib.ExecContext and avoid future churn: If a
public wrapper calls a protected method, define the protected
method above the respective wrapper.
- sudo(): Make cmd_input default equal to run(): InputMode.OptInteractive
- CallContext: Expose parameters throw, wd, cmd as properties for
later use
Signed-off-by: Jan Lindemann <jan@janware.com>
CmdCanonicalizeRemotes / canonicalize-remotes and the respective
target in topdir.mk remove the /srv/git portion from all remotes'
URLs pointing to git.janware.com.
Signed-off-by: Jan Lindemann <jan@janware.com>
Without --backtrace, the outmost try-catch block logs exceptions
plainly as their text. If it catches a key error, the exception text
only consists of the key itself, which can be easily mistaken for a
normal program output, so prefix it with a "Failed:".
Signed-off-by: Jan Lindemann <jan@janware.com>
To be able to use secret handling code from other modules, move the
bulk of it from the "secrets"-command centric implementation in
cmds.secrets.Cmd into a new module cmds.secrets.lib.util.
Signed-off-by: Jan Lindemann <jan@janware.com>
_run_ssh() of ssh.Exec doesn't pass throw=False to run_cmd(), which
makes it throw exceptions, and effectively strips the caller of any
chance to get hold of stdout and stderr. Pass throw=False and let
run() decide according the the caller-provided throw parameter
whether or not a problem should propagate up as exception or return
value.
Signed-off-by: Jan Lindemann <jan@janware.com>
ssh_client() tries a predefined order of client class implementations
until it finds a workable candidate. For testing all, it's desirable
to be able to target the exact class. Add a "type" parameter to
achieve that.
I'm aware that type is also a function. But the semantics look so
compelling to me that I'm using the variable name anyway.
Signed-off-by: Jan Lindemann <jan@janware.com>
Naively join()ing a command list to be executed remotely via SSH also
quotes shell operators which doesn't work, of course. Work around
that. The workaround will not always work but covers lots of cases.
Signed-off-by: Jan Lindemann <jan@janware.com>
Instantiating a SSHClient-derived class with an invalid or missing
uri parameter is accepted and fails later down the road. Raise an
Exception early on to make the error log more comprehensible.
Signed-off-by: Jan Lindemann <jan@janware.com>
The SSHClient classes Paramiko and Exec are exported via # export.
This is a bad idea, because if Paramiko is not installed, none of the
other's can be instantiated either: On the attempt to load them,
__init__.py is loaded first and fails. SSHClient.ssh_client() knows
what to do, no need to auto-import them into the lib.ec.ssh module.
Signed-off-by: Jan Lindemann <jan@janware.com>
Installation type SCRIPT has beed disabled long ago because of its
overlap with EXE. Remove the dead code around it.
Signed-off-by: Jan Lindemann <jan@janware.com>
/usr/bin/file <candidate> | grep text is used to detect if a file is
a text file or not. Replace that with grep -I., because that adds
some files left out by /usr/bin/file, notably systemd service files.
Signed-off-by: Jan Lindemann <jan@janware.com>
