As of now, install() passes the "names" parameter on to _install(), which is expected to pass the list of package names on to the package manager for having it handle package download and installation. This commit makes it easier for Distro instances to support directly installing packages via an URL instead by providing a few callback methods to be overridden, in case the package manager doesn't handle package URLs the same way as package names.
Add class Curl as the first pure FileTransfer class without _run() / _sudo(). It doesn't use any PycURL / libcurl-like binding, but runs the curl binary in a subprocess. That looks the most portable still.
ExecContext has get() / _get() and put() / _put(), which make a fine API for a file transfer class. A class supporting file transfer should not, however, be forced to implement _run() and _sudo(), so place this functionality in a new class FileTransfer, and derive ExecContext from it.
For now, move the definiions of Result, Input and InputMode from ExecContext into lib.base. Having to import them from the ExecContect module is too heavy-handed for those simple types.
Add CmdCopy, designed to copy data from the filesystem to another location in the filesystem. Not necessarily local file systems, the URLs can be all URLs supported by ExecContext.run().
Add a new group of commands - "posix". The current command categories are not a good fit for that: "projects" is for CI, "distro" is distribution-specific for CD, and secrets is for handling secrets specifically. Introduce the more general command group "posix", a class of commands not POSIX compliant in the exposed API, but primarily using POSIX utilities as workhorse.
Add wrapper methods get() and put(), plus their wrapped methods _get() and _put(). The wrapped methods have default implementations, using POSIX utilities on the target machine over _run().
ExecContext.create() relies on properly formed URLs with a schema for deciding which backend gets created. Create a Local instance if an URL doesn't have schema.
The Input instance passed as cmd_input to ExecContext.run() and .sudo() currently may be of type str. Allow to pass bytes, too.
At the same time, disallow None to be passed as cmd_input. Force the caller to be more explicit how it wants input to be handled, notably with respect to interactivity.
Along the way fix a bug: Content in cmd_input should result in CallContext.interactive == False but doesn't. Fix that.
To have a pattern in lib.ExecContext and avoid future churn: If a
public wrapper calls a protected method, define the protected
method above the respective wrapper.
- sudo(): Make cmd_input default equal to run(): InputMode.OptInteractive
- CallContext: Expose parameters throw, wd, cmd as properties for
later use
CmdCanonicalizeRemotes / canonicalize-remotes and the respective target in topdir.mk remove the /srv/git portion from all remotes' URLs pointing to git.janware.com.
Without --backtrace, the outmost try-catch block logs exceptions plainly as their text. If it catches a key error, the exception text only consists of the key itself, which can be easily mistaken for a normal program output, so prefix it with a "Failed:".
To be able to use secret handling code from other modules, move the bulk of it from the "secrets"-command centric implementation in cmds.secrets.Cmd into a new module cmds.secrets.lib.util.
_run_ssh() of ssh.Exec doesn't pass throw=False to run_cmd(), which makes it throw exceptions, and effectively strips the caller of any chance to get hold of stdout and stderr. Pass throw=False and let run() decide according the the caller-provided throw parameter whether or not a problem should propagate up as exception or return value.
ssh_client() tries a predefined order of client class implementations until it finds a workable candidate. For testing all, it's desirable to be able to target the exact class. Add a "type" parameter to achieve that.
I'm aware that type is also a function. But the semantics look so compelling to me that I'm using the variable name anyway.
Naively join()ing a command list to be executed remotely via SSH also quotes shell operators which doesn't work, of course. Work around that. The workaround will not always work but covers lots of cases.
Instantiating a SSHClient-derived class with an invalid or missing uri parameter is accepted and fails later down the road. Raise an Exception early on to make the error log more comprehensible.
The SSHClient classes Paramiko and Exec are exported via # export. This is a bad idea, because if Paramiko is not installed, none of the other's can be instantiated either: On the attempt to load them, __init__.py is loaded first and fails. SSHClient.ssh_client() knows what to do, no need to auto-import them into the lib.ec.ssh module.
run_curl() has no clear API of whether or not the return values should be decoded. It has parse_json, which should imply decoding, but there's no way to specify that explicitly. Moreover, when it tries to decode, it decodes on the coroutine returned from run_cmd(), not the awaited coroutine return value.
Add a decode parameter, defaulting to False, change the parse_json parameter's default from True to False, and fix the run_cmd() return value evaluation.
Make variable names a little more readable and searchable within the long pkg_relations_list() method by making their names longer and truer to what they actually mean.
CmdPrereq was mostly redundant to PkgRequired all along. CmdPrereq has grown more versatile, and CmdPrereq is not used throughout jw-pkg anylonger by now. Remove it.
print() should be used to output information requested by a certain command, but not for logging the process to achieve it. log() should be used for the latter. The current code has the distinction not down clearly, fix that.
By default, argcomplete uses argcomplete.FilesCompleter as default for every argument. This mixes accessible files into the list of possible completions. For most of jw-pkg's commands, that's unwanted, so turn it off by defining a NoopCompleter class which does nothing, and by set every arguments's default completer to a NoopCompleter instance. If desired, completing files can be restored for an argument by
Every module derived from lib.Cmd implements its own parser.add_argument() logic. As a consequence, all Cmd derived modules need to be loaded to have the full argument tree available. This is avoided during normal program startup because it takes some time. It's not necessary during normal program execution, nor for showing help messages. It is, however, needed for argcomplete to do its thing, so fully parse the command line if the program runs in argcomplete mode, as determined by checking if _ARGCOMPLETE is present in the environment.
lib.ExecContext.log_delim() logs a header not designed for enclosing command output, and, hence, no footer should be output. This commit suppresses it.
print() should be used to output information requested by a certain command, but not for logging the process to achieve it. log() should be used for the latter. The current code has the distinction not down clearly, fix that.
Request a remote PTY from AsyncSSH, and wire the local terminal's stdin up with it if interactive == True. This gives a real interactive session if local stdin belongs to a terminal. Also, thanks to AsyncSSH understanding that, forward terminal size changes to the remote end.
Add a SSHClient implementation using AsyncSSH. This is the first and currently only class derived from SSHClient which implements SSHClient.Cap.LogOutput, designed to consume and log command output as it streams in. It felt like the lower hanging fruit not to do that with Paramiko: Paramiko doesn't provide a native async API, so it would need to spawn additional worker threads. I think.
Add an optional caps ("capabilities") argument to the constructor of SSHClient. It is meant to be used by derived classes in order to declare that they don't want the base class to handle a default behaviour for a certain capability, but that they want to implement it themselves instead.
Also, give the _run_ssh() callbacks the necessary info as parameters, so that the derived classes have the means to do so.
Python's platform.system() outputs 'Linux', and to use it is tempting. Sadly, that's wrong, because it reflects the host's idea of the target system, not the execution context's, so replace it with straight 'linux' if the distro is known, or, failing that, the output of uname -s.
Even with --interactive=[true|auto], there's no point in trying to read /etc/os-release interactively, so don't do that. Most notably, this commit keeps the property method from spilling /etc/os-release's content over the terminal.
