janware/jw-pkg
1
Code Pull requests Activity Actions

jw.pkg: Fix "make check" static code check fallout

Browse source 
The previous commits have put rules for linting and formatting via ruff, yapf, mypy and pyright into place. They are checked with the make check target, and this commit adds the fixes for the target to succeed.
It does some refactoring where type checking dug up dirty bits, and also adds lots of churn in the Python code. To a good deal, that's owed to mere formatting changes. It would have been better to seperate those from syntax and refactoring fixes into multiple commits, so that the interesting changes don't drown in the formatting nose. However, that would have been a lot of additional work only to be thrown away by later commits, hence this commit has a big diff in one piece. The size of the diff is regrettable but hopefully a one-off: What it buys is automatic format checking for CI and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
      - ruff check .
      - mypy .
      - pyright
- Format check:
     - yapf --diff --recursive .
The refactoring includes:
  - Turn the Result class into a more elaborate object, capable of
    doing more heavy lifting around stderr and stdout decoding,
    summarizing outcome, and matching error strings.
    Aside from fixing broken type checks, this also removes lots of
    boilerplate calling code which is currently used for handling
    possible call outcome scenarios. Trying to access an inexistent,
    decoded string should raise a meaningful exception by itself now,
    which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
    - Add the AbstractCmd class above Cmd. This is necessary because
      the checker rightfully complains it can't instantiate a Cmd
      instance where constructor arguments were needed. They never
      were, but the type used at the instantiating code's location in
      jw.pkg.App so claims.
    - Lots of sub- and sub-subcommands are derived from the base
      class of the invoking command. That provides some properties
      shared across the ancestor hierarchy of a command, but is
      semantically unsound. Fix that by introducing jw.pkg.BaseCmd
      class as a place to provide basic helpers shared across all
      commands used in a jw.pkg.App's context, and derive all command
      classes from that afresh. The parent command is still reachable
      via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
    The line was needed by Python 2 which is not supported anylonger.
    For Python 3, the default encoding is UTF-8, anyway.
  - Allow to run "make py-format" without having it produce any
    changes. It's basically "yapf --in-place --recursive ." with some
    code style settings, see conf/topdir/pyproject.toml. The settings
    may be debatable. I've had custom tweaks in place on that target,
    too, but then again, IDEs would have more hassle to integrate
    that.
- Introduce a 88 character line length limit
  - One import per line, reshuffle them semantically, see
    [tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
      if TYPE_CHECKING
  - Spaces around assignments accounts for much churn. Having having
    no spaces in inline parameter list assignments and default
    parameter values would arguably be more compact where it's
    useful. On the other hand, I have not found a code formatter
    which allows spaces around assignments in parameter lists broken
    into one per line and that's often better than a wall of text.
  - Add two spaces before # export, as this seems to be mandated by
    PEP-8
- Use single quotes by default
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit is contained in:
Jan Lindemann 2026-05-27 07:16:05 +02:00
commit 6db73873e7
Signed by: Jan Lindemann
GPG key ID: 3750640C9E25DD61
97 changed files with 3229 additions and 1893 deletions
Download patch file Download diff file Expand all files Collapse all files

187
src/python/jw/pkg/lib/App.py
View file

@ -1,76 +1,119 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import Any, TYPE_CHECKING
import asyncio
import cProfile
import os
import sys
import os, sys, argparse, re, asyncio, cProfile
from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace
from typing import TYPE_CHECKING, Any
from .AsyncRunner import AsyncRunner
from .log import *
from .log import DEBUG, ERR, NOTICE, log, set_log_flags, set_log_level
from .Types import LoadTypes
if TYPE_CHECKING:
from typing import TypeVar
from collections.abc import Awaitable
T = TypeVar("T")
from typing import TypeVar
class App: # export
T = TypeVar('T')
class App: # export
def _add_arguments(self, parser):
self.__parser.add_argument('--log-flags', help='Log flags', default=self.__default_log_flags)
self.__parser.add_argument('--log-level', help='Log level', default=self.__default_log_level)
self.__parser.add_argument('--log-file', help='Log file', default=self.__default_log_file)
self.__parser.add_argument('--backtrace', help='Show exception backtraces', action='store_true', default=self.__back_trace)
self.__parser.add_argument('--write-profile', help='Profile code and store output to file', default=None)
self.__parser.add_argument(
'--log-flags', help = 'Log flags', default = self.__default_log_flags
)
self.__parser.add_argument(
'--log-level', help = 'Log level', default = self.__default_log_level
)
self.__parser.add_argument(
'--log-file', help = 'Log file', default = self.__default_log_file
)
self.__parser.add_argument(
'--backtrace',
help = 'Show exception backtraces',
action = 'store_true',
default = self.__back_trace,
)
self.__parser.add_argument(
'--write-profile',
help = 'Profile code and store output to file',
default = None,
)
def __init__(self, description: str = '', name_filter: str = '^Cmd.*', modules: None=None, eloop: None=None) -> None:
def __init__(
self,
description: str = '',
name_filter: str = '^Cmd.*',
modules: list[str] | None = None,
eloop: None = None,
) -> None:
def add_cmd_to_parser(cmd, parsers):
parser = parsers.add_parser(
cmd.name,
help = cmd.help,
description = cmd.description,
formatter_class = argparse.ArgumentDefaultsHelpFormatter,
formatter_class = ArgumentDefaultsHelpFormatter,
)
parser.set_defaults(func=cmd.run)
parser.set_defaults(func = cmd.run)
cmd.add_arguments(parser)
cmd.set_parser(parser)
return parser
def add_cmds_to_parser(parent, parser, cmds, all=False):
def add_cmds_to_parser(
parent: AbstractCmd | App,
parser: ArgumentParser,
cmds,
all = False
) -> None:
if not cmds:
return
class SubCommand:
def __init__(self, cmd: Cmd, parser: Any):
def __init__(self, cmd: AbstractCmd, parser: Any):
self.cmd = cmd
self.parser = parser
title = 'Available subcommands'
if hasattr(parent, 'name'):
title += ' of ' + getattr(parent, 'name')
subparsers = parser.add_subparsers(title=title, metavar='', dest='command')
subparsers = parser.add_subparsers(
title = title, metavar = '', dest = 'command'
)
scs: dict[str, SubCommand] = {}
for cmd in cmds:
cmd.set_parent(parent)
scs[cmd.name] = SubCommand(cmd, add_cmd_to_parser(cmd, subparsers))
if all:
for sc in scs.values():
add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all=all)
add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all = all)
return
args, unknown = self.__parser.parse_known_args()
if args.command in scs:
sc = scs[args.command]
add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all=all)
add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all = all)
from .Cmd import Cmd
from .Cmd import AbstractCmd
self.__args: Namespace|None = None
self.__cmdline: str|None = None
self.__default_log_flags: str = os.getenv('JW_DEFAULT_LOG_FLAGS', default='stderr,position,prio,color')
self.__default_log_level: str|int|None = os.getenv('JW_DEFAULT_LOG_LEVEL', default=NOTICE)
self.__default_log_file: str|None = os.getenv('JW_DEFAULT_LOG_FILE', default=None)
backtrace: str|bool = os.getenv('JW_DEFAULT_SHOW_BACKTRACE', False)
self.__back_trace = True if isinstance(backtrace, str) and backtrace.lower() in ['1', 'true'] else False
self.__args: Namespace | None = None
self.__cmdline: str | None = None
self.__default_log_flags: str = os.getenv(
'JW_DEFAULT_LOG_FLAGS', default = 'stderr,position,prio,color'
)
self.__default_log_level: str | int | None = os.getenv(
'JW_DEFAULT_LOG_LEVEL', default = NOTICE
)
self.__default_log_file: str | None = os.getenv(
'JW_DEFAULT_LOG_FILE', default = None
)
backtrace: str | bool = os.getenv('JW_DEFAULT_SHOW_BACKTRACE', False)
self.__back_trace = isinstance(backtrace, str) and backtrace.lower() in {
'1',
'true',
}
set_log_flags(self.__default_log_flags)
set_log_level(self.__default_log_level)
@ -79,10 +122,13 @@ class App: # export
if eloop is None:
self.__eloop = asyncio.get_event_loop()
self.__own_eloop = True
self.__async_runner: AsyncRunner|None = None
self.__async_runner: AsyncRunner | None = None
self.__parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter,
description=description, add_help=False)
self.__parser = ArgumentParser(
formatter_class = ArgumentDefaultsHelpFormatter,
description = description,
add_help = False,
)
self._add_arguments(self.__parser)
args, unknown = self.__parser.parse_known_args()
@ -91,12 +137,26 @@ class App: # export
log(DEBUG, '-------------- Running: >' + ' '.join(sys.argv) + '<')
cmd_classes = LoadTypes(modules if modules else ['__main__'], type_name_filter=name_filter, type_filter=[Cmd])
add_all_parsers = '-h' in sys.argv or '--help' in sys.argv or '_ARGCOMPLETE' in os.environ
add_cmds_to_parser(self, self.__parser, [cmd_class(self) for cmd_class in cmd_classes], all=add_all_parsers)
cmd_classes = LoadTypes(
modules if modules else ['__main__'],
type_name_filter = name_filter,
type_filter = [AbstractCmd], # type: ignore[type-abstract]
)
add_all_parsers = (
'-h' in sys.argv or '--help' in sys.argv or '_ARGCOMPLETE' in os.environ
)
add_cmds_to_parser(
self,
self.__parser,
[cmd_class(self) for cmd_class in cmd_classes],
all = add_all_parsers,
)
# -- Add help only now, wouldn't want to have parse_known_args() exit on --help with subcommands missing
self.__parser.add_argument('-h', '--help', action='help', help='Show this help message and exit')
# -- Add help only now, wouldn't want to have parse_known_args() exit
# on --help with subcommands missing
self.__parser.add_argument(
'-h', '--help', action = 'help', help = 'Show this help message and exit'
)
def __del__(self):
if self.__own_eloop:
@ -105,24 +165,32 @@ class App: # export
self.__eloop = None
self.__own_eloop = False
async def __aenter__(self) ->None:
return self
async def __aenter__(self) -> None:
pass
async def __aexit__(self, exc_type, exc, tb) -> None:
pass
async def __run(self, argv=None) -> None:
async def __run(self, argv = None) -> None:
try:
class NoopCompleter:
# Import argcomplete only here to not require it to be compatible
# with minimal environments
from argcomplete.completers import BaseCompleter
class NoopCompleter(BaseCompleter):
def __call__(self, **kwargs):
return ()
import argcomplete # Don't require it to be compatible with minimal environments
argcomplete.autocomplete(self.__parser, default_completer=NoopCompleter())
except:
import argcomplete
argcomplete.autocomplete(self.__parser, default_completer = NoopCompleter())
except Exception:
pass
self.__args = self.__parser.parse_args(args=argv)
self.__args = self.__parser.parse_args(args = argv)
set_log_flags(self.__args.log_flags)
set_log_level(self.__args.log_level)
@ -152,14 +220,17 @@ class App: # export
finally:
if pr is not None:
pr.disable()
log(NOTICE, f'Writing profile statistics to {self.__args.write_profile}')
log(
NOTICE,
f'Writing profile statistics to {self.__args.write_profile}'
)
pr.dump_stats(self.__args.write_profile)
if exit_status:
sys.exit(exit_status)
# Run sub-command. Overwrite if you want to do anything before or after
async def _run(self, args: argparse.Namespace) -> None:
async def _run(self, args: Namespace) -> None | int:
return await self.args.func(args)
def call_async(self, awaitable: Awaitable[T], timeout: float | None = None) -> T:
@ -167,6 +238,8 @@ class App: # export
@property
def eloop(self) -> asyncio.AbstractEventLoop:
if self.__eloop is None:
raise Exception('Tried to get inexistent event loop from application')
return self.__eloop
@property
@ -178,28 +251,34 @@ class App: # export
@property
def cmdline(self) -> str:
if self.__cmdline is None:
import shlex
with open('/proc/self/cmdline', 'rb') as f:
raw = f.read().split(b'\0')[:-1]
self.__cmdline = ' '.join(shlex.quote(arg.decode()) for arg in raw)
return self.__cmdline
@property
def args(self) -> argparse.Namespace:
def args(self) -> Namespace:
if self.__args is None:
raise Exception('Tried to get inexistent argument list from application')
return self.__args
@property
def parser(self) -> argparse.ArgumentParser:
def parser(self) -> ArgumentParser:
return self.__parser
def run(self, argv=None) -> None:
def run(self, argv = None) -> None:
try:
ret = self.__eloop.run_until_complete(self.__run(argv)) # type: ignore
ret = self.eloop.run_until_complete(self.__run(argv))
finally:
if self.__async_runner:
self.__async_runner.close()
self.__async_runner = None
return ret
def run_sub_commands(description = '', name_filter = '^Cmd.*', modules=None, argv=None): # export
with App(description, name_filter, modules) as app:
return app.run(argv=argv)
def run_sub_commands(
description = '', name_filter = '^Cmd.*', modules = None, argv = None
): # export
app = App(description, name_filter, modules)
return app.run(argv = argv)

16
src/python/jw/pkg/lib/AsyncRunner.py
View file

@ -1,10 +1,13 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
import asyncio
import concurrent.futures
import contextlib
import abc, asyncio, contextlib, concurrent.futures
from collections.abc import Awaitable, Generator
from typing import TypeVar
T = TypeVar("T")
T = TypeVar('T')
@contextlib.contextmanager
def loop_in_thread() -> Generator[asyncio.AbstractEventLoop, None, None]:
@ -19,8 +22,7 @@ def loop_in_thread() -> Generator[asyncio.AbstractEventLoop, None, None]:
loop_fut.set_result(asyncio.get_running_loop())
await stop_event.wait()
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as tpe:
with concurrent.futures.ThreadPoolExecutor(max_workers = 1) as tpe:
complete_fut = tpe.submit(asyncio.run, main())
for fut in concurrent.futures.as_completed((loop_fut, complete_fut)):
@ -40,13 +42,13 @@ class AsyncRunner:
self._loop = self._cm.__enter__()
def call(self, awaitable: Awaitable[T], timeout: float | None = None) -> T:
fut = asyncio.run_coroutine_threadsafe(awaitable, self._loop)
fut = asyncio.run_coroutine_threadsafe(awaitable, self._loop) # type: ignore
return fut.result(timeout)
def close(self) -> None:
self._cm.__exit__(None, None, None)
def __enter__(self) -> "AsyncRunner":
def __enter__(self) -> AsyncRunner:
return self
def __exit__(self, exc_type, exc, tb) -> None:

162
src/python/jw/pkg/lib/Cmd.py
View file

@ -1,38 +1,44 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import Any
import abc
import sys
import inspect, sys, re, abc, argparse
from argparse import ArgumentParser, _SubParsersAction
from argparse import ArgumentParser
from typing import TYPE_CHECKING
from .log import *
from .Types import Types, LoadTypes
from .log import ERR
from .Types import LoadTypes, Types
class Cmd(abc.ABC): # export
if TYPE_CHECKING:
from typing import Any
def __init__(self, parent: App|Cmd, name: str, help: str, description: str|None=None) -> None:
from . import App
self.__parent: App|Cmd|None = parent
self.__app: App|None = None
self.__name = name
self.__help = help
self.__description = description if description else help
from .App import App
class AbstractCmd(abc.ABC):
def __init__(
self,
parent: App | AbstractCmd,
) -> None:
self.__parent: App | AbstractCmd | None = parent
self.__app: App | None = None
self.__children: list[Cmd] = []
self.__child_classes: list[type[Cmd]] = []
self.__parser: ArgumentParser|None = None
self.__parser: ArgumentParser | None = None
@abc.abstractmethod
async def _run(self, args) -> None:
if isinstance(self.__parent, Cmd): # Calling App.run() would loop
return await self.__parent._run(args)
def set_parent(self, parent: Any|Cmd):
def set_parent(self, parent: Any | Cmd):
self.__parent = parent
@property
def parent(self) -> App|Cmd:
def name(self) -> str:
return self._name()
@abc.abstractmethod
def _name(self) -> str:
raise NotImplementedError('Called pure virtual base class method')
@property
def parent(self) -> App | AbstractCmd:
if self.__parent is None:
raise Exception(f'Tried to access inexistent parent of command {self.name}')
return self.__parent
@ -40,55 +46,45 @@ class Cmd(abc.ABC): # export
@property
def app(self) -> App:
from .App import App
if self.__app is None:
parent = self.__parent
while True:
if parent is None:
raise Exception("Can't get application object from command without parent")
raise Exception(
"Can't get application object from command without parent"
)
if isinstance(parent, App):
self.__app = parent
break
assert parent != parent.__parent, f'Assertion failed: Parent mismatch'
assert parent != parent.__parent, 'Assertion failed: Parent mismatch'
parent = parent.__parent
return self.__app
@property
def children(self) -> tuple[Cmd, ...]:
return tuple(self.__children)
@property
def child_classes(self) -> tuple[type[Cmd], ...]:
return tuple(self.__child_classes)
@property
def parser(self) -> ArgumentParser:
if self.__parser is None:
raise Exception(f'Tried to get a non-existing parser from {self}')
return self.__parser
# Don't use a setter decorator to force using a grepable method
def set_parser(self, parser: ArgumentParser):
self.__parser = parser
@property
def parser(self) -> str:
return self.__parser
@property
def name(self) -> str:
return self.__name
@property
def help(self) -> str:
return self.__help
@property
def description(self) -> str:
return self.__description
@property
def children(self) -> list[Cmd]:
return tuple(self.__children)
@property
def child_classes(self) -> list[type[Cmd]]:
return tuple(self.__child_classes)
def print_help(self, exit_status: int|None=None) -> None:
def print_help(self, exit_status: int | None = None) -> None:
self.parser.print_help()
if exit_status is not None:
sys.exit(exit_status)
async def run(self, args):
return await self._run(args)
def add_subcommands(self, cmds: Cmd|list[Cmds]|Types|list[Types]) -> None:
def add_subcommands(self, cmds: Cmd | list[Cmd] | Types | list[Types]) -> None:
if isinstance(cmds, Cmd):
assert False
return
@ -106,21 +102,73 @@ class Cmd(abc.ABC): # export
self.__children.append(cmd)
assert len(self.__children) == len(self.__child_classes)
except Exception as e:
cmds.dump(ERR, f"Failed to add subcommands ({str(e)})")
cmds.dump(ERR, f'Failed to add subcommands ({str(e)})')
raise
return
raise Exception(f'Tried to add sub-commands of unknown type {type(cmds)}')
def load_subcommands(self, modules: str|list[str]|None=None, name_filter: str=r'Cmd[^.]') -> None:
def load_subcommands(
self,
modules: str | list[str] | None = None,
name_filter: str = r'Cmd[^.]'
) -> None:
if modules is None:
# Derive module search path for the calling module's subcommands
# from the module path of the calling module itself
modules = [type(self).__module__.replace('Cmd', '').lower()]
elif isinstance(modules, str):
modules = [modules]
self.add_subcommands(LoadTypes(modules, type_name_filter=name_filter))
self.add_subcommands(LoadTypes(modules, type_name_filter = name_filter))
# -- Interface to derived classes
# To be overridden by derived class in case the command does take arguments.
# Will be called from App base class constructor and set up the parser hierarchy
def add_arguments(self, parser: ArgumentParser) -> None:
pass
@abc.abstractmethod
async def _run(self, args) -> None:
if isinstance(self.__parent, Cmd): # Calling App.run() would loop
return await self.__parent._run(args)
async def run(self, args):
return await self._run(args)
@abc.abstractmethod
def _help(self) -> str:
raise NotImplementedError('Called pure virtual base class method')
@property
def help(self) -> str:
return self._help()
def _description(self) -> str:
raise NotImplementedError('Called pure virtual base class method')
@property
def description(self) -> str:
return self._description()
class Cmd(AbstractCmd): # export
def __init__(
self,
parent: App | Cmd,
name: str,
help: str,
description: str | None = None
) -> None:
super().__init__(parent)
self.__name = name
self.__help = help
self.__description = description if description else help
def _name(self) -> str:
return self.__name
def _help(self) -> str:
return self.__help
def _description(self) -> str:
return self.__description

53
src/python/jw/pkg/lib/CopyContext.py
View file

@ -1,32 +1,47 @@
# -*- coding: utf-8 -*-
from typing import Self
from .FileContext import FileContext
from .Uri import Uri
class CopyContext:
def __init__(self, src: str|FileContext, dst: str|FileContext, chroot=False) -> None:
if isinstance(src, FileContext):
self.__src = src
self.__src_uri = src.uri
else:
self.__src: FileContext|None = None
self.__src_uri = src
if isinstance(dst, FileContext):
self.__dst = dst
self.__dst_uri = dst.uri
else:
self.__dst: FileContext|None = None
self.__dst_uri = dst
def __init__(
self,
src: Uri | str | FileContext,
dst: Uri | str | FileContext,
chroot = False
) -> None:
def __uri(ctx: FileContext | Uri | str) -> Uri | str | None:
if ctx is None:
return None
if isinstance(ctx, Uri):
return ctx
if isinstance(ctx, str):
return ctx
assert isinstance(ctx, FileContext)
return ctx.uri
def __info(
ctx: FileContext | Uri | str,
) -> tuple[FileContext | None, str | Uri | None]:
fc: FileContext | None = ctx if isinstance(ctx, FileContext) else None
return fc, __uri(ctx)
self.__src, self.__src_uri = __info(src)
self.__dst, self.__dst_uri = __info(dst)
self.__chroot = chroot
async def __aenter__(self) -> Self:
if self.__src is None:
self.__src = FileContext.create(self.__src_uri, chroot=self.__chroot)
if self.__src_uri is None:
raise Exception('Tried to create source context without URI')
self.__src = FileContext.create(self.__src_uri, chroot = self.__chroot)
await self.__src.open()
if self.__dst is None:
self.__dst = FileContext.create(self.__dst_uri, chroot=self.__chroot)
if self.__dst_uri is None:
raise Exception('Tried to create destination context without URI')
self.__dst = FileContext.create(self.__dst_uri, chroot = self.__chroot)
await self.__dst.open()
return self
@ -40,10 +55,14 @@ class CopyContext:
@property
def src(self) -> FileContext:
if self.__src is None:
raise Exception('Tried to access inexistent source context')
return self.__src
@property
def dst(self) -> FileContext:
if self.__dst is None:
raise Exception('Tried to access inexistent destination context')
return self.__dst
async def _run(self) -> None:

222
src/python/jw/pkg/lib/Distro.py
View file

@ -1,38 +1,41 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
import abc
import importlib
import re
import sys
from functools import cached_property
from typing import TYPE_CHECKING
from .log import ERR, INFO, WARNING, log
if TYPE_CHECKING:
import Iterable
from typing import Iterable
from .base import InputMode, Result
from .ExecContext import ExecContext
from .Package import Package
from .PackageFilter import PackageFilter
import abc, importlib, re
from .PackageFilter import PackageFilter
from .ExecContext import ExecContext
from .base import Result, InputMode
from .Package import Package
from .log import *
class Distro(abc.ABC):
def __init__(
self,
ec: ExecContext,
id: str|None=None,
os_release_str: str|None=None,
default_pkg_filter: PackageFilter|None=None,
) -> None:
self,
ec: ExecContext,
id: str | None = None,
os_release_str: str | None = None,
default_pkg_filter: PackageFilter | None = None,
) -> None:
if id is None:
raise ValueError(f'Tried to instaniate Distro without id')
raise ValueError('Tried to instaniate Distro without id')
if ec is None:
raise ValueError(f'Tried to instaniate Distro "{id}" without execution context')
raise ValueError(
f'Tried to instaniate Distro "{id}" without execution context'
)
self.__exec_context = ec
self.__id: str|None = None
self.__os_release_str: str|None = os_release_str
self.__id: str | None = None
self.__os_release_str: str | None = os_release_str
self.__default_pkg_filter = default_pkg_filter
# Names that can be used by code outside this class to retrieve
@ -52,47 +55,60 @@ class Distro(abc.ABC):
# == Load
@classmethod
async def read_os_release_str(cls, ec: ExecContext) -> None:
async def read_os_release_str(cls, ec: ExecContext) -> str:
release_file = '/etc/os-release'
try:
result = await ec.get(release_file, throw=True)
ret = result.stdout.decode().strip()
result = await ec.get(release_file, throw = True)
return result.stdout_str
except Exception as e:
log(INFO, f'Failed to read {release_file} ({str(e)}), falling back to uname')
log(
INFO,
f'Failed to read {release_file} ({str(e)}), falling back to uname'
)
result = await ec.run(
['uname', '-s'],
throw=False,
cmd_input=InputMode.NonInteractive
)
['uname', '-s'], throw = False, cmd_input = InputMode.NonInteractive
)
if result.status != 0:
log(ERR, f'/etc/os-release and uname both failed, the latter with exit status {result.status}')
log(
ERR,
(
'/etc/os-release and uname both failed, '
f'the latter with {result.summary}'
),
)
raise
uname = result.decode().stdout.strip().lower()
uname = result.stdout_str.lower()
ret = f'ID={uname}\nVERSION_CODENAME=unknown'
return ret
@classmethod
def parse_os_release_field(self, key: str, os_release_str: str, throw: bool=False) -> str:
m = re.search(r'^\s*' + key + r'\s*=\s*("?)([^"\n]+)\1\s*$', os_release_str, re.MULTILINE)
def parse_os_release_field(cls, key: str, os_release_str: str) -> str:
m = re.search(
r'^\s*' + key + r'\s*=\s*("?)([^"\n]+)\1\s*$', os_release_str, re.MULTILINE
)
if m is None:
if throw:
raise Exception(f'Could not read "{key}=" from /etc/os-release')
return None
raise Exception(f'Could not read "{key}=" from /etc/os-release')
return m.group(2)
@classmethod
def parse_os_release_field_id(cls, os_release_str: str, throw: bool=False) -> str:
ret = cls.parse_os_release_field('ID', os_release_str, throw=throw)
def parse_os_release_field_id(cls, os_release_str: str) -> str:
ret = cls.parse_os_release_field('ID', os_release_str)
match ret:
case 'opensuse-tumbleweed':
return 'suse'
return ret
@classmethod
async def instantiate(cls, ec: ExecContext, *args, id: str|None=None, os_release_str: str|None=None, **kwargs):
async def instantiate(
cls,
ec: ExecContext,
id: str | None = None,
os_release_str: str | None = None,
**kwargs,
):
if id is None:
os_release_str = await cls.read_os_release_str(ec)
id = cls.parse_os_release_field_id(os_release_str, throw=True)
id = cls.parse_os_release_field_id(os_release_str)
backend_id = id.lower().replace('-', '_')
match backend_id:
case 'ubuntu' | 'raspbian' | 'kali':
@ -108,11 +124,11 @@ class Distro(abc.ABC):
log(ERR, f'Failed to import Distro module {module_path} ({str(e)})')
raise
cls = getattr(module, 'Distro')
ret = cls(ec, *args, id=id, os_release_str=os_release_str, **kwargs)
ret = cls(ec, id = id, os_release_str = os_release_str, **kwargs)
return ret
def os_release_field(self, key: str, throw: bool=False) -> str:
return self.parse_os_release_field(key, self.os_release_str, throw)
def os_release_field(self, key: str) -> str:
return self.parse_os_release_field(key, self.os_release_str)
async def cache(self) -> None:
if self.__os_release_str is None:
@ -120,10 +136,12 @@ class Distro(abc.ABC):
@cached_property
def os_cascade(self) -> list[str]:
def __append(entry: str):
if not entry in ret:
if entry not in ret:
ret.append(entry)
ret = [ 'os' ]
ret = ['os']
match self.id:
case 'centos':
__append('linux')
@ -177,27 +195,32 @@ class Distro(abc.ABC):
@property
def os_release_str(self) -> str:
if self.__os_release_str is None:
raise Exception(f'Tried to access OS release from an incompletely loaded Distro instance. Call reacache() before')
raise Exception(
'Tried to access OS release from an incompletely loaded Distro '
'instance. Call cache() before'
)
return self.__os_release_str
@cached_property
def name(self) -> str:
return self.os_release_field('NAME', throw=True)
return self.os_release_field('NAME')
@cached_property
def id(self) -> str:
return self.parse_os_release_field_id(self.__os_release_str, throw=True)
return self.parse_os_release_field_id(self.os_release_str)
@cached_property
def codename(self) -> str:
match self.id:
case 'suse':
return self.os_release_field('ID', throw=True).split('-')[1]
return self.os_release_field('ID').split('-')[1]
case 'kali':
return self.os_release_field('VERSION_CODENAME', throw=True).split('-')[1]
return self.os_release_field('VERSION_CODENAME').split('-')[1]
case _:
return self.os_release_field('VERSION_CODENAME', throw=True)
raise NotImplementedError(f'Can\'t determine code name from distribution ID {self.id}')
return self.os_release_field('VERSION_CODENAME')
raise NotImplementedError(
f"Can't determine code name from distribution ID {self.id}"
)
@cached_property
def os(self) -> str:
@ -214,33 +237,35 @@ class Distro(abc.ABC):
@cached_property
def gnu_triplet(self) -> str:
import sysconfig
import shutil
import subprocess
import sysconfig
# Best: GNU host triplet Python was built for
for key in ("HOST_GNU_TYPE", "BUILD_GNU_TYPE"): # BUILD_GNU_TYPE can exist too
for key in ('HOST_GNU_TYPE', 'BUILD_GNU_TYPE'): # BUILD_GNU_TYPE can exist too
ret = sysconfig.get_config_var(key)
if isinstance(ret, str) and ret:
return ret
# Common on Debian/Ubuntu: multiarch component (often looks like a triplet)
ret = sysconfig.get_config_var("MULTIARCH")
ret = sysconfig.get_config_var('MULTIARCH')
if isinstance(ret, str) and ret:
return ret
# Sometimes exposed (privately) by CPython
ret = getattr(sys.implementation, "_multiarch", None)
ret = getattr(sys.implementation, '_multiarch', None)
if isinstance(ret, str) and ret:
return ret
# Last resort: ask the system compiler
for cc in ("gcc", "cc", "clang"):
for cc in ('gcc', 'cc', 'clang'):
path = shutil.which(cc)
if not path:
continue
try:
ret = subprocess.check_output([path, "-dumpmachine"], text=True, stderr=subprocess.DEVNULL).strip()
ret = subprocess.check_output(
[path, '-dumpmachine'], text = True, stderr = subprocess.DEVNULL
).strip()
if ret:
return ret
except Exception:
@ -252,14 +277,21 @@ class Distro(abc.ABC):
def macros(cls) -> list[str]:
return ['%%{' + name + '}' for name in cls.macro_names]
def expand_macros(self, fmt: str|Iterable) -> str|Iterable:
def expand_macros(self, fmt: str | Iterable) -> str | list[str]:
ret: str | list[str]
if not isinstance(fmt, str):
ret: list[str] = []
ret = []
for entry in fmt:
ret.append(self.expand_macros(entry))
rv = self.expand_macros(entry)
if isinstance(rv, str):
ret.append(rv)
continue
raise NotImplementedError(
f'Expanding macros in nested lists is not supported: {rv}'
)
return ret
ret = fmt
for macro in re.findall("%{([A-Za-z_-]+)}", fmt):
for macro in re.findall('%{([A-Za-z_-]+)}', fmt):
try:
name = macro.replace('-', '_')
val = getattr(self, name)
@ -279,7 +311,7 @@ class Distro(abc.ABC):
return self.__exec_context
@property
def default_pkg_filter(self) -> str:
def default_pkg_filter(self) -> PackageFilter | None:
return self.__default_pkg_filter
async def run(self, *args, **kwargs) -> Result:
@ -289,7 +321,7 @@ class Distro(abc.ABC):
return await self.__exec_context.sudo(*args, **kwargs)
@property
def interactive(self) -> bool:
def interactive(self) -> bool | None:
return self.__exec_context.interactive
# == Distribution abstraction methods
@ -309,8 +341,8 @@ class Distro(abc.ABC):
async def _dup(self, download_only: bool) -> None:
pass
async def dup(self, download_only: bool=False) -> None:
return await self._dup(download_only=download_only)
async def dup(self, download_only: bool = False) -> None:
return await self._dup(download_only = download_only)
# -- reboot_required
@ -318,10 +350,10 @@ class Distro(abc.ABC):
async def _reboot_required(self, verbose: bool) -> bool:
pass
async def reboot_required(self, verbose: bool|None=None) -> bool:
async def reboot_required(self, verbose: bool | None = None) -> bool:
if verbose is None:
verbose = self.ctx.verbose_default
return await self._reboot_required(verbose=verbose)
return await self._reboot_required(verbose = verbose)
# -- select
@ -329,11 +361,16 @@ class Distro(abc.ABC):
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
pass
async def _select(self, names: Iterable[str], filter: PackageFilter) -> Iterable[Package]:
assert filter, "No filter in _select()"
async def _select(self, names: Iterable[str],
filter: PackageFilter) -> Iterable[Package]:
assert filter, 'No filter in _select()'
return [p for p in await self._select_by_name(names) if filter.match(p)]
async def select(self, names: Iterable[str] = [], filter: PackageFilter|None=None) -> Iterable[Package]:
async def select(
self,
names: Iterable[str] = [],
filter: PackageFilter | None = None
) -> Iterable[Package]:
if not filter:
filter = self.__default_pkg_filter
if not filter:
@ -349,17 +386,28 @@ class Distro(abc.ABC):
# Default implementation assumes package manager can handle local files.
# Not true for all distros. Override if Distro knows better.
async def _install_local_files(self, paths: Iterable[str], only_update: bool) -> None:
await self._install(paths, only_update=only_update)
async def _install_local_files(
self, paths: Iterable[str], only_update: bool
) -> None:
await self._install(paths, only_update = only_update)
# Download first and then install. Override if Distro knows better.
async def _install_urls(self, urls: Iterable[str], only_update: bool) -> None:
from .util import copy
tmp: str|None = None
tmp: str | None = None
try:
tmp = await self.__exec_context.mktemp('/tmp/jw-pkg-XXXXXX', directory=True)
paths = await copy(urls, self.__exec_context.uri.scheme_plus_authority + tmp)
await self._install_local_files(paths, only_update=only_update)
tmp = await self.__exec_context.mktemp(
'/tmp/jw-pkg-XXXXXX', directory = True
)
paths = await copy(
urls, self.__exec_context.uri.scheme_plus_authority + tmp
)
if isinstance(paths, Exception):
raise paths
if isinstance(paths, str):
paths = [paths]
await self._install_local_files(paths, only_update = only_update)
finally:
if tmp is not None:
await self.__exec_context.erase(tmp)
@ -368,7 +416,9 @@ class Distro(abc.ABC):
# - Download URLs into local directories and install
# - Pass names to package manager
# Override if Distro knows better.
async def _install_urls_and_names(self, packages: Iterable[str], only_update: bool) -> None:
async def _install_urls_and_names(
self, packages: Iterable[str], only_update: bool
) -> None:
urls: list[str] = []
names: list[str] = []
for package in packages:
@ -380,15 +430,15 @@ class Distro(abc.ABC):
continue
names.append(package)
if urls:
await self._install_urls(urls, only_update=only_update)
await self._install_urls(urls, only_update = only_update)
if names:
await self._install(names, only_update=only_update)
await self._install(names, only_update = only_update)
async def install(self, names: Iterable[str], only_update: bool=False) -> None:
async def install(self, names: Iterable[str], only_update: bool = False) -> None:
if not names:
log(WARNING, f'No packages specified for installation')
log(WARNING, 'No packages specified for installation')
return
await self._install_urls_and_names(names, only_update=only_update)
await self._install_urls_and_names(names, only_update = only_update)
# -- delete
@ -398,7 +448,7 @@ class Distro(abc.ABC):
async def delete(self, names: Iterable[str]) -> None:
if not names:
log(WARNING, f'No packages specified for deletion')
log(WARNING, 'No packages specified for deletion')
return
return await self._delete(names)
@ -410,6 +460,6 @@ class Distro(abc.ABC):
async def pkg_files(self, name: str) -> Iterable[str]:
if not name:
log(WARNING, f'No package specified for inspection')
log(WARNING, 'No package specified for inspection')
return []
return await self._pkg_files(name)

463
src/python/jw/pkg/lib/ExecContext.py
View file

@ -1,42 +1,43 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
import abc, re, sys, errno
from enum import Enum, auto
from typing import NamedTuple, TYPE_CHECKING
from decimal import Decimal, ROUND_FLOOR
import abc
import errno
import sys
from decimal import ROUND_FLOOR, Decimal
from typing import TYPE_CHECKING, NamedTuple
if TYPE_CHECKING:
from typing import Self, Type
from typing import Type
from types import TracebackType
from .log import *
from .base import Input, InputMode, Result, StatResult
from .FileContext import FileContext as Base
from .log import DEBUG, ERR, NOTICE, log
_US = "\x1f" # unlikely to appear in numeric output
_US = '\x1f' # unlikely to appear in numeric output
_BILLION = Decimal(1_000_000_000)
def _looks_like_option_error(stderr: str) -> bool:
def _looks_like_option_error(stderr: str | None) -> bool:
if stderr is None:
return False
s = stderr.lower()
return any(
needle in s
for needle in (
"unrecognized option",
"illegal option",
"unknown option",
"invalid option",
"option requires an argument",
)
needle in s for needle in (
'unrecognized option',
'illegal option',
'unknown option',
'invalid option',
'option requires an argument', )
)
def _raise_stat_error(path: str, stderr: str, returncode: int) -> None:
msg = (stderr or "").strip() or f"stat exited with status {returncode}"
def _raise_stat_error(path: str, result: Result) -> None:
stderr = result.stderr_str_or_none or f'stat exited with status {result.status}'
msg = stderr.strip()
lower = msg.lower()
if "no such file" in lower:
if 'no such file' in lower:
raise FileNotFoundError(errno.ENOENT, msg, path)
if "permission denied" in lower or "operation not permitted" in lower:
if 'permission denied' in lower or 'operation not permitted' in lower:
raise PermissionError(errno.EACCES, msg, path)
raise OSError(errno.EIO, msg, path)
@ -46,14 +47,14 @@ def _parse_epoch(value: str) -> tuple[int, float, int]:
(integer seconds for tuple slot, float seconds for attribute, ns for *_ns)
"""
dec = Decimal(value.strip())
sec = int(dec.to_integral_value(rounding=ROUND_FLOOR))
ns = int((dec * _BILLION).to_integral_value(rounding=ROUND_FLOOR))
sec = int(dec.to_integral_value(rounding = ROUND_FLOOR))
ns = int((dec * _BILLION).to_integral_value(rounding = ROUND_FLOOR))
return sec, float(dec), ns
def _build_stat_result(fields: list[str], mode_base: int) -> StatResult:
if len(fields) != 13:
raise ValueError(
f"unexpected stat output: expected 13 fields, got {len(fields)}: {fields!r}"
f'unexpected stat output: expected 13 fields, got {len(fields)}: {fields!r}'
)
(
@ -73,9 +74,9 @@ def _build_stat_result(fields: list[str], mode_base: int) -> StatResult:
) = fields
st_mode = int(mode_s, mode_base)
st_ino = int(ino_s)
st_dev = int(dev_s)
st_nlink = int(nlink_s)
# st_ino = int(ino_s)
# st_dev = int(dev_s)
# st_nlink = int(nlink_s)
st_uid = uid_s
st_gid = gid_s
st_size = int(size_s)
@ -84,9 +85,9 @@ def _build_stat_result(fields: list[str], mode_base: int) -> StatResult:
st_mtime_i, st_mtime_f, st_mtime_ns = _parse_epoch(mtime_s)
st_ctime_i, st_ctime_f, st_ctime_ns = _parse_epoch(ctime_s)
st_blksize = int(blksize_s)
st_blocks = int(blocks_s)
st_rdev = int(rdev_s)
# st_blksize = int(blksize_s)
# st_blocks = int(blocks_s)
# st_rdev = int(rdev_s)
return StatResult(
mode = st_mode,
@ -103,38 +104,38 @@ class ExecContext(Base):
class CallContext:
def __init__(
self,
parent: ExecContext,
title: str|None,
cmd: list[str],
cmd_input: Input,
mod_env: dict[str, str]|None,
wd: str|None,
log_prefix: str,
throw: bool,
verbose: bool,
) -> None:
self,
parent: ExecContext,
title: str | None,
cmd: list[str],
cmd_input: Input,
mod_env: dict[str, str] | None,
wd: str | None,
log_prefix: str,
throw: bool,
verbose: bool | None,
) -> None:
self.__cmd = cmd
self.__wd = wd
self.__log_prefix = log_prefix
self.__parent = parent
self.__title = title
self.__pretty_cmd: str|None = None
self.__delim = title if title is not None else f'---- {parent.uri}: Running {self.pretty_cmd} -'
self.__pretty_cmd: str | None = None
self.__delim = (
title if title is not None else
f'---- {parent.uri}: Running {self.pretty_cmd} -'
)
delim_len = 120
self.__delim += '-' * max(0, delim_len - len(self.__delim))
self.__mod_env = {'LC_ALL': 'C'} if mod_env is None else mod_env
self.__cmd_input: bytes | None = None
# -- At the end of this dance, interactive needs to be either True
# or False
interactive: bool|None = None
if not isinstance(cmd_input, InputMode):
interactive = False
self.__cmd_input = (
cmd_input if isinstance(cmd_input, bytes) else
cmd_input.encode(sys.stdout.encoding or "utf-8")
)
else:
interactive: bool | None = None
cmd_input_bytes: None | bytes
if isinstance(cmd_input, InputMode):
cmd_input_bytes = None
match cmd_input:
case InputMode.Interactive:
interactive = True
@ -148,25 +149,34 @@ class ExecContext(Base):
interactive = parent.interactive
if interactive is None:
interactive = sys.stdin.isatty()
self.__cmd_input = None
assert interactive in [ True, False ], f'Invalid: interactive = {invalid}'
else:
interactive = False
if cmd_input is None:
cmd_input_bytes = None
elif isinstance(cmd_input, str):
cmd_input_bytes = cmd_input.encode(sys.stdout.encoding or 'utf-8')
else:
cmd_input_bytes = cmd_input
self.__cmd_input = cmd_input_bytes
assert interactive in [True, False], f'Invalid: interactive = {interactive}'
self.__interactive = interactive
self.__cmd_input = cmd_input if not isinstance(cmd_input, InputMode) else None
self.__throw = throw
self.__verbose = verbose if verbose is not None else parent.verbose_default
def __enter__(self) -> CallContext:
self.log_delim(start=True)
def __enter__(self) -> ExecContext.CallContext:
self.log_delim(start = True)
return self
def __exit__(
self,
exc_type: Type[BaseException]|None,
exc_value: BaseException|None,
traceback: TracebackType|None
exc_type: Type[BaseException] | None,
exc_value: BaseException | None,
traceback: TracebackType | None,
) -> bool:
self.log_delim(start=False)
self.log_delim(start = False)
return True
@property
def log_prefix(self) -> str:
@ -181,7 +191,7 @@ class ExecContext(Base):
return self.__verbose
@property
def cmd_input(self) -> bytes|None:
def cmd_input(self) -> bytes | None:
return self.__cmd_input
@property
@ -193,7 +203,7 @@ class ExecContext(Base):
return self.__throw
@property
def wd(self) -> str|None:
def wd(self) -> str | None:
return self.__wd
@property
@ -204,16 +214,17 @@ class ExecContext(Base):
def pretty_cmd(self) -> str:
if self.__pretty_cmd is None:
from .util import pretty_cmd
self.__pretty_cmd = pretty_cmd(self.__cmd, self.__wd)
return self.__pretty_cmd
def log(prio: int, *args, **kwargs) -> None:
def log(self, prio: int, *args, **kwargs) -> None:
log(prio, self.__log_prefix, *args, **kwargs)
def log_delim(self, start: bool) -> None:
if not self.__verbose:
return None
if self.__interactive: # Don't log footer in interative mode
if self.__interactive: # Don't log footer in interative mode
if start:
log(NOTICE, self.__delim)
return
@ -223,12 +234,9 @@ class ExecContext(Base):
def check_exit_code(self, result: Result) -> None:
if result.status == 0:
return
if (self.__throw or self.__verbose):
msg = f'Command exited with status {result.status}: {self.pretty_cmd}'
if result.stderr:
msg += ': ' + result.decode().stderr.strip()
if self.__throw or self.__verbose:
if self.__throw:
raise RuntimeError(msg)
raise RuntimeError(result.summary)
def exception(self, result: Result, e: Exception) -> Result:
log(ERR, self.__log_prefix, f'Failed to run {self.pretty_cmd}')
@ -243,15 +251,35 @@ class ExecContext(Base):
def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
@classmethod
def create(cls, *args, **kwargs) -> ExecContext:
ret = super().create(*args, **kwargs)
if not isinstance(ret, cls):
raise TypeError(f'Expected {cls.__name__}, got {type(ret).__name__}')
return ret
@abc.abstractmethod
async def _run(
self,
cmd: list[str],
wd: str | None,
verbose: bool,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str,
) -> Result:
raise NotImplementedError('Called pure virtual method _run()')
async def run(
self,
cmd: list[str],
wd: str|None = None,
wd: str | None = None,
throw: bool = True,
verbose: bool|None = None,
verbose: bool | None = None,
cmd_input: Input = InputMode.OptInteractive,
mod_env: dict[str, str]|None = None,
title: str = None
mod_env: dict[str, str] | None = None,
title: str | None = None,
) -> Result:
"""
Run a command asynchronously and return its output
@ -262,13 +290,16 @@ class ExecContext(Base):
throw: Raise an exception on non-zero exit status if True
verbose: Emit log output while the command runs
cmd_input:
- "InputMode.OptInteractive" -> Let --interactive govern how to handle interactivity (default)
- "InputMode.OptInteractive" -> Let --interactive govern how to handle
interactivity (default)
- "InputMode.Interactive" -> Inherit terminal stdin
- "InputMode.Auto" -> Inherit terminal stdin if it is a TTY
- "InputMode.NonInteractive" -> stdin from /dev/null
- None -> Alias for InputMode.NonInteractive
- otherwise -> Feed cmd_input to stdin
mod_env: Change set to command's environment. key: val adds a variable, key: None removes it
mod_env: Change set to command's environment:
- key: val adds a variable,
- key: None removes it
Returns:
A Result instance
@ -285,8 +316,17 @@ class ExecContext(Base):
try:
ret = Result(None, None, 1)
with self.CallContext(self, title=title, cmd=cmd, cmd_input=cmd_input, mod_env=mod_env, wd=wd,
log_prefix='|', throw=throw, verbose=verbose) as cc:
with self.CallContext(
self,
title = title,
cmd = cmd,
cmd_input = cmd_input,
mod_env = mod_env,
wd = wd,
log_prefix = '|',
throw = throw,
verbose = verbose,
) as cc:
try:
ret = await self._run(
cmd = cc.cmd,
@ -295,7 +335,7 @@ class ExecContext(Base):
cmd_input = cc.cmd_input,
mod_env = cc.mod_env,
interactive = cc.interactive,
log_prefix = cc.log_prefix
log_prefix = cc.log_prefix,
)
except Exception as e:
return cc.exception(ret, e)
@ -308,11 +348,11 @@ class ExecContext(Base):
async def _sudo(
self,
cmd: list[str],
opts: list[str]|None,
wd: str|None,
mod_env_sudo: dict[str, str]|None,
mod_env_cmd: dict[str, str]|None,
cmd_input: bytes|None,
opts: list[str] | None,
wd: str | None,
mod_env_sudo: dict[str, str] | None,
mod_env_cmd: dict[str, str] | None,
cmd_input: bytes | None,
verbose: bool,
interactive: bool,
log_prefix: str,
@ -320,20 +360,22 @@ class ExecContext(Base):
def __check_equal_values(d1: dict[str, str], d2: dict[str, str]) -> None:
for key, val in d1.items():
if not d2.get(key, None) in [None, val]:
raise ValueError(f'Outer and inner environments differ at least for {key}: "{val}" != "{d2.get(key)}"')
if d2.get(key, None) not in [None, val]:
raise ValueError(
'Outer and inner environments differ at least for '
f'{key}: "{val}" != "{d2.get(key)}"'
)
fw_cmd: list[str] = []
fw_env: dict[str, str] = {}
if opts is None:
opts = {}
opts = []
if mod_env_cmd:
fw_env.update(mod_env_cmd)
if self.username != 'root':
if mod_env_sudo and mod_env_cmd:
__check_equal_values(mod_env_sudo, mod_env_cmd)
__check_equal_values(mod_env_cmd, mod_env_sudo)
@ -345,7 +387,7 @@ class ExecContext(Base):
fw_cmd.append('--preserve-env=' + ','.join(mod_env_cmd.keys()))
if wd is not None:
opts.extend('-D', wd)
opts.extend(['-D', wd])
wd = None
fw_cmd.extend(opts)
@ -361,20 +403,20 @@ class ExecContext(Base):
verbose = verbose,
cmd_input = cmd_input,
interactive = interactive,
log_prefix = log_prefix
log_prefix = log_prefix,
)
async def sudo(
self,
cmd: list[str],
opts: list[str]|None = None,
wd: str|None = None,
mod_env_sudo: dict[str, str]|None = None,
mod_env_cmd: dict[str, str]|None = None,
opts: list[str] | None = None,
wd: str | None = None,
mod_env_sudo: dict[str, str] | None = None,
mod_env_cmd: dict[str, str] | None = None,
throw: bool = True,
verbose: bool|None = None,
verbose: bool | None = None,
cmd_input: Input = InputMode.OptInteractive,
title: str = None
title: str | None = None,
) -> Result:
# Note that in the calls to the wrapped method, cmd_input == None can
@ -382,9 +424,17 @@ class ExecContext(Base):
assert cmd_input is not None, 'Invalid: cmd_input is None'
ret = Result(None, None, 1)
with self.CallContext(self, title=title, cmd=cmd, cmd_input=cmd_input,
mod_env=mod_env_cmd, wd=wd,
log_prefix='|', throw=throw, verbose=verbose) as cc:
with self.CallContext(
self,
title = title,
cmd = cmd,
cmd_input = cmd_input,
mod_env = mod_env_cmd,
wd = wd,
log_prefix = '|',
throw = throw,
verbose = verbose,
) as cc:
try:
ret = await self._sudo(
cmd = cc.cmd,
@ -416,19 +466,22 @@ class ExecContext(Base):
)
async def _get(
self,
path: str,
wd: str|None,
throw: bool,
verbose: bool|None,
title: str
self, path: str, wd: str | None, throw: bool, verbose: bool | None, title: str
) -> Result:
ret = Result(None, None, 1)
if wd is not None:
path = wd + '/' + path
with self.CallContext(self, title=title, cmd=['cat', path],
cmd_input=InputMode.NonInteractive, wd=None, mod_env=None,
log_prefix='|', throw=throw, verbose=verbose) as cc:
with self.CallContext(
self,
title = title,
cmd = ['cat', path],
cmd_input = InputMode.NonInteractive,
wd = None,
mod_env = None,
log_prefix = '|',
throw = throw,
verbose = verbose,
) as cc:
try:
ret = await self._run(
cmd = cc.cmd,
@ -437,12 +490,12 @@ class ExecContext(Base):
cmd_input = cc.cmd_input,
mod_env = cc.mod_env,
interactive = cc.interactive,
log_prefix = cc.log_prefix
log_prefix = cc.log_prefix,
)
except Exception as e:
return cc.exception(ret, e)
if ret.status != 0 and ret.stderr.decode().find('No such file') != -1:
raise FileNotFoundError(ret.stderr)
if ret.matches_error('No such file'):
raise FileNotFoundError(ret.summarize(cc.cmd, wd = cc.wd))
cc.check_exit_code(ret)
return ret
@ -450,138 +503,176 @@ class ExecContext(Base):
self,
path: str,
content: bytes,
wd: str|None,
wd: str | None,
throw: bool,
verbose: bool|None,
verbose: bool | None,
title: str,
owner: str|None,
group: str|None,
mode: str|None,
owner: str | None,
group: str | None,
mode: str | None,
atomic: bool,
) -> Result:
from .util import pretty_cmd
async def __run(cmd: list[str], cmd_input: Input=InputMode.NonInteractive, **kwargs) -> Result:
return await self.run(cmd, cmd_input=cmd_input, **kwargs)
async def __run(
cmd: list[str],
cmd_input: Input = InputMode.NonInteractive,
**kwargs
) -> Result:
return await self.run(cmd, cmd_input = cmd_input, **kwargs)
ret = Result(None, None, 1)
try:
class RemoteCmd(NamedTuple):
cmd: list[str]
cmd_input: Input = InputMode.NonInteractive
if wd is not None:
path = wd + '/' + path
cmds: list[dict[str, str|list[str]|bool]] = []
out = (await __run(['mktemp', path + '.XXXXXX'])).stdout.decode().strip() if atomic else path
cmds.append({'cmd': ['tee', out], 'cmd_input': content})
cmds: list[RemoteCmd] = []
stdout = (await __run(['mktemp', path + '.XXXXXX'])).stdout_str
if stdout is None:
raise Exception(f'Failed to create tmp-directory on {self.root}')
out = stdout.strip() if atomic else path
cmds.append(RemoteCmd(
cmd = ['tee', out],
cmd_input = content,
))
if owner is not None and group is not None:
cmds.append({'cmd': ['chown', f'{owner}:{group}', out]})
cmds.append(RemoteCmd(
cmd = ['chown', f'{owner}:{group}', out],
))
elif owner is not None:
cmds.append({'cmd': ['chown', owner, out]})
cmds.append(RemoteCmd(
cmd = ['chown', owner, out],
))
elif group is not None:
cmds.append({'cmd': ['chgrp', group, out]})
cmds.append(RemoteCmd(
cmd = ['chgrp', group, out],
))
if mode is not None:
cmds.append({'cmd': ['chmod', mode, out]})
cmds.append(RemoteCmd(
cmd = ['chmod', mode, out],
))
if atomic:
cmds.append({'cmd': ['mv', out, path]})
cmds.append(RemoteCmd(
cmd = ['mv', out, path],
))
await self.open()
try:
for cmd in cmds:
log(DEBUG, f'{self.log_name}: Running {pretty_cmd(cmd['cmd'], wd)}')
ret = await __run(**cmd)
log(DEBUG, f'{self.log_name}: Running {pretty_cmd(cmd.cmd, wd)}')
ret = await __run(cmd.cmd)
return ret
finally:
await self.close()
except:
except Exception as e:
msg = f'Failed to get {path} from {self.root} ({str(e)})'
if throw:
raise
return cc.exception(ret, e)
raise Exception(msg)
log(ERR, msg)
return ret
async def _unlink(self, path: str) -> None:
cmd = ['rm', '-f', path]
await self.run(cmd, cmd_input=InputMode.NonInteractive)
await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _erase(self, path: str) -> None:
cmd = ['rm', '-rf', path]
await self.run(cmd, cmd_input=InputMode.NonInteractive)
await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _rename(self, src: str, dst: str) -> None:
cmd = ['mv', src, dst]
await self.run(cmd, cmd_input=InputMode.NonInteractive)
await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _mkdir(self, name: str, mode: int) -> None:
cmd = ['mkdir', name, '-m', self.__mode_str(mode)]
await self.run(cmd, cmd_input=InputMode.NonInteractive)
await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _mktemp(self, tmpl: str, directory: bool) -> str:
cmd = ['mktemp']
if directory:
cmd.append('-d')
cmd.append(tmpl)
result = await self.run(cmd, cmd_input=InputMode.NonInteractive)
return result.stdout.strip().decode()
result = await self.run(cmd, cmd_input = InputMode.NonInteractive, throw = True)
if result.status != 0 or result.stdout is None:
raise Exception(
f'Failed to create temporary file on {self.root}: {result.summary}'
)
return result.stdout_str
async def _stat(self, path: str, follow_symlinks: bool) -> StatResult:
async def __stat(opts: list[str]) -> str:
mod_env = {
'LC_ALL': 'C'
}
async def __stat(opts: list[str]) -> Result:
mod_env = {'LC_ALL': 'C'}
cmd = ['stat']
if follow_symlinks:
cmd.append('-L')
cmd.extend(opts)
cmd.append(path)
return (await self.run(cmd, mod_env=mod_env, throw=False,
cmd_input=InputMode.NonInteractive)).decode()
return await self.run(
cmd,
mod_env = mod_env,
throw = False,
cmd_input = InputMode.NonInteractive
)
# GNU coreutils stat
gnu_format = _US.join([
"%f", # st_mode in hex
"%i", # st_ino
"%d", # st_dev
"%h", # st_nlink
"%U", # st_uid
"%G", # st_gid
"%s", # st_size
"%.9X", # st_atime
"%.9Y", # st_mtime
"%.9Z", # st_ctime
"%o", # st_blksize hint
"%b", # st_blocks
"%r", # st_rdev
])
gnu_format = _US.join(
[
'%f', # st_mode in hex
'%i', # st_ino
'%d', # st_dev
'%h', # st_nlink
'%U', # st_uid
'%G', # st_gid
'%s', # st_size
'%.9X', # st_atime
'%.9Y', # st_mtime
'%.9Z', # st_ctime
'%o', # st_blksize hint
'%b', # st_blocks
'%r', # st_rdev
]
)
result = await __stat(['--printf', gnu_format])
if result.status == 0:
return _build_stat_result(result.stdout.split(_US), mode_base=16)
if result.status == 0 and result.stdout is not None:
return _build_stat_result(result.stdout_str.split(_US), mode_base = 16)
if not _looks_like_option_error(result.stderr):
if not _looks_like_option_error(result.stderr_str_or_none):
# log(DEBUG, f'GNU stat attempt failed on "{path}" ({str(e)})')
_raise_stat_error(path, result.stderr, result.status)
_raise_stat_error(path, result)
# BSD / macOS / OpenBSD / NetBSD stat
bsd_format = _US.join([
"%p", # st_mode in octal
"%i", # st_ino
"%d", # st_dev
"%l", # st_nlink
"%U", # st_uid
"%G", # st_gid
"%z", # st_size
"%.9Fa", # st_atime
"%.9Fm", # st_mtime
"%.9Fc", # st_ctime
"%k", # st_blksize
"%b", # st_blocks
"%r", # st_rdev
])
bsd_format = _US.join(
[
'%p', # st_mode in octal
'%i', # st_ino
'%d', # st_dev
'%l', # st_nlink
'%U', # st_uid
'%G', # st_gid
'%z', # st_size
'%.9Fa', # st_atime
'%.9Fm', # st_mtime
'%.9Fc', # st_ctime
'%k', # st_blksize
'%b', # st_blocks
'%r', # st_rdev
]
)
result = await __stat(['-n', '-f', bst_format])
if proc.returncode == 0:
return _build_stat_result(proc.stdout.rstrip('\n').split(_US), mode_base=8)
_raise_stat_error(path, result.stderr, result.status)
result = await __stat(['-n', '-f', bsd_format])
stdout = result.stdout_str_or_none
if result.status != 0 or stdout is None:
_raise_stat_error(path, result)
assert stdout is not None # Just there to pacify the linter
return _build_stat_result(stdout.rstrip('\n').split(_US), mode_base = 8)
async def _chown(self, path: str, owner: str|None, group: str|None) -> None:
async def _chown(self, path: str, owner: str | None, group: str | None) -> None:
if owner is None and group is None:
raise ValueError(f'Tried to chown("{path}") without owner and group')
if group is None:
@ -590,7 +681,11 @@ class ExecContext(Base):
ownership = ':' + group
else:
ownership = owner + ':' + group
await self.run(['chown', ownership, path], cmd_input=InputMode.NonInteractive)
assert ownership is not None # Impossible, just there to calm the linter
await self.run(['chown', ownership, path], cmd_input = InputMode.NonInteractive)
async def _chmod(self, path: str, mode: int) -> None:
await self.run(['chmod', self.__mode_str(mode), path], cmd_input=InputMode.NonInteractive)
await self.run(
['chmod', self.__mode_str(mode), path],
cmd_input = InputMode.NonInteractive
)

173
src/python/jw/pkg/lib/FileContext.py
View file

@ -1,19 +1,17 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
import abc, re
import abc
from enum import Enum, auto
from typing import TYPE_CHECKING, Self
from functools import cached_property, cache
from functools import cached_property
from typing import TYPE_CHECKING
from .log import DEBUG, ERR, log
from .Uri import Uri
if TYPE_CHECKING:
from typing import Self
from .log import *
from .base import Input, InputMode, Result, StatResult
from .Uri import Uri
from .ProcFilter import ProcPipeline
from .base import Result, StatResult
from .ProcFilter import ProcFilter, ProcPipeline
class FileContext(abc.ABC):
@ -22,24 +20,27 @@ class FileContext(abc.ABC):
Out = auto()
def __init__(
self,
uri: str|Uri,
interactive: bool|None = None,
verbose_default = False,
chroot: bool = False,
in_pipe: ProcPipeline|None = None,
out_pipe: ProcPipeline|None = None,
):
self,
uri: str | Uri,
interactive: bool | None = None,
verbose_default = False,
chroot: bool = False,
in_pipe: ProcPipeline | None = None,
out_pipe: ProcPipeline | None = None,
):
self.__uri = Uri.pimp(uri)
self.__chroot = chroot
self.__interactive = interactive
self.__verbose_default = verbose_default
self.__log_name: str|None = None
self.__log_name: str | None = None
self.__in_pipe = in_pipe
self.__out_pipe = out_pipe
self.__open_count = 0
if not verbose_default in [True, False]:
raise ValueError(f'Tried to instantiate FileContext with verbose_default = "{verbose_default}"')
if verbose_default not in [True, False]:
raise ValueError(
'Tried to instantiate FileContext with verbose_default '
f'= "{verbose_default}"'
)
async def __aenter__(self):
await self.open()
@ -91,7 +92,9 @@ class FileContext(abc.ABC):
if self.__open_count == 1:
await self._close()
self.__open_count -= 1
assert self.__open_count >= 0, f'Closed file context "{self}" more often than opened'
assert self.__open_count >= 0, (
f'Closed file context "{self}" more often than opened'
)
@property
def uri(self) -> Uri:
@ -106,7 +109,7 @@ class FileContext(abc.ABC):
return self.__uri.path
@property
def username(self) -> str|None:
def username(self) -> str | None:
return self.__uri.username
@property
@ -114,7 +117,7 @@ class FileContext(abc.ABC):
return self.__uri.id
@property
def interactive(self) -> bool|None:
def interactive(self) -> bool | None:
return self.__interactive
@property
@ -123,29 +126,24 @@ class FileContext(abc.ABC):
@abc.abstractmethod
async def _get(
self,
path: str,
wd: str|None,
throw: bool,
verbose: bool|None,
title: str
self, path: str, wd: str | None, throw: bool, verbose: bool | None, title: str
) -> Result:
raise NotImplementedError()
async def get(
self,
path: str,
wd: str|None = None,
wd: str | None = None,
throw: bool = True,
verbose: bool|None = None,
title: str=None,
verbose: bool | None = None,
title: str | None = None,
) -> Result:
ret = await self._get(
self._chroot(path),
wd = wd,
throw = throw,
verbose = verbose,
title = title,
title = title or f'Fetching {path} from {self.uri}',
)
return await self.__in_pipe.run(ret) if self.__in_pipe else ret
@ -153,13 +151,13 @@ class FileContext(abc.ABC):
self,
path: str,
content: bytes,
wd: str|None,
wd: str | None,
throw: bool,
verbose: bool|None,
verbose: bool | None,
title: str,
owner: str|None,
group: str|None,
mode: str|None,
owner: str | None,
group: str | None,
mode: str | None,
atomic: bool,
) -> Result:
raise NotImplementedError()
@ -167,26 +165,26 @@ class FileContext(abc.ABC):
async def put(
self,
path: str,
content: str,
wd: str|None = None,
content: bytes,
wd: str | None = None,
throw: bool = True,
verbose: bool|None = None,
title: str = None,
owner: str|None = None,
group: str|None = None,
mode: int|None = None,
atomic: bool = False
verbose: bool | None = None,
title: str | None = None,
owner: str | None = None,
group: str | None = None,
mode: int | None = None,
atomic: bool = False,
) -> Result:
mode_str = None if mode is None else oct(mode).replace('0o', '0')
if self.__out_pipe is not None:
content = self.__out_pipe.run(content).stdout
result = await self.__out_pipe.run(content)
return await self._put(
self._chroot(path),
content,
result.stdout,
wd = wd,
throw = throw,
verbose = verbose,
title = title,
title = title or f'Pushing content to {path} on {self.uri}',
owner = owner,
group = group,
mode = mode_str,
@ -194,55 +192,73 @@ class FileContext(abc.ABC):
)
async def _unlink(self, path: str) -> None:
raise NotImplementedError(f'{self.log_name}: unlink("{path}") is not implemented')
raise NotImplementedError(
f'{self.log_name}: unlink("{path}") is not implemented'
)
async def unlink(self, path: str) -> None:
return await self._unlink(self._chroot(path))
async def _erase(self, path: str) -> None:
raise NotImplementedError(f'{self.log_name}: erase("{path}") is not implemented')
raise NotImplementedError(
f'{self.log_name}: erase("{path}") is not implemented'
)
async def erase(self, path: str) -> None:
return await self._erase(self._chroot(path))
async def _rename(self, src: str, dst: str) -> None:
raise NotImplementedError(f'{self.log_name}: rename("{path}") is not implemented')
raise NotImplementedError(
f'{self.log_name}: rename("{src}" -> "{dst}") is not implemented'
)
async def rename(self, src: str, dst: str) -> None:
return await self._rename(src, dst)
async def _mkdir(self, path: str, mode: int) -> None:
raise NotImplementedError(f'{self.log_path}: mkdir({path}) is not implemented')
raise NotImplementedError(f'{self.log_name}: mkdir({path}) is not implemented')
async def mkdir(self, path: str, mode: int=0o777) -> None:
async def mkdir(self, path: str, mode: int = 0o777) -> None:
return await self._mkdir(path, mode)
async def _mktemp(self, tmpl: str, directory: bool) -> None:
raise NotImplementedError(f'{self.log_name}: mktemp("{path}") is not implemented')
async def _mktemp(self, tmpl: str, directory: bool) -> str:
raise NotImplementedError(
f'{self.log_name}: mktemp("{tmpl}") is not implemented'
)
async def mktemp(self, tmpl: str, directory: bool=False) -> None:
async def mktemp(self, tmpl: str, directory: bool = False) -> str:
return await self._mktemp(self._chroot(tmpl), directory)
async def _chown(self, path: str, owner: str|None, group: str|None) -> None:
raise NotImplementedError(f'{self.log_name}: chown("{path}") is not implemented')
async def _chown(self, path: str, owner: str | None, group: str | None) -> None:
raise NotImplementedError(
f'{self.log_name}: chown("{path}") is not implemented'
)
async def chown(self, path: str, owner: str|None=None, group: str|None=None) -> None:
async def chown(
self, path: str, owner: str | None = None, group: str | None = None
) -> None:
if owner is None and group is None:
raise ValueError(f'Tried to change ownership of {path} specifying neither owner nor group')
raise ValueError(
f'Tried to change ownership of {path} with neither owner nor group'
)
return await self._chown(self._chroot(path), owner, group)
async def _chmod(self, path: str, mode: int) -> None:
raise NotImplementedError(f'{self.log_name}: chmod("{path}") is not implemented')
raise NotImplementedError(
f'{self.log_name}: chmod("{path}") is not implemented'
)
async def chmod(self, path: str, mode: int) -> None:
return await self._chmod(self._chroot(path), mode)
async def _stat(self, path: str, follow_symlinks: bool) -> StatResult:
raise NotImplementedError(f'{self.log_name}: lstat("{path}") is not implemented')
raise NotImplementedError(
f'{self.log_name}: lstat("{path}") is not implemented'
)
async def stat(self, path: str, follow_symlinks: bool=True) -> StatResult:
async def stat(self, path: str, follow_symlinks: bool = True) -> StatResult:
if not isinstance(path, str):
raise TypeError(f"path must be str, got {type(path).__name__}")
raise TypeError(f'path must be str, got {type(path).__name__}')
return await self._stat(self._chroot(path), follow_symlinks)
async def _file_exists(self, path: str) -> bool:
@ -261,10 +277,17 @@ class FileContext(abc.ABC):
async def _is_dir(self, path: str, follow_symlinks: bool) -> bool:
import stat
try:
return stat.S_ISDIR((await self._stat(path, follow_symlinks)).mode)
except NotImplementedError:
log(DEBUG, f'{self.log_name} doesn\'t implement stat(), judging by trailing slash if {path} is a directory')
log(
DEBUG,
(
f"{self.log_name} doesn't implement stat(), judging by trailing "
'slash if {path} is a directory'
),
)
return path[-1] == '/'
except FileNotFoundError as e:
log(DEBUG, f'{self.log_name}: Failed to stat({path}) ({str(e)})')
@ -274,22 +297,28 @@ class FileContext(abc.ABC):
raise
return False
async def is_dir(self, path: str, follow_symlinks=True) -> bool:
return await self._is_dir(self._chroot(path), follow_symlinks=follow_symlinks)
async def is_dir(self, path: str, follow_symlinks = True) -> bool:
return await self._is_dir(self._chroot(path), follow_symlinks = follow_symlinks)
@classmethod
def create(cls, uri: str|Uri, *args, **kwargs) -> Self:
def create(cls, uri: str | Uri, *args, **kwargs) -> FileContext:
uri = Uri.pimp(uri)
match uri.protocol:
case 'local' | 'file':
from .ec.Local import Local
return Local(uri, *args, **kwargs)
case 'ssh':
from .ec.SSHClient import ssh_client
return ssh_client(uri, *args, **kwargs)
case 'http' | 'https':
from .ec.Curl import Curl
return Curl(uri, *args, **kwargs)
case _:
pass
raise Exception(f'Can\'t create file context instance for "{uri}" with unsupported protocol "{uri.protocol}"')
raise Exception(
f'Can\'t create file context instance for "{uri}" with unsupported '
f'protocol "{uri.protocol}"'
)

44
src/python/jw/pkg/lib/Package.py
View file

@ -1,32 +1,35 @@
# -*- coding: utf-8 -*-
from typing import Any
meta_tags = [
"name",
"vendor",
"packager",
"url",
"maintainer",
'name',
'vendor',
'packager',
'url',
'maintainer',
]
class Package:
name: str = None
vendor: str|None = None
packager: str|None = None
url: str|None = None
maintainer: str|None = None
name: str
vendor: str | None = None
packager: str | None = None
url: str | None = None
maintainer: str | None = None
@classmethod
def parse_spec_str(cls, spec: str, delimiter='|'):
def parse_spec_str(cls, spec: str, delimiter = '|'):
tags = spec.split(delimiter)
if len(tags) != 5:
raise ValueError(f'Invalid package spec string "{spec}"')
return cls(name=tags[0], vendor=tags[1], packager=tags[2], url=tags[3], maintainer=tags[4])
return cls(
name = tags[0],
vendor = tags[1],
packager = tags[2],
url = tags[3],
maintainer = tags[4],
)
@classmethod
def parse_specs_str(cls, specs: str, delimiter='|'):
def parse_specs_str(cls, specs: str, delimiter = '|'):
ret: list[Package] = []
for spec in specs.splitlines():
ret.append(cls.parse_spec_str(spec))
@ -39,7 +42,14 @@ class Package:
ret[tag] = mapping.get(tag, '')
return ret
def __init__(self, name: str, vendor: str|None=None, packager: str|None=None, url: str|None=None, maintainer: str|None=None):
def __init__(
self,
name: str,
vendor: str | None = None,
packager: str | None = None,
url: str | None = None,
maintainer: str | None = None,
):
self.name = name
self.vendor = vendor
self.packager = packager

10
src/python/jw/pkg/lib/PackageFilter.py
View file

@ -1,6 +1,5 @@
# -*- coding: utf-8 -*-
import abc, re
import abc
import re
from .Package import Package
@ -23,4 +22,7 @@ class PackageFilterString(PackageFilter):
self.__definition = url_rx_str
def _match(self, package: Package) -> bool:
return re.search(self.__definition, package.url) is not None
url = package.url
if url is None:
return False
return re.search(self.__definition, url) is not None

22
src/python/jw/pkg/lib/ProcFilter.py
View file

@ -1,10 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
import abc
from typing import TYPE_CHECKING
from .base import Result
if TYPE_CHECKING:
@ -13,37 +12,40 @@ if TYPE_CHECKING:
class ProcFilter(abc.ABC):
@abc.abstractmethod
async def _run(self, data: bytes) -> Result:
async def _run(self, data: bytes | None) -> Result:
raise NotImplementedError()
async def run(self, data: bytes) -> Result:
async def run(self, data: bytes | None) -> Result:
return await self._run(data)
class ProcFilterIdentity(ProcFilter):
async def _run(self, data: bytes) -> Result:
async def _run(self, data: bytes | None) -> Result:
return Result(data, None, 0)
class ProcPipeline:
def __init__(self, f: Iterable[ProcFilter]|ProcFilter = []) -> None:
def __init__(self, f: Iterable[ProcFilter] | ProcFilter = []) -> None:
self.__filters: list[ProcFilter] = []
self.append(f)
def append(self, f: ProcFilter|Iterable[ProcFilter]) -> None:
def append(self, f: ProcFilter | Iterable[ProcFilter]) -> None:
if not isinstance(f, ProcFilter):
for e in f:
self.append(e)
return
self.__filters.append(f)
async def run(self, data: bytes|Result) -> Result:
async def run(self, data: None | bytes | Result) -> Result:
ret = data if isinstance(data, Result) else Result(data, None, 0)
for f in self.__filters:
ret = await f.run(ret.stdout)
return ret
async def run(data: bytes|Result, chain: ProcFilter|list[ProcFilter]|ProcPipeline|None = None) -> Result:
async def run(
data: bytes | Result,
chain: ProcFilter | list[ProcFilter] | ProcPipeline | None = None,
) -> Result:
if chain is None:
if isinstance(data, Result):
return data

16
src/python/jw/pkg/lib/ProcFilterGpg.py
View file

@ -1,10 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
from .ProcFilter import ProcFilter
from .base import Result
from .ProcFilter import ProcFilter
if TYPE_CHECKING:
from .ExecContext import ExecContext
@ -14,9 +13,12 @@ class ProcFilterGpg(ProcFilter):
def __init__(self, ec: ExecContext) -> None:
self.__ec = ec
async def _run(self, data: bytes) -> Result:
return await self.__ec.run([
"gpg",
async def _run(self, data: bytes | None) -> Result:
if data is None:
raise Exception('No data for GPG to decrypt')
return await self.__ec.run(
[
'gpg',
'--batch',
'--yes',
'--quiet',
@ -24,5 +26,5 @@ class ProcFilterGpg(ProcFilter):
'--decrypt',
],
cmd_input = data,
throw = True
throw = True,
)

85
src/python/jw/pkg/lib/TarIo.py
View file

@ -1,29 +1,34 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import Self
import abc, io
import abc
import io
import tarfile
from tarfile import TarFile
from tarfile import TarFile, TarInfo
from .base import StatResult
from .CopyContext import CopyContext
from .FileContext import FileContext
from .log import *
from .ExecContext import ExecContext
from .log import DEBUG, ERR, log
class TarIo(CopyContext):
def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs, chroot=False)
kwargs['chroot'] = False
super().__init__(*args, **kwargs)
def _match(self, path: str, path_filter: list[str]) -> bool:
return path in path_filter
def _filter_tar_file(self, blob: bytes, path_filter: list[str]|None=None, matched: list[str]|None=None) -> bytes:
def _filter_tar_file(
self,
blob: bytes,
path_filter: list[str] | None = None,
matched: list[str] | None = None,
) -> bytes:
ret = io.BytesIO()
with tarfile.open(fileobj=ret, mode='w') as tf_out:
tf_in = TarFile(fileobj=io.BytesIO(blob))
with tarfile.open(fileobj = ret, mode = 'w') as tf_out:
tf_in = TarFile(fileobj = io.BytesIO(blob))
for info in tf_in.getmembers():
if path_filter is not None and not self._match(info.name, path_filter):
continue
@ -34,13 +39,18 @@ class TarIo(CopyContext):
tf_out.addfile(info, buf)
return ret.getvalue()
async def _read_filtered(self, path, path_filter: list[str]|None=None, matched: list[str]|None=None) -> bytes:
async def _read_filtered(
self,
path,
path_filter: list[str] | None = None,
matched: list[str] | None = None,
) -> bytes:
try:
blob = (await self.src.get(path)).stdout
except Exception as e:
log(ERR, f'Failed to read tar file "{path}" ({str(e)}')
raise
return self._filter_tar_file(blob, path_filter, matched=matched)
return self._filter_tar_file(blob, path_filter, matched = matched)
def _add(self, tf: TarFile, path: str, st: StatResult, contents: bytes) -> None:
file_obj = io.BytesIO(contents)
@ -50,37 +60,34 @@ class TarIo(CopyContext):
info.uname = st.owner
info.gname = st.group
info.size = st.size
info.atime = st.atime
info.mtime = st.mtime
info.ctime = st.ctime
tf.addfile(info, io.BytesIO(file_obj))
async def _add_from_path(self, src: FileContext, tf: TarFile, path: str) -> None:
contents = await src.get(path)
st = await self.stat(path)
self._add(tf, path, st, contents)
info.mtime = int(st.mtime)
tf.addfile(info, file_obj)
@abc.abstractmethod
async def _extract(self, blob: bytes, root: str|None=None) -> None:
async def _extract(self, blob: bytes, root: str | None = None) -> None:
raise NotImplementedError()
async def extract(self, root: str|None=None, path_filter: list[str]|None=None) -> list[str]:
async def extract(
self,
root: str | None = None,
path_filter: list[str] | None = None
) -> list[str]:
ret: list[str] = []
filtered = await self._read_filtered(self.src.root, path_filter, matched=ret)
await self._extract(blob=filtered, root=root)
filtered = await self._read_filtered(self.src.root, path_filter, matched = ret)
await self._extract(blob = filtered, root = root)
return ret
@classmethod
def create(cls, *args, type: str=None, **kwargs):
def create(cls, *args, type: str | None = None, **kwargs):
if type is not None:
raise NotImplementedError
#return TarIoTarFile(*args, **kwargs)
# return TarIoTarFile(*args, **kwargs)
return TarIoTarExec(*args, **kwargs)
class TarIoTarFile(TarIo):
async def _extract(self, blob: bytes, root: str|None=None) -> None:
tf = TarFile(fileobj=io.BytesIO(blob))
async def _extract(self, blob: bytes, root: str | None = None) -> None:
tf = TarFile(fileobj = io.BytesIO(blob))
for info in tf.getmembers():
log(DEBUG, f'Extracting {info.name}')
path = root + '/' + info.name if root else info.name
@ -96,14 +103,24 @@ class TarIoTarFile(TarIo):
buf.read(),
owner = info.uname,
group = info.gname,
mode = info.mode,
mode = info.mode,
)
class TarIoTarExec(TarIo):
async def _extract(self, blob: bytes, root: str|None=None) -> None:
@property
def dst(self) -> ExecContext:
ret = super().dst
if not isinstance(ret, ExecContext):
raise Exception(
'Tried to get executable destination context from copy '
'context, which only has a file context'
)
return ret
async def _extract(self, blob: bytes, root: str | None = None) -> None:
cmd = ['tar']
if root is not None:
cmd += ['-C', root]
cmd += ['-x', '-f', '-']
await self.dst.run(cmd, cmd_input=blob)
await self.dst.run(cmd, cmd_input = blob)

86
src/python/jw/pkg/lib/Types.py
View file

@ -1,24 +1,29 @@
# -*- coding: utf-8 -*-
import abc
import os
import re
import sys
from typing import TypeVar, Generic
import abc, re, sys, os
from collections.abc import Iterable, Iterator
from collections.abc import Iterator
from typing import TYPE_CHECKING, Generic, Iterable, TypeVar
from .log import *
from .log import OFF, log, parse_log_level
T = TypeVar("T")
if TYPE_CHECKING:
from typing import Any
class Types(abc.ABC, Iterable[T], Generic[T]): # export
T = TypeVar('T')
def __iter__(self) -> Iterator[T]:
class Types(abc.ABC, Iterable[type[T]], Generic[T]): # export
def __iter__(self) -> Iterator[type[T]]:
return iter(self._classes())
@abc.abstractmethod
def _classes(self) -> Iterable[T]:
def _classes(self) -> Iterable[type[T]]:
pass
@property
def classes(self) -> Iterable[T]:
def classes(self) -> Iterable[type[T]]:
return self._classes()
@abc.abstractmethod
@ -27,14 +32,20 @@ class Types(abc.ABC, Iterable[T], Generic[T]): # export
def dump(self, prio: int, *args, **kwargs) -> None:
contents = self._stringify()
log(prio, ",--- ", *args, **kwargs)
log(prio, ',--- ', *args, **kwargs)
for line in contents:
log(prio, "| " + line)
log(prio, "`--- ", *args, **kwargs)
log(prio, '| ' + line)
log(prio, '`--- ', *args, **kwargs)
class LoadTypes(Types): # export
class LoadTypes(Types[T]): # export
def __init__(self, mod_names: list[str], type_name_filter: str=None, type_filter: list[T]=[], debug_level=None):
def __init__(
self,
mod_names: list[str],
type_name_filter: str | None = None,
type_filter: list[type[T]] = [],
debug_level = None,
):
if debug_level is None:
val = os.getenv('JW_LOG_LEVEL_LOAD_TYPES')
if val is not None:
@ -45,7 +56,7 @@ class LoadTypes(Types): # export
self.__type_name_filter = type_name_filter
self.__type_filter = type_filter
self.__mod_names = mod_names
self.__classes: list[type[Any]]|None = None
self.__classes: list[type[T]] | None = None
def _debug(self, *args, **kwargs) -> None:
if self.__debug_level != OFF:
@ -53,37 +64,56 @@ class LoadTypes(Types): # export
def _stringify(self):
return [
"type_name_filter: " + str(self.__type_name_filter),
"type_filter: " + ', '.join([str(f) for f in self.__type_filter]),
"mod_names: " + ', '.join(self.__mod_names)
'type_name_filter: ' + str(self.__type_name_filter),
'type_filter: ' + ', '.join([str(f) for f in self.__type_filter]),
'mod_names: ' + ', '.join(self.__mod_names),
]
def _classes(self) -> Iterable[T]:
def _classes(self) -> Iterable[type[T]]:
if self.__classes is None:
import importlib, inspect
rx: Any|None = None
import importlib
import inspect
rx: Any | None = None
if self.__type_name_filter is not None:
rx = re.compile(self.__type_name_filter)
ret: list[Any] = []
for mod_name in self.__mod_names:
if mod_name != '__main__':
importlib.import_module(mod_name)
for member_name, c in inspect.getmembers(sys.modules[mod_name], inspect.isclass):
for member_name, c in inspect.getmembers(
sys.modules[mod_name], inspect.isclass
):
if rx is not None and not re.match(rx, member_name):
self._debug('o "{}.{}" has wrong name'.format(mod_name, member_name))
self._debug(
'o "{}.{}" has wrong name'.format(mod_name, member_name)
)
continue
if inspect.isabstract(c):
self._debug('o "{}.{}" is abstract'.format(mod_name, member_name))
self._debug(
'o "{}.{}" is abstract'.format(mod_name, member_name)
)
continue
if self.__type_filter:
for tp in self.__type_filter:
if issubclass(c, tp):
break
self._debug('o "{}.{}" is not of type {}'.format(mod_name, member_name, tp))
self._debug(
'o "{}.{}" is not of type {}'.format(
mod_name, member_name, tp
)
)
else:
self._debug('o "{}.{}" doesn\'t match type filter'.format(mod_name, member_name))
self._debug(
'o "{}.{}" doesn\'t match type filter'.format(
mod_name, member_name
)
)
continue
self._debug('o "{}.{}" is fine, adding'.format(mod_name, member_name))
self._debug(
'o "{}.{}" is fine, adding'.format(mod_name, member_name)
)
ret.append(c)
self.__classes = ret
return self.__classes

54
src/python/jw/pkg/lib/Uri.py
View file

@ -1,20 +1,21 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
from functools import cached_property
import copy
from functools import cached_property
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import urllib.parse
from typing import Self
import urllib
# Make sure URIs are interpreted indentically everywhere
class Uri:
def __assemble(self, scheme: bool, credentials: bool, secure: bool, path: bool) -> str:
def __assemble(
self, scheme: bool, credentials: bool, secure: bool, path: bool
) -> str:
ret = ''
if scheme:
ret += f'{self.protocol}://'
@ -33,8 +34,8 @@ class Uri:
def __init__(self, string: str) -> None:
self.__string = string
self.__username: str|None = None
self.__password: str|None = None
self.__username: str | None = None
self.__password: str | None = None
def __repr__(self) -> str:
return self.full
@ -45,10 +46,11 @@ class Uri:
@cached_property
def __p(self) -> urllib.parse.ParseResult:
from urllib.parse import urlparse
return urlparse(self.__string)
@classmethod
def pimp(cls, url: str|Self) -> Uri:
def pimp(cls, url: str | Self) -> Uri:
if isinstance(url, Uri):
return url
return Uri(url)
@ -69,7 +71,7 @@ class Uri:
return self.scheme.replace('://', '')
@property
def username(self) -> str|None:
def username(self) -> str | None:
if self.__username is None:
return self.__p.username
return self.__username
@ -78,7 +80,7 @@ class Uri:
self.__username = username
@property
def password(self) -> str|None:
def password(self) -> str | None:
if self.__password is None:
return self.__p.password
return self.__password
@ -87,15 +89,15 @@ class Uri:
self.__password = password
@cached_property
def hostname(self) -> str|None:
def hostname(self) -> str | None:
return self.__p.hostname
@cached_property
def port(self) -> int|None:
def port(self) -> int | None:
return self.__p.port
@cached_property
def port_str(self) -> str|None:
def port_str(self) -> str | None:
if self.port is None:
return None
return str(self.port)
@ -110,11 +112,15 @@ class Uri:
@cached_property
def authority(self) -> str:
return self.__assemble(scheme=False, credentials=True, secure=False, path=False)
return self.__assemble(
scheme = False, credentials = True, secure = False, path = False
)
@cached_property
def origin(self) -> str:
return self.__assemble(scheme=False, credentials=False, secure=True, path=False)
return self.__assemble(
scheme = False, credentials = False, secure = True, path = False
)
@cached_property
def scheme_plus_authority(self) -> str:
@ -122,15 +128,21 @@ class Uri:
@cached_property
def id(self) -> str:
return self.__assemble(scheme=True, credentials=True, secure=True, path=False)
return self.__assemble(
scheme = True, credentials = True, secure = True, path = False
)
@cached_property
def full(self) -> str:
return self.__assemble(scheme=True, credentials=True, secure=False, path=True)
return self.__assemble(
scheme = True, credentials = True, secure = False, path = True
)
@cached_property
def safe_full_with_username(self) -> str:
return self.__assemble(scheme=True, credentials=True, secure=True, path=True)
return self.__assemble(
scheme = True, credentials = True, secure = True, path = True
)
def __new_with_path(self, base: str, path: str) -> Self:
ret = copy.deepcopy(self)
@ -155,4 +167,4 @@ class Uri:
return self.__new_with_path(self.__string, path)
def new_replace_path(self, path: str) -> Self:
return self.__new_with_path(self.schema_plus_authority, path)
return self.__new_with_path(self.scheme_plus_authority, path)

165
src/python/jw/pkg/lib/base.py
View file

@ -1,12 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from enum import Enum, auto
from typing import NamedTuple, TypeAlias, TYPE_CHECKING
import os
if TYPE_CHECKING:
from typing import Type
from enum import Enum, auto
from typing import NamedTuple, TypeAlias
class InputMode(Enum):
Interactive = auto()
@ -16,32 +13,158 @@ class InputMode(Enum):
Input: TypeAlias = InputMode | bytes | str
class Result(NamedTuple):
class Result:
stdout: str|None
stderr: str|None
status: int|None
def __init__(
self,
stdout: bytes | None,
stderr: bytes | None,
status: int,
encoding: str = 'UTF-8',
strip: bool = True,
cmd: list[str] | None = None,
wd: str | None = None,
) -> None:
self.__stdout = stdout
self.__stderr = stderr
self.__status = status
self.__encoding = encoding
self.__strip = strip
self.__cmd = cmd
self.__wd = wd
def decode(self, encoding='UTF-8', errors='replace') -> Result:
return Result(
self.stdout.decode(encoding, errors=errors) if self.stdout is not None else None,
self.stderr.decode(encoding, errors=errors) if self.stderr is not None else None,
self.status
)
def __decode(self, stdxxx: bytes | None) -> str | None:
if stdxxx is None:
return None
ret = stdxxx.decode(self.encoding)
if self.strip:
return ret.strip()
return ret
@property
def status(self) -> int | None:
return self.__status
@property
def encoding(self) -> str:
return self.__encoding
@encoding.setter
def encoding(self, value: str) -> None:
self.__encoding = value
@property
def strip(self) -> bool:
return self.__strip
@strip.setter
def strip(self, value: bool) -> None:
self.__strip = value
@property
def cmd(self) -> list[str] | None:
return self.__cmd
@cmd.setter
def cmd(self, value: list[str]) -> None:
self.__cmd = value
@property
def wd(self) -> str | None:
return self.__wd
@wd.setter
def wd(self, value: str) -> None:
self.__wd = value
def matches_error(self, pattern: str) -> bool:
if self.status == 0:
return False
err = self.stderr_str
if err is None:
return False
import re
return re.search(pattern, err) is not None
def __summarize(self, cmd: list[str] | None, wd: str | None = None) -> str:
if cmd is None:
cmd = self.__cmd
call = ''
if cmd is not None:
from .util import pretty_cmd
if wd is None:
wd = self.__wd
call = f'"{pretty_cmd(cmd, wd)}" '
ret = f'Command {call}has exited with status {self.__status}'
call = pretty_cmd(cmd, wd)
if self.status != 0:
ret += f' -> stderr="{self.__stderr!r}"'
else:
if self.__stdout:
ret += f' -> stdout has {len(self.__stdout)} bytes'
else:
ret += ' -> stdout = None'
return ret
def summarize(self, cmd: list[str] | None = None, wd: str | None = None) -> str:
return self.__summarize(cmd, wd)
@property
def summary(self) -> str:
return self.__summarize(None, None)
@property
def stdout(self) -> bytes:
if self.__stdout is None:
raise Exception(f'Result has no standard output stream: {self.summary}')
return self.__stdout
@property
def stdout_or_none(self) -> bytes | None:
return self.__stdout
@property
def stdout_str_or_none(self) -> str | None:
return self.__decode(self.__stdout)
@property
def stdout_str(self) -> str:
return self.stdout.decode(self.__encoding)
@property
def stderr(self) -> bytes:
if self.__stderr is None:
raise Exception(f'Result has no standard error stream: {self.summary}')
return self.__stderr
@property
def stderr_or_none(self) -> bytes | None:
return self.__stderr
@property
def stderr_str_or_none(self) -> str | None:
return self.__decode(self.__stderr)
@property
def stderr_str(self) -> str:
return self.stderr.decode(self.__encoding)
class StatResult(NamedTuple):
mode: int
owner: str
group: str
size: int
atime: int
mtime: int
ctime: int
atime: float
mtime: float
ctime: float
@classmethod
def from_os(cls, rhs: os.stat_result) -> StatResult:
import pwd, grp
import grp
import pwd
return StatResult(
rhs.st_mode,
pwd.getpwuid(rhs.st_uid).pw_name,

38
src/python/jw/pkg/lib/distros/arch/Distro.py
View file

@ -1,53 +1,65 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
from ...Distro import Distro as Base
if TYPE_CHECKING:
from typing import Iterable
from ...base import Result
from ...Package import Package
class Distro(Base):
async def pacman(self, args: list[str], verbose: bool=True, sudo: bool=True) -> Result:
async def pacman(
self, args: list[str], verbose: bool = True, sudo: bool = True
) -> Result:
cmd = ['/usr/bin/pacman']
if not self.interactive:
cmd.extend(['--noconfirm'])
cmd.extend(args)
if sudo:
return await self.sudo(cmd, verbose=verbose)
return await self.run(cmd, verbose=verbose)
return await self.sudo(cmd, verbose = verbose)
return await self.run(cmd, verbose = verbose)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
async def _ref(self) -> None:
raise NotImplementedError('distro refresh is not yet implemented for Arch-like distributions')
raise NotImplementedError(
'distro refresh is not yet implemented for Arch-like distributions'
)
async def _dup(self, download_only: bool) -> None:
args = ['-Su']
if args.download_only:
if download_only:
args.append('-w')
return await self.pacman(args)
await self.pacman(args)
async def _reboot_required(self, verbose: bool) -> bool:
raise NotImplementedError('distro reboot-required is not yet implemented for Arch-like distributions')
raise NotImplementedError(
'distro reboot-required is not yet implemented for Arch-like distributions'
)
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
raise NotImplementedError('distro select is not yet implemented for Arch-like distributions')
raise NotImplementedError(
'distro select is not yet implemented for Arch-like distributions'
)
async def _install(self, names: Iterable[str], only_update: bool) -> None:
if only_update:
raise NotImplementedError('--only-update is not yet implemented for pacman')
args = ['-S', '--needed']
args.extend(args.packages)
args.extend(names)
await self.pacman(args)
async def _delete(self, names: Iterable[str]) -> None:
raise NotImplementedError('distro delete not yet implemented for Arch-like distributions')
raise NotImplementedError(
'distro delete not yet implemented for Arch-like distributions'
)
async def _pkg_files(self, name: str) -> Iterable[str]:
raise NotImplementedError('distro pkg ls yet implemented for Arch-like distributions')
raise NotImplementedError(
'distro pkg ls yet implemented for Arch-like distributions'
)

43
src/python/jw/pkg/lib/distros/debian/Distro.py
View file

@ -1,47 +1,52 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
import os
from ...log import *
from typing import TYPE_CHECKING
from ...Distro import Distro as Base
from ...pm.dpkg import run_dpkg, run_dpkg_query, query_packages, list_files
from ...log import NOTICE, log
from ...pm.dpkg import list_files, query_packages, run_dpkg
if TYPE_CHECKING:
from typing import Iterable
from ...base import Result
from ...Package import Package
class Distro(Base):
async def apt_get(self, args: list[str], verbose: bool=True, sudo: bool=True):
async def apt_get(
self, args: list[str], verbose: bool = True, sudo: bool = True
) -> Result:
cmd = ['/usr/bin/apt-get']
mod_env_cmd = None
if not self.interactive:
cmd.extend(['--yes', '--quiet'])
mod_env_cmd = { 'DEBIAN_FRONTEND': 'noninteractive' }
mod_env_cmd = {'DEBIAN_FRONTEND': 'noninteractive'}
cmd.extend(args)
if sudo:
return await self.sudo(cmd, verbose=verbose, mod_env_cmd=mod_env_cmd)
return await self.run(cmd, verbose=verbose)
return (
await
self.sudo(cmd, verbose = verbose, mod_env_cmd = mod_env_cmd, throw = True)
if sudo else await self.run(cmd, verbose = verbose)
)
async def dpkg(self, *args, **kwargs):
return await run_dpkg(*args, ec=self.ctx, **kwargs)
async def dpkg(self, *args, **kwargs) -> str:
kwargs.setdefault('ec', self.ctx)
return await run_dpkg(*args, **kwargs)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
async def _ref(self) -> None:
return await self.apt_get(['update'])
await self.apt_get(['update'])
async def _dup(self, download_only: bool) -> None:
args: list[str] = []
if download_only:
args.append('--download-only')
args.append('upgrade')
return await self.apt_get(args)
await self.apt_get(args)
async def _reboot_required(self, verbose: bool) -> bool:
reboot_required = '/run/reboot_required'
@ -56,11 +61,11 @@ class Distro(Base):
print(content.strip())
return True
if verbose:
log(NOTICE, f'No. {reboot_required} doesn\'t exist.')
log(NOTICE, f"No. {reboot_required} doesn't exist.")
return False
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
return await query_packages(names, ec=self.ctx)
return await query_packages(names, ec = self.ctx)
async def _install(self, names: Iterable[str], only_update: bool) -> None:
args = ['install']
@ -68,10 +73,10 @@ class Distro(Base):
args.append('--only-upgrade')
args.append('--no-install-recommends')
args.extend(names)
return await self.apt_get(args)
await self.apt_get(args)
async def _delete(self, names: Iterable[str]) -> None:
return await self.dpkg(['-P', *names], sudo=True)
await self.dpkg(['-P', *names], sudo = True)
async def _pkg_files(self, name: str) -> Iterable[str]:
return await list_files(name, ec=self.ctx)
return await list_files(name, ec = self.ctx)

45
src/python/jw/pkg/lib/distros/suse/Distro.py
View file

@ -1,62 +1,71 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
from ...Distro import Distro as Base
from ...pm.rpm import run_rpm, query_packages, list_files
from ...pm.rpm import list_files, query_packages, run_rpm
if TYPE_CHECKING:
from typing import Iterable
from ...base import Result
from ...ExecContext import ExecContext
from ...Package import Package
class Distro(Base):
async def zypper(self, args: list[str], verbose: bool=True, sudo: bool=True) -> Result:
async def zypper(
self, args: list[str], verbose: bool = True, sudo: bool = True
) -> Result:
cmd = ['/usr/bin/zypper']
if not self.interactive:
cmd.extend(['--non-interactive', '--gpg-auto-import-keys', '--no-gpg-checks'])
cmd.extend(
['--non-interactive', '--gpg-auto-import-keys', '--no-gpg-checks']
)
cmd.extend(args)
if sudo:
return await self.sudo(cmd, verbose=verbose)
return await self.run(cmd, verbose=verbose)
return (
await self.sudo(cmd, verbose = verbose)
if sudo else await self.run(cmd, verbose = verbose)
)
async def rpm(self, *args, **kwargs) -> Result:
return await run_rpm(*args, ec=self.ctx, **kwargs)
async def rpm(self, *args, ec: ExecContext | None = None, **kwargs) -> str:
if ec is None:
ec = self.ctx
kwargs['ec'] = ec
return await run_rpm(*args, **kwargs)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
async def _ref(self) -> None:
return await self.zypper(['refresh'])
await self.zypper(['refresh'])
async def _dup(self, download_only: bool) -> None:
args = ['dup', '--force-resolution', '--auto-agree-with-licenses']
if download_only:
args.append('--download-only')
return await self.zypper(args)
await self.zypper(args)
async def _reboot_required(self, verbose: bool) -> bool:
opts = []
if not verbose:
pass
#opts.append('--quiet')
# opts.append('--quiet')
opts.append('needs-rebooting')
stdout, stderr, ret = await self.zypper(opts, sudo=False, verbose=verbose)
ret = await self.zypper(opts, sudo = False, verbose = verbose)
if ret != 0:
return True
return False
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
return await query_packages(names, ec=self.ctx)
return await query_packages(names, ec = self.ctx)
async def _install(self, names: Iterable[str], only_update: bool) -> None:
cmd = 'update' if only_update else 'install'
return await self.zypper([cmd, *names])
await self.zypper([cmd, *names])
async def _delete(self, names: Iterable[str]) -> None:
return await self.rpm(['-e', *names], sudo=True)
await self.rpm(['-e', *names], sudo = True)
async def _pkg_files(self, name: str) -> Iterable[str]:
return await list_files(name, ec=self.ctx)
return await list_files(name, ec = self.ctx)

35
src/python/jw/pkg/lib/ec/Curl.py
View file

@ -1,32 +1,35 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
from ..FileContext import FileContext as Base
from ..base import Result
from ..FileContext import FileContext as Base
if TYPE_CHECKING:
from ..ExecContext import ExecContext
from ..Uri import Uri
from .Local import Local
class Curl(Base):
def __init__(self, uri: str|Uri, *args, ec: ExecContext|None=None, **kwargs) -> None:
super().__init__(uri=uri, *args, **kwargs)
self.__ec: ExecContext|None = ec
if ec is None:
def __init__(
self, uri: str | Uri, *args, ec: ExecContext | None = None, **kwargs
) -> None:
def __local() -> Local:
from .Local import Local
self.__ec = Local(interactive=False, *args, **kwargs)
return Local(interactive = False, *args, **kwargs)
# MyPy complains for reasons I don't understand:
# E: "__init__" of "FileContext" gets multiple values for keyword # argument
# "uri" [misc]
super().__init__(uri = uri, *args, **kwargs) # type: ignore[misc]
self.__ec = ec if ec else __local()
async def _get(
self,
path: str,
wd: str|None,
throw: bool,
verbose: bool|None,
title: str
self, path: str, wd: str | None, throw: bool, verbose: bool | None, title: str
) -> Result:
cmd = ['curl']
if verbose is None:
@ -38,5 +41,5 @@ class Curl(Base):
path = wd + '/' + path
if not len(path) or path[0] != '/':
path = '/' + path
cmd.append(self.url.to_string + self._chroot(path))
return await self.__ec.run(cmd, throw=throw, verbose=verbose)
cmd.append(self.uri.to_string + self._chroot(path))
return await self.__ec.run(cmd, throw = throw, verbose = verbose)

92
src/python/jw/pkg/lib/ec/Local.py
View file

@ -1,91 +1,96 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING
import os, sys, subprocess, asyncio, pwd, grp, stat
import asyncio
import grp
import os
import pwd
import sys
from functools import cache
from typing import TYPE_CHECKING
from ..ExecContext import ExecContext as Base
from ..base import Result, StatResult
from ..log import *
from ..util import pretty_cmd
from ..ExecContext import ExecContext as Base
from ..log import ERR, NOTICE, log
if TYPE_CHECKING:
from ..Uri import Uri
class Local(Base):
def __init__(self, uri: str|Uri='local', *args, **kwargs) -> None:
def __init__(self, uri: str | Uri = 'local', *args, **kwargs) -> None:
super().__init__(uri, *args, **kwargs)
@cache
def _username(self) -> str:
return pwd.getpwuid(os.getuid()).pw_name,
return pwd.getpwuid(os.getuid()).pw_name
async def _run(
self,
cmd: list[str],
wd: str|None,
wd: str | None,
verbose: bool,
cmd_input: bytes|None,
mod_env: dict[str, str]|None,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str
log_prefix: str,
) -> Result:
def __log(prio, *args, verbose=verbose):
def __log(prio, *args, verbose = verbose):
if verbose:
log(prio, log_prefix, *args)
def __make_pty_reader(collector: list[bytes], enc_for_verbose: str):
def _read(fd):
ret = os.read(fd, 1024)
if not ret:
return ret
collector.append(ret)
return ret
return _read
cwd: str|None = None
cwd: str | None = None
if wd is not None:
cwd = os.getcwd()
os.chdir(wd)
try:
# -- interactive mode
if interactive:
import pty
def _spawn():
# Apply env in PTY mode by temporarily updating os.environ around spawn.
# Apply env in PTY mode by temporarily updating os.environ
# around spawn.
if mod_env:
old_env = os.environ.copy()
try:
os.environ.update(mod_env)
return pty.spawn(cmd, master_read=reader)
return pty.spawn(cmd, master_read = reader)
finally:
os.environ.clear()
os.environ.update(old_env)
return pty.spawn(cmd, master_read=reader)
return pty.spawn(cmd, master_read = reader)
stdout_chunks: list[bytes] = []
enc_for_verbose = sys.stdout.encoding or "utf-8"
enc_for_verbose = sys.stdout.encoding or 'utf-8'
reader = __make_pty_reader(stdout_chunks, enc_for_verbose)
exit_code = await asyncio.to_thread(_spawn)
# PTY merges stdout/stderr
stdout = b"".join(stdout_chunks) if stdout_chunks else None
stdout = b''.join(stdout_chunks) if stdout_chunks else None
return Result(stdout, None, exit_code)
# -- non-interactive mode
stdin = asyncio.subprocess.DEVNULL if cmd_input is None else asyncio.subprocess.PIPE
stdin = (
asyncio.subprocess.DEVNULL
if cmd_input is None else asyncio.subprocess.PIPE
)
if mod_env:
new_env = os.environ.copy()
@ -94,21 +99,21 @@ class Local(Base):
proc = await asyncio.create_subprocess_exec(
*cmd,
stdin=stdin,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
env=mod_env,
stdin = stdin,
stdout = asyncio.subprocess.PIPE,
stderr = asyncio.subprocess.PIPE,
env = mod_env,
)
stdout_parts: list[bytes] = []
stderr_parts: list[bytes] = []
# -- decoding for verbose output in pipe mode
stdout_log_enc = sys.stdout.encoding or "utf-8"
stderr_log_enc = sys.stderr.encoding or "utf-8"
stdout_log_enc = sys.stdout.encoding or 'utf-8'
stderr_log_enc = sys.stderr.encoding or 'utf-8'
async def read_stream(stream, prio, collector: list[bytes], log_enc: str):
buf = b""
buf = b''
while True:
chunk = await stream.read(4096)
if not chunk:
@ -116,12 +121,12 @@ class Local(Base):
collector.append(chunk)
if verbose:
buf += chunk
while b"\n" in buf:
line, buf = buf.split(b"\n", 1)
__log(prio, line.decode(log_enc, errors="replace"))
while b'\n' in buf:
line, buf = buf.split(b'\n', 1)
__log(prio, line.decode(log_enc, errors = 'replace'))
if verbose and buf:
# flush trailing partial line (no newline)
__log(prio, buf.decode(log_enc, errors="replace"))
__log(prio, buf.decode(log_enc, errors = 'replace'))
tasks = [
asyncio.create_task(
@ -132,7 +137,8 @@ class Local(Base):
),
]
if stdin is asyncio.subprocess.PIPE:
if (cmd_input is not None and stdin is asyncio.subprocess.PIPE
and proc.stdin is not None):
proc.stdin.write(cmd_input)
await proc.stdin.drain()
proc.stdin.close()
@ -140,8 +146,8 @@ class Local(Base):
exit_code = await proc.wait()
await asyncio.gather(*tasks)
stdout = b"".join(stdout_parts) if stdout_parts else None
stderr = b"".join(stderr_parts) if stderr_parts else None
stdout = b''.join(stdout_parts) if stdout_parts else None
stderr = b''.join(stderr_parts) if stderr_parts else None
return Result(stdout, stderr, exit_code)
@ -153,7 +159,9 @@ class Local(Base):
os.unlink(path)
async def _erase(self, path: str) -> None:
if os.isdir(path):
if os.path.isdir(path):
import shutil
shutil.rmtree(path)
return
os.unlink(path)
@ -165,12 +173,12 @@ class Local(Base):
os.mkdir(name, mode)
async def _stat(self, path: str, follow_symlinks: bool) -> StatResult:
return StatResult.from_os(os.stat(path, follow_symlinks=follow_symlinks))
return StatResult.from_os(os.stat(path, follow_symlinks = follow_symlinks))
async def _file_exists(self, path: str) -> bool:
return os.path.exists(path)
async def _chown(self, path: str, owner: str|None, group: str|None) -> None:
async def _chown(self, path: str, owner: str | None, group: str | None) -> None:
uid = pwd.getpwnam(owner).pw_uid if owner else -1
gid = grp.getgrnam(group).gr_gid if group else -1
os.chown(path, uid, gid)
@ -179,6 +187,6 @@ class Local(Base):
os.chmod(path, mode)
async def _is_dir(self, path: str, follow_symlinks: bool) -> bool:
if (not follow_symlinks) and os.islink(path):
if (not follow_symlinks) and os.path.islink(path):
return False
return os.path.isdir(path)

84
src/python/jw/pkg/lib/ec/SSHClient.py
View file

@ -1,16 +1,16 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import Any, TYPE_CHECKING
import abc
import os
import pwd
import sys
import os, abc, sys, pwd
from enum import Flag, auto
from typing import TYPE_CHECKING
from ..util import pretty_cmd
from ..log import *
from ..base import Result
from ..ExecContext import ExecContext
from ..log import DEBUG, ERR, INFO, NOTICE, WARNING, log
from ..Uri import Uri
if TYPE_CHECKING:
@ -24,48 +24,50 @@ class SSHClient(ExecContext):
ModEnv = auto()
Wd = auto()
def __init__(self, uri: Uri|str, caps: Caps=Caps(0), *args, **kwargs) -> None:
def __init__(self, uri: Uri | str, caps: Caps = Caps(0), *args, **kwargs) -> None:
uri = Uri.pimp(uri)
if uri.username is None:
uri.set_username(pwd.getpwuid(os.getuid()).pw_name)
super().__init__(uri=uri, *args, **kwargs)
super().__init__(uri = uri, *args, **kwargs)
self.__caps = caps
@abc.abstractmethod
async def _run_ssh(
self,
cmd: list[str],
wd: str|None,
wd: str | None,
verbose: bool,
cmd_input: bytes|None,
mod_env: dict[str, str]|None,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str
log_prefix: str,
) -> Result:
pass
async def _run(
self,
cmd: list[str],
wd: str|None,
wd: str | None,
verbose: bool,
cmd_input: bytes|None,
mod_env: dict[str, str]|None,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str
log_prefix: str,
) -> Result:
def __log(prio: int, *args):
log(prio, log_prefix, *args)
def __log_block(prio: int, title: str, block: str):
def __log_block(prio: int, title: str, block: bytes | str | None):
if self.__caps & self.Caps.LogOutput:
return
if not block:
return
encoding = sys.stdout.encoding or 'utf-8'
block = block.decode(encoding).strip()
if not block:
return
if isinstance(block, bytes):
encoding = sys.stdout.encoding or 'utf-8'
block = block.decode(encoding).strip()
# Needed to pacify pyright: block can't be anything else at this point
assert isinstance(block, str)
delim = f'---- {title} ----'
__log(prio, f',{delim}')
for line in block.splitlines():
@ -79,44 +81,49 @@ class SSHClient(ExecContext):
raise NotImplementedError('Interactive SSH is not yet implemented')
if mod_env is not None and not self.__caps & self.Caps.ModEnv:
raise NotImplementedError('Passing an environment to SSH commands is not yet implemented')
raise NotImplementedError(
'Passing an environment to SSH commands is not yet implemented'
)
ret = await self._run_ssh(
cmd=cmd,
wd=wd,
verbose=verbose,
cmd_input=cmd_input,
mod_env=mod_env,
interactive=interactive,
log_prefix=log_prefix
cmd = cmd,
wd = wd,
verbose = verbose,
cmd_input = cmd_input,
mod_env = mod_env,
interactive = interactive,
log_prefix = log_prefix,
)
if verbose:
__log_block(NOTICE, 'stdout', ret.stdout)
__log_block(NOTICE, 'stderr', ret.stderr)
__log_block(NOTICE, 'stdout', ret.stdout_str_or_none)
__log_block(NOTICE, 'stderr', ret.stderr_str_or_none)
if ret.status != 0:
__log(WARNING, f'Exit code {ret.status}')
return ret
@property
def hostname(self) -> str|None:
def hostname(self) -> str | None:
return self.uri.hostname
@property
def port(self) -> int|None:
def port(self) -> int | None:
return self.uri.port
@property
def username(self) -> str|None:
def username(self) -> str | None:
return self.uri.username
@property
def password(self) -> str|None:
def password(self) -> str | None:
return self.uri.password
def ssh_client(*args, type: str|list[str]|None=None, **kwargs) -> SSHClient: # export
def ssh_client(
*args, type: str | list[str] | None = None, **kwargs
) -> SSHClient: # export
from importlib import import_module
errors: list[str] = []
if type is None:
val = os.getenv('JW_DEFAULT_SSH_CLIENT')
@ -128,11 +135,12 @@ def ssh_client(*args, type: str|list[str]|None=None, **kwargs) -> SSHClient: # e
type = [type]
for name in type:
try:
ret = getattr(import_module(f'jw.pkg.lib.ec.ssh.{name}'), name)(*args, **kwargs)
ret = getattr(import_module(f'jw.pkg.lib.ec.ssh.{name}'),
name)(*args, **kwargs)
log(INFO, f'Using SSH-client "{name}"')
return ret
except Exception as e:
msg = f'Can\'t instantiate SSH client class {name} ({str(e)})'
msg = f"Can't instantiate SSH client class {name} ({str(e)})"
errors.append(msg)
log(DEBUG, f'{msg}, trying next')
msg = f'No working SSH clients for {" ".join([str(arg) for arg in args])}'

55
src/python/jw/pkg/lib/ec/ssh/AsyncSSH.py
View file

@ -1,11 +1,15 @@
# -*- coding: utf-8 -*-
import asyncio
import os
import shlex
import shutil
import signal
import sys
import os, sys, shlex, asyncio, asyncssh, shutil, signal
import asyncssh
from ...log import *
from ...base import Result
from ...log import DEBUG, ERR, NOTICE, log
from ..SSHClient import SSHClient as Base
from .util import join_cmd
_USE_DEFAULT_KNOWN_HOSTS = object()
@ -25,15 +29,18 @@ class AsyncSSH(Base):
super().__init__(
uri,
caps = self.Caps.LogOutput | self.Caps.Wd | self.Caps.Interactive | self.Caps.ModEnv,
**kwargs
caps = self.Caps.LogOutput
| self.Caps.Wd
| self.Caps.Interactive
| self.Caps.ModEnv,
**kwargs,
)
self.__client_keys = client_keys
self.__known_hosts = known_hosts
self.__term_type = term_type or os.environ.get('TERM', 'xterm')
self.__connect_timeout = connect_timeout
self.__conn: asyncssh.SSHClientConnection|None = None
self.__conn: asyncssh.SSHClientConnection | None = None
async def _open(self) -> None:
await super()._open()
@ -48,7 +55,7 @@ class AsyncSSH(Base):
log(DEBUG, f'Failed to close connection ({str(e)}, ignored)')
self.__conn = None
def _connect_kwargs(self, hide_secrets: bool=False) -> dict:
def _connect_kwargs(self, hide_secrets: bool = False) -> dict:
kwargs: dict = {
'host': self.hostname,
'port': self.port,
@ -72,7 +79,7 @@ class AsyncSSH(Base):
except Exception as e:
msg = f'-------------------- Failed to connect ({str(e)})'
log(ERR, ',', msg)
for key, val in self._connect_kwargs(hide_secrets=True).items():
for key, val in self._connect_kwargs(hide_secrets = True).items():
log(ERR, f'| {key:<20} = {val}')
log(ERR, '`', msg)
raise
@ -94,10 +101,13 @@ class AsyncSSH(Base):
@staticmethod
def _get_local_term_size() -> tuple[int, int, int, int]:
cols, rows = shutil.get_terminal_size(fallback=(80, 24))
cols, rows = shutil.get_terminal_size(fallback = (80, 24))
xpixel = ypixel = 0
try:
import fcntl, termios, struct
import fcntl
import struct
import termios
packed = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, b'\0' * 8)
rows2, cols2, xpixel, ypixel = struct.unpack('HHHH', packed)
if cols2 > 0 and rows2 > 0:
@ -126,9 +136,9 @@ class AsyncSSH(Base):
buf += chunk
while b'\n' in buf:
line, buf = buf.split(b'\n', 1)
log(prio, log_prefix, line.decode(log_enc, errors='replace'))
log(prio, log_prefix, line.decode(log_enc, errors = 'replace'))
if verbose and buf:
log(prio, log_prefix, buf.decode(log_enc, errors='replace'))
log(prio, log_prefix, buf.decode(log_enc, errors = 'replace'))
async def _run_interactive_on_conn(
self,
@ -222,7 +232,8 @@ class AsyncSSH(Base):
sys.stderr.flush()
try:
import termios, tty
import termios
import tty
old_tty_state = termios.tcgetattr(stdin_fd)
tty.setraw(stdin_fd)
@ -257,7 +268,9 @@ class AsyncSSH(Base):
exit_code = completed.exit_status
if exit_code is None:
exit_code = completed.returncode if completed.returncode is not None else -1
exit_code = (
completed.returncode if completed.returncode is not None else -1
)
stdout = b''.join(stdout_parts) if stdout_parts else None
return Result(stdout, None, exit_code)
@ -278,6 +291,7 @@ class AsyncSSH(Base):
if old_tty_state is not None:
try:
import termios
termios.tcsetattr(stdin_fd, termios.TCSADRAIN, old_tty_state)
except Exception:
pass
@ -331,7 +345,7 @@ class AsyncSSH(Base):
await proc.stdin.drain()
proc.stdin.write_eof()
completed = await proc.wait(check=False)
completed = await proc.wait(check = False)
await task
exit_code = completed.exit_status
@ -346,14 +360,13 @@ class AsyncSSH(Base):
cmd: list[str],
wd: str | None,
verbose: bool,
cmd_input: str | None,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str,
) -> Result:
try:
if interactive:
if self._has_local_tty():
return await self._run_interactive_on_conn(
@ -421,7 +434,7 @@ class AsyncSSH(Base):
await proc.stdin.drain()
proc.stdin.write_eof()
completed = await proc.wait(check=False)
completed = await proc.wait(check = False)
await asyncio.gather(*tasks)
stdout = b''.join(stdout_parts) if stdout_parts else None
@ -429,7 +442,9 @@ class AsyncSSH(Base):
exit_code = completed.exit_status
if exit_code is None:
exit_code = completed.returncode if completed.returncode is not None else -1
exit_code = (
completed.returncode if completed.returncode is not None else -1
)
return Result(stdout, stderr, exit_code)

55
src/python/jw/pkg/lib/ec/ssh/Exec.py
View file

@ -1,5 +1,7 @@
from __future__ import annotations
import os
from typing import TYPE_CHECKING
from ...base import InputMode
@ -8,18 +10,14 @@ from ..SSHClient import SSHClient as Base
from .util import join_cmd
if TYPE_CHECKING:
from ...base import Result
from ...base import Input, Result
class Exec(Base):
def __init__(self, uri, *args, **kwargs) -> None:
self.__askpass: str|None = None
self.__askpass_orig: dict[str, str|None] = dict()
super().__init__(
uri = uri,
caps = self.Caps.ModEnv,
**kwargs
)
self.__askpass: str | None = None
self.__askpass_orig: dict[str, str | None] = dict()
super().__init__(uri = uri, caps = self.Caps.ModEnv, **kwargs)
def __del__(self):
for key, val in self.__askpass_orig.items():
@ -32,36 +30,53 @@ class Exec(Base):
def __init_askpass(self):
if self.__askpass is None and self.password is not None:
import sys, tempfile
import sys
import tempfile
prefix = os.path.basename(sys.argv[0]) + '-'
f = tempfile.NamedTemporaryFile(mode='w+t', prefix=prefix, delete=False)
f = tempfile.NamedTemporaryFile(
mode = 'w+t', prefix = prefix, delete = False
)
os.chmod(f.name, 0o0700)
self.__askpass = f.name
f.write(f'#!/bin/bash\n\necho -n "{self.password}\n"')
f.close()
for key, val in {'SSH_ASKPASS': self.__askpass, 'SSH_ASKPASS_REQUIRE': 'force'}.items():
for key, val in {
'SSH_ASKPASS': self.__askpass,
'SSH_ASKPASS_REQUIRE': 'force',
}.items():
self.__askpass_orig[key] = os.getenv(key)
os.environ[key] = val
async def _run_ssh(
self,
cmd: list[str],
wd: str|None,
wd: str | None,
verbose: bool,
cmd_input: bytes|None,
mod_env: dict[str, str]|None,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str
log_prefix: str,
) -> Result:
def __pub_cmd_input(cmd_input: bytes | None) -> Input:
if cmd_input is None:
if interactive:
return InputMode.Interactive
return InputMode.NonInteractive
return cmd_input
self.__init_askpass()
if cmd_input is None:
cmd_input = InputMode.Interactive if interactive else InputMode.NonInteractive
opts: dict[str, str] = []
opts: list[str] = []
if mod_env:
for key, val in mod_env.items():
opts.extend(['-o', f'SetEnv {key}="{val}"'])
if self.username:
opts.extend(['-l', self.username])
if self.port is not None:
pots.extend(['-p', str(self.port)])
return await run_cmd(['ssh', *opts, self.hostname, join_cmd(cmd)], cmd_input=cmd_input, throw=False)
opts.extend(['-p', str(self.port)])
return await run_cmd(
['ssh', *opts, self.hostname, join_cmd(cmd)],
cmd_input = __pub_cmd_input(cmd_input),
throw = False,
)

40
src/python/jw/pkg/lib/ec/ssh/Paramiko.py
View file

@ -2,44 +2,46 @@ from __future__ import annotations
from typing import TYPE_CHECKING
import paramiko # type: ignore # error: Library stubs not installed for "paramiko"
import paramiko # type: ignore[import-untyped] # error: Library stubs not installed for "paramiko"
from ...log import *
from ...base import Result
from ...log import ERR, log
from ..SSHClient import SSHClient as Base
from .util import join_cmd
if TYPE_CHECKING:
from typing import Any
import paramiko.agent # type: ignore[import-untyped]
import paramiko.SCPClient # type: ignore[import-untyped]
class Paramiko(Base):
def __init__(self, uri, *args, **kwargs) -> None:
super().__init__(
uri,
*args,
caps = self.Caps.ModEnv,
**kwargs
)
self.__timeout: float|None = None # Untested
self.___client: Any|None = None
kwargs['caps'] = (self.Caps.ModEnv, )
super().__init__(uri, *args, **kwargs)
self.__timeout: float | None = None # Untested
self.___client: Any | None = None
@property
def __client(self) -> Any:
if self.___client is None:
ret = paramiko.SSHClient()
ret.set_missing_host_key_policy(paramiko.AutoAddPolicy())
hostname = self.hostname
if hostname is None:
raise Exception('Tried to run connect without target hostname')
try:
ret.connect(
hostname = self.hostname,
username = self.username,
allow_agent = True
hostname = hostname, username = self.username, allow_agent = True
)
except Exception as e:
log(ERR, f'Failed to connect to {self.hostname} ({str(e)})')
raise
s = ret.get_transport().open_session()
transport = ret.get_transport()
if transport is None:
raise Exception(f'Failed to get SSH transport for {hostname}')
s = transport.open_session()
# set up the agent request handler to handle agent requests from the server
paramiko.agent.AgentRequestHandler(s)
self.___client = ret
@ -47,7 +49,7 @@ class Paramiko(Base):
@property
def __scp(self) -> Any:
return SCPClient(self.__client.get_transport())
return paramiko.SCPClient(self.__client.get_transport())
async def _open(self) -> None:
await super()._open()
@ -63,13 +65,13 @@ class Paramiko(Base):
cmd: list[str],
wd: str | None,
verbose: bool,
cmd_input: str | None,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str,
) -> Result:
try:
kwargs: [str, Any] = {}
kwargs: dict[str, Any] = {}
if mod_env is not None:
kwargs['environment'] = mod_env
stdin, stdout, stderr = self.__client.exec_command(
@ -78,7 +80,7 @@ class Paramiko(Base):
**kwargs,
)
except Exception as e:
log(ERR, f'Command failed for {self.uri}: "{join_cmd(cmd)}"')
log(ERR, f'Command failed for {self.uri}: "{join_cmd(cmd)}" ({str(e)})')
raise
if cmd_input is not None:
stdin.write(cmd_input)

32
src/python/jw/pkg/lib/ec/ssh/util.py
View file

@ -1,19 +1,29 @@
# -*- coding: utf-8 -*-
import shlex
from typing import Iterable
import shlex
DEFAULT_SHELL_OPERATORS = {
# redirections
">", ">>", "<", "<<", "<<-", "<&", ">&", "<>", ">|",
"1>", "1>>", "2>", "2>>",
# pipelines / control
"|", "||", "&", "&&", ";",
# grouping
"(", ")",
'>',
'>>',
'<',
'<<',
'<<-',
'<&',
'>&',
'<>',
'>|',
'1>',
'1>>',
'2>',
'2>>', # pipelines / control
'|',
'||',
'&',
'&&',
';', # grouping
'(',
')',
}
def join_cmd(

68
src/python/jw/pkg/lib/log.py
View file

@ -1,38 +1,41 @@
# -*- coding: utf-8 -*-
import datetime
import sys
import syslog
import sys, syslog, datetime
# fmt: disable # don't conflate
EMERG = int(syslog.LOG_EMERG)
ALERT = int(syslog.LOG_ALERT)
CRIT = int(syslog.LOG_CRIT)
ERR = int(syslog.LOG_ERR)
WARNING = int(syslog.LOG_WARNING)
NOTICE = int(syslog.LOG_NOTICE)
INFO = int(syslog.LOG_INFO)
DEBUG = int(syslog.LOG_DEBUG)
DEVEL = int(syslog.LOG_DEBUG + 1)
OFF = DEVEL + 1
EMERG = int(syslog.LOG_EMERG)
ALERT = int(syslog.LOG_ALERT)
CRIT = int(syslog.LOG_CRIT)
ERR = int(syslog.LOG_ERR)
WARNING = int(syslog.LOG_WARNING)
NOTICE = int(syslog.LOG_NOTICE)
INFO = int(syslog.LOG_INFO)
DEBUG = int(syslog.LOG_DEBUG)
DEVEL = int(syslog.LOG_DEBUG + 1)
OFF = DEVEL + 1
_log_level = NOTICE
_last_tstamp = datetime.datetime.now()
_first_tstamp = _last_tstamp
_log_level = NOTICE
_last_tstamp = datetime.datetime.now()
_first_tstamp = _last_tstamp
# fmt: enable
def _log_level_name_by_value():
if _log_level_name_by_value.map is None:
_log_level_name_by_value.map = {
EMERG: "EMERG",
ALERT: "ALERT",
CRIT: "CRIT",
ERR: "ERR",
WARNING: "WARNING",
NOTICE: "NOTICE",
INFO: "INFO",
DEBUG: "DEBUG",
DEVEL: "DEVEL",
OFF: "OFF"
EMERG: 'EMERG',
ALERT: 'ALERT',
CRIT: 'CRIT',
ERR: 'ERR',
WARNING: 'WARNING',
NOTICE: 'NOTICE',
INFO: 'INFO',
DEBUG: 'DEBUG',
DEVEL: 'DEVEL',
OFF: 'OFF',
}
return _log_level_name_by_value.map
_log_level_name_by_value.map: dict[int, str]|None = None
_log_level_name_by_value.map: dict[int, str] | None = None # type: ignore
def _log_level_value_by_name():
if _log_level_value_by_name.map is None:
@ -41,21 +44,22 @@ def _log_level_value_by_name():
_log_level_value_by_name.map[name] = value
_log_level_value_by_name.map[name.lower()] = value
return _log_level_value_by_name.map
_log_level_value_by_name.map: dict[str, int]|None = None
_log_level_value_by_name.map: dict[str, int] | None = None # type: ignore
def get_log_level_name(level: int) -> str:
return _log_level_name_by_value()[level]
def parse_log_level(level: str|int) -> int:
def parse_log_level(level: str | int) -> int:
try:
ret = int(level)
if ret >= 0 and ret <= DEVEL:
return ret
except ValueError:
return _log_level_value_by_name()[level]
raise Exception("Invalid log level ", level)
return _log_level_value_by_name()[level]
raise Exception('Invalid log level ', level)
def set_log_level(level: int|None=None) -> int:
def set_log_level(level: str | int | None = None) -> int:
global _log_level
ret = _log_level
if level is not None:

66
src/python/jw/pkg/lib/pm/dpkg.py
View file

@ -1,50 +1,66 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import Iterable, TYPE_CHECKING
from typing import TYPE_CHECKING, Iterable
if TYPE_CHECKING:
from ..ExecContext import ExecContext
from ..base import InputMode
from ..Package import Package
from ..util import run_cmd, run_sudo
from ..Package import Package, meta_tags
_meta_map: dict[str, str]|None = None
_meta_map: dict[str, str] | None = None
def meta_map():
global _meta_map
if _meta_map is None:
_meta_map = Package.order_tags({
'name': 'binary:Package',
'vendor': None, # deb doesn't have vendor field
'packager': None, # -- packager --
'url': 'Homepage',
'maintainer': 'Maintainer',
})
_meta_map = Package.order_tags(
{
'name': 'binary:Package',
'vendor': None, # deb doesn't have vendor field
'packager': None, # -- packager --
'url': 'Homepage',
'maintainer': 'Maintainer',
}
)
return _meta_map
async def run_dpkg(args: list[str], sudo: bool=False, ec: ExecContext=None): # export
async def _run(
cmd: list[str], sudo: bool = False, ec: ExecContext | None = None
) -> str:
return (
await run_sudo(cmd)
if sudo else await run_cmd(cmd, ec = ec, cmd_input = InputMode.NonInteractive)
).stdout_str
async def run_dpkg(
args: list[str],
sudo: bool = False,
ec: ExecContext | None = None
) -> str: # export
cmd = ['/usr/bin/dpkg']
cmd.extend(args)
if sudo:
return await run_sudo(cmd, ec=ec)
return (await run_cmd(cmd, ec=ec)).decode()
return await _run(cmd, sudo, ec)
async def run_dpkg_query(args: list[str], sudo: bool=False, ec: ExecContext=None): # export
async def run_dpkg_query(
args: list[str],
sudo: bool = False,
ec: ExecContext | None = None
) -> str: # export
cmd = ['/usr/bin/dpkg-query']
cmd.extend(args)
if sudo:
return await run_sudo(cmd)
return (await run_cmd(cmd, ec=ec, cmd_input=InputMode.NonInteractive)).decode()
return await _run(cmd, sudo, ec)
async def query_packages(names: Iterable[str] = [], ec: ExecContext=None) -> Iterable[Package]:
fmt_str = '|'.join([(f'${{{tag}}}' if tag else '') for tag in meta_map().values()]) + r'\n'
async def query_packages(names: Iterable[str] = [],
ec: ExecContext | None = None) -> Iterable[Package]:
fmt_str = (
'|'.join([(f'${{{tag}}}' if tag else '')
for tag in meta_map().values()]) + r'\n'
)
# dpkg-query -W -f='${binary:Package}|${Maintainer}| ... \n'
specs, stderr, status = await run_dpkg_query(['-W', '-f=' + fmt_str, *names], sudo=False, ec=ec)
specs = await run_dpkg_query(['-W', '-f=' + fmt_str, *names], sudo = False, ec = ec)
return Package.parse_specs_str(specs)
async def list_files(pkg: str, ec: ExecContext=None) -> list[str]:
file_list_str, stderr, status = await run_dpkg(['-L', pkg], sudo=False, ec=ec)
async def list_files(pkg: str, ec: ExecContext | None = None) -> list[str]:
file_list_str = await run_dpkg(['-L', pkg], sudo = False, ec = ec)
return file_list_str.splitlines()

78
src/python/jw/pkg/lib/pm/rpm.py
View file

@ -1,45 +1,71 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import Iterable, TYPE_CHECKING
from typing import TYPE_CHECKING, Iterable
from ..base import InputMode
from ..Package import Package
from ..util import run_cmd, run_sudo
if TYPE_CHECKING:
from ..ExecContext import ExecContext
from ..util import run_cmd, run_sudo
from ..base import InputMode
from ..Package import Package, meta_tags
_meta_map: dict[str, str]|None = None
_meta_map: dict[str, str] | None = None
def meta_map():
global _meta_map
if _meta_map is None:
_meta_map = Package.order_tags({
'name': 'Name',
'vendor': 'Vendor',
'packager': 'Packager',
'url': 'URL',
'maintainer': None, # RPM doesn't have a maintainer field
})
_meta_map = Package.order_tags(
{
'name': 'Name',
'vendor': 'Vendor',
'packager': 'Packager',
'url': 'URL',
'maintainer': None, # RPM doesn't have a maintainer field
}
)
return _meta_map
async def run_rpm(args: list[str], sudo: bool=False, ec: ExecContext=None, mode: InputMode=InputMode.OptInteractive, **kwargs): # export
async def run_rpm(
args: list[str],
sudo: bool = False,
ec: ExecContext | None = None,
mode: InputMode = InputMode.OptInteractive,
**kwargs,
) -> str: # export
cmd = ['/usr/bin/rpm']
cmd.extend(args)
if sudo:
return await run_sudo(cmd, ec=ec, cmd_input=mode, **kwargs)
return await run_cmd(cmd, ec=ec, cmd_input=mode, **kwargs)
result = (
await run_sudo(cmd, ec = ec, cmd_input = mode, **kwargs)
if sudo else await run_cmd(cmd, ec = ec, cmd_input = mode, **kwargs)
)
return result.stdout_str
async def query_packages(names: Iterable[str] = [], ec: ExecContext=None) -> Iterable[Package]:
fmt_str = '|'.join([(f'%{{{tag}}}' if tag else '') for tag in meta_map().values()]) + r'\n'
async def query_packages(
names: Iterable[str] = [],
ec: ExecContext | None = None,
) -> Iterable[Package]: # export
fmt_str = (
'|'.join([(f'%{{{tag}}}' if tag else '')
for tag in meta_map().values()]) + r'\n'
)
opts = ['-q', '--queryformat', fmt_str]
if not names:
opts.append('-a')
specs, stderr, status = await run_rpm([*opts, *names], throw=True, sudo=False, mode=InputMode.NonInteractive, ec=ec)
return Package.parse_specs_str(specs.decode())
specs = await run_rpm(
[*opts, *names],
throw = True,
sudo = False,
mode = InputMode.NonInteractive,
ec = ec
)
return Package.parse_specs_str(specs)
async def list_files(pkg: str, ec: ExecContext=None) -> list[str]:
stdout, stderr, status = await run_rpm(['-ql', pkg], throw=True, sudo=False, mode=InputMode.NonInteractive, ec=ec)
return stdout.decode().splitlines()
async def list_files(pkg: str, ec: ExecContext | None = None) -> list[str]:
stdout = await run_rpm(
['-ql', pkg],
throw = True,
sudo = False,
mode = InputMode.NonInteractive,
ec = ec
)
return stdout.splitlines()

243
src/python/jw/pkg/lib/util.py
View file

@ -1,28 +1,31 @@
# -*- coding: utf-8 -*-
from __future__ import annotations
from typing import TYPE_CHECKING, Iterable
if TYPE_CHECKING:
from typing import Sequence
from .ExecContext import ExecContext
from .ProcFilter import ProcFilter, ProcPipeline
import os, sys, json
import json
import os
import sys
from argparse import Namespace
from enum import Enum, auto
from typing import TYPE_CHECKING, Iterable, TypeVar, cast
from .log import *
from .base import InputMode
from .base import Input, InputMode, Result
from .log import DEBUG, ERR, log
from .Uri import Uri
if TYPE_CHECKING:
from .ExecContext import ExecContext
from .FileContext import FileContext
from .ProcFilter import ProcFilter, ProcPipeline
T = TypeVar('T')
class AskpassKey(Enum):
Username = auto()
Password = auto()
def pretty_cmd(cmd: list[str], wd=None):
def pretty_cmd(cmd: list[str] | None = None, wd = None):
if cmd is None:
cmd = sys.argv
tokens = [cmd[0]]
for token in cmd[1:]:
if token.find(' ') != -1:
@ -34,44 +37,72 @@ def pretty_cmd(cmd: list[str], wd=None):
return ret
# See ExecContext.run() for what this function does
async def run_cmd(*args, ec: ExecContext|None=None, verbose: bool|None=None, cmd_input: Input=InputMode.NonInteractive, **kwargs) -> Result:
async def run_cmd(
*args,
ec: ExecContext | None = None,
verbose: bool | None = None,
cmd_input: Input = InputMode.NonInteractive,
**kwargs,
) -> Result:
if verbose is None:
verbose = False if ec is None else ec.verbose_default
if ec is None:
from .ec.Local import Local
interactive = cmd_input == InputMode.Interactive
ec = Local(verbose_default=verbose, interactive=interactive)
return await ec.run(verbose=verbose, *args, **kwargs)
async def run_curl(args: list[str], parse_json: bool=False, wd=None, throw=None, verbose=None, cmd_input=InputMode.NonInteractive, ec: ExecContext|None=None, decode=False) -> dict|str: # export
interactive = cmd_input == InputMode.Interactive
ec = Local(verbose_default = verbose, interactive = interactive)
kwargs['verbose'] = verbose
return await ec.run(*args, **kwargs)
async def run_curl(
args: list[str],
wd = None,
throw = None,
verbose = None,
cmd_input = InputMode.NonInteractive,
ec: ExecContext | None = None,
decode = False,
) -> Result:
if verbose is None:
verbose = False if ec is None else ec.verbose_default
cmd = ['curl']
if not verbose:
cmd.append('-s')
cmd.extend(args)
if parse_json:
decode = True
output = await run_cmd(cmd, wd=wd, throw=throw, verbose=verbose, cmd_input=cmd_input, ec=ec)
stdout, stderr, status = output.decode() if decode else output
if not parse_json:
ret = stdout
else:
try:
ret = json.loads(stdout)
except Exception as e:
size = 'unknown number of'
try:
size = len(stdout)
except:
pass
log(ERR, f'Failed to parse {size} bytes output of command '
+ f'>{pretty_cmd(cmd, wd)}< ({str(e)}): "{stdout}"', file=sys.stderr)
raise
return ret, stderr, status
return await run_cmd(
cmd, wd = wd, throw = throw, verbose = verbose, cmd_input = cmd_input, ec = ec
)
async def run_askpass(askpass_env: list[str], key: AskpassKey, host: str|None=None, ec: ExecContext|None=None):
if host is not None: # Currently unsupported
async def run_curl_into(
expected_type: type[T],
args: list[str],
**kwargs,
) -> T:
result = await run_curl(args, **kwargs)
stdout = result.stdout_str
try:
ret = json.loads(stdout)
except Exception as e:
log(
ERR,
f'Failed to parse {len(stdout)} bytes of Curl output ({str(e)})',
file = sys.stderr,
)
raise
if not isinstance(ret, expected_type):
raise TypeError(
f'Expected {expected_type.__name__}, got {type(ret).__name__} from Curl'
)
return cast(T, ret)
async def run_askpass(
askpass_env: list[str],
key: AskpassKey,
host: str | None = None,
ec: ExecContext | None = None,
throw: bool = False,
) -> str | None:
if host is not None: # Currently unsupported
raise NotImplementedError(f'Tried to run askpass with host "{host}"')
for var in askpass_env:
exe = os.getenv(var)
@ -88,50 +119,88 @@ async def run_askpass(askpass_env: list[str], key: AskpassKey, host: str|None=No
case 'SSH_ASKPASS':
match key:
case AskpassKey.Username:
continue # Can't get user name from SSH_ASKPASS
continue # Can't get user name from SSH_ASKPASS
case AskpassKey.Password:
exe_arg += 'Password'
ret, stderr, status = await run_cmd([exe, exe_arg], throw=False, ec=ec).decode()
if ret is not None:
return ret
result = await run_cmd([exe, exe_arg], throw = throw, ec = ec)
if result.status == 0 and result.stdout_or_none is not None:
ret = result.stdout_str_or_none
if ret:
return ret
msg = (
f"Trying to get user data from {', '.join(askpass_env)} didn't produce anything"
)
if throw:
raise Exception(msg)
log(DEBUG, msg)
return None
async def run_sudo(cmd: list[str], *args, interactive: bool=True, ec: ExecContext|None=None, **kwargs):
async def run_sudo(
cmd: list[str],
*args,
interactive: bool = True,
ec: ExecContext | None = None,
**kwargs,
):
if ec is None:
from .ec.Local import Local
ec = Local(interactive=interactive)
ec = Local(interactive = interactive)
return await ec.sudo(cmd, *args, **kwargs)
async def get(
uri: str|Uri,
*args,
ctx: FileContext|None=None,
content_filter: ProcFilter|list[ProcFilter]|ProcPipeline|None = None,
**kwargs
) -> Result:
uri: str | Uri,
*args,
ctx: FileContext | None = None,
content_filter: ProcFilter | list[ProcFilter] | ProcPipeline | None = None,
**kwargs,
) -> Result:
uri = Uri.pimp(uri)
if ctx is None or uri.id != ctx.uri.id:
from .FileContext import FileContext
ctx = FileContext.create(uri)
from .ProcFilter import run as run_pipeline
return await run_pipeline(await ctx.get(uri.path, *args, **kwargs), content_filter)
async def copy(src_uri: str|Iterable[str], dst: str|FileContext, owner: str|None=None, group: str|None=None, mode: int|None=None, throw=True) -> Exception|str|list[str]:
async def copy(
src_uri: str | Iterable[str],
dst: str | FileContext,
owner: str | None = None,
group: str | None = None,
mode: int | None = None,
throw = True,
) -> Exception | str | list[str]:
if not isinstance(src_uri, str):
ret: list[str] = []
for uri in src_uri: # TODO: Group identical netlocs into one CopyContext
rr = ret.append(await copy(uri, dst, owner, group, mode, throw))
for uri in src_uri: # TODO: Group identical netlocs into one CopyContext
rr = await copy(uri, dst, owner, group, mode, throw)
if isinstance(rr, Exception):
return rr
if isinstance(rr, list):
ret.extend(rr)
if isinstance(rr, str):
ret.append(rr)
else:
raise Exception(f'copy() returned unexpected type {type(rr)}')
return ret
from .CopyContext import CopyContext
async with CopyContext(src_uri, dst) as ctx:
try:
content = (await ctx.src.get(ctx.src.root, throw=True)).stdout
result = await ctx.src.get(ctx.src.root, throw = True)
dst_path = ctx.dst.root
if await ctx.dst.is_dir(ctx.dst.root):
dst_path += '/' + os.path.basename(src_uri)
await ctx.dst.put(path=dst_path, content=content, owner=owner, group=group, mode=mode, throw=True)
await ctx.dst.put(
path = dst_path,
content = result.stdout,
owner = owner,
group = group,
mode = mode,
throw = True,
)
return dst_path
except Exception as e:
if throw:
@ -140,21 +209,39 @@ async def copy(src_uri: str|Iterable[str], dst: str|FileContext, owner: str|None
return e
assert False, 'Unreachable code'
async def get_username(args: Namespace|None=None, url: str|None=None, askpass_env: list[str]=[], ec: ExecContext|None=None) -> str: # export
async def get_username(
args: Namespace | None = None,
url: str | None = None,
askpass_env: list[str] = [],
ec: ExecContext | None = None,
) -> str | None: # export
url_user = None if url is None else Uri(url).username
if args is not None:
if args.username is not None:
if url_user is not None and url_user != args.username:
raise Exception(f'Username mismatch: called with --username="{args.username}", URL has user name "{url_user}"')
raise Exception(
f'Username mismatch: called with --username="{args.username}", '
f'URL has user name "{url_user}"'
)
return args.username
if url_user is not None:
return url_user
return await run_askpass(askpass_env, AskpassKey.Username, ec=ec)
return await run_askpass(askpass_env, AskpassKey.Username, ec = ec)
async def get_password(args: Namespace|None=None, url: str|None=None, askpass_env: list[str]=[], ec: ExecContext|None=None) -> str: # export
async def get_password(
args: Namespace | None = None,
url: str | None = None,
askpass_env: list[str] = [],
ec: ExecContext | None = None,
) -> str | None: # export
if args is None and url is None and not askpass_env:
raise Exception(f'Neither URL nor command-line arguments nor askpass environment variable available, can\'t get password')
if args is not None and hasattr(args, 'password'): # use getattr(), because we don't necessarily want to have insecure --password among options
raise Exception(
'Neither URL nor command-line arguments nor askpass environment variable '
"available, can't get password"
)
if args is not None and hasattr(args, 'password'):
# use getattr(), because we don't necessarily want to have insecure
# --password among options
ret = getattr(args, 'password')
if ret is not None:
return ret
@ -162,9 +249,13 @@ async def get_password(args: Namespace|None=None, url: str|None=None, askpass_en
ret = Uri(url).password
if ret is not None:
return ret
return await run_askpass(askpass_env, AskpassKey.Password, ec=ec)
return await run_askpass(askpass_env, AskpassKey.Password, ec = ec)
async def get_profile_env(throw: bool=True, keep: Iterable[str]|bool=False, ec: ExecContext|None=None) -> dict[str, str]: # export
async def get_profile_env(
throw: bool = True,
keep: Iterable[str] | bool = False,
ec: ExecContext | None = None,
) -> dict[str, str]: # export
"""
Get a fresh environment from /etc/profile
@ -177,22 +268,28 @@ async def get_profile_env(throw: bool=True, keep: Iterable[str]|bool=False, ec:
Returns:
Dictionary with fresh environment
"""
mod_env: dict[str,str]|None = None
if keep == False or isinstance(keep, Iterable):
mod_env: dict[str, str] | None = None
if (not keep) or isinstance(keep, Iterable):
mod_env = {
'HOME': os.environ.get('HOME', '/'),
'USER': os.environ.get('USER', ''),
'PATH': '/usr/bin:/bin',
}
# Run bash as a login shell, which sources /etc/profile, then print environment as NUL-separated key=value pairs
# Run bash as a login shell, which sources /etc/profile, then print
# environment as NUL-separated key=value pairs
cmd = ['/usr/bin/env', '-i', '/bin/bash', '-lc', 'env -0']
result = await run_cmd(cmd, throw=throw, verbose=True, mod_env=mod_env, ec=ec)
result = await run_cmd(
cmd, throw = throw, verbose = True, mod_env = mod_env, ec = ec
)
ret: dict[str, str] = {}
for entry in result.stdout.rstrip(b"\0").split(b"\0"):
if not entry:
continue
key, val = entry.split(b"=", 1)
ret[key.decode()] = val.decode()
stdout = result.stdout_or_none
if stdout is not None:
for entry in stdout.rstrip(b'\0').split(b'\0'):
if not entry:
continue
bkey, bval = entry.split(b'=', 1)
ret[bkey.decode()] = bval.decode()
if isinstance(keep, Iterable):
for key in keep:
val = os.getenv(key)