janware/jw-pkg
1
Code Pull requests Activity Actions

jw.pkg: Fix "make check" static code check fallout

Browse source 
The previous commits have put rules for linting and formatting via ruff, yapf, mypy and pyright into place. They are checked with the make check target, and this commit adds the fixes for the target to succeed.
It does some refactoring where type checking dug up dirty bits, and also adds lots of churn in the Python code. To a good deal, that's owed to mere formatting changes. It would have been better to seperate those from syntax and refactoring fixes into multiple commits, so that the interesting changes don't drown in the formatting nose. However, that would have been a lot of additional work only to be thrown away by later commits, hence this commit has a big diff in one piece. The size of the diff is regrettable but hopefully a one-off: What it buys is automatic format checking for CI and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
      - ruff check .
      - mypy .
      - pyright
- Format check:
     - yapf --diff --recursive .
The refactoring includes:
  - Turn the Result class into a more elaborate object, capable of
    doing more heavy lifting around stderr and stdout decoding,
    summarizing outcome, and matching error strings.
    Aside from fixing broken type checks, this also removes lots of
    boilerplate calling code which is currently used for handling
    possible call outcome scenarios. Trying to access an inexistent,
    decoded string should raise a meaningful exception by itself now,
    which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
    - Add the AbstractCmd class above Cmd. This is necessary because
      the checker rightfully complains it can't instantiate a Cmd
      instance where constructor arguments were needed. They never
      were, but the type used at the instantiating code's location in
      jw.pkg.App so claims.
    - Lots of sub- and sub-subcommands are derived from the base
      class of the invoking command. That provides some properties
      shared across the ancestor hierarchy of a command, but is
      semantically unsound. Fix that by introducing jw.pkg.BaseCmd
      class as a place to provide basic helpers shared across all
      commands used in a jw.pkg.App's context, and derive all command
      classes from that afresh. The parent command is still reachable
      via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
    The line was needed by Python 2 which is not supported anylonger.
    For Python 3, the default encoding is UTF-8, anyway.
  - Allow to run "make py-format" without having it produce any
    changes. It's basically "yapf --in-place --recursive ." with some
    code style settings, see conf/topdir/pyproject.toml. The settings
    may be debatable. I've had custom tweaks in place on that target,
    too, but then again, IDEs would have more hassle to integrate
    that.
- Introduce a 88 character line length limit
  - One import per line, reshuffle them semantically, see
    [tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
      if TYPE_CHECKING
  - Spaces around assignments accounts for much churn. Having having
    no spaces in inline parameter list assignments and default
    parameter values would arguably be more compact where it's
    useful. On the other hand, I have not found a code formatter
    which allows spaces around assignments in parameter lists broken
    into one per line and that's often better than a wall of text.
  - Add two spaces before # export, as this seems to be mandated by
    PEP-8
- Use single quotes by default
Signed-off-by: Jan Lindemann <jan@janware.com>
This commit is contained in:
Jan Lindemann 2026-05-27 07:16:05 +02:00
commit 6db73873e7
Signed by: Jan Lindemann
GPG key ID: 3750640C9E25DD61
97 changed files with 3229 additions and 1893 deletions
Download patch file Download diff file Expand all files Collapse all files

1
scripts/jw-pkg.py
View file

@ -1,5 +1,4 @@
#!/usr/bin/python3 #!/usr/bin/python3
# -*- coding: utf-8 -*-
# PYTHON_ARGCOMPLETE_OK # PYTHON_ARGCOMPLETE_OK
from jw.pkg.App import App from jw.pkg.App import App

1
src/python/jw/__init__.py
View file

@ -1,2 +1,3 @@
from pkgutil import extend_path from pkgutil import extend_path
__path__ = extend_path(__path__, __name__) __path__ = extend_path(__path__, __name__)

451
src/python/jw/pkg/App.py
View file

@ -1,29 +1,40 @@
# -*- coding: utf-8 -*-
# #
# This source code file is a merge of various build tools and a horrible mess. # This source code file is a merge of various build tools and a horrible mess.
# #
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING, Iterable import argparse
import os
import pwd
import re
import sys
if TYPE_CHECKING:
from typing import TypeAlias
import os, sys, pwd, re
import os, sys, argparse, pwd, re
from functools import lru_cache, cached_property
from enum import Enum, auto from enum import Enum, auto
from functools import lru_cache
from typing import TYPE_CHECKING
from .lib.App import App as Base from .lib.App import App as Base
from .lib.log import *
from .lib.Distro import Distro from .lib.Distro import Distro
from .lib.base import InputMode from .lib.log import DEBUG, ERR, log
if TYPE_CHECKING:
import os
import pwd
import re
import sys
from typing import TypeAlias
from .lib.ExecContext import ExecContext
from .lib.PackageFilter import PackageFilter
# Meaning of pkg.requires.xxx variables # Meaning of pkg.requires.xxx variables
# build: needs to be built and installed before this can be built # build: needs to be built and installed before this can be built
# devel: needs to be installed before this-devel can be installed, i.e. before _other_ packages can be built against this # devel: needs to be installed before this-devel can be installed,
# run: needs to be installed before this-run can be installed, i.e. before this and other packages can run with this # i.e. before _other_ packages can be built against this
# run: needs to be installed before this-run can be installed,
# i.e. before this and other packages can run with this
# --------------------------------------------------------------------- Helpers # --------------------------------------------------------------------- Helpers
@ -35,32 +46,34 @@ class ResultCache(object):
def run(self, func, args): def run(self, func, args):
d = self.__cache d = self.__cache
depth = 0 depth = 0
keys = [ func.__name__ ] + args keys = [func.__name__] + args
l = len(keys) sz = len(keys)
for k in keys: for k in keys:
if k is None: if k is None:
k = 'None' k = 'None'
else: else:
k = str(k) k = str(k)
depth += 1 depth += 1
#log(DEBUG, 'depth = ', depth, 'key = ', k, 'd = ', str(d)) # log(DEBUG, 'depth = ', depth, 'key = ', k, 'd = ', str(d))
if k in d: if k in d:
if l == depth: if sz == depth:
return d[k] return d[k]
d = d[k] d = d[k]
continue continue
if l == depth: if sz == depth:
r = func(*args) r = func(*args)
d[k] = r d[k] = r
return r return r
d = d[k] = {} d = d[k] = {}
#d = d[k] # d = d[k]
raise Exception('cache algorithm failed for function', func.__name__, 'in depth', depth) raise Exception(
'cache algorithm failed for function', func.__name__, 'in depth', depth
)
class Scope(Enum): class Scope(Enum):
Self = auto() Self = auto()
One = auto() One = auto()
Subtree = auto() Subtree = auto()
Graph: TypeAlias = dict[str, set[str]] Graph: TypeAlias = dict[str, set[str]]
@ -68,27 +81,41 @@ Graph: TypeAlias = dict[str, set[str]]
class App(Base): class App(Base):
def __format_topdir(self, topdir: None|str, fmt: str) -> str: def __format_topdir(self, path: None | str, fmt: str) -> str | None:
if topdir is None: if path is None:
return None return None
match fmt: match fmt:
case 'unaltered': case 'unaltered':
return topdir return path
case None | 'absolute': case None | 'absolute':
return os.path.abspath(self.__topdir) return os.path.abspath(path)
case _: case _:
m = re.search(r'^make:(\S+)$', fmt) m = re.search(r'^make:(\S+)$', fmt)
if m is None: if m is None:
raise Exception(f'Can\'t interpret "{fmt}" as valid topdir ' + raise Exception(
'reference, expecting "unaltered", "absolute", or "make:<variable-name>"') f'Can\'t interpret "{fmt}" as valid topdir reference, '
'expecting "unaltered", "absolute", or "make:<variable-name>"'
)
return '$(' + m.group(1) + ')' return '$(' + m.group(1) + ')'
def __proj_dir(self, name: str, pretty) -> str: @property
def __topdir(self) -> str:
if self.___topdir is None:
raise Exception('Tried to access undefined top directory')
return self.___topdir
@property
def __pretty_topdir(self) -> str:
if self.___pretty_topdir is None:
raise Exception('Tried to access undefined pretty top directory')
return self.___pretty_topdir
def __proj_dir(self, name: str, pretty: bool) -> str | None:
if name == self.__top_name: if name == self.__top_name:
if pretty: if pretty:
return self.__pretty_topdir return self.__pretty_topdir
return self.__topdir return self.__topdir
for d in [ self.__projs_root, '/opt' ]: for d in [self.__projs_root, '/opt']:
ret = d + '/' + name ret = d + '/' + name
if os.path.exists(ret): if os.path.exists(ret):
return ret return ret
@ -97,7 +124,14 @@ class App(Base):
return None return None
raise Exception('No project path found for module "{}"'.format(name)) raise Exception('No project path found for module "{}"'.format(name))
def __find_dir(self, name: str, search_subdirs: list[str]=[], search_absdirs: list[str]=[], pretty: bool=True) -> str|None: def __find_dir(
self,
name: str,
search_subdirs: list[str] = [],
search_absdirs: list[str] = [],
pretty: bool = True,
) -> str | None:
def format_pd(name: str, pd: str, pretty: bool): def format_pd(name: str, pd: str, pretty: bool):
if not pretty: if not pretty:
return pd return pd
@ -107,7 +141,10 @@ class App(Base):
return pd return pd
if name == self.__top_name: if name == self.__top_name:
return self.__pretty_topdir return self.__pretty_topdir
raise NotImplementedError(f'Tried to pretty-format directory {pd}, not implemented') raise NotImplementedError(
f'Tried to pretty-format directory {pd}, not implemented'
)
pd = self.__proj_dir(name, False) pd = self.__proj_dir(name, False)
if pd is None: if pd is None:
return None return None
@ -126,11 +163,25 @@ class App(Base):
return ret return ret
return None return None
def __get_project_refs_cached(self, buf, visited, spec, section, key, add_self, scope, names_only): def __get_project_refs_cached(
return self.__res_cache.run(self.__get_project_refs, [buf, visited, spec, section, key, add_self, scope, names_only]) self, buf, visited, spec, section, key, add_self, scope, names_only
):
return self.__res_cache.run(
self.__get_project_refs,
[buf, visited, spec, section, key, add_self, scope, names_only],
)
def __get_project_refs(self, buf: list[str], visited: set[str], spec: str, def __get_project_refs(
section: str, key: str, add_self: bool, scope: Scope, names_only: bool) -> None: self,
buf: list[str],
visited: set[str],
spec: str,
section: str,
key: str,
add_self: bool,
scope: Scope,
names_only: bool,
) -> None:
name = self.strip_module_from_spec(spec) name = self.strip_module_from_spec(spec)
if names_only: if names_only:
spec = name spec = name
@ -142,59 +193,95 @@ class App(Base):
return return
visited.add(spec) visited.add(spec)
deps = self.get_value(name, section, key) deps = self.get_value(name, section, key)
log(DEBUG, 'name = ', name, 'section = ', section, 'key = ', key, 'deps = ', deps, 'scope = ', scope.name, 'visited = ', visited) log(
DEBUG,
(
f'name={name}, section={section}, key={key}, deps={deps}, '
f'scope={scope.name}, visited={visited}'
),
)
if deps and scope != Scope.Self: if deps and scope != Scope.Self:
if scope == Scope.One: if scope == Scope.One:
subscope = Scope.Self subscope = Scope.Self
else: else:
subscope = Scope.Subtree subscope = Scope.Subtree
deps = deps.split(',') for dep in deps.split(','):
for dep in deps:
dep = dep.strip() dep = dep.strip()
if not(len(dep)): if not (len(dep)):
continue continue
self.__get_project_refs_cached(buf, visited, dep, self.__get_project_refs_cached(
section, key, add_self=True, scope=subscope, buf,
names_only=names_only) visited,
dep,
section,
key,
add_self = True,
scope = subscope,
names_only = names_only,
)
if add_self: if add_self:
buf.append(spec) buf.append(spec)
def __read_dep_graph(self, projects: list[str], section: str, graph: Graph) -> None: def __read_dep_graph(
self,
projects: list[str],
sections: str | list[str],
graph: Graph,
) -> None:
if isinstance(sections, str):
sections = [sections]
for project in projects: for project in projects:
if project in graph: if project in graph:
continue continue
deps = self.get_project_refs([ project ], ['pkg.requires.jw'], section, for section in sections:
scope = Scope.One, add_self=False, names_only=True) deps = self.get_project_refs(
if not deps is None: [project],
['pkg.requires.jw'],
sections,
scope = Scope.One,
add_self = False,
names_only = True,
)
if deps is None:
continue
graph[project] = set(deps) graph[project] = set(deps)
for dep in deps: for dep in deps:
self.__read_dep_graph([ dep ], section, graph) self.__read_dep_graph([dep], sections, graph)
def __flip_dep_graph(self, graph: Graph): def __flip_dep_graph(self, graph: Graph):
ret: Graph = {} ret: Graph = {}
for project, deps in graph.items(): for project, deps in graph.items():
for d in deps: for d in deps:
if not d in ret: if d not in ret:
ret[d] = set() ret[d] = set()
ret[d].add(project) ret[d].add(project)
return ret return ret
def __find_circular_deps_recursive(self, project: str, graph: Graph, unvisited: list[str], def __find_circular_deps_recursive(
temp: set[str], path: str) -> str|None: self,
project: str,
graph: Graph,
unvisited: list[str],
temp: set[str],
path: list[str],
) -> str | None:
if project in temp: if project in temp:
log(DEBUG, 'found circular dependency at project', project) log(DEBUG, 'found circular dependency at project', project)
return project return project
if not project in unvisited: if project not in unvisited:
return None return None
temp.add(project) temp.add(project)
if project in graph: if project in graph:
for dep in graph[project]: for dep in graph[project]:
last = self.__find_circular_deps_recursive(dep, graph, unvisited, temp, path) last = self.__find_circular_deps_recursive(
dep, graph, unvisited, temp, path
)
if last is not None: if last is not None:
path.insert(0, dep) path.insert(0, dep)
return last return last
unvisited.remove(project) unvisited.remove(project)
temp.remove(project) temp.remove(project)
return None
def __find_circular_deps(self, projects: list[str], flavours: list[str]) -> bool: def __find_circular_deps(self, projects: list[str], flavours: list[str]) -> bool:
graph: Graph = {} graph: Graph = {}
@ -205,25 +292,27 @@ class App(Base):
while unvisited: while unvisited:
project = unvisited[0] project = unvisited[0]
log(DEBUG, 'Checking circular dependency of', project) log(DEBUG, 'Checking circular dependency of', project)
last = self.__find_circular_deps_recursive(project, self.__flip_dep_graph(graph), unvisited, temp, ret) last = self.__find_circular_deps_recursive(
project, self.__flip_dep_graph(graph), unvisited, temp, ret
)
if last is not None: if last is not None:
log(DEBUG, f'Found circular dependency below {project}, last is {last}') log(DEBUG, f'Found circular dependency below {project}, last is {last}')
return True return True
return False return False
def __init__(self, distro: Distro|None=None) -> None: def __init__(self, distro: Distro | None = None) -> None:
super().__init__('jw-pkg swiss army knife', modules=['jw.pkg.cmds']) super().__init__('jw-pkg swiss army knife', modules = ['jw.pkg.cmds'])
# -- Members without default values # -- Members without default values
self.__opt_interactive: bool|None = None self.__opt_interactive: bool | None = None
self.__opt_verbose: bool|None = None self.__opt_verbose: bool | None = None
self.__top_name: str|None = None self.__top_name: str | None = None
self.__distro = distro self.__distro = distro
self.__res_cache = ResultCache() self.__res_cache = ResultCache()
self.__topdir: str|None = None self.___topdir: str | None = None
self.__pretty_topdir: str|None = None self.___pretty_topdir: str | None = None
self.__exec_context: ExecContext|None = None self.__exec_context: ExecContext | None = None
# -- Members with default values # -- Members with default values
self.__topdir_fmt = 'absolute' self.__topdir_fmt = 'absolute'
@ -235,48 +324,83 @@ class App(Base):
pkg_filter_str = self.args.pkg_filter pkg_filter_str = self.args.pkg_filter
if pkg_filter_str is None: if pkg_filter_str is None:
pkg_filter_str = os.getenv('JW_DEFAULT_PKG_FILTER') pkg_filter_str = os.getenv('JW_DEFAULT_PKG_FILTER')
pkg_filter: PackageFilter|None = None pkg_filter: PackageFilter | None = None
if pkg_filter_str is not None: if pkg_filter_str is not None:
from .lib.PackageFilter import PackageFilterString from .lib.PackageFilter import PackageFilterString
pkg_filter = PackageFilterString(pkg_filter_str) pkg_filter = PackageFilterString(pkg_filter_str)
self.__distro = await Distro.instantiate( self.__distro = await Distro.instantiate(
ec = self.exec_context, ec = self.exec_context,
id = self.args.distro_id, id = self.args.distro_id,
default_pkg_filter = pkg_filter, default_pkg_filter = pkg_filter,
) )
async def __aexit__(self, exc_type, exc, tb) -> None: async def __aexit__(self, exc_type, exc, tb) -> None:
if self.__exec_context is not None: if self.__exec_context is not None:
await self.__exec_context.close() await self.__exec_context.close()
self.__exec_context = None self.__exec_context = None
return super().__aexit__(exc_type, exc, tb)
def _add_arguments(self, parser) -> None: def _add_arguments(self, parser) -> None:
super()._add_arguments(parser) super()._add_arguments(parser)
parser.add_argument('-t', '--topdir', default = None, help='Project Path') parser.add_argument('-t', '--topdir', default = None, help = 'Project Path')
parser.add_argument('--topdir-format', default = 'absolute', help='Output references to topdir as ' parser.add_argument(
+ 'one of "make:<var-name>", "unaltered", "absolute". Absolute topdir by default') '--topdir-format',
parser.add_argument('-p', '--prefix', default = None, default = 'absolute',
help='Parent directory of project source directories') help = (
parser.add_argument('--distro-id', default=None, help='Distribution ID (default is taken from /etc/os-release)') 'Output references to topdir as one of "make:<var-name>", '
parser.add_argument('--interactive', choices=['true', 'false', 'auto'], default='true', help='Wait for user input or try to proceed unattended') '"unaltered", "absolute". Absolute topdir by default'
parser.add_argument('--verbose', action='store_true', default=False, help='Be verbose on stderr about what\'s being done on the distro level') ),
parser.add_argument('--target', default='local', help='Run commands on this host') )
parser.add_argument('--pkg-filter', help='Default filter for all distribution package-related operations') parser.add_argument(
'-p',
'--prefix',
default = None,
help = 'Parent directory of project source directories',
)
parser.add_argument(
'--distro-id',
default = None,
help = 'Distribution ID (default is taken from /etc/os-release)',
)
parser.add_argument(
'--interactive',
choices = ['true', 'false', 'auto'],
default = 'true',
help = 'Wait for user input or try to proceed unattended',
)
parser.add_argument(
'--verbose',
action = 'store_true',
default = False,
help = "Be verbose on stderr about what's being done on the distro level",
)
parser.add_argument(
'--target', default = 'local', help = 'Run commands on this host'
)
parser.add_argument(
'--pkg-filter',
help = 'Default filter for all distribution package-related operations',
)
async def _run(self, args: argparse.Namespace) -> None: async def _run(self, args: argparse.Namespace) -> None:
self.__topdir = args.topdir self.___topdir = args.topdir
self.__pretty_topdir = self.__format_topdir(self.__topdir, args.topdir_format) self.___pretty_topdir = self.__format_topdir(self.___topdir, args.topdir_format)
self.__topdir_fmt = args.topdir_format self.__topdir_fmt = args.topdir_format
if self.__topdir is not None: if self.___topdir is not None:
self.__top_name = self.read_value(self.__topdir + '/make/project.conf', 'build', 'name') self.__top_name = self.read_value(
self.___topdir + '/make/project.conf', 'build', 'name'
)
if not self.__top_name: if not self.__top_name:
self.__top_name = re.sub('-[0-9.-]*$', '', os.path.basename(os.path.realpath(self.__topdir))) self.__top_name = re.sub(
'-[0-9.-]*$',
'',
os.path.basename(os.path.realpath(self.___topdir))
)
if args.prefix is not None: if args.prefix is not None:
self.__projs_root = args.prefix self.__projs_root = args.prefix
self.__pretty_projs_root = args.prefix self.__pretty_projs_root = args.prefix
await self.__init_async() await self.__init_async()
return await super()._run(args) await super()._run(args)
@property @property
def interactive(self) -> bool: def interactive(self) -> bool:
@ -288,20 +412,32 @@ class App(Base):
self.__opt_interactive = False self.__opt_interactive = False
case 'auto': case 'auto':
self.__opt_interactive = sys.stdin.isatty() self.__opt_interactive = sys.stdin.isatty()
case _:
raise ValueError(
f'Unknown --interactive value: {self.args.interactive}'
)
# Not logically possible to fail, but this keeps pyright happy
assert self.__opt_interactive is not None
return self.__opt_interactive return self.__opt_interactive
@property @property
def verbose(self) -> bool: def verbose(self) -> bool:
if self.__opt_verbose is None: if self.__opt_verbose is None:
self.__opt_verbose = self.args.verbose self.__opt_verbose = self.args.verbose
# Not logically possible to fail, but this keeps pyright happy
assert self.__opt_verbose is not None
return self.__opt_verbose return self.__opt_verbose
@property @property
def exec_context(self) -> str: def exec_context(self) -> ExecContext:
if self.__exec_context is None: if self.__exec_context is None:
from .lib.ExecContext import ExecContext from .lib.ExecContext import ExecContext
self.__exec_context = ExecContext.create(self.args.target, interactive=self.interactive,
verbose_default=self.verbose) self.__exec_context = ExecContext.create(
self.args.target,
interactive = self.interactive,
verbose_default = self.verbose,
)
return self.__exec_context return self.__exec_context
@property @property
@ -318,29 +454,51 @@ class App(Base):
raise Exception('No distro object') raise Exception('No distro object')
return self.__distro return self.__distro
def find_dir(self, name: str, search_subdirs: list[str]=[], search_absdirs: list[str]=[], pretty: bool=True): def find_dir(
return self.__find_dir(name, search_subdirs, search_absdirs, pretty) self,
name: str,
search_subdirs: list[str] = [],
search_absdirs: list[str] = [],
pretty: bool = True,
) -> str:
ret = self.__find_dir(name, search_subdirs, search_absdirs, pretty)
if ret is None:
msg = f'Failed to find directory for "{name}"'
log(ERR, msg)
for search_name, search in [
('subdirs', search_subdirs),
('absdirs', search_absdirs),
]:
if search:
log(ERR, f' Searched {search_name} in:')
for d in search:
log(ERR, f' - {d}')
raise FileNotFoundError(msg)
return ret
# TODO: add support for customizing this in project.conf # TODO: add support for customizing this in project.conf
def htdocs_dir(self, project: str) -> str: def htdocs_dir(self, project: str) -> str:
return self.find_dir(project, ['/src/html/htdocs', '/tools/html/htdocs', '/htdocs'], return self.find_dir(
['/srv/www/proj/' + project]) project,
['/src/html/htdocs', '/tools/html/htdocs', '/htdocs'],
['/srv/www/proj/' + project],
)
# TODO: add support for customizing this in project.conf # TODO: add support for customizing this in project.conf
def tmpl_dir(self, name: str) -> str: def tmpl_dir(self, name: str) -> str:
return self.find_dir(name, ['/tmpl'], ['/opt/' + name + '/share/tmpl']) return self.find_dir(name, ['/tmpl'], ['/opt/' + name + '/share/tmpl'])
def strip_module_from_spec(self, mod): def strip_module_from_spec(self, mod):
return re.sub(r'-dev$|-devel$|-run$', '', re.split('([=><]+)', mod)[0].strip()) return re.sub(r'-dev$|-devel$|-run$', '', re.split('([=><]+)', mod)[0].strip())
@lru_cache(maxsize=None) @lru_cache(maxsize = None)
def get_section(self, path: str, section: str) -> str: def get_section(self, path: str, section: str) -> str:
ret = '' ret = ''
pat = '[' + section + ']' pat = '[' + section + ']'
in_section = False in_section = False
file = open(path) file = open(path)
for line in file: for line in file:
if (line.rstrip() == pat): if line.rstrip() == pat:
in_section = True in_section = True
continue continue
if in_section: if in_section:
@ -350,10 +508,10 @@ class App(Base):
file.close() file.close()
return ret.rstrip() return ret.rstrip()
@lru_cache(maxsize=None) @lru_cache(maxsize = None)
def read_value(self, path: str, section: str, key: str) -> str|None: def read_value(self, path: str, section: str, key: str) -> str | None:
def scan_section(f, key: str) -> str|None: def scan_section(f, key: str) -> str | None:
if key is None: if key is None:
ret = '' ret = ''
for line in f: for line in f:
@ -374,43 +532,44 @@ class App(Base):
cont_line = '' cont_line = ''
rx = re.compile(r'^\s*' + key + r'\s*=\s*(.*)\s*$') rx = re.compile(r'^\s*' + key + r'\s*=\s*(.*)\s*$')
for line in lines: for line in lines:
#log(DEBUG, ' looking for "%s" in line="%s"' % (key, line)) # log(DEBUG, ' looking for "%s" in line="%s"' % (key, line))
m = re.search(rx, line) m = re.search(rx, line)
if m is not None: if m is not None:
return m.group(1) return m.group(1)
return None return None
def scan_section_debug(f, key: str) -> str|None: def scan_section_debug(f, key: str) -> str | None:
ret = scan_section(f, key) ret = scan_section(f, key)
#log(DEBUG, ' returning', rr) # log(DEBUG, ' returning', rr)
return ret return ret
try: try:
#log(DEBUG, 'looking for {}::[{}].{}'.format(path, section, key)) # log(DEBUG, 'looking for {}::[{}].{}'.format(path, section, key))
# TODO: Parse
with open(path, 'r') as f: with open(path, 'r') as f:
if not len(section): if not len(section):
rr = scan_section(f, key) return scan_section(f, key)
pat = '[' + section + ']' pat = '[' + section + ']'
for line in f: for line in f:
if line.rstrip() == pat: if line.rstrip() == pat:
return scan_section(f, key) return scan_section(f, key)
return None return None
except: except Exception:
log(DEBUG, path, 'not found') log(DEBUG, f'Not found: {path}')
# TODO: handle this special case cleaner somewhere up the stack # TODO: handle this special case cleaner somewhere up the stack
if section == 'build' and key == 'libname': if section == 'build' and key == 'libname':
return 'none' return 'none'
return None return None
@lru_cache(maxsize=None) @lru_cache(maxsize = None)
def get_value(self, project: str, section: str, key: str) -> str: def get_value(self, project: str, section: str, key: str) -> str | None:
if self.__top_name and project == self.__top_name: ret: str | None
proj_root = self.__topdir proj_dir = self.__proj_dir(project, pretty = False)
else: if proj_dir is None:
proj_root = self.__projs_root + '/' + project raise Exception(f"Can't get project directory for {project}")
if section == 'version': if section == 'version':
proj_version_dirs = [ proj_root ] proj_version_dirs = [proj_dir]
if proj_root != self.__topdir: if proj_dir != self.___topdir:
proj_version_dirs.append('/usr/share/doc/packages/' + project) proj_version_dirs.append('/usr/share/doc/packages/' + project)
for d in proj_version_dirs: for d in proj_version_dirs:
version_path = d + '/VERSION' version_path = d + '/VERSION'
@ -423,13 +582,24 @@ class App(Base):
log(DEBUG, f'Ignoring unreadable file "{version_path}"') log(DEBUG, f'Ignoring unreadable file "{version_path}"')
continue continue
raise Exception(f'No version file found for project "{project}"') raise Exception(f'No version file found for project "{project}"')
path = proj_root + '/make/project.conf' path = proj_dir + '/make/project.conf'
ret = self.read_value(path, section, key) ret = self.read_value(path, section, key)
log(DEBUG, 'Lookup %s -> %s / [%s%s] -> "%s"' % log(
(self.__top_name, project, section, '.' + key if key else '', ret)) DEBUG,
'Lookup %s -> %s / [%s%s] -> "%s"' %
(self.__top_name, project, section, '.' + key if key else '', ret),
)
return ret return ret
def get_values(self, projects: list[str], sections: list[str], keys: list[str]) -> list[str]: @lru_cache(maxsize = None)
def get_version(self, project) -> str:
ret = self.get_value(project, 'version', '')
if ret is None:
raise Exception(f"Can't get version of project {project}")
return ret
def get_values(self, projects: list[str], sections: list[str],
keys: list[str]) -> list[str]:
""" """
Collect a list of values from a list of given projects, sections and Collect a list of values from a list of given projects, sections and
keys, maintaining order keys, maintaining order
@ -441,39 +611,60 @@ class App(Base):
vals = self.get_value(p, section, key) vals = self.get_value(p, section, key)
if vals: if vals:
ret += [val.strip() for val in vals.split(',')] ret += [val.strip() for val in vals.split(',')]
return list(dict.fromkeys(ret)) # Remove duplicates, keep ordering return list(dict.fromkeys(ret)) # Remove duplicates, keep ordering
def get_project_refs(self, projects: list[str], sections: list[str], def get_project_refs(
keys: str|list[str], add_self: bool, scope: Scope, names_only=True) -> list[str]: self,
projects: list[str],
sections: list[str],
keys: str | list[str],
add_self: bool,
scope: Scope,
names_only = True,
) -> list[str]:
if isinstance(keys, str): if isinstance(keys, str):
keys = [ keys ] keys = [keys]
ret: list[str] = [] ret: list[str] = []
for section in sections: for section in sections:
for key in keys: for key in keys:
visited = set() visited: set[str] = set()
for name in projects: for name in projects:
rr: list[str] = [] rr: list[str] = []
self.__get_project_refs_cached(rr, visited, name, section, key, add_self, scope, names_only) self.__get_project_refs_cached(
rr, visited, name, section, key, add_self, scope, names_only
)
# TODO: this looks like a performance hogger # TODO: this looks like a performance hogger
for m in rr: for m in rr:
if not m in ret: if m not in ret:
ret.append(m) ret.append(m)
return ret return ret
def get_libname(self, projects) -> str: def get_libname(self, projects) -> str:
vals = self.get_project_refs(projects, ['build'], 'libname', vals = self.get_project_refs(
scope = Scope.One, add_self=False, names_only=True) projects,
['build'],
'libname',
scope = Scope.One,
add_self = False,
names_only = True,
)
if not vals: if not vals:
return ' '.join(projects) return ' '.join(projects)
if 'none' in vals: if 'none' in vals:
vals.remove('none') vals.remove('none')
return ' '.join(reversed(vals)) return ' '.join(reversed(vals))
def is_excluded_from_build(self, project: str) -> str|None: def is_excluded_from_build(self, project: str) -> str | None:
log(DEBUG, 'checking if project ' + project + ' is excluded from build') log(DEBUG, 'checking if project ' + project + ' is excluded from build')
exclude = self.get_project_refs([ project ], ['build'], 'exclude', exclude = self.get_project_refs(
scope = Scope.One, add_self=False, names_only=True) [project],
cascade = self.distro.os_cascade + [ 'all' ] ['build'],
'exclude',
scope = Scope.One,
add_self = False,
names_only = True,
)
cascade = self.distro.os_cascade + ['all']
for p1 in exclude: for p1 in exclude:
for p2 in cascade: for p2 in cascade:
if p1 == p2: if p1 == p2:

15
src/python/jw/pkg/cmds/Cmd.py
View file

@ -1,15 +1,16 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from argparse import ArgumentParser from typing import TYPE_CHECKING
from ..App import App from ..CmdBase import CmdBase
from ..lib.Cmd import Cmd as Base
class Cmd(Base): # export if TYPE_CHECKING:
from ..App import App
from ..lib.Distro import Distro
def __init__(self, parent: App|Base, name: str, help: str) -> None: class Cmd(CmdBase): # export
def __init__(self, parent: App | CmdBase, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)
async def _run(self, args): async def _run(self, args):

6
src/python/jw/pkg/cmds/CmdPkg.py
View file

@ -1,14 +1,12 @@
# -*- coding: utf-8 -*-
from argparse import ArgumentParser from argparse import ArgumentParser
from ..App import App from ..App import App
from .Cmd import Cmd as CmdBase from .Cmd import Cmd as CmdBase
class CmdPkg(CmdBase): # export class CmdPkg(CmdBase): # export
def __init__(self, parent: App) -> None: def __init__(self, parent: App) -> None:
super().__init__(parent, 'pkg', help="System package manager wrapper") super().__init__(parent, 'pkg', help = 'System package manager wrapper')
self.load_subcommands() self.load_subcommands()
def add_arguments(self, p: ArgumentParser) -> None: def add_arguments(self, p: ArgumentParser) -> None:

8
src/python/jw/pkg/cmds/CmdPlatform.py
View file

@ -1,14 +1,14 @@
# -*- coding: utf-8 -*-
from argparse import ArgumentParser from argparse import ArgumentParser
from ..App import App from ..App import App
from .Cmd import Cmd as CmdBase from .Cmd import Cmd as CmdBase
class CmdPlatform(CmdBase): # export class CmdPlatform(CmdBase): # export
def __init__(self, parent: App) -> None: def __init__(self, parent: App) -> None:
super().__init__(parent, 'platform', help="Miscellaneous platform-related comamnds") super().__init__(
parent, 'platform', help = 'Miscellaneous platform-related comamnds'
)
self.load_subcommands() self.load_subcommands()
def add_arguments(self, p: ArgumentParser) -> None: def add_arguments(self, p: ArgumentParser) -> None:

13
src/python/jw/pkg/cmds/CmdPosix.py
View file

@ -1,14 +1,19 @@
# -*- coding: utf-8 -*-
from argparse import ArgumentParser from argparse import ArgumentParser
from ..App import App from ..App import App
from .Cmd import Cmd as CmdBase from .Cmd import Cmd as CmdBase
class CmdPosix(CmdBase): # export class CmdPosix(CmdBase): # export
def __init__(self, parent: App) -> None: def __init__(self, parent: App) -> None:
super().__init__(parent, 'posix', help='Perform various operations on a distro through its POSIX utility interface') super().__init__(
parent,
'posix',
help = (
'Perform various operations on a distro through its '
'POSIX utility interface'
),
)
self.load_subcommands() self.load_subcommands()
def add_arguments(self, p: ArgumentParser) -> None: def add_arguments(self, p: ArgumentParser) -> None:

11
src/python/jw/pkg/cmds/CmdProjects.py
View file

@ -1,15 +1,18 @@
# -*- coding: utf-8 -*-
import sys import sys
from argparse import ArgumentParser from argparse import ArgumentParser
from ..App import App from ..App import App
from .Cmd import Cmd as CmdBase from .Cmd import Cmd as CmdBase
class CmdProjects(CmdBase): # export class CmdProjects(CmdBase): # export
def __init__(self, parent: App) -> None: def __init__(self, parent: App) -> None:
super().__init__(parent, 'projects', help='Project metadata evaluation for building packages') super().__init__(
parent,
'projects',
help = 'Project metadata evaluation for building packages'
)
self.load_subcommands() self.load_subcommands()
def add_arguments(self, p: ArgumentParser) -> None: def add_arguments(self, p: ArgumentParser) -> None:

6
src/python/jw/pkg/cmds/CmdSecrets.py
View file

@ -1,14 +1,12 @@
# -*- coding: utf-8 -*-
from argparse import ArgumentParser from argparse import ArgumentParser
from ..App import App from ..App import App
from .Cmd import Cmd as CmdBase from .Cmd import Cmd as CmdBase
class CmdSecrets(CmdBase): # export class CmdSecrets(CmdBase): # export
def __init__(self, parent: App) -> None: def __init__(self, parent: App) -> None:
super().__init__(parent, 'secrets', help="Manage package secrets") super().__init__(parent, 'secrets', help = 'Manage package secrets')
self.load_subcommands() self.load_subcommands()
def add_arguments(self, p: ArgumentParser) -> None: def add_arguments(self, p: ArgumentParser) -> None:

6
src/python/jw/pkg/cmds/__init__.py
View file

@ -4,6 +4,6 @@ __all__ = detect_modules(
package_name = __name__, package_name = __name__,
package_path = __path__, package_path = __path__,
namespace = globals(), namespace = globals(),
prefix = "Cmd", prefix = 'Cmd',
skip = {"Cmd"}, skip = {'Cmd'},
) # pyright: ignore[reportUnsupportedDunderAll] ) # pyright: ignore[reportUnsupportedDunderAll]

5
src/python/jw/pkg/cmds/pkg/Cmd.py
View file

@ -1,16 +1,13 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
if TYPE_CHECKING: if TYPE_CHECKING:
from ...lib.Distro import Distro
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from ..Cmd import Cmd as Base from ..Cmd import Cmd as Base
class Cmd(Base): # export class Cmd(Base): # export
def __init__(self, parent: CmdPkg, name: str, help: str) -> None: def __init__(self, parent: CmdPkg, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)

14
src/python/jw/pkg/cmds/pkg/CmdDelete.py
View file

@ -1,18 +1,18 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .Cmd import Cmd
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .Cmd import Cmd
class CmdDelete(Cmd): # export class CmdDelete(Cmd): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'delete', help="Delete packages by name") super().__init__(parent, 'delete', help = 'Delete packages by name')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument("names", nargs="*", help="Names of packages to be deleted") parser.add_argument(
'names', nargs = '*', help = 'Names of packages to be deleted'
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
return await self.distro.delete(args.names) return await self.distro.delete(args.names)

21
src/python/jw/pkg/cmds/pkg/CmdDup.py
View file

@ -1,18 +1,21 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .Cmd import Cmd
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .Cmd import Cmd
class CmdDup(Cmd): # export class CmdDup(Cmd): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'dup', help="Upgrade distribution") super().__init__(parent, 'dup', help = 'Upgrade distribution')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('--download-only', default=False, action='store_true', parser.add_argument(
help='Only download packages from the repos, don\'t install them, yet') '--download-only',
default = False,
action = 'store_true',
help = "Only download packages from the repos, don't install them, yet",
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
return await self.distro.dup(download_only=args.download_only) return await self.distro.dup(download_only = args.download_only)

37
src/python/jw/pkg/cmds/pkg/CmdInstall.py
View file

@ -1,22 +1,35 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .Cmd import Cmd
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .Cmd import Cmd
class CmdInstall(Cmd): # export class CmdInstall(Cmd): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'install', help="Install the distribution's notion of available packages") super().__init__(
parent,
'install',
help = "Install the distribution's notion of available packages",
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument("names", nargs="*", help="Packages to be installed") parser.add_argument('names', nargs = '*', help = 'Packages to be installed')
parser.add_argument('--only-update', default=False, action='store_true', help='Only update the listed packages, don\'t install them') parser.add_argument(
parser.add_argument('-F', '--fixed-strings', action='store_true', '--only-update',
help='Don\'t expand platform.expand_macros macros in <names>') default = False,
action = 'store_true',
help = "Only update the listed packages, don't install them",
)
parser.add_argument(
'-F',
'--fixed-strings',
action = 'store_true',
help = "Don't expand platform.expand_macros macros in <names>",
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
names = names if args.fixed_strings else self.app.distro.expand_macros(args.names) names = (
return await self.distro.install(names, only_update=args.only_update) args.names if args.fixed_strings else self.distro.expand_macros(args.names)
)
return await self.distro.install(names, only_update = args.only_update)

12
src/python/jw/pkg/cmds/pkg/CmdLs.py
View file

@ -1,18 +1,16 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .NamedPkgsCmd import NamedPkgsCmd as Base
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .NamedPkgsCmd import NamedPkgsCmd as Base
class CmdLs(Base): # export class CmdLs(Base): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'ls', help="List package contents") super().__init__(parent, 'ls', help = 'List package contents')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
for name in args.names: for name in args.names:
print('\n'.join(await self.parent.distro.pkg_files(name))) print('\n'.join(await self.distro.pkg_files(name)))

16
src/python/jw/pkg/cmds/pkg/CmdMeta.py
View file

@ -1,21 +1,19 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .NamedPkgsCmd import NamedPkgsCmd as Base
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .NamedPkgsCmd import NamedPkgsCmd as Base
class CmdMeta(Base): # export class CmdMeta(Base): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'meta', help="List package metadata") super().__init__(parent, 'meta', help = 'List package metadata')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
packages = await self.distro.select(args.names) names = await self.distro.select(args.names)
for package in packages: for name in names:
if len(args.names) > 1: if len(args.names) > 1:
print(f'-- {name}') print(f'-- {name}')
print(package) print(name)

16
src/python/jw/pkg/cmds/pkg/CmdRebootRequired.py
View file

@ -1,17 +1,19 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .Cmd import Cmd
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .Cmd import Cmd
class CmdRebootRequired(Cmd): # export class CmdRebootRequired(Cmd): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'reboot-required', help="Check whether the machine needs rebooting") super().__init__(
parent,
'reboot-required',
help = 'Check whether the machine needs rebooting'
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
return await self.distro.reboot_required() await self.distro.reboot_required()

14
src/python/jw/pkg/cmds/pkg/CmdRefresh.py
View file

@ -1,14 +1,16 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from .Cmd import Cmd
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .Cmd import Cmd
class CmdRefresh(Cmd): # export class CmdRefresh(Cmd): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'refresh', help="Refresh the distribution's notion of available packages") super().__init__(
parent,
'refresh',
help = "Refresh the distribution's notion of available packages",
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)

14
src/python/jw/pkg/cmds/pkg/CmdSelect.py
View file

@ -1,23 +1,19 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
import re
from ...lib.Package import Package
from ...lib.PackageFilter import PackageFilterString from ...lib.PackageFilter import PackageFilterString
from ..CmdPkg import CmdPkg from ..CmdPkg import CmdPkg
from .Cmd import Cmd from .Cmd import Cmd
class CmdSelect(Cmd): # export class CmdSelect(Cmd): # export
def __init__(self, parent: CmdPkg) -> None: def __init__(self, parent: CmdPkg) -> None:
super().__init__(parent, 'select', help="Select packages by filter") super().__init__(parent, 'select', help = 'Select packages by filter')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument("filter", help="Package filter string") parser.add_argument('filter', help = 'Package filter string')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
filter = PackageFilterString(args.filter) if args.filter else None filter = PackageFilterString(args.filter) if args.filter else None
for p in await self.distro.select(filter=filter): for p in await self.distro.select(filter = filter):
print(p.name) print(p.name)

10
src/python/jw/pkg/cmds/pkg/NamedPkgsCmd.py
View file

@ -1,15 +1,13 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser
from argparse import Namespace, ArgumentParser
from .Cmd import Cmd as Base
from ..CmdPkg import CmdPkg as Parent from ..CmdPkg import CmdPkg as Parent
from .Cmd import Cmd as Base
class NamedPkgsCmd(Base): # export class NamedPkgsCmd(Base): # export
def __init__(self, parent: Parent, name: str, help: str) -> None: def __init__(self, parent: Parent, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('names', nargs='*', help='Package names') parser.add_argument('names', nargs = '*', help = 'Package names')

6
src/python/jw/pkg/cmds/pkg/__init__.py
View file

@ -4,6 +4,6 @@ __all__ = detect_modules(
package_name = __name__, package_name = __name__,
package_path = __path__, package_path = __path__,
namespace = globals(), namespace = globals(),
prefix = "Cmd", prefix = 'Cmd',
skip = {"Cmd"}, skip = {'Cmd'},
) # pyright: ignore[reportUnsupportedDunderAll] ) # pyright: ignore[reportUnsupportedDunderAll]

5
src/python/jw/pkg/cmds/platform/Cmd.py
View file

@ -1,16 +1,13 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
if TYPE_CHECKING: if TYPE_CHECKING:
from ...lib.Distro import Distro
from ..CmdPlatform import CmdPlatform from ..CmdPlatform import CmdPlatform
from ..Cmd import Cmd as Base from ..Cmd import Cmd as Base
class Cmd(Base): # export class Cmd(Base): # export
def __init__(self, parent: CmdPlatform, name: str, help: str) -> None: def __init__(self, parent: CmdPlatform, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)

18
src/python/jw/pkg/cmds/platform/CmdInfo.py
View file

@ -1,21 +1,23 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...lib.log import *
from ...lib.Distro import Distro from ...lib.Distro import Distro
from ..Cmd import Cmd from ..Cmd import Cmd
from ..CmdPlatform import CmdPlatform from ..CmdPlatform import CmdPlatform
class CmdInfo(Cmd): # export class CmdInfo(Cmd): # export
def __init__(self, parent: CmdPlatform) -> None: def __init__(self, parent: CmdPlatform) -> None:
super().__init__(parent, 'info', help='Retrieve information about target platform') super().__init__(
parent, 'info', help = 'Retrieve information about target platform'
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('--format', default='%{cascade}', parser.add_argument(
help=f'Format string, expanding macros {", ".join(Distro.macros())}') '--format',
default = '%{cascade}',
help = f'Format string, expanding macros {", ".join(Distro.macros())}',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
print(self.app.distro.expand_macros(args.format)) print(self.app.distro.expand_macros(args.format))

6
src/python/jw/pkg/cmds/platform/__init__.py
View file

@ -4,6 +4,6 @@ __all__ = detect_modules(
package_name = __name__, package_name = __name__,
package_path = __path__, package_path = __path__,
namespace = globals(), namespace = globals(),
prefix = "Cmd", prefix = 'Cmd',
skip = {"Cmd"}, skip = {'Cmd'},
) # pyright: ignore[reportUnsupportedDunderAll] ) # pyright: ignore[reportUnsupportedDunderAll]

11
src/python/jw/pkg/cmds/posix/Cmd.py
View file

@ -1,17 +1,16 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from argparse import ArgumentParser
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ..Cmd import Cmd as Base from ...CmdBase import CmdBase
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdPosix import CmdPosix from ..CmdPosix import CmdPosix as Parent
class Cmd(Base): # export class Cmd(CmdBase): # export
def __init__(self, parent: CmdPosix, name: str, help: str) -> None: def __init__(self, parent: Parent, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:

54
src/python/jw/pkg/cmds/posix/CmdCopy.py
View file

@ -1,34 +1,56 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ...lib.util import copy from ...lib.util import copy
from .Cmd import Cmd from .Cmd import Cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdPosix import CmdPosix from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
class CmdCopy(Cmd): # export from ..CmdPosix import CmdPosix
class CmdCopy(Cmd): # export
def __init__(self, parent: CmdPosix) -> None: def __init__(self, parent: CmdPosix) -> None:
super().__init__(parent, 'copy', help="Copy files") super().__init__(parent, 'copy', help = 'Copy files')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('src', help='Source file URI') parser.add_argument('src', help = 'Source file URI')
parser.add_argument('dst', help='Destination file URI') parser.add_argument('dst', help = 'Destination file URI')
parser.add_argument('-o', '--owner', default=None, help='Destination file owner') parser.add_argument(
parser.add_argument('-g', '--group', default=None, help='Destination file group') '-o', '--owner', default = None, help = 'Destination file owner'
parser.add_argument('-m', '--mode', default=None, help='Destination file mode') )
parser.add_argument('-F', '--fixed-strings', action='store_true', parser.add_argument(
help='Don\'t expand platform.expand_macros macros in <src> and <dst>') '-g', '--group', default = None, help = 'Destination file group'
)
parser.add_argument(
'-m', '--mode', default = None, help = 'Destination file mode'
)
parser.add_argument(
'-F',
'--fixed-strings',
action = 'store_true',
help = "Don't expand platform.expand_macros macros in <src> and <dst>",
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
def __expand(url: str) -> str: def __expand(url: str) -> str:
if args.fixed_strings: if args.fixed_strings:
return url return url
return self.app.distro.expand_macros(url) ret = self.app.distro.expand_macros(url)
await copy(__expand(args.src), __expand(args.dst), if not isinstance(ret, str):
owner=args.owner, group=args.group, mode=int(args.mode, 0)) raise Exception(
f'Expanding macros in "{url}" returned unexpected ret "{ret}"'
)
return ret
await copy(
__expand(args.src),
__expand(args.dst),
owner = args.owner,
group = args.group,
mode = int(args.mode, 0),
)

14
src/python/jw/pkg/cmds/posix/CmdTar.py
View file

@ -1,14 +1,12 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from ..CmdPosix import CmdPosix as Parent
from .Cmd import Cmd as Base
from .Cmd import Cmd class CmdTar(Base): # export
from ..CmdPosix import CmdPosix
class CmdTar(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'tar', help = 'Handle tar archives')
def __init__(self, parent: CmdPosix) -> None:
super().__init__(parent, 'tar', help='Handle tar archives')
self.load_subcommands() self.load_subcommands()
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:

28
src/python/jw/pkg/cmds/posix/tar/Cmd.py
View file

@ -1,27 +1,29 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser
from collections.abc import AsyncIterator
from argparse import Namespace, ArgumentParser
from contextlib import asynccontextmanager from contextlib import asynccontextmanager
from ..Cmd import Cmd as Base from ....CmdBase import CmdBase
from ....lib.FileContext import FileContext
from ....lib.ProcFilterGpg import ProcFilterGpg
from ....lib.TarIo import TarIo
from ..CmdTar import CmdTar as Parent from ..CmdTar import CmdTar as Parent
from ....lib.TarIo import TarIo class Cmd(CmdBase): # export
from ....lib.ProcFilterGpg import ProcFilterGpg
from ....lib.FileContext import FileContext
class Cmd(Base): # export
def __init__(self, parent: Parent, name: str, help: str) -> None: def __init__(self, parent: Parent, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)
self.__tar_io: None = None self.__tar_io: None = None
@asynccontextmanager @asynccontextmanager
async def ctx(self, **kwargs) -> TarIo: async def ctx(self, **kwargs) -> AsyncIterator[TarIo]:
async with TarIo.create(src=self.app.args.archive_path, **kwargs) as ret: async with TarIo.create(src = self.app.args.archive_path, **kwargs) as ret:
ret.src.add_proc_filter(FileContext.Direction.In, ProcFilterGpg(ec=self.app.exec_context)) ret.src.add_proc_filter(
FileContext.Direction.In, ProcFilterGpg(ec = self.app.exec_context)
)
yield ret yield ret
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('-f', '--archive-path', required=True, help='Archive path') parser.add_argument(
'-f', '--archive-path', required = True, help = 'Archive path'
)

23
src/python/jw/pkg/cmds/posix/tar/CmdExtract.py
View file

@ -1,27 +1,20 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from ....lib.log import DEBUG, log
from .Cmd import Cmd, Parent
from .Cmd import Cmd class CmdExtract(Cmd): # export
from ..CmdTar import CmdTar
from ....lib.FileContext import FileContext def __init__(self, parent: Parent) -> None:
from ....lib.log import * super().__init__(parent, 'x', help = 'Extract a tar archive')
class CmdExtract(Cmd): # export
def __init__(self, parent: CmdTar) -> None:
super().__init__(parent, 'x', help="Extract a tar archive")
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('dst', help='Destination root URI') parser.add_argument('dst', help = 'Destination root URI')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
async with self.ctx(dst=args.dst) as ctx: async with self.ctx(dst = args.dst) as ctx:
paths = await ctx.extract(ctx.dst.root) paths = await ctx.extract(ctx.dst.root)
log(DEBUG, f'Extracted {len(paths)} files') log(DEBUG, f'Extracted {len(paths)} files')

4
src/python/jw/pkg/cmds/projects/BaseCmdPkgRelations.py
View file

@ -106,8 +106,8 @@ class BaseCmdPkgRelations(Cmd):
default = False, default = False,
help = ( help = (
"Don't consider or output modules matching the os cascade in their " "Don't consider or output modules matching the os cascade in their "
"[build].exclude config" '[build].exclude config'
) ),
) )
parser.add_argument( parser.add_argument(
'--hide-self', '--hide-self',

208
src/python/jw/pkg/cmds/projects/CmdBuild.py
View file

@ -1,65 +1,120 @@
# -*- coding: utf-8 -*- import datetime
import os
import re
import os, re, sys, subprocess, datetime, time from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from functools import lru_cache from functools import lru_cache
from ...lib.util import get_profile_env
from ...lib.log import *
from ..Cmd import Cmd
from ..CmdProjects import CmdProjects
from ...App import Scope from ...App import Scope
from ...lib.log import DEBUG, ERR, NOTICE, log
from ...lib.util import get_profile_env, pretty_cmd
from .Cmd import Cmd, Parent
class CmdBuild(Cmd): # export class CmdBuild(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'build', help='janware software project build tool') super().__init__(parent, 'build', help = 'janware software project build tool')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('--exclude', default='', help='Space seperated ist of modules to be excluded from build') parser.add_argument(
parser.add_argument('-n', '--dry-run', action='store_true', '--exclude',
default=False, help='Don\'t build anything, just print what would be done.') default = '',
parser.add_argument('-O', '--build-order', action='store_true', help = 'Space seperated ist of modules to be excluded from build',
default=False, help='Don\'t build anything, just print the build order.') )
parser.add_argument('-I', '--ignore-deps', action='store_true', parser.add_argument(
default=False, help='Don\'t build dependencies, i.e. build only modules specified on the command line') '-n',
parser.add_argument('--env-reinit', action='store_true', '--dry-run',
default=False, help='Source /etc/profile before each build step. Discard environment unless --env-keep is specified') action = 'store_true',
parser.add_argument('--env-keep', default='none', help='Comma seperated list of environment variables to keep, ' default = False,
+ '"all" or "none", only meaningful if --env-reinit is specified') help = "Don't build anything, just print what would be done.",
parser.add_argument('target', default='all', help='Build target') )
parser.add_argument('modules', nargs='+', default='', help='Modules to be built') parser.add_argument(
'-O',
'--build-order',
action = 'store_true',
default = False,
help = "Don't build anything, just print the build order.",
)
parser.add_argument(
'-I',
'--ignore-deps',
action = 'store_true',
default = False,
help = (
"Don't build dependencies, i.e. build only modules specified "
'on the command line'
),
)
parser.add_argument(
'--env-reinit',
action = 'store_true',
default = False,
help = (
'Source /etc/profile before each build step. Discard environment '
'unless --env-keep is specified'
),
)
parser.add_argument(
'--env-keep',
default = 'none',
help = (
'Comma seperated list of environment variables to keep, '
'"all" or "none", only meaningful if --env-reinit is specified'
),
)
parser.add_argument(
'target',
default = 'all',
help = 'Build target',
)
parser.add_argument(
'modules',
nargs = '+',
default = '',
help = 'Modules to be built',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
@lru_cache(maxsize=None) @lru_cache(maxsize = None)
def read_deps(cur, prereq_type): def read_deps(cur, prereq_type: str) -> list[str]:
# dep cache doesn't make a difference at all # dep cache doesn't make a difference at all
if prereq_type in dep_cache: if prereq_type in dep_cache:
if cur in dep_cache[prereq_type]: if cur in dep_cache[prereq_type]:
return dep_cache[prereq_type][cur] return dep_cache[prereq_type][cur]
else: else:
dep_cache[prereq_type]: dict[str, str] = {} dep_cache[prereq_type] = {}
ret = self.app.get_project_refs([ cur ], ['pkg.requires.jw'], ret = self.app.get_project_refs(
prereq_type, scope = Scope.Subtree, add_self=False, names_only=True) [cur],
['pkg.requires.jw'],
prereq_type,
scope = Scope.Subtree,
add_self = False,
names_only = True,
)
log(DEBUG, 'prerequisites = ' + ' '.join(ret)) log(DEBUG, 'prerequisites = ' + ' '.join(ret))
if cur in ret: if cur in ret:
ret.remove(cur) ret.remove(cur)
log(DEBUG, 'inserting', prereq_type, "prerequisites of", cur, ":", ' '.join(ret))
log(
DEBUG,
(f'Inserting {prereq_type}, prerequisites of {cur}: {" ".join(ret)}'),
)
dep_cache[prereq_type][cur] = ret dep_cache[prereq_type][cur] = ret
return ret return ret
def add_dep_tree(cur, prereq_types, tree, all_deps): def add_dep_tree(cur, prereq_types, tree, all_deps):
log(DEBUG, "adding prerequisites " + ' '.join(prereq_types) + " of module " + cur) log(DEBUG, 'Adding deps "{" ".join(prereq_types)}" of module {cur)')
if cur in all_deps: if cur in all_deps:
log(DEBUG, 'already handled module ' + cur) log(DEBUG, 'Already handled module "{cur}"')
return 0 return 0
deps = set() deps = set()
all_deps.add(cur) all_deps.add(cur)
for t in prereq_types: for t in prereq_types:
log(DEBUG, "checking prereqisites of type " + t) log(DEBUG, 'Checking deps of type "{t}"')
deps.update(read_deps(cur, t)) deps.update(read_deps(cur, t))
for d in deps: for d in deps:
add_dep_tree(d, prereq_types, tree, all_deps) add_dep_tree(d, prereq_types, tree, all_deps)
@ -70,17 +125,20 @@ class CmdBuild(Cmd): # export
all_deps = set() all_deps = set()
dep_tree = {} dep_tree = {}
for m in modules: for m in modules:
log(DEBUG, "--- adding dependency tree of module " + m) log(DEBUG, '--- Adding dependency tree of module "{m}"')
add_dep_tree(m, prereq_types, dep_tree, all_deps) add_dep_tree(m, prereq_types, dep_tree, all_deps)
while len(all_deps): while len(all_deps):
# Find any leaf # Find any leaf
for d in all_deps: for d in all_deps:
if not len(dep_tree[d]): # Dependency d doesn't have dependencies itself # Dependency d doesn't have dependencies itself
break # found if not len(dep_tree[d]):
else: # no Leaf found break # found
else: # no Leaf found
print(all_deps) print(all_deps)
raise Exception("fatal: the dependencies between these modules are unresolvable") raise Exception(
order.append(d) # do it 'Fatal: the dependencies between these modules are unresolvable'
)
order.append(d) # do it
# bookkeep it # bookkeep it
all_deps.remove(d) all_deps.remove(d)
for k in dep_tree.keys(): for k in dep_tree.keys():
@ -88,56 +146,68 @@ class CmdBuild(Cmd): # export
dep_tree[k].remove(d) dep_tree[k].remove(d)
return 1 return 1
async def run_make(module, target, cur_project, num_projects): async def run_make(module, target, cur_project, num_projects) -> None:
patt = self.app.is_excluded_from_build(module) patt = self.app.is_excluded_from_build(module)
if patt is not None: if patt is not None:
title = f'---- {module}'
log(NOTICE, f',{title} >') log(NOTICE, f',{title} >')
log(NOTICE, f'| Configured to skip build on platform >{patt}<') log(NOTICE, f'| Configured to skip build on platform >{patt}<')
log(NOTICE, f'`{title} <') log(NOTICE, f'`{title} <')
return return
make_cmd = [ "make", target ] make_cmd = ['make', target]
wd = self.app.find_dir(module, pretty=False) wd = self.app.find_dir(module, pretty = False)
title = '---- [%d/%d]: Running "%s" in %s -' % (cur_project, num_projects, ' '.join(make_cmd), wd) title = '---- [%d/%d]: Running "%s" in %s -' % (
cur_project,
num_projects,
' '.join(make_cmd),
wd,
)
mod_env = None mod_env = None
if args.env_reinit: if args.env_reinit:
keep: bool|list[str] = False keep: bool | list[str] = False
if args.env_keep is not None: if args.env_keep is not None:
match args.env_keep: match args.env_keep:
case 'all': case 'all':
keep=True keep = True
case 'none': case 'none':
keep=False keep = False
case _: case _:
keep = args.env_keep.split(',') keep = args.env_keep.split(',')
mod_env = await get_profile_env(keep=keep) mod_env = await get_profile_env(keep = keep)
try: try:
await self.app.exec_context.run( await self.app.exec_context.run(
make_cmd, make_cmd,
wd=wd, wd = wd,
throw=True, throw = True,
verbose=True, verbose = True,
mod_env=mod_env, mod_env = mod_env,
title=title title = title,
) )
except Exception as e: except Exception as e:
log(ERR, f'Failed to make target "{target}" in module "{module}" below base {self.app.projs_root}: {str(e)}') log(
ERR,
(
f'Failed to make target "{target}" in module "{module}" '
f'below base {self.app.projs_root}: {str(e)}'
),
)
raise raise
async def run_make_on_modules(modules, order, target): async def run_make_on_modules(modules, order, target):
cur_project = 0 cur_project = 0
num_projects = len(order) num_projects = len(order)
if target in ["clean", "distclean"]: if target in ['clean', 'distclean']:
for m in reversed(order): for m in reversed(order):
cur_project += 1 cur_project += 1
await run_make(m, target, cur_project, num_projects) await run_make(m, target, cur_project, num_projects)
if m in modules: if m in modules:
modules.remove(m) modules.remove(m)
if not len(modules): if not len(modules):
log(NOTICE, "All modules cleaned") log(NOTICE, 'All modules cleaned')
return return
else: else:
for m in order: for m in order:
@ -146,7 +216,7 @@ class CmdBuild(Cmd): # export
async def run(args): async def run(args):
log(DEBUG, "----------------------------------------- running ", ' '.join(sys.argv)) log(DEBUG, f'-------------------------------------- running {pretty_cmd()}')
modules = args.modules modules = args.modules
exclude = args.exclude.split() exclude = args.exclude.split()
@ -154,19 +224,19 @@ class CmdBuild(Cmd): # export
env_exclude = os.getenv('BUILD_EXCLUDE', '') env_exclude = os.getenv('BUILD_EXCLUDE', '')
if len(env_exclude): if len(env_exclude):
log(NOTICE, "Exluding modules from environment: " + env_exclude) log(NOTICE, 'Exluding modules from environment: ' + env_exclude)
exclude += " " + env_exclude exclude += ' ' + env_exclude
# -- build # -- build
order = [] order = []
glob_prereq_types = [ "build" ] glob_prereq_types = ['build']
if re.match("pkg-.*", target) is not None: if re.match('pkg-.*', target) is not None:
glob_prereq_types = [ "build", "run", "release", "devel" ] glob_prereq_types = ['build', 'run', 'release', 'devel']
if target != 'order' and not args.build_order: if target != 'order' and not args.build_order:
log(NOTICE, "Using prerequisite types " + ' '.join(glob_prereq_types)) log(NOTICE, 'Using prerequisite types ' + ' '.join(glob_prereq_types))
log(NOTICE, "Calculating order for modules ... ") log(NOTICE, 'Calculating order for modules ... ')
calculate_order(order, modules, glob_prereq_types) calculate_order(order, modules, glob_prereq_types)
if args.ignore_deps: if args.ignore_deps:
@ -177,18 +247,24 @@ class CmdBuild(Cmd): # export
exit(0) exit(0)
cur_project = 0 cur_project = 0
log(NOTICE, "Building target %s in %d projects:" % (target, len(order))) log(NOTICE, 'Building target %s in %d projects:' % (target, len(order)))
for m in order: for m in order:
cur_project += 1 cur_project += 1
log(NOTICE, " %3d %s" % (cur_project, m)) log(NOTICE, ' %3d %s' % (cur_project, m))
if args.dry_run: if args.dry_run:
exit(0) exit(0)
await run_make_on_modules(modules, order, target) await run_make_on_modules(modules, order, target)
log(NOTICE, 'Build done at %s' % (datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))) log(
NOTICE,
(
'Build done at %s' %
(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
),
)
dep_cache: dict[dict[str, str]] = {} dep_cache: dict[str, dict[str, list[str]]] = {}
await run(args) await run(args)

62
src/python/jw/pkg/cmds/projects/CmdCanonicalizeRemotes.py
View file

@ -1,54 +1,72 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from argparse import ArgumentParser, Namespace
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from argparse import Namespace, ArgumentParser
from ...lib.log import *
from ...lib.base import InputMode from ...lib.base import InputMode
from ..Cmd import Cmd from ...lib.log import NOTICE, log
from ..CmdProjects import CmdProjects from .Cmd import Cmd, Parent
from ...App import Scope
if TYPE_CHECKING: if TYPE_CHECKING:
from ...lib.base import Result from ...lib.base import Result
class CmdCanonicalizeRemotes(Cmd): # export class CmdCanonicalizeRemotes(Cmd): # export
def __rewrite_url(self, url: str) -> str: def __rewrite_url(self, url: str) -> str:
return url.replace('/srv/git', '') return url.replace('/srv/git', '')
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'canonicalize-remotes', help='Streamline janware Git remotes') super().__init__(
parent, 'canonicalize-remotes', help = 'Streamline janware Git remotes'
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('-n', '--dry-run', default=False, action='store_true', help='Only log what would be done') parser.add_argument(
'-n',
'--dry-run',
default = False,
action = 'store_true',
help = 'Only log what would be done',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
async def git(cmd: list[str], ro=False, throw=True) -> Result: async def git(cmd: list[str], ro = False, throw = True) -> Result:
cmd = ['/usr/bin/git', *cmd] cmd = ['/usr/bin/git', *cmd]
log(NOTICE, f'-- {" ".join(cmd)}') log(NOTICE, f'-- {" ".join(cmd)}')
if ro or not args.dry_run: if ro or not args.dry_run:
return await self.app.exec_context.run(cmd, cmd_input=InputMode.NonInteractive, throw=throw) return await self.app.exec_context.run(
remotes: dict[str, dict[str, str]] = {} cmd, cmd_input = InputMode.NonInteractive, throw = throw
stdout, stderr, status = await git(['remote', '-v'], ro=True) )
for line in stdout.decode().splitlines(): return Result(b'', None, 0)
remotes: dict[str, dict[str, str | list[str]]] = {}
result = await git(['remote', '-v'], ro = True)
for line in result.stdout_str.splitlines():
name, url, fp = line.split() name, url, fp = line.split()
remote = remotes.setdefault(name, {}) remote = remotes.setdefault(name, {})
key = 'url' if fp == '(fetch)' else 'pushurl' key = 'url' if fp == '(fetch)' else 'pushurl'
fpurls = remote.setdefault(key, []) fpurls = remote.setdefault(key, [])
assert isinstance(fpurls, list)
fpurls.append(url) fpurls.append(url)
for remote, config in remotes.items(): for name, remote in remotes.items():
dirty_keys: set[str] = set() dirty_keys: set[str] = set()
for key, urls in config.items(): for key, urls in remote.items():
for url in urls: for url in urls:
if url != self.__rewrite_url(url): if url != self.__rewrite_url(url):
dirty_keys.add(key) dirty_keys.add(key)
for key in dirty_keys: for key in dirty_keys:
urls = config[key] urls = remote[key]
await git(['config', '--unset-all', f'remote.{remote}.{key}'], throw=False) await git(
['config', '--unset-all', f'remote.{name}.{key}'], throw = False
)
for url in urls: for url in urls:
await git(['config', '--add', f'remote.{remote}.{key}', self.__rewrite_url(url)]) await git(
[
'config',
'--add',
f'remote.{name}.{key}',
self.__rewrite_url(url),
]
)

25
src/python/jw/pkg/cmds/projects/CmdCflags.py
View file

@ -1,23 +1,26 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ..Cmd import Cmd
from ..CmdProjects import CmdProjects
from ...App import Scope from ...App import Scope
from .Cmd import Cmd, Parent
class CmdCflags(Cmd): # export class CmdCflags(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'cflags', help='cflags') super().__init__(parent, 'cflags', help = 'cflags')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], 'build', deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) args.module,
['pkg.requires.jw'],
'build',
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
out = [] out = []
for m in reversed(deps): for m in reversed(deps):
path = self.app.find_dir(m, ['/include']) path = self.app.find_dir(m, ['/include'])

34
src/python/jw/pkg/cmds/projects/CmdCheck.py
View file

@ -1,24 +1,28 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from ...lib.log import NOTICE, log
from .Cmd import Cmd, Parent
from ...lib.log import * class CmdCheck(Cmd): # export
from ..Cmd import Cmd
from ..CmdProjects import CmdProjects
class CmdCheck(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(
def __init__(self, parent: CmdProjects) -> None: parent,
super().__init__(parent, 'check', help='Check for circular dependencies between given modules') 'check',
help = 'Check for circular dependencies between given modules',
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
parser.add_argument('-f', '--flavour', nargs='?', default = 'build') parser.add_argument('-f', '--flavour', nargs = '?', default = 'build')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
path = self.app.find_circular_deps(args.module, args.flavour) if self.app.find_circular_deps(args.module, args.flavour):
if path: log(NOTICE, f'Found circular dependency in flavour {args.flavour}')
log(NOTICE, f'Found circular dependency in flavour {args.flavour}:', ' -> '.join(path))
exit(1) exit(1)
log(NOTICE, f'No circular dependency found for flavour {args.flavour} in modules:', ' '.join(args.module)) log(
NOTICE,
f'No circular dependency found for flavour {args.flavour} in modules:',
' '.join(args.module),
)

22
src/python/jw/pkg/cmds/projects/CmdCommands.py
View file

@ -1,23 +1,23 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdCommands(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdCommands(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'commands', help = 'List available commands')
def __init__(self, parent: CmdProjects) -> None:
super().__init__(parent, 'commands', help='List available commands')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
import sys, re, os, glob import glob
this_dir = os.path.dirname(sys.modules[__name__].__file__) import os
import re
this_dir = os.path.dirname(__file__)
ret = [] ret = []
for file_name in glob.glob('Cmd*.py', root_dir=this_dir): for file_name in glob.glob('Cmd*.py', root_dir = this_dir):
cc_name = re.sub(r'^Cmd|\.py', '', file_name) cc_name = re.sub(r'^Cmd|\.py', '', file_name)
name = re.sub(r'(?<!^)(?=[A-Z])', '-', cc_name).lower() name = re.sub(r'(?<!^)(?=[A-Z])', '-', cc_name).lower()
ret.append(name) ret.append(name)

25
src/python/jw/pkg/cmds/projects/CmdExepath.py
View file

@ -1,23 +1,26 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...App import Scope from ...App import Scope
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdExepath(Cmd): # export class CmdExepath(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'exepath', help='exepath') super().__init__(parent, 'exepath', help = 'exepath')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], [ 'run', 'build', 'devel' ], deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) args.module,
['pkg.requires.jw'],
['run', 'build', 'devel'],
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
out = [] out = []
for m in deps: for m in deps:
path = self.app.find_dir(m, ['/bin']) path = self.app.find_dir(m, ['/bin'])

76
src/python/jw/pkg/cmds/projects/CmdGetAuthInfo.py
View file

@ -1,44 +1,72 @@
# -*- coding: utf-8 -*- import os
import re
import re, os from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...lib.log import * from ...lib.log import DEBUG, log
from ...lib.Uri import Uri from ...lib.Uri import Uri
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdGetAuthInfo(Cmd): # export class CmdGetAuthInfo(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'get-auth-info', help='Try to retrieve authentication information from the source tree') super().__init__(
parent,
'get-auth-info',
help = 'Try to retrieve authentication information from the source tree',
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('--only-values', default=False, action='store_true', parser.add_argument(
help='Don\'t prefix values by "<field-name>="') '--only-values',
parser.add_argument('--username', default=False, action='store_true', default = False,
help='Show user name') action = 'store_true',
parser.add_argument('--password', default=False, action='store_true', help = 'Don\'t prefix values by "<field-name>="',
help='Show password') )
parser.add_argument('--remote-owner-base', default=False, action='store_true', parser.add_argument(
help='Show remote base URL for owner jw-pkg was cloned from') '--username',
parser.add_argument('--remote-base', default=False, action='store_true', default = False,
help='Show remote base URL') action = 'store_true',
help = 'Show user name'
)
parser.add_argument(
'--password',
default = False,
action = 'store_true',
help = 'Show password'
)
parser.add_argument(
'--remote-owner-base',
default = False,
action = 'store_true',
help = 'Show remote base URL for owner jw-pkg was cloned from',
)
parser.add_argument(
'--remote-base',
default = False,
action = 'store_true',
help = 'Show remote base URL',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
keys = ['username', 'password'] keys = ['username', 'password']
# --- Milk jw-pkg repo # --- Milk jw-pkg repo
jw_pkg_dir = self.app.find_dir('jw-pkg', pretty=False) jw_pkg_dir = self.app.find_dir('jw-pkg', pretty = False)
if not os.path.isdir(jw_pkg_dir + '/.git'): if not os.path.isdir(jw_pkg_dir + '/.git'):
log(DEBUG, f'jw-pkg directory is not a Git repo: {jw_pkg_dir}') log(DEBUG, f'jw-pkg directory is not a Git repo: {jw_pkg_dir}')
return return
remotes, stderr, status = (await self.app.exec_context.run(['git', '-C', jw_pkg_dir, 'remote', '-v'])).decode() git_result = await self.app.exec_context.run(
['git', '-C', jw_pkg_dir, 'remote', '-v']
)
result: dict[str, str] = {} result: dict[str, str] = {}
for line in remotes.splitlines(): for line in git_result.stdout_str.splitlines():
name, url, typ = re.split(r'\s+', line) name, url, typ = re.split(r'\s+', line)
if name == 'origin' and typ in ['(pull)', '(fetch)']: # TODO: Use other remotes, too? if name == 'origin' and typ in [
'(pull)',
'(fetch)',
]: # TODO: Use other remotes, too?
parsed = Uri(url) parsed = Uri(url)
for key in keys: for key in keys:
result[key] = getattr(parsed, key) result[key] = getattr(parsed, key)
@ -52,7 +80,7 @@ class CmdGetAuthInfo(Cmd): # export
# --- Print results # --- Print results
for key, val in result.items(): for key, val in result.items():
if getattr(args, key, None) != True: if not getattr(args, key, None):
continue continue
if val is None: if val is None:
continue continue

19
src/python/jw/pkg/cmds/projects/CmdGetval.py
View file

@ -1,18 +1,19 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdGetval(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdGetval(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'getval', help = 'Get value from project config')
def __init__(self, parent: CmdProjects) -> None:
super().__init__(parent, 'getval', help='Get value from project config')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('--project', default = None, help = 'Project name, default is name of project\'s topdir') parser.add_argument(
'--project',
default = None,
help = "Project name, default is name of project's topdir",
)
parser.add_argument('section', default = '', help = 'Config section') parser.add_argument('section', default = '', help = 'Config section')
parser.add_argument('key', default = '', help = 'Config key') parser.add_argument('key', default = '', help = 'Config key')

20
src/python/jw/pkg/cmds/projects/CmdHtdocsDir.py
View file

@ -1,21 +1,21 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdHtdocsDir(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdHtdocsDir(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(
def __init__(self, parent: CmdProjects) -> None: parent,
super().__init__(parent, 'htdocs-dir', help='Print source directory containing document root of a given module') 'htdocs-dir',
help = 'Print source directory containing document root of a given module',
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
r = [] r = []
for m in args.module: for m in args.module:
r.append(self.app.htdocs_dir(m)) r.append(self.app.htdocs_dir(m))
print(' '.join(r))

57
src/python/jw/pkg/cmds/projects/CmdLdflags.py
View file

@ -1,27 +1,42 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...App import Scope from ...App import Scope
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdLdflags(Cmd): # export class CmdLdflags(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'ldflags', help='ldflags') super().__init__(parent, 'ldflags', help = 'ldflags')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
parser.add_argument('--exclude', action='append', help='Exclude Modules', default=[]) parser.add_argument(
parser.add_argument('-s', '--add-self', action='store_true', '--exclude', action = 'append', help = 'Exclude Modules', default = []
default=False, help='Include libflags of specified modules, too, not only their dependencies') )
parser.add_argument(
'-s',
'--add-self',
action = 'store_true',
default = False,
help = (
'Include libflags of specified modules, too, '
'not only their dependencies'
),
)
# -L needs to contain more paths than libs linked with -l would require # -L needs to contain more paths than libs linked with -l would require
def __get_ldpathflags(self, names: list[str], exclude: list[str] = []) -> str: def __get_ldpathflags(
deps = self.app.get_project_refs(names, ['pkg.requires.jw'], 'build', self, names: list[str], exclude: list[str] = []
scope = Scope.Subtree, add_self=True, names_only=True) ) -> str | None:
deps = self.app.get_project_refs(
names,
['pkg.requires.jw'],
'build',
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
ret = [] ret = []
for m in deps: for m in deps:
if m in exclude: if m in exclude:
@ -35,11 +50,17 @@ class CmdLdflags(Cmd): # export
ret.append('-L' + path) ret.append('-L' + path)
if not ret: if not ret:
return None return None
return(' '.join(ret)) return ' '.join(ret)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], 'build', deps = self.app.get_project_refs(
scope = Scope.One, add_self=args.add_self, names_only=True) args.module,
['pkg.requires.jw'],
'build',
scope = Scope.One,
add_self = args.add_self,
names_only = True,
)
out = [] out = []
for m in reversed(deps): for m in reversed(deps):
if m in args.exclude: if m in args.exclude:

25
src/python/jw/pkg/cmds/projects/CmdLdlibpath.py
View file

@ -1,23 +1,26 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...App import Scope from ...App import Scope
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdLdlibpath(Cmd): # export class CmdLdlibpath(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'ldlibpath', help='ldlibpath') super().__init__(parent, 'ldlibpath', help = 'ldlibpath')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], [ 'run', 'build', 'devel' ], deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) args.module,
['pkg.requires.jw'],
['run', 'build', 'devel'],
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
out = [] out = []
for m in deps: for m in deps:
path = self.app.find_dir(m, ['/lib']) path = self.app.find_dir(m, ['/lib'])

15
src/python/jw/pkg/cmds/projects/CmdLibname.py
View file

@ -1,18 +1,15 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdLibname(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdLibname(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'libname', help = 'libname')
def __init__(self, parent: CmdProjects) -> None:
super().__init__(parent, 'libname', help='libname')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
print(self.app.get_libname(args.module)) print(self.app.get_libname(args.module))

126
src/python/jw/pkg/cmds/projects/CmdListRepos.py
View file

@ -1,62 +1,105 @@
# -*- coding: utf-8 -*- import os
import re
import re, os from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...lib.util import get_username, get_password, run_curl from ...lib.base import Input
from ...lib.log import * from ...lib.log import DEBUG, log
from ...lib.Uri import Uri from ...lib.Uri import Uri
from ..Cmd import Cmd from ...lib.util import get_password, get_username, run_curl_into
from ..CmdProjects import CmdProjects from .Cmd import Cmd, Parent
class CmdListRepos(Cmd): # export class CmdListRepos(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'list-repos', help='Query a remote GIT server for repositories') super().__init__(
parent, 'list-repos', help = 'Query a remote GIT server for repositories'
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('base_url', help='Base URL of all Git repositories without user part') parser.add_argument(
parser.add_argument('--username', help='Username for SSH or HTTP authentication, don\'t specify for unauthenticated', default=None) 'base_url', help = 'Base URL of all Git repositories without user part'
parser.add_argument('--askpass', help='Program to echo password for SSH or HTTP authentication, don\'t specify for unauthenticated', default=None) )
parser.add_argument('--from-owner', help='List from-owner\'s projects', default='janware') parser.add_argument(
'--username',
help = (
"Username for SSH or HTTP authentication, don't "
'specify for unauthenticated'
),
default = None,
)
parser.add_argument(
'--askpass',
help = (
'Program to echo password for SSH or HTTP authentication, '
"don't specify for unauthenticated"
),
default = None,
)
parser.add_argument(
'--from-owner', help = "List from-owner's projects", default = 'janware'
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
base_url = Uri(args.base_url) base_url = Uri(args.base_url)
askpass_env=['GIT_ASKPASS', 'SSH_ASKPASS'] askpass_env = ['GIT_ASKPASS', 'SSH_ASKPASS']
username = await get_username(args=args, url=args.base_url, askpass_env=askpass_env) username = await get_username(
args = args, url = args.base_url, askpass_env = askpass_env
)
password = None password = None
if username is not None: if username is not None:
password = await get_password(args=args, url=args.base_url, askpass_env=askpass_env) password = await get_password(
args = args, url = args.base_url, askpass_env = askpass_env
)
match base_url.scheme: match base_url.scheme:
case 'ssh': case 'ssh':
if re.match(r'ssh://.*devgit\.janware\.com/', args.base_url): if re.match(r'ssh://.*devgit\.janware\.com/', args.base_url):
from jw.pkg.lib.ec.SSHClient import SSHClient, ssh_client from jw.pkg.lib.ec.SSHClient import ssh_client
if username is not None: if username is not None:
base_url.set_username(username) base_url.set_username(username)
if password is not None: if password is not None:
base_url.set_password(password) base_url.set_password(password)
ssh = ssh_client(base_url, interactive=self.app.interactive, verbose_default=self.app.verbose) ssh = ssh_client(
cmd = ['/opt/jw-pkg/bin/git-srv-admin.sh', '-u', args.from_owner, '-j', 'list-personal-projects'] base_url,
interactive = self.app.interactive,
verbose_default = self.app.verbose,
)
cmd = [
'/opt/jw-pkg/bin/git-srv-admin.sh',
'-u',
args.from_owner,
'-j',
'list-personal-projects',
]
result = await ssh.run(cmd) result = await ssh.run(cmd)
print('\n'.join(result.stdout.decode().splitlines())) print('\n'.join(result.stdout_str.splitlines()))
return return
case 'https': case 'https':
from jw.pkg.lib.base import InputMode from jw.pkg.lib.base import InputMode
cmd_input = InputMode.NonInteractive
cmd_input: Input = InputMode.NonInteractive
if re.match(r'https://github.com', args.base_url): if re.match(r'https://github.com', args.base_url):
curl_args = [ curl_args = [
'-f', '-f',
'-H', 'Accept: application/vnd.github+json', '-H',
'-H', 'X-GitHub-Api-Version: 2022-11-28', 'Accept: application/vnd.github+json',
'-H',
'X-GitHub-Api-Version: 2022-11-28',
] ]
if password is not None: if password is not None:
assert username is not None, f'Assertion failed: username is empty but password isn\'t for "{args.base_url}"' assert username is not None, (
'Assertion failed: username is empty but password '
'isn\'t for "{args.base_url}"'
)
cmd_input = (f'-u {username}:{password}').encode('utf-8') cmd_input = (f'-u {username}:{password}').encode('utf-8')
curl_args.extend(['-K-']) curl_args.extend(['-K-'])
curl_args.append(f'https://api.github.com/users/{args.from_owner}/repos') curl_args.append(
repos, stderr, status = await run_curl(curl_args, cmd_input=cmd_input, parse_json=True) f'https://api.github.com/users/{args.from_owner}/repos'
)
repos = await run_curl_into(list, curl_args, cmd_input = cmd_input)
for repo in repos: for repo in repos:
print(repo['name']) print(repo['name'])
return return
@ -69,33 +112,44 @@ class CmdListRepos(Cmd): # export
cmd_input = (f'-u {username}:{password}').encode('utf-8') cmd_input = (f'-u {username}:{password}').encode('utf-8')
curl_args.extend(['-K-']) curl_args.extend(['-K-'])
for entities_dir in ['orgs', 'users']: for entities_dir in ['orgs', 'users']:
api_url = f'{args.base_url}/api/v1/{entities_dir}/{args.from_owner}/repos' api_url = (
f'{args.base_url}/api/v1/{entities_dir}/'
f'{args.from_owner}/repos'
)
try: try:
tried.append(api_url) tried.append(api_url)
repos, stderr, status = await run_curl(curl_args + [api_url], cmd_input=cmd_input, parse_json=True) repos = await run_curl_into(
list,
curl_args + [api_url],
cmd_input = cmd_input,
)
for repo in repos: for repo in repos:
print(repo['name']) print(repo['name'])
break break
except Exception as e: except Exception as e:
msg = 'curl {} failed ({}), trying next'.format( msg = 'curl {} failed ({}), trying next'.format(
' '.join(curl_args + [api_url]), ' '.join(curl_args + [api_url]), str(e)
str(e)
) )
log(DEBUG, msg) log(DEBUG, msg)
tried[-1] += ': ' + msg tried[-1] += ': ' + msg
raise raise
else: else:
raise RuntimeError(f'Failed to fetch repository list from assumed Forgejo instance at {args.base_url}, tried {', '.join(tried)}') raise RuntimeError(
f'Failed to fetch repository list from assumed Forgejo '
f'instance at {args.base_url}, tried {", ".join(tried)}'
)
return return
if os.path.isdir(args.base_url): if os.path.isdir(args.base_url):
for subdir in ["." , args.from_owner]: for subdir in ['.', args.from_owner]:
out = [] out = []
for entry in os.scandir(args.base_url + "/" + subdir): for entry in os.scandir(args.base_url + '/' + subdir):
path = entry.path path = entry.path
if os.path.isdir(path + "/.git") or os.path.exists(path + "/HEAD"): if os.path.isdir(path + '/.git') or os.path.exists(path + '/HEAD'):
out.append(path) out.append(path)
if out: if out:
print('\n'.join(out)) print('\n'.join(out))
break break
return return
raise Exception(f'Don\'t know how to enumerate Git repos at base url {args.base_url}') raise Exception(
f"Don't know how to enumerate Git repos at base url {args.base_url}"
)

67
src/python/jw/pkg/cmds/projects/CmdModules.py
View file

@ -1,44 +1,57 @@
# -*- coding: utf-8 -*- import re
from argparse import Namespace, ArgumentParser from argparse import ArgumentParser, Namespace
from ...lib.log import * from ...lib.log import DEBUG, log
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdModules(Cmd): # export class CmdModules(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'modules', help='Query existing janware packages') super().__init__(parent, 'modules', help = 'Query existing janware packages')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('-F', '--filter', nargs='?', default=None, help='Key-value pairs, seperated by commas, to be searched for in project.conf') parser.add_argument(
'-F',
'--filter',
nargs = '?',
default = None,
help =
'Key-value pairs, seperated by commas, to be searched for in project.conf',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
import pathlib import pathlib
proj_root = self.app.projs_root proj_root = self.app.projs_root
log(DEBUG, "proj_root = " + proj_root) log(DEBUG, 'proj_root = ' + proj_root)
path = pathlib.Path(self.app.projs_root) path = pathlib.Path(self.app.projs_root)
modules = [p.parents[1].name for p in path.glob('*/make/project.conf')] modules = [p.parents[1].name for p in path.glob('*/make/project.conf')]
log(DEBUG, "modules = ", modules) log(DEBUG, 'modules = ', modules)
out = [] out = []
filters = None if args.filter is None else [re.split("=", f) for f in re.split(",", args.filter)] filters = (
None if args.filter is None else
[re.split('=', f) for f in re.split(',', args.filter)]
)
for m in modules: for m in modules:
if filters: if not filters:
for f in filters:
path = f[0].rsplit('.')
if len(path) > 1:
sec = path[0]
key = path[1]
else:
sec = None
key = path[0]
val = self.app.get_value(m, sec, key)
log(DEBUG, 'Checking in {} if {}="{}", is "{}"'.format(m, f[0], f[1], val))
if val and val == f[1]:
out.append(m)
break
else:
out.append(m) out.append(m)
continue
for f in filters:
path_str = f[0].rsplit('.')
if len(path_str) > 1:
sec = path_str[0]
key = path_str[1]
else:
sec = None
key = path_str[0]
val = self.app.get_value(m, sec, key)
log(
DEBUG,
'Checking in {} if {}="{}", is "{}"'.format(m, f[0], f[1], val),
)
if val and val == f[1]:
out.append(m)
break
print(' '.join(out)) print(' '.join(out))

27
src/python/jw/pkg/cmds/projects/CmdPath.py
View file

@ -1,26 +1,29 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...App import Scope from ...App import Scope
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdPath(Cmd): # export class CmdPath(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'path', help='path') super().__init__(parent, 'path', help = 'path')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], 'run', deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) args.module,
['pkg.requires.jw'],
'run',
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
out = [] out = []
for m in deps: for m in deps:
path = self.app.find_dir(m, '/bin') path = self.app.find_dir(m, ['/bin'])
if path is not None: if path is not None:
out.append(path) out.append(path)
print(':'.join(out)) print(':'.join(out))

15
src/python/jw/pkg/cmds/projects/CmdPkgConflicts.py
View file

@ -1,10 +1,11 @@
# -*- coding: utf-8 -*-
from argparse import Namespace, ArgumentParser
from .BaseCmdPkgRelations import BaseCmdPkgRelations as Base from .BaseCmdPkgRelations import BaseCmdPkgRelations as Base
from .Cmd import Parent
class CmdPkgConflicts(Base): # export class CmdPkgConflicts(Base): # export
def __init__(self, parent: Base) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'conflicts', help='Print packages conflicting with a given package') super().__init__(
parent,
'conflicts',
help = 'Print packages conflicting with a given package'
)

15
src/python/jw/pkg/cmds/projects/CmdPkgProvides.py
View file

@ -1,10 +1,11 @@
# -*- coding: utf-8 -*-
from argparse import Namespace, ArgumentParser
from .BaseCmdPkgRelations import BaseCmdPkgRelations as Base from .BaseCmdPkgRelations import BaseCmdPkgRelations as Base
from .Cmd import Parent
class CmdPkgProvides(Base): # export class CmdPkgProvides(Base): # export
def __init__(self, parent: Base) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'provides', help='Print packages and capabilities provided by a given package') super().__init__(
parent,
'provides',
help = 'Print packages and capabilities provided by a given package',
)

13
src/python/jw/pkg/cmds/projects/CmdPkgRequires.py
View file

@ -1,10 +1,9 @@
# -*- coding: utf-8 -*-
from argparse import Namespace, ArgumentParser
from .BaseCmdPkgRelations import BaseCmdPkgRelations as Base from .BaseCmdPkgRelations import BaseCmdPkgRelations as Base
from .Cmd import Parent
class CmdPkgRequires(Base): # export class CmdPkgRequires(Base): # export
def __init__(self, parent: Base) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'requires', help='Print packages required for a given package') super().__init__(
parent, 'requires', help = 'Print packages required for a given package'
)

19
src/python/jw/pkg/cmds/projects/CmdProjDir.py
View file

@ -1,19 +1,18 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from ...lib.log import WARNING, log
from .Cmd import Cmd, Parent
from ...lib.log import * class CmdProjDir(Cmd): # export
from ..Cmd import Cmd
from ..CmdProjects import CmdProjects
class CmdProjDir(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(
def __init__(self, parent: CmdProjects) -> None: parent, 'proj-dir', help = 'Print directory of a given package'
super().__init__(parent, 'proj-dir', help='Print directory of a given package') )
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
out = [] out = []

27
src/python/jw/pkg/cmds/projects/CmdPythonpath.py
View file

@ -1,23 +1,28 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
from ...App import Scope from ...App import Scope
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdPythonpath(Cmd): # export class CmdPythonpath(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'pythonpath', help='Generate PYTHONPATH for given modules') super().__init__(
parent, 'pythonpath', help = 'Generate PYTHONPATH for given modules'
)
def add_arguments(self, p: ArgumentParser) -> None: def add_arguments(self, p: ArgumentParser) -> None:
super().add_arguments(p) super().add_arguments(p)
p.add_argument('module', help='Modules', nargs='*') p.add_argument('module', help = 'Modules', nargs = '*')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], [ 'run', 'build' ], deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) args.module,
['pkg.requires.jw'],
['run', 'build'],
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
out = [] out = []
for m in deps: for m in deps:
path = self.app.find_dir(m, ['src/python', 'tools/python']) path = self.app.find_dir(m, ['src/python', 'tools/python'])

33
src/python/jw/pkg/cmds/projects/CmdPythonpathOrig.py
View file

@ -1,30 +1,35 @@
# -*- coding: utf-8 -*- import os
from argparse import Namespace, ArgumentParser from argparse import ArgumentParser, Namespace
from ...App import Scope from ...App import Scope
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
class CmdPythonpathOrig(Cmd): # export class CmdPythonpathOrig(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'pythonpath_orig', help='pythonpath') super().__init__(parent, 'pythonpath_orig', help = 'pythonpath')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
deps = self.app.get_project_refs(args.module, ['pkg.requires.jw'], [ 'run', 'build' ], deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) args.module,
['pkg.requires.jw'],
['run', 'build'],
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
r = '' r = ''
for m in deps: for m in deps:
pd = self.app.find_dir(m, pretty=False) pd = self.app.find_dir(m, pretty = False)
if pd is None: if pd is None:
continue continue
for subdir in [ 'src/python', 'tools/python' ]: for subdir in ['src/python', 'tools/python']:
cand = pd + "/" + subdir cand = pd + '/' + subdir
if isdir(cand): if os.path.isdir(cand):
r = r + ':' + cand r = r + ':' + cand
print(r[1:]) print(r[1:])

61
src/python/jw/pkg/cmds/projects/CmdRequiredOsPkg.py
View file

@ -1,45 +1,60 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from typing import Iterable
from argparse import Namespace, ArgumentParser
from ...App import Scope from ...App import Scope
from ...lib.log import * from ...lib.log import DEBUG, log
from ..Cmd import Cmd from .Cmd import Cmd, Parent
from ..CmdProjects import CmdProjects
# TODO: seems at least partly redundant to CmdPkgRequires / print_pkg_relations # TODO: seems at least partly redundant to CmdPkgRequires / print_pkg_relations
class CmdRequiredOsPkg(Cmd): # export class CmdRequiredOsPkg(Cmd): # export
def __init__(self, parent: CmdProjects) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'required-os-pkg', help='List distribution packages required for a package') super().__init__(
parent,
'required-os-pkg',
help = 'List distribution packages required for a package',
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('flavours', help='Dependency flavours', default='build') parser.add_argument('flavours', help = 'Dependency flavours', default = 'build')
parser.add_argument('modules', nargs='*', help='Modules') parser.add_argument('modules', nargs = '*', help = 'Modules')
parser.add_argument('--skip-excluded', action='store_true', default=False, parser.add_argument(
help='Output empty prerequisite list for excluded modules') '--skip-excluded',
parser.add_argument('--quote', action='store_true', default=False, action = 'store_true',
help='Put double quotes around each listed dependency') default = False,
help = 'Output empty prerequisite list for excluded modules',
)
parser.add_argument(
'--quote',
action = 'store_true',
default = False,
help = 'Put double quotes around each listed dependency',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
modules = args.modules modules = args.modules
flavours = set(args.flavours.split(',')) flavours = set(args.flavours.split(','))
if 'build' in flavours: if 'build' in flavours:
# TODO: This adds too much. Only the run dependencies of the build dependencies would be needed. # TODO: This adds too much. Only the run dependencies of the build
# dependencies would be needed.
flavours.add('run') flavours.add('run')
if 'release' in flavours: if 'release' in flavours:
flavours |= set(['run', 'devel', 'build']) flavours |= set(['run', 'devel', 'build'])
log(DEBUG, "flavours = " + args.flavours) log(DEBUG, 'flavours = ' + args.flavours)
deps = self.app.get_project_refs(modules, ['pkg.requires.jw'], flavours, deps = self.app.get_project_refs(
scope = Scope.Subtree, add_self=True, names_only=True) modules,
['pkg.requires.jw'],
list(flavours),
scope = Scope.Subtree,
add_self = True,
names_only = True,
)
if args.skip_excluded: if args.skip_excluded:
for d in deps: for d in deps:
if self.app.is_excluded_from_build(d) is not None: if self.app.is_excluded_from_build(d) is not None:
deps.remove(d) deps.remove(d)
subsecs = self.app.distro.os_cascade subsecs = self.app.distro.os_cascade
log(DEBUG, "subsecs = ", subsecs) log(DEBUG, 'subsecs = ', subsecs)
requires: set[str] = set() requires: set[str] = set()
for sec in subsecs: for sec in subsecs:
for flavour in flavours: for flavour in flavours:
@ -47,6 +62,6 @@ class CmdRequiredOsPkg(Cmd): # export
if vals: if vals:
requires |= set(vals) requires |= set(vals)
if args.quote: if args.quote:
requires = [f'"{dep}"' for dep in requires] out = [f'"{dep}"' for dep in requires]
# TODO: add all not in build tree as -devel # TODO: add all not in build tree as -devel
print(' '.join(requires)) print(' '.join(out))

19
src/python/jw/pkg/cmds/projects/CmdSummary.py
View file

@ -1,23 +1,22 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdSummary(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdSummary(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(
def __init__(self, parent: CmdProjects) -> None: parent, 'summary', help = 'Print summary description of given modules'
super().__init__(parent, 'summary', help='Print summary description of given modules') )
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
r = [] r = []
for m in args.module: for m in args.module:
summary = self.app.get_value(m, "summary", None) summary = self.app.get_value(m, 'summary', None)
if summary is not None: if summary is not None:
r.append(summary) r.append(summary)
print(' '.join(r)) print(' '.join(r))

17
src/python/jw/pkg/cmds/projects/CmdTest.py
View file

@ -1,18 +1,15 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdTest(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdTest(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'test', help = 'Test')
def __init__(self, parent: CmdProjects) -> None:
super().__init__(parent, 'test', help='Test')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('blah', default='', help='The blah argument') parser.add_argument('blah', default = '', help = 'The blah argument')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
print("blah = " + args.blah) print('blah = ' + args.blah)

19
src/python/jw/pkg/cmds/projects/CmdTmplDir.py
View file

@ -1,18 +1,19 @@
# -*- coding: utf-8 -*- from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser from .Cmd import Cmd, Parent
from ..Cmd import Cmd class CmdTmplDir(Cmd): # export
from ..CmdProjects import CmdProjects
class CmdTmplDir(Cmd): # export def __init__(self, parent: Parent) -> None:
super().__init__(
def __init__(self, parent: CmdProjects) -> None: parent,
super().__init__(parent, 'tmpl-dir', help='Print directory containing templates of a given module') 'tmpl-dir',
help = 'Print directory containing templates of a given module',
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('module', nargs='*', help='Modules') parser.add_argument('module', nargs = '*', help = 'Modules')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
r = [] r = []

6
src/python/jw/pkg/cmds/projects/__init__.py
View file

@ -4,6 +4,6 @@ __all__ = detect_modules(
package_name = __name__, package_name = __name__,
package_path = __path__, package_path = __path__,
namespace = globals(), namespace = globals(),
prefix = "Cmd", prefix = 'Cmd',
skip = {"Cmd"}, skip = {'Cmd'},
) # pyright: ignore[reportUnsupportedDunderAll] ) # pyright: ignore[reportUnsupportedDunderAll]

32
src/python/jw/pkg/cmds/secrets/Cmd.py
View file

@ -1,24 +1,22 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from argparse import ArgumentParser
from functools import cached_property from functools import cached_property
from typing import TYPE_CHECKING
from ..Cmd import Cmd as Base from ...CmdBase import CmdBase
from ..CmdSecrets import CmdSecrets as Parent
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Iterable from typing import Iterable
from ...lib.Distro import Distro from .lib.base import Attrs
from ...lib.ExecContext import ExecContext
from ..CmdSecrets import CmdSecrets
from .lib.DistroContext import DistroContext from .lib.DistroContext import DistroContext
class Cmd(Base): # export class Cmd(CmdBase): # export
@cached_property @cached_property
def ctx(self) -> ExecContext: def ctx(self) -> DistroContext:
return DistroContext(self.app.distro) return DistroContext(self.app.distro)
async def _match_files(self, packages: Iterable[str], pattern: str) -> list[str]: async def _match_files(self, packages: Iterable[str], pattern: str) -> list[str]:
@ -27,21 +25,27 @@ class Cmd(Base): # export
async def _list_template_files(self, packages: Iterable[str]) -> list[str]: async def _list_template_files(self, packages: Iterable[str]) -> list[str]:
return await self.ctx.list_template_files(packages) return await self.ctx.list_template_files(packages)
async def _list_secret_paths(self, packages: Iterable[str], ignore_missing: bool=False) -> list[str]: async def _list_secret_paths(
self, packages: Iterable[str], ignore_missing: bool = False
) -> list[str]:
return await self.ctx.list_secret_paths(packages, ignore_missing) return await self.ctx.list_secret_paths(packages, ignore_missing)
async def _list_compilation_targets(self, packages: Iterable[str], ignore_missing: bool=False) -> list[str]: async def _list_compilation_targets(
self, packages: Iterable[str], ignore_missing: bool = False
) -> list[str]:
return await self.ctx.list_compilation_targets(packages, ignore_missing) return await self.ctx.list_compilation_targets(packages, ignore_missing)
async def _remove_compilation_targets(self, packages: Iterable[str]) -> list[str]: async def _remove_compilation_targets(self, packages: Iterable[str]) -> list[str]:
return await self.ctx.remove_compilation_targets(packages) return await self.ctx.remove_compilation_targets(packages)
async def _compile_template_files(self, packages: Iterable[str], default_attrs: Attrs) -> list[str]: async def _compile_template_files(
self, packages: Iterable[str], default_attrs: Attrs
) -> list[str]:
return await self.ctx.compile_template_files(packages, default_attrs) return await self.ctx.compile_template_files(packages, default_attrs)
def __init__(self, parent: CmdSecrets, name: str, help: str) -> None: def __init__(self, parent: Parent, name: str, help: str) -> None:
super().__init__(parent, name, help) super().__init__(parent, name, help)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument("packages", nargs='*', help="Package names") parser.add_argument('packages', nargs = '*', help = 'Package names')

29
src/python/jw/pkg/cmds/secrets/CmdCompileTemplates.py
View file

@ -1,20 +1,21 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .Cmd import Cmd
from .lib.base import Attrs from .lib.base import Attrs
from .Cmd import Cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdSecrets import CmdSecrets from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
class CmdCompileTemplates(Cmd): # export from ..CmdSecrets import CmdSecrets
class CmdCompileTemplates(Cmd): # export
def __init__(self, parent: CmdSecrets) -> None: def __init__(self, parent: CmdSecrets) -> None:
super().__init__(parent, 'compile-templates', help="Compile package template files") super().__init__(
parent, 'compile-templates', help = 'Compile package template files'
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
attrs = Attrs(args.mode, args.owner, args.group, None) attrs = Attrs(args.mode, args.owner, args.group, None)
@ -22,6 +23,12 @@ class CmdCompileTemplates(Cmd): # export
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument('--owner', '-o', default=None, help='Default output file owner') parser.add_argument(
parser.add_argument('--group', '-g', default=None, help='Default output file group') '--owner', '-o', default = None, help = 'Default output file owner'
parser.add_argument('--mode', '-m', default=None, help='Default output file mode') )
parser.add_argument(
'--group', '-g', default = None, help = 'Default output file group'
)
parser.add_argument(
'--mode', '-m', default = None, help = 'Default output file mode'
)

27
src/python/jw/pkg/cmds/secrets/CmdInstall.py
View file

@ -1,22 +1,33 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .Cmd import Cmd from .Cmd import Cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdSecrets import CmdSecrets from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
class CmdInstall(Cmd): # export from ..CmdSecrets import CmdSecrets
class CmdInstall(Cmd): # export
def __init__(self, parent: CmdSecrets) -> None: def __init__(self, parent: CmdSecrets) -> None:
super().__init__(parent, 'install', help='Install secrets from various sources as static secrets onto the target') super().__init__(
parent,
'install',
help = (
'Install secrets from various sources as static secrets onto the target'
),
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
parser.add_argument('src', help='URI of secret source') parser.add_argument('src', help = 'URI of secret source')
parser.add_argument('--only-missing', action='store_true', default=False, help='Install only secrets not already on the target') parser.add_argument(
'--only-missing',
action = 'store_true',
default = False,
help = 'Install only secrets not already on the target',
)
super().add_arguments(parser) super().add_arguments(parser)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:

31
src/python/jw/pkg/cmds/secrets/CmdListCompilationOutput.py
View file

@ -1,22 +1,37 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .Cmd import Cmd from .Cmd import Cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdSecrets import CmdSecrets from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
class CmdListCompilationOutput(Cmd): # export from ..CmdSecrets import CmdSecrets
class CmdListCompilationOutput(Cmd): # export
def __init__(self, parent: CmdSecrets) -> None: def __init__(self, parent: CmdSecrets) -> None:
super().__init__(parent, 'list-compilation-output', help="List package compilation output files") super().__init__(
parent,
'list-compilation-output',
help = 'List package compilation output files',
)
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument("--all", action='store_true', default=False, help="Show all output targets, including non-existent files") parser.add_argument(
'--all',
action = 'store_true',
default = False,
help = 'Show all output targets, including non-existent files',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
print('\n'.join(await self._list_compilation_targets(args.packages, ignore_missing=(not args.all)))) print(
'\n'.join(
await self._list_compilation_targets(
args.packages, ignore_missing = (not args.all)
)
)
)

26
src/python/jw/pkg/cmds/secrets/CmdListSecrets.py
View file

@ -1,22 +1,32 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .Cmd import Cmd from .Cmd import Cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdSecrets import CmdSecrets from argparse import ArgumentParser, Namespace
from argparse import Namespace, ArgumentParser
class CmdListSecrets(Cmd): # export from ..CmdSecrets import CmdSecrets
class CmdListSecrets(Cmd): # export
def __init__(self, parent: CmdSecrets) -> None: def __init__(self, parent: CmdSecrets) -> None:
super().__init__(parent, 'list-secrets', help="List package secret files") super().__init__(parent, 'list-secrets', help = 'List package secret files')
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
super().add_arguments(parser) super().add_arguments(parser)
parser.add_argument("--all", action='store_true', default=False, help="Show all secret paths, including non-existent files") parser.add_argument(
'--all',
action = 'store_true',
default = False,
help = 'Show all secret paths, including non-existent files',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
print('\n'.join(await self._list_secret_paths(args.packages, ignore_missing=(not args.all)))) print(
'\n'.join(
await
self._list_secret_paths(args.packages, ignore_missing = (not args.all))
)
)

12
src/python/jw/pkg/cmds/secrets/CmdListTemplates.py
View file

@ -1,18 +1,18 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .Cmd import Cmd from .Cmd import Cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdSecrets import CmdSecrets from argparse import Namespace
from argparse import Namespace, ArgumentParser
class CmdListTemplates(Cmd): # export from ..CmdSecrets import CmdSecrets
class CmdListTemplates(Cmd): # export
def __init__(self, parent: CmdSecrets) -> None: def __init__(self, parent: CmdSecrets) -> None:
super().__init__(parent, 'list-templates', help="List package template files") super().__init__(parent, 'list-templates', help = 'List package template files')
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
print('\n'.join(await self._list_template_files(args.packages))) print('\n'.join(await self._list_template_files(args.packages)))

18
src/python/jw/pkg/cmds/secrets/CmdRmCompilationOutput.py
View file

@ -1,18 +1,20 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .Cmd import Cmd from .Cmd import Cmd, Parent
if TYPE_CHECKING: if TYPE_CHECKING:
from ..CmdSecrets import CmdSecrets from argparse import Namespace
from argparse import Namespace, ArgumentParser
class CmdRmCompilationOutput(Cmd): # export class CmdRmCompilationOutput(Cmd): # export
def __init__(self, parent: CmdSecrets) -> None: def __init__(self, parent: Parent) -> None:
super().__init__(parent, 'rm-compilation-output', help="Remove package compilation output files") super().__init__(
parent,
'rm-compilation-output',
help = 'Remove package compilation output files',
)
async def _run(self, args: Namespace) -> None: async def _run(self, args: Namespace) -> None:
await self._remove_compilation_targets(args.packages) await self._remove_compilation_targets(args.packages)

135
src/python/jw/pkg/cmds/secrets/lib/DistroContext.py
View file

@ -1,24 +1,20 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
import re import re
import sys
from pathlib import Path from pathlib import Path
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ....lib.log import DEBUG, NOTICE, WARNING, log
from ....lib.ProcFilterGpg import ProcFilterGpg
from .base import Attrs
from .FilesContext import FilesContext
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Iterable from typing import Iterable
from ....lib.FileContext import FileContext
from ....lib.log import * from ....lib.Distro import Distro
from ....lib.util import run_cmd
from ....lib.TarIo import TarIo
from ....lib.ProcFilterGpg import ProcFilterGpg
from .base import Attrs
from .FilesContext import FilesContext
class DistroContext(FilesContext): class DistroContext(FilesContext):
@ -37,18 +33,22 @@ class DistroContext(FilesContext):
async def list_template_files(self, pkg_names: Iterable[str]) -> list[str]: async def list_template_files(self, pkg_names: Iterable[str]) -> list[str]:
if not pkg_names: if not pkg_names:
pkg_names = [p.name for p in await self.__distro.select()] pkg_names = [p.name for p in await self.__distro.select()]
return await self.match_files(pkg_names, pattern=r'.*\.jw-tmpl$') return await self.match_files(pkg_names, pattern = r'.*\.jw-tmpl$')
async def list_secret_paths(self, pkg_names: Iterable[str], ignore_missing: bool=False) -> list[str]: async def list_secret_paths(
self, pkg_names: Iterable[str], ignore_missing: bool = False
) -> list[str]:
ret = [] ret = []
for tmpl in await self.list_template_files(pkg_names): for tmpl in await self.list_template_files(pkg_names):
path = str(Path(tmpl).with_suffix(".jw-secret")) path = str(Path(tmpl).with_suffix('.jw-secret'))
if ignore_missing and not await self.ctx.file_exists(path): if ignore_missing and not await self.ctx.file_exists(path):
continue continue
ret.append(path) ret.append(path)
return ret return ret
async def list_compilation_targets(self, pkg_names: Iterable[str], ignore_missing: bool=False) -> list[str]: async def list_compilation_targets(
self, pkg_names: Iterable[str], ignore_missing: bool = False
) -> list[str]:
ret = [] ret = []
for tmpl in await self.list_template_files(pkg_names): for tmpl in await self.list_template_files(pkg_names):
path = tmpl.removesuffix('.jw-tmpl') path = tmpl.removesuffix('.jw-tmpl')
@ -58,72 +58,119 @@ class DistroContext(FilesContext):
return ret return ret
async def remove_compilation_targets(self, pkg_names: Iterable[str]) -> list[str]: async def remove_compilation_targets(self, pkg_names: Iterable[str]) -> list[str]:
ret: list[str] = []
for path in await self.list_compilation_targets(pkg_names): for path in await self.list_compilation_targets(pkg_names):
try: try:
self.ctx.stat(path) await self.ctx.stat(path)
log(NOTICE, f'Removing {path}') log(NOTICE, f'Removing {path}')
await self.ctx.unlink(path) await self.ctx.unlink(path)
except FileNotFoundError as e: ret.append(path)
log(DEBUG, f'Compilation target {path} doesn\'t exist (ignored)') except FileNotFoundError:
log(DEBUG, f"Compilation target {path} doesn't exist (ignored)")
continue continue
return ret
async def compile_template_files(self, pkg_names: Iterable[str], default_attrs: Attrs) -> list[str]: async def compile_template_files(
self, pkg_names: Iterable[str], default_attrs: Attrs
) -> list[str]:
ret: list[str] = []
missing = 0 missing = 0
for target in await self.list_compilation_targets(pkg_names): for target in await self.list_compilation_targets(pkg_names):
if not await self.compile_template_file(target, default_attrs): if await self.compile_template_file(target, default_attrs):
ret.append(target)
else:
missing += 1 missing += 1
if missing > 0: if missing > 0:
log(WARNING, f'{missing} missing secrets found. You might want to add them and run sudo {app.cmdline} again') from ....lib.util import pretty_cmd
async def install(self, src_uri: str, pkg_names: Iterable[str], only_missing: bool=False, verbose: bool=False) -> None: cmdline = pretty_cmd(sys.argv)
log(
WARNING,
(
f'{missing} missing secrets found. You might want to add them and '
f'run sudo {cmdline} again'
),
)
return ret
async def _read_secret_tar_blob(src_uri: str): async def install(
self,
src_uri: str,
pkg_names: Iterable[str],
only_missing: bool = False,
verbose: bool = False,
) -> None:
async def _read_secret_tar_blob(src_uri: str) -> bytes:
ec = self.ctx ec = self.ctx
from ....lib.ec.Local import Local from ....lib.ec.Local import Local
from ....lib.util import get from ....lib.util import get
if not isinstance(ec, Local): if not isinstance(ec, Local):
ec = Local() # Security: Use a local exec context for decrypting and filtering secrets # Security: Use a local exec context for decrypting and
return (await get(src_uri, content_filter=ProcFilterGpg(ec=ec))).stdout # filtering secrets
ec = Local()
return (await get(src_uri, content_filter = ProcFilterGpg(ec = ec))).stdout
def _matches_host_prefix(path: str) -> bool: def _matches_host_prefix(path: str) -> bool:
return re.match(r'^' + root_in_tar, path) return re.match(r'^' + host_root_in_tar, path) is not None
def _crop_host_prefix(path: str) -> bool: def _crop_host_prefix(path: str) -> bool:
return re.sub(r'^' + root_in_tar, '', path) return re.sub(r'^' + host_root_in_tar, '', path) is not None
def _crop_default_prefix(path: str) -> bool: def _crop_default_prefix(path: str) -> bool:
return re.sub(r'^' + default, '', path) return re.sub(default_rx, '', path) is not None
def _matches_default_prefix(path: str) -> bool: def _matches_default_prefix(path: str) -> bool:
return re.match(r'^default', path) return re.match(r'^default', path) is not None
def _is_needed_secret(path: str) -> bool: def _is_needed_secret(path: str) -> bool:
return path in secret_paths return path in secret_paths
from .tar import filter as tar_filter, rewrite as tar_rewrite, extract as tar_extract, merge as tar_merge from .tar import extract as tar_extract
from .tar import filter as tar_filter
from .tar import merge as tar_merge
from .tar import rewrite as tar_rewrite
if only_missing: if only_missing:
raise NotImplementedError('--only-missing is not yet implemented') raise NotImplementedError('--only-missing is not yet implemented')
secret_paths = await self.list_secret_paths(pkg_names) secret_paths = await self.list_secret_paths(pkg_names)
hostname = self.ctx.uri.hostname
host_root_in_tar = '/'.join(reversed(self.ctx.hostname.split('.'))) if not hostname:
raise Exception('Have no hostname to find secrets in tar file')
host_root_in_tar = '/'.join(reversed(hostname.split('.')))
host_rx = re.compile(r'^' + host_root_in_tar) host_rx = re.compile(r'^' + host_root_in_tar)
default_rx = re.compile(r'^default') default_rx = re.compile(r'^default')
filtered_paths: list[str] = [] filtered_paths: list[str] = []
extracted_paths: list[str] = [] extracted_paths: list[str] = []
blob_all = await _read_secret_tar_blob(src_uri) blob_all = await _read_secret_tar_blob(src_uri)
blob_host_filtered = tar_filter(blob_all, lambda p: re.match(host_rx, p), filtered_paths) if blob_all is None:
blob_host_transformed = tar_rewrite(blob_host_filtered, lambda p: re.sub(host_rx, '', p)) raise Exception(f'Tar blob {src_uri} is empty')
blob_default_filtered = tar_filter(blob_all, lambda p: re.match(default_rx, p), filtered_paths) blob_host_filtered = tar_filter(
blob_default_transformed = tar_rewrite(blob_default_filtered, lambda p: re.sub(default_rx, '', p)) blob_all, lambda p: re.match(host_rx, p) is not None, filtered_paths
blob_secret_material = tar_merge([blob_host_transformed, blob_default_transformed], overwrite=False) )
blob_needed = tar_filter(blob_secret_material, _is_needed_secret, extracted_paths) blob_host_transformed = tar_rewrite(
blob_host_filtered, lambda p: re.sub(host_rx, '', p)
)
blob_default_filtered = tar_filter(
blob_all, lambda p: re.match(default_rx, p) is not None, filtered_paths
)
blob_default_transformed = tar_rewrite(
blob_default_filtered, lambda p: re.sub(default_rx, '', p)
)
blob_secret_material = tar_merge(
[blob_host_transformed, blob_default_transformed], overwrite = False
)
blob_needed = tar_filter(
blob_secret_material, _is_needed_secret, extracted_paths
)
await tar_extract(self.ctx, blob_needed, root='/', verbose=verbose) await tar_extract(self.ctx, blob_needed, root = '/', verbose = verbose)
for path in secret_paths: for path in secret_paths:
if not path in extracted_paths: if path not in extracted_paths:
log(NOTICE, f'not extracted: {path}') log(NOTICE, f'not extracted: {path}')
else: else:
log(NOTICE, f'extracted: {path}') log(NOTICE, f'extracted: {path}')

116
src/python/jw/pkg/cmds/secrets/lib/FilesContext.py
View file

@ -1,21 +1,17 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
import re, stat, copy import copy
import re
import stat
from contextlib import suppress from contextlib import suppress
from pathlib import Path
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Iterable from typing import Iterable
from ....lib.FileContext import FileContext from ....lib.FileContext import FileContext
from ....lib.log import * from ....lib.log import DEBUG, NOTICE, WARNING, log
from ....lib.util import run_cmd
from .base import Attrs from .base import Attrs
class FilesContext: class FilesContext:
@ -27,17 +23,17 @@ class FilesContext:
def ctx(self) -> FileContext: def ctx(self) -> FileContext:
return self.__ctx return self.__ctx
async def _read_key_value_file(self, path: str, throw=False) -> dict[str, str]: async def _read_key_value_file(self, path: str, throw = False) -> dict[str, str]:
ret: dict[str, str] = {} ret: dict[str, str] = {}
try: try:
result = await self.ctx.get(path) result = await self.ctx.get(path)
for line in result.stdout.decode().splitlines(): for line in result.stdout_str.splitlines():
line = line.strip() line = line.strip()
if not line or line.startswith("#"): if not line or line.startswith('#'):
continue continue
if "=" not in line: if '=' not in line:
continue continue
key, val = line.split("=", 1) key, val = line.split('=', 1)
key = key.strip() key = key.strip()
val = val.strip() val = val.strip()
if key: if key:
@ -46,39 +42,39 @@ class FilesContext:
log(DEBUG, f'File not found {path}') log(DEBUG, f'File not found {path}')
return ret return ret
def _parse_attributes(self, content: str) -> Attrs: def _parse_attributes(self, content: str) -> Attrs | None:
if not content: if not content:
return None return None
first_line = content.splitlines()[0] first_line = content.splitlines()[0]
if not re.match(r"^\s*#\s*conf\s*:", first_line): if not re.match(r'^\s*#\s*conf\s*:', first_line):
return None return None
ret = Attrs() ret = Attrs()
ret.conf = first_line ret.conf = first_line
m = re.match(r"^\s*#\s*conf\s*:\s*(.*?)\s*$", first_line) m = re.match(r'^\s*#\s*conf\s*:\s*(.*?)\s*$', first_line)
if not m: if not m:
return ret return ret
for part in re.split(r'[; ,]', m.group(1)): for part in re.split(r'[; ,]', m.group(1)):
part = part.strip() part = part.strip()
if not part or "=" not in part: if not part or '=' not in part:
continue continue
key, val = part.split("=", 1) key, val = part.split('=', 1)
key = key.strip() key = key.strip()
val = val.strip() val = val.strip()
if key == "owner": if key == 'owner':
ret.owner = val or None ret.owner = val or None
elif key == "group": elif key == 'group':
ret.group = val or None ret.group = val or None
elif key == "mode": elif key == 'mode':
if val: if val:
try: try:
if re.fullmatch(r"0[0-7]+", val): if re.fullmatch(r'0[0-7]+', val):
ret.mode = int(val, 8) ret.mode = int(val, 8)
else: else:
ret.mode = int(val, 0) ret.mode = int(val, 0)
@ -87,20 +83,26 @@ class FilesContext:
return ret return ret
async def _read_attributes(self, paths: Iterable[str]) -> Attrs|None: async def _read_attributes(self, paths: Iterable[str]) -> Attrs | None:
ret = Attrs() ret = Attrs()
for path in paths: for path in paths:
try: try:
result = await self.ctx.get(path) result = await self.ctx.get(path)
lines = result.stdout.decode().splitlines() lines = result.stdout_str.splitlines()
if lines: if lines:
ret.update(self._parse_attributes(lines[0])) ret.update(self._parse_attributes(lines[0]))
except FileNotFoundError: except FileNotFoundError:
log(DEBUG, f'Can\'t parse "{path}" for attributes, file doesn\'t exist (ignored)') log(
DEBUG,
(
f'Can\'t parse "{path}" for attributes, '
"file doesn't exist (ignored)"
),
)
return ret return ret
def _format_metadata(self, owner: str, group: str, mode: int) -> str: def _format_metadata(self, owner: str, group: str, mode: int) -> str:
return f"{owner}:{group} {mode:o}" return f'{owner}:{group} {mode:o}'
async def _compile_one_template_file( async def _compile_one_template_file(
self, self,
@ -110,11 +112,12 @@ class FilesContext:
replace: dict[str, str] = {}, replace: dict[str, str] = {},
) -> None: ) -> None:
owner = "root" owner = 'root'
group = "root" group = 'root'
mode = 0o400 mode = 0o400
new_content = (await self.ctx.get(src)).stdout.decode() result = await self.ctx.get(src)
new_content = result.stdout_str
attrs = self._parse_attributes(new_content) attrs = self._parse_attributes(new_content)
if attrs is None: if attrs is None:
@ -133,15 +136,21 @@ class FilesContext:
for key, val in replace.items(): for key, val in replace.items():
new_content = new_content.replace(key, val) new_content = new_content.replace(key, val)
tmp_path: str|None = None tmp_path: str | None = None
try: try:
tmp_path = await self.ctx.mktemp(dst + '.jw-pkg.XXXXX') tmp_path = await self.ctx.mktemp(dst + '.jw-pkg.XXXXX')
await self.ctx.put(tmp_path, new_content.encode('utf-8'), owner=owner, group=group, mode=mode) await self.ctx.put(
tmp_path,
new_content.encode('utf-8'),
owner = owner,
group = group,
mode = mode,
)
content_changed = True content_changed = True
metadata_changed = True metadata_changed = True
old_meta = "<missing>" old_meta = '<missing>'
try: try:
st = await self.ctx.stat(dst) st = await self.ctx.stat(dst)
@ -150,23 +159,21 @@ class FilesContext:
else: else:
old_mode = stat.S_IMODE(st.mode) old_mode = stat.S_IMODE(st.mode)
old_meta = self._format_metadata(st.owner, st.group, old_mode) old_meta = self._format_metadata(st.owner, st.group, old_mode)
old_content = (await self.ctx.get(dst)).stdout.decode() old_content = (await self.ctx.get(dst)).stdout_str
content_changed = old_content != new_content content_changed = old_content != new_content
metadata_changed = ( metadata_changed = (
st.owner != owner st.owner != owner or st.group != group or old_mode != mode
or st.group != group
or old_mode != mode
) )
changes = [] changes = []
if content_changed: if content_changed:
changes.append("@content") changes.append('@content')
if metadata_changed: if metadata_changed:
changes.append(f"@metadata ({old_meta} -> {new_meta})") changes.append(f'@metadata ({old_meta} -> {new_meta})')
details = ", ".join(changes) if changes else "no changes" details = ', '.join(changes) if changes else 'no changes'
log(NOTICE, f"Applying macros in {src} to {dst}: {details}") log(NOTICE, f'Applying macros in {src} to {dst}: {details}')
if not changes: if not changes:
await self.ctx.unlink(tmp_path) await self.ctx.unlink(tmp_path)
@ -181,23 +188,28 @@ class FilesContext:
with suppress(FileNotFoundError): with suppress(FileNotFoundError):
await self.ctx.unlink(tmp_path) await self.ctx.unlink(tmp_path)
async def compile_template_file(self, target_path: str, default_attrs: Attrs|None=None) -> bool: async def compile_template_file(
path_tmpl = target_path + '.jw-tmpl' self, target_path: str, default_attrs: Attrs | None = None
) -> bool:
path_tmpl = target_path + '.jw-tmpl'
path_secret_file = target_path + '.jw-secret-file' path_secret_file = target_path + '.jw-secret-file'
path_secret = target_path + '.jw-secret' path_secret = target_path + '.jw-secret'
attrs = copy.deepcopy(default_attrs if default_attrs is not None else Attrs()) attrs = copy.deepcopy(default_attrs if default_attrs is not None else Attrs())
attrs.update(await self._read_attributes([ attrs.update(
path_tmpl, await self._read_attributes([path_tmpl, path_secret_file, path_secret])
path_secret_file, )
path_secret
]))
replace = await self._read_key_value_file(path_secret) replace = await self._read_key_value_file(path_secret)
for src in [ path_secret_file, path_tmpl ]: for src in [path_secret_file, path_tmpl]:
try: try:
await self._compile_one_template_file(src=src, dst=target_path, default_attrs=attrs, replace=replace) await self._compile_one_template_file(
src = src,
dst = target_path,
default_attrs = attrs,
replace = replace
)
return True return True
except FileNotFoundError as e: except FileNotFoundError:
log(DEBUG, f'Compilation source {src} doesn\'t exist (ignored)') log(DEBUG, f"Compilation source {src} doesn't exist (ignored)")
continue continue
log(WARNING, f'No secret found for target {target_path}, not compiling') log(WARNING, f'No secret found for target {target_path}, not compiling')

6
src/python/jw/pkg/cmds/secrets/lib/base.py
View file

@ -1,18 +1,15 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from dataclasses import dataclass from dataclasses import dataclass
@dataclass @dataclass
class Attrs: class Attrs:
mode: int | None = None mode: int | None = None
owner: str | None = None owner: str | None = None
group: str | None = None group: str | None = None
conf: str | None = None conf: str | None = None
def update(self, rhs: Args|None) -> Args: def update(self, rhs: Attrs | None) -> Attrs:
if rhs is not None: if rhs is not None:
if rhs.mode: if rhs.mode:
self.mode = rhs.mode self.mode = rhs.mode
@ -32,4 +29,3 @@ class Attrs:
if self.group is not None: if self.group is not None:
return False return False
return True return True

50
src/python/jw/pkg/cmds/secrets/lib/tar.py
View file

@ -1,21 +1,27 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
import io
import tarfile
from tarfile import TarFile
from typing import TYPE_CHECKING, Callable from typing import TYPE_CHECKING, Callable
import tarfile, io from ....lib.log import DEBUG, log
from tarfile import TarFile
from ....lib.log import *
if TYPE_CHECKING: if TYPE_CHECKING:
from ....lib.ExecContext import ExecContext from typing import Iterable
def filter(blob: bytes, path_filter: Callable[[str], bool]|None, matched: list[str]|None=None) -> bytes: from ....lib.ExecContext import ExecContext
from ....lib.FileContext import FileContext
def filter(
blob: bytes,
path_filter: Callable[[str], bool] | None,
matched: list[str] | None = None,
) -> bytes:
ret = io.BytesIO() ret = io.BytesIO()
with tarfile.open(fileobj=ret, mode='w') as tf_out: with tarfile.open(fileobj = ret, mode = 'w') as tf_out:
tf_in = TarFile(fileobj=io.BytesIO(blob)) tf_in = TarFile(fileobj = io.BytesIO(blob))
for info in tf_in.getmembers(): for info in tf_in.getmembers():
if path_filter is not None and not path_filter(info.name): if path_filter is not None and not path_filter(info.name):
continue continue
@ -28,8 +34,8 @@ def filter(blob: bytes, path_filter: Callable[[str], bool]|None, matched: list[s
def rewrite(blob: bytes, rewrite_filter: Callable[[str], str]) -> bytes: def rewrite(blob: bytes, rewrite_filter: Callable[[str], str]) -> bytes:
ret = io.BytesIO() ret = io.BytesIO()
with tarfile.open(fileobj=ret, mode='w') as tf_out: with tarfile.open(fileobj = ret, mode = 'w') as tf_out:
tf_in = TarFile(fileobj=io.BytesIO(blob)) tf_in = TarFile(fileobj = io.BytesIO(blob))
for info in tf_in.getmembers(): for info in tf_in.getmembers():
new_name = rewrite_filter(info.name) new_name = rewrite_filter(info.name)
log(DEBUG, f'Rewriting {info.name} -> {new_name}') log(DEBUG, f'Rewriting {info.name} -> {new_name}')
@ -38,11 +44,11 @@ def rewrite(blob: bytes, rewrite_filter: Callable[[str], str]) -> bytes:
tf_out.addfile(info, buf) tf_out.addfile(info, buf)
return ret.getvalue() return ret.getvalue()
def merge(blobs: Iterable[bytes], overwrite: bool=False) -> bytes: def merge(blobs: Iterable[bytes], overwrite: bool = False) -> bytes:
ret = io.BytesIO() ret = io.BytesIO()
with tarfile.open(fileobj=ret, mode='w') as tf_out: with tarfile.open(fileobj = ret, mode = 'w') as tf_out:
for blob in blobs: for blob in blobs:
tf_in = TarFile(fileobj=io.BytesIO(blob)) tf_in = TarFile(fileobj = io.BytesIO(blob))
existing_names = tf_out.getnames() existing_names = tf_out.getnames()
for info in tf_in.getmembers(): for info in tf_in.getmembers():
if not overwrite and info.name in existing_names: if not overwrite and info.name in existing_names:
@ -51,11 +57,21 @@ def merge(blobs: Iterable[bytes], overwrite: bool=False) -> bytes:
tf_out.addfile(info, buf) tf_out.addfile(info, buf)
return ret.getvalue() return ret.getvalue()
async def extract(dst: ExecContext, blob: bytes, root: str|None=None, verbose: bool=False) -> None: async def extract(
dst: FileContext,
blob: bytes,
root: str | None = None,
verbose: bool = False
) -> None:
cmd = ['tar'] cmd = ['tar']
if root is not None: if root is not None:
cmd += ['-C', root] cmd += ['-C', root]
if verbose: if verbose:
cmd += '-v' cmd += '-v'
cmd += ['-x', '-f', '-'] cmd += ['-x', '-f', '-']
await dst.run(cmd, verbose=verbose, cmd_input=blob) if not isinstance(dst, ExecContext):
raise NotImplementedError(
'Extracting tar files to a non-executable '
f'context is not yet implemented: {dst}'
)
await dst.run(cmd, verbose = verbose, cmd_input = blob)

14
src/python/jw/pkg/cmds/secrets/lib/util.py
View file

@ -1,17 +1,21 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ....lib.FileContext import FileContext
from ....lib.ec.Local import Local from ....lib.ec.Local import Local
from ....lib.FileContext import FileContext
from .FilesContext import FilesContext from .FilesContext import FilesContext
if TYPE_CHECKING: if TYPE_CHECKING:
from .base import Attrs from .base import Attrs
async def compile_template_file(target_path: str, default_attrs: Attrs|None=None, ctx: FileContext|None=None) -> bool: async def compile_template_file(
target_path: str,
default_attrs: Attrs | None = None,
ctx: FileContext | None = None
) -> bool:
if ctx is None: if ctx is None:
ctx = Local() ctx = Local()
await FilesContext(ctx).compile_template_file(target_path, default_attrs=default_attrs) return await FilesContext(ctx).compile_template_file(
target_path, default_attrs = default_attrs
)

187
src/python/jw/pkg/lib/App.py
View file

@ -1,76 +1,119 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import Any, TYPE_CHECKING import asyncio
import cProfile
import os
import sys
import os, sys, argparse, re, asyncio, cProfile from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, Namespace
from typing import TYPE_CHECKING, Any
from .AsyncRunner import AsyncRunner from .AsyncRunner import AsyncRunner
from .log import * from .log import DEBUG, ERR, NOTICE, log, set_log_flags, set_log_level
from .Types import LoadTypes from .Types import LoadTypes
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import TypeVar
from collections.abc import Awaitable from collections.abc import Awaitable
T = TypeVar("T") from typing import TypeVar
class App: # export T = TypeVar('T')
class App: # export
def _add_arguments(self, parser): def _add_arguments(self, parser):
self.__parser.add_argument('--log-flags', help='Log flags', default=self.__default_log_flags) self.__parser.add_argument(
self.__parser.add_argument('--log-level', help='Log level', default=self.__default_log_level) '--log-flags', help = 'Log flags', default = self.__default_log_flags
self.__parser.add_argument('--log-file', help='Log file', default=self.__default_log_file) )
self.__parser.add_argument('--backtrace', help='Show exception backtraces', action='store_true', default=self.__back_trace) self.__parser.add_argument(
self.__parser.add_argument('--write-profile', help='Profile code and store output to file', default=None) '--log-level', help = 'Log level', default = self.__default_log_level
)
self.__parser.add_argument(
'--log-file', help = 'Log file', default = self.__default_log_file
)
self.__parser.add_argument(
'--backtrace',
help = 'Show exception backtraces',
action = 'store_true',
default = self.__back_trace,
)
self.__parser.add_argument(
'--write-profile',
help = 'Profile code and store output to file',
default = None,
)
def __init__(self, description: str = '', name_filter: str = '^Cmd.*', modules: None=None, eloop: None=None) -> None: def __init__(
self,
description: str = '',
name_filter: str = '^Cmd.*',
modules: list[str] | None = None,
eloop: None = None,
) -> None:
def add_cmd_to_parser(cmd, parsers): def add_cmd_to_parser(cmd, parsers):
parser = parsers.add_parser( parser = parsers.add_parser(
cmd.name, cmd.name,
help = cmd.help, help = cmd.help,
description = cmd.description, description = cmd.description,
formatter_class = argparse.ArgumentDefaultsHelpFormatter, formatter_class = ArgumentDefaultsHelpFormatter,
) )
parser.set_defaults(func=cmd.run) parser.set_defaults(func = cmd.run)
cmd.add_arguments(parser) cmd.add_arguments(parser)
cmd.set_parser(parser) cmd.set_parser(parser)
return parser return parser
def add_cmds_to_parser(parent, parser, cmds, all=False): def add_cmds_to_parser(
parent: AbstractCmd | App,
parser: ArgumentParser,
cmds,
all = False
) -> None:
if not cmds: if not cmds:
return return
class SubCommand: class SubCommand:
def __init__(self, cmd: Cmd, parser: Any):
def __init__(self, cmd: AbstractCmd, parser: Any):
self.cmd = cmd self.cmd = cmd
self.parser = parser self.parser = parser
title = 'Available subcommands' title = 'Available subcommands'
if hasattr(parent, 'name'): if hasattr(parent, 'name'):
title += ' of ' + getattr(parent, 'name') title += ' of ' + getattr(parent, 'name')
subparsers = parser.add_subparsers(title=title, metavar='', dest='command') subparsers = parser.add_subparsers(
title = title, metavar = '', dest = 'command'
)
scs: dict[str, SubCommand] = {} scs: dict[str, SubCommand] = {}
for cmd in cmds: for cmd in cmds:
cmd.set_parent(parent) cmd.set_parent(parent)
scs[cmd.name] = SubCommand(cmd, add_cmd_to_parser(cmd, subparsers)) scs[cmd.name] = SubCommand(cmd, add_cmd_to_parser(cmd, subparsers))
if all: if all:
for sc in scs.values(): for sc in scs.values():
add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all=all) add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all = all)
return return
args, unknown = self.__parser.parse_known_args() args, unknown = self.__parser.parse_known_args()
if args.command in scs: if args.command in scs:
sc = scs[args.command] sc = scs[args.command]
add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all=all) add_cmds_to_parser(sc.cmd, sc.parser, sc.cmd.children, all = all)
from .Cmd import Cmd from .Cmd import AbstractCmd
self.__args: Namespace|None = None self.__args: Namespace | None = None
self.__cmdline: str|None = None self.__cmdline: str | None = None
self.__default_log_flags: str = os.getenv('JW_DEFAULT_LOG_FLAGS', default='stderr,position,prio,color') self.__default_log_flags: str = os.getenv(
self.__default_log_level: str|int|None = os.getenv('JW_DEFAULT_LOG_LEVEL', default=NOTICE) 'JW_DEFAULT_LOG_FLAGS', default = 'stderr,position,prio,color'
self.__default_log_file: str|None = os.getenv('JW_DEFAULT_LOG_FILE', default=None) )
backtrace: str|bool = os.getenv('JW_DEFAULT_SHOW_BACKTRACE', False) self.__default_log_level: str | int | None = os.getenv(
self.__back_trace = True if isinstance(backtrace, str) and backtrace.lower() in ['1', 'true'] else False 'JW_DEFAULT_LOG_LEVEL', default = NOTICE
)
self.__default_log_file: str | None = os.getenv(
'JW_DEFAULT_LOG_FILE', default = None
)
backtrace: str | bool = os.getenv('JW_DEFAULT_SHOW_BACKTRACE', False)
self.__back_trace = isinstance(backtrace, str) and backtrace.lower() in {
'1',
'true',
}
set_log_flags(self.__default_log_flags) set_log_flags(self.__default_log_flags)
set_log_level(self.__default_log_level) set_log_level(self.__default_log_level)
@ -79,10 +122,13 @@ class App: # export
if eloop is None: if eloop is None:
self.__eloop = asyncio.get_event_loop() self.__eloop = asyncio.get_event_loop()
self.__own_eloop = True self.__own_eloop = True
self.__async_runner: AsyncRunner|None = None self.__async_runner: AsyncRunner | None = None
self.__parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter, self.__parser = ArgumentParser(
description=description, add_help=False) formatter_class = ArgumentDefaultsHelpFormatter,
description = description,
add_help = False,
)
self._add_arguments(self.__parser) self._add_arguments(self.__parser)
args, unknown = self.__parser.parse_known_args() args, unknown = self.__parser.parse_known_args()
@ -91,12 +137,26 @@ class App: # export
log(DEBUG, '-------------- Running: >' + ' '.join(sys.argv) + '<') log(DEBUG, '-------------- Running: >' + ' '.join(sys.argv) + '<')
cmd_classes = LoadTypes(modules if modules else ['__main__'], type_name_filter=name_filter, type_filter=[Cmd]) cmd_classes = LoadTypes(
add_all_parsers = '-h' in sys.argv or '--help' in sys.argv or '_ARGCOMPLETE' in os.environ modules if modules else ['__main__'],
add_cmds_to_parser(self, self.__parser, [cmd_class(self) for cmd_class in cmd_classes], all=add_all_parsers) type_name_filter = name_filter,
type_filter = [AbstractCmd], # type: ignore[type-abstract]
)
add_all_parsers = (
'-h' in sys.argv or '--help' in sys.argv or '_ARGCOMPLETE' in os.environ
)
add_cmds_to_parser(
self,
self.__parser,
[cmd_class(self) for cmd_class in cmd_classes],
all = add_all_parsers,
)
# -- Add help only now, wouldn't want to have parse_known_args() exit on --help with subcommands missing # -- Add help only now, wouldn't want to have parse_known_args() exit
self.__parser.add_argument('-h', '--help', action='help', help='Show this help message and exit') # on --help with subcommands missing
self.__parser.add_argument(
'-h', '--help', action = 'help', help = 'Show this help message and exit'
)
def __del__(self): def __del__(self):
if self.__own_eloop: if self.__own_eloop:
@ -105,24 +165,32 @@ class App: # export
self.__eloop = None self.__eloop = None
self.__own_eloop = False self.__own_eloop = False
async def __aenter__(self) ->None: async def __aenter__(self) -> None:
return self pass
async def __aexit__(self, exc_type, exc, tb) -> None: async def __aexit__(self, exc_type, exc, tb) -> None:
pass pass
async def __run(self, argv=None) -> None: async def __run(self, argv = None) -> None:
try: try:
class NoopCompleter: # Import argcomplete only here to not require it to be compatible
# with minimal environments
from argcomplete.completers import BaseCompleter
class NoopCompleter(BaseCompleter):
def __call__(self, **kwargs): def __call__(self, **kwargs):
return () return ()
import argcomplete # Don't require it to be compatible with minimal environments
argcomplete.autocomplete(self.__parser, default_completer=NoopCompleter()) import argcomplete
except:
argcomplete.autocomplete(self.__parser, default_completer = NoopCompleter())
except Exception:
pass pass
self.__args = self.__parser.parse_args(args=argv) self.__args = self.__parser.parse_args(args = argv)
set_log_flags(self.__args.log_flags) set_log_flags(self.__args.log_flags)
set_log_level(self.__args.log_level) set_log_level(self.__args.log_level)
@ -152,14 +220,17 @@ class App: # export
finally: finally:
if pr is not None: if pr is not None:
pr.disable() pr.disable()
log(NOTICE, f'Writing profile statistics to {self.__args.write_profile}') log(
NOTICE,
f'Writing profile statistics to {self.__args.write_profile}'
)
pr.dump_stats(self.__args.write_profile) pr.dump_stats(self.__args.write_profile)
if exit_status: if exit_status:
sys.exit(exit_status) sys.exit(exit_status)
# Run sub-command. Overwrite if you want to do anything before or after # Run sub-command. Overwrite if you want to do anything before or after
async def _run(self, args: argparse.Namespace) -> None: async def _run(self, args: Namespace) -> None | int:
return await self.args.func(args) return await self.args.func(args)
def call_async(self, awaitable: Awaitable[T], timeout: float | None = None) -> T: def call_async(self, awaitable: Awaitable[T], timeout: float | None = None) -> T:
@ -167,6 +238,8 @@ class App: # export
@property @property
def eloop(self) -> asyncio.AbstractEventLoop: def eloop(self) -> asyncio.AbstractEventLoop:
if self.__eloop is None:
raise Exception('Tried to get inexistent event loop from application')
return self.__eloop return self.__eloop
@property @property
@ -178,28 +251,34 @@ class App: # export
@property @property
def cmdline(self) -> str: def cmdline(self) -> str:
if self.__cmdline is None: if self.__cmdline is None:
import shlex
with open('/proc/self/cmdline', 'rb') as f: with open('/proc/self/cmdline', 'rb') as f:
raw = f.read().split(b'\0')[:-1] raw = f.read().split(b'\0')[:-1]
self.__cmdline = ' '.join(shlex.quote(arg.decode()) for arg in raw) self.__cmdline = ' '.join(shlex.quote(arg.decode()) for arg in raw)
return self.__cmdline return self.__cmdline
@property @property
def args(self) -> argparse.Namespace: def args(self) -> Namespace:
if self.__args is None:
raise Exception('Tried to get inexistent argument list from application')
return self.__args return self.__args
@property @property
def parser(self) -> argparse.ArgumentParser: def parser(self) -> ArgumentParser:
return self.__parser return self.__parser
def run(self, argv=None) -> None: def run(self, argv = None) -> None:
try: try:
ret = self.__eloop.run_until_complete(self.__run(argv)) # type: ignore ret = self.eloop.run_until_complete(self.__run(argv))
finally: finally:
if self.__async_runner: if self.__async_runner:
self.__async_runner.close() self.__async_runner.close()
self.__async_runner = None self.__async_runner = None
return ret return ret
def run_sub_commands(description = '', name_filter = '^Cmd.*', modules=None, argv=None): # export def run_sub_commands(
with App(description, name_filter, modules) as app: description = '', name_filter = '^Cmd.*', modules = None, argv = None
return app.run(argv=argv) ): # export
app = App(description, name_filter, modules)
return app.run(argv = argv)

16
src/python/jw/pkg/lib/AsyncRunner.py
View file

@ -1,10 +1,13 @@
# -*- coding: utf-8 -*- from __future__ import annotations
import asyncio
import concurrent.futures
import contextlib
import abc, asyncio, contextlib, concurrent.futures
from collections.abc import Awaitable, Generator from collections.abc import Awaitable, Generator
from typing import TypeVar from typing import TypeVar
T = TypeVar("T") T = TypeVar('T')
@contextlib.contextmanager @contextlib.contextmanager
def loop_in_thread() -> Generator[asyncio.AbstractEventLoop, None, None]: def loop_in_thread() -> Generator[asyncio.AbstractEventLoop, None, None]:
@ -19,8 +22,7 @@ def loop_in_thread() -> Generator[asyncio.AbstractEventLoop, None, None]:
loop_fut.set_result(asyncio.get_running_loop()) loop_fut.set_result(asyncio.get_running_loop())
await stop_event.wait() await stop_event.wait()
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as tpe: with concurrent.futures.ThreadPoolExecutor(max_workers = 1) as tpe:
complete_fut = tpe.submit(asyncio.run, main()) complete_fut = tpe.submit(asyncio.run, main())
for fut in concurrent.futures.as_completed((loop_fut, complete_fut)): for fut in concurrent.futures.as_completed((loop_fut, complete_fut)):
@ -40,13 +42,13 @@ class AsyncRunner:
self._loop = self._cm.__enter__() self._loop = self._cm.__enter__()
def call(self, awaitable: Awaitable[T], timeout: float | None = None) -> T: def call(self, awaitable: Awaitable[T], timeout: float | None = None) -> T:
fut = asyncio.run_coroutine_threadsafe(awaitable, self._loop) fut = asyncio.run_coroutine_threadsafe(awaitable, self._loop) # type: ignore
return fut.result(timeout) return fut.result(timeout)
def close(self) -> None: def close(self) -> None:
self._cm.__exit__(None, None, None) self._cm.__exit__(None, None, None)
def __enter__(self) -> "AsyncRunner": def __enter__(self) -> AsyncRunner:
return self return self
def __exit__(self, exc_type, exc, tb) -> None: def __exit__(self, exc_type, exc, tb) -> None:

162
src/python/jw/pkg/lib/Cmd.py
View file

@ -1,38 +1,44 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import Any import abc
import sys
import inspect, sys, re, abc, argparse from argparse import ArgumentParser
from argparse import ArgumentParser, _SubParsersAction from typing import TYPE_CHECKING
from .log import * from .log import ERR
from .Types import Types, LoadTypes from .Types import LoadTypes, Types
class Cmd(abc.ABC): # export if TYPE_CHECKING:
from typing import Any
def __init__(self, parent: App|Cmd, name: str, help: str, description: str|None=None) -> None: from .App import App
from . import App
self.__parent: App|Cmd|None = parent class AbstractCmd(abc.ABC):
self.__app: App|None = None
self.__name = name def __init__(
self.__help = help self,
self.__description = description if description else help parent: App | AbstractCmd,
) -> None:
self.__parent: App | AbstractCmd | None = parent
self.__app: App | None = None
self.__children: list[Cmd] = [] self.__children: list[Cmd] = []
self.__child_classes: list[type[Cmd]] = [] self.__child_classes: list[type[Cmd]] = []
self.__parser: ArgumentParser|None = None self.__parser: ArgumentParser | None = None
@abc.abstractmethod def set_parent(self, parent: Any | Cmd):
async def _run(self, args) -> None:
if isinstance(self.__parent, Cmd): # Calling App.run() would loop
return await self.__parent._run(args)
def set_parent(self, parent: Any|Cmd):
self.__parent = parent self.__parent = parent
@property @property
def parent(self) -> App|Cmd: def name(self) -> str:
return self._name()
@abc.abstractmethod
def _name(self) -> str:
raise NotImplementedError('Called pure virtual base class method')
@property
def parent(self) -> App | AbstractCmd:
if self.__parent is None: if self.__parent is None:
raise Exception(f'Tried to access inexistent parent of command {self.name}') raise Exception(f'Tried to access inexistent parent of command {self.name}')
return self.__parent return self.__parent
@ -40,55 +46,45 @@ class Cmd(abc.ABC): # export
@property @property
def app(self) -> App: def app(self) -> App:
from .App import App from .App import App
if self.__app is None: if self.__app is None:
parent = self.__parent parent = self.__parent
while True: while True:
if parent is None: if parent is None:
raise Exception("Can't get application object from command without parent") raise Exception(
"Can't get application object from command without parent"
)
if isinstance(parent, App): if isinstance(parent, App):
self.__app = parent self.__app = parent
break break
assert parent != parent.__parent, f'Assertion failed: Parent mismatch' assert parent != parent.__parent, 'Assertion failed: Parent mismatch'
parent = parent.__parent parent = parent.__parent
return self.__app return self.__app
@property
def children(self) -> tuple[Cmd, ...]:
return tuple(self.__children)
@property
def child_classes(self) -> tuple[type[Cmd], ...]:
return tuple(self.__child_classes)
@property
def parser(self) -> ArgumentParser:
if self.__parser is None:
raise Exception(f'Tried to get a non-existing parser from {self}')
return self.__parser
# Don't use a setter decorator to force using a grepable method # Don't use a setter decorator to force using a grepable method
def set_parser(self, parser: ArgumentParser): def set_parser(self, parser: ArgumentParser):
self.__parser = parser self.__parser = parser
@property def print_help(self, exit_status: int | None = None) -> None:
def parser(self) -> str:
return self.__parser
@property
def name(self) -> str:
return self.__name
@property
def help(self) -> str:
return self.__help
@property
def description(self) -> str:
return self.__description
@property
def children(self) -> list[Cmd]:
return tuple(self.__children)
@property
def child_classes(self) -> list[type[Cmd]]:
return tuple(self.__child_classes)
def print_help(self, exit_status: int|None=None) -> None:
self.parser.print_help() self.parser.print_help()
if exit_status is not None: if exit_status is not None:
sys.exit(exit_status) sys.exit(exit_status)
async def run(self, args): def add_subcommands(self, cmds: Cmd | list[Cmd] | Types | list[Types]) -> None:
return await self._run(args)
def add_subcommands(self, cmds: Cmd|list[Cmds]|Types|list[Types]) -> None:
if isinstance(cmds, Cmd): if isinstance(cmds, Cmd):
assert False assert False
return return
@ -106,21 +102,73 @@ class Cmd(abc.ABC): # export
self.__children.append(cmd) self.__children.append(cmd)
assert len(self.__children) == len(self.__child_classes) assert len(self.__children) == len(self.__child_classes)
except Exception as e: except Exception as e:
cmds.dump(ERR, f"Failed to add subcommands ({str(e)})") cmds.dump(ERR, f'Failed to add subcommands ({str(e)})')
raise raise
return return
raise Exception(f'Tried to add sub-commands of unknown type {type(cmds)}') raise Exception(f'Tried to add sub-commands of unknown type {type(cmds)}')
def load_subcommands(self, modules: str|list[str]|None=None, name_filter: str=r'Cmd[^.]') -> None: def load_subcommands(
self,
modules: str | list[str] | None = None,
name_filter: str = r'Cmd[^.]'
) -> None:
if modules is None: if modules is None:
# Derive module search path for the calling module's subcommands # Derive module search path for the calling module's subcommands
# from the module path of the calling module itself # from the module path of the calling module itself
modules = [type(self).__module__.replace('Cmd', '').lower()] modules = [type(self).__module__.replace('Cmd', '').lower()]
elif isinstance(modules, str): elif isinstance(modules, str):
modules = [modules] modules = [modules]
self.add_subcommands(LoadTypes(modules, type_name_filter=name_filter)) self.add_subcommands(LoadTypes(modules, type_name_filter = name_filter))
# -- Interface to derived classes
# To be overridden by derived class in case the command does take arguments. # To be overridden by derived class in case the command does take arguments.
# Will be called from App base class constructor and set up the parser hierarchy # Will be called from App base class constructor and set up the parser hierarchy
def add_arguments(self, parser: ArgumentParser) -> None: def add_arguments(self, parser: ArgumentParser) -> None:
pass pass
@abc.abstractmethod
async def _run(self, args) -> None:
if isinstance(self.__parent, Cmd): # Calling App.run() would loop
return await self.__parent._run(args)
async def run(self, args):
return await self._run(args)
@abc.abstractmethod
def _help(self) -> str:
raise NotImplementedError('Called pure virtual base class method')
@property
def help(self) -> str:
return self._help()
def _description(self) -> str:
raise NotImplementedError('Called pure virtual base class method')
@property
def description(self) -> str:
return self._description()
class Cmd(AbstractCmd): # export
def __init__(
self,
parent: App | Cmd,
name: str,
help: str,
description: str | None = None
) -> None:
super().__init__(parent)
self.__name = name
self.__help = help
self.__description = description if description else help
def _name(self) -> str:
return self.__name
def _help(self) -> str:
return self.__help
def _description(self) -> str:
return self.__description

53
src/python/jw/pkg/lib/CopyContext.py
View file

@ -1,32 +1,47 @@
# -*- coding: utf-8 -*-
from typing import Self from typing import Self
from .FileContext import FileContext from .FileContext import FileContext
from .Uri import Uri
class CopyContext: class CopyContext:
def __init__(self, src: str|FileContext, dst: str|FileContext, chroot=False) -> None: def __init__(
if isinstance(src, FileContext): self,
self.__src = src src: Uri | str | FileContext,
self.__src_uri = src.uri dst: Uri | str | FileContext,
else: chroot = False
self.__src: FileContext|None = None ) -> None:
self.__src_uri = src
if isinstance(dst, FileContext): def __uri(ctx: FileContext | Uri | str) -> Uri | str | None:
self.__dst = dst if ctx is None:
self.__dst_uri = dst.uri return None
else: if isinstance(ctx, Uri):
self.__dst: FileContext|None = None return ctx
self.__dst_uri = dst if isinstance(ctx, str):
return ctx
assert isinstance(ctx, FileContext)
return ctx.uri
def __info(
ctx: FileContext | Uri | str,
) -> tuple[FileContext | None, str | Uri | None]:
fc: FileContext | None = ctx if isinstance(ctx, FileContext) else None
return fc, __uri(ctx)
self.__src, self.__src_uri = __info(src)
self.__dst, self.__dst_uri = __info(dst)
self.__chroot = chroot self.__chroot = chroot
async def __aenter__(self) -> Self: async def __aenter__(self) -> Self:
if self.__src is None: if self.__src is None:
self.__src = FileContext.create(self.__src_uri, chroot=self.__chroot) if self.__src_uri is None:
raise Exception('Tried to create source context without URI')
self.__src = FileContext.create(self.__src_uri, chroot = self.__chroot)
await self.__src.open() await self.__src.open()
if self.__dst is None: if self.__dst is None:
self.__dst = FileContext.create(self.__dst_uri, chroot=self.__chroot) if self.__dst_uri is None:
raise Exception('Tried to create destination context without URI')
self.__dst = FileContext.create(self.__dst_uri, chroot = self.__chroot)
await self.__dst.open() await self.__dst.open()
return self return self
@ -40,10 +55,14 @@ class CopyContext:
@property @property
def src(self) -> FileContext: def src(self) -> FileContext:
if self.__src is None:
raise Exception('Tried to access inexistent source context')
return self.__src return self.__src
@property @property
def dst(self) -> FileContext: def dst(self) -> FileContext:
if self.__dst is None:
raise Exception('Tried to access inexistent destination context')
return self.__dst return self.__dst
async def _run(self) -> None: async def _run(self) -> None:

222
src/python/jw/pkg/lib/Distro.py
View file

@ -1,38 +1,41 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING import abc
import importlib
import re
import sys
from functools import cached_property from functools import cached_property
from typing import TYPE_CHECKING
from .log import ERR, INFO, WARNING, log
if TYPE_CHECKING: if TYPE_CHECKING:
import Iterable from typing import Iterable
from .base import InputMode, Result
from .ExecContext import ExecContext
from .Package import Package
from .PackageFilter import PackageFilter from .PackageFilter import PackageFilter
import abc, importlib, re
from .PackageFilter import PackageFilter
from .ExecContext import ExecContext
from .base import Result, InputMode
from .Package import Package
from .log import *
class Distro(abc.ABC): class Distro(abc.ABC):
def __init__( def __init__(
self, self,
ec: ExecContext, ec: ExecContext,
id: str|None=None, id: str | None = None,
os_release_str: str|None=None, os_release_str: str | None = None,
default_pkg_filter: PackageFilter|None=None, default_pkg_filter: PackageFilter | None = None,
) -> None: ) -> None:
if id is None: if id is None:
raise ValueError(f'Tried to instaniate Distro without id') raise ValueError('Tried to instaniate Distro without id')
if ec is None: if ec is None:
raise ValueError(f'Tried to instaniate Distro "{id}" without execution context') raise ValueError(
f'Tried to instaniate Distro "{id}" without execution context'
)
self.__exec_context = ec self.__exec_context = ec
self.__id: str|None = None self.__id: str | None = None
self.__os_release_str: str|None = os_release_str self.__os_release_str: str | None = os_release_str
self.__default_pkg_filter = default_pkg_filter self.__default_pkg_filter = default_pkg_filter
# Names that can be used by code outside this class to retrieve # Names that can be used by code outside this class to retrieve
@ -52,47 +55,60 @@ class Distro(abc.ABC):
# == Load # == Load
@classmethod @classmethod
async def read_os_release_str(cls, ec: ExecContext) -> None: async def read_os_release_str(cls, ec: ExecContext) -> str:
release_file = '/etc/os-release' release_file = '/etc/os-release'
try: try:
result = await ec.get(release_file, throw=True) result = await ec.get(release_file, throw = True)
ret = result.stdout.decode().strip() return result.stdout_str
except Exception as e: except Exception as e:
log(INFO, f'Failed to read {release_file} ({str(e)}), falling back to uname') log(
INFO,
f'Failed to read {release_file} ({str(e)}), falling back to uname'
)
result = await ec.run( result = await ec.run(
['uname', '-s'], ['uname', '-s'], throw = False, cmd_input = InputMode.NonInteractive
throw=False, )
cmd_input=InputMode.NonInteractive
)
if result.status != 0: if result.status != 0:
log(ERR, f'/etc/os-release and uname both failed, the latter with exit status {result.status}') log(
ERR,
(
'/etc/os-release and uname both failed, '
f'the latter with {result.summary}'
),
)
raise raise
uname = result.decode().stdout.strip().lower() uname = result.stdout_str.lower()
ret = f'ID={uname}\nVERSION_CODENAME=unknown' ret = f'ID={uname}\nVERSION_CODENAME=unknown'
return ret return ret
@classmethod @classmethod
def parse_os_release_field(self, key: str, os_release_str: str, throw: bool=False) -> str: def parse_os_release_field(cls, key: str, os_release_str: str) -> str:
m = re.search(r'^\s*' + key + r'\s*=\s*("?)([^"\n]+)\1\s*$', os_release_str, re.MULTILINE) m = re.search(
r'^\s*' + key + r'\s*=\s*("?)([^"\n]+)\1\s*$', os_release_str, re.MULTILINE
)
if m is None: if m is None:
if throw: raise Exception(f'Could not read "{key}=" from /etc/os-release')
raise Exception(f'Could not read "{key}=" from /etc/os-release')
return None
return m.group(2) return m.group(2)
@classmethod @classmethod
def parse_os_release_field_id(cls, os_release_str: str, throw: bool=False) -> str: def parse_os_release_field_id(cls, os_release_str: str) -> str:
ret = cls.parse_os_release_field('ID', os_release_str, throw=throw) ret = cls.parse_os_release_field('ID', os_release_str)
match ret: match ret:
case 'opensuse-tumbleweed': case 'opensuse-tumbleweed':
return 'suse' return 'suse'
return ret return ret
@classmethod @classmethod
async def instantiate(cls, ec: ExecContext, *args, id: str|None=None, os_release_str: str|None=None, **kwargs): async def instantiate(
cls,
ec: ExecContext,
id: str | None = None,
os_release_str: str | None = None,
**kwargs,
):
if id is None: if id is None:
os_release_str = await cls.read_os_release_str(ec) os_release_str = await cls.read_os_release_str(ec)
id = cls.parse_os_release_field_id(os_release_str, throw=True) id = cls.parse_os_release_field_id(os_release_str)
backend_id = id.lower().replace('-', '_') backend_id = id.lower().replace('-', '_')
match backend_id: match backend_id:
case 'ubuntu' | 'raspbian' | 'kali': case 'ubuntu' | 'raspbian' | 'kali':
@ -108,11 +124,11 @@ class Distro(abc.ABC):
log(ERR, f'Failed to import Distro module {module_path} ({str(e)})') log(ERR, f'Failed to import Distro module {module_path} ({str(e)})')
raise raise
cls = getattr(module, 'Distro') cls = getattr(module, 'Distro')
ret = cls(ec, *args, id=id, os_release_str=os_release_str, **kwargs) ret = cls(ec, id = id, os_release_str = os_release_str, **kwargs)
return ret return ret
def os_release_field(self, key: str, throw: bool=False) -> str: def os_release_field(self, key: str) -> str:
return self.parse_os_release_field(key, self.os_release_str, throw) return self.parse_os_release_field(key, self.os_release_str)
async def cache(self) -> None: async def cache(self) -> None:
if self.__os_release_str is None: if self.__os_release_str is None:
@ -120,10 +136,12 @@ class Distro(abc.ABC):
@cached_property @cached_property
def os_cascade(self) -> list[str]: def os_cascade(self) -> list[str]:
def __append(entry: str): def __append(entry: str):
if not entry in ret: if entry not in ret:
ret.append(entry) ret.append(entry)
ret = [ 'os' ]
ret = ['os']
match self.id: match self.id:
case 'centos': case 'centos':
__append('linux') __append('linux')
@ -177,27 +195,32 @@ class Distro(abc.ABC):
@property @property
def os_release_str(self) -> str: def os_release_str(self) -> str:
if self.__os_release_str is None: if self.__os_release_str is None:
raise Exception(f'Tried to access OS release from an incompletely loaded Distro instance. Call reacache() before') raise Exception(
'Tried to access OS release from an incompletely loaded Distro '
'instance. Call cache() before'
)
return self.__os_release_str return self.__os_release_str
@cached_property @cached_property
def name(self) -> str: def name(self) -> str:
return self.os_release_field('NAME', throw=True) return self.os_release_field('NAME')
@cached_property @cached_property
def id(self) -> str: def id(self) -> str:
return self.parse_os_release_field_id(self.__os_release_str, throw=True) return self.parse_os_release_field_id(self.os_release_str)
@cached_property @cached_property
def codename(self) -> str: def codename(self) -> str:
match self.id: match self.id:
case 'suse': case 'suse':
return self.os_release_field('ID', throw=True).split('-')[1] return self.os_release_field('ID').split('-')[1]
case 'kali': case 'kali':
return self.os_release_field('VERSION_CODENAME', throw=True).split('-')[1] return self.os_release_field('VERSION_CODENAME').split('-')[1]
case _: case _:
return self.os_release_field('VERSION_CODENAME', throw=True) return self.os_release_field('VERSION_CODENAME')
raise NotImplementedError(f'Can\'t determine code name from distribution ID {self.id}') raise NotImplementedError(
f"Can't determine code name from distribution ID {self.id}"
)
@cached_property @cached_property
def os(self) -> str: def os(self) -> str:
@ -214,33 +237,35 @@ class Distro(abc.ABC):
@cached_property @cached_property
def gnu_triplet(self) -> str: def gnu_triplet(self) -> str:
import sysconfig
import shutil import shutil
import subprocess import subprocess
import sysconfig
# Best: GNU host triplet Python was built for # Best: GNU host triplet Python was built for
for key in ("HOST_GNU_TYPE", "BUILD_GNU_TYPE"): # BUILD_GNU_TYPE can exist too for key in ('HOST_GNU_TYPE', 'BUILD_GNU_TYPE'): # BUILD_GNU_TYPE can exist too
ret = sysconfig.get_config_var(key) ret = sysconfig.get_config_var(key)
if isinstance(ret, str) and ret: if isinstance(ret, str) and ret:
return ret return ret
# Common on Debian/Ubuntu: multiarch component (often looks like a triplet) # Common on Debian/Ubuntu: multiarch component (often looks like a triplet)
ret = sysconfig.get_config_var("MULTIARCH") ret = sysconfig.get_config_var('MULTIARCH')
if isinstance(ret, str) and ret: if isinstance(ret, str) and ret:
return ret return ret
# Sometimes exposed (privately) by CPython # Sometimes exposed (privately) by CPython
ret = getattr(sys.implementation, "_multiarch", None) ret = getattr(sys.implementation, '_multiarch', None)
if isinstance(ret, str) and ret: if isinstance(ret, str) and ret:
return ret return ret
# Last resort: ask the system compiler # Last resort: ask the system compiler
for cc in ("gcc", "cc", "clang"): for cc in ('gcc', 'cc', 'clang'):
path = shutil.which(cc) path = shutil.which(cc)
if not path: if not path:
continue continue
try: try:
ret = subprocess.check_output([path, "-dumpmachine"], text=True, stderr=subprocess.DEVNULL).strip() ret = subprocess.check_output(
[path, '-dumpmachine'], text = True, stderr = subprocess.DEVNULL
).strip()
if ret: if ret:
return ret return ret
except Exception: except Exception:
@ -252,14 +277,21 @@ class Distro(abc.ABC):
def macros(cls) -> list[str]: def macros(cls) -> list[str]:
return ['%%{' + name + '}' for name in cls.macro_names] return ['%%{' + name + '}' for name in cls.macro_names]
def expand_macros(self, fmt: str|Iterable) -> str|Iterable: def expand_macros(self, fmt: str | Iterable) -> str | list[str]:
ret: str | list[str]
if not isinstance(fmt, str): if not isinstance(fmt, str):
ret: list[str] = [] ret = []
for entry in fmt: for entry in fmt:
ret.append(self.expand_macros(entry)) rv = self.expand_macros(entry)
if isinstance(rv, str):
ret.append(rv)
continue
raise NotImplementedError(
f'Expanding macros in nested lists is not supported: {rv}'
)
return ret return ret
ret = fmt ret = fmt
for macro in re.findall("%{([A-Za-z_-]+)}", fmt): for macro in re.findall('%{([A-Za-z_-]+)}', fmt):
try: try:
name = macro.replace('-', '_') name = macro.replace('-', '_')
val = getattr(self, name) val = getattr(self, name)
@ -279,7 +311,7 @@ class Distro(abc.ABC):
return self.__exec_context return self.__exec_context
@property @property
def default_pkg_filter(self) -> str: def default_pkg_filter(self) -> PackageFilter | None:
return self.__default_pkg_filter return self.__default_pkg_filter
async def run(self, *args, **kwargs) -> Result: async def run(self, *args, **kwargs) -> Result:
@ -289,7 +321,7 @@ class Distro(abc.ABC):
return await self.__exec_context.sudo(*args, **kwargs) return await self.__exec_context.sudo(*args, **kwargs)
@property @property
def interactive(self) -> bool: def interactive(self) -> bool | None:
return self.__exec_context.interactive return self.__exec_context.interactive
# == Distribution abstraction methods # == Distribution abstraction methods
@ -309,8 +341,8 @@ class Distro(abc.ABC):
async def _dup(self, download_only: bool) -> None: async def _dup(self, download_only: bool) -> None:
pass pass
async def dup(self, download_only: bool=False) -> None: async def dup(self, download_only: bool = False) -> None:
return await self._dup(download_only=download_only) return await self._dup(download_only = download_only)
# -- reboot_required # -- reboot_required
@ -318,10 +350,10 @@ class Distro(abc.ABC):
async def _reboot_required(self, verbose: bool) -> bool: async def _reboot_required(self, verbose: bool) -> bool:
pass pass
async def reboot_required(self, verbose: bool|None=None) -> bool: async def reboot_required(self, verbose: bool | None = None) -> bool:
if verbose is None: if verbose is None:
verbose = self.ctx.verbose_default verbose = self.ctx.verbose_default
return await self._reboot_required(verbose=verbose) return await self._reboot_required(verbose = verbose)
# -- select # -- select
@ -329,11 +361,16 @@ class Distro(abc.ABC):
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]: async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
pass pass
async def _select(self, names: Iterable[str], filter: PackageFilter) -> Iterable[Package]: async def _select(self, names: Iterable[str],
assert filter, "No filter in _select()" filter: PackageFilter) -> Iterable[Package]:
assert filter, 'No filter in _select()'
return [p for p in await self._select_by_name(names) if filter.match(p)] return [p for p in await self._select_by_name(names) if filter.match(p)]
async def select(self, names: Iterable[str] = [], filter: PackageFilter|None=None) -> Iterable[Package]: async def select(
self,
names: Iterable[str] = [],
filter: PackageFilter | None = None
) -> Iterable[Package]:
if not filter: if not filter:
filter = self.__default_pkg_filter filter = self.__default_pkg_filter
if not filter: if not filter:
@ -349,17 +386,28 @@ class Distro(abc.ABC):
# Default implementation assumes package manager can handle local files. # Default implementation assumes package manager can handle local files.
# Not true for all distros. Override if Distro knows better. # Not true for all distros. Override if Distro knows better.
async def _install_local_files(self, paths: Iterable[str], only_update: bool) -> None: async def _install_local_files(
await self._install(paths, only_update=only_update) self, paths: Iterable[str], only_update: bool
) -> None:
await self._install(paths, only_update = only_update)
# Download first and then install. Override if Distro knows better. # Download first and then install. Override if Distro knows better.
async def _install_urls(self, urls: Iterable[str], only_update: bool) -> None: async def _install_urls(self, urls: Iterable[str], only_update: bool) -> None:
from .util import copy from .util import copy
tmp: str|None = None
tmp: str | None = None
try: try:
tmp = await self.__exec_context.mktemp('/tmp/jw-pkg-XXXXXX', directory=True) tmp = await self.__exec_context.mktemp(
paths = await copy(urls, self.__exec_context.uri.scheme_plus_authority + tmp) '/tmp/jw-pkg-XXXXXX', directory = True
await self._install_local_files(paths, only_update=only_update) )
paths = await copy(
urls, self.__exec_context.uri.scheme_plus_authority + tmp
)
if isinstance(paths, Exception):
raise paths
if isinstance(paths, str):
paths = [paths]
await self._install_local_files(paths, only_update = only_update)
finally: finally:
if tmp is not None: if tmp is not None:
await self.__exec_context.erase(tmp) await self.__exec_context.erase(tmp)
@ -368,7 +416,9 @@ class Distro(abc.ABC):
# - Download URLs into local directories and install # - Download URLs into local directories and install
# - Pass names to package manager # - Pass names to package manager
# Override if Distro knows better. # Override if Distro knows better.
async def _install_urls_and_names(self, packages: Iterable[str], only_update: bool) -> None: async def _install_urls_and_names(
self, packages: Iterable[str], only_update: bool
) -> None:
urls: list[str] = [] urls: list[str] = []
names: list[str] = [] names: list[str] = []
for package in packages: for package in packages:
@ -380,15 +430,15 @@ class Distro(abc.ABC):
continue continue
names.append(package) names.append(package)
if urls: if urls:
await self._install_urls(urls, only_update=only_update) await self._install_urls(urls, only_update = only_update)
if names: if names:
await self._install(names, only_update=only_update) await self._install(names, only_update = only_update)
async def install(self, names: Iterable[str], only_update: bool=False) -> None: async def install(self, names: Iterable[str], only_update: bool = False) -> None:
if not names: if not names:
log(WARNING, f'No packages specified for installation') log(WARNING, 'No packages specified for installation')
return return
await self._install_urls_and_names(names, only_update=only_update) await self._install_urls_and_names(names, only_update = only_update)
# -- delete # -- delete
@ -398,7 +448,7 @@ class Distro(abc.ABC):
async def delete(self, names: Iterable[str]) -> None: async def delete(self, names: Iterable[str]) -> None:
if not names: if not names:
log(WARNING, f'No packages specified for deletion') log(WARNING, 'No packages specified for deletion')
return return
return await self._delete(names) return await self._delete(names)
@ -410,6 +460,6 @@ class Distro(abc.ABC):
async def pkg_files(self, name: str) -> Iterable[str]: async def pkg_files(self, name: str) -> Iterable[str]:
if not name: if not name:
log(WARNING, f'No package specified for inspection') log(WARNING, 'No package specified for inspection')
return [] return []
return await self._pkg_files(name) return await self._pkg_files(name)

463
src/python/jw/pkg/lib/ExecContext.py
View file

@ -1,42 +1,43 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
import abc, re, sys, errno import abc
from enum import Enum, auto import errno
from typing import NamedTuple, TYPE_CHECKING import sys
from decimal import Decimal, ROUND_FLOOR
from decimal import ROUND_FLOOR, Decimal
from typing import TYPE_CHECKING, NamedTuple
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Self, Type from typing import Type
from types import TracebackType from types import TracebackType
from .log import *
from .base import Input, InputMode, Result, StatResult from .base import Input, InputMode, Result, StatResult
from .FileContext import FileContext as Base from .FileContext import FileContext as Base
from .log import DEBUG, ERR, NOTICE, log
_US = "\x1f" # unlikely to appear in numeric output _US = '\x1f' # unlikely to appear in numeric output
_BILLION = Decimal(1_000_000_000) _BILLION = Decimal(1_000_000_000)
def _looks_like_option_error(stderr: str) -> bool: def _looks_like_option_error(stderr: str | None) -> bool:
if stderr is None:
return False
s = stderr.lower() s = stderr.lower()
return any( return any(
needle in s needle in s for needle in (
for needle in ( 'unrecognized option',
"unrecognized option", 'illegal option',
"illegal option", 'unknown option',
"unknown option", 'invalid option',
"invalid option", 'option requires an argument', )
"option requires an argument",
)
) )
def _raise_stat_error(path: str, stderr: str, returncode: int) -> None: def _raise_stat_error(path: str, result: Result) -> None:
msg = (stderr or "").strip() or f"stat exited with status {returncode}" stderr = result.stderr_str_or_none or f'stat exited with status {result.status}'
msg = stderr.strip()
lower = msg.lower() lower = msg.lower()
if "no such file" in lower: if 'no such file' in lower:
raise FileNotFoundError(errno.ENOENT, msg, path) raise FileNotFoundError(errno.ENOENT, msg, path)
if "permission denied" in lower or "operation not permitted" in lower: if 'permission denied' in lower or 'operation not permitted' in lower:
raise PermissionError(errno.EACCES, msg, path) raise PermissionError(errno.EACCES, msg, path)
raise OSError(errno.EIO, msg, path) raise OSError(errno.EIO, msg, path)
@ -46,14 +47,14 @@ def _parse_epoch(value: str) -> tuple[int, float, int]:
(integer seconds for tuple slot, float seconds for attribute, ns for *_ns) (integer seconds for tuple slot, float seconds for attribute, ns for *_ns)
""" """
dec = Decimal(value.strip()) dec = Decimal(value.strip())
sec = int(dec.to_integral_value(rounding=ROUND_FLOOR)) sec = int(dec.to_integral_value(rounding = ROUND_FLOOR))
ns = int((dec * _BILLION).to_integral_value(rounding=ROUND_FLOOR)) ns = int((dec * _BILLION).to_integral_value(rounding = ROUND_FLOOR))
return sec, float(dec), ns return sec, float(dec), ns
def _build_stat_result(fields: list[str], mode_base: int) -> StatResult: def _build_stat_result(fields: list[str], mode_base: int) -> StatResult:
if len(fields) != 13: if len(fields) != 13:
raise ValueError( raise ValueError(
f"unexpected stat output: expected 13 fields, got {len(fields)}: {fields!r}" f'unexpected stat output: expected 13 fields, got {len(fields)}: {fields!r}'
) )
( (
@ -73,9 +74,9 @@ def _build_stat_result(fields: list[str], mode_base: int) -> StatResult:
) = fields ) = fields
st_mode = int(mode_s, mode_base) st_mode = int(mode_s, mode_base)
st_ino = int(ino_s) # st_ino = int(ino_s)
st_dev = int(dev_s) # st_dev = int(dev_s)
st_nlink = int(nlink_s) # st_nlink = int(nlink_s)
st_uid = uid_s st_uid = uid_s
st_gid = gid_s st_gid = gid_s
st_size = int(size_s) st_size = int(size_s)
@ -84,9 +85,9 @@ def _build_stat_result(fields: list[str], mode_base: int) -> StatResult:
st_mtime_i, st_mtime_f, st_mtime_ns = _parse_epoch(mtime_s) st_mtime_i, st_mtime_f, st_mtime_ns = _parse_epoch(mtime_s)
st_ctime_i, st_ctime_f, st_ctime_ns = _parse_epoch(ctime_s) st_ctime_i, st_ctime_f, st_ctime_ns = _parse_epoch(ctime_s)
st_blksize = int(blksize_s) # st_blksize = int(blksize_s)
st_blocks = int(blocks_s) # st_blocks = int(blocks_s)
st_rdev = int(rdev_s) # st_rdev = int(rdev_s)
return StatResult( return StatResult(
mode = st_mode, mode = st_mode,
@ -103,38 +104,38 @@ class ExecContext(Base):
class CallContext: class CallContext:
def __init__( def __init__(
self, self,
parent: ExecContext, parent: ExecContext,
title: str|None, title: str | None,
cmd: list[str], cmd: list[str],
cmd_input: Input, cmd_input: Input,
mod_env: dict[str, str]|None, mod_env: dict[str, str] | None,
wd: str|None, wd: str | None,
log_prefix: str, log_prefix: str,
throw: bool, throw: bool,
verbose: bool, verbose: bool | None,
) -> None: ) -> None:
self.__cmd = cmd self.__cmd = cmd
self.__wd = wd self.__wd = wd
self.__log_prefix = log_prefix self.__log_prefix = log_prefix
self.__parent = parent self.__parent = parent
self.__title = title self.__title = title
self.__pretty_cmd: str|None = None self.__pretty_cmd: str | None = None
self.__delim = title if title is not None else f'---- {parent.uri}: Running {self.pretty_cmd} -' self.__delim = (
title if title is not None else
f'---- {parent.uri}: Running {self.pretty_cmd} -'
)
delim_len = 120 delim_len = 120
self.__delim += '-' * max(0, delim_len - len(self.__delim)) self.__delim += '-' * max(0, delim_len - len(self.__delim))
self.__mod_env = {'LC_ALL': 'C'} if mod_env is None else mod_env self.__mod_env = {'LC_ALL': 'C'} if mod_env is None else mod_env
self.__cmd_input: bytes | None = None
# -- At the end of this dance, interactive needs to be either True # -- At the end of this dance, interactive needs to be either True
# or False # or False
interactive: bool|None = None interactive: bool | None = None
if not isinstance(cmd_input, InputMode): cmd_input_bytes: None | bytes
interactive = False if isinstance(cmd_input, InputMode):
self.__cmd_input = ( cmd_input_bytes = None
cmd_input if isinstance(cmd_input, bytes) else
cmd_input.encode(sys.stdout.encoding or "utf-8")
)
else:
match cmd_input: match cmd_input:
case InputMode.Interactive: case InputMode.Interactive:
interactive = True interactive = True
@ -148,25 +149,34 @@ class ExecContext(Base):
interactive = parent.interactive interactive = parent.interactive
if interactive is None: if interactive is None:
interactive = sys.stdin.isatty() interactive = sys.stdin.isatty()
self.__cmd_input = None else:
assert interactive in [ True, False ], f'Invalid: interactive = {invalid}' interactive = False
if cmd_input is None:
cmd_input_bytes = None
elif isinstance(cmd_input, str):
cmd_input_bytes = cmd_input.encode(sys.stdout.encoding or 'utf-8')
else:
cmd_input_bytes = cmd_input
self.__cmd_input = cmd_input_bytes
assert interactive in [True, False], f'Invalid: interactive = {interactive}'
self.__interactive = interactive self.__interactive = interactive
self.__cmd_input = cmd_input if not isinstance(cmd_input, InputMode) else None
self.__throw = throw self.__throw = throw
self.__verbose = verbose if verbose is not None else parent.verbose_default self.__verbose = verbose if verbose is not None else parent.verbose_default
def __enter__(self) -> CallContext: def __enter__(self) -> ExecContext.CallContext:
self.log_delim(start=True) self.log_delim(start = True)
return self return self
def __exit__( def __exit__(
self, self,
exc_type: Type[BaseException]|None, exc_type: Type[BaseException] | None,
exc_value: BaseException|None, exc_value: BaseException | None,
traceback: TracebackType|None traceback: TracebackType | None,
) -> bool: ) -> bool:
self.log_delim(start=False) self.log_delim(start = False)
return True
@property @property
def log_prefix(self) -> str: def log_prefix(self) -> str:
@ -181,7 +191,7 @@ class ExecContext(Base):
return self.__verbose return self.__verbose
@property @property
def cmd_input(self) -> bytes|None: def cmd_input(self) -> bytes | None:
return self.__cmd_input return self.__cmd_input
@property @property
@ -193,7 +203,7 @@ class ExecContext(Base):
return self.__throw return self.__throw
@property @property
def wd(self) -> str|None: def wd(self) -> str | None:
return self.__wd return self.__wd
@property @property
@ -204,16 +214,17 @@ class ExecContext(Base):
def pretty_cmd(self) -> str: def pretty_cmd(self) -> str:
if self.__pretty_cmd is None: if self.__pretty_cmd is None:
from .util import pretty_cmd from .util import pretty_cmd
self.__pretty_cmd = pretty_cmd(self.__cmd, self.__wd) self.__pretty_cmd = pretty_cmd(self.__cmd, self.__wd)
return self.__pretty_cmd return self.__pretty_cmd
def log(prio: int, *args, **kwargs) -> None: def log(self, prio: int, *args, **kwargs) -> None:
log(prio, self.__log_prefix, *args, **kwargs) log(prio, self.__log_prefix, *args, **kwargs)
def log_delim(self, start: bool) -> None: def log_delim(self, start: bool) -> None:
if not self.__verbose: if not self.__verbose:
return None return None
if self.__interactive: # Don't log footer in interative mode if self.__interactive: # Don't log footer in interative mode
if start: if start:
log(NOTICE, self.__delim) log(NOTICE, self.__delim)
return return
@ -223,12 +234,9 @@ class ExecContext(Base):
def check_exit_code(self, result: Result) -> None: def check_exit_code(self, result: Result) -> None:
if result.status == 0: if result.status == 0:
return return
if (self.__throw or self.__verbose): if self.__throw or self.__verbose:
msg = f'Command exited with status {result.status}: {self.pretty_cmd}'
if result.stderr:
msg += ': ' + result.decode().stderr.strip()
if self.__throw: if self.__throw:
raise RuntimeError(msg) raise RuntimeError(result.summary)
def exception(self, result: Result, e: Exception) -> Result: def exception(self, result: Result, e: Exception) -> Result:
log(ERR, self.__log_prefix, f'Failed to run {self.pretty_cmd}') log(ERR, self.__log_prefix, f'Failed to run {self.pretty_cmd}')
@ -243,15 +251,35 @@ class ExecContext(Base):
def __init__(self, *args, **kwargs) -> None: def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
@classmethod
def create(cls, *args, **kwargs) -> ExecContext:
ret = super().create(*args, **kwargs)
if not isinstance(ret, cls):
raise TypeError(f'Expected {cls.__name__}, got {type(ret).__name__}')
return ret
@abc.abstractmethod
async def _run(
self,
cmd: list[str],
wd: str | None,
verbose: bool,
cmd_input: bytes | None,
mod_env: dict[str, str] | None,
interactive: bool,
log_prefix: str,
) -> Result:
raise NotImplementedError('Called pure virtual method _run()')
async def run( async def run(
self, self,
cmd: list[str], cmd: list[str],
wd: str|None = None, wd: str | None = None,
throw: bool = True, throw: bool = True,
verbose: bool|None = None, verbose: bool | None = None,
cmd_input: Input = InputMode.OptInteractive, cmd_input: Input = InputMode.OptInteractive,
mod_env: dict[str, str]|None = None, mod_env: dict[str, str] | None = None,
title: str = None title: str | None = None,
) -> Result: ) -> Result:
""" """
Run a command asynchronously and return its output Run a command asynchronously and return its output
@ -262,13 +290,16 @@ class ExecContext(Base):
throw: Raise an exception on non-zero exit status if True throw: Raise an exception on non-zero exit status if True
verbose: Emit log output while the command runs verbose: Emit log output while the command runs
cmd_input: cmd_input:
- "InputMode.OptInteractive" -> Let --interactive govern how to handle interactivity (default) - "InputMode.OptInteractive" -> Let --interactive govern how to handle
interactivity (default)
- "InputMode.Interactive" -> Inherit terminal stdin - "InputMode.Interactive" -> Inherit terminal stdin
- "InputMode.Auto" -> Inherit terminal stdin if it is a TTY - "InputMode.Auto" -> Inherit terminal stdin if it is a TTY
- "InputMode.NonInteractive" -> stdin from /dev/null - "InputMode.NonInteractive" -> stdin from /dev/null
- None -> Alias for InputMode.NonInteractive - None -> Alias for InputMode.NonInteractive
- otherwise -> Feed cmd_input to stdin - otherwise -> Feed cmd_input to stdin
mod_env: Change set to command's environment. key: val adds a variable, key: None removes it mod_env: Change set to command's environment:
- key: val adds a variable,
- key: None removes it
Returns: Returns:
A Result instance A Result instance
@ -285,8 +316,17 @@ class ExecContext(Base):
try: try:
ret = Result(None, None, 1) ret = Result(None, None, 1)
with self.CallContext(self, title=title, cmd=cmd, cmd_input=cmd_input, mod_env=mod_env, wd=wd, with self.CallContext(
log_prefix='|', throw=throw, verbose=verbose) as cc: self,
title = title,
cmd = cmd,
cmd_input = cmd_input,
mod_env = mod_env,
wd = wd,
log_prefix = '|',
throw = throw,
verbose = verbose,
) as cc:
try: try:
ret = await self._run( ret = await self._run(
cmd = cc.cmd, cmd = cc.cmd,
@ -295,7 +335,7 @@ class ExecContext(Base):
cmd_input = cc.cmd_input, cmd_input = cc.cmd_input,
mod_env = cc.mod_env, mod_env = cc.mod_env,
interactive = cc.interactive, interactive = cc.interactive,
log_prefix = cc.log_prefix log_prefix = cc.log_prefix,
) )
except Exception as e: except Exception as e:
return cc.exception(ret, e) return cc.exception(ret, e)
@ -308,11 +348,11 @@ class ExecContext(Base):
async def _sudo( async def _sudo(
self, self,
cmd: list[str], cmd: list[str],
opts: list[str]|None, opts: list[str] | None,
wd: str|None, wd: str | None,
mod_env_sudo: dict[str, str]|None, mod_env_sudo: dict[str, str] | None,
mod_env_cmd: dict[str, str]|None, mod_env_cmd: dict[str, str] | None,
cmd_input: bytes|None, cmd_input: bytes | None,
verbose: bool, verbose: bool,
interactive: bool, interactive: bool,
log_prefix: str, log_prefix: str,
@ -320,20 +360,22 @@ class ExecContext(Base):
def __check_equal_values(d1: dict[str, str], d2: dict[str, str]) -> None: def __check_equal_values(d1: dict[str, str], d2: dict[str, str]) -> None:
for key, val in d1.items(): for key, val in d1.items():
if not d2.get(key, None) in [None, val]: if d2.get(key, None) not in [None, val]:
raise ValueError(f'Outer and inner environments differ at least for {key}: "{val}" != "{d2.get(key)}"') raise ValueError(
'Outer and inner environments differ at least for '
f'{key}: "{val}" != "{d2.get(key)}"'
)
fw_cmd: list[str] = [] fw_cmd: list[str] = []
fw_env: dict[str, str] = {} fw_env: dict[str, str] = {}
if opts is None: if opts is None:
opts = {} opts = []
if mod_env_cmd: if mod_env_cmd:
fw_env.update(mod_env_cmd) fw_env.update(mod_env_cmd)
if self.username != 'root': if self.username != 'root':
if mod_env_sudo and mod_env_cmd: if mod_env_sudo and mod_env_cmd:
__check_equal_values(mod_env_sudo, mod_env_cmd) __check_equal_values(mod_env_sudo, mod_env_cmd)
__check_equal_values(mod_env_cmd, mod_env_sudo) __check_equal_values(mod_env_cmd, mod_env_sudo)
@ -345,7 +387,7 @@ class ExecContext(Base):
fw_cmd.append('--preserve-env=' + ','.join(mod_env_cmd.keys())) fw_cmd.append('--preserve-env=' + ','.join(mod_env_cmd.keys()))
if wd is not None: if wd is not None:
opts.extend('-D', wd) opts.extend(['-D', wd])
wd = None wd = None
fw_cmd.extend(opts) fw_cmd.extend(opts)
@ -361,20 +403,20 @@ class ExecContext(Base):
verbose = verbose, verbose = verbose,
cmd_input = cmd_input, cmd_input = cmd_input,
interactive = interactive, interactive = interactive,
log_prefix = log_prefix log_prefix = log_prefix,
) )
async def sudo( async def sudo(
self, self,
cmd: list[str], cmd: list[str],
opts: list[str]|None = None, opts: list[str] | None = None,
wd: str|None = None, wd: str | None = None,
mod_env_sudo: dict[str, str]|None = None, mod_env_sudo: dict[str, str] | None = None,
mod_env_cmd: dict[str, str]|None = None, mod_env_cmd: dict[str, str] | None = None,
throw: bool = True, throw: bool = True,
verbose: bool|None = None, verbose: bool | None = None,
cmd_input: Input = InputMode.OptInteractive, cmd_input: Input = InputMode.OptInteractive,
title: str = None title: str | None = None,
) -> Result: ) -> Result:
# Note that in the calls to the wrapped method, cmd_input == None can # Note that in the calls to the wrapped method, cmd_input == None can
@ -382,9 +424,17 @@ class ExecContext(Base):
assert cmd_input is not None, 'Invalid: cmd_input is None' assert cmd_input is not None, 'Invalid: cmd_input is None'
ret = Result(None, None, 1) ret = Result(None, None, 1)
with self.CallContext(self, title=title, cmd=cmd, cmd_input=cmd_input, with self.CallContext(
mod_env=mod_env_cmd, wd=wd, self,
log_prefix='|', throw=throw, verbose=verbose) as cc: title = title,
cmd = cmd,
cmd_input = cmd_input,
mod_env = mod_env_cmd,
wd = wd,
log_prefix = '|',
throw = throw,
verbose = verbose,
) as cc:
try: try:
ret = await self._sudo( ret = await self._sudo(
cmd = cc.cmd, cmd = cc.cmd,
@ -416,19 +466,22 @@ class ExecContext(Base):
) )
async def _get( async def _get(
self, self, path: str, wd: str | None, throw: bool, verbose: bool | None, title: str
path: str,
wd: str|None,
throw: bool,
verbose: bool|None,
title: str
) -> Result: ) -> Result:
ret = Result(None, None, 1) ret = Result(None, None, 1)
if wd is not None: if wd is not None:
path = wd + '/' + path path = wd + '/' + path
with self.CallContext(self, title=title, cmd=['cat', path], with self.CallContext(
cmd_input=InputMode.NonInteractive, wd=None, mod_env=None, self,
log_prefix='|', throw=throw, verbose=verbose) as cc: title = title,
cmd = ['cat', path],
cmd_input = InputMode.NonInteractive,
wd = None,
mod_env = None,
log_prefix = '|',
throw = throw,
verbose = verbose,
) as cc:
try: try:
ret = await self._run( ret = await self._run(
cmd = cc.cmd, cmd = cc.cmd,
@ -437,12 +490,12 @@ class ExecContext(Base):
cmd_input = cc.cmd_input, cmd_input = cc.cmd_input,
mod_env = cc.mod_env, mod_env = cc.mod_env,
interactive = cc.interactive, interactive = cc.interactive,
log_prefix = cc.log_prefix log_prefix = cc.log_prefix,
) )
except Exception as e: except Exception as e:
return cc.exception(ret, e) return cc.exception(ret, e)
if ret.status != 0 and ret.stderr.decode().find('No such file') != -1: if ret.matches_error('No such file'):
raise FileNotFoundError(ret.stderr) raise FileNotFoundError(ret.summarize(cc.cmd, wd = cc.wd))
cc.check_exit_code(ret) cc.check_exit_code(ret)
return ret return ret
@ -450,138 +503,176 @@ class ExecContext(Base):
self, self,
path: str, path: str,
content: bytes, content: bytes,
wd: str|None, wd: str | None,
throw: bool, throw: bool,
verbose: bool|None, verbose: bool | None,
title: str, title: str,
owner: str|None, owner: str | None,
group: str|None, group: str | None,
mode: str|None, mode: str | None,
atomic: bool, atomic: bool,
) -> Result: ) -> Result:
from .util import pretty_cmd from .util import pretty_cmd
async def __run(cmd: list[str], cmd_input: Input=InputMode.NonInteractive, **kwargs) -> Result: async def __run(
return await self.run(cmd, cmd_input=cmd_input, **kwargs) cmd: list[str],
cmd_input: Input = InputMode.NonInteractive,
**kwargs
) -> Result:
return await self.run(cmd, cmd_input = cmd_input, **kwargs)
ret = Result(None, None, 1) ret = Result(None, None, 1)
try: try:
class RemoteCmd(NamedTuple):
cmd: list[str]
cmd_input: Input = InputMode.NonInteractive
if wd is not None: if wd is not None:
path = wd + '/' + path path = wd + '/' + path
cmds: list[dict[str, str|list[str]|bool]] = [] cmds: list[RemoteCmd] = []
out = (await __run(['mktemp', path + '.XXXXXX'])).stdout.decode().strip() if atomic else path stdout = (await __run(['mktemp', path + '.XXXXXX'])).stdout_str
cmds.append({'cmd': ['tee', out], 'cmd_input': content}) if stdout is None:
raise Exception(f'Failed to create tmp-directory on {self.root}')
out = stdout.strip() if atomic else path
cmds.append(RemoteCmd(
cmd = ['tee', out],
cmd_input = content,
))
if owner is not None and group is not None: if owner is not None and group is not None:
cmds.append({'cmd': ['chown', f'{owner}:{group}', out]}) cmds.append(RemoteCmd(
cmd = ['chown', f'{owner}:{group}', out],
))
elif owner is not None: elif owner is not None:
cmds.append({'cmd': ['chown', owner, out]}) cmds.append(RemoteCmd(
cmd = ['chown', owner, out],
))
elif group is not None: elif group is not None:
cmds.append({'cmd': ['chgrp', group, out]}) cmds.append(RemoteCmd(
cmd = ['chgrp', group, out],
))
if mode is not None: if mode is not None:
cmds.append({'cmd': ['chmod', mode, out]}) cmds.append(RemoteCmd(
cmd = ['chmod', mode, out],
))
if atomic: if atomic:
cmds.append({'cmd': ['mv', out, path]}) cmds.append(RemoteCmd(
cmd = ['mv', out, path],
))
await self.open() await self.open()
try: try:
for cmd in cmds: for cmd in cmds:
log(DEBUG, f'{self.log_name}: Running {pretty_cmd(cmd['cmd'], wd)}') log(DEBUG, f'{self.log_name}: Running {pretty_cmd(cmd.cmd, wd)}')
ret = await __run(**cmd) ret = await __run(cmd.cmd)
return ret return ret
finally: finally:
await self.close() await self.close()
except: except Exception as e:
msg = f'Failed to get {path} from {self.root} ({str(e)})'
if throw: if throw:
raise raise Exception(msg)
return cc.exception(ret, e) log(ERR, msg)
return ret return ret
async def _unlink(self, path: str) -> None: async def _unlink(self, path: str) -> None:
cmd = ['rm', '-f', path] cmd = ['rm', '-f', path]
await self.run(cmd, cmd_input=InputMode.NonInteractive) await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _erase(self, path: str) -> None: async def _erase(self, path: str) -> None:
cmd = ['rm', '-rf', path] cmd = ['rm', '-rf', path]
await self.run(cmd, cmd_input=InputMode.NonInteractive) await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _rename(self, src: str, dst: str) -> None: async def _rename(self, src: str, dst: str) -> None:
cmd = ['mv', src, dst] cmd = ['mv', src, dst]
await self.run(cmd, cmd_input=InputMode.NonInteractive) await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _mkdir(self, name: str, mode: int) -> None: async def _mkdir(self, name: str, mode: int) -> None:
cmd = ['mkdir', name, '-m', self.__mode_str(mode)] cmd = ['mkdir', name, '-m', self.__mode_str(mode)]
await self.run(cmd, cmd_input=InputMode.NonInteractive) await self.run(cmd, cmd_input = InputMode.NonInteractive)
async def _mktemp(self, tmpl: str, directory: bool) -> str: async def _mktemp(self, tmpl: str, directory: bool) -> str:
cmd = ['mktemp'] cmd = ['mktemp']
if directory: if directory:
cmd.append('-d') cmd.append('-d')
cmd.append(tmpl) cmd.append(tmpl)
result = await self.run(cmd, cmd_input=InputMode.NonInteractive) result = await self.run(cmd, cmd_input = InputMode.NonInteractive, throw = True)
return result.stdout.strip().decode() if result.status != 0 or result.stdout is None:
raise Exception(
f'Failed to create temporary file on {self.root}: {result.summary}'
)
return result.stdout_str
async def _stat(self, path: str, follow_symlinks: bool) -> StatResult: async def _stat(self, path: str, follow_symlinks: bool) -> StatResult:
async def __stat(opts: list[str]) -> str: async def __stat(opts: list[str]) -> Result:
mod_env = { mod_env = {'LC_ALL': 'C'}
'LC_ALL': 'C'
}
cmd = ['stat'] cmd = ['stat']
if follow_symlinks: if follow_symlinks:
cmd.append('-L') cmd.append('-L')
cmd.extend(opts) cmd.extend(opts)
cmd.append(path) cmd.append(path)
return (await self.run(cmd, mod_env=mod_env, throw=False, return await self.run(
cmd_input=InputMode.NonInteractive)).decode() cmd,
mod_env = mod_env,
throw = False,
cmd_input = InputMode.NonInteractive
)
# GNU coreutils stat # GNU coreutils stat
gnu_format = _US.join([ gnu_format = _US.join(
"%f", # st_mode in hex [
"%i", # st_ino '%f', # st_mode in hex
"%d", # st_dev '%i', # st_ino
"%h", # st_nlink '%d', # st_dev
"%U", # st_uid '%h', # st_nlink
"%G", # st_gid '%U', # st_uid
"%s", # st_size '%G', # st_gid
"%.9X", # st_atime '%s', # st_size
"%.9Y", # st_mtime '%.9X', # st_atime
"%.9Z", # st_ctime '%.9Y', # st_mtime
"%o", # st_blksize hint '%.9Z', # st_ctime
"%b", # st_blocks '%o', # st_blksize hint
"%r", # st_rdev '%b', # st_blocks
]) '%r', # st_rdev
]
)
result = await __stat(['--printf', gnu_format]) result = await __stat(['--printf', gnu_format])
if result.status == 0: if result.status == 0 and result.stdout is not None:
return _build_stat_result(result.stdout.split(_US), mode_base=16) return _build_stat_result(result.stdout_str.split(_US), mode_base = 16)
if not _looks_like_option_error(result.stderr): if not _looks_like_option_error(result.stderr_str_or_none):
# log(DEBUG, f'GNU stat attempt failed on "{path}" ({str(e)})') # log(DEBUG, f'GNU stat attempt failed on "{path}" ({str(e)})')
_raise_stat_error(path, result.stderr, result.status) _raise_stat_error(path, result)
# BSD / macOS / OpenBSD / NetBSD stat # BSD / macOS / OpenBSD / NetBSD stat
bsd_format = _US.join([ bsd_format = _US.join(
"%p", # st_mode in octal [
"%i", # st_ino '%p', # st_mode in octal
"%d", # st_dev '%i', # st_ino
"%l", # st_nlink '%d', # st_dev
"%U", # st_uid '%l', # st_nlink
"%G", # st_gid '%U', # st_uid
"%z", # st_size '%G', # st_gid
"%.9Fa", # st_atime '%z', # st_size
"%.9Fm", # st_mtime '%.9Fa', # st_atime
"%.9Fc", # st_ctime '%.9Fm', # st_mtime
"%k", # st_blksize '%.9Fc', # st_ctime
"%b", # st_blocks '%k', # st_blksize
"%r", # st_rdev '%b', # st_blocks
]) '%r', # st_rdev
]
)
result = await __stat(['-n', '-f', bst_format]) result = await __stat(['-n', '-f', bsd_format])
if proc.returncode == 0: stdout = result.stdout_str_or_none
return _build_stat_result(proc.stdout.rstrip('\n').split(_US), mode_base=8) if result.status != 0 or stdout is None:
_raise_stat_error(path, result.stderr, result.status) _raise_stat_error(path, result)
assert stdout is not None # Just there to pacify the linter
return _build_stat_result(stdout.rstrip('\n').split(_US), mode_base = 8)
async def _chown(self, path: str, owner: str|None, group: str|None) -> None: async def _chown(self, path: str, owner: str | None, group: str | None) -> None:
if owner is None and group is None: if owner is None and group is None:
raise ValueError(f'Tried to chown("{path}") without owner and group') raise ValueError(f'Tried to chown("{path}") without owner and group')
if group is None: if group is None:
@ -590,7 +681,11 @@ class ExecContext(Base):
ownership = ':' + group ownership = ':' + group
else: else:
ownership = owner + ':' + group ownership = owner + ':' + group
await self.run(['chown', ownership, path], cmd_input=InputMode.NonInteractive) assert ownership is not None # Impossible, just there to calm the linter
await self.run(['chown', ownership, path], cmd_input = InputMode.NonInteractive)
async def _chmod(self, path: str, mode: int) -> None: async def _chmod(self, path: str, mode: int) -> None:
await self.run(['chmod', self.__mode_str(mode), path], cmd_input=InputMode.NonInteractive) await self.run(
['chmod', self.__mode_str(mode), path],
cmd_input = InputMode.NonInteractive
)

173
src/python/jw/pkg/lib/FileContext.py
View file

@ -1,19 +1,17 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
import abc, re import abc
from enum import Enum, auto from enum import Enum, auto
from typing import TYPE_CHECKING, Self from functools import cached_property
from functools import cached_property, cache from typing import TYPE_CHECKING
from .log import DEBUG, ERR, log
from .Uri import Uri
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Self from .base import Result, StatResult
from .ProcFilter import ProcFilter, ProcPipeline
from .log import *
from .base import Input, InputMode, Result, StatResult
from .Uri import Uri
from .ProcFilter import ProcPipeline
class FileContext(abc.ABC): class FileContext(abc.ABC):
@ -22,24 +20,27 @@ class FileContext(abc.ABC):
Out = auto() Out = auto()
def __init__( def __init__(
self, self,
uri: str|Uri, uri: str | Uri,
interactive: bool|None = None, interactive: bool | None = None,
verbose_default = False, verbose_default = False,
chroot: bool = False, chroot: bool = False,
in_pipe: ProcPipeline|None = None, in_pipe: ProcPipeline | None = None,
out_pipe: ProcPipeline|None = None, out_pipe: ProcPipeline | None = None,
): ):
self.__uri = Uri.pimp(uri) self.__uri = Uri.pimp(uri)
self.__chroot = chroot self.__chroot = chroot
self.__interactive = interactive self.__interactive = interactive
self.__verbose_default = verbose_default self.__verbose_default = verbose_default
self.__log_name: str|None = None self.__log_name: str | None = None
self.__in_pipe = in_pipe self.__in_pipe = in_pipe
self.__out_pipe = out_pipe self.__out_pipe = out_pipe
self.__open_count = 0 self.__open_count = 0
if not verbose_default in [True, False]: if verbose_default not in [True, False]:
raise ValueError(f'Tried to instantiate FileContext with verbose_default = "{verbose_default}"') raise ValueError(
'Tried to instantiate FileContext with verbose_default '
f'= "{verbose_default}"'
)
async def __aenter__(self): async def __aenter__(self):
await self.open() await self.open()
@ -91,7 +92,9 @@ class FileContext(abc.ABC):
if self.__open_count == 1: if self.__open_count == 1:
await self._close() await self._close()
self.__open_count -= 1 self.__open_count -= 1
assert self.__open_count >= 0, f'Closed file context "{self}" more often than opened' assert self.__open_count >= 0, (
f'Closed file context "{self}" more often than opened'
)
@property @property
def uri(self) -> Uri: def uri(self) -> Uri:
@ -106,7 +109,7 @@ class FileContext(abc.ABC):
return self.__uri.path return self.__uri.path
@property @property
def username(self) -> str|None: def username(self) -> str | None:
return self.__uri.username return self.__uri.username
@property @property
@ -114,7 +117,7 @@ class FileContext(abc.ABC):
return self.__uri.id return self.__uri.id
@property @property
def interactive(self) -> bool|None: def interactive(self) -> bool | None:
return self.__interactive return self.__interactive
@property @property
@ -123,29 +126,24 @@ class FileContext(abc.ABC):
@abc.abstractmethod @abc.abstractmethod
async def _get( async def _get(
self, self, path: str, wd: str | None, throw: bool, verbose: bool | None, title: str
path: str,
wd: str|None,
throw: bool,
verbose: bool|None,
title: str
) -> Result: ) -> Result:
raise NotImplementedError() raise NotImplementedError()
async def get( async def get(
self, self,
path: str, path: str,
wd: str|None = None, wd: str | None = None,
throw: bool = True, throw: bool = True,
verbose: bool|None = None, verbose: bool | None = None,
title: str=None, title: str | None = None,
) -> Result: ) -> Result:
ret = await self._get( ret = await self._get(
self._chroot(path), self._chroot(path),
wd = wd, wd = wd,
throw = throw, throw = throw,
verbose = verbose, verbose = verbose,
title = title, title = title or f'Fetching {path} from {self.uri}',
) )
return await self.__in_pipe.run(ret) if self.__in_pipe else ret return await self.__in_pipe.run(ret) if self.__in_pipe else ret
@ -153,13 +151,13 @@ class FileContext(abc.ABC):
self, self,
path: str, path: str,
content: bytes, content: bytes,
wd: str|None, wd: str | None,
throw: bool, throw: bool,
verbose: bool|None, verbose: bool | None,
title: str, title: str,
owner: str|None, owner: str | None,
group: str|None, group: str | None,
mode: str|None, mode: str | None,
atomic: bool, atomic: bool,
) -> Result: ) -> Result:
raise NotImplementedError() raise NotImplementedError()
@ -167,26 +165,26 @@ class FileContext(abc.ABC):
async def put( async def put(
self, self,
path: str, path: str,
content: str, content: bytes,
wd: str|None = None, wd: str | None = None,
throw: bool = True, throw: bool = True,
verbose: bool|None = None, verbose: bool | None = None,
title: str = None, title: str | None = None,
owner: str|None = None, owner: str | None = None,
group: str|None = None, group: str | None = None,
mode: int|None = None, mode: int | None = None,
atomic: bool = False atomic: bool = False,
) -> Result: ) -> Result:
mode_str = None if mode is None else oct(mode).replace('0o', '0') mode_str = None if mode is None else oct(mode).replace('0o', '0')
if self.__out_pipe is not None: if self.__out_pipe is not None:
content = self.__out_pipe.run(content).stdout result = await self.__out_pipe.run(content)
return await self._put( return await self._put(
self._chroot(path), self._chroot(path),
content, result.stdout,
wd = wd, wd = wd,
throw = throw, throw = throw,
verbose = verbose, verbose = verbose,
title = title, title = title or f'Pushing content to {path} on {self.uri}',
owner = owner, owner = owner,
group = group, group = group,
mode = mode_str, mode = mode_str,
@ -194,55 +192,73 @@ class FileContext(abc.ABC):
) )
async def _unlink(self, path: str) -> None: async def _unlink(self, path: str) -> None:
raise NotImplementedError(f'{self.log_name}: unlink("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: unlink("{path}") is not implemented'
)
async def unlink(self, path: str) -> None: async def unlink(self, path: str) -> None:
return await self._unlink(self._chroot(path)) return await self._unlink(self._chroot(path))
async def _erase(self, path: str) -> None: async def _erase(self, path: str) -> None:
raise NotImplementedError(f'{self.log_name}: erase("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: erase("{path}") is not implemented'
)
async def erase(self, path: str) -> None: async def erase(self, path: str) -> None:
return await self._erase(self._chroot(path)) return await self._erase(self._chroot(path))
async def _rename(self, src: str, dst: str) -> None: async def _rename(self, src: str, dst: str) -> None:
raise NotImplementedError(f'{self.log_name}: rename("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: rename("{src}" -> "{dst}") is not implemented'
)
async def rename(self, src: str, dst: str) -> None: async def rename(self, src: str, dst: str) -> None:
return await self._rename(src, dst) return await self._rename(src, dst)
async def _mkdir(self, path: str, mode: int) -> None: async def _mkdir(self, path: str, mode: int) -> None:
raise NotImplementedError(f'{self.log_path}: mkdir({path}) is not implemented') raise NotImplementedError(f'{self.log_name}: mkdir({path}) is not implemented')
async def mkdir(self, path: str, mode: int=0o777) -> None: async def mkdir(self, path: str, mode: int = 0o777) -> None:
return await self._mkdir(path, mode) return await self._mkdir(path, mode)
async def _mktemp(self, tmpl: str, directory: bool) -> None: async def _mktemp(self, tmpl: str, directory: bool) -> str:
raise NotImplementedError(f'{self.log_name}: mktemp("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: mktemp("{tmpl}") is not implemented'
)
async def mktemp(self, tmpl: str, directory: bool=False) -> None: async def mktemp(self, tmpl: str, directory: bool = False) -> str:
return await self._mktemp(self._chroot(tmpl), directory) return await self._mktemp(self._chroot(tmpl), directory)
async def _chown(self, path: str, owner: str|None, group: str|None) -> None: async def _chown(self, path: str, owner: str | None, group: str | None) -> None:
raise NotImplementedError(f'{self.log_name}: chown("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: chown("{path}") is not implemented'
)
async def chown(self, path: str, owner: str|None=None, group: str|None=None) -> None: async def chown(
self, path: str, owner: str | None = None, group: str | None = None
) -> None:
if owner is None and group is None: if owner is None and group is None:
raise ValueError(f'Tried to change ownership of {path} specifying neither owner nor group') raise ValueError(
f'Tried to change ownership of {path} with neither owner nor group'
)
return await self._chown(self._chroot(path), owner, group) return await self._chown(self._chroot(path), owner, group)
async def _chmod(self, path: str, mode: int) -> None: async def _chmod(self, path: str, mode: int) -> None:
raise NotImplementedError(f'{self.log_name}: chmod("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: chmod("{path}") is not implemented'
)
async def chmod(self, path: str, mode: int) -> None: async def chmod(self, path: str, mode: int) -> None:
return await self._chmod(self._chroot(path), mode) return await self._chmod(self._chroot(path), mode)
async def _stat(self, path: str, follow_symlinks: bool) -> StatResult: async def _stat(self, path: str, follow_symlinks: bool) -> StatResult:
raise NotImplementedError(f'{self.log_name}: lstat("{path}") is not implemented') raise NotImplementedError(
f'{self.log_name}: lstat("{path}") is not implemented'
)
async def stat(self, path: str, follow_symlinks: bool=True) -> StatResult: async def stat(self, path: str, follow_symlinks: bool = True) -> StatResult:
if not isinstance(path, str): if not isinstance(path, str):
raise TypeError(f"path must be str, got {type(path).__name__}") raise TypeError(f'path must be str, got {type(path).__name__}')
return await self._stat(self._chroot(path), follow_symlinks) return await self._stat(self._chroot(path), follow_symlinks)
async def _file_exists(self, path: str) -> bool: async def _file_exists(self, path: str) -> bool:
@ -261,10 +277,17 @@ class FileContext(abc.ABC):
async def _is_dir(self, path: str, follow_symlinks: bool) -> bool: async def _is_dir(self, path: str, follow_symlinks: bool) -> bool:
import stat import stat
try: try:
return stat.S_ISDIR((await self._stat(path, follow_symlinks)).mode) return stat.S_ISDIR((await self._stat(path, follow_symlinks)).mode)
except NotImplementedError: except NotImplementedError:
log(DEBUG, f'{self.log_name} doesn\'t implement stat(), judging by trailing slash if {path} is a directory') log(
DEBUG,
(
f"{self.log_name} doesn't implement stat(), judging by trailing "
'slash if {path} is a directory'
),
)
return path[-1] == '/' return path[-1] == '/'
except FileNotFoundError as e: except FileNotFoundError as e:
log(DEBUG, f'{self.log_name}: Failed to stat({path}) ({str(e)})') log(DEBUG, f'{self.log_name}: Failed to stat({path}) ({str(e)})')
@ -274,22 +297,28 @@ class FileContext(abc.ABC):
raise raise
return False return False
async def is_dir(self, path: str, follow_symlinks=True) -> bool: async def is_dir(self, path: str, follow_symlinks = True) -> bool:
return await self._is_dir(self._chroot(path), follow_symlinks=follow_symlinks) return await self._is_dir(self._chroot(path), follow_symlinks = follow_symlinks)
@classmethod @classmethod
def create(cls, uri: str|Uri, *args, **kwargs) -> Self: def create(cls, uri: str | Uri, *args, **kwargs) -> FileContext:
uri = Uri.pimp(uri) uri = Uri.pimp(uri)
match uri.protocol: match uri.protocol:
case 'local' | 'file': case 'local' | 'file':
from .ec.Local import Local from .ec.Local import Local
return Local(uri, *args, **kwargs) return Local(uri, *args, **kwargs)
case 'ssh': case 'ssh':
from .ec.SSHClient import ssh_client from .ec.SSHClient import ssh_client
return ssh_client(uri, *args, **kwargs) return ssh_client(uri, *args, **kwargs)
case 'http' | 'https': case 'http' | 'https':
from .ec.Curl import Curl from .ec.Curl import Curl
return Curl(uri, *args, **kwargs) return Curl(uri, *args, **kwargs)
case _: case _:
pass pass
raise Exception(f'Can\'t create file context instance for "{uri}" with unsupported protocol "{uri.protocol}"') raise Exception(
f'Can\'t create file context instance for "{uri}" with unsupported '
f'protocol "{uri.protocol}"'
)

44
src/python/jw/pkg/lib/Package.py
View file

@ -1,32 +1,35 @@
# -*- coding: utf-8 -*-
from typing import Any from typing import Any
meta_tags = [ meta_tags = [
"name", 'name',
"vendor", 'vendor',
"packager", 'packager',
"url", 'url',
"maintainer", 'maintainer',
] ]
class Package: class Package:
name: str
name: str = None vendor: str | None = None
vendor: str|None = None packager: str | None = None
packager: str|None = None url: str | None = None
url: str|None = None maintainer: str | None = None
maintainer: str|None = None
@classmethod @classmethod
def parse_spec_str(cls, spec: str, delimiter='|'): def parse_spec_str(cls, spec: str, delimiter = '|'):
tags = spec.split(delimiter) tags = spec.split(delimiter)
if len(tags) != 5: if len(tags) != 5:
raise ValueError(f'Invalid package spec string "{spec}"') raise ValueError(f'Invalid package spec string "{spec}"')
return cls(name=tags[0], vendor=tags[1], packager=tags[2], url=tags[3], maintainer=tags[4]) return cls(
name = tags[0],
vendor = tags[1],
packager = tags[2],
url = tags[3],
maintainer = tags[4],
)
@classmethod @classmethod
def parse_specs_str(cls, specs: str, delimiter='|'): def parse_specs_str(cls, specs: str, delimiter = '|'):
ret: list[Package] = [] ret: list[Package] = []
for spec in specs.splitlines(): for spec in specs.splitlines():
ret.append(cls.parse_spec_str(spec)) ret.append(cls.parse_spec_str(spec))
@ -39,7 +42,14 @@ class Package:
ret[tag] = mapping.get(tag, '') ret[tag] = mapping.get(tag, '')
return ret return ret
def __init__(self, name: str, vendor: str|None=None, packager: str|None=None, url: str|None=None, maintainer: str|None=None): def __init__(
self,
name: str,
vendor: str | None = None,
packager: str | None = None,
url: str | None = None,
maintainer: str | None = None,
):
self.name = name self.name = name
self.vendor = vendor self.vendor = vendor
self.packager = packager self.packager = packager

10
src/python/jw/pkg/lib/PackageFilter.py
View file

@ -1,6 +1,5 @@
# -*- coding: utf-8 -*- import abc
import re
import abc, re
from .Package import Package from .Package import Package
@ -23,4 +22,7 @@ class PackageFilterString(PackageFilter):
self.__definition = url_rx_str self.__definition = url_rx_str
def _match(self, package: Package) -> bool: def _match(self, package: Package) -> bool:
return re.search(self.__definition, package.url) is not None url = package.url
if url is None:
return False
return re.search(self.__definition, url) is not None

22
src/python/jw/pkg/lib/ProcFilter.py
View file

@ -1,10 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING
import abc import abc
from typing import TYPE_CHECKING
from .base import Result from .base import Result
if TYPE_CHECKING: if TYPE_CHECKING:
@ -13,37 +12,40 @@ if TYPE_CHECKING:
class ProcFilter(abc.ABC): class ProcFilter(abc.ABC):
@abc.abstractmethod @abc.abstractmethod
async def _run(self, data: bytes) -> Result: async def _run(self, data: bytes | None) -> Result:
raise NotImplementedError() raise NotImplementedError()
async def run(self, data: bytes) -> Result: async def run(self, data: bytes | None) -> Result:
return await self._run(data) return await self._run(data)
class ProcFilterIdentity(ProcFilter): class ProcFilterIdentity(ProcFilter):
async def _run(self, data: bytes) -> Result: async def _run(self, data: bytes | None) -> Result:
return Result(data, None, 0) return Result(data, None, 0)
class ProcPipeline: class ProcPipeline:
def __init__(self, f: Iterable[ProcFilter]|ProcFilter = []) -> None: def __init__(self, f: Iterable[ProcFilter] | ProcFilter = []) -> None:
self.__filters: list[ProcFilter] = [] self.__filters: list[ProcFilter] = []
self.append(f) self.append(f)
def append(self, f: ProcFilter|Iterable[ProcFilter]) -> None: def append(self, f: ProcFilter | Iterable[ProcFilter]) -> None:
if not isinstance(f, ProcFilter): if not isinstance(f, ProcFilter):
for e in f: for e in f:
self.append(e) self.append(e)
return return
self.__filters.append(f) self.__filters.append(f)
async def run(self, data: bytes|Result) -> Result: async def run(self, data: None | bytes | Result) -> Result:
ret = data if isinstance(data, Result) else Result(data, None, 0) ret = data if isinstance(data, Result) else Result(data, None, 0)
for f in self.__filters: for f in self.__filters:
ret = await f.run(ret.stdout) ret = await f.run(ret.stdout)
return ret return ret
async def run(data: bytes|Result, chain: ProcFilter|list[ProcFilter]|ProcPipeline|None = None) -> Result: async def run(
data: bytes | Result,
chain: ProcFilter | list[ProcFilter] | ProcPipeline | None = None,
) -> Result:
if chain is None: if chain is None:
if isinstance(data, Result): if isinstance(data, Result):
return data return data

16
src/python/jw/pkg/lib/ProcFilterGpg.py
View file

@ -1,10 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from .ProcFilter import ProcFilter
from .base import Result from .base import Result
from .ProcFilter import ProcFilter
if TYPE_CHECKING: if TYPE_CHECKING:
from .ExecContext import ExecContext from .ExecContext import ExecContext
@ -14,9 +13,12 @@ class ProcFilterGpg(ProcFilter):
def __init__(self, ec: ExecContext) -> None: def __init__(self, ec: ExecContext) -> None:
self.__ec = ec self.__ec = ec
async def _run(self, data: bytes) -> Result: async def _run(self, data: bytes | None) -> Result:
return await self.__ec.run([ if data is None:
"gpg", raise Exception('No data for GPG to decrypt')
return await self.__ec.run(
[
'gpg',
'--batch', '--batch',
'--yes', '--yes',
'--quiet', '--quiet',
@ -24,5 +26,5 @@ class ProcFilterGpg(ProcFilter):
'--decrypt', '--decrypt',
], ],
cmd_input = data, cmd_input = data,
throw = True throw = True,
) )

85
src/python/jw/pkg/lib/TarIo.py
View file

@ -1,29 +1,34 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import Self import abc
import io
import abc, io
import tarfile import tarfile
from tarfile import TarFile
from tarfile import TarFile, TarInfo
from .base import StatResult
from .CopyContext import CopyContext from .CopyContext import CopyContext
from .FileContext import FileContext from .ExecContext import ExecContext
from .log import * from .log import DEBUG, ERR, log
class TarIo(CopyContext): class TarIo(CopyContext):
def __init__(self, *args, **kwargs) -> None: def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs, chroot=False) kwargs['chroot'] = False
super().__init__(*args, **kwargs)
def _match(self, path: str, path_filter: list[str]) -> bool: def _match(self, path: str, path_filter: list[str]) -> bool:
return path in path_filter return path in path_filter
def _filter_tar_file(self, blob: bytes, path_filter: list[str]|None=None, matched: list[str]|None=None) -> bytes: def _filter_tar_file(
self,
blob: bytes,
path_filter: list[str] | None = None,
matched: list[str] | None = None,
) -> bytes:
ret = io.BytesIO() ret = io.BytesIO()
with tarfile.open(fileobj=ret, mode='w') as tf_out: with tarfile.open(fileobj = ret, mode = 'w') as tf_out:
tf_in = TarFile(fileobj=io.BytesIO(blob)) tf_in = TarFile(fileobj = io.BytesIO(blob))
for info in tf_in.getmembers(): for info in tf_in.getmembers():
if path_filter is not None and not self._match(info.name, path_filter): if path_filter is not None and not self._match(info.name, path_filter):
continue continue
@ -34,13 +39,18 @@ class TarIo(CopyContext):
tf_out.addfile(info, buf) tf_out.addfile(info, buf)
return ret.getvalue() return ret.getvalue()
async def _read_filtered(self, path, path_filter: list[str]|None=None, matched: list[str]|None=None) -> bytes: async def _read_filtered(
self,
path,
path_filter: list[str] | None = None,
matched: list[str] | None = None,
) -> bytes:
try: try:
blob = (await self.src.get(path)).stdout blob = (await self.src.get(path)).stdout
except Exception as e: except Exception as e:
log(ERR, f'Failed to read tar file "{path}" ({str(e)}') log(ERR, f'Failed to read tar file "{path}" ({str(e)}')
raise raise
return self._filter_tar_file(blob, path_filter, matched=matched) return self._filter_tar_file(blob, path_filter, matched = matched)
def _add(self, tf: TarFile, path: str, st: StatResult, contents: bytes) -> None: def _add(self, tf: TarFile, path: str, st: StatResult, contents: bytes) -> None:
file_obj = io.BytesIO(contents) file_obj = io.BytesIO(contents)
@ -50,37 +60,34 @@ class TarIo(CopyContext):
info.uname = st.owner info.uname = st.owner
info.gname = st.group info.gname = st.group
info.size = st.size info.size = st.size
info.atime = st.atime info.mtime = int(st.mtime)
info.mtime = st.mtime tf.addfile(info, file_obj)
info.ctime = st.ctime
tf.addfile(info, io.BytesIO(file_obj))
async def _add_from_path(self, src: FileContext, tf: TarFile, path: str) -> None:
contents = await src.get(path)
st = await self.stat(path)
self._add(tf, path, st, contents)
@abc.abstractmethod @abc.abstractmethod
async def _extract(self, blob: bytes, root: str|None=None) -> None: async def _extract(self, blob: bytes, root: str | None = None) -> None:
raise NotImplementedError() raise NotImplementedError()
async def extract(self, root: str|None=None, path_filter: list[str]|None=None) -> list[str]: async def extract(
self,
root: str | None = None,
path_filter: list[str] | None = None
) -> list[str]:
ret: list[str] = [] ret: list[str] = []
filtered = await self._read_filtered(self.src.root, path_filter, matched=ret) filtered = await self._read_filtered(self.src.root, path_filter, matched = ret)
await self._extract(blob=filtered, root=root) await self._extract(blob = filtered, root = root)
return ret return ret
@classmethod @classmethod
def create(cls, *args, type: str=None, **kwargs): def create(cls, *args, type: str | None = None, **kwargs):
if type is not None: if type is not None:
raise NotImplementedError raise NotImplementedError
#return TarIoTarFile(*args, **kwargs) # return TarIoTarFile(*args, **kwargs)
return TarIoTarExec(*args, **kwargs) return TarIoTarExec(*args, **kwargs)
class TarIoTarFile(TarIo): class TarIoTarFile(TarIo):
async def _extract(self, blob: bytes, root: str|None=None) -> None: async def _extract(self, blob: bytes, root: str | None = None) -> None:
tf = TarFile(fileobj=io.BytesIO(blob)) tf = TarFile(fileobj = io.BytesIO(blob))
for info in tf.getmembers(): for info in tf.getmembers():
log(DEBUG, f'Extracting {info.name}') log(DEBUG, f'Extracting {info.name}')
path = root + '/' + info.name if root else info.name path = root + '/' + info.name if root else info.name
@ -96,14 +103,24 @@ class TarIoTarFile(TarIo):
buf.read(), buf.read(),
owner = info.uname, owner = info.uname,
group = info.gname, group = info.gname,
mode = info.mode, mode = info.mode,
) )
class TarIoTarExec(TarIo): class TarIoTarExec(TarIo):
async def _extract(self, blob: bytes, root: str|None=None) -> None: @property
def dst(self) -> ExecContext:
ret = super().dst
if not isinstance(ret, ExecContext):
raise Exception(
'Tried to get executable destination context from copy '
'context, which only has a file context'
)
return ret
async def _extract(self, blob: bytes, root: str | None = None) -> None:
cmd = ['tar'] cmd = ['tar']
if root is not None: if root is not None:
cmd += ['-C', root] cmd += ['-C', root]
cmd += ['-x', '-f', '-'] cmd += ['-x', '-f', '-']
await self.dst.run(cmd, cmd_input=blob) await self.dst.run(cmd, cmd_input = blob)

86
src/python/jw/pkg/lib/Types.py
View file

@ -1,24 +1,29 @@
# -*- coding: utf-8 -*- import abc
import os
import re
import sys
from typing import TypeVar, Generic from collections.abc import Iterator
import abc, re, sys, os from typing import TYPE_CHECKING, Generic, Iterable, TypeVar
from collections.abc import Iterable, Iterator
from .log import * from .log import OFF, log, parse_log_level
T = TypeVar("T") if TYPE_CHECKING:
from typing import Any
class Types(abc.ABC, Iterable[T], Generic[T]): # export T = TypeVar('T')
def __iter__(self) -> Iterator[T]: class Types(abc.ABC, Iterable[type[T]], Generic[T]): # export
def __iter__(self) -> Iterator[type[T]]:
return iter(self._classes()) return iter(self._classes())
@abc.abstractmethod @abc.abstractmethod
def _classes(self) -> Iterable[T]: def _classes(self) -> Iterable[type[T]]:
pass pass
@property @property
def classes(self) -> Iterable[T]: def classes(self) -> Iterable[type[T]]:
return self._classes() return self._classes()
@abc.abstractmethod @abc.abstractmethod
@ -27,14 +32,20 @@ class Types(abc.ABC, Iterable[T], Generic[T]): # export
def dump(self, prio: int, *args, **kwargs) -> None: def dump(self, prio: int, *args, **kwargs) -> None:
contents = self._stringify() contents = self._stringify()
log(prio, ",--- ", *args, **kwargs) log(prio, ',--- ', *args, **kwargs)
for line in contents: for line in contents:
log(prio, "| " + line) log(prio, '| ' + line)
log(prio, "`--- ", *args, **kwargs) log(prio, '`--- ', *args, **kwargs)
class LoadTypes(Types): # export class LoadTypes(Types[T]): # export
def __init__(self, mod_names: list[str], type_name_filter: str=None, type_filter: list[T]=[], debug_level=None): def __init__(
self,
mod_names: list[str],
type_name_filter: str | None = None,
type_filter: list[type[T]] = [],
debug_level = None,
):
if debug_level is None: if debug_level is None:
val = os.getenv('JW_LOG_LEVEL_LOAD_TYPES') val = os.getenv('JW_LOG_LEVEL_LOAD_TYPES')
if val is not None: if val is not None:
@ -45,7 +56,7 @@ class LoadTypes(Types): # export
self.__type_name_filter = type_name_filter self.__type_name_filter = type_name_filter
self.__type_filter = type_filter self.__type_filter = type_filter
self.__mod_names = mod_names self.__mod_names = mod_names
self.__classes: list[type[Any]]|None = None self.__classes: list[type[T]] | None = None
def _debug(self, *args, **kwargs) -> None: def _debug(self, *args, **kwargs) -> None:
if self.__debug_level != OFF: if self.__debug_level != OFF:
@ -53,37 +64,56 @@ class LoadTypes(Types): # export
def _stringify(self): def _stringify(self):
return [ return [
"type_name_filter: " + str(self.__type_name_filter), 'type_name_filter: ' + str(self.__type_name_filter),
"type_filter: " + ', '.join([str(f) for f in self.__type_filter]), 'type_filter: ' + ', '.join([str(f) for f in self.__type_filter]),
"mod_names: " + ', '.join(self.__mod_names) 'mod_names: ' + ', '.join(self.__mod_names),
] ]
def _classes(self) -> Iterable[T]: def _classes(self) -> Iterable[type[T]]:
if self.__classes is None: if self.__classes is None:
import importlib, inspect import importlib
rx: Any|None = None import inspect
rx: Any | None = None
if self.__type_name_filter is not None: if self.__type_name_filter is not None:
rx = re.compile(self.__type_name_filter) rx = re.compile(self.__type_name_filter)
ret: list[Any] = [] ret: list[Any] = []
for mod_name in self.__mod_names: for mod_name in self.__mod_names:
if mod_name != '__main__': if mod_name != '__main__':
importlib.import_module(mod_name) importlib.import_module(mod_name)
for member_name, c in inspect.getmembers(sys.modules[mod_name], inspect.isclass): for member_name, c in inspect.getmembers(
sys.modules[mod_name], inspect.isclass
):
if rx is not None and not re.match(rx, member_name): if rx is not None and not re.match(rx, member_name):
self._debug('o "{}.{}" has wrong name'.format(mod_name, member_name)) self._debug(
'o "{}.{}" has wrong name'.format(mod_name, member_name)
)
continue continue
if inspect.isabstract(c): if inspect.isabstract(c):
self._debug('o "{}.{}" is abstract'.format(mod_name, member_name)) self._debug(
'o "{}.{}" is abstract'.format(mod_name, member_name)
)
continue continue
if self.__type_filter: if self.__type_filter:
for tp in self.__type_filter: for tp in self.__type_filter:
if issubclass(c, tp): if issubclass(c, tp):
break break
self._debug('o "{}.{}" is not of type {}'.format(mod_name, member_name, tp)) self._debug(
'o "{}.{}" is not of type {}'.format(
mod_name, member_name, tp
)
)
else: else:
self._debug('o "{}.{}" doesn\'t match type filter'.format(mod_name, member_name)) self._debug(
'o "{}.{}" doesn\'t match type filter'.format(
mod_name, member_name
)
)
continue continue
self._debug('o "{}.{}" is fine, adding'.format(mod_name, member_name)) self._debug(
'o "{}.{}" is fine, adding'.format(mod_name, member_name)
)
ret.append(c) ret.append(c)
self.__classes = ret self.__classes = ret
return self.__classes return self.__classes

54
src/python/jw/pkg/lib/Uri.py
View file

@ -1,20 +1,21 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING
from functools import cached_property
import copy import copy
from functools import cached_property
from typing import TYPE_CHECKING
if TYPE_CHECKING: if TYPE_CHECKING:
import urllib.parse
from typing import Self from typing import Self
import urllib
# Make sure URIs are interpreted indentically everywhere # Make sure URIs are interpreted indentically everywhere
class Uri: class Uri:
def __assemble(self, scheme: bool, credentials: bool, secure: bool, path: bool) -> str: def __assemble(
self, scheme: bool, credentials: bool, secure: bool, path: bool
) -> str:
ret = '' ret = ''
if scheme: if scheme:
ret += f'{self.protocol}://' ret += f'{self.protocol}://'
@ -33,8 +34,8 @@ class Uri:
def __init__(self, string: str) -> None: def __init__(self, string: str) -> None:
self.__string = string self.__string = string
self.__username: str|None = None self.__username: str | None = None
self.__password: str|None = None self.__password: str | None = None
def __repr__(self) -> str: def __repr__(self) -> str:
return self.full return self.full
@ -45,10 +46,11 @@ class Uri:
@cached_property @cached_property
def __p(self) -> urllib.parse.ParseResult: def __p(self) -> urllib.parse.ParseResult:
from urllib.parse import urlparse from urllib.parse import urlparse
return urlparse(self.__string) return urlparse(self.__string)
@classmethod @classmethod
def pimp(cls, url: str|Self) -> Uri: def pimp(cls, url: str | Self) -> Uri:
if isinstance(url, Uri): if isinstance(url, Uri):
return url return url
return Uri(url) return Uri(url)
@ -69,7 +71,7 @@ class Uri:
return self.scheme.replace('://', '') return self.scheme.replace('://', '')
@property @property
def username(self) -> str|None: def username(self) -> str | None:
if self.__username is None: if self.__username is None:
return self.__p.username return self.__p.username
return self.__username return self.__username
@ -78,7 +80,7 @@ class Uri:
self.__username = username self.__username = username
@property @property
def password(self) -> str|None: def password(self) -> str | None:
if self.__password is None: if self.__password is None:
return self.__p.password return self.__p.password
return self.__password return self.__password
@ -87,15 +89,15 @@ class Uri:
self.__password = password self.__password = password
@cached_property @cached_property
def hostname(self) -> str|None: def hostname(self) -> str | None:
return self.__p.hostname return self.__p.hostname
@cached_property @cached_property
def port(self) -> int|None: def port(self) -> int | None:
return self.__p.port return self.__p.port
@cached_property @cached_property
def port_str(self) -> str|None: def port_str(self) -> str | None:
if self.port is None: if self.port is None:
return None return None
return str(self.port) return str(self.port)
@ -110,11 +112,15 @@ class Uri:
@cached_property @cached_property
def authority(self) -> str: def authority(self) -> str:
return self.__assemble(scheme=False, credentials=True, secure=False, path=False) return self.__assemble(
scheme = False, credentials = True, secure = False, path = False
)
@cached_property @cached_property
def origin(self) -> str: def origin(self) -> str:
return self.__assemble(scheme=False, credentials=False, secure=True, path=False) return self.__assemble(
scheme = False, credentials = False, secure = True, path = False
)
@cached_property @cached_property
def scheme_plus_authority(self) -> str: def scheme_plus_authority(self) -> str:
@ -122,15 +128,21 @@ class Uri:
@cached_property @cached_property
def id(self) -> str: def id(self) -> str:
return self.__assemble(scheme=True, credentials=True, secure=True, path=False) return self.__assemble(
scheme = True, credentials = True, secure = True, path = False
)
@cached_property @cached_property
def full(self) -> str: def full(self) -> str:
return self.__assemble(scheme=True, credentials=True, secure=False, path=True) return self.__assemble(
scheme = True, credentials = True, secure = False, path = True
)
@cached_property @cached_property
def safe_full_with_username(self) -> str: def safe_full_with_username(self) -> str:
return self.__assemble(scheme=True, credentials=True, secure=True, path=True) return self.__assemble(
scheme = True, credentials = True, secure = True, path = True
)
def __new_with_path(self, base: str, path: str) -> Self: def __new_with_path(self, base: str, path: str) -> Self:
ret = copy.deepcopy(self) ret = copy.deepcopy(self)
@ -155,4 +167,4 @@ class Uri:
return self.__new_with_path(self.__string, path) return self.__new_with_path(self.__string, path)
def new_replace_path(self, path: str) -> Self: def new_replace_path(self, path: str) -> Self:
return self.__new_with_path(self.schema_plus_authority, path) return self.__new_with_path(self.scheme_plus_authority, path)

165
src/python/jw/pkg/lib/base.py
View file

@ -1,12 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from enum import Enum, auto import os
from typing import NamedTuple, TypeAlias, TYPE_CHECKING
if TYPE_CHECKING: from enum import Enum, auto
from typing import Type from typing import NamedTuple, TypeAlias
class InputMode(Enum): class InputMode(Enum):
Interactive = auto() Interactive = auto()
@ -16,32 +13,158 @@ class InputMode(Enum):
Input: TypeAlias = InputMode | bytes | str Input: TypeAlias = InputMode | bytes | str
class Result(NamedTuple): class Result:
stdout: str|None def __init__(
stderr: str|None self,
status: int|None stdout: bytes | None,
stderr: bytes | None,
status: int,
encoding: str = 'UTF-8',
strip: bool = True,
cmd: list[str] | None = None,
wd: str | None = None,
) -> None:
self.__stdout = stdout
self.__stderr = stderr
self.__status = status
self.__encoding = encoding
self.__strip = strip
self.__cmd = cmd
self.__wd = wd
def decode(self, encoding='UTF-8', errors='replace') -> Result: def __decode(self, stdxxx: bytes | None) -> str | None:
return Result( if stdxxx is None:
self.stdout.decode(encoding, errors=errors) if self.stdout is not None else None, return None
self.stderr.decode(encoding, errors=errors) if self.stderr is not None else None, ret = stdxxx.decode(self.encoding)
self.status if self.strip:
) return ret.strip()
return ret
@property
def status(self) -> int | None:
return self.__status
@property
def encoding(self) -> str:
return self.__encoding
@encoding.setter
def encoding(self, value: str) -> None:
self.__encoding = value
@property
def strip(self) -> bool:
return self.__strip
@strip.setter
def strip(self, value: bool) -> None:
self.__strip = value
@property
def cmd(self) -> list[str] | None:
return self.__cmd
@cmd.setter
def cmd(self, value: list[str]) -> None:
self.__cmd = value
@property
def wd(self) -> str | None:
return self.__wd
@wd.setter
def wd(self, value: str) -> None:
self.__wd = value
def matches_error(self, pattern: str) -> bool:
if self.status == 0:
return False
err = self.stderr_str
if err is None:
return False
import re
return re.search(pattern, err) is not None
def __summarize(self, cmd: list[str] | None, wd: str | None = None) -> str:
if cmd is None:
cmd = self.__cmd
call = ''
if cmd is not None:
from .util import pretty_cmd
if wd is None:
wd = self.__wd
call = f'"{pretty_cmd(cmd, wd)}" '
ret = f'Command {call}has exited with status {self.__status}'
call = pretty_cmd(cmd, wd)
if self.status != 0:
ret += f' -> stderr="{self.__stderr!r}"'
else:
if self.__stdout:
ret += f' -> stdout has {len(self.__stdout)} bytes'
else:
ret += ' -> stdout = None'
return ret
def summarize(self, cmd: list[str] | None = None, wd: str | None = None) -> str:
return self.__summarize(cmd, wd)
@property
def summary(self) -> str:
return self.__summarize(None, None)
@property
def stdout(self) -> bytes:
if self.__stdout is None:
raise Exception(f'Result has no standard output stream: {self.summary}')
return self.__stdout
@property
def stdout_or_none(self) -> bytes | None:
return self.__stdout
@property
def stdout_str_or_none(self) -> str | None:
return self.__decode(self.__stdout)
@property
def stdout_str(self) -> str:
return self.stdout.decode(self.__encoding)
@property
def stderr(self) -> bytes:
if self.__stderr is None:
raise Exception(f'Result has no standard error stream: {self.summary}')
return self.__stderr
@property
def stderr_or_none(self) -> bytes | None:
return self.__stderr
@property
def stderr_str_or_none(self) -> str | None:
return self.__decode(self.__stderr)
@property
def stderr_str(self) -> str:
return self.stderr.decode(self.__encoding)
class StatResult(NamedTuple): class StatResult(NamedTuple):
mode: int mode: int
owner: str owner: str
group: str group: str
size: int size: int
atime: int atime: float
mtime: int mtime: float
ctime: int ctime: float
@classmethod @classmethod
def from_os(cls, rhs: os.stat_result) -> StatResult: def from_os(cls, rhs: os.stat_result) -> StatResult:
import pwd, grp import grp
import pwd
return StatResult( return StatResult(
rhs.st_mode, rhs.st_mode,
pwd.getpwuid(rhs.st_uid).pw_name, pwd.getpwuid(rhs.st_uid).pw_name,

38
src/python/jw/pkg/lib/distros/arch/Distro.py
View file

@ -1,53 +1,65 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ...Distro import Distro as Base from ...Distro import Distro as Base
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Iterable from typing import Iterable
from ...base import Result from ...base import Result
from ...Package import Package from ...Package import Package
class Distro(Base): class Distro(Base):
async def pacman(self, args: list[str], verbose: bool=True, sudo: bool=True) -> Result: async def pacman(
self, args: list[str], verbose: bool = True, sudo: bool = True
) -> Result:
cmd = ['/usr/bin/pacman'] cmd = ['/usr/bin/pacman']
if not self.interactive: if not self.interactive:
cmd.extend(['--noconfirm']) cmd.extend(['--noconfirm'])
cmd.extend(args) cmd.extend(args)
if sudo: if sudo:
return await self.sudo(cmd, verbose=verbose) return await self.sudo(cmd, verbose = verbose)
return await self.run(cmd, verbose=verbose) return await self.run(cmd, verbose = verbose)
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
async def _ref(self) -> None: async def _ref(self) -> None:
raise NotImplementedError('distro refresh is not yet implemented for Arch-like distributions') raise NotImplementedError(
'distro refresh is not yet implemented for Arch-like distributions'
)
async def _dup(self, download_only: bool) -> None: async def _dup(self, download_only: bool) -> None:
args = ['-Su'] args = ['-Su']
if args.download_only: if download_only:
args.append('-w') args.append('-w')
return await self.pacman(args) await self.pacman(args)
async def _reboot_required(self, verbose: bool) -> bool: async def _reboot_required(self, verbose: bool) -> bool:
raise NotImplementedError('distro reboot-required is not yet implemented for Arch-like distributions') raise NotImplementedError(
'distro reboot-required is not yet implemented for Arch-like distributions'
)
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]: async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
raise NotImplementedError('distro select is not yet implemented for Arch-like distributions') raise NotImplementedError(
'distro select is not yet implemented for Arch-like distributions'
)
async def _install(self, names: Iterable[str], only_update: bool) -> None: async def _install(self, names: Iterable[str], only_update: bool) -> None:
if only_update: if only_update:
raise NotImplementedError('--only-update is not yet implemented for pacman') raise NotImplementedError('--only-update is not yet implemented for pacman')
args = ['-S', '--needed'] args = ['-S', '--needed']
args.extend(args.packages) args.extend(names)
await self.pacman(args) await self.pacman(args)
async def _delete(self, names: Iterable[str]) -> None: async def _delete(self, names: Iterable[str]) -> None:
raise NotImplementedError('distro delete not yet implemented for Arch-like distributions') raise NotImplementedError(
'distro delete not yet implemented for Arch-like distributions'
)
async def _pkg_files(self, name: str) -> Iterable[str]: async def _pkg_files(self, name: str) -> Iterable[str]:
raise NotImplementedError('distro pkg ls yet implemented for Arch-like distributions') raise NotImplementedError(
'distro pkg ls yet implemented for Arch-like distributions'
)

43
src/python/jw/pkg/lib/distros/debian/Distro.py
View file

@ -1,47 +1,52 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING
import os import os
from ...log import * from typing import TYPE_CHECKING
from ...Distro import Distro as Base from ...Distro import Distro as Base
from ...pm.dpkg import run_dpkg, run_dpkg_query, query_packages, list_files from ...log import NOTICE, log
from ...pm.dpkg import list_files, query_packages, run_dpkg
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Iterable from typing import Iterable
from ...base import Result from ...base import Result
from ...Package import Package from ...Package import Package
class Distro(Base): class Distro(Base):
async def apt_get(self, args: list[str], verbose: bool=True, sudo: bool=True): async def apt_get(
self, args: list[str], verbose: bool = True, sudo: bool = True
) -> Result:
cmd = ['/usr/bin/apt-get'] cmd = ['/usr/bin/apt-get']
mod_env_cmd = None mod_env_cmd = None
if not self.interactive: if not self.interactive:
cmd.extend(['--yes', '--quiet']) cmd.extend(['--yes', '--quiet'])
mod_env_cmd = { 'DEBIAN_FRONTEND': 'noninteractive' } mod_env_cmd = {'DEBIAN_FRONTEND': 'noninteractive'}
cmd.extend(args) cmd.extend(args)
if sudo: return (
return await self.sudo(cmd, verbose=verbose, mod_env_cmd=mod_env_cmd) await
return await self.run(cmd, verbose=verbose) self.sudo(cmd, verbose = verbose, mod_env_cmd = mod_env_cmd, throw = True)
if sudo else await self.run(cmd, verbose = verbose)
)
async def dpkg(self, *args, **kwargs): async def dpkg(self, *args, **kwargs) -> str:
return await run_dpkg(*args, ec=self.ctx, **kwargs) kwargs.setdefault('ec', self.ctx)
return await run_dpkg(*args, **kwargs)
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
async def _ref(self) -> None: async def _ref(self) -> None:
return await self.apt_get(['update']) await self.apt_get(['update'])
async def _dup(self, download_only: bool) -> None: async def _dup(self, download_only: bool) -> None:
args: list[str] = [] args: list[str] = []
if download_only: if download_only:
args.append('--download-only') args.append('--download-only')
args.append('upgrade') args.append('upgrade')
return await self.apt_get(args) await self.apt_get(args)
async def _reboot_required(self, verbose: bool) -> bool: async def _reboot_required(self, verbose: bool) -> bool:
reboot_required = '/run/reboot_required' reboot_required = '/run/reboot_required'
@ -56,11 +61,11 @@ class Distro(Base):
print(content.strip()) print(content.strip())
return True return True
if verbose: if verbose:
log(NOTICE, f'No. {reboot_required} doesn\'t exist.') log(NOTICE, f"No. {reboot_required} doesn't exist.")
return False return False
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]: async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
return await query_packages(names, ec=self.ctx) return await query_packages(names, ec = self.ctx)
async def _install(self, names: Iterable[str], only_update: bool) -> None: async def _install(self, names: Iterable[str], only_update: bool) -> None:
args = ['install'] args = ['install']
@ -68,10 +73,10 @@ class Distro(Base):
args.append('--only-upgrade') args.append('--only-upgrade')
args.append('--no-install-recommends') args.append('--no-install-recommends')
args.extend(names) args.extend(names)
return await self.apt_get(args) await self.apt_get(args)
async def _delete(self, names: Iterable[str]) -> None: async def _delete(self, names: Iterable[str]) -> None:
return await self.dpkg(['-P', *names], sudo=True) await self.dpkg(['-P', *names], sudo = True)
async def _pkg_files(self, name: str) -> Iterable[str]: async def _pkg_files(self, name: str) -> Iterable[str]:
return await list_files(name, ec=self.ctx) return await list_files(name, ec = self.ctx)

45
src/python/jw/pkg/lib/distros/suse/Distro.py
View file

@ -1,62 +1,71 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ...Distro import Distro as Base from ...Distro import Distro as Base
from ...pm.rpm import run_rpm, query_packages, list_files from ...pm.rpm import list_files, query_packages, run_rpm
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Iterable from typing import Iterable
from ...base import Result from ...base import Result
from ...ExecContext import ExecContext
from ...Package import Package from ...Package import Package
class Distro(Base): class Distro(Base):
async def zypper(self, args: list[str], verbose: bool=True, sudo: bool=True) -> Result: async def zypper(
self, args: list[str], verbose: bool = True, sudo: bool = True
) -> Result:
cmd = ['/usr/bin/zypper'] cmd = ['/usr/bin/zypper']
if not self.interactive: if not self.interactive:
cmd.extend(['--non-interactive', '--gpg-auto-import-keys', '--no-gpg-checks']) cmd.extend(
['--non-interactive', '--gpg-auto-import-keys', '--no-gpg-checks']
)
cmd.extend(args) cmd.extend(args)
if sudo: return (
return await self.sudo(cmd, verbose=verbose) await self.sudo(cmd, verbose = verbose)
return await self.run(cmd, verbose=verbose) if sudo else await self.run(cmd, verbose = verbose)
)
async def rpm(self, *args, **kwargs) -> Result: async def rpm(self, *args, ec: ExecContext | None = None, **kwargs) -> str:
return await run_rpm(*args, ec=self.ctx, **kwargs) if ec is None:
ec = self.ctx
kwargs['ec'] = ec
return await run_rpm(*args, **kwargs)
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
async def _ref(self) -> None: async def _ref(self) -> None:
return await self.zypper(['refresh']) await self.zypper(['refresh'])
async def _dup(self, download_only: bool) -> None: async def _dup(self, download_only: bool) -> None:
args = ['dup', '--force-resolution', '--auto-agree-with-licenses'] args = ['dup', '--force-resolution', '--auto-agree-with-licenses']
if download_only: if download_only:
args.append('--download-only') args.append('--download-only')
return await self.zypper(args) await self.zypper(args)
async def _reboot_required(self, verbose: bool) -> bool: async def _reboot_required(self, verbose: bool) -> bool:
opts = [] opts = []
if not verbose: if not verbose:
pass pass
#opts.append('--quiet') # opts.append('--quiet')
opts.append('needs-rebooting') opts.append('needs-rebooting')
stdout, stderr, ret = await self.zypper(opts, sudo=False, verbose=verbose) ret = await self.zypper(opts, sudo = False, verbose = verbose)
if ret != 0: if ret != 0:
return True return True
return False return False
async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]: async def _select_by_name(self, names: Iterable[str]) -> Iterable[Package]:
return await query_packages(names, ec=self.ctx) return await query_packages(names, ec = self.ctx)
async def _install(self, names: Iterable[str], only_update: bool) -> None: async def _install(self, names: Iterable[str], only_update: bool) -> None:
cmd = 'update' if only_update else 'install' cmd = 'update' if only_update else 'install'
return await self.zypper([cmd, *names]) await self.zypper([cmd, *names])
async def _delete(self, names: Iterable[str]) -> None: async def _delete(self, names: Iterable[str]) -> None:
return await self.rpm(['-e', *names], sudo=True) await self.rpm(['-e', *names], sudo = True)
async def _pkg_files(self, name: str) -> Iterable[str]: async def _pkg_files(self, name: str) -> Iterable[str]:
return await list_files(name, ec=self.ctx) return await list_files(name, ec = self.ctx)

35
src/python/jw/pkg/lib/ec/Curl.py
View file

@ -1,32 +1,35 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ..FileContext import FileContext as Base
from ..base import Result from ..base import Result
from ..FileContext import FileContext as Base
if TYPE_CHECKING: if TYPE_CHECKING:
from ..ExecContext import ExecContext from ..ExecContext import ExecContext
from ..Uri import Uri from ..Uri import Uri
from .Local import Local
class Curl(Base): class Curl(Base):
def __init__(self, uri: str|Uri, *args, ec: ExecContext|None=None, **kwargs) -> None: def __init__(
super().__init__(uri=uri, *args, **kwargs) self, uri: str | Uri, *args, ec: ExecContext | None = None, **kwargs
self.__ec: ExecContext|None = ec ) -> None:
if ec is None:
def __local() -> Local:
from .Local import Local from .Local import Local
self.__ec = Local(interactive=False, *args, **kwargs)
return Local(interactive = False, *args, **kwargs)
# MyPy complains for reasons I don't understand:
# E: "__init__" of "FileContext" gets multiple values for keyword # argument
# "uri" [misc]
super().__init__(uri = uri, *args, **kwargs) # type: ignore[misc]
self.__ec = ec if ec else __local()
async def _get( async def _get(
self, self, path: str, wd: str | None, throw: bool, verbose: bool | None, title: str
path: str,
wd: str|None,
throw: bool,
verbose: bool|None,
title: str
) -> Result: ) -> Result:
cmd = ['curl'] cmd = ['curl']
if verbose is None: if verbose is None:
@ -38,5 +41,5 @@ class Curl(Base):
path = wd + '/' + path path = wd + '/' + path
if not len(path) or path[0] != '/': if not len(path) or path[0] != '/':
path = '/' + path path = '/' + path
cmd.append(self.url.to_string + self._chroot(path)) cmd.append(self.uri.to_string + self._chroot(path))
return await self.__ec.run(cmd, throw=throw, verbose=verbose) return await self.__ec.run(cmd, throw = throw, verbose = verbose)

92
src/python/jw/pkg/lib/ec/Local.py
View file

@ -1,91 +1,96 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING
import os, sys, subprocess, asyncio, pwd, grp, stat import asyncio
import grp
import os
import pwd
import sys
from functools import cache from functools import cache
from typing import TYPE_CHECKING
from ..ExecContext import ExecContext as Base
from ..base import Result, StatResult from ..base import Result, StatResult
from ..ExecContext import ExecContext as Base
from ..log import * from ..log import ERR, NOTICE, log
from ..util import pretty_cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ..Uri import Uri from ..Uri import Uri
class Local(Base): class Local(Base):
def __init__(self, uri: str|Uri='local', *args, **kwargs) -> None: def __init__(self, uri: str | Uri = 'local', *args, **kwargs) -> None:
super().__init__(uri, *args, **kwargs) super().__init__(uri, *args, **kwargs)
@cache @cache
def _username(self) -> str: def _username(self) -> str:
return pwd.getpwuid(os.getuid()).pw_name, return pwd.getpwuid(os.getuid()).pw_name
async def _run( async def _run(
self, self,
cmd: list[str], cmd: list[str],
wd: str|None, wd: str | None,
verbose: bool, verbose: bool,
cmd_input: bytes|None, cmd_input: bytes | None,
mod_env: dict[str, str]|None, mod_env: dict[str, str] | None,
interactive: bool, interactive: bool,
log_prefix: str log_prefix: str,
) -> Result: ) -> Result:
def __log(prio, *args, verbose=verbose): def __log(prio, *args, verbose = verbose):
if verbose: if verbose:
log(prio, log_prefix, *args) log(prio, log_prefix, *args)
def __make_pty_reader(collector: list[bytes], enc_for_verbose: str): def __make_pty_reader(collector: list[bytes], enc_for_verbose: str):
def _read(fd): def _read(fd):
ret = os.read(fd, 1024) ret = os.read(fd, 1024)
if not ret: if not ret:
return ret return ret
collector.append(ret) collector.append(ret)
return ret return ret
return _read return _read
cwd: str|None = None cwd: str | None = None
if wd is not None: if wd is not None:
cwd = os.getcwd() cwd = os.getcwd()
os.chdir(wd) os.chdir(wd)
try: try:
# -- interactive mode # -- interactive mode
if interactive: if interactive:
import pty import pty
def _spawn(): def _spawn():
# Apply env in PTY mode by temporarily updating os.environ around spawn. # Apply env in PTY mode by temporarily updating os.environ
# around spawn.
if mod_env: if mod_env:
old_env = os.environ.copy() old_env = os.environ.copy()
try: try:
os.environ.update(mod_env) os.environ.update(mod_env)
return pty.spawn(cmd, master_read=reader) return pty.spawn(cmd, master_read = reader)
finally: finally:
os.environ.clear() os.environ.clear()
os.environ.update(old_env) os.environ.update(old_env)
return pty.spawn(cmd, master_read=reader) return pty.spawn(cmd, master_read = reader)
stdout_chunks: list[bytes] = [] stdout_chunks: list[bytes] = []
enc_for_verbose = sys.stdout.encoding or "utf-8" enc_for_verbose = sys.stdout.encoding or 'utf-8'
reader = __make_pty_reader(stdout_chunks, enc_for_verbose) reader = __make_pty_reader(stdout_chunks, enc_for_verbose)
exit_code = await asyncio.to_thread(_spawn) exit_code = await asyncio.to_thread(_spawn)
# PTY merges stdout/stderr # PTY merges stdout/stderr
stdout = b"".join(stdout_chunks) if stdout_chunks else None stdout = b''.join(stdout_chunks) if stdout_chunks else None
return Result(stdout, None, exit_code) return Result(stdout, None, exit_code)
# -- non-interactive mode # -- non-interactive mode
stdin = asyncio.subprocess.DEVNULL if cmd_input is None else asyncio.subprocess.PIPE
stdin = (
asyncio.subprocess.DEVNULL
if cmd_input is None else asyncio.subprocess.PIPE
)
if mod_env: if mod_env:
new_env = os.environ.copy() new_env = os.environ.copy()
@ -94,21 +99,21 @@ class Local(Base):
proc = await asyncio.create_subprocess_exec( proc = await asyncio.create_subprocess_exec(
*cmd, *cmd,
stdin=stdin, stdin = stdin,
stdout=asyncio.subprocess.PIPE, stdout = asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE, stderr = asyncio.subprocess.PIPE,
env=mod_env, env = mod_env,
) )
stdout_parts: list[bytes] = [] stdout_parts: list[bytes] = []
stderr_parts: list[bytes] = [] stderr_parts: list[bytes] = []
# -- decoding for verbose output in pipe mode # -- decoding for verbose output in pipe mode
stdout_log_enc = sys.stdout.encoding or "utf-8" stdout_log_enc = sys.stdout.encoding or 'utf-8'
stderr_log_enc = sys.stderr.encoding or "utf-8" stderr_log_enc = sys.stderr.encoding or 'utf-8'
async def read_stream(stream, prio, collector: list[bytes], log_enc: str): async def read_stream(stream, prio, collector: list[bytes], log_enc: str):
buf = b"" buf = b''
while True: while True:
chunk = await stream.read(4096) chunk = await stream.read(4096)
if not chunk: if not chunk:
@ -116,12 +121,12 @@ class Local(Base):
collector.append(chunk) collector.append(chunk)
if verbose: if verbose:
buf += chunk buf += chunk
while b"\n" in buf: while b'\n' in buf:
line, buf = buf.split(b"\n", 1) line, buf = buf.split(b'\n', 1)
__log(prio, line.decode(log_enc, errors="replace")) __log(prio, line.decode(log_enc, errors = 'replace'))
if verbose and buf: if verbose and buf:
# flush trailing partial line (no newline) # flush trailing partial line (no newline)
__log(prio, buf.decode(log_enc, errors="replace")) __log(prio, buf.decode(log_enc, errors = 'replace'))
tasks = [ tasks = [
asyncio.create_task( asyncio.create_task(
@ -132,7 +137,8 @@ class Local(Base):
), ),
] ]
if stdin is asyncio.subprocess.PIPE: if (cmd_input is not None and stdin is asyncio.subprocess.PIPE
and proc.stdin is not None):
proc.stdin.write(cmd_input) proc.stdin.write(cmd_input)
await proc.stdin.drain() await proc.stdin.drain()
proc.stdin.close() proc.stdin.close()
@ -140,8 +146,8 @@ class Local(Base):
exit_code = await proc.wait() exit_code = await proc.wait()
await asyncio.gather(*tasks) await asyncio.gather(*tasks)
stdout = b"".join(stdout_parts) if stdout_parts else None stdout = b''.join(stdout_parts) if stdout_parts else None
stderr = b"".join(stderr_parts) if stderr_parts else None stderr = b''.join(stderr_parts) if stderr_parts else None
return Result(stdout, stderr, exit_code) return Result(stdout, stderr, exit_code)
@ -153,7 +159,9 @@ class Local(Base):
os.unlink(path) os.unlink(path)
async def _erase(self, path: str) -> None: async def _erase(self, path: str) -> None:
if os.isdir(path): if os.path.isdir(path):
import shutil
shutil.rmtree(path) shutil.rmtree(path)
return return
os.unlink(path) os.unlink(path)
@ -165,12 +173,12 @@ class Local(Base):
os.mkdir(name, mode) os.mkdir(name, mode)
async def _stat(self, path: str, follow_symlinks: bool) -> StatResult: async def _stat(self, path: str, follow_symlinks: bool) -> StatResult:
return StatResult.from_os(os.stat(path, follow_symlinks=follow_symlinks)) return StatResult.from_os(os.stat(path, follow_symlinks = follow_symlinks))
async def _file_exists(self, path: str) -> bool: async def _file_exists(self, path: str) -> bool:
return os.path.exists(path) return os.path.exists(path)
async def _chown(self, path: str, owner: str|None, group: str|None) -> None: async def _chown(self, path: str, owner: str | None, group: str | None) -> None:
uid = pwd.getpwnam(owner).pw_uid if owner else -1 uid = pwd.getpwnam(owner).pw_uid if owner else -1
gid = grp.getgrnam(group).gr_gid if group else -1 gid = grp.getgrnam(group).gr_gid if group else -1
os.chown(path, uid, gid) os.chown(path, uid, gid)
@ -179,6 +187,6 @@ class Local(Base):
os.chmod(path, mode) os.chmod(path, mode)
async def _is_dir(self, path: str, follow_symlinks: bool) -> bool: async def _is_dir(self, path: str, follow_symlinks: bool) -> bool:
if (not follow_symlinks) and os.islink(path): if (not follow_symlinks) and os.path.islink(path):
return False return False
return os.path.isdir(path) return os.path.isdir(path)

84
src/python/jw/pkg/lib/ec/SSHClient.py
View file

@ -1,16 +1,16 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import Any, TYPE_CHECKING import abc
import os
import pwd
import sys
import os, abc, sys, pwd
from enum import Flag, auto from enum import Flag, auto
from typing import TYPE_CHECKING
from ..util import pretty_cmd
from ..log import *
from ..base import Result from ..base import Result
from ..ExecContext import ExecContext from ..ExecContext import ExecContext
from ..log import DEBUG, ERR, INFO, NOTICE, WARNING, log
from ..Uri import Uri from ..Uri import Uri
if TYPE_CHECKING: if TYPE_CHECKING:
@ -24,48 +24,50 @@ class SSHClient(ExecContext):
ModEnv = auto() ModEnv = auto()
Wd = auto() Wd = auto()
def __init__(self, uri: Uri|str, caps: Caps=Caps(0), *args, **kwargs) -> None: def __init__(self, uri: Uri | str, caps: Caps = Caps(0), *args, **kwargs) -> None:
uri = Uri.pimp(uri) uri = Uri.pimp(uri)
if uri.username is None: if uri.username is None:
uri.set_username(pwd.getpwuid(os.getuid()).pw_name) uri.set_username(pwd.getpwuid(os.getuid()).pw_name)
super().__init__(uri=uri, *args, **kwargs) super().__init__(uri = uri, *args, **kwargs)
self.__caps = caps self.__caps = caps
@abc.abstractmethod @abc.abstractmethod
async def _run_ssh( async def _run_ssh(
self,
cmd: list[str], cmd: list[str],
wd: str|None, wd: str | None,
verbose: bool, verbose: bool,
cmd_input: bytes|None, cmd_input: bytes | None,
mod_env: dict[str, str]|None, mod_env: dict[str, str] | None,
interactive: bool, interactive: bool,
log_prefix: str log_prefix: str,
) -> Result: ) -> Result:
pass pass
async def _run( async def _run(
self, self,
cmd: list[str], cmd: list[str],
wd: str|None, wd: str | None,
verbose: bool, verbose: bool,
cmd_input: bytes|None, cmd_input: bytes | None,
mod_env: dict[str, str]|None, mod_env: dict[str, str] | None,
interactive: bool, interactive: bool,
log_prefix: str log_prefix: str,
) -> Result: ) -> Result:
def __log(prio: int, *args): def __log(prio: int, *args):
log(prio, log_prefix, *args) log(prio, log_prefix, *args)
def __log_block(prio: int, title: str, block: str): def __log_block(prio: int, title: str, block: bytes | str | None):
if self.__caps & self.Caps.LogOutput: if self.__caps & self.Caps.LogOutput:
return return
if not block: if not block:
return return
encoding = sys.stdout.encoding or 'utf-8' if isinstance(block, bytes):
block = block.decode(encoding).strip() encoding = sys.stdout.encoding or 'utf-8'
if not block: block = block.decode(encoding).strip()
return # Needed to pacify pyright: block can't be anything else at this point
assert isinstance(block, str)
delim = f'---- {title} ----' delim = f'---- {title} ----'
__log(prio, f',{delim}') __log(prio, f',{delim}')
for line in block.splitlines(): for line in block.splitlines():
@ -79,44 +81,49 @@ class SSHClient(ExecContext):
raise NotImplementedError('Interactive SSH is not yet implemented') raise NotImplementedError('Interactive SSH is not yet implemented')
if mod_env is not None and not self.__caps & self.Caps.ModEnv: if mod_env is not None and not self.__caps & self.Caps.ModEnv:
raise NotImplementedError('Passing an environment to SSH commands is not yet implemented') raise NotImplementedError(
'Passing an environment to SSH commands is not yet implemented'
)
ret = await self._run_ssh( ret = await self._run_ssh(
cmd=cmd, cmd = cmd,
wd=wd, wd = wd,
verbose=verbose, verbose = verbose,
cmd_input=cmd_input, cmd_input = cmd_input,
mod_env=mod_env, mod_env = mod_env,
interactive=interactive, interactive = interactive,
log_prefix=log_prefix log_prefix = log_prefix,
) )
if verbose: if verbose:
__log_block(NOTICE, 'stdout', ret.stdout) __log_block(NOTICE, 'stdout', ret.stdout_str_or_none)
__log_block(NOTICE, 'stderr', ret.stderr) __log_block(NOTICE, 'stderr', ret.stderr_str_or_none)
if ret.status != 0: if ret.status != 0:
__log(WARNING, f'Exit code {ret.status}') __log(WARNING, f'Exit code {ret.status}')
return ret return ret
@property @property
def hostname(self) -> str|None: def hostname(self) -> str | None:
return self.uri.hostname return self.uri.hostname
@property @property
def port(self) -> int|None: def port(self) -> int | None:
return self.uri.port return self.uri.port
@property @property
def username(self) -> str|None: def username(self) -> str | None:
return self.uri.username return self.uri.username
@property @property
def password(self) -> str|None: def password(self) -> str | None:
return self.uri.password return self.uri.password
def ssh_client(*args, type: str|list[str]|None=None, **kwargs) -> SSHClient: # export def ssh_client(
*args, type: str | list[str] | None = None, **kwargs
) -> SSHClient: # export
from importlib import import_module from importlib import import_module
errors: list[str] = [] errors: list[str] = []
if type is None: if type is None:
val = os.getenv('JW_DEFAULT_SSH_CLIENT') val = os.getenv('JW_DEFAULT_SSH_CLIENT')
@ -128,11 +135,12 @@ def ssh_client(*args, type: str|list[str]|None=None, **kwargs) -> SSHClient: # e
type = [type] type = [type]
for name in type: for name in type:
try: try:
ret = getattr(import_module(f'jw.pkg.lib.ec.ssh.{name}'), name)(*args, **kwargs) ret = getattr(import_module(f'jw.pkg.lib.ec.ssh.{name}'),
name)(*args, **kwargs)
log(INFO, f'Using SSH-client "{name}"') log(INFO, f'Using SSH-client "{name}"')
return ret return ret
except Exception as e: except Exception as e:
msg = f'Can\'t instantiate SSH client class {name} ({str(e)})' msg = f"Can't instantiate SSH client class {name} ({str(e)})"
errors.append(msg) errors.append(msg)
log(DEBUG, f'{msg}, trying next') log(DEBUG, f'{msg}, trying next')
msg = f'No working SSH clients for {" ".join([str(arg) for arg in args])}' msg = f'No working SSH clients for {" ".join([str(arg) for arg in args])}'

55
src/python/jw/pkg/lib/ec/ssh/AsyncSSH.py
View file

@ -1,11 +1,15 @@
# -*- coding: utf-8 -*- import asyncio
import os
import shlex
import shutil
import signal
import sys
import os, sys, shlex, asyncio, asyncssh, shutil, signal import asyncssh
from ...log import *
from ...base import Result from ...base import Result
from ...log import DEBUG, ERR, NOTICE, log
from ..SSHClient import SSHClient as Base from ..SSHClient import SSHClient as Base
from .util import join_cmd from .util import join_cmd
_USE_DEFAULT_KNOWN_HOSTS = object() _USE_DEFAULT_KNOWN_HOSTS = object()
@ -25,15 +29,18 @@ class AsyncSSH(Base):
super().__init__( super().__init__(
uri, uri,
caps = self.Caps.LogOutput | self.Caps.Wd | self.Caps.Interactive | self.Caps.ModEnv, caps = self.Caps.LogOutput
**kwargs | self.Caps.Wd
| self.Caps.Interactive
| self.Caps.ModEnv,
**kwargs,
) )
self.__client_keys = client_keys self.__client_keys = client_keys
self.__known_hosts = known_hosts self.__known_hosts = known_hosts
self.__term_type = term_type or os.environ.get('TERM', 'xterm') self.__term_type = term_type or os.environ.get('TERM', 'xterm')
self.__connect_timeout = connect_timeout self.__connect_timeout = connect_timeout
self.__conn: asyncssh.SSHClientConnection|None = None self.__conn: asyncssh.SSHClientConnection | None = None
async def _open(self) -> None: async def _open(self) -> None:
await super()._open() await super()._open()
@ -48,7 +55,7 @@ class AsyncSSH(Base):
log(DEBUG, f'Failed to close connection ({str(e)}, ignored)') log(DEBUG, f'Failed to close connection ({str(e)}, ignored)')
self.__conn = None self.__conn = None
def _connect_kwargs(self, hide_secrets: bool=False) -> dict: def _connect_kwargs(self, hide_secrets: bool = False) -> dict:
kwargs: dict = { kwargs: dict = {
'host': self.hostname, 'host': self.hostname,
'port': self.port, 'port': self.port,
@ -72,7 +79,7 @@ class AsyncSSH(Base):
except Exception as e: except Exception as e:
msg = f'-------------------- Failed to connect ({str(e)})' msg = f'-------------------- Failed to connect ({str(e)})'
log(ERR, ',', msg) log(ERR, ',', msg)
for key, val in self._connect_kwargs(hide_secrets=True).items(): for key, val in self._connect_kwargs(hide_secrets = True).items():
log(ERR, f'| {key:<20} = {val}') log(ERR, f'| {key:<20} = {val}')
log(ERR, '`', msg) log(ERR, '`', msg)
raise raise
@ -94,10 +101,13 @@ class AsyncSSH(Base):
@staticmethod @staticmethod
def _get_local_term_size() -> tuple[int, int, int, int]: def _get_local_term_size() -> tuple[int, int, int, int]:
cols, rows = shutil.get_terminal_size(fallback=(80, 24)) cols, rows = shutil.get_terminal_size(fallback = (80, 24))
xpixel = ypixel = 0 xpixel = ypixel = 0
try: try:
import fcntl, termios, struct import fcntl
import struct
import termios
packed = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, b'\0' * 8) packed = fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, b'\0' * 8)
rows2, cols2, xpixel, ypixel = struct.unpack('HHHH', packed) rows2, cols2, xpixel, ypixel = struct.unpack('HHHH', packed)
if cols2 > 0 and rows2 > 0: if cols2 > 0 and rows2 > 0:
@ -126,9 +136,9 @@ class AsyncSSH(Base):
buf += chunk buf += chunk
while b'\n' in buf: while b'\n' in buf:
line, buf = buf.split(b'\n', 1) line, buf = buf.split(b'\n', 1)
log(prio, log_prefix, line.decode(log_enc, errors='replace')) log(prio, log_prefix, line.decode(log_enc, errors = 'replace'))
if verbose and buf: if verbose and buf:
log(prio, log_prefix, buf.decode(log_enc, errors='replace')) log(prio, log_prefix, buf.decode(log_enc, errors = 'replace'))
async def _run_interactive_on_conn( async def _run_interactive_on_conn(
self, self,
@ -222,7 +232,8 @@ class AsyncSSH(Base):
sys.stderr.flush() sys.stderr.flush()
try: try:
import termios, tty import termios
import tty
old_tty_state = termios.tcgetattr(stdin_fd) old_tty_state = termios.tcgetattr(stdin_fd)
tty.setraw(stdin_fd) tty.setraw(stdin_fd)
@ -257,7 +268,9 @@ class AsyncSSH(Base):
exit_code = completed.exit_status exit_code = completed.exit_status
if exit_code is None: if exit_code is None:
exit_code = completed.returncode if completed.returncode is not None else -1 exit_code = (
completed.returncode if completed.returncode is not None else -1
)
stdout = b''.join(stdout_parts) if stdout_parts else None stdout = b''.join(stdout_parts) if stdout_parts else None
return Result(stdout, None, exit_code) return Result(stdout, None, exit_code)
@ -278,6 +291,7 @@ class AsyncSSH(Base):
if old_tty_state is not None: if old_tty_state is not None:
try: try:
import termios import termios
termios.tcsetattr(stdin_fd, termios.TCSADRAIN, old_tty_state) termios.tcsetattr(stdin_fd, termios.TCSADRAIN, old_tty_state)
except Exception: except Exception:
pass pass
@ -331,7 +345,7 @@ class AsyncSSH(Base):
await proc.stdin.drain() await proc.stdin.drain()
proc.stdin.write_eof() proc.stdin.write_eof()
completed = await proc.wait(check=False) completed = await proc.wait(check = False)
await task await task
exit_code = completed.exit_status exit_code = completed.exit_status
@ -346,14 +360,13 @@ class AsyncSSH(Base):
cmd: list[str], cmd: list[str],
wd: str | None, wd: str | None,
verbose: bool, verbose: bool,
cmd_input: str | None, cmd_input: bytes | None,
mod_env: dict[str, str] | None, mod_env: dict[str, str] | None,
interactive: bool, interactive: bool,
log_prefix: str, log_prefix: str,
) -> Result: ) -> Result:
try: try:
if interactive: if interactive:
if self._has_local_tty(): if self._has_local_tty():
return await self._run_interactive_on_conn( return await self._run_interactive_on_conn(
@ -421,7 +434,7 @@ class AsyncSSH(Base):
await proc.stdin.drain() await proc.stdin.drain()
proc.stdin.write_eof() proc.stdin.write_eof()
completed = await proc.wait(check=False) completed = await proc.wait(check = False)
await asyncio.gather(*tasks) await asyncio.gather(*tasks)
stdout = b''.join(stdout_parts) if stdout_parts else None stdout = b''.join(stdout_parts) if stdout_parts else None
@ -429,7 +442,9 @@ class AsyncSSH(Base):
exit_code = completed.exit_status exit_code = completed.exit_status
if exit_code is None: if exit_code is None:
exit_code = completed.returncode if completed.returncode is not None else -1 exit_code = (
completed.returncode if completed.returncode is not None else -1
)
return Result(stdout, stderr, exit_code) return Result(stdout, stderr, exit_code)

55
src/python/jw/pkg/lib/ec/ssh/Exec.py
View file

@ -1,5 +1,7 @@
from __future__ import annotations from __future__ import annotations
import os
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
from ...base import InputMode from ...base import InputMode
@ -8,18 +10,14 @@ from ..SSHClient import SSHClient as Base
from .util import join_cmd from .util import join_cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from ...base import Result from ...base import Input, Result
class Exec(Base): class Exec(Base):
def __init__(self, uri, *args, **kwargs) -> None: def __init__(self, uri, *args, **kwargs) -> None:
self.__askpass: str|None = None self.__askpass: str | None = None
self.__askpass_orig: dict[str, str|None] = dict() self.__askpass_orig: dict[str, str | None] = dict()
super().__init__( super().__init__(uri = uri, caps = self.Caps.ModEnv, **kwargs)
uri = uri,
caps = self.Caps.ModEnv,
**kwargs
)
def __del__(self): def __del__(self):
for key, val in self.__askpass_orig.items(): for key, val in self.__askpass_orig.items():
@ -32,36 +30,53 @@ class Exec(Base):
def __init_askpass(self): def __init_askpass(self):
if self.__askpass is None and self.password is not None: if self.__askpass is None and self.password is not None:
import sys, tempfile import sys
import tempfile
prefix = os.path.basename(sys.argv[0]) + '-' prefix = os.path.basename(sys.argv[0]) + '-'
f = tempfile.NamedTemporaryFile(mode='w+t', prefix=prefix, delete=False) f = tempfile.NamedTemporaryFile(
mode = 'w+t', prefix = prefix, delete = False
)
os.chmod(f.name, 0o0700) os.chmod(f.name, 0o0700)
self.__askpass = f.name self.__askpass = f.name
f.write(f'#!/bin/bash\n\necho -n "{self.password}\n"') f.write(f'#!/bin/bash\n\necho -n "{self.password}\n"')
f.close() f.close()
for key, val in {'SSH_ASKPASS': self.__askpass, 'SSH_ASKPASS_REQUIRE': 'force'}.items(): for key, val in {
'SSH_ASKPASS': self.__askpass,
'SSH_ASKPASS_REQUIRE': 'force',
}.items():
self.__askpass_orig[key] = os.getenv(key) self.__askpass_orig[key] = os.getenv(key)
os.environ[key] = val os.environ[key] = val
async def _run_ssh( async def _run_ssh(
self, self,
cmd: list[str], cmd: list[str],
wd: str|None, wd: str | None,
verbose: bool, verbose: bool,
cmd_input: bytes|None, cmd_input: bytes | None,
mod_env: dict[str, str]|None, mod_env: dict[str, str] | None,
interactive: bool, interactive: bool,
log_prefix: str log_prefix: str,
) -> Result: ) -> Result:
def __pub_cmd_input(cmd_input: bytes | None) -> Input:
if cmd_input is None:
if interactive:
return InputMode.Interactive
return InputMode.NonInteractive
return cmd_input
self.__init_askpass() self.__init_askpass()
if cmd_input is None: opts: list[str] = []
cmd_input = InputMode.Interactive if interactive else InputMode.NonInteractive
opts: dict[str, str] = []
if mod_env: if mod_env:
for key, val in mod_env.items(): for key, val in mod_env.items():
opts.extend(['-o', f'SetEnv {key}="{val}"']) opts.extend(['-o', f'SetEnv {key}="{val}"'])
if self.username: if self.username:
opts.extend(['-l', self.username]) opts.extend(['-l', self.username])
if self.port is not None: if self.port is not None:
pots.extend(['-p', str(self.port)]) opts.extend(['-p', str(self.port)])
return await run_cmd(['ssh', *opts, self.hostname, join_cmd(cmd)], cmd_input=cmd_input, throw=False) return await run_cmd(
['ssh', *opts, self.hostname, join_cmd(cmd)],
cmd_input = __pub_cmd_input(cmd_input),
throw = False,
)

40
src/python/jw/pkg/lib/ec/ssh/Paramiko.py
View file

@ -2,44 +2,46 @@ from __future__ import annotations
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
import paramiko # type: ignore # error: Library stubs not installed for "paramiko" import paramiko # type: ignore[import-untyped] # error: Library stubs not installed for "paramiko"
from ...log import *
from ...base import Result from ...base import Result
from ...log import ERR, log
from ..SSHClient import SSHClient as Base from ..SSHClient import SSHClient as Base
from .util import join_cmd from .util import join_cmd
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Any from typing import Any
import paramiko.agent # type: ignore[import-untyped]
import paramiko.SCPClient # type: ignore[import-untyped]
class Paramiko(Base): class Paramiko(Base):
def __init__(self, uri, *args, **kwargs) -> None: def __init__(self, uri, *args, **kwargs) -> None:
super().__init__( kwargs['caps'] = (self.Caps.ModEnv, )
uri, super().__init__(uri, *args, **kwargs)
*args, self.__timeout: float | None = None # Untested
caps = self.Caps.ModEnv, self.___client: Any | None = None
**kwargs
)
self.__timeout: float|None = None # Untested
self.___client: Any|None = None
@property @property
def __client(self) -> Any: def __client(self) -> Any:
if self.___client is None: if self.___client is None:
ret = paramiko.SSHClient() ret = paramiko.SSHClient()
ret.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ret.set_missing_host_key_policy(paramiko.AutoAddPolicy())
hostname = self.hostname
if hostname is None:
raise Exception('Tried to run connect without target hostname')
try: try:
ret.connect( ret.connect(
hostname = self.hostname, hostname = hostname, username = self.username, allow_agent = True
username = self.username,
allow_agent = True
) )
except Exception as e: except Exception as e:
log(ERR, f'Failed to connect to {self.hostname} ({str(e)})') log(ERR, f'Failed to connect to {self.hostname} ({str(e)})')
raise raise
s = ret.get_transport().open_session() transport = ret.get_transport()
if transport is None:
raise Exception(f'Failed to get SSH transport for {hostname}')
s = transport.open_session()
# set up the agent request handler to handle agent requests from the server # set up the agent request handler to handle agent requests from the server
paramiko.agent.AgentRequestHandler(s) paramiko.agent.AgentRequestHandler(s)
self.___client = ret self.___client = ret
@ -47,7 +49,7 @@ class Paramiko(Base):
@property @property
def __scp(self) -> Any: def __scp(self) -> Any:
return SCPClient(self.__client.get_transport()) return paramiko.SCPClient(self.__client.get_transport())
async def _open(self) -> None: async def _open(self) -> None:
await super()._open() await super()._open()
@ -63,13 +65,13 @@ class Paramiko(Base):
cmd: list[str], cmd: list[str],
wd: str | None, wd: str | None,
verbose: bool, verbose: bool,
cmd_input: str | None, cmd_input: bytes | None,
mod_env: dict[str, str] | None, mod_env: dict[str, str] | None,
interactive: bool, interactive: bool,
log_prefix: str, log_prefix: str,
) -> Result: ) -> Result:
try: try:
kwargs: [str, Any] = {} kwargs: dict[str, Any] = {}
if mod_env is not None: if mod_env is not None:
kwargs['environment'] = mod_env kwargs['environment'] = mod_env
stdin, stdout, stderr = self.__client.exec_command( stdin, stdout, stderr = self.__client.exec_command(
@ -78,7 +80,7 @@ class Paramiko(Base):
**kwargs, **kwargs,
) )
except Exception as e: except Exception as e:
log(ERR, f'Command failed for {self.uri}: "{join_cmd(cmd)}"') log(ERR, f'Command failed for {self.uri}: "{join_cmd(cmd)}" ({str(e)})')
raise raise
if cmd_input is not None: if cmd_input is not None:
stdin.write(cmd_input) stdin.write(cmd_input)

32
src/python/jw/pkg/lib/ec/ssh/util.py
View file

@ -1,19 +1,29 @@
# -*- coding: utf-8 -*- import shlex
from typing import Iterable from typing import Iterable
import shlex
DEFAULT_SHELL_OPERATORS = { DEFAULT_SHELL_OPERATORS = {
# redirections # redirections
">", ">>", "<", "<<", "<<-", "<&", ">&", "<>", ">|", '>',
"1>", "1>>", "2>", "2>>", '>>',
'<',
# pipelines / control '<<',
"|", "||", "&", "&&", ";", '<<-',
'<&',
# grouping '>&',
"(", ")", '<>',
'>|',
'1>',
'1>>',
'2>',
'2>>', # pipelines / control
'|',
'||',
'&',
'&&',
';', # grouping
'(',
')',
} }
def join_cmd( def join_cmd(

68
src/python/jw/pkg/lib/log.py
View file

@ -1,38 +1,41 @@
# -*- coding: utf-8 -*- import datetime
import sys
import syslog
import sys, syslog, datetime # fmt: disable # don't conflate
EMERG = int(syslog.LOG_EMERG)
ALERT = int(syslog.LOG_ALERT)
CRIT = int(syslog.LOG_CRIT)
ERR = int(syslog.LOG_ERR)
WARNING = int(syslog.LOG_WARNING)
NOTICE = int(syslog.LOG_NOTICE)
INFO = int(syslog.LOG_INFO)
DEBUG = int(syslog.LOG_DEBUG)
DEVEL = int(syslog.LOG_DEBUG + 1)
OFF = DEVEL + 1
EMERG = int(syslog.LOG_EMERG) _log_level = NOTICE
ALERT = int(syslog.LOG_ALERT) _last_tstamp = datetime.datetime.now()
CRIT = int(syslog.LOG_CRIT) _first_tstamp = _last_tstamp
ERR = int(syslog.LOG_ERR) # fmt: enable
WARNING = int(syslog.LOG_WARNING)
NOTICE = int(syslog.LOG_NOTICE)
INFO = int(syslog.LOG_INFO)
DEBUG = int(syslog.LOG_DEBUG)
DEVEL = int(syslog.LOG_DEBUG + 1)
OFF = DEVEL + 1
_log_level = NOTICE
_last_tstamp = datetime.datetime.now()
_first_tstamp = _last_tstamp
def _log_level_name_by_value(): def _log_level_name_by_value():
if _log_level_name_by_value.map is None: if _log_level_name_by_value.map is None:
_log_level_name_by_value.map = { _log_level_name_by_value.map = {
EMERG: "EMERG", EMERG: 'EMERG',
ALERT: "ALERT", ALERT: 'ALERT',
CRIT: "CRIT", CRIT: 'CRIT',
ERR: "ERR", ERR: 'ERR',
WARNING: "WARNING", WARNING: 'WARNING',
NOTICE: "NOTICE", NOTICE: 'NOTICE',
INFO: "INFO", INFO: 'INFO',
DEBUG: "DEBUG", DEBUG: 'DEBUG',
DEVEL: "DEVEL", DEVEL: 'DEVEL',
OFF: "OFF" OFF: 'OFF',
} }
return _log_level_name_by_value.map return _log_level_name_by_value.map
_log_level_name_by_value.map: dict[int, str]|None = None
_log_level_name_by_value.map: dict[int, str] | None = None # type: ignore
def _log_level_value_by_name(): def _log_level_value_by_name():
if _log_level_value_by_name.map is None: if _log_level_value_by_name.map is None:
@ -41,21 +44,22 @@ def _log_level_value_by_name():
_log_level_value_by_name.map[name] = value _log_level_value_by_name.map[name] = value
_log_level_value_by_name.map[name.lower()] = value _log_level_value_by_name.map[name.lower()] = value
return _log_level_value_by_name.map return _log_level_value_by_name.map
_log_level_value_by_name.map: dict[str, int]|None = None
_log_level_value_by_name.map: dict[str, int] | None = None # type: ignore
def get_log_level_name(level: int) -> str: def get_log_level_name(level: int) -> str:
return _log_level_name_by_value()[level] return _log_level_name_by_value()[level]
def parse_log_level(level: str|int) -> int: def parse_log_level(level: str | int) -> int:
try: try:
ret = int(level) ret = int(level)
if ret >= 0 and ret <= DEVEL: if ret >= 0 and ret <= DEVEL:
return ret return ret
except ValueError: except ValueError:
return _log_level_value_by_name()[level] return _log_level_value_by_name()[level]
raise Exception("Invalid log level ", level) raise Exception('Invalid log level ', level)
def set_log_level(level: int|None=None) -> int: def set_log_level(level: str | int | None = None) -> int:
global _log_level global _log_level
ret = _log_level ret = _log_level
if level is not None: if level is not None:

66
src/python/jw/pkg/lib/pm/dpkg.py
View file

@ -1,50 +1,66 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import Iterable, TYPE_CHECKING from typing import TYPE_CHECKING, Iterable
if TYPE_CHECKING: if TYPE_CHECKING:
from ..ExecContext import ExecContext from ..ExecContext import ExecContext
from ..base import InputMode from ..base import InputMode
from ..Package import Package
from ..util import run_cmd, run_sudo from ..util import run_cmd, run_sudo
from ..Package import Package, meta_tags
_meta_map: dict[str, str]|None = None _meta_map: dict[str, str] | None = None
def meta_map(): def meta_map():
global _meta_map global _meta_map
if _meta_map is None: if _meta_map is None:
_meta_map = Package.order_tags({ _meta_map = Package.order_tags(
'name': 'binary:Package', {
'vendor': None, # deb doesn't have vendor field 'name': 'binary:Package',
'packager': None, # -- packager -- 'vendor': None, # deb doesn't have vendor field
'url': 'Homepage', 'packager': None, # -- packager --
'maintainer': 'Maintainer', 'url': 'Homepage',
}) 'maintainer': 'Maintainer',
}
)
return _meta_map return _meta_map
async def run_dpkg(args: list[str], sudo: bool=False, ec: ExecContext=None): # export async def _run(
cmd: list[str], sudo: bool = False, ec: ExecContext | None = None
) -> str:
return (
await run_sudo(cmd)
if sudo else await run_cmd(cmd, ec = ec, cmd_input = InputMode.NonInteractive)
).stdout_str
async def run_dpkg(
args: list[str],
sudo: bool = False,
ec: ExecContext | None = None
) -> str: # export
cmd = ['/usr/bin/dpkg'] cmd = ['/usr/bin/dpkg']
cmd.extend(args) cmd.extend(args)
if sudo: return await _run(cmd, sudo, ec)
return await run_sudo(cmd, ec=ec)
return (await run_cmd(cmd, ec=ec)).decode()
async def run_dpkg_query(args: list[str], sudo: bool=False, ec: ExecContext=None): # export async def run_dpkg_query(
args: list[str],
sudo: bool = False,
ec: ExecContext | None = None
) -> str: # export
cmd = ['/usr/bin/dpkg-query'] cmd = ['/usr/bin/dpkg-query']
cmd.extend(args) cmd.extend(args)
if sudo: return await _run(cmd, sudo, ec)
return await run_sudo(cmd)
return (await run_cmd(cmd, ec=ec, cmd_input=InputMode.NonInteractive)).decode()
async def query_packages(names: Iterable[str] = [], ec: ExecContext=None) -> Iterable[Package]: async def query_packages(names: Iterable[str] = [],
fmt_str = '|'.join([(f'${{{tag}}}' if tag else '') for tag in meta_map().values()]) + r'\n' ec: ExecContext | None = None) -> Iterable[Package]:
fmt_str = (
'|'.join([(f'${{{tag}}}' if tag else '')
for tag in meta_map().values()]) + r'\n'
)
# dpkg-query -W -f='${binary:Package}|${Maintainer}| ... \n' # dpkg-query -W -f='${binary:Package}|${Maintainer}| ... \n'
specs, stderr, status = await run_dpkg_query(['-W', '-f=' + fmt_str, *names], sudo=False, ec=ec) specs = await run_dpkg_query(['-W', '-f=' + fmt_str, *names], sudo = False, ec = ec)
return Package.parse_specs_str(specs) return Package.parse_specs_str(specs)
async def list_files(pkg: str, ec: ExecContext=None) -> list[str]: async def list_files(pkg: str, ec: ExecContext | None = None) -> list[str]:
file_list_str, stderr, status = await run_dpkg(['-L', pkg], sudo=False, ec=ec) file_list_str = await run_dpkg(['-L', pkg], sudo = False, ec = ec)
return file_list_str.splitlines() return file_list_str.splitlines()

78
src/python/jw/pkg/lib/pm/rpm.py
View file

@ -1,45 +1,71 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import Iterable, TYPE_CHECKING from typing import TYPE_CHECKING, Iterable
from ..base import InputMode
from ..Package import Package
from ..util import run_cmd, run_sudo
if TYPE_CHECKING: if TYPE_CHECKING:
from ..ExecContext import ExecContext from ..ExecContext import ExecContext
from ..util import run_cmd, run_sudo _meta_map: dict[str, str] | None = None
from ..base import InputMode
from ..Package import Package, meta_tags
_meta_map: dict[str, str]|None = None
def meta_map(): def meta_map():
global _meta_map global _meta_map
if _meta_map is None: if _meta_map is None:
_meta_map = Package.order_tags({ _meta_map = Package.order_tags(
'name': 'Name', {
'vendor': 'Vendor', 'name': 'Name',
'packager': 'Packager', 'vendor': 'Vendor',
'url': 'URL', 'packager': 'Packager',
'maintainer': None, # RPM doesn't have a maintainer field 'url': 'URL',
}) 'maintainer': None, # RPM doesn't have a maintainer field
}
)
return _meta_map return _meta_map
async def run_rpm(args: list[str], sudo: bool=False, ec: ExecContext=None, mode: InputMode=InputMode.OptInteractive, **kwargs): # export async def run_rpm(
args: list[str],
sudo: bool = False,
ec: ExecContext | None = None,
mode: InputMode = InputMode.OptInteractive,
**kwargs,
) -> str: # export
cmd = ['/usr/bin/rpm'] cmd = ['/usr/bin/rpm']
cmd.extend(args) cmd.extend(args)
if sudo: result = (
return await run_sudo(cmd, ec=ec, cmd_input=mode, **kwargs) await run_sudo(cmd, ec = ec, cmd_input = mode, **kwargs)
return await run_cmd(cmd, ec=ec, cmd_input=mode, **kwargs) if sudo else await run_cmd(cmd, ec = ec, cmd_input = mode, **kwargs)
)
return result.stdout_str
async def query_packages(names: Iterable[str] = [], ec: ExecContext=None) -> Iterable[Package]: async def query_packages(
fmt_str = '|'.join([(f'%{{{tag}}}' if tag else '') for tag in meta_map().values()]) + r'\n' names: Iterable[str] = [],
ec: ExecContext | None = None,
) -> Iterable[Package]: # export
fmt_str = (
'|'.join([(f'%{{{tag}}}' if tag else '')
for tag in meta_map().values()]) + r'\n'
)
opts = ['-q', '--queryformat', fmt_str] opts = ['-q', '--queryformat', fmt_str]
if not names: if not names:
opts.append('-a') opts.append('-a')
specs, stderr, status = await run_rpm([*opts, *names], throw=True, sudo=False, mode=InputMode.NonInteractive, ec=ec) specs = await run_rpm(
return Package.parse_specs_str(specs.decode()) [*opts, *names],
throw = True,
sudo = False,
mode = InputMode.NonInteractive,
ec = ec
)
return Package.parse_specs_str(specs)
async def list_files(pkg: str, ec: ExecContext=None) -> list[str]: async def list_files(pkg: str, ec: ExecContext | None = None) -> list[str]:
stdout, stderr, status = await run_rpm(['-ql', pkg], throw=True, sudo=False, mode=InputMode.NonInteractive, ec=ec) stdout = await run_rpm(
return stdout.decode().splitlines() ['-ql', pkg],
throw = True,
sudo = False,
mode = InputMode.NonInteractive,
ec = ec
)
return stdout.splitlines()

243
src/python/jw/pkg/lib/util.py
View file

@ -1,28 +1,31 @@
# -*- coding: utf-8 -*-
from __future__ import annotations from __future__ import annotations
from typing import TYPE_CHECKING, Iterable import json
import os
if TYPE_CHECKING: import sys
from typing import Sequence
from .ExecContext import ExecContext
from .ProcFilter import ProcFilter, ProcPipeline
import os, sys, json
from argparse import Namespace from argparse import Namespace
from enum import Enum, auto from enum import Enum, auto
from typing import TYPE_CHECKING, Iterable, TypeVar, cast
from .log import * from .base import Input, InputMode, Result
from .base import InputMode from .log import DEBUG, ERR, log
from .Uri import Uri from .Uri import Uri
if TYPE_CHECKING:
from .ExecContext import ExecContext
from .FileContext import FileContext
from .ProcFilter import ProcFilter, ProcPipeline
T = TypeVar('T')
class AskpassKey(Enum): class AskpassKey(Enum):
Username = auto() Username = auto()
Password = auto() Password = auto()
def pretty_cmd(cmd: list[str], wd=None): def pretty_cmd(cmd: list[str] | None = None, wd = None):
if cmd is None:
cmd = sys.argv
tokens = [cmd[0]] tokens = [cmd[0]]
for token in cmd[1:]: for token in cmd[1:]:
if token.find(' ') != -1: if token.find(' ') != -1:
@ -34,44 +37,72 @@ def pretty_cmd(cmd: list[str], wd=None):
return ret return ret
# See ExecContext.run() for what this function does # See ExecContext.run() for what this function does
async def run_cmd(*args, ec: ExecContext|None=None, verbose: bool|None=None, cmd_input: Input=InputMode.NonInteractive, **kwargs) -> Result: async def run_cmd(
*args,
ec: ExecContext | None = None,
verbose: bool | None = None,
cmd_input: Input = InputMode.NonInteractive,
**kwargs,
) -> Result:
if verbose is None: if verbose is None:
verbose = False if ec is None else ec.verbose_default verbose = False if ec is None else ec.verbose_default
if ec is None: if ec is None:
from .ec.Local import Local from .ec.Local import Local
interactive = cmd_input == InputMode.Interactive
ec = Local(verbose_default=verbose, interactive=interactive)
return await ec.run(verbose=verbose, *args, **kwargs)
async def run_curl(args: list[str], parse_json: bool=False, wd=None, throw=None, verbose=None, cmd_input=InputMode.NonInteractive, ec: ExecContext|None=None, decode=False) -> dict|str: # export interactive = cmd_input == InputMode.Interactive
ec = Local(verbose_default = verbose, interactive = interactive)
kwargs['verbose'] = verbose
return await ec.run(*args, **kwargs)
async def run_curl(
args: list[str],
wd = None,
throw = None,
verbose = None,
cmd_input = InputMode.NonInteractive,
ec: ExecContext | None = None,
decode = False,
) -> Result:
if verbose is None: if verbose is None:
verbose = False if ec is None else ec.verbose_default verbose = False if ec is None else ec.verbose_default
cmd = ['curl'] cmd = ['curl']
if not verbose: if not verbose:
cmd.append('-s') cmd.append('-s')
cmd.extend(args) cmd.extend(args)
if parse_json: return await run_cmd(
decode = True cmd, wd = wd, throw = throw, verbose = verbose, cmd_input = cmd_input, ec = ec
output = await run_cmd(cmd, wd=wd, throw=throw, verbose=verbose, cmd_input=cmd_input, ec=ec) )
stdout, stderr, status = output.decode() if decode else output
if not parse_json:
ret = stdout
else:
try:
ret = json.loads(stdout)
except Exception as e:
size = 'unknown number of'
try:
size = len(stdout)
except:
pass
log(ERR, f'Failed to parse {size} bytes output of command '
+ f'>{pretty_cmd(cmd, wd)}< ({str(e)}): "{stdout}"', file=sys.stderr)
raise
return ret, stderr, status
async def run_askpass(askpass_env: list[str], key: AskpassKey, host: str|None=None, ec: ExecContext|None=None): async def run_curl_into(
if host is not None: # Currently unsupported expected_type: type[T],
args: list[str],
**kwargs,
) -> T:
result = await run_curl(args, **kwargs)
stdout = result.stdout_str
try:
ret = json.loads(stdout)
except Exception as e:
log(
ERR,
f'Failed to parse {len(stdout)} bytes of Curl output ({str(e)})',
file = sys.stderr,
)
raise
if not isinstance(ret, expected_type):
raise TypeError(
f'Expected {expected_type.__name__}, got {type(ret).__name__} from Curl'
)
return cast(T, ret)
async def run_askpass(
askpass_env: list[str],
key: AskpassKey,
host: str | None = None,
ec: ExecContext | None = None,
throw: bool = False,
) -> str | None:
if host is not None: # Currently unsupported
raise NotImplementedError(f'Tried to run askpass with host "{host}"') raise NotImplementedError(f'Tried to run askpass with host "{host}"')
for var in askpass_env: for var in askpass_env:
exe = os.getenv(var) exe = os.getenv(var)
@ -88,50 +119,88 @@ async def run_askpass(askpass_env: list[str], key: AskpassKey, host: str|None=No
case 'SSH_ASKPASS': case 'SSH_ASKPASS':
match key: match key:
case AskpassKey.Username: case AskpassKey.Username:
continue # Can't get user name from SSH_ASKPASS continue # Can't get user name from SSH_ASKPASS
case AskpassKey.Password: case AskpassKey.Password:
exe_arg += 'Password' exe_arg += 'Password'
ret, stderr, status = await run_cmd([exe, exe_arg], throw=False, ec=ec).decode() result = await run_cmd([exe, exe_arg], throw = throw, ec = ec)
if ret is not None: if result.status == 0 and result.stdout_or_none is not None:
return ret ret = result.stdout_str_or_none
if ret:
return ret
msg = (
f"Trying to get user data from {', '.join(askpass_env)} didn't produce anything"
)
if throw:
raise Exception(msg)
log(DEBUG, msg)
return None return None
async def run_sudo(cmd: list[str], *args, interactive: bool=True, ec: ExecContext|None=None, **kwargs): async def run_sudo(
cmd: list[str],
*args,
interactive: bool = True,
ec: ExecContext | None = None,
**kwargs,
):
if ec is None: if ec is None:
from .ec.Local import Local from .ec.Local import Local
ec = Local(interactive=interactive)
ec = Local(interactive = interactive)
return await ec.sudo(cmd, *args, **kwargs) return await ec.sudo(cmd, *args, **kwargs)
async def get( async def get(
uri: str|Uri, uri: str | Uri,
*args, *args,
ctx: FileContext|None=None, ctx: FileContext | None = None,
content_filter: ProcFilter|list[ProcFilter]|ProcPipeline|None = None, content_filter: ProcFilter | list[ProcFilter] | ProcPipeline | None = None,
**kwargs **kwargs,
) -> Result: ) -> Result:
uri = Uri.pimp(uri) uri = Uri.pimp(uri)
if ctx is None or uri.id != ctx.uri.id: if ctx is None or uri.id != ctx.uri.id:
from .FileContext import FileContext from .FileContext import FileContext
ctx = FileContext.create(uri) ctx = FileContext.create(uri)
from .ProcFilter import run as run_pipeline from .ProcFilter import run as run_pipeline
return await run_pipeline(await ctx.get(uri.path, *args, **kwargs), content_filter) return await run_pipeline(await ctx.get(uri.path, *args, **kwargs), content_filter)
async def copy(src_uri: str|Iterable[str], dst: str|FileContext, owner: str|None=None, group: str|None=None, mode: int|None=None, throw=True) -> Exception|str|list[str]: async def copy(
src_uri: str | Iterable[str],
dst: str | FileContext,
owner: str | None = None,
group: str | None = None,
mode: int | None = None,
throw = True,
) -> Exception | str | list[str]:
if not isinstance(src_uri, str): if not isinstance(src_uri, str):
ret: list[str] = [] ret: list[str] = []
for uri in src_uri: # TODO: Group identical netlocs into one CopyContext for uri in src_uri: # TODO: Group identical netlocs into one CopyContext
rr = ret.append(await copy(uri, dst, owner, group, mode, throw)) rr = await copy(uri, dst, owner, group, mode, throw)
if isinstance(rr, Exception): if isinstance(rr, Exception):
return rr return rr
if isinstance(rr, list):
ret.extend(rr)
if isinstance(rr, str):
ret.append(rr)
else:
raise Exception(f'copy() returned unexpected type {type(rr)}')
return ret return ret
from .CopyContext import CopyContext from .CopyContext import CopyContext
async with CopyContext(src_uri, dst) as ctx: async with CopyContext(src_uri, dst) as ctx:
try: try:
content = (await ctx.src.get(ctx.src.root, throw=True)).stdout result = await ctx.src.get(ctx.src.root, throw = True)
dst_path = ctx.dst.root dst_path = ctx.dst.root
if await ctx.dst.is_dir(ctx.dst.root): if await ctx.dst.is_dir(ctx.dst.root):
dst_path += '/' + os.path.basename(src_uri) dst_path += '/' + os.path.basename(src_uri)
await ctx.dst.put(path=dst_path, content=content, owner=owner, group=group, mode=mode, throw=True) await ctx.dst.put(
path = dst_path,
content = result.stdout,
owner = owner,
group = group,
mode = mode,
throw = True,
)
return dst_path return dst_path
except Exception as e: except Exception as e:
if throw: if throw:
@ -140,21 +209,39 @@ async def copy(src_uri: str|Iterable[str], dst: str|FileContext, owner: str|None
return e return e
assert False, 'Unreachable code' assert False, 'Unreachable code'
async def get_username(args: Namespace|None=None, url: str|None=None, askpass_env: list[str]=[], ec: ExecContext|None=None) -> str: # export async def get_username(
args: Namespace | None = None,
url: str | None = None,
askpass_env: list[str] = [],
ec: ExecContext | None = None,
) -> str | None: # export
url_user = None if url is None else Uri(url).username url_user = None if url is None else Uri(url).username
if args is not None: if args is not None:
if args.username is not None: if args.username is not None:
if url_user is not None and url_user != args.username: if url_user is not None and url_user != args.username:
raise Exception(f'Username mismatch: called with --username="{args.username}", URL has user name "{url_user}"') raise Exception(
f'Username mismatch: called with --username="{args.username}", '
f'URL has user name "{url_user}"'
)
return args.username return args.username
if url_user is not None: if url_user is not None:
return url_user return url_user
return await run_askpass(askpass_env, AskpassKey.Username, ec=ec) return await run_askpass(askpass_env, AskpassKey.Username, ec = ec)
async def get_password(args: Namespace|None=None, url: str|None=None, askpass_env: list[str]=[], ec: ExecContext|None=None) -> str: # export async def get_password(
args: Namespace | None = None,
url: str | None = None,
askpass_env: list[str] = [],
ec: ExecContext | None = None,
) -> str | None: # export
if args is None and url is None and not askpass_env: if args is None and url is None and not askpass_env:
raise Exception(f'Neither URL nor command-line arguments nor askpass environment variable available, can\'t get password') raise Exception(
if args is not None and hasattr(args, 'password'): # use getattr(), because we don't necessarily want to have insecure --password among options 'Neither URL nor command-line arguments nor askpass environment variable '
"available, can't get password"
)
if args is not None and hasattr(args, 'password'):
# use getattr(), because we don't necessarily want to have insecure
# --password among options
ret = getattr(args, 'password') ret = getattr(args, 'password')
if ret is not None: if ret is not None:
return ret return ret
@ -162,9 +249,13 @@ async def get_password(args: Namespace|None=None, url: str|None=None, askpass_en
ret = Uri(url).password ret = Uri(url).password
if ret is not None: if ret is not None:
return ret return ret
return await run_askpass(askpass_env, AskpassKey.Password, ec=ec) return await run_askpass(askpass_env, AskpassKey.Password, ec = ec)
async def get_profile_env(throw: bool=True, keep: Iterable[str]|bool=False, ec: ExecContext|None=None) -> dict[str, str]: # export async def get_profile_env(
throw: bool = True,
keep: Iterable[str] | bool = False,
ec: ExecContext | None = None,
) -> dict[str, str]: # export
""" """
Get a fresh environment from /etc/profile Get a fresh environment from /etc/profile
@ -177,22 +268,28 @@ async def get_profile_env(throw: bool=True, keep: Iterable[str]|bool=False, ec:
Returns: Returns:
Dictionary with fresh environment Dictionary with fresh environment
""" """
mod_env: dict[str,str]|None = None mod_env: dict[str, str] | None = None
if keep == False or isinstance(keep, Iterable): if (not keep) or isinstance(keep, Iterable):
mod_env = { mod_env = {
'HOME': os.environ.get('HOME', '/'), 'HOME': os.environ.get('HOME', '/'),
'USER': os.environ.get('USER', ''), 'USER': os.environ.get('USER', ''),
'PATH': '/usr/bin:/bin', 'PATH': '/usr/bin:/bin',
} }
# Run bash as a login shell, which sources /etc/profile, then print environment as NUL-separated key=value pairs
# Run bash as a login shell, which sources /etc/profile, then print
# environment as NUL-separated key=value pairs
cmd = ['/usr/bin/env', '-i', '/bin/bash', '-lc', 'env -0'] cmd = ['/usr/bin/env', '-i', '/bin/bash', '-lc', 'env -0']
result = await run_cmd(cmd, throw=throw, verbose=True, mod_env=mod_env, ec=ec) result = await run_cmd(
cmd, throw = throw, verbose = True, mod_env = mod_env, ec = ec
)
ret: dict[str, str] = {} ret: dict[str, str] = {}
for entry in result.stdout.rstrip(b"\0").split(b"\0"): stdout = result.stdout_or_none
if not entry: if stdout is not None:
continue for entry in stdout.rstrip(b'\0').split(b'\0'):
key, val = entry.split(b"=", 1) if not entry:
ret[key.decode()] = val.decode() continue
bkey, bval = entry.split(b'=', 1)
ret[bkey.decode()] = bval.decode()
if isinstance(keep, Iterable): if isinstance(keep, Iterable):
for key in keep: for key in keep:
val = os.getenv(key) val = os.getenv(key)