The previous commits have put rules for linting and formatting via ruff, yapf, mypy and pyright into place. They are checked with the make check target, and this commit adds the fixes for the target to succeed.
It does some refactoring where type checking dug up dirty bits, and also adds lots of churn in the Python code. To a good deal, that's owed to mere formatting changes. It would have been better to seperate those from syntax and refactoring fixes into multiple commits, so that the interesting changes don't drown in the formatting nose. However, that would have been a lot of additional work only to be thrown away by later commits, hence this commit has a big diff in one piece. The size of the diff is regrettable but hopefully a one-off: What it buys is automatic format checking for CI and predictble formats for smaller diffs in the future.
Rules that "make check" enforces are, in the following order
- Syntax checkers:
- ruff check .
- mypy .
- pyright
- Format check:
- yapf --diff --recursive .
The refactoring includes:
- Turn the Result class into a more elaborate object, capable of
doing more heavy lifting around stderr and stdout decoding,
summarizing outcome, and matching error strings.
Aside from fixing broken type checks, this also removes lots of
boilerplate calling code which is currently used for handling
possible call outcome scenarios. Trying to access an inexistent,
decoded string should raise a meaningful exception by itself now,
which removes lots of code with case distinctions.
- Fix Cmd type hierarchy:
- Add the AbstractCmd class above Cmd. This is necessary because
the checker rightfully complains it can't instantiate a Cmd
instance where constructor arguments were needed. They never
were, but the type used at the instantiating code's location in
jw.pkg.App so claims.
- Lots of sub- and sub-subcommands are derived from the base
class of the invoking command. That provides some properties
shared across the ancestor hierarchy of a command, but is
semantically unsound. Fix that by introducing jw.pkg.BaseCmd
class as a place to provide basic helpers shared across all
commands used in a jw.pkg.App's context, and derive all command
classes from that afresh. The parent command is still reachable
via a common parent property.
Formatting changes are conforming to PEP-8, mostly, with minor tweaks. All in all they include the following changes.
- Remove # -*- coding: utf-8 -*-
The line was needed by Python 2 which is not supported anylonger.
For Python 3, the default encoding is UTF-8, anyway.
- Allow to run "make py-format" without having it produce any
changes. It's basically "yapf --in-place --recursive ." with some
code style settings, see conf/topdir/pyproject.toml. The settings
may be debatable. I've had custom tweaks in place on that target,
too, but then again, IDEs would have more hassle to integrate
that.
- Introduce a 88 character line length limit
- One import per line, reshuffle them semantically, see
[tool.isort] in pyproject.toml.
- Hide imports needed for type-checking only behind
if TYPE_CHECKING
- Spaces around assignments accounts for much churn. Having having
no spaces in inline parameter list assignments and default
parameter values would arguably be more compact where it's
useful. On the other hand, I have not found a code formatter
which allows spaces around assignments in parameter lists broken
into one per line and that's often better than a wall of text.
- Add two spaces before # export, as this seems to be mandated by
PEP-8
The __init__.py files as gnerated by python-tools.sh contain multiple issues, fix them:
- Make the machinery fail if the same type name is imported from
different modules
- Support relative imports from .Module import Module instead of
having to use the entire module path as import source
- Import types explicitly re-exported with "as":
from .Module import Module as Module
Otherwise ruff will regard the type as "imported but not used"
- Add "# ruff: noqa: E501" near the top. The import lines can get
long and are beyond manual control (except for renaming the
modules themselves, that is). This can cause ruff to fail, so get
it to accept long lines in __init__.py. The style violation
doesn't make much of a difference in generated code, anyway,
because nobody reads that. Plus what's happening in the code
isn't rocket science, so good style wouldn't help much with
understanding, either.
This promptly digs up two symbol name conflicts lib.pm.dpkg and lib.pm.rpm. Fix them along with this commit to keep it from breaking the build.
Currently, completing a release works with a plain git push. It may push to several repos, depending on how the client repo's origin's pushurl is configured. Those repos may have different user names, and if the ssh wrapper added -l via JW_PKG_SSH_EXTRA_OPTS, the push would fail. Hence, disable JW_PKG_SSH_EXTRA_OPTS for that case.
Rename command "distro" to "pkg" together with "info", its last remaining subcommand. "distro" is often used in the sense of "Linux distribution", which would be too narrow for the targets jw-pkg could theoretically support.
jw-pkg is copied into $(TOPDIR)/bin during build, that's wrong. Write a rule precisely targeted at installing /usr/bin/jw-pkg, and cut all the scripts.mk machinery.
Also, make jw-pkg a relative link to avoid the respective RPM warning.
log_start_stop() is responsible for logging markers at the beginning and end of a decorated log. They should not be applied if pgit.sh is run with --porcelain. In fact, they are, and vice versa. Fix that.
Set -o pipefail at the start of the script. This makes pgit.sh commit work. Before it didn't, because run_git() doesn't return a proper return value when it's used in a pipe with a cosmetic sed afterwards.
Log to stderr and add some ASCII-art around the output. Also, add a --porcelain option to allow more stable output parsing. Subsequently, use that option in make targets parsing the output, notably make diff and make git-show-xxx.
Add the name of the operating system ID as taken from /etc/os-release to $INSTALL_LOG. This should allow for easier post-mortem debugging if multiple builds used the same file in /tmp and possibly also prevent conflicts.
jw-pkg supports more than RPM-based package managers, but for historic reasons, lots of its Makefile variables still have "RPM" in their names. This is misleading. Replace "RPM" in variable names by the more generic "PKG" where appropriate.
Maintainer scripts often mess with systemd services via systemctl. In Docker containers, chroot environments or other environments not governed by Systemd, systemctl will not exist or complain. This is a frequent use case, worthy of providing a wrapper to catch and ignore these cases conveniently.
pkg.run is not evaluated on Debian, fix that. For now it's hacked into pkg.sh, which is bound to be replaced by Python. The limited hassle is still worth the detour.
Rename the --from-user option to --from-owner. Forgejo supports users and organizations under the more general term "owner", so that's the better term.
PGIT_SH_PROJECTS currently only sets the projects to operate on for the get command. Extend this to all commands. And replace the environment variable by a command-line option, likely after this script has been ported to Python.
"clone" in the Git sense means to copy a remote project over from scratch. pgit.sh clone has come from that, but has since evolved into something different, a mixture of clone, pull and fetch, so find a different name. "get" seems generic enough and doesn't clash with a Git meaning. Adapt variable names accordingly across the project.
Continue to name variables in pgit.sh somewhat more consistently, notably turn somevar into some_var. Plus some additional cleanup. Still not a beauty.
If "current-branch" is specified within --refspec, either as from-ref or as to-ref, expand that to the branch the working directory has currently checked out.
dc945537 (pkg.sh log-install: Log %attr(0777, ...) for links) fixed packaging symlink for Debian-based distros, but produces a warning on RPM based distros: Links may not have explicit attributes, so go back to not specifying them at all for RPM.
This commit allows pgit.sh to target not only multiple projects below a projects-directory, but also one single project. If invoked from the toplevel directory of a project, it uses that as the only project it should deal with. This is meant to facilitate running the same VCS abstraction logic for one project as for many projects. The project or projects to deal with should probably be specified on the command line, but changing the auto-detection mechanism buys us what we want for now with low hassle.
Some variable names are too short for global scope ($p, $pdir, $pdirs), among others. For those mentioned: Make them longer and more descriptive.
Also add a variable project_name, which denotes what a project is in a remote repository, and which is currently but not necessarily always the same as the project directory.
Not logging any attribute for links, as it's now, breaks Debian's parser. So, log %attr(0777, $owner, $mode). This fixes the parser on the Debian side and hopefully leaves the RPM side intact.
Calling make git-pull-xxx from a projects directory stops iterating projects if one has a dirty workspace. Calling --autostash fixes that.
With this in place, a failed rebase leaves the local changes behind stashed. So, after manually fixing the rebase, the stash needs to be manually reapplied. The commands that led up to the failure are logged right before, so I have hope that this is learnable, and not too much of a footgun.
Before merging the remote branch, do a rebase. This may fail and prompt conflict resolution, but that seems the canonical outcome for the common use case "interactive make git pull-xxx" with master out-of-sync.
