janware/jw-pkg
1
Code Pull requests Activity Actions

Improve Python config file template substitution #8

Merged
Jan Lindemann merged 10 commits from jan/feature/20260609-pyproject-toml-add-mypypath into master 2026-06-09 08:13:09 +02:00 AGit
Conversation 0 Commits 10 Files changed 7 +219 -64
Jan Lindemann commented 2026-06-09 08:06:50 +02:00
Owner
Copy link

Improve Python config file template substitution

This PR introduces a series of commits leading up to automatic management of dependency paths in the Python devtools configuration files pyproject.toml and pyrightconfig.json in the root of every jw-pkg - compatible repository.

py-topdir.mk: Introduce $(PY_CHECK_ROOTS)

Replace variable PY_SRC_ROOT by PY_CHECK_ROOTS. The name PY_SRC_ROOT was a bad choice, given that it isn't immediately obvious that it a) can contain multiple root locations to be checked, and that it b) specifically concerns static type checking.

As of this commit, it's possible to limit the type checking scope with PY_CHECK_ROOTS as in

PY_CHECK_ROOTS="src/python/jw/pkg/CmdBase.py src/python/jw/lib" make check

cmds.projects.CmdCreateFile: Add subdirs src/python

In order to allow Pyright to check the types provided by dependant repositories without installing them, pyrightconfig.json contains a list of paths to their root directories in "extraPaths". These paths are unusable for Pyright, though: For type checking to work, it needs to be pointed to the jw namespace package paths within those repos. This commit achieves that by appending the subdirs src/python and tools/python to them, provided they exist.

TODO: This fix hardcodes the current project directory structure. Better would be a way to customize that via makefiles, where the paths can be more easily customized.

conf/templates: Add subdir

Add $(TOPDIR)/conf/templates as a location for templates, i.e. input files to the CmdCreateFile template rendering command.

cmds.projects.lib.templates.tmpl_render(): search_path

Add an additional keyword-argument search_path to templates.tmpl_render(). It allows to specifiy a list of directory paths in which a template of a given name can be found. It defaults to [], in which case only the built-in templates are considered. Otherwise file locations are tried first, then the built-in templates.

cmds.projects.lib.templates.tmpl_render(): RenderValues

tmpl_render()'s "values" argument currently understands dict[str,str|list[str]]. Enhance that to understand the broader RenderValues type, which also includes Iterable[tuple[str, str]], as produced by argparse.add_argument(action='append').

This commit also adds proper type-checking for the values argument. Before, its type had gone unchecked entirely.

cmds.projects.CmdCreateFile: Support --search-path

Add --search-path to the list of CmdCreateFile's supported options. Its value is split by ":" and subsequently passed to the tmpl_render()'s search_path argument.

cmds.projects.CmdCreateFile: Add more options

  • Add an additional, more generic value that --format understands: tmpl. If chosen, the template selected by the new option --template-name is rendered by replacing --field key=value pairs.

  • This commit also adds the option --quote, which makes the renderer enclose the rendered variable values in double quotes.

py-topdir.mk: Add --config pyproject.toml

ruff tries to recursivley use every config file it finds and stumbles over a template:

  /usr/bin/ruff check --select TC,FA --fix --unsafe-fixes .
  ruff failed
    Cause: Failed to parse /home/jan/local/src/jw.dev/proj/jw-pkg/conf/templates/pyproject.toml
    Cause: TOML parse error at line 3, column 3
    |
  3 |   {mypypath}
    |   ^
  invalid key-value pair, expected key

Limiting it to the toplevel pyproject.toml by explicitly specifying --config fixes the behaviour, so that's what this commit does.

py-topdir.mk: Add machinery to generate pyproject.toml

pyproject.toml is currently copied unchanged from conf/topdir to the toplevel directory. Set up machinery in py-topdir.mk to render it from a template in conf/templates instead, replacing {mypypath} in the process.

pyproject.toml: Add {mypypath}

Add a [tool.mypy] section to pyproject.toml with a {mypypath} template variable. The already existing template generation mechanism in py-topdir.mk should fill that in with a path pointing to all Python modules managed by jw-pkg:

[tool.mypy]

{mypypath}
# Improve Python config file template substitution This PR introduces a series of commits leading up to automatic management of dependency paths in the Python devtools configuration files `pyproject.toml` and `pyrightconfig.json` in the root of every jw-pkg - compatible repository. #### py-topdir.mk: Introduce $(PY_CHECK_ROOTS) Replace variable `PY_SRC_ROOT` by `PY_CHECK_ROOTS`. The name `PY_SRC_ROOT` was a bad choice, given that it isn't immediately obvious that it a) can contain multiple root locations to be checked, and that it b) specifically concerns static type checking. As of this commit, it's possible to limit the type checking scope with `PY_CHECK_ROOTS` as in `PY_CHECK_ROOTS="src/python/jw/pkg/CmdBase.py src/python/jw/lib" make check` #### cmds.projects.CmdCreateFile: Add subdirs src/python In order to allow Pyright to check the types provided by dependant repositories without installing them, pyrightconfig.json contains a list of paths to their root directories in "extraPaths". These paths are unusable for Pyright, though: For type checking to work, it needs to be pointed to the `jw` namespace package paths within those repos. This commit achieves that by appending the subdirs src/python and tools/python to them, provided they exist. TODO: This fix hardcodes the current project directory structure. Better would be a way to customize that via makefiles, where the paths can be more easily customized. #### conf/templates: Add subdir Add $(TOPDIR)/conf/templates as a location for templates, i.e. input files to the CmdCreateFile template rendering command. #### cmds.projects.lib.templates.tmpl_render(): search_path Add an additional keyword-argument `search_path` to `templates.tmpl_render()`. It allows to specifiy a list of directory paths in which a template of a given name can be found. It defaults to `[]`, in which case only the built-in templates are considered. Otherwise file locations are tried first, then the built-in templates. #### cmds.projects.lib.templates.tmpl_render(): RenderValues `tmpl_render()`'s "values" argument currently understands `dict[str,str|list[str]]`. Enhance that to understand the broader `RenderValues` type, which also includes `Iterable[tuple[str, str]]`, as produced by `argparse.add_argument(action='append')`. This commit also adds proper type-checking for the values argument. Before, its type had gone unchecked entirely. #### cmds.projects.CmdCreateFile: Support --search-path Add --search-path to the list of CmdCreateFile's supported options. Its value is split by ":" and subsequently passed to the tmpl_render()'s search_path argument. #### cmds.projects.CmdCreateFile: Add more options - Add an additional, more generic value that `--format `understands: `tmpl`. If chosen, the template selected by the new option `--template-name` is rendered by replacing `--field key=value` pairs. - This commit also adds the option --quote, which makes the renderer enclose the rendered variable values in double quotes. #### py-topdir.mk: Add --config pyproject.toml ruff tries to recursivley use every config file it finds and stumbles over a template: ``` /usr/bin/ruff check --select TC,FA --fix --unsafe-fixes . ruff failed Cause: Failed to parse /home/jan/local/src/jw.dev/proj/jw-pkg/conf/templates/pyproject.toml Cause: TOML parse error at line 3, column 3 | 3 | {mypypath} | ^ invalid key-value pair, expected key ``` Limiting it to the toplevel pyproject.toml by explicitly specifying --config fixes the behaviour, so that's what this commit does. #### py-topdir.mk: Add machinery to generate pyproject.toml pyproject.toml is currently copied unchanged from conf/topdir to the toplevel directory. Set up machinery in py-topdir.mk to render it from a template in conf/templates instead, replacing {mypypath} in the process. #### pyproject.toml: Add {mypypath} Add a [tool.mypy] section to pyproject.toml with a {mypypath} template variable. The already existing template generation mechanism in py-topdir.mk should fill that in with a path pointing to all Python modules managed by jw-pkg: ``` [tool.mypy] {mypypath} ``` 
Replace variable PY_SRC_ROOT by PY_CHECK_ROOTS. The name PY_SRC_ROOT was a bad choice, given that it isn't immediately obvious that it a) can contain multiple root locations to be checked, and that it b) specifically concerns static type checking.
As of this commit, it's possible to limit the type checking scope with PY_CHECK_ROOTS as in
  PY_CHECK_ROOTS="src/python/jw/pkg/CmdBase.py src/python/jw/lib" \
    make check
Signed-off-by: Jan Lindemann <jan@janware.com>
In order to allow Pyright to check the types provided by dependant repositories without installing them, pyrightconfig.json contains a list of paths to their root directories in "extraPaths".  These paths are unusable for Pyright, though: For type checking to work, it needs to be pointed to the "jw" namespace package paths within those repos. This commit achieves that by appending the subdirs src/python and tools/python to them, provided they exist.
TODO: This fix hardcodes the current project directory structure. Better would be a way to customize that via makefiles, where the paths can be more easily customized.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add $(TOPDIR)/conf/templates as a location for templates, i.e. input files to the CmdCreateFile template rendering command.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add an additional keyword-argument search_path to templates.tmpl_render(). It allows to specifiy a list of directory paths in which a template of a given name can be found. It defaults to [], in which case only the built-in templates are considered. Otherwise file locations are tried first, then the built-in templates.
Signed-off-by: Jan Lindemann <jan@janware.com>
tmpl_render()'s "values" argument currently understands dict[str,str|list[str]]. Enhance that to understand the broader RenderValues type, which also includes Iterable[tuple[str, str]], as produced by argparse.add_argument(action='append').
This commit also adds proper type-checking for the values argument. Before, its type had gone unchecked entirely.
Signed-off-by: Jan Lindemann <jan@janware.com>
Add --search-path to the list of CmdCreateFile's supported options. Its value is split by ":" and subsequently passed to the tmpl_render()'s search_path argument.
Signed-off-by: Jan Lindemann <jan@janware.com>
- Add an additional, more generic value that --format understands:
    "tmpl". If chosen, the template selected by the new option
    --template-name is rendered by replacing --field key=value pairs.
  - This commit also adds the option --quote, which makes the
    renderer enclose the rendered variable values in double quotes.
Signed-off-by: Jan Lindemann <jan@janware.com>
ruff tries to recursivley use every config file it finds and stumbles over a template:
  /usr/bin/ruff check --select TC,FA --fix --unsafe-fixes .
  ruff failed
    Cause: Failed to parse /home/jan/local/src/jw.dev/proj/jw-pkg/conf/templates/pyproject.toml
    Cause: TOML parse error at line 3, column 3
    |
  3 |   {mypypath}
    |   ^
  invalid key-value pair, expected key
Limiting it to the toplevel pyproject.toml by explicitly specifying --config fixes the behaviour, so that's what this commit does.
Signed-off-by: Jan Lindemann <jan@janware.com>
pyproject.toml is currently copied unchanged from conf/topdir to the toplevel directory. Set up machinery in py-topdir.mk to render it from a template in conf/templates instead, replacing {mypypath} in the process.
Signed-off-by: Jan Lindemann <jan@janware.com>
pyproject.toml: Add {mypypath}
All checks were successful
CI / Packaging - Kali Linux (pull_request) Successful in 3m7s
Details
CI / Packaging - OpenSUSE Tumbleweed (pull_request) Successful in 3m8s
Details
CI / Packaging test (pull_request) Successful in 0s
Details
CI / Packaging - Kali Linux (push) Successful in 3m9s
Details
CI / Packaging - OpenSUSE Tumbleweed (push) Successful in 3m12s
Details
CI / Packaging test (push) Successful in 0s
Details
4b6966c480
 
Add a [tool.mypy] section to pyproject.toml with a {mypypath} template variable. The already existing template generation mechanism in py-topdir.mk should fill that in with a path pointing to all Python modules managed by jw-pkg:
[tool.mypy]
    {mypypath}
Signed-off-by: Jan Lindemann <jan@janware.com>
Jan Lindemann scheduled this pull request to auto merge when all checks succeed 2026-06-09 08:06:51 +02:00
Jan Lindemann changed title from pyproject.toml: Add {mypypath} to Improve Python config file template substitution 2026-06-09 08:13:38 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
janware/jw-pkg!8
No description provided.