janware/jw-pkg
Watch
1
Code Pull requests Activity Actions

scm.sh: Honor -f in git, derive release hash from VCS file ids #110

Merged
Jan Lindemann merged 2 commits from jan/fix/20260916-scm-sh-honor-f-in-git-derive-release-hash-from-vcs-file-ids into master 2026-09-16 14:55:06 +02:00 AGit
Conversation 0 Commits 2 Files changed 5 +59 -12
Jan Lindemann commented 2026-09-16 14:45:01 +02:00
Owner
Copy link

This PR makes the release hash depend only on the committed tree. The old scheme md5summed the working tree, and md5sum follows symlinks, so a project whose tree links to files outside the repo, such as jw-ws's pi config, hashed differently depending on which targets happened to exist on the machine, and jw-ws was released afresh every night. The first commit makes scm.sh ls-files -f honest in git (a mode filter instead of a silent no-op), and the second derives the hash from the committed file ids via a new -s option, so the result is the same on every machine. Expect one extra release per project the first time it runs against the new scheme, then stable hashes.

scm.sh ls-files: Honor -f in Git repositories

In CVS, the -f option keeps directory entries out of scm.sh ls-file's listing; in Git, it's a silent no-op. Notably symlinks are not excluded despite -f.

Implement -f for git with a mode filter on git ls-files -s output: only 100644 and 100755 entries are listed, so the option means the same thing in both code paths.

For Git, this actually means a behaviour change which needs to be fixed:

The dist archive targets tar-files and cpio-files in make/list-files.mk and the scm_files() helper in scripts/pkg.sh then may no longer pass it to get unchanged behaviour.

The targets list-files and list-text-files keep it, so the text-files cache now matches its documented regular-files-only membership.

calculate_hash(): Derive hash from VCS file ids

The source hash is an md5sum over all versioned files in the working tree. md5sum follows symlinks, so a project with links to files outside the tree may produce different hashes depending on files outside the repo.

Compute the hash over the VCS file ids instead. Add a -s option to scm.sh ls-files that lists each versioned file together with its content id. The listing is sorted by path rather than by id, so an entry keeps its position when its content changes.

For Git repos, git ls-tree -r HEAD reports the blob of a symlink's target string and the pinned commit of a submodule, so the resulting hash depends only on the committed tree, never on the working tree or on the machine.

calculate_hash() and the HASH variable in make/pkg-dist.mk feed the id listing, minus the release metadata files, directly to md5sum.

calculate_hash() fails loudly when the pipeline fails, with its callers refusing to proceed on an empty result.

The scheme change invalidates the stored HASH values, so every project gets one more release the first time it runs against the new code, and the hashes are stable again afterwards.

This PR makes the release hash depend only on the committed tree. The old scheme md5summed the working tree, and `md5sum` follows symlinks, so a project whose tree links to files outside the repo, such as `jw-ws`'s pi config, hashed differently depending on which targets happened to exist on the machine, and `jw-ws` was released afresh every night. The first commit makes `scm.sh ls-files -f` honest in git (a mode filter instead of a silent no-op), and the second derives the hash from the committed file ids via a new `-s` option, so the result is the same on every machine. Expect one extra release per project the first time it runs against the new scheme, then stable hashes. #### scm.sh ls-files: Honor -f in Git repositories In CVS, the `-f` option keeps directory entries out of `scm.sh` ls-file's listing; in Git, it's a silent no-op. Notably symlinks are not excluded despite `-f`. Implement `-f` for git with a mode filter on `git ls-files -s` output: only `100644` and `100755` entries are listed, so the option means the same thing in both code paths. For Git, this actually means a behaviour change which needs to be fixed: The dist archive targets `tar-files` and `cpio-files` in `make/list-files.mk` and the `scm_files()` helper in `scripts/pkg.sh` then may no longer pass it to get unchanged behaviour. The targets `list-files` and `list-text-files` keep it, so the text-files cache now matches its documented regular-files-only membership. #### calculate_hash(): Derive hash from VCS file ids The source hash is an `md5sum` over all versioned files in the working tree. `md5sum` follows symlinks, so a project with links to files outside the tree may produce different hashes depending on files outside the repo. Compute the hash over the VCS file ids instead. Add a `-s` option to `scm.sh ls-files` that lists each versioned file together with its content id. The listing is sorted by path rather than by id, so an entry keeps its position when its content changes. For Git repos, `git ls-tree -r HEAD` reports the blob of a symlink's target string and the pinned commit of a submodule, so the resulting hash depends only on the committed tree, never on the working tree or on the machine. `calculate_hash()` and the `HASH` variable in `make/pkg-dist.mk` feed the id listing, minus the release metadata files, directly to `md5sum`. `calculate_hash()` fails loudly when the pipeline fails, with its callers refusing to proceed on an empty result. The scheme change invalidates the stored `HASH` values, so every project gets one more release the first time it runs against the new code, and the hashes are stable again afterwards.
In CVS, the -f option keeps directory entries out of scm.sh ls-file's
listing; in Git, it's a silent no-op. Notably symlinks are not excluded
despite -f.

Implement -f for git with a mode filter on git ls-files -s output: only
100644 and 100755 entries are listed, so the option means the same thing in
both code paths.

For Git, this actually means a behaviour change which needs to be fixed:

The dist archive targets tar-files and cpio-files in make/list-files.mk and
the scm_files() helper in scripts/pkg.sh then may no longer pass it to get
unchanged behaviour.

The targets list-files and list-text-files keep it, so the text-files cache
now matches its documented regular-files-only membership.

Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Signed-off-by: Jan Lindemann <jan@janware.com>
calculate_hash(): Derive hash from VCS file ids
All checks were successful
CI / Packaging - Kali Linux (pull_request) Successful in 5m28s
Details
CI / Packaging - OpenSUSE Tumbleweed (pull_request) Successful in 4m34s
Details
CI / Packaging test (pull_request) Successful in 0s
Details
CI / Packaging - Kali Linux (push) Successful in 5m12s
Details
CI / Packaging - OpenSUSE Tumbleweed (push) Successful in 4m22s
Details
CI / Packaging test (push) Successful in 0s
Details
f738951642
 
The source hash is an md5sum over all versioned files in the working tree.
md5sum follows symlinks, so a project with links to files outside the tree
may produce different hashes depending on files outside the repo.

Compute the hash over the VCS file ids instead. Add a -s option to scm.sh
ls-files that lists each versioned file together with its content id. The
listing is sorted by path rather than by id, so an entry keeps its position
when its content changes.

For Git repos, git ls-tree -r HEAD reports the blob of a symlink's target
string and the pinned commit of a submodule, so the resulting hash depends
only on the committed tree, never on the working tree or on the machine.

calculate_hash() and the HASH variable in make/pkg-dist.mk feed the id
listing, minus the release metadata files, directly to md5sum.

calculate_hash() fails loudly when the pipeline fails, with its callers
refusing to proceed on an empty result.

The scheme change invalidates the stored HASH values, so every project
gets one more release the first time it runs against the new code, and the
hashes are stable again afterwards.

Signed-off-by: Jan Lindemann <jan@janware.com>
Assisted-by: unsloth/Qwen3.8-27B-GGUF:Q4_K_M with pi.dev v0.85.1
Jan Lindemann scheduled this pull request to auto merge when all checks succeed 2026-09-16 14:45:02 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
janware/jw-pkg!110
No description provided.