From 92a07cfa89cc1873807a6bbc4b399beea44e8068 Mon Sep 17 00:00:00 2001 From: Keerthana KT Date: Fri, 13 Nov 2026 21:21:32 +0530 Subject: [PATCH 1/2] fix(gix): use Git for `.gitmodules` lines that Git rejects On the Gix backend, `_backend.submodule_config()` reads `.gitmodules` with GixPython's `ModulesFile.from_bytes()` for `Repo.submodules`, `Submodule.iter_items()` and the cached `path`, `url` and `branch` fields. GixPython 0.3.0 accepts lines that Git rejects with `bad config line` and still returns settings from them. A line holding more than one name is read as several settings: after `url = ../good`, the line `x url = ../evil` is read as an implicit `x` plus `url = ../evil`, and `[submodule "module"] x url = ../evil` does the same on a header line. `br#anch = main` is read as an implicit `br` followed by a comment. `.gitmodules` is untrusted repository content. For such a file, `git submodule update --init` stops with `fatal: bad config line`, `git fsck --strict` only warns `gitmodulesParse` and exits 0, and the CLI backend raises `configparser.ParsingError`. The Gix backend instead listed the submodule with `../evil`, a URL Git never reads, and `Submodule.update(init=True)` cloned from it. GixPython has no strict parsing mode, so the adapter now selects the CLI path unless every line is blank, a comment, a bare `[submodule "..."]` header or a single `key` or `key = value` setting. Git then reports the error exactly as on the CLI backend. The guard only chooses the fallback and produces no values. Continued values and settings on header lines also go to Git, which keeps the guard simple at the cost of one CLI call for those uncommon files. The mismatch is recorded as GIX-35 in `doc/gix-backend.md`; this guard does not close it. Validation on macOS arm64 with CPython 3.13.17, Git 2.54.0 and GixPython 0.3.0: - `test_submodule_configuration_rejects_lines_git_rejects`, using the new `submodule-rejected-lines.sh` fixture, fails for all three scenarios on the Gix backend before the change and passes on both backends after it. - A differential fuzzer comparing the adapter with `git config --file .gitmodules --list` found rejected files with native values within the first 3,000 mutated inputs before the change, and none in 13,000 after it. Where Git accepts a file, it found no value differences in 7,500 inputs. - Full suite: 2103 passed, 155 skipped, 1 xfailed on the CLI backend; 2206 passed, 52 skipped, 1 xfailed on the Gix backend, where native submodule reads still serve 943 calls and the new fallback serves only the 6 calls from the new test. - Ruff check and format, mypy, shellcheck and `git diff --check` pass. Agent-Model: claude-opus-5-5 --- doc/gix-backend.md | 3 ++- git/_backend.py | 11 +++++++++++ test/test_submodule.py | 12 ++++++++++++ tests/fixtures/submodule-rejected-lines.sh | 18 ++++++++++++++++++ 4 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 tests/fixtures/submodule-rejected-lines.sh diff --git a/doc/gix-backend.md b/doc/gix-backend.md index 8a5b2ec71..7861054f8 100644 --- a/doc/gix-backend.md +++ b/doc/gix-backend.md @@ -373,7 +373,7 @@ and predate the current fallback; they do not establish native coverage. | Managed `symbolic_ref` | Nonrecursive target lookup | Mutation, command-level missing-reference diagnostics | | Reference enumeration, `for_each_ref` | Sorted reference names and literal prefixes, preserving symbolic aliases | Dangling symbolic refs, root refs, glob patterns, other formats/options; GIX-25 | | Managed `config --get KEY` | Merged repository config snapshot | Files/streams, enumeration, mutation; GIX-12 | -| Submodule enumeration and cached fields | Dedicated native `.gitmodules` parser over the existing worktree/blob source, with raw path/URL/branch values | Non-UTF-8 subsection names, other/duplicate sections, ambiguous brackets, missing/implicit fields and parser errors; GIX-12 | +| Submodule enumeration and cached fields | Dedicated native `.gitmodules` parser over the existing worktree/blob source, with raw path/URL/branch values | Non-UTF-8 subsection names, other/duplicate sections, ambiguous brackets, missing/implicit fields and parser errors; GIX-12. Lines other than comments, bare headers and single settings; GIX-35 | | Commit hooks | No bound native lookup/execution API | All hooks, including absent hooks; GIX-23 | | Worktree inventory, `worktree list` | Main and linked worktree metadata, including bare main repositories and locks | Prunable entries; GIX-14 | | `Repo.merge_base`, `Repo.is_ancestor` | Native graph queries for two revisions | Octopus/fork-point and other options | @@ -990,6 +990,7 @@ identify the historical fixture where needed. | GIX-32 | Missing capability (open); no demonstrated Git compatibility mismatch | Windows CI on GixPython 0.3.0, Git 2.56.0.windows.2 and CPython 3.12.10 enumerates the non-UTF-8 packed names in `legacy-refs.sh`, but exact byte-name lookup selects `Reference.read: CLI: native read diagnostics`; the no-CLI guard fails for SHA-1 and SHA-256. POSIX lookup succeeds. Git's Windows CLI likewise cannot address those original bytes through UTF-16 arguments, so there is no demonstrated equivalent Windows CLI operand. Git enumeration does preserve both names. | Provide native Windows packed-reference lookup for byte names without first converting the name to a loose-reference filesystem path. Keep `reference_info()`'s existing diagnostic fallback; `test_non_utf8_tag_names_preserve_bytes` checks it on Windows. Resolution coverage and native claims remain POSIX-only. The upstream lookup source attempts loose-path conversion before consulting packed refs; CI demonstrates the fallback, but does not capture the underlying Gix error text. | | GIX-33 | Bug (open); missing exact-name mode | With Git 2.54.0 and GixPython 0.3.0, `exact-refs.sh sha1 files symbolic` makes `refs/tags/REBASE_HEAD` a symbolic ref to another tag while the root `REBASE_HEAD` is absent. `Repository.try_find_reference("REBASE_HEAD")` returns that tag; native `_symbolic_ref` formerly returned its target, whereas `git symbolic-ref --quiet --no-recurse -- REBASE_HEAD` exits 1. GitPython's root-reference object read consequently returned a commit through Gix and raised `ValueError` through CLI. SHA-256 reproduces the same mismatch. | Expose an exact-name lookup mode, including root references, whose absence and error behavior matches Git. `_symbolic_ref()` compares the native reference's name with the requested bytes and selects CLI for a lookalike. `test_missing_root_reference_does_not_follow_symbolic_lookalike_tag` verifies parity and forced CLI fallback. This adapter guard does not close the upstream compatibility bug. | | GIX-34 | Bug (open) | With Git 2.54.0 and GixPython 0.3.0, an author, committer or tagger line with fields after its time zone, such as `A 1112591163 +0200 1700000000 -0500`, decodes through `Commit.author()`/`committer()` and `Tag.tagger()` with the correct seconds but an offset of 0. Git reads only the first date and time zone after the last `>` (`split_ident_line`) and reports `1112591163 +0200` through `git show --format=%ad --date=raw` and `%(taggerdate:raw)`. `git fsck` reports `badTimezone` for such objects, but the default `transfer.fsckObjects=false` accepts them. The `trailing-date` read-only `commit.sh` fixture in `test_deserialization_reads_date_after_email` reproduces both object kinds. | Report the time zone Git reads for identity lines with trailing fields, or reject them consistently. `object_metadata()` selects CLI when the decoded signature does not reproduce the end of its raw header line. This adapter guard does not close the upstream compatibility bug. | +| GIX-35 | Bug (open) | With Git 2.54.0 and GixPython 0.3.0, `ModulesFile.from_bytes()` accepts `.gitmodules` lines that Git rejects with `bad config line`, so `git submodule update --init` stops while `git fsck` reports only `gitmodulesParse`. A line holding more than one name is read as several settings: after `url = ../good`, `x url = ../evil` makes `url()` return `../evil` with `submodule..x` true, and `[submodule "module"] x url = ../evil` does the same on the header line. `br#anch = main` is read as an implicit `br` followed by a comment. The `submodule-rejected-lines.sh` read-only fixture in `test_submodule_configuration_rejects_lines_git_rejects` reproduces all three. | Reject the lines Git rejects, at least in a Git-strict parsing mode, with a typed error. `submodule_config()` selects CLI unless every line is blank, a comment, a bare submodule header or a single `key` or `key = value` setting, so Git's parser reports the error. This adapter guard does not close the upstream compatibility bug. | GIX-14 also covers configured worktree symlink resolution. On macOS with GixPython 0.3.0, CPython 3.12.14 and Git 2.54.0, set a submodule's diff --git a/git/_backend.py b/git/_backend.py index 51d25c436..a239f4007 100644 --- a/git/_backend.py +++ b/git/_backend.py @@ -403,6 +403,15 @@ def _for_each_ref(repo: Any, args: List[str], kwargs: Dict[str, Any]) -> bytes: ) +# The native parser accepts lines Git rejects, such as ``x url = ...`` (GIX-35). +# Keep only blank and comment lines, bare submodule headers and single settings +# native; Git parses everything else, including continued values. +_NATIVE_SUBMODULE_CONFIG_LINE = re.compile( + rb'[ \t\r]*(?:[#;].*|\[submodule[ \t]+"(?:[^"\\]|\\.)*"\][ \t\r]*(?:[#;].*)?|[a-z][a-z0-9-]*[ \t]*(?:=.*|\r?))?', + re.I, +) + + def submodule_config(source: Any) -> Any: """Read the fields used by submodule objects from their selected config source.""" if gix is None: @@ -417,6 +426,8 @@ def submodule_config(source: Any) -> Any: # occurrences and duplicate sections until GIX-12 can preserve their order. if b"\0" in data or re.search(rb"\[(?!submodule[ \t])", data, re.I): raise _Unsupported("non-submodule configuration sections (GIX-12)") + if not all(map(_NATIVE_SUBMODULE_CONFIG_LINE.fullmatch, data.split(b"\n"))): + raise _Unsupported("configuration lines Git may reject (GIX-35)") modules = gix.ModulesFile.from_bytes(data) with modules.names() as cursor: names = list(cursor) diff --git a/test/test_submodule.py b/test/test_submodule.py index 751235e21..54ce27668 100644 --- a/test/test_submodule.py +++ b/test/test_submodule.py @@ -61,6 +61,18 @@ def test_native_submodule_configuration_preserves_non_utf8_values(): assert native == {"module": {"path": "path-\udcfe", "url": "url-\udcfd", "branch": "refs/heads/branch-\udcfc"}} +@pytest.mark.parametrize("scenario", ["extra-setting", "header-setting", "key-comment"]) +def test_submodule_configuration_rejects_lines_git_rejects(scenario): + with fixture_repo("submodule-rejected-lines.sh", scenario) as repo: + with pytest.raises(GitCommandError, match="bad config line"): + repo.git.config("list", "--file", ".gitmodules") + with pytest.raises(cp.ParsingError): + repo.submodules + submodule = Submodule(repo, repo.head.commit.tree["module"].binsha, name="module") + with pytest.raises(cp.ParsingError): + submodule.url + + @pytest.mark.parametrize("state", ["unborn", "untracked", "detached"]) @pytest.mark.parametrize("remote", [None, ".", "origin"], ids=["working-tree", "local-origin", "origin"]) @patch_git_config("protocol.file.allow", "always") diff --git a/tests/fixtures/submodule-rejected-lines.sh b/tests/fixtures/submodule-rejected-lines.sh new file mode 100644 index 000000000..488817e1d --- /dev/null +++ b/tests/fixtures/submodule-rejected-lines.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -eu +git init -q repo +cd repo +git -c user.name=fixture -c user.email=fixture@example.invalid commit --allow-empty -qm initial +# Git rejects each of these files with "bad config line". +case "$1" in + extra-setting) gitmodules='[submodule "module"]\npath = module\nurl = ../good\nx url = ../evil\n' ;; + header-setting) gitmodules='[submodule "module"] x url = ../evil\npath = module\n' ;; + key-comment) gitmodules='[submodule "module"]\npath = module\nurl = ../good\nbr#anch = main\n' ;; + *) exit 1 ;; +esac +printf '%b' "$gitmodules" > .gitmodules +modules=$(git hash-object -w .gitmodules) +commit=$(git rev-parse HEAD) +tree=$(printf '100644 blob %s\t.gitmodules\000160000 commit %s\tmodule\000' "$modules" "$commit" | git mktree -z) +commit=$(printf 'Add submodule\n' | git -c user.name=fixture -c user.email=fixture@example.invalid commit-tree "$tree" -p "$commit") +git update-ref HEAD "$commit" From e3ca695320251182539ceca90224a10ed98fa8fb Mon Sep 17 00:00:00 2001 From: Byron Date: Sat, 14 Nov 2026 21:50:51 +0100 Subject: [PATCH 2/2] review Keept the note about the Gix shortcoming without protecting against it. Plan is to have this fixed before release. --- git/_backend.py | 11 ----------- test/test_submodule.py | 12 ------------ tests/fixtures/submodule-rejected-lines.sh | 18 ------------------ 3 files changed, 41 deletions(-) delete mode 100644 tests/fixtures/submodule-rejected-lines.sh diff --git a/git/_backend.py b/git/_backend.py index a239f4007..51d25c436 100644 --- a/git/_backend.py +++ b/git/_backend.py @@ -403,15 +403,6 @@ def _for_each_ref(repo: Any, args: List[str], kwargs: Dict[str, Any]) -> bytes: ) -# The native parser accepts lines Git rejects, such as ``x url = ...`` (GIX-35). -# Keep only blank and comment lines, bare submodule headers and single settings -# native; Git parses everything else, including continued values. -_NATIVE_SUBMODULE_CONFIG_LINE = re.compile( - rb'[ \t\r]*(?:[#;].*|\[submodule[ \t]+"(?:[^"\\]|\\.)*"\][ \t\r]*(?:[#;].*)?|[a-z][a-z0-9-]*[ \t]*(?:=.*|\r?))?', - re.I, -) - - def submodule_config(source: Any) -> Any: """Read the fields used by submodule objects from their selected config source.""" if gix is None: @@ -426,8 +417,6 @@ def submodule_config(source: Any) -> Any: # occurrences and duplicate sections until GIX-12 can preserve their order. if b"\0" in data or re.search(rb"\[(?!submodule[ \t])", data, re.I): raise _Unsupported("non-submodule configuration sections (GIX-12)") - if not all(map(_NATIVE_SUBMODULE_CONFIG_LINE.fullmatch, data.split(b"\n"))): - raise _Unsupported("configuration lines Git may reject (GIX-35)") modules = gix.ModulesFile.from_bytes(data) with modules.names() as cursor: names = list(cursor) diff --git a/test/test_submodule.py b/test/test_submodule.py index 54ce27668..751235e21 100644 --- a/test/test_submodule.py +++ b/test/test_submodule.py @@ -61,18 +61,6 @@ def test_native_submodule_configuration_preserves_non_utf8_values(): assert native == {"module": {"path": "path-\udcfe", "url": "url-\udcfd", "branch": "refs/heads/branch-\udcfc"}} -@pytest.mark.parametrize("scenario", ["extra-setting", "header-setting", "key-comment"]) -def test_submodule_configuration_rejects_lines_git_rejects(scenario): - with fixture_repo("submodule-rejected-lines.sh", scenario) as repo: - with pytest.raises(GitCommandError, match="bad config line"): - repo.git.config("list", "--file", ".gitmodules") - with pytest.raises(cp.ParsingError): - repo.submodules - submodule = Submodule(repo, repo.head.commit.tree["module"].binsha, name="module") - with pytest.raises(cp.ParsingError): - submodule.url - - @pytest.mark.parametrize("state", ["unborn", "untracked", "detached"]) @pytest.mark.parametrize("remote", [None, ".", "origin"], ids=["working-tree", "local-origin", "origin"]) @patch_git_config("protocol.file.allow", "always") diff --git a/tests/fixtures/submodule-rejected-lines.sh b/tests/fixtures/submodule-rejected-lines.sh deleted file mode 100644 index 488817e1d..000000000 --- a/tests/fixtures/submodule-rejected-lines.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash -set -eu -git init -q repo -cd repo -git -c user.name=fixture -c user.email=fixture@example.invalid commit --allow-empty -qm initial -# Git rejects each of these files with "bad config line". -case "$1" in - extra-setting) gitmodules='[submodule "module"]\npath = module\nurl = ../good\nx url = ../evil\n' ;; - header-setting) gitmodules='[submodule "module"] x url = ../evil\npath = module\n' ;; - key-comment) gitmodules='[submodule "module"]\npath = module\nurl = ../good\nbr#anch = main\n' ;; - *) exit 1 ;; -esac -printf '%b' "$gitmodules" > .gitmodules -modules=$(git hash-object -w .gitmodules) -commit=$(git rev-parse HEAD) -tree=$(printf '100644 blob %s\t.gitmodules\000160000 commit %s\tmodule\000' "$modules" "$commit" | git mktree -z) -commit=$(printf 'Add submodule\n' | git -c user.name=fixture -c user.email=fixture@example.invalid commit-tree "$tree" -p "$commit") -git update-ref HEAD "$commit"