Skip to content

Rewind Hadoop and raw streams after format detection - #156

Merged
martindurant merged 2 commits into
intake:masterfrom
CAOShurong:codex/rewind-auto-formats
Nov 16, 2026
Merged

martindurant merged 2 commits into
intake:masterfrom
CAOShurong:codex/rewind-auto-formats

Conversation

@CAOShurong

Copy link
Copy Markdown
Contributor

After format detection, python -m snappy -d input.snappy output produces an empty file with exit status 0 for a Hadoop stream, and raises UncompressError for a raw stream. Both format probes leave the input at EOF. This is reproducible on current master with bytes(range(256)) compressed using the corresponding stream compressor.

Rewind after those two seek-dependent probes before returning the decoder. Framed streams keep their existing behavior, including decompression from a pipe.

Adds API and CLI regressions, a framed-stdin check, and includes test_formats.py in the CI test commands. On Windows, CPython 3.13.1 and cramjam 2.14.0: the four new Hadoop/raw cases fail on unchanged master; the source and installed-wheel suites each pass 27 tests and four subtests. Flake8's CI error selection, sdist/wheel builds and Twine checks pass (existing packaging warnings remain). Other platforms were not run locally.

Related to #153 and #154. The read-ahead problem was also noted in #155; this fixes the remaining Hadoop/raw paths after #154 was merged.

@martindurant

Copy link
Copy Markdown
Member

The cancelled job is due to old CI system requirements. Please update.

@CAOShurong

Copy link
Copy Markdown
Contributor Author

Updated the macOS runner to macos-15-intel in a54df27, keeping the x64 configuration. The new workflow runs are awaiting approval.

@martindurant
martindurant merged commit 30a4237 into intake:master Nov 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants