Skip to content

fix: make stage4 files sort-private, enable sub-buffer stage4 sorts - #911

Open
jodavies wants to merge 3 commits into
form-dev:masterfrom
jodavies:sub-buffer-stagesort
Open

jodavies wants to merge 3 commits into
form-dev:masterfrom
jodavies:sub-buffer-stagesort

Conversation

@jodavies

Copy link
Copy Markdown
Collaborator

For a long time, Stage4 sorting in the sub-buffers (function arguments, dollar variables, term environments) has been forbidden due to crashes.

Resolve this by moving the Stage4 file handles into the sorting struct; every sort has its own files.

File-to-file based sorting on a per-term basis will be horrible for performance so print a Warning for the user. But FORM will not simply terminate during a potentially long-running program, when the going gets tough.

This resolves #211 and #215

The fix is basically the same deal as the "nearly stage 4" problem I fixed in f09853d

@jodavies

Copy link
Copy Markdown
Collaborator Author

While preparing slides for the last workshop, I also noticed that only one of the stage4 files is ever used. The stagesort ping-pong is between one stage4 file, and the main sort file. So also remove the unused file handle.

@coveralls

coveralls commented Sep 15, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 66.041% (-0.03%) from 66.069% — jodavies:sub-buffer-stagesort into form-dev:master

@jodavies
jodavies force-pushed the sub-buffer-stagesort branch from 9cca600 to 69726c5 Compare September 19, 2026 08:07
@jodavies

Copy link
Copy Markdown
Collaborator Author

At the moment, the warning to the user that a sub-buffer stage sort has occurred is printed a single time. Printing it per-term is too much spam, but a single time is probably not prominent enough in the log file. What is a better balance? Once per module? Once per expression per module? Once per expression per thread per module?

@jodavies

jodavies commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

The new third commit resolves buffer size warnings in the new tests.

It can manifest as a TFORM "performance regression" in particular cases. For example, currently in the mincer test of form-bench, we request 32 filepatches. When running with 12 workers, we called AllocSort in threads.c with AM.S0->MaxFpatches/numberofworkers (2), which is then promoted to 4 which causes LargeSize to be silently increased. So the test runs 8% slower due to the implicit LargeSize difference, not due to the change in how the patch caches are organised.

I think it is nonetheless OK to make this change. I want to work towards having no unexpected silent internal buffer-resizing, so that it is more clear what the user's buffer size settings are doing.

For a long time, Stage4 sorting in the sub-buffers (function arguments, dollar
variables, term environments) has been forbidden due to crashes.

Resolve this by moving the Stage4 file handles into the sorting struct; every
sort has its own files.

File-to-file based sorting on a per-term basis will be horrible for performance
so print a Warning for the user. But FORM will not simply terminate during a
potentially long-running program, when the going gets tough.
To maximise use of large+small buffers for caching file patches during sorting,
FORM used to update SortIOSize relect the maximum possible cache size. This had
the effect of also making the output "PObuffer" rather large. Confusingly, also
setting FilePatches smaller could sometimes cause an increased LargeSize alloc
for TFORM workers.

Decouple these sizes: now the PObuffer is sized according to the user's
SortIOSize (and at least MaxTermSize), and when reading file patches we share
the large+small buffer space between the current number of active patches
(which might be less than file patches: in this case the per-patch cache size
is larger than before).
@jodavies
jodavies force-pushed the sub-buffer-stagesort branch from 4ac01d6 to 105f3a8 Compare November 19, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unexpected code in ReNumber

2 participants