Repository navigation
Conversation
Sum/count of point columns by shapes now joins each Dask partition with a shared shapely.STRtree on a thread pool (shapely releases the GIL) and reduces it to a sparse matrix, instead of computing all points into a GeoDataFrame and grouping with observed=False over every region x category pair. Same signature, same output; other cases keep the geopandas path.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1320 +/- ##
==========================================
- Coverage 91.43% 91.41% -0.03%
==========================================
Files 53 54 +1
Lines 8395 8570 +175
==========================================
+ Hits 7676 7834 +158
- Misses 719 736 +17
🚀 New features to boost your workflow:
|
Creating shapely point geometries holds the GIL, so the threaded STRtree join stopped scaling. A grid index plus crossing-number test compiled with numba (nogil) needs no geometries and matches shapely intersects, including holes, multipolygons, overlaps and boundary points.
Every points-by-shapes aggregation except fractions=True now joins the points to the shapes with the numba grid index, one Dask partition at a time on the active scheduler. sum, count, mean, min and max reduce per chunk; other aggregations collect the matched values and run one pandas groupby, as the sjoin path does, so both paths resolve keys and boundaries the same way. Fixes from review: - holes: one even-odd test over all rings of a geometry, so labels inside labels and touching multipolygon parts count once - NaN and inf coordinates are skipped before the float-to-int conversion - empty and non-polygonal geometries match nothing; polygons with more than 500 vertices are tested with prepared GEOS (intersects_xy) on their grid candidates only - the grid cell size is bounded by the number of geometries - integer sums are widened like pandas; ambiguous keys raise through the checks of get_values (now _locate_values) - no repartition (it sorted the whole index), no scheduler override, lazy import of the kernel module, and jit cache failures fall back to no cache - the column of ones is removed even if the aggregation raises
…isions PointsModel.parse() called dd.from_pandas(sort=True) once for the coordinates and again for every column assigned afterwards. dask computes the divisions in pure Python over the whole index, about 80 s per call at 624M rows. Cast the feature column in pandas, convert the frame once, and take the divisions of a sorted RangeIndex directly.
- Prepared GEOS geometries are not thread-safe: concurrent intersects_xy aborted the process. Each thread now prepares its own copy, the check is one vectorised call, and only geometries that are large in vertices and in grid cells (masks, not cells) go to GEOS. - Per-chunk results are partial pandas groupbys of the matched values, so dtypes match the sjoin path (int64 exact, bool min/max, mean in float) and memory scales with the matched regions, not regions x columns. Categorical counts are still summed sparsely. - The default column of ones is located on the transformed points. - Several categorical value keys raise instead of dropping all but one. - PointsModel.parse() casts the coordinates of a pandas frame to one dtype again.
timtreis
marked this pull request as ready for review
November 18, 2026 21:31
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aggregate()of points by shapes now joins the points to the shapes with a point-in-polygon kernel compiled with numba, one Dask partition at a time. The signature and the output are unchanged, and numba, Dask and scipy are already dependencies. On a 740M-transcript Atera breast FFPE dataset read withread_zarr(), a cells x genes count takes 27 s in a fresh process and 25 s on a repeated call. Main took 406 s for 0.6M transcripts and did not finish larger sizes.Why main is slow
The old path computes all points into a
GeoDataFrame, runssjoin(), and then callsgroupby([region, category], observed=False). Withobserved=False, pandas builds every (region, category) pair, so for 18,028 genes the grouped table has billions of rows, and profiling showed 262 of 373 s in that groupby.What changed
Every points-by-shapes aggregation except
fractions=Truetakes the new path. Shapes by shapes andfractions=Truekeep thesjoin()andoverlay()path, which is now the function_aggregate_shapes_sjoin().PolygonIndex(_points_in_polygons.py) puts the geometries in a uniform grid and tests each point with the even-odd rule over all rings of a geometry, so holes, labels inside labels and touching multipolygon parts are handled. A point on an edge counts as inside, as with shapely'sintersects. Empty and non-polygonal geometries match nothing, and NaN or infinite coordinates are skipped.shapely.intersects_xy), so no shapely point objects are created. Prepared GEOS geometries are not thread-safe, so each thread prepares its own copy."sum","count","mean","min"and"max"are reduced per chunk with a pandas groupby over the matched values, and the partial results are combined at the end. Other functions, including callables, collect the matched values of all chunks and run one pandas groupby, as thesjoin()path does.get_values()(moved to_locate_values()), so ambiguous keys still raise. Values from a table are loaded into memory and aligned by position, as before.sjoin()path: integer sums are widened, so auint8column does not wrap,minandmaxkeep the column's dtype, andmeanis computed in float.PointsModel.parse()converts a pandas frame to Dask once. Before, it calleddd.from_pandas(sort=True)for the coordinates and again for every column assigned afterwards, and Dask computes the divisions in pure Python over the whole index (about 80 s per call at 624M rows). For a sortedRangeIndexthe divisions are now taken directly. The coordinates still get one common dtype.Benchmark
Hardware: 28 cores of a Xeon Platinum 8480+. Data: Atera FFPE Human Breast Cancer, 170,057 cells.
read_zarr(), 740M transcripts, count by gene, fresh processThe counts match main at 0.6M (488,124 transcripts in cells), and every (point, geometry) pair matches
shapely.STRtree.query(predicate="intersects")in the tests.Tests
test_aggregate_points_by_shapes_matches_sjoincompares the new path with_aggregate_shapes_sjoin()for sum, count, mean, min, max, median and a callable, on categorical, string, float,uint8, boolean, int64 (above 2^53) and coordinate columns, and checks that the dtypes match. The test data includes boundary points, overlaps, an empty polygon, missing values, a region with only missing values and a region without points. Other new tests cover a label inside another label, the defaultvalue_keyon transformed points, several categorical keys (which raise), and the compiled kernels in a subprocess, because the test suite disables numba: NaN and infinite coordinates, and many-vertex geometries queried from 8 threads at once.Limits