Skip to content

perf(aggregate): numba point-in-polygon join for points by shapes - #1320

Open
timtreis wants to merge 6 commits into
mainfrom
perf/aggregate-points-strtree
Open

timtreis wants to merge 6 commits into
mainfrom
perf/aggregate-points-strtree

Conversation

@timtreis

@timtreis timtreis commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

aggregate() of points by shapes now joins the points to the shapes with a point-in-polygon kernel compiled with numba, one Dask partition at a time. The signature and the output are unchanged, and numba, Dask and scipy are already dependencies. On a 740M-transcript Atera breast FFPE dataset read with read_zarr(), a cells x genes count takes 27 s in a fresh process and 25 s on a repeated call. Main took 406 s for 0.6M transcripts and did not finish larger sizes.

Why main is slow

The old path computes all points into a GeoDataFrame, runs sjoin(), and then calls groupby([region, category], observed=False). With observed=False, pandas builds every (region, category) pair, so for 18,028 genes the grouped table has billions of rows, and profiling showed 262 of 373 s in that groupby.

What changed

Every points-by-shapes aggregation except fractions=True takes the new path. Shapes by shapes and fractions=True keep the sjoin() and overlay() path, which is now the function _aggregate_shapes_sjoin().

  • PolygonIndex (_points_in_polygons.py) puts the geometries in a uniform grid and tests each point with the even-odd rule over all rings of a geometry, so holes, labels inside labels and touching multipolygon parts are handled. A point on an edge counts as inside, as with shapely's intersects. Empty and non-polygonal geometries match nothing, and NaN or infinite coordinates are skipped.
  • Geometries with more than 500 vertices that span more than 64 grid cells, e.g. a tissue mask, stay in the grid, but only their candidate points are tested with prepared GEOS (shapely.intersects_xy), so no shapely point objects are created. Prepared GEOS geometries are not thread-safe, so each thread prepares its own copy.
  • Each partition is split into chunks so that every core is busy, and the chunks run on the active Dask scheduler, a distributed client included. "sum", "count", "mean", "min" and "max" are reduced per chunk with a pandas groupby over the matched values, and the partial results are combined at the end. Other functions, including callables, collect the matched values of all chunks and run one pandas groupby, as the sjoin() path does.
  • Keys are located with the checks of get_values() (moved to _locate_values()), so ambiguous keys still raise. Values from a table are loaded into memory and aligned by position, as before.
  • The results have the dtypes of the sjoin() path: integer sums are widened, so a uint8 column does not wrap, min and max keep the column's dtype, and mean is computed in float.
  • PointsModel.parse() converts a pandas frame to Dask once. Before, it called dd.from_pandas(sort=True) for the coordinates and again for every column assigned afterwards, and Dask computes the divisions in pure Python over the whole index (about 80 s per call at 624M rows). For a sorted RangeIndex the divisions are now taken directly. The coordinates still get one common dtype.

Benchmark

Hardware: 28 cores of a Xeon Platinum 8480+. Data: Atera FFPE Human Breast Cancer, 170,057 cells.

case main this PR shapely STRtree, 28 processes cuspa (H100)
read_zarr(), 740M transcripts, count by gene, fresh process not run 27 s (47 s with numba compilation)
624M transcripts (qv >= 20, genes) parsed from pandas, first call in a fresh process not run 28 to 34 s (54 s with numba compilation) 63 s 1.97 s
same, repeated call not run 26 s 63 s 1.97 s
0.6M transcripts 406 s 0.76 s 0.98 s 0.17 s
2,500 cells plus one 20k-vertex mask, 200k points 0.43 s 0.27 s

The counts match main at 0.6M (488,124 transcripts in cells), and every (point, geometry) pair matches shapely.STRtree.query(predicate="intersects") in the tests.

Tests

test_aggregate_points_by_shapes_matches_sjoin compares the new path with _aggregate_shapes_sjoin() for sum, count, mean, min, max, median and a callable, on categorical, string, float, uint8, boolean, int64 (above 2^53) and coordinate columns, and checks that the dtypes match. The test data includes boundary points, overlaps, an empty polygon, missing values, a region with only missing values and a region without points. Other new tests cover a label inside another label, the default value_key on transformed points, several categorical keys (which raise), and the compiled kernels in a subprocess, because the test suite disables numba: NaN and infinite coordinates, and many-vertex geometries queried from 8 threads at once.

Limits

  • The first call in a new environment compiles the kernels, which takes about 20 s. Numba caches them, and without a writable cache directory it compiles once per process.
  • Polygon edges are tested in floating point, so a point within rounding error of a slanted edge can differ from GEOS.

Sum/count of point columns by shapes now joins each Dask partition with a shared
shapely.STRtree on a thread pool (shapely releases the GIL) and reduces it to a
sparse matrix, instead of computing all points into a GeoDataFrame and grouping
with observed=False over every region x category pair. Same signature, same
output; other cases keep the geopandas path.
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.86758% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.41%. Comparing base (c359bf5) to head (23a9e67).

Files with missing lines Patch % Lines
...patialdata/_core/operations/_points_in_polygons.py 88.04% 11 Missing ⚠️
src/spatialdata/_core/operations/aggregate.py 94.25% 5 Missing ⚠️
src/spatialdata/models/models.py 85.71% 3 Missing ⚠️
src/spatialdata/_core/query/relational_query.py 94.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1320      +/-   ##
==========================================
- Coverage   91.43%   91.41%   -0.03%     
==========================================
  Files          53       54       +1     
  Lines        8395     8570     +175     
==========================================
+ Hits         7676     7834     +158     
- Misses        719      736      +17     
Files with missing lines Coverage Δ
src/spatialdata/_core/query/relational_query.py 93.37% <94.73%> (+0.02%) ⬆️
src/spatialdata/models/models.py 87.81% <85.71%> (-0.03%) ⬇️
src/spatialdata/_core/operations/aggregate.py 90.28% <94.25%> (+1.07%) ⬆️
...patialdata/_core/operations/_points_in_polygons.py 88.04% <88.04%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Creating shapely point geometries holds the GIL, so the threaded STRtree join
stopped scaling. A grid index plus crossing-number test compiled with numba
(nogil) needs no geometries and matches shapely intersects, including holes,
multipolygons, overlaps and boundary points.
@timtreis timtreis changed the title perf(aggregate): faster points-by-shapes aggregation with a threaded STRtree join perf(aggregate): stream points-by-shapes aggregation through a numba point-in-polygon join Nov 18, 2026
Every points-by-shapes aggregation except fractions=True now joins the points
to the shapes with the numba grid index, one Dask partition at a time on the
active scheduler. sum, count, mean, min and max reduce per chunk; other
aggregations collect the matched values and run one pandas groupby, as the
sjoin path does, so both paths resolve keys and boundaries the same way.

Fixes from review:
- holes: one even-odd test over all rings of a geometry, so labels inside
  labels and touching multipolygon parts count once
- NaN and inf coordinates are skipped before the float-to-int conversion
- empty and non-polygonal geometries match nothing; polygons with more than
  500 vertices are tested with prepared GEOS (intersects_xy) on their grid
  candidates only
- the grid cell size is bounded by the number of geometries
- integer sums are widened like pandas; ambiguous keys raise through the
  checks of get_values (now _locate_values)
- no repartition (it sorted the whole index), no scheduler override, lazy
  import of the kernel module, and jit cache failures fall back to no cache
- the column of ones is removed even if the aggregation raises
@timtreis timtreis changed the title perf(aggregate): stream points-by-shapes aggregation through a numba point-in-polygon join perf(aggregate): numba point-in-polygon join for points by shapes Nov 18, 2026
…isions

PointsModel.parse() called dd.from_pandas(sort=True) once for the
coordinates and again for every column assigned afterwards. dask computes
the divisions in pure Python over the whole index, about 80 s per call at
624M rows. Cast the feature column in pandas, convert the frame once, and
take the divisions of a sorted RangeIndex directly.
- Prepared GEOS geometries are not thread-safe: concurrent intersects_xy
  aborted the process. Each thread now prepares its own copy, the check is
  one vectorised call, and only geometries that are large in vertices and
  in grid cells (masks, not cells) go to GEOS.
- Per-chunk results are partial pandas groupbys of the matched values, so
  dtypes match the sjoin path (int64 exact, bool min/max, mean in float)
  and memory scales with the matched regions, not regions x columns.
  Categorical counts are still summed sparsely.
- The default column of ones is located on the transformed points.
- Several categorical value keys raise instead of dropping all but one.
- PointsModel.parse() casts the coordinates of a pandas frame to one dtype
  again.
@timtreis
timtreis marked this pull request as ready for review November 18, 2026 21:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant