Skip to content

Speed up HeatMap redraws for large datasets - #2351

Open
KhadeerBasha1232 wants to merge 1 commit into
python-visualization:mainfrom
KhadeerBasha1232:heatmap-cache-projection
Open

KhadeerBasha1232 wants to merge 1 commit into
python-visualization:mainfrom
KhadeerBasha1232:heatmap-cache-projection

Conversation

@KhadeerBasha1232

Copy link
Copy Markdown
Contributor

Speed up HeatMap redraws for large datasets

Closes #1430

Summary

HeatMap gets slow with many points, especially when zoomed in: with 20,000 points spread over the world, a single redraw at zoom 13 took ~5 seconds. This PR patches the leaflet_heat.min.js file folium hosts so redraws are much faster, without changing what is drawn.

Before After
20k points, zoom 13 ~4.9 s ~14 ms
1M points, zoom 9 ~1.5 s ~0.26 s
Output — identical (same points, weights, max and pixels)

Why it was slow

Every pan or zoom runs _redraw, which:

  1. Re-projects every point with map.latLngToContainerPoint, re-running the map projection (trigonometry plus several object allocations per point) even though a point's projected position never changes.
  2. Bins the points into a grid stored as sparse arrays. At high zoom, off-screen points get row/column indices in the millions.
  3. Scans those sparse arrays end to end to find the on-screen cells. At high zoom this loop runs over millions of empty slots, and it is what made zoomed-in maps take seconds.

What this PR changes

The patch is an addition at the end of the file (the original code is untouched). It overrides _redraw to:

  • Project each point once and cache it. On redraw, only Leaflet's per-zoom scaling is applied, using the same arithmetic and rounding as latLngToContainerPoint, so pixel positions are bit-identical. The cache is rebuilt by setLatLngs and extended by addLatLng.
  • Store the grid in Maps instead of sparse arrays.
  • Only visit the on-screen part of the grid in the final pass. A cell is only drawn if its first point is inside the bounds, so cells outside that row/column range can never be drawn.

What stays the same

  • The drawn heatmap. The intensity scale (_max) is still computed from all points at the current zoom, as the weights patch (Leaflet.heat#78) in this file expects. I deliberately did not add the spatial index (R-tree) the issue suggests: skipping off-screen points would change _max, so colours would shift as you pan.
  • Custom projections. Maps whose CRS overrides latLngToPoint (e.g. some Proj4Leaflet setups) fall back to the original _redraw.
  • The Python side. folium/plugins/heat_map.py is unchanged.

Benchmarks

Measured in Chromium against the current file, with seeded random data (80% clustered around six cities, 20% spread over the world, random weights). Each view compares the exact point list passed to simpleheat, the computed _max, and a hash of the canvas pixels.

20,000 points

View Before After Speed-up Identical
World, zoom 2 34.6 ms 31.8 ms 1.1× ✅
Europe, zoom 5 26.8 ms 12.9 ms 2.1× ✅
Paris, zoom 9 132 ms 24.3 ms 5.4× ✅
NYC, zoom 11 1,149 ms 18 ms 64× ✅
Paris, zoom 13 4,934 ms 13.5 ms 365× ✅
Paris, zoom 13 (panned) 4,869 ms 14.8 ms 329× ✅
Empty ocean, zoom 8 55.9 ms 12 ms 4.6× ✅

1,000,000 points

View Before After Speed-up Identical
World, zoom 2 279 ms 102 ms 2.7× ✅
Europe, zoom 5 432 ms 140 ms 3.1× ✅
Paris, zoom 9 1,517 ms 261 ms 5.8× ✅
Empty ocean, zoom 8 878 ms 184 ms 4.8× ✅
NYC, zoom 11 not run* 490 ms — —
Paris, zoom 13 not run* 592 ms — —

* The old code already took ~5 s at these zoom levels with only 20k points, so I didn't run it with 1M.

At world zoom, drawing the canvas dominates, so the gain there is small.

How I verified it

  • Output identical to the current file in all views above, comparing the point list, _max and canvas pixels.
  • Also identical for: addLatLng, setLatLngs, L.LatLng objects with alt weights, points without weights, L.CRS.Simple and L.CRS.EPSG4326.
  • A CRS that overrides latLngToPoint correctly falls back to the original _redraw.
  • End to end: a folium HeatMap (50k points) loading the patched file draws correctly while zooming and panning, with no console errors.

Notes for reviewers

  • heat_map.py loads this file from cdn.jsdelivr.net/gh/python-visualization/folium@main/..., so once merged it applies to all HeatMaps right away, not only after the next release.
  • Like the original, the cache assumes points are changed through setLatLngs / addLatLng. Code that edits layer._latlngs[i] in place would need to call setLatLngs afterwards.
  • The pre-commit.ci failure (if it shows) comes from tests/test_features.py on main and is fixed in Fix black formatting in test_features.py #2350.

Leaflet.heat's _redraw re-projected every point with
map.latLngToContainerPoint on every pan and zoom, stored its grid in
sparse arrays whose indices grow with the zoom level, and then scanned
those arrays end to end. At high zoom with points spread over the
world, that final scan alone took seconds.

Patch the hosted leaflet_heat.min.js so that it:
- projects each point once and caches it, applying only Leaflet's
  per-zoom scaling on redraw (same arithmetic, same rounding);
- keeps the grid in Maps instead of sparse arrays;
- only visits the on-screen part of the grid in the final pass.

The drawn heatmap is unchanged: the same points, weights, maximum and
pixels as before. Maps whose CRS overrides latLngToPoint keep using
the original code.

Closes python-visualization#1430

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
pypi — 26c3fb1c Deployed Nov 16, 2026 by KhadeerBasha1232 via packages #2098
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Heatmap scalability

1 participant