Repository navigation
Don't walk route_ht while holding peertab->lock - #147
randomizedcoder wants to merge 1 commit into
Conversation
|
Heads-up: this conflicts with #144 at the end of |
The rhashtable walk functions acquire the table's internal lock with spin_lock, and the rhashtable rehash worker holds that lock with softirqs enabled. peertab->lock is acquired in softirq context (homa_route_get, homa_route_free), so holding it while walking the table can deadlock: lockdep reports "SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected" from homa_peer_free_net at netns exit, and homa_route_gc has the same problem. Add peertab->walk_mutex to serialize the two walks (and protect route_ht_iter), and acquire peertab->lock only around the removals. homa_route_gc now holds the RCU read lock from choosing victims until it has removed them, since another thread may remove a victim in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4583bad to
dc8c5fa
Compare
|
I'm finally getting around to looking at this PR, and I am wondering if there is a better fix. Part of the problem is that the peertab is shared across network namespaces, so homa_peer_free_net can be cleaning up one namespace while garbage collection is occurring in another. I think that it is actually incorrect to share peer and routing information across namespaces; this information should be private to each namespace. If I make peertabs private, I assume this would eliminate the need for synchronization in homa_peer_free, since there should be no competing access to the peertab at this point. Would the changes to homa_route_gc still be needed (hash table resizing could be running concurrently)? |
rhashtable_walk_enter,_startand_stoptake the table'sht->lockwith plainspin_lock, and the rehash worker holds it with softirqs enabled.peertab->lockis taken in softirq (homa_route_getfromhoma_rpc_alloc_server,homa_route_freefrom an RCU callback). Two places holdpeertab->lockwhile walkingroute_ht:homa_peer_free_net(netns exit);homa_route_gc→homa_route_pick_victims.That can hard-hang: CPU0's rehash worker holds
ht->lockand a Homa softirq on CPU0 spins onpeertab->lock, while CPU1 holdspeertab->lockand spins onht->lock. It's rare (a rehash, a netns exit or GC, and traffic at the same moment), but lockdep reports it on every run that exits a netns:Fix:
peertab->walk_mutexserializes the two walks and protectsroute_ht_iter(it's acquired beforepeertab->lock; both callers run in process context).homa_peer_free_nettakespeertab->lockonly around each removal and counter update.homa_route_gcpicks victims under the mutex andrcu_read_lock(), then takespeertab->lockfor the removals.rhashtable_remove_fast's existing check skips a victim someone else removed in between, and RCU keeps it valid until then.READ_ONCEon the locklessnum_routesreads.If you'd rather not add a lock: a fresh local iterator per GC pass would also work, but it loses the persistent cursor, so GC would always restart from the beginning of the table.
Checked:
homa_peer_free_net__*,homa_route_gc__*,homa_route_pick_victims__*).__STRIP__=1.No test: the unit harness has no lockdep, and the lock order isn't otherwise observable.
🤖 Generated with Claude Code