Repository navigation
Conversation
* homa_rpc_reap takes RPCs off dead_rpcs under the socket lock and frees them after unlocking; since d00a93b that is also when it subtracts their frag_bytes from sk_wmem_alloc. homa_sock_destroy only waited for dead_rpcs to be empty (its comment wrongly said that all references were gone), so while homa_timer or softirq still freed the last batch it hit WARN_ON_ONCE(sk_wmem_alloc != 1) and freed the buffer pool under the reaper. * Count the reaps in progress in hsk->reapers. homa_sock_destroy now increments protect_count so that no new reap starts, and waits for hsk->reapers to reach zero before it checks sk_wmem_alloc and frees the pool.
|
Thanks for pointing out the bug, and for the suggested fix. However, I'm a bit uncomfortable with this PR because it adds yet another synchronization variable; Homa's synchronization is already way too complicated, so I hate to make it even more so. Is there any other way to fix this problem without introducing a new synchronization variable? Homa already has a bunch of variables related to synchronizing sockets, such as its lock, its reference count, hsk->shutdown, and protect_count. Is there any way that some combination of these could be used to solve the problem? |
|
Hi, Thanks for your reply! To be honest, I am also uncomfortable with that. I will double check and get you back. Cheers, |
|
One way to do this with the existing reference count: free the pool in sk->sk_destruct instead of homa_sock_destroy. homa_timer (through the socktab scan) and softirq (through homa_sock_find) both hold a reference on the socket while they reap, and sk_destruct only runs after the last reference is dropped (and an RCU grace period, since Homa sets SOCK_RCU_FREE). So a reap in progress always finishes before the pool is freed. homa_sock_destroy still shuts the pool down and cancels its wakeup work, since that has to sleep. The sk_wmem_alloc WARN in homa_sock_destroy is racy for the same reason, so I removed it. I have this on a branch, with a unit test that hits the use-after-free under ASan without the change: Stuartwastaken#14. Tianyi, feel free to fold it into #153 if it works for your repro; I haven't run your xl170 test. |
Hi Stuart, Thanks a lot for looking into this and for putting the branch together. Using the socket reference count is a neat idea, and it does keep the pool alive for a reaper in progress. I have two concerns:
I will post a v2 soon. Comments and reviews are very welcome. Cheers, |
homa_sock_destroywaits only untildead_rpcsis empty. Buthoma_timeror SoftIRQ can still free the last batch that it took off the list. That reaper returns the buffers to the pool and subtracts theirfrag_bytesfromsk_wmem_alloc. Destroy then hitsWARN_ON_ONCE(sk_wmem_alloc != 1)and frees the pool under that reaper.This patch counts the reaps in progress in
hsk->reapers. Destroy incrementsprotect_count, so a new reap returns at once, and then waits untilreapersis 0.Test on two xl170 nodes: 16 threads each keep 150 8 KB RPCs in flight, then all close their sockets together. On upstream, destroy found
sk_wmem_alloc != 1in 318 of 128000 closes. In one run,homa_timerthen oopsed inhoma_pool_free_bufs(NULLbuffer_pool) and 2 sockets leaked. With this patch: 0 of 144000. cp_node throughput does not change.Reproduction, logs and the oops: https://github.com/breakertt/HomaModule/tree/fix-artifact/destroy-wait-for-reapers