Skip to content

Wait for concurrent reapers in homa_sock_destroy - #153

Open
breakertt wants to merge 1 commit into
PlatformLab:mainfrom
breakertt:fix/destroy-wait-for-reapers
Open

breakertt wants to merge 1 commit into
PlatformLab:mainfrom
breakertt:fix/destroy-wait-for-reapers

Conversation

@breakertt

Copy link
Copy Markdown
Contributor

homa_sock_destroy waits only until dead_rpcs is empty. But homa_timer or SoftIRQ can still free the last batch that it took off the list. That reaper returns the buffers to the pool and subtracts their frag_bytes from sk_wmem_alloc. Destroy then hits WARN_ON_ONCE(sk_wmem_alloc != 1) and frees the pool under that reaper.

This patch counts the reaps in progress in hsk->reapers. Destroy increments protect_count, so a new reap returns at once, and then waits until reapers is 0.

Test on two xl170 nodes: 16 threads each keep 150 8 KB RPCs in flight, then all close their sockets together. On upstream, destroy found sk_wmem_alloc != 1 in 318 of 128000 closes. In one run, homa_timer then oopsed in homa_pool_free_bufs (NULL buffer_pool) and 2 sockets leaked. With this patch: 0 of 144000. cp_node throughput does not change.

Reproduction, logs and the oops: https://github.com/breakertt/HomaModule/tree/fix-artifact/destroy-wait-for-reapers

* homa_rpc_reap takes RPCs off dead_rpcs under the socket lock and frees
  them after unlocking; since d00a93b that is also when it subtracts
  their frag_bytes from sk_wmem_alloc. homa_sock_destroy only waited for
  dead_rpcs to be empty (its comment wrongly said that all references
  were gone), so while homa_timer or softirq still freed the last batch
  it hit WARN_ON_ONCE(sk_wmem_alloc != 1) and freed the buffer pool under
  the reaper.
* Count the reaps in progress in hsk->reapers. homa_sock_destroy now
  increments protect_count so that no new reap starts, and waits for
  hsk->reapers to reach zero before it checks sk_wmem_alloc and frees
  the pool.
@johnousterhout

johnousterhout commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Thanks for pointing out the bug, and for the suggested fix. However, I'm a bit uncomfortable with this PR because it adds yet another synchronization variable; Homa's synchronization is already way too complicated, so I hate to make it even more so.

Is there any other way to fix this problem without introducing a new synchronization variable? Homa already has a bunch of variables related to synchronizing sockets, such as its lock, its reference count, hsk->shutdown, and protect_count. Is there any way that some combination of these could be used to solve the problem?

@breakertt

Copy link
Copy Markdown
Contributor Author

Hi,

Thanks for your reply! To be honest, I am also uncomfortable with that. I will double check and get you back.

Cheers,
Tianyi

@Stuartwastaken

Copy link
Copy Markdown
Contributor

One way to do this with the existing reference count: free the pool in sk->sk_destruct instead of homa_sock_destroy. homa_timer (through the socktab scan) and softirq (through homa_sock_find) both hold a reference on the socket while they reap, and sk_destruct only runs after the last reference is dropped (and an RCU grace period, since Homa sets SOCK_RCU_FREE). So a reap in progress always finishes before the pool is freed. homa_sock_destroy still shuts the pool down and cancels its wakeup work, since that has to sleep.

The sk_wmem_alloc WARN in homa_sock_destroy is racy for the same reason, so I removed it.

I have this on a branch, with a unit test that hits the use-after-free under ASan without the change: Stuartwastaken#14. Tianyi, feel free to fold it into #153 if it works for your repro; I haven't run your xl170 test.

@breakertt

Copy link
Copy Markdown
Contributor Author

One way to do this with the existing reference count: free the pool in sk->sk_destruct instead of homa_sock_destroy. homa_timer (through the socktab scan) and softirq (through homa_sock_find) both hold a reference on the socket while they reap, and sk_destruct only runs after the last reference is dropped (and an RCU grace period, since Homa sets SOCK_RCU_FREE). So a reap in progress always finishes before the pool is freed. homa_sock_destroy still shuts the pool down and cancels its wakeup work, since that has to sleep.

The sk_wmem_alloc WARN in homa_sock_destroy is racy for the same reason, so I removed it.

I have this on a branch, with a unit test that hits the use-after-free under ASan without the change: Stuartwastaken#14. Tianyi, feel free to fold it into #153 if it works for your repro; I haven't run your xl170 test.

Hi Stuart,

Thanks a lot for looking into this and for putting the branch together. Using the socket reference count is a neat idea, and it does keep the pool alive for a reaper in progress. I have two concerns:

  1. homa_sock_destroy still does not wait for a reaper in progress. After ->destroy returns, sock_orphan sets sk->sk_wq to NULL, and the reaper calls homa_sock_wakeup_wmem after it unlocks the socket, which can then dereference NULL. In the version I am testing, the reaper does the wakeup before it unlocks the socket, and destroy waits for the socket lock, so this window is closed.

  2. Since John asked for less complexity, I would like to keep the change as small as possible, and mine ends up quite a bit smaller than moving the pool free to sk_destruct.

I will post a v2 soon. Comments and reviews are very welcome.

Cheers,
Tianyi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants