Skip to content

ssl: clear callback state after a failed handshake - #1157

Closed
wtn wants to merge 1 commit into
ruby:masterfrom
wtn:callback
Closed

wtn wants to merge 1 commit into
ruby:masterfrom
wtn:callback

Conversation

@wtn

@wtn wtn commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

When a callback raises during SSLSocket#connect or SSLSocket#accept, the socket keeps the stored callback state. Since aac9ce1 (#821), the next SSLSocket#sysread or SSLSocket#syswrite on the socket calls rb_jump_tag() with the stored callback state while no exception is pending, and the VM crashes.

require "openssl"
require "socket"

sctx = OpenSSL::SSL::SSLContext.new
sctx.servername_cb = proc { raise "boom" }
sock1, sock2 = UNIXSocket.pair
Thread.new {
  client = OpenSSL::SSL::SSLSocket.new(sock2)
  client.hostname = "localhost"
  client.connect rescue nil
}

server = OpenSSL::SSL::SSLSocket.new(sock1, sctx)
server.accept rescue p $!
#=> #<RuntimeError: boom>
server.syswrite("abc") rescue p $!
#=> [BUG] Segmentation fault at 0x000000000000000c

Releases from 3.3.0 on likewise crash.

Fix

ossl_start_ssl() now clears the stored callback state before calling rb_jump_tag(). ossl_ssl_read() and ossl_ssl_write() already clear the stored callback state before calling rb_jump_tag(). The syswrite in the example raises OpenSSL::SSL::SSLError.

When a callback raises during SSLSocket#connect or #accept, the stored
callback state is kept, and the next SSLSocket#sysread or #syswrite
calls rb_jump_tag() with the stored callback state and crashes the VM.
Clear the callback state in ossl_start_ssl() before rb_jump_tag().

Co-authored-by: Claude Mythos 5.2 <noreply@anthropic.com>
@rhenium

rhenium commented Nov 18, 2026

Copy link
Copy Markdown
Member

Thanks for spotting this!

I'm closing this PR since it would cause conflicts with #1080, which will remove the ivar ID_callback_state entirely.

OpenSSL::SSL actually has quite a few similar issues with jump tag handling when used in unexpected ways. This one should be fixed as part of the larger refactoring in that PR.

@rhenium rhenium closed this Nov 18, 2026
@wtn
wtn deleted the callback branch November 18, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants