Skip to content

Refactor callbacks in OpenSSL::SSL - #1080

Open
rhenium wants to merge 7 commits into
ruby:masterfrom
rhenium:ky/ssl-callback-errors
Open

rhenium wants to merge 7 commits into
ruby:masterfrom
rhenium:ky/ssl-callback-errors

Conversation

@rhenium

@rhenium rhenium commented Jul 5, 2026 •

Copy link
Copy Markdown
Member

Extracted from #736


x509store: do not suppress exceptions in verify_callback

The verify_callback uses rb_protect() to execute Ruby code, but since
there is no common location it can store the jump tag, it suppresses it
and simply emits a warning to stderr.

Use Fiber-local storage for this purpose. All callers that may invoke
the verify_callback should now use ossl_cb_state_{begin,end}() to
handle the suspended jump.


ossl.c: fix possible exceptions in ossl_clear_error()

When OpenSSL.debug is enabled, ossl_clear_error() allocates String for
the log text and prints it with rb_warn(), which technically may fail.


ssl: use ossl_cb_state_*() for handling exceptions in callbacks

Simplify handling of the temporary rb_protect() jump tags. As a side
effect, this also avoids corruption of OpenSSL's internal state if the
SSLSocket object is frozen in the middle of SSL_*() calls.


ssl: do not call another callback when an exception is pending

Do not invoke another Ruby callback while we are waiting for a chance
to do rb_jump_tag().

As far as I know, this cannot currently happen. Handle it explicitly
in preparation for an upcoming patch that uses a custom BIO_METHOD
backed by Ruby methods in SSLSocket. These callbacks may be called
after Ruby code called from BIO_METHOD has failed, or vice versa.


ssl: harden info_callback

Use rb_protect() to prevent exiting from SSL_accept() prematurely.

Also set the callback to SSL_CTX instead of SSL, similarly to what we
do for other callbacks configurable on SSLContext.


ssl: harden client_cert_cb

Currently, if the callback raises an exception or returns an invalid
value, the handshake is resumed as if the callback has not been called.
This is contrary to the documentation.

Propagate exceptions raised by the callback to the originating method.

Raise an exception if the callback returns an invalid value. It is
immediately caught by rb_protect() and will similarly be re-raised in
the originating method.


ssl: harden session_{get,new,remove}_cb

Prepare the Array objects for the argument inside rb_protect(). This
object allocation may potentially raise an exception and corrupt
OpenSSL's internal state.

Also look up the callback Proc objects directly from SSLContext's
instance variables to simplify.

@rhenium
rhenium force-pushed the ky/ssl-callback-errors branch 2 times, most recently from a05a89f to 5ae433b Compare August 31, 2026 12:32
@rhenium
rhenium force-pushed the ky/ssl-callback-errors branch from 5ae433b to 11bf938 Compare October 13, 2026 09:58
@rhenium
rhenium force-pushed the ky/ssl-callback-errors branch 2 times, most recently from deb1dfb to 79b6a35 Compare November 12, 2026 13:34
The verify_callback uses rb_protect() to execute Ruby code, but since
there is no common location it can store the jump tag, it suppresses it
and simply emits a warning to stderr.

Use Fiber-local storage for this purpose. All callers that may invoke
the verify_callback should now use ossl_cb_state_{begin,end}() to
handle the suspended jump.
When OpenSSL.debug is enabled, ossl_clear_error() allocates String for
the log text and prints it with rb_warn(), which technically may fail.
Simplify handling of the temporary rb_protect() jump tags. As a side
effect, this also avoids corruption of OpenSSL's internal state if the
SSLSocket object is frozen in the middle of SSL_*() calls.
Do not invoke another Ruby callback while we are waiting for a chance
to do rb_jump_tag().

As far as I know, this cannot currently happen. Handle it explicitly
in preparation for an upcoming patch that uses a custom BIO_METHOD
backed by Ruby methods in SSLSocket. These callbacks may be called
after Ruby code called from BIO_METHOD has failed, or vice versa.
Use rb_protect() to prevent exiting from SSL_accept() prematurely.

Also set the callback to SSL_CTX instead of SSL, similarly to what we
do for other callbacks configurable on SSLContext.
Currently, if the callback raises an exception or returns an invalid
value, the handshake is resumed as if the callback has not been called.
This is contrary to the documentation.

Propagate exceptions raised by the callback to the originating method.

Raise an exception if the callback returns an invalid value. It is
immediately caught by rb_protect() and will similarly be re-raised in
the originating method.
Prepare the Array objects for the argument inside rb_protect(). This
object allocation may potentially raise an exception and corrupt
OpenSSL's internal state.

Also look up the callback Proc objects directly from SSLContext's
instance variables to simplify.
@rhenium
rhenium force-pushed the ky/ssl-callback-errors branch from 49451a2 to 9c54359 Compare November 18, 2026 10:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant