Skip to content

GH-51816: [GLib] Add filesystem GType registration mechanism - #55464

Open
alytantawyy wants to merge 3 commits into
apache:mainfrom
alytantawyy:fix-glib-filesystem-gtype-registration
Open

alytantawyy wants to merge 3 commits into
apache:mainfrom
alytantawyy:fix-glib-filesystem-gtype-registration

Conversation

@alytantawyy

@alytantawyy alytantawyy commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Provide filesystem wrapper registration so separately packaged bindings can supply their own GLib types. This prepares for the S3 GLib bindings split in #51608.

Closes #51816.

What changes are included in this PR?

  • Add garrow_file_system_register_type() backed by GIO's arrow-file-system extension point.
  • Select wrappers through GIO while preserving built-in mappings and the generic GArrowFileSystem fallback.
  • Validate names and concrete wrapper types, reject duplicate names and wrapper aliases, and synchronize registration and lookup.
  • Add native and Ruby regression tests, including registration directly through GIO, and run the native tests in CI.

Are these changes tested?

Verified locally on macOS arm64:

  • 5 native tests passed.
  • 85 Ruby filesystem tests passed; 12 optional Dataset tests were omitted.
  • Build, formatting, and all applicable pre-commit checks passed.

These results are local. GitHub CI will verify the updated revision after it is pushed.

Are there any user-facing changes?

Adds a filesystem type-registration API and allows unregistered C++ filesystems to use a generic GLib wrapper. Each wrapper type has one registration name within the extension point. Moving the S3 bindings into a separate library is outside this PR's scope.

Was AI used for this PR?

Codex generated the implementation and tests and helped draft this description.

I personally reviewed the full implementation and tests.

Comment thread c_glib/arrow-glib/file-system.cpp Outdated
Comment on lines +552 to +563
struct GArrowFileSystemTypeRegistry
{
std::mutex mutex;
std::unordered_map<std::string, GType> types = {
{"local", GARROW_TYPE_LOCAL_FILE_SYSTEM},
{"hdfs", GARROW_TYPE_HDFS_FILE_SYSTEM},
{"s3", GARROW_TYPE_S3_FILE_SYSTEM},
{"gcs", GARROW_TYPE_GCS_FILE_SYSTEM},
{"abfs", GARROW_TYPE_AZURE_FILE_SYSTEM},
{"mock", GARROW_TYPE_MOCK_FILE_SYSTEM},
};
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using GIOExtensionPoint instead?
https://docs.gtk.org/gio/struct.IOExtensionPoint.html

If you aren't familiar with GIOExtensionPoint, I can push some commits to this branch.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou Addressed. Please check

@kou

kou commented Nov 13, 2026

Copy link
Copy Markdown
Member

BTW, we don't need no human reviewed PR.
Read https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code carefully.

If you use only AI, I'll open a separated PR for the issue.

@alytantawyy

Copy link
Copy Markdown
Contributor Author

separated

@kou the code was fully reviewed and design was implemented by me - just the wording in the PR description and code syntax was by AI - I updated the description apologies for the confusion

@kou

kou commented Nov 15, 2026

Copy link
Copy Markdown
Member

OK. Could you explain the following?

  1. Why don't we need a mutex for garrow_file_system_get_extension_point()?
  2. Why do we need GRecMutex instead of GMutex for garrow_file_system_register_type()?
  3. Is this still thread-safe with "registration directly through GIO"?

@alytantawyy

Copy link
Copy Markdown
Contributor Author

OK. Could you explain the following?

  1. Why don't we need a mutex for garrow_file_system_get_extension_point()?
  2. Why do we need GRecMutex instead of GMutex for garrow_file_system_register_type()?
  3. Is this still thread-safe with "registration directly through GIO"?

Thanks for your questions @kou , I looked into GIO's implementation and here's what I found:

1- garrow_file_system_get_extension_point() uses a function local static variable where its initialization is thread safe in C++11 and later - this ensures that the initialization lambda runs only once even when multiple threads call it concurrently. That's why a mutex isn't needed.

2- I used GRecMutex to account for potential reentrant registration during GIO's lazy module loading - g_io_extension_point_get_extension_by_name() can trigger lazy module loading which may execute initialization code that calls garrow_file_system_register_type() again on the same thread - GMutex would cause a deadlock in this situation whereas GRecMutex allows recursive locking. I chose GRecMutex defensively in case of a possible callback.

3- For direct registration through GIO, the current implementation cannot guarantee thread safety when another module callsg_io_extension_point_implement()directly while Arrow GLib is performing a lookup - from my inspection of GIO's source code, I saw that registration modifies the extension list and lookup traverses it without shared synchronization around those operations. One thing we could do here is clarify the intended registration contract: require external file system modules to register through garrow_file_system_register_type() where thread safety is guaranteed through the mutexes we implemented. Another approach here could be to require filesystem modules to complete registration during initialization before concurrent lookups begin - this could also be documented and clarified as the intended registration contract.

Please let me know what you think the best approach is whenever you get the chance, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[GLib] Add filesystem GType registration mechanism

2 participants