Skip to content

Migrate gemma_cpp SentencePieceTokenizer to SentencePiece Lite (SentencePieceLiteProcessor), supporting both serialized ModelProto and FlatBuffer (.spm.fb) model blobs. - #1087

Open
copybara-service[bot] wants to merge 1 commit into
devfrom
test_1017935613

Conversation

@copybara-service

Copy link
Copy Markdown

Migrate gemma_cpp SentencePieceTokenizer to SentencePiece Lite (SentencePieceLiteProcessor), supporting both serialized ModelProto and FlatBuffer (.spm.fb) model blobs.

Performance & Memory Improvements on Gemma 3 (256K BPE):

  • Encoding throughput:
    • English (FineWeb2): 41.70 MB/s vs 1.42 MB/s with standard SentencePiece (~29.4x faster)
    • Multilingual Mixed (FineWeb2): 26.50 MB/s vs 2.64 MB/s (~10.0x faster)
    • Japanese (FineWeb2): 31.84 MB/s vs 9.15 MB/s (~3.5x faster)
  • Decoding throughput: up to 9.4x faster
  • Runtime heap memory:
    • On-the-fly ModelProto -> FlatBuffer conversion: 10.3 MB vs 43.4 MB (~4.2x smaller / ~76% reduction)
    • Note: SentencePiece Lite can also run zero-copy on a FlatBuffer (.spm.fb) with ~1.2 KB of heap, but this is not realized by this CL yet because the tokenizer still copies the given model buffer.

Load time:

  • When a serialized ModelProto is given, it is converted to a FlatBuffer on the fly, so initialization becomes slightly slower for BPE models because BPE direct mappings are precomputed during the conversion (Gemma 3 262K BPE: ~334 ms vs ~211 ms with standard SentencePiece, per sentencepiece_model_converters.h).
  • Loading a pre-converted FlatBuffer (.spm.fb) skips this conversion.

Open-source build:

  • Bump the pinned SentencePiece to v0.2.3, the first release that ships SentencePiece Lite.
  • CMake: link sentencepiece_lite and sentencepiece_lite_fb_converter instead of sentencepiece-static.
  • Bazel: depend on the sentencepiece module through archive_override (it is not in the Bazel Central Registry yet) and drop the v0.1.96 overlay BUILD file and patch.

@google-cla

google-cla Bot commented Nov 11, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@copybara-service
copybara-service Bot force-pushed the test_1017935613 branch 4 times, most recently from 976b66a to 71ecd5f Compare November 17, 2026 05:03
…encePieceLiteProcessor`), supporting both serialized `ModelProto` and FlatBuffer (`.spm.fb`) model blobs.

Performance & Memory Improvements on Gemma 3 (256K BPE):
- Encoding throughput:
  - English (FineWeb2): 41.70 MB/s vs 1.42 MB/s with standard SentencePiece (~29.4x faster)
  - Multilingual Mixed (FineWeb2): 26.50 MB/s vs 2.64 MB/s (~10.0x faster)
  - Japanese (FineWeb2): 31.84 MB/s vs 9.15 MB/s (~3.5x faster)
- Decoding throughput: up to 9.4x faster
- Runtime heap memory:
  - On-the-fly `ModelProto` -> FlatBuffer conversion: 10.3 MB vs 43.4 MB (~4.2x smaller / ~76% reduction)
  - Note: SentencePiece Lite can also run zero-copy on a FlatBuffer (`.spm.fb`) with ~1.2 KB of heap, but this is not realized by this CL yet because the tokenizer still copies the given model buffer.

Load time:
- When a serialized `ModelProto` is given, it is converted to a FlatBuffer on the fly, so initialization becomes slightly slower for BPE models because BPE direct mappings are precomputed during the conversion (Gemma 3 262K BPE: ~334 ms vs ~211 ms with standard SentencePiece, per sentencepiece_model_converters.h).
- Loading a pre-converted FlatBuffer (`.spm.fb`) skips this conversion.

Open-source build:
- Bump the pinned SentencePiece to v0.2.3, the first release that ships SentencePiece Lite.
- CMake: link `sentencepiece_lite` and `sentencepiece_lite_fb_converter` instead of `sentencepiece-static`.
- Bazel: depend on the `sentencepiece` module through `archive_override` (it is not in the Bazel Central Registry yet) and drop the v0.1.96 overlay BUILD file and patch.

PiperOrigin-RevId: 1017935613
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants