Skip to content

Optimize tiled-attention memory footprint, AMX fallback workspace plumbing, and Python bindings in gemma.cpp: - #1091

Open
copybara-service[bot] wants to merge 1 commit into
devfrom
test_1023101399
Open

copybara-service[bot] wants to merge 1 commit into
devfrom
test_1023101399

Conversation

@copybara-service

Copy link
Copy Markdown

Optimize tiled-attention memory footprint, AMX fallback workspace plumbing, and Python bindings in gemma.cpp:

  • Plumb preallocated worker_workspace through DispatchTileFlashAttentionReturnExpSumsAndMaxLogitsAMX for non-AMX fallback execution.
  • Skip unused flat/transposed KVCache buffers when IsTiledAttention is active, gate ViT KV buffers on vocab_size == 0, and add KVCache::ZeroInitPrefix.
  • Expose seq_len, prefill_tbatch_size, attention_impl, map_weights, and tokenize in Python GemmaModel bindings.

…mbing, and Python bindings in gemma.cpp:

- Plumb preallocated worker_workspace through DispatchTileFlashAttentionReturnExpSumsAndMaxLogitsAMX for non-AMX fallback execution.
- Skip unused flat/transposed KVCache buffers when IsTiledAttention is active, gate ViT KV buffers on vocab_size == 0, and add KVCache::ZeroInitPrefix.
- Expose seq_len, prefill_tbatch_size, attention_impl, map_weights, and tokenize in Python GemmaModel bindings.

PiperOrigin-RevId: 1023101399
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant