* Fix GIL starvation in _generate thread when batch is idle
After a chat completion, the batch_generator stays alive but has no in-flight
requests (batch_results is empty). The original timeout logic used get_nowait()
when batch_generator existed, causing a tight loop that starved the GIL.
This fix uses a 0.1s timeout when the batch is idle (no in-flight requests),
preventing GIL starvation while preserving batching capability.
* format
---------
Co-authored-by: Awni Hannun <awni@apple.com>
* fix(rope): handle batched offsets in SuScaledRoPE for batch_generate
When batch_generate processes variable-length prompts with left-padding,
cache.offset becomes an array. The seq_len comparison now extracts the
max value as a scalar to determine which frequencies to use.
* only use long rope
---------
Co-authored-by: Awni Hannun <awni@apple.com>
* Enhance load_config function to check for config file existence and incorporate eos_token_id from generation_config.json if available
* nits
---------
Co-authored-by: Awni Hannun <awni@apple.com>
* Fix for Devstral-2
Convert cache offset to int for mx.arange compatibility in attention scale
* fix
---------
Co-authored-by: Awni Hannun <awni@apple.com>
* add support for Trinity/AfMoE model
- Implement AfMoE architecture with MoE (128 experts, 8 active per token)
- Dual normalization pattern (4 layer norms per decoder layer)
- Attention with Q/K normalization and learned sigmoid gating
- RoPE only for sliding window attention layers
- muP embedding scaling
- Shared experts support
- Custom quant_predicate for 4-bit quantization (keeps attention/embeddings at 8-bit)
* nits
---------
Co-authored-by: Awni Hannun <awni@apple.com>
* in. com.
* update
* better inference
* update
* updas
* upd.
* closer
* updates
* updates
* nits
* upd. ackn.
* format
* correct masking like the torch version
* add to test
* format
* optimization + format
* nits
* Fast path for generation
* remove linear attetnion cache
* adding it back
* speedbump + format
* clean up ackn.
* Store GLA state as float32 in metal kernel
* Fix operation order in Simple GLA recurrence
* nits
* fix
---------
Co-authored-by: Tarjei Mandt <kernelpool@gmail.com>
Co-authored-by: Awni Hannun <awni@apple.com>
Tarjei Mandt