electron-rare/mlx-lm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
828 Commits 18 Branches 29 Tags
main
Commit Graph

215 Commits

Author SHA1 Message Date
Angelos Katharopoulos ed1fca4cef Thread local generation stream (#1090) 2026-04-22 00:34:09 -07:00
glyphVault 4f5cbd2a4f Fix Gemma 4 KV-shared layers creating unused projections (#1158) 2026-04-21 16:44:13 -07:00
Angelos Katharopoulos 3cd9a52df2 Fix ArraysCache extend (#1177) 2026-04-21 16:41:49 -07:00
Eyüp Can Akman 2f1ab85aec Fix Mistral empty tool_call_end flipping state machine to normal (#1151) 2026-04-21 01:36:56 -07:00
AkashKhamkar f3bb10c141 Fix Gemma4 tool parser: support hyphenated names and braces in strings (#1150) 2026-04-21 01:15:58 -07:00
Sherry Lo e1c24b3237 fix: handle NoneType check for think tokens in TokenizerWrapper (#1167) 2026-04-21 01:13:15 -07:00
TechToboggan a9856b485d Fix batch dimension mismatch in ArraysCache extend() (#1169) 
Co-authored-by: Tristan Stahnke <tristan@melchior.lan>
Co-authored-by: Angelos Katharopoulos <a_katharopoulos@apple.com>
 2026-04-20 23:42:03 -07:00
Tarjei Mandt 6d114686e5 Fix MiniMax M2 parallel tool calling (#1171) 2026-04-20 10:57:20 -07:00
Tarjei Mandt aa4f880fb3 Fix parallel tool call handling in server (#1170) 2026-04-19 23:58:25 -07:00
razorback16 62f38aeb51 Fix batch dimension mismatch in BatchKVCache and BatchRotatingKVCache extend() (#1141) 2026-04-14 17:21:31 -07:00
Neil Mehta dcbf6e33d1 Align batch logits processor token contract (#1115) 2026-04-06 18:07:35 -07:00
Angelos Katharopoulos f26fddfd3b Gemma4 final fixes and multi-token think/tool start/end (#1114) 2026-04-06 17:40:48 -07:00
spicyneuron c65c27b450 Fix Gemma 4 quantized per-layer projection loading (#1112) 2026-04-06 13:26:07 -07:00
Nic Davidson 3257c3df17 Add Gemma 4 tool call parser (#1105) 2026-04-04 17:21:45 -07:00
Angelos Katharopoulos d4eb136d44 Bring back max-kv-size to the batch generator (#1106) 2026-04-04 17:12:43 -07:00
Prince Canuma 4469ad4647 Add gemma 4 (#1093) 
Co-authored-by: N8 <n8@n8programs.com>
Co-authored-by: Angelos Katharopoulos <a_katharopoulos@apple.com>
 2026-04-04 04:47:03 -07:00
Angelos Katharopoulos 3f9d179fd1 Batch generation refactoring and various fixes (#1072) 2026-04-01 15:07:50 -07:00
Lik Xun Yuan (Lx) 9dc023beed Fix PromptTrie.pop_prefixes() off-by-one when pruning immediate prefixes (#1078) 
Signed-off-by: Yuan Lik Xun <lxyuan0420@gmail.com>
Co-authored-by: Angelos Katharopoulos <a_katharopoulos@apple.com>
 2026-04-01 11:51:19 -07:00
Angelos Katharopoulos 4d3af3cebc Refactor LRUPromptCache (#1019) 2026-03-26 10:04:22 -07:00
Angelos Katharopoulos f8019f7769 Fix flaky test (#1020) 2026-03-18 12:51:22 -07:00
mm65x ed69f837e6 fall back to ast.literal_eval for malformed JSON in qwen3_coder tool parser (#1004) 2026-03-15 23:00:51 -07:00
mm65x 2146e4ed18 avoid mutating input in SuScaledRoPE and YarnRoPE (#1003) 2026-03-15 18:13:47 -07:00
Angelos Katharopoulos 735a43b275 Delta net precision (#997) 2026-03-15 15:39:18 -07:00
Ryo Ota 332d94ca6f Add allowed-origins to the server (#987) 2026-03-13 19:22:23 -07:00
Angelos Katharopoulos ab157c2d18 Fix test after latest MLX update (#996) 2026-03-13 17:07:05 -07:00
Eyüp Can Akman 760c5abcc8 Fix CompletionsDataset mask_prompt crash (#967) 2026-03-10 18:10:06 -07:00
Angelos Katharopoulos 4a21ffdf4b Presence and frequency penalties (#971) 2026-03-09 22:26:39 -07:00
Angelos Katharopoulos 2105aaf9c3 Better caching in the server (#911) 2026-03-06 13:42:56 -08:00
Yongyue Sun 35fa620279 Add --prefill-step-size as cmd line argument (#943) 2026-03-04 17:40:01 -08:00
Noah Lyons 8162aaad56 step3p5: use rotating cache for sliding attention layers (#949) 2026-03-04 17:17:29 -08:00
Angelos Katharopoulos 83ff9c96d5 Improve the cache size limits (#906) 2026-02-19 10:13:48 -08:00
Angelos Katharopoulos bcf630614f Fix save/load of CacheList (#886) 2026-02-12 18:41:48 -08:00
JJJYmmm 0fd3126496 [MODEL] support qwen3.5 series w/o vision (#869) 
* support text-only qwen3.5 series

Co-authored-by: johnmai-dev <johnmai-dev@users.noreply.github.com>

* add test

* fix sanitize and add test

* make it more readable

* fix lint

---------

Co-authored-by: johnmai-dev <johnmai-dev@users.noreply.github.com>
 2026-02-12 07:23:51 -08:00
Awni Hannun 82edd51a1e Devstral tool parser (#874) 2026-02-11 09:41:23 -08:00
viktike 84ae19e675 Pythonic tool calling for LFM2 models (#864) 
* Fix tool calling 404 Error: content with non-thinking (instruct) models.

* Add pythonic style tool call parser for LFM2

* test + format

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-02-09 15:23:07 -08:00
Tarjei Mandt fd6959dca7 Fix Kimi Linear (#853) 
* Fix Kimi Linear

* Avoid concat/split

* Use fused rms_norm
 2026-02-06 17:31:32 -08:00
Awni Hannun f18526f8d6 DSV3 MLA (#839) 
* mla

* try to speed up prefill

* update dsv32 as well
 2026-02-04 12:06:42 -08:00
Tarjei Mandt 25a4c8369e Fix sliding window mask during generation (#843) 
* Fix sliding window mask during generation

* make window mask for regular cache

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-02-04 08:44:01 -08:00
Awni Hannun e08ec15b72 Fix batch mamba (#842) 
* fix batch mamba

* remove mamba cache
 2026-02-03 19:31:42 -08:00
Josh Lehman 942b3ed4b6 fix: handle GLM 4.7 tool call fallbacks (#792) 
* fix: handle glm47 tool call fallbacks

- add JSON/plain-text fallback parsing for GLM 4.7 tool calls
- normalize fallback argument values using tool schema
- add tests covering JSON/plain-text fallback paths
- Refs: n/a (pebbles not initialized)

Regeneration-Prompt: |
  Fix GLM 4.7 tool parser crash when the <arg_key> regex does not match. Keep the
  existing arg_key/arg_value parsing path intact, but add defensive fallbacks:
  first try JSON tool-call shapes (name+arguments, function+arguments, or nested
  tool objects), then a plain-text form like "name {json}" or "name key=value".
  If none parse, return a safe unknown tool with raw text in arguments. Use tool
  schema types to preserve string arguments and deserialize non-string values.
  Add tests in tests/test_tool_parsing.py that reproduce the crash with JSON
  input and verify the plain-text fallback behavior. Avoid dependency changes.

* simplify test

* rebase

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-02-03 06:48:22 -08:00
Tarjei Mandt 1630f9bf16 Add Step 3.5 Flash (#836) 
* Add Step 3.5 Flash

* Shard model

* Feedback
 2026-02-02 18:48:46 -08:00
Angelos Katharopoulos 1fe1b3c901 Support distributed inference in the server (#741) 2026-01-30 11:03:55 -08:00
Tarjei Mandt 56b8c0f383 Add LongCat Flash Lite (#819) 
* Add LongCat Flash Lite

* Fix integer overflow

* Reuse LongCat Flash components

* proper cache

* use arrays cache

* add test for longcat

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-01-29 13:31:18 -08:00
Tarjei Mandt 04fd7ccb97 Fix Kimi K2.5 tool call handling (#821) 
* Fix Kimi K2 tool call handling

* Add unit test
 2026-01-29 12:35:51 -08:00
Tarjei Mandt b012e1a1e9 Add LongCat Flash tool parser (#810) 
* Add LongCat Flash tool parser

* Add unit tests for both xml and json formats
 2026-01-26 11:51:44 -08:00
lpalbou beceb5c16d Fix ArraysCache.from_state not initializing left_padding and lengths (#807) 
* Fix ArraysCache.from_state not initializing left_padding and lengths

The base class from_state uses __new__ to bypass __init__, but
ArraysCache.state.setter only sets self.cache, not left_padding
or lengths. This causes AttributeError when using loaded caches
with models that use MambaCache (e.g., Qwen3-Next).

The fix overrides from_state in ArraysCache to properly initialize
these attributes before setting state.

* Add test

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-01-25 06:46:35 -08:00
Evan Quiney e5ddaff99b add kimi tool parser (#791) 
* add kimi parser

* fix + test

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-01-22 07:46:07 -08:00
Gökdeniz Gülmez 14b9faf1bc Adding TeleChat3 (#773) 
* in. com.

* Add Telechat3 MLP and Decoder Layer implementations

* Implement Telechat3 model and decoder layer with attention mechanism

* Add support for 'telechat3-yarn' in initialize_rope function

* Update Acknowledgments and add Telechat3 model tests

* format

* fix test

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-01-22 07:35:58 -08:00
n8programs 1423702019 Update glm4_moe_lite To Store KV Latent in Cache (#780) 
* update glm4_moe_lite

* Fix GLM4 MoE Lite RoPE and MoE layer selection

* Apply PR #780 MLA KV latent cache

* Use full KV for prefill with MLA cache

* Remove unused GLM4 MoE lite config fields

* simplify

* updates

* try and load already quantized models

---------

Co-authored-by: N8 <n8@n8programs.com>
Co-authored-by: ivanfioravanti <ivan.fioravanti@gmail.com>
Co-authored-by: Awni Hannun <awni@apple.com>
 2026-01-21 13:55:14 -08:00
Ivan Fioravanti 02228601cd Add glm4 moe lite model (#776) 
* Add glm4 moe lite model

* Format glm4_moe_lite with black

* version

---------

Co-authored-by: Awni Hannun <awni@apple.com>
 2026-01-19 08:13:13 -08:00