clear cache on prompt ingestion in server (#917)

Co-authored-by: N8 <[email protected]>
2026-02-23 12:13:25 -08:00
parent 321e764e0a
commit d4701ba513
1 changed files with 1 additions and 0 deletions
@@ -1081,6 +1081,7 @@ class BatchGenerator:
                        for uid, length in zip(uids, lengths)
                    ]
                )
+                mx.clear_cache()

        # Further prompt processing so we need to
        #   1. Merge the KV caches and prepare for right padded prompts