feat: implement clip_grad_norm (#1043)

* feat: implement `clip_grad_norm` * pre-commit * Add test for clip_grad_norm function in test_optimizers.py * small fixes * fix * lint * Update tree_reduce * Update python/mlx/utils.py Co-authored-by: Awni Hannun <[email protected]> * Update python/mlx/utils.py Co-authored-by: Awni Hannun <[email protected]> * Update python/mlx/utils.py Co-authored-by: Awni Hannun <[email protected]> * Update python/mlx/utils.py Co-authored-by: Awni Hannun <[email protected]> * Update python/mlx/utils.py Co-authored-by: Awni Hannun <[email protected]> * Update python/mlx/utils.py Co-authored-by: Awni Hannun <[email protected]> * Refactor clip_grad_norm function to include documentation and improve readability * format docstring * Add acknowlegements * text wrap * pre-commit * nits in docs --------- Co-authored-by: Awni Hannun <[email protected]> Co-authored-by: Awni Hannun <[email protected]>
2024-05-03 09:07:02 -07:00
parent b00ac960b4
commit 79c859e2e0
7 changed files with 127 additions and 3 deletions
@@ -7,7 +7,7 @@ with a short description of your contribution(s) below. For example:

 MLX was developed with contributions from the following individuals:

- Nripesh Niketan: Added `softsign`, `softmax`, `hardswish`, `logsoftmax` activation functions. Added `dropout3d` ops. Added `LogicalAnd` and `LogicalOR` ops.
+- Nripesh Niketan: Added `softsign`, `softmax`, `hardswish`, `logsoftmax` activation functions. Added `dropout3d` ops. Added `LogicalAnd` and `LogicalOR` ops. Added `clip_grad_norm` along with `tree_reduce`.
 - Juarez Bochi: Fixed bug in cross attention.
 - Justin Deschenaux: Sine, Cosine, arange, randint, truncated normal, bernoulli, lion optimizer, Dropout2d, linear and logistic regression python example.
 - Diogo Da Cruz: Added `tri`, `tril`, `triu`, `tensordot`, `inner`, `outer`, `tile`, `StreamContext`, `stream` and safetensor support.