Activations LeakyReLU / PReLU / Softplus / Mish (#109)

* Leaky_relu / prelu / softplus / mish * added tests * updated bench * remove torch refs, add init to PReLU * added arvix reference to mish * added missing docs
2023-12-11 19:40:57 -08:00
parent f5df47ec6e
commit 02de234ef0
8 changed files with 132 additions and 30 deletions
@@ -146,10 +146,12 @@ Neural Network Layers

   Embedding
   ReLU
+   PReLU
   GELU
   SiLU
   Step
   SELU
+   Mish
   Linear
   Conv1d
   Conv2d
@@ -171,9 +173,11 @@ simple functions.
   gelu_approx
   gelu_fast_approx
   relu
+   prelu
   silu
   step
   selu
+   mish

 Loss Functions
 --------------