Transformer fix (#167)

* add transformer with dropout, fix transformer ffm, layernorm order * precommit changes * precommit changes * add docstring, activation, norm_first * run precommit * run precommit * add doctstring * precommit * style nits in docs --------- Co-authored-by: junwoo-yun <[email protected]> Co-authored-by: Awni Hannun <[email protected]>
2023-12-27 08:48:36 -08:00
parent 79c95b6919
commit 4417e37ede
5 changed files with 220 additions and 77 deletions
@@ -9,7 +9,7 @@ Layers
   :toctree: _autosummary
   :template: nn-module-template.rst

-   Embedding
+   Sequential
   ReLU
   PReLU
   GELU
@@ -17,17 +17,19 @@ Layers
   Step
   SELU
   Mish
+   Embedding
   Linear
+   QuantizedLinear
   Conv1d
   Conv2d
   BatchNorm
   LayerNorm
   RMSNorm
   GroupNorm
-   RoPE
-   MultiHeadAttention
-   Sequential
-   QuantizedLinear
   Dropout
   Dropout2d
-
+   Transformer
+   MultiHeadAttention
+   ALiBi
+   RoPE
+   SinusoidalPositionalEncoding