sentencepiece bpe compatible tokenizer (#252)

* potential out of bounds read * fix quantize * style * Update convert-pth-to-ggml.py * mild cleanup * don't need the space-prefixing here rn since main.cpp already does it * new file magic + version header field * readme notice * missing newlines Co-authored-by: slaren <[email protected]>
2023-03-20 03:17:23 -07:00
parent 5cb63e2493
commit 074bea2eb1
7 changed files with 180 additions and 44 deletions
@@ -31,7 +31,7 @@ endif
 #

 CFLAGS   = -I.              -O3 -DNDEBUG -std=c11   -fPIC
-CXXFLAGS = -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC
+CXXFLAGS = -I. -I./examples -O3 -DNDEBUG -std=c++17 -fPIC
 LDFLAGS  =

 # OS specific