2 Commits

Author	SHA1	Message	Date
stronk7	cfc673c112	Now the tokeniser is using the new textlib->entities_to_utf8() ... own function (PHP4 and PHP compatible)	2007-06-13 23:21:36 +00:00
stronk7	1c09e6fe0c	First cut of the tokeniser library. Modified from drupal search.module ... code (see copyrights). The library converts any text/html into an array of tokens with their score (weight). Supports stop_words, cjk basic tokeniser (for indexers) and different modes of handling numbers.	2007-06-12 18:15:05 +00:00