PHP Classes


Recommend this page to a friend!
  Classes of Gonzalo Chumillas   Ses Tokenizer   Download  
Role: Documentation
Content type: text/plain
Description: Documentation
Class: Ses Tokenizer
Parse and split a string into tokens
Author: By
Last change:
Date: 9 years ago
Size: 930 bytes


Class file image Download
tokenizer ========= The Tokenizer class allows us to split an string into tokens. Unlike other classes, it is based on regular expressions. The 'match' function is the most important function of the class. It allows to split an string into tokens and accepts a regular expression as parameter. For example: ```php // splits an string into 'words' $t = new Tokenizer("Lorem ipsum dolor sit amet"); while (list($token) = $t->match("\w+")) { echo "$token-"; } ``` Note that you DO NOT NEED to write an explicit regular expression. In the above example, instead of typing "/^\s*\w+/" we can write "\w+". In this case, the function ignores the left spaces and start searching from the current offset position. In any case, you can use an explicit regular expresion: ```php // uses an explicit regular expression $t = new Tokenizer("I'm 35 years old"); if (list($years) = $t->match("/\d+/")) { echo "You are $years old"; } ```