PHP Classes

Ses Tokenizer: Parse and split a string into tokens

Recommend this page to a friend!
  Info   View files View files (4)   DownloadInstall with Composer Download .zip   Reputation   Support forum   Blog    
Ratings Unique User Downloads Download Rankings
Not enough user ratingsTotal: 315 This week: 1All time: 7,257 This week: 560Up
Version License PHP version Categories
ses-tokenizer 1.0GNU General Publi...5.3PHP 5, Text processing, Parsers
Description 

Author

This class can parse and split a string into tokens.

It can take a string and split it to retrieve smaller tokens one by one.

The format of the tokens is defined by regular expressions passed to the class as parameters.

Picture of Gonzalo Chumillas
Name: Gonzalo Chumillas <contact>
Classes: 8 packages by
Country: Spain Spain
Age: 49
All time rank: 135129 in Spain Spain
Week rank: 416 Up14 in Spain Spain Up
Innovation award
Innovation award
Nominee: 2x

Details

tokenizer ========= The Tokenizer class allows us to split an string into tokens. Unlike other classes, it is based on regular expressions. The 'match' function is the most important function of the class. It allows to split an string into tokens and accepts a regular expression as parameter. For example: ```php // splits an string into 'words' $t = new Tokenizer("Lorem ipsum dolor sit amet"); while (list($token) = $t->match("\w+")) { echo "$token-"; } ``` Note that you DO NOT NEED to write an explicit regular expression. In the above example, instead of typing "/^\s*\w+/" we can write "\w+". In this case, the function ignores the left spaces and start searching from the current offset position. In any case, you can use an explicit regular expresion: ```php // uses an explicit regular expression $t = new Tokenizer("I'm 35 years old"); if (list($years) = $t->match("/\d+/")) { echo "You are $years old"; } ```

  Files folder image Files  
File Role Description
Files folder imageclasses (1 directory)
Accessible without login Plain text file LICENSE Lic. License text
Accessible without login Plain text file README.md Doc. Documentation
Accessible without login Plain text file test.php Example Examples

  Files folder image Files  /  classes  
File Role Description
Files folder imageparser (1 file)

  Files folder image Files  /  classes  /  parser  
File Role Description
  Accessible without login Plain text file tokenizer.php Class Tokenizer class

 Version Control Unique User Downloads Download Rankings  
 100%
Total:315
This week:1
All time:7,257
This week:560Up