# AOWP

- Overview
- Package
- Class
- Tree
- Deprecated
- Index

Frames No frames
Summary: Field | Method | Constr Detail: Field | Method | Constr

aowp.parser.parser.PHP_Parser_Tokenizer
/parser/Parser/Tokenizer.php at line 54

# Class PHP_Parser_Tokenizer

    PHP_Parser_Tokenizer

public class PHP_Parser_Tokenizer

The tokenizer wrapper for parser - implements the 'standard?' yylex interface

2 main methods:

- constructor, which takes the data to parse calls php's internal tokenizer, then tidies up the array a little (key=>value) rather than mixed type.
- advance, which returns true while tokens are available - sets `$value` - sets `$token`
- parseError, which returns a string to appear on parser error messages. (could also display some of the code that has an error)

uses a few flags like: - `$line` - current line number - `$pos` - current token id - `$N` - total no. of tokens

Version:
  $Id: Tokenizer.php,v 1.13 2007/03/05 05:05:50 cellog Exp $

Field Summary
int | $N Total Number of tokens.
private mixed | $_docParser
private string | $_globalSearch String of global variable to search forphpDocumentor-specific usage, extracted from documentation's @global tag
private mixed | $_options Tokenizing options
private mixed | $_trackingWhitespace
private mixed | $_whitespace
boolean | $debug Debugging on/off
string | $lastComment The string of the last Comment Token
int | $lastCommentLine ID of the last Comment Token
int | $lastCommentToken ID of the last Comment Token
string | $lastParsedComment The string of the last Comment Token
int | $line Current line.
int | $pos Current token position.
int | $token The current token (either a ord(';') or token numer - see php tokenizer.
mixed | $tokenValue
array | $tokens Tokens - array of all the tokens.
string | $value The value associated with a token - eg.
string | $valueWithWhitespace The value associated with a token plus preceding whitespace, if anyThis is only filled if whitespace attachment is turned on, for performance reasons

Constructor Summary
PHP_Parser_Tokenizer(mixed data, mixed docparser, string PHP) ConstructorLoad the tokenizer - with a string to tokenize.

Method Summary
private void | _handleDocumentation() Helper function for advance(), parses and publishes doc comments as necessary
private void | _setGlobalSearch(mixed var, string )
boolean | advance() The main advance call required by the parser return true if a token is available, false if no more are available.
array | getLastComment() get the last comment block (and reset it)
void | getValue()
string | getWhitespace() Return the whitespace (if any) that preceded the current token
void | globalSearch(mixed var, string global) Compare global variable to search value, to see if we've found a variable that must be documented
void | haltLexing()
void | handleMessage(mixed msg, MsgServer_Msg )
string | parseError() return something useful, when a parse error occurs.
void | releaseInstance()
void | stopTrackingWhitespace()
void | trackWhitespace()

## Field Detail

/parser/Parser/Tokenizer.php at line 78

### N

`public int $N`

Total Number of tokens.

/parser/Parser/Tokenizer.php at line 174

### _docParser

`private mixed $_docParser`

/parser/Parser/Tokenizer.php at line 165

### _globalSearch

`private string $_globalSearch = false`

String of global variable to search for

phpDocumentor-specific usage, extracted from documentation's @global tag

/parser/Parser/Tokenizer.php at line 171

### _options

`private mixed $_options`

Tokenizing options

/parser/Parser/Tokenizer.php at line 173

### _trackingWhitespace

`private mixed $_trackingWhitespace`

/parser/Parser/Tokenizer.php at line 172

### _whitespace

`private mixed $_whitespace`

/parser/Parser/Tokenizer.php at line 64

### debug

`public boolean $debug = false`

Debugging on/off

/parser/Parser/Tokenizer.php at line 146

### lastComment

`public string $lastComment`

The string of the last Comment Token

/parser/Parser/Tokenizer.php at line 137

### lastCommentLine

`public int $lastCommentLine`

ID of the last Comment Token

/parser/Parser/Tokenizer.php at line 128

### lastCommentToken

`public int $lastCommentToken`

ID of the last Comment Token

/parser/Parser/Tokenizer.php at line 155

### lastParsedComment

`public string $lastParsedComment`

The string of the last Comment Token

/parser/Parser/Tokenizer.php at line 85

### line

`public int $line`

Current line.

/parser/Parser/Tokenizer.php at line 92

### pos

`public int $pos = -1`

Current token position.

/parser/Parser/Tokenizer.php at line 100

### token

`public int $token`

The current token (either a ord(';') or token numer - see php tokenizer.

/parser/Parser/Tokenizer.php at line 56

### tokenValue

`public mixed $tokenValue`

/parser/Parser/Tokenizer.php at line 71

### tokens

`public array $tokens`

Tokens - array of all the tokens.

/parser/Parser/Tokenizer.php at line 109

### value

`public string $value`

The value associated with a token - eg. for T_STRING it's the string

/parser/Parser/Tokenizer.php at line 119

### valueWithWhitespace

`public string $valueWithWhitespace`

The value associated with a token plus preceding whitespace, if any

This is only filled if whitespace attachment is turned on, for performance reasons

## Constructor Detail

/parser/Parser/Tokenizer.php at line 188

### PHP_Parser_Tokenizer

`public PHP_Parser_Tokenizer(mixed data, mixed docparser, string PHP)`

Constructor

Load the tokenizer - with a string to tokenize. tidies up array, sets vars pos, line, N and tokens

Parameters:
  PHP - code to serialize

## Method Detail

/parser/Parser/Tokenizer.php at line 298

### _handleDocumentation

`private void _handleDocumentation()`

Helper function for advance(), parses and publishes doc comments as necessary

/parser/Parser/Tokenizer.php at line 237

### _setGlobalSearch

`private void _setGlobalSearch(mixed var, string )`

/parser/Parser/Tokenizer.php at line 327

### advance

`public boolean advance()`

The main advance call required by the parser

return true if a token is available, false if no more are available. skips stuff that is not a valid token stores lastcomment, lastcommenttoken

Returns:
  - true = have tokens

/parser/Parser/Tokenizer.php at line 279

### getLastComment

`public array getLastComment()`

get the last comment block (and reset it)

Returns:
  array('comment' => $commmentstring, 'parsed' => $parsed, 'line' => $linenumber)

/parser/Parser/Tokenizer.php at line 429

### getValue

`public void getValue()`

/parser/Parser/Tokenizer.php at line 218

### getWhitespace

`public string getWhitespace()`

Return the whitespace (if any) that preceded the current token

/parser/Parser/Tokenizer.php at line 257

### globalSearch

`public void globalSearch(mixed var, string global)`

Compare global variable to search value, to see if we've found a variable that must be documented

Parameters:
  global - variable found in source code

/parser/Parser/Tokenizer.php at line 208

### haltLexing

`public void haltLexing()`

/parser/Parser/Tokenizer.php at line 226

### handleMessage

`public void handleMessage(mixed msg, MsgServer_Msg )`

/parser/Parser/Tokenizer.php at line 442

### parseError

`public string parseError()`

return something useful, when a parse error occurs.

used to build error messages if the parser fails, and needs to know the line number..

/parser/Parser/Tokenizer.php at line 448

### releaseInstance

`public void releaseInstance()`

/parser/Parser/Tokenizer.php at line 247

### stopTrackingWhitespace

`public void stopTrackingWhitespace()`

/parser/Parser/Tokenizer.php at line 242

### trackWhitespace

`public void trackWhitespace()`

# AOWP

- Overview
- Package
- Class
- Tree
- Deprecated
- Index

Frames No frames
Summary: Field | Method | Constr Detail: Field | Method | Constr