Parses a string containing BASIC+ into an array of discrete BASIC+ tokens.
Syntax
Tokens = SRP_String("TokenizeCode", WhitespaceOptions, IncludeComments)
Parameters
Parameter | Description |
---|---|
WhitespaceOptions | Whitespace tokenizing options. (OPTIONAL) |
IncludeComments | TRUE to include comments, FALSE to omit them. (OPTIONAL) |
Returns
An @FM delimited array of strings, each one a BASIC+ token.
Remarks
This service takes a string of BASIC+ code and returns an @FM delimited array of tokens. A token is a discrete element of code: a string, an operator, whitespace, comment, identifier, etc. This was written to support the SRP Precompiler, which parses BASIC+ code in order to provide language enhancements. This routine does all the work of breaking up the code into tokens for easier analysis. There are some options to control the output.
WhitespaceOptions: This parameter supports one of three values: "None", "AllWhitespace", or "LineBreaksOnly". The default option is "None", in which case, whitespace tokens are never included in the final array. "AllWhitespace" has the exact opposite effect: all whitespace tokens are included in the output. "LineBreaksOnly" will omit whitespace composed of spaces and tabs but include carriage return and/or line feeds. This last option could be useful for detecting line breaks in code while still ignoring all other whitespace.
IncludeComments: This true/false parameter determines whether or not comment tokens are included in the output. The default is FALSE.
Example
// Sample code Code = "A = B + C; // This is a simple expression" // Output will be: "A":@FM:"=":@FM:"B":@FM:"+":@FM:"C":@FM:";" Tokens = SRP_String("TokenizeCode") // Output will be: "A":@FM:" ":@FM:"=":@FM:" ":@FM:"B":@FM:" ":@FM:"+"::@FM:" "@FM:"C":@FM:";":@FM:" ":@FM:\0D0A\ Tokens = SRP_String("TokenizeCode", "AllWhitespace") // Output will be: "A":@FM:"=":@FM:"B":@FM:"+":@FM:"C":@FM:";":@FM:\0D0A\ Tokens = SRP_String("TokenizeCode", "LineBreaksOnly") // Output will be: "A":@FM:"=":@FM:"B":@FM:"+":@FM:"C":@FM:";":@FM:"// This is a simple expression" Tokens = SRP_String("TokenizeCode", "None", 1)
See Also
DetokenizeCode