Parses a string containing BASIC+ into an array of discrete BASIC+ tokens.
Syntax
Tokens = SRP_String("TokenizeCode", String, WhitespaceOptions, IncludeComments)
Parameters
| Parameter | Description |
|---|---|
| String | A string of BASIC+ code. (REQUIRED) |
| WhitespaceOptions | Whitespace tokenizing options. (OPTIONAL, Default="None") |
| IncludeComments | TRUE to include comments, FALSE to omit them. (OPTIONAL, Default=0) |
Returns
An @FM delimited array of strings, each one a BASIC+ token.
Remarks
This service takes a string of BASIC+ code and returns an @FM delimited array of tokens. A token is a discrete element of code: a string, an operator, whitespace, comment, identifier, etc. This was written to support the SRP Precompiler, which parses BASIC+ code in order to provide language enhancements. This routine does all the work of breaking up the code into tokens for easier analysis. To use this service, pass your BASIC+ code to the String parameter. The remaining parameters provide some options for controlling the output.
WhitespaceOptions: This parameter supports one of three values: "None", "AllWhitespace", or "LineBreaksOnly". The default option is "None", in which case, whitespace tokens are never included in the final array. "AllWhitespace" has the exact opposite effect: all whitespace tokens are included in the output. "LineBreaksOnly" will omit whitespace composed of spaces and tabs but include carriage return and/or line feeds. This last option could be useful for detecting line breaks in code while still ignoring all other whitespace.
IncludeComments: This true/false parameter determines whether or not comment tokens are included in the output. The default is FALSE.
Example
// Sample code
Code = "A = B + C; // This is a simple expression"
// Output will be: "A":@FM:"=":@FM:"B":@FM:"+":@FM:"C":@FM:";"
Tokens = SRP_String("TokenizeCode")
// Output will be: "A":@FM:" ":@FM:"=":@FM:" ":@FM:"B":@FM:" ":@FM:"+"::@FM:" "@FM:"C":@FM:";":@FM:" ":@FM:\0D0A\
Tokens = SRP_String("TokenizeCode", "AllWhitespace")
// Output will be: "A":@FM:"=":@FM:"B":@FM:"+":@FM:"C":@FM:";":@FM:\0D0A\
Tokens = SRP_String("TokenizeCode", "LineBreaksOnly")
// Output will be: "A":@FM:"=":@FM:"B":@FM:"+":@FM:"C":@FM:";":@FM:"// This is a simple expression"
Tokens = SRP_String("TokenizeCode", "None", 1)
See Also
