Token

A Token is a sequence of characters recognized by the lexical analyzer.

Members

Functions

errorLocationOfEnd Location errorLocationOfEnd(cstring filePath): Returns the location of the character past the end of this token.
getLocation Location getLocation(cstring filePath): Returns the Location of this token.
isAsmStatementStart bool isAsmStatementStart(): Returns true if this token starts an AsmStatement.
isDeclDefStart bool isDeclDefStart(): Returns true if this token starts a DeclarationDefinition.
isIntegralType bool isIntegralType(): Returns true if this is an integral type token.
isKeyword bool isKeyword(): Returns true if this is a keyword token.
isMultiline bool isMultiline(): Returns true if this is a token that can have newlines in it.
isSpecialToken bool isSpecialToken(): Returns true if this is a special token.
isStatementStart bool isStatementStart(): Returns true if this token starts a Statement.
isTokenStringLiteral bool isTokenStringLiteral(): Returns true if this is a token string literal.
isWhitespace bool isWhitespace(): Returns true if this is a whitespace token.
lineCount uint lineCount(): Counts the newlines in this token.
next Token* next(): Returns the next token.
nextNWS Token* nextNWS(): Finds the next non-whitespace token. Does not go past TOK.EOF.
opCmp int opCmp(Token* rhs): Compares the position of two tokens. Assumes they are from the same source text.
opEquals int opEquals(TOK kind2): Compares a token's kind to kind2.
prev Token* prev(): Returns the previous token.
prevNWS Token* prevNWS(): Finds the previous non-whitespace token. Does not go past TOK.HEAD.
text cstring text(): Returns the text of the token.
text void text(cstring s): Sets the text of the token.
textSpan cstring textSpan(Token* right): Return the source text enclosed by the left and right token.
toString cstring toString(): Returns the text of this token.
wsChars cstring wsChars(): Returns the preceding whitespace of the token.

Static functions

textSpan cstring textSpan(Token* left, Token* right): Return the source text enclosed by the left and right token.

Structs

HashLineInfo struct HashLineInfo: Represents the info of a #line token. Used for error messages.
HashLineValue struct HashLineValue: Represents the value of a "#line Number Filespec?" token.
NewlineValue struct NewlineValue: Represents the data of a newline token.
StringValue struct StringValue: Represents the string value of a single string literal, where possible escape sequences have been converted to their values.

Unions

IntegerValue union IntegerValue: Represents the long/ulong value of a number literal.
__anonymous union __anonymous: Data associated with this token.

Variables

end cchar* end;: Points one character past the end of the token.
kind TOK kind;: The token kind.
start cchar* start;: Points to the first character of the token.
ws cchar* ws;: Points to the preceding whitespace characters if present.

Examples

‘    StringValue’
/ ^$(Token ws, ws) ^$(Token start, start)     ^$(Token end, end)$(Token kind, kind)  = TOK.Identifier
$(Token flags, flags) = Flags.None
$(Token union.ident, ident) = $(Identifier)("StringValue", kind)

Token

Members

Functions

Static functions

Structs

Unions

Variables

Examples

Meta

Source