dil ~master (2015-07-05T21:12:10Z)
Home
Dub
Repo
Lexer.newToken
dil
lexer
Lexer
Lexer
Returns the next free token from the array. NB: The bytes are not zeroed out.
class
Lexer
Token
*
newToken
(
)
Meta
Source
See Implementation
dil
lexer
Lexer
Lexer
constructors
this
functions
decodeUTF8
dlxCallback
endX
error
errorFilePath
errorLineNumber
finalizeFloat
finalizeSpecialToken
firstToken
fromDLXFile
getBuffer
head
isInText
isNewlineEnd
lastToken
lineNum
lookupFloat
lookupNewline
lookupString
newToken
new_
peek
scan
scanAll
scanBlockComment
scanCharacter
scanDelimitedString
scanEscapeSequence
scanEscapeString
scanFloat
scanHexFloat
scanHexString
scanNestedComment
scanNormalString
scanNumber
scanRawString
scanShebang
scanSpecialTokenSequence
scanTokenString
scan_
setBuffer
text
tokenList
static functions
cases
copySansUnderscores
encodeUTF8
findInvalidUTF8Sequence
formatBytes
scanPostfix
scanUnicodeAlpha
static variables
chars_line
chars_q
chars_q2
chars_r
chars_shebang
chars_x
structs
LineLoc
variables
allocator
buffer
diag
end
errors
hlinfo
inTokenString
lineLoc
p
srcText
tables
tokens
Returns the next free token from the array. NB: The bytes are not zeroed out.