The main method which recognizes the characters that make up a token.
Complicated tokens are scanned in separate methods.
See Implementation
The main method which recognizes the characters that make up a token.
Complicated tokens are scanned in separate methods.