Lexertakes a given input and returns a series of analyzed tokens.
Provides a generalized token lexer capability.
Contains the token's type, corresponding Java object, raw text and the input line number where the token was found.
The lexer will either analyze the tokens identifying the type or collect raw input until a terminating delimiter is found.
Lexertakes a given input and returns a series of analyzed tokens. These tokens contain:
The most important difference between
java.util.StringTokenizer is that Lexer analyzes the
tokens into types. Further, tokens are not created by
breaking a line based on inter-field separator characters or
by regular expression. Instead, Lexer searches for tokens
based on types. This lexical package is designed to work with
a higher level parser rather than a line-by-line text