CMPUT174 Lecture Notes - Delimiter
ceruleanzebra152 and 30 others unlocked
25
CMPUT174 Full Course Notes
Verified Note
25 documents
Document Summary
Token - the smallest meaningful unit of information of data for a compiler. Natural language - a character, punctuation and a word is part of a token which is a linguistic unit. Programming language - a character is part of a token which is an information unit. Keyword token(can act as a delimiter, literal or operator depending on the keyword) No token can belong to more than one token kind. Lexical analyzer is a part of python interpreter which reads python code characters and creates token. Each kind of token is either described by a lexical rule or defined by a lexical table.