lexer module

src.lexer.tokenize(code)

Tokenizes the given code into a list of tokens.

Args: code (str): The source code to be tokenized.

Returns: list: A list of tokens.