dj_angles.tokenizer

Module Contents

dj_angles.tokenizer.yield_tokens(s: str, breaking_character: str, *, handle_quotes: bool = True, handle_parenthesis: bool = False) → collections.abc.Generator[str, None, None]

Yields tokens from s by reading the string from left to right until a breaking_character, then continuing to read the s until the next breaking_character, ad infinitum.

Parameters:

  • s (param) – The string to parse.

  • breaking_character (param) – The character that signifies the end of the token.

  • handle_quotes (param) – Whether to ignore the breaking_character when inside single or double quotes.

  • handle_parenthesis (param) – Whether to ignore the breaking_character when inside parenthesis.

Returns:

A generator of tokens.