dj_angles.tokenizer

Module Contents

dj_angles.tokenizer.yield_tokens(s: str, breaking_character: str) → collections.abc.Generator[str, None, None]

Yields tokens from s by reading the string from left to right until a breaking_character, then continuing to read the s until the next breaking_character, ad infinitum.

Parameters:

• s (param) – The string to parse.

• breaking_character (param) – The character that signifies the end of the token.

Returns:

A generator of tokens.