dj_angles.tokenizer
===================

.. py:module:: dj_angles.tokenizer




Module Contents
---------------

.. py:function:: yield_tokens(s: str, breaking_character: str) -> collections.abc.Generator[str, None, None]

   Yields tokens from `s` by reading the string from left to right until a `breaking_character`,
   then continuing to read the `s` until the next `breaking_character`, ad infinitum.

   :param param s: The string to parse.
   :param param breaking_character: The character that signifies the end of the token.

   :returns: A generator of tokens.