rust_macro.util

class rust_macro.util.Token

Canonical: tokenize.TokenInfo

A class that represents a token from the lexer. Iterables that yield these are taken in and returned by macros. This class is the same as tokenize.TokenInfo in the standard library.

type

Type: int

The type of token. See the Python token module for all of the different options.

string

Type: str

The text that the token contains

start

Type: int

end

Type: int

line

Type: str

The line the token is located in

property exact_type

Type: int

The exact type of token. See the Python token module for all of the different options.

rust_macro.util.splitargs(tokens: Iterable[Token], *, delimiter: str = ',') → List[List[Token]]

Splits a group of tokens into parts by a delimiter string

Example:

from rust_macro.util import tokenize_string, splitargs

tokens = tokenize_string("'Hello, World', Hello There")

args = splitargs(tokens, delimiter=',')

assert len(args) == 2
assert args[0][0].string == "'Hello, World'"
assert [i.string for i in args[1]] == ['Hello', 'There']

rust_macro.util.fix(text: str) → str: Fixes a some wonky text created by tokenize.untokenize

rust_macro.util.untokenize(tokens: Iterable[Token]) → str: Converts an interable of Tokens back into a string.

rust_macro.util.tokenize_string(s: str) → List[Token]:: Converts a string into its tokens

More may come soon!