Enum muse_lang::compiler::syntax::token::Token

pub enum Token {
    Whitespace,
    Comment,
    Identifier(Symbol),
    Symbol(Symbol),
    Sigil(Symbol),
    Label(Symbol),
    Int(i64),
    UInt(u64),
    Float(f64),
    Char(char),
    String(Symbol),
    Regex(RegexLiteral),
    FormatString(FormatString),
    Power,
    LessThanOrEqual,
    GreaterThanOrEqual,
    Equals,
    AddAssign,
    SubtractAssign,
    MultiplyAssign,
    DivideAssign,
    IntegerDivide,
    IntegerDivideAssign,
    RemainderAssign,
    ShiftLeft,
    ShiftLeftAssign,
    ShiftRight,
    ShiftRightAssign,
    NotEqual,
    Range,
    RangeInclusive,
    Ellipses,
    SlimArrow,
    FatArrow,
    NilCoalesce,
    Open(Paired),
    Close(Paired),
}

A Muse token.

A token is the smallest unit of syntax that forms Muse syntax. A series of tokens can be parsed into an expression using Muse’s grammar.

Variants

Whitespace

A sequence of ASCII whitespace.

Whitespace is insignificant to Muse and is ignored by the compiler. While being insignificant, it can still serve as a way to ensure a sequence of characters is broken into multiple tokens instead of interpreted as a single token (e.g., forx parses as a single identifier token while for x parses as two identifier tokens regardless of if you ignore whitespace tokens).

Comment

A source code comment.

Comments are ignored by the Muse compiler.

Identifier(Symbol)

A name or keyword (foo).

Symbol(Symbol)

A symbol (:foo).

Sigil(Symbol)

A sigil ($foo).

Label(Symbol)

A source label name (@foo).

Int(i64)

A signed integer.

UInt(u64)

An unsigned integer.

Float(f64)

A floating point number.

Char(char)

A single character (not a character literal).

String(Symbol)

A string literal.

Regex(RegexLiteral)

A regular expression literal.

FormatString(FormatString)

A format string.

Power

The power token **

LessThanOrEqual

The less than or equal token <=

GreaterThanOrEqual

The greater than or equal token >=

Equals

The equals token ==

AddAssign

The add assign token +=

SubtractAssign

The add assign token -=

MultiplyAssign

The add assign token *=

DivideAssign

The add assign token /=

IntegerDivide

The add assign token //

IntegerDivideAssign

The add assign token //=

RemainderAssign

The add assign token %=

ShiftLeft

The shift left token <<

ShiftLeftAssign

The shift left asign token <<=

ShiftRight

The shift right token >>

ShiftRightAssign

The shift right token assign >>=

NotEqual

The not equal token !=

Range

The range token ..

RangeInclusive

The range inclusive token ..=

Ellipses

The elipses token ...

SlimArrow

The slim arrow token ->

FatArrow

The fat arrow token =>

NilCoalesce

Nil coalesce token ??

Open(Paired)

An opening of a paired token.

Close(Paired)

A closing of a paired token.

Implementations

impl Token

pub fn is_likely_end(&self) -> bool

Returns true if this token is likely the end of an expression.

This function is used in some locations to make the grammar not require as many “hard” keywords. Many grammar keywords only match if the next token is not a likely end, which allows for “escaping” ambiguous usage by surrounding the keyword in parentheses.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Token

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Hash for Token

fn hash<H: Hasher>(&self, state: &mut H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Token

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Token

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for Token