Enum Token 

pub enum Token {
    Literal(Literal),
    Identifier(Ident),
    Punct {
        op: PunctOp,
        span: Span,
    },
    OpenDelim {
        delimiter: Delimiter,
        span: Span,
    },
    CloseDelim {
        delimiter: Delimiter,
        span: Span,
    },
    DocComment {
        text: String,
        inner: bool,
        span: Span,
    },
    Error {
        message: String,
        span: Span,
    },
}

A flattened token that represents elements from a TokenTree stream.

Groups are flattened into OpenDelim and CloseDelim tokens, making parsing simpler by removing nesting from the token stream.

Examples

use cel_parser::lex_lexer::{LexLexer, Token};

let input: proc_macro2::TokenStream = "name".parse()?;
let token = LexLexer::new(input.into_iter()).next().unwrap();
assert!(matches!(token, Token::Identifier(id) if id == "name"));

Variants

Literal(Literal)

A literal value (integer, string, boolean, or float) with eager discrimination.

Identifier(Ident)

An identifier.

Punct

A punctuation operator storing all one to three characters inline.

Fields

op: PunctOp

The operator (e.g., “+”, “&&”, “<=”).

span: Span

Span for error reporting.

OpenDelim

Opening delimiter (flattened from Group).

Fields

delimiter: Delimiter

The type of delimiter (Parenthesis, Brace, Bracket).

span: Span

Span for error reporting.

CloseDelim

Closing delimiter (flattened from Group).

Fields

delimiter: Delimiter

The type of delimiter (Parenthesis, Brace, Bracket).

span: Span

Span for error reporting.

DocComment

A recognized //////! doc comment, unwrapped from the #[doc = "..."]/ #![doc = "..."] attribute-shaped token sequence proc_macro2 produces for it. General #[...] attribute syntax is not supported — see Token::Error.

Fields

text: String

The comment’s text: the doc-attribute string literal’s value verbatim (including its leading space), e.g. " the total" for /// the total.

inner: bool

true for //!/#![doc] (inner), false for ////#[doc] (outer).

span: Span

The # token’s span.

Error

An unsupported attribute shape or a literal Syn cannot parse.

Fields

message: String

A description of what was expected instead.

span: Span

Where the mismatch was detected.

Trait Implementations

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl HasSpan for Token

fn span(&self) -> Span

Returns the token’s preserved source or diagnostic span.