Module lex_lexer 

Flattens Rust token trees into CEL tokens while preserving source spans.

Construct a LexLexer from a proc_macro2::TokenStream iterator and consume it as an iterator of Token values. Groups emit matching opening and closing delimiters, booleans become literals, and adjacent punctuation combines into the supported operators. Documentation attributes become comment tokens.

use cel_parser::lex_lexer::{LexLexer, Token};

let input: proc_macro2::TokenStream = "value && true".parse()?;
let mut lexer = LexLexer::new(input.into_iter());
assert!(matches!(lexer.next(), Some(Token::Identifier(id)) if id == "value"));
assert!(matches!(lexer.next(), Some(Token::Punct { op, .. }) if op == "&&"));
assert!(matches!(lexer.next(), Some(Token::Literal(_))));
assert!(lexer.next().is_none());

Error Handling

A # commits to a documentation attribute: #[doc = "..."] or #![doc = "..."]. Other attribute shapes and literals Syn cannot parse produce Token::Error. Errors consume the inspected token trees; iteration may continue with the remaining input, but does not roll back a malformed attribute.

Structs

LexLexer

Lazily flattens token trees and combines adjacent supported punctuation.

Enums

PunctOp

Stores one to three operator characters inline.

Token

A flattened token that represents elements from a TokenTree stream.

Traits

HasSpan

Provides a token’s source span for diagnostics.

Type Aliases

Literal

Represents a parsed literal, including boolean keywords.

TokenStreamIter

Owns a token stream’s consuming iterator.