Trait Token 

pub trait Token:
    Clone
    + Eq
    + 'static
    + Debug
    + Sized {
    type CS: CSName<Self::Char>;
    type Char: Character;

    // Required methods
    fn to_enum(&self) -> StandardToken<Self::Char, Self::CS>;
    fn from_cs(cs: Self::CS) -> Self;
    fn space() -> Self;
    fn primitive(id: PrimitiveIdentifier) -> Self;
    fn argument_marker(i: u8) -> Self;
    fn eof() -> Self;
    fn from_char_cat(c: Self::Char, cat: CommandCode) -> Self;

    // Provided methods
    fn char_value(&self) -> Option<Self::Char> { ... }
    fn command_code(&self) -> CommandCode { ... }
    fn is_cs_or_active(&self) -> bool { ... }
    fn is_cs(&self, name: &Self::CS) -> bool { ... }
    fn is_primitive(&self) -> Option<PrimitiveIdentifier> { ... }
    fn is_argument_marker(&self) -> Option<u8> { ... }
    fn display_fmt<W: Write>(
        &self,
        int: &<Self::CS as CSName<Self::Char>>::Handler,
        cc: &CategoryCodeScheme<Self::Char>,
        escapechar: Option<Self::Char>,
        f: &mut W,
    ) -> Result { ... }
    fn display<'a>(
        &'a self,
        int: &'a <Self::CS as CSName<Self::Char>>::Handler,
        cc: &'a CategoryCodeScheme<Self::Char>,
        escapechar: Option<Self::Char>,
    ) -> DisplayToken<'a, Self> { ... }
}

Trait for Tokens, to be implemented for an engine (see above). Note that two Space tokens are always considered equal.

Required Associated Types

type CS: CSName<Self::Char>

The CSName type used for control sequence names (e.g. Rc<str> or something interned).

type Char: Character

The Character type for char/catcode-pair tokens.

Required Methods

fn to_enum(&self) -> StandardToken<Self::Char, Self::CS>

Converts to the canonical enum representation of a token, i.e. StandardToken.

fn from_cs(cs: Self::CS) -> Self

Create a new token from a control sequence name.

fn space() -> Self

Create a new space token.

fn primitive(id: PrimitiveIdentifier) -> Self

Create a new token representing a primitive Command.

fn argument_marker(i: u8) -> Self

Create a new argument marker token. i needs to be in the range 0..=8.

fn eof() -> Self

Create a new end-of-file token.

fn from_char_cat(c: Self::Char, cat: CommandCode) -> Self

Create a new character token with given CommandCode (i.e. conceptually the CategoryCode).

Provided Methods

fn char_value(&self) -> Option<Self::Char>

The Character value of this token, if it is a character token.

fn command_code(&self) -> CommandCode

The CommandCode (i.e. conceptually the CategoryCode) of this token.

fn is_cs_or_active(&self) -> bool

Check if this token is a control sequence or an active character

fn is_cs(&self, name: &Self::CS) -> bool

Check if this token is a control sequence with the given name.

fn is_primitive(&self) -> Option<PrimitiveIdentifier>

fn is_argument_marker(&self) -> Option<u8>

Check if this token is a argument token, and if so, return its number (in the range 0..=8).

fn display_fmt<W: Write>( &self, int: &<Self::CS as CSName<Self::Char>>::Handler, cc: &CategoryCodeScheme<Self::Char>, escapechar: Option<Self::Char>, f: &mut W, ) -> Result

Display this token to a writer, using the given CSHandler (in case it is a control sequence). In that case, we also need the current \escapechar to optionally insert it in front of the control sequence name, and the current CategoryCodeScheme to determine whether or not to insert a space afterwards - which we do unless the control sequence name is a single character with any CommandCode other than Letter.

fn display<'a>( &'a self, int: &'a <Self::CS as CSName<Self::Char>>::Handler, cc: &'a CategoryCodeScheme<Self::Char>, escapechar: Option<Self::Char>, ) -> DisplayToken<'a, Self>

Returns a helper struct implementing Display for this token.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl Token for CompactToken

type CS = (NonZero<u32>, PhantomData<u8>)

type Char = u8

impl<Char: Character, CS: CSName<Char>> Token for StandardToken<Char, CS>

type CS = CS

type Char = Char