Struct CompactToken 

pub struct CompactToken(/* private fields */);

A compact representation of a Token with Char==u8 and CS==InternedCSName as a single u32 (similar to the way plain TeX does it) – i.e. it is small and Copy, which yields a significant performance improvement in the most performance critical parts of the code.

Values up to 0x8000_0000 are interpreted as interned control sequences, and the rest as character tokens. The downside is that we need an interning table for control sequences, that needs passing around whenever we want to make a Token from a control sequence name or display a CompactToken to the user in a comprehensible way.

(Also, we can only have 2³¹ control sequences in total, but that limit is ridiculously large.)

Trait Implementations

impl Clone for CompactToken

fn clone(&self) -> CompactToken

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CompactToken

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for CompactToken

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Token for CompactToken

type CS = (NonZero<u32>, PhantomData<u8>)

The CSName type used for control sequence names (e.g. Rc<str> or something interned).

type Char = u8

The Character type for char/catcode-pair tokens.

fn to_enum(&self) -> StandardToken<u8, InternedCSName<u8>>

Converts to the canonical enum representation of a token, i.e. StandardToken.

fn from_cs(cs: Self::CS) -> Self

Create a new token from a control sequence name.

fn from_char_cat(c: u8, cat: CommandCode) -> Self

Create a new character token with given CommandCode (i.e. conceptually the CategoryCode).

fn space() -> Self

Create a new space token.

fn eof() -> Self

Create a new end-of-file token.

fn primitive(id: PrimitiveIdentifier) -> Self

Create a new token representing a primitive Command.

fn is_primitive(&self) -> Option<PrimitiveIdentifier>

fn argument_marker(i: u8) -> Self

Create a new argument marker token. i needs to be in the range 0..=8.

fn command_code(&self) -> CommandCode

The CommandCode (i.e. conceptually the CategoryCode) of this token.

fn char_value(&self) -> Option<Self::Char>

The Character value of this token, if it is a character token.

fn is_cs_or_active(&self) -> bool

Check if this token is a control sequence or an active character

fn is_cs(&self, name: &Self::CS) -> bool

Check if this token is a control sequence with the given name.

fn is_argument_marker(&self) -> Option<u8>

Check if this token is a argument token, and if so, return its number (in the range 0..=8).

fn display_fmt<W: Write>( &self, int: &<Self::CS as CSName<Self::Char>>::Handler, cc: &CategoryCodeScheme<Self::Char>, escapechar: Option<Self::Char>, f: &mut W, ) -> Result

Display this token to a writer, using the given CSHandler (in case it is a control sequence). In that case, we also need the current \escapechar to optionally insert it in front of the control sequence name, and the current CategoryCodeScheme to determine whether or not to insert a space afterwards - which we do unless the control sequence name is a single character with any CommandCode other than Letter.

fn display<'a>( &'a self, int: &'a <Self::CS as CSName<Self::Char>>::Handler, cc: &'a CategoryCodeScheme<Self::Char>, escapechar: Option<Self::Char>, ) -> DisplayToken<'a, Self>

Returns a helper struct implementing Display for this token.

impl Copy for CompactToken

impl Eq for CompactToken