Class Tokenizer

constructor

• new Tokenizer(input: string, handler: TokenizerHandler): Tokenizer

  Parameters

  • input: string
  • handler: TokenizerHandler

  Returns Tokenizer

buffer

curr

Readonlyhandler

indexMask

input

position

tail

consume

• consume(): void

  Returns void

nthToken

• nthToken(n: number): Token

  Parameters

  • n: number

  Returns Token

token

• token(): Token

  Returns Token