Module tokenizer

SQL Tokenizer

The tokenizer (a.k.a. lexer) converts a string into a sequence of tokens.

The tokens then form the input for the parser, which outputs an Abstract Syntax Tree (AST).

Structs

• Location
  Location in input string
• TokenWithLocation
  A Token with Location attached to it
• Tokenizer
  SQL Tokenizer
• TokenizerError
  Tokenizer error
• Word
  A keyword (like SELECT) or an optionally quoted SQL identifier

Enums

• Token
  SQL Token enumeration
• Whitespace

Functions

• is_delimited_identifier_start 🔒
  Determine if a character starts a quoted identifier. The default implementation, accepting “double quoted” ids is both ANSI-compliant and appropriate for most dialects (with the notable exception of MySQL, MS SQL, and sqlite). You can accept one of characters listed in Word::matching_end_quote here
• is_identifier_part 🔒
  Determine if a character is a valid unquoted identifier character
• is_identifier_start 🔒
  Determine if a character is a valid start character for an unquoted identifier