Module tokenizer

SQL Tokenizer

The tokenizer (a.k.a. lexer) converts a string into a sequence of tokens.

The tokens then form the input for the parser, which outputs an Abstract Syntax Tree (AST).

Structs

Location

Location in input string

TokenWithLocation

A Token with Location attached to it

Tokenizer

SQL Tokenizer

TokenizerError

Tokenizer error

Word

A keyword (like SELECT) or an optionally quoted SQL identifier

Enums

Token

SQL Token enumeration

Whitespace

Functions

is_delimited_identifier_start 🔒

Determine if a character starts a quoted identifier. The default implementation, accepting “double quoted” ids is both ANSI-compliant and appropriate for most dialects (with the notable exception of MySQL, MS SQL, and sqlite). You can accept one of characters listed in Word::matching_end_quote here

is_identifier_part 🔒

Determine if a character is a valid unquoted identifier character

is_identifier_start 🔒

Determine if a character is a valid start character for an unquoted identifier