Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to understand the lexical analysis of PostgreSQL". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to understand the lexical analysis of PostgreSQL".

I. Lexical analysis

Basic concept

First of all, let's sort out some basic concepts.

Lexical analysis scans input SQL statements from left to right and divides its character stream into words (called token). These token are inseparable strings of characters in the input stream, similar to words in English or Chinese.

The category of token in SQL statements is limited, generally speaking, there are constants (numeric / character / string, etc.), operators (arithmetic operators / logical operators, etc.), delimiters (comma / semicolon / parentheses, etc.), reserved keywords, identifiers (function name / procedure name, etc.). For example: 1 and 200.13 are numeric constants token,' Zhang San 'and' Guangzhou 'are string constants token,+/- and so on are operators token and so on.

Introduction to Flex

In PostgreSQL, the open source Flex is used for lexical analysis of SQL.

The full name of Flex is Fast LEXical analyser generator-scanner generator for lexing in C and Clearing.

The input file format for Flex is:

{Declarations (declaration)%} Definitions (definition)% Rules (rules)% User subroutines (user subprocedures)

Such as:

% {# define T_ZEOR 0 int I = 0 NUM%} NUM ([0-9] +)% {NUM} printf ("?"); / / encountered a number, printed? # return title ZEOR; / / encountered the character # and returned 0. ECHO; / / encountered other characters, print this character% int main (int argc, char* argv []) {yylex (); return return ZEOR;} int yywrap () {return 1;}

The effect of this routine is as follows:

[root@localhost mytest] #. / mytest 1t33..q?t?..q#

SQL lexical analyzer

A simple SQL lexical analyzer can be implemented using Flex, which is divided into the following steps:

1. List all types of token in SQL

two。 Assign a unique number to each token and write a regular expression for that token

3. Write out the rule of each token

Sql.l

% {int current_linenum = 1: void init (); void elog (char* msg, int line); typedef enum {T_EQUAL = 128,T_SELECT, T_CONST, T_STRING, T_ID} TokeType;static char* string_token [] = {"T_EQUAL", "T_SELECT", "T_CONST", "T_STRING", "T_ID"} %} INTEGER ([0-9] +) UNTERM_STRING ("'" [^'\ n] *) STRING ("'" [^'\ n] * "'") IDENTIFIER ([_ a-zA-Z] [_ a-zA-Z0-9] *) OPERATOR ([+ *-/% = ! () {}] SINGLE_COMMENT ("/ /" [^\ n] *)% [\ n] {current_linenum++;} [\ t\ r\ a] + {/ * ignore all spaces * /} {SINGLE_COMMENT} {/ * skip for single line comment * /} {OPERATOR} {return yytext [0] } "=" {return titled EQUALs;} "select" {return Tunable select;} {INTEGER} {return Tunable connect;} {STRING} {return Tunable STRING;} {IDENTIFIER} {return Tunable ID;} {return 0 {UNTERM_STRING} {elog ("Unterminated string constant", current_linenum);}. {elog ("Unrecognized character", current_linenum);}% int main (int argc, char* argv []) {int token; init (); while (token = yylex ()) {if (token < 128c) printf ("%-20c", token); else printf ("%-20s", string_ token [token-128s]); puts (yytext);} return 0 } void init () {printf ("%-20s%s\ n", "TOKEN-TYPE", "TOKEN-VALUE"); printf ("- -\ n") } void elog (char* msg, int line) {printf ("\ nError at line%-3D:% s\ n\ n", line, msg);} int yywrap (void) {return 1;}

Makefile

Run: sql. / sql < test.sqlsql: lex.yy.c gcc-o $@ $

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.