How to use yacc to test for ambiguity (Compilers)
There is no algorithm that decides definitively whether a given context free grammar is ambiguous, but that does not prevent us from using yacc pragmatically. If we submit a grammar (unannotated with precedence declarations) to yacc, and yacc accepts it, then we know the grammar is in the class LALR(1), and all such grammars are unambiguous. If yacc says No and reports parsing conflicts, then either the grammar is ambiguous, or it is unambiguous, but "too difficult" for yacc to handle. To find out which, we can look at yacc's diagnostic output and use it to find a situation where a conflict is encountered: that can help us to find a string that can be derived in more than one way.
Consider, for example, the following interesting grammar provided by Matteo Cati. It purports to be an unambiguous grammar for the classic Algol--60 dangling else problem.
stmt : inner | IF expr THEN stmt inner : BASIC | IF expr THEN inner ELSE stmt
In order to submit this to yacc, we must surround it with some apparatus declaring various tokens, and supply semantic actions, all of which can be written as the trivial unit expression {()}. A complete input is shown towards the end of this note. Invoking ocamlyacc with
$ ocamlyacc -v cati.mly
produces the error message
1 shift/reduce conflict
and also (because of the -v flag) a report file cati.output that gives details of parsing automaton ocamlyacc compiled, and where it discovered the conflict. In what follows, I've used (because why wouldn't I?) the output of my own yacc variant, but the output of ocamlyacc is equivalent, even if the states are numbered differently. The whole of the report file is shown at the end of this note.
Examining the report, we see the conflict occurs in state 11:
State 11: shift/reduce conflict (shift 12, reduce 1) on ELSE State 11 inner --> IF expr THEN inner . ELSE stmt (4) stmt --> inner . (1) EOF reduce 1 ELSE shift 12
If the machine reaches state 11 and the lookahead is ELSE, then the machine cannot decide between shifting the ELSE (in the hope of reducing with the production inner --> IF expr THEN inner ELSE stmt), or reducing immediately with the production stmt --> inner (presumably so that it can then reduce with stmt -> IF expr THEN stmt).
We can look for an ambiguous string by asking what sequence of moves in the machine can lead to state 11. Sadly, the simplifications from the full LR(1) process incorporated into yacc mean that the shortest such sequence does not yield an ambiguous string, but a longer sequence does.[1] Consider the following string of tokens and variables:
IF expr THEN IF expr THEN inner ELSE IF expr THEN inner
Starting in state 2 (states 0 and 1 are present for housekeeping purposes, as is the magic token *1*), we can trace out a trajectory in the machine as follows. In the report from yacc, state 2 contains the instruction
State 2: IF shift 4
And state 4 contains
State 4: expr goto 8
Continuing in this way, we can trace out the following path that ends at state 11, incidentally visiting state 11 along the way.
2 IF 4 expr 8 THEN 9 IF 4 expr 8 THEN 9 inner 11 ELSE 12 IF 4 expr 8 THEN 9 inner 11
The shorter sequence that stops after reaching state 11 for the first time also leaves the yacc-based parser unable to decide what to do next, but it does not allow both decisions to lead to a succesful parsing outcome.
Now, with the longer string, and with ELSE BASIC as the remaining input, the conflict gives two possibilities: one option is to shift the ELSE, in which case we can complete a parse by shifting the token BASIC on top of it, then reducing in several stages as follows.
IF expr THEN IF expr THEN inner ELSE IF expr THEN inner | ELSE BASIC IF expr THEN IF expr THEN inner ELSE IF expr THEN inner ELSE | BASIC IF expr THEN IF expr THEN inner ELSE IF expr THEN inner ELSE BASIC | IF expr THEN IF expr THEN inner ELSE IF expr THEN inner ELSE inner | IF expr THEN IF expr THEN inner ELSE IF expr THEN inner ELSE stmt | IF expr THEN IF expr THEN inner ELSE inner | IF expr THEN IF expr THEN inner ELSE stmt | IF expr THEN inner | IF expr THEN stmt | stmt |
The other option is immediately to reduce by stmt --> inner, beginning a sequence that goes like this:
IF expr THEN IF expr THEN inner ELSE IF expr THEN inner | ELSE BASIC IF expr THEN IF expr THEN inner ELSE IF expr THEN stmt | ELSE BASIC IF expr THEN IF expr THEN inner ELSE stmt | ELSE BASIC IF expr THEN inner | ELSE BASIC IF expr THEN inner ELSE | BASIC IF expr THEN inner ELSE BASIC | IF expr THEN inner ELSE inner | IF expr THEN inner ELSE stmt | inner | stmt |
The string
IF expr THEN IF expr THEN inner ELSE IF expr THEN inner ELSE BASIC
is not a string of tokens, but if X is an expression, the sequence
IF X THEN IF X THEN BASIC ELSE IF X THEN BASIC ELSE BASIC
leads, with some reduction along the way, to the same configuration, and is therefore a witness that the grammar is ambiguous.
Ocamlyacc input
%token BASIC X IF THEN ELSE
%type<unit> stmt
%start stmt
%%
stmt : inner {()}
| IF expr THEN stmt {()} ;
inner : BASIC {()}
| IF expr THEN inner ELSE stmt {()} ;
expr : X {()} ;
Report file
This is the contents of the file cati.output that is produced by Yacc when given the -v flag. Rules 1 to 5 represent the input grammar; rule 6 is added by Yacc with a semantic action that makes the parser return the AST for the statement it has parsed, and rule 0 is added in order to establish the end-of-file token EOF as a lookahead expected at the end of the input. A Yacc parser can have several start symbols, each giving a parser for part of the language that can be invoked from the main program; the magic token *1* tells the parser which entry point has been chosen, and there would be other magic tokens *2*, etc., and multiple variations on rule 6, in a parser with multiple entry points.
Following the summary of the rules appears a list of states in the LR(0) automaton compiled by Yacc. The administrative housekeeping just mentioned means that the real work of parsing starts in state 2. Each state has a number of LR(0) items (production rules with a marked place in the RHS, denoted by a dot). It also has a list of actions, each labelled with a lookahead token, and either shift s for some state s, or reduce r for a numbered rule r; for some states there is a default reduction rule labelled with a dot. States can also have goto entries labelled with a nonterminal, and with the shift entries, these are used to compute a trajectory through the state space in the way explained above.
Rules: 0: *start* --> *entry* EOF 1: stmt --> inner 2: stmt --> IF expr THEN stmt 3: inner --> BASIC 4: inner --> IF expr THEN inner ELSE stmt 5: expr --> X 6: *entry* --> *1* stmt State 0 *start* --> . *entry* EOF (0) *1* shift 2 *entry* goto 1 State 1 *start* --> *entry* . EOF (0) State 2 *entry* --> *1* . stmt (6) BASIC shift 3 IF shift 4 stmt goto 5 inner goto 6 State 3 inner --> BASIC . (3) . reduce 3 State 4 stmt --> IF . expr THEN stmt (2) inner --> IF . expr THEN inner ELSE stmt (4) X shift 7 expr goto 8 State 5 *entry* --> *1* stmt . (6) . reduce 6 State 6 stmt --> inner . (1) . reduce 1 State 7 expr --> X . (5) . reduce 5 State 8 stmt --> IF expr . THEN stmt (2) inner --> IF expr . THEN inner ELSE stmt (4) THEN shift 9 State 9 stmt --> IF expr THEN . stmt (2) inner --> IF expr THEN . inner ELSE stmt (4) BASIC shift 3 IF shift 4 stmt goto 10 inner goto 11 State 10 stmt --> IF expr THEN stmt . (2) . reduce 2 State 11: shift/reduce conflict (shift 12, reduce 1) on ELSE State 11 inner --> IF expr THEN inner . ELSE stmt (4) stmt --> inner . (1) EOF reduce 1 ELSE shift 12 State 12 inner --> IF expr THEN inner ELSE . stmt (4) BASIC shift 3 IF shift 4 stmt goto 13 inner goto 6 State 13 inner --> IF expr THEN inner ELSE stmt . (4) . reduce 4
- ↑ The longer sequence can be found immediately by using an LR(1) parser generator, such as the one at http://jsmachines.sourceforge.net/machines/lr1.html . Here is a suitable grammar:
S' -> S S -> I S -> if E then S I -> b I -> if E then I else S E -> x
The analysis reveals a parsing conflict, and the shortest string that leads to the state with a conflict is the one shown as an ambiguous input above.