flex+bison problem: syntax error that shouldn't be

[Rui Maciel]

I've just started looking into flex+bison and unfortunately I've stumbled
on a problem that I'm not able to figure out. I have a grammar definition
which I believe is sane and a lexer which works fine but when I try to
feed the parser a valid entry, the parser complains about a syntax error.

Could anyone please point out to me what is wrong with this code? Thanks
in advance.

The code is as follows:

the flex source code:

%{
#include <stdio.h>
#include <string.h>

#include "definition.h"
extern char* yylval;
%}
char [A-Za-z]
num [0-9]
eq [=]
name {char}+
age {num}+
%%
{name} { yylval = strdup(yytext); return NAME; }
{eq} { return EQ; }
{age} { yylval = strdup(yytext); return AGE; }
[ \t] /* ignore */
. {yyerror("invalid character\n");}
%%

int yywrap()
{
return 1;
}


The bison source code:
%{
#include <stdio.h>

typedef char* string;
#define YYSTYPE string
%}

%token NAME EQ AGE

%start file

%%
file: record file
| record
;
record: '\n'
| NAME EQ AGE { printf("%s is %s years old!!!\n", $1, $3); }
;
%%

int main(void)
{
yyparse();
return 0;
}

int yyerror(char *msg)
{
yydebug=1;
printf("Error encountered: %s \n", msg);
}

[f***@yahoo.com]

Post by Rui Maciel
I've just started looking into flex+bison and unfortunately I've stumbled
on a problem that I'm not able to figure out. I have a grammar definition
which I believe is sane and a lexer which works fine but when I try to
feed the parser a valid entry, the parser complains about a syntax error.
Could anyone please point out to me what is wrong with this code? Thanks
in advance.
%{
#include <stdio.h>
#include <string.h>
#include "definition.h"

[...]

You didn't include definition.h, so I can't compile this to try it.

Also, please include an example of some input that demonstrates the
problem, and say what error you get.

[Rui Maciel]

Post by f***@yahoo.com

Post by Rui Maciel
#include "definition.h"

[...]
You didn't include definition.h, so I can't compile this to try it.

You are right. Here it is:

#ifndef DEFINITION
#define DEFINITION

#define NAME 257
#define EQ 258
#define AGE 259

#endif

Post by f***@yahoo.com
Also, please include an example of some input that demonstrates the
problem, and say what error you get.

With the following input it appears that the parser simply chokes at the
first token (NAME):

joe=10

On the other hand, it appears that the parser doesn't complain, at least
initially, with input such as:

=10

which, to my newbie brain, doesn't make sense.


So, can anyone help?

Rui Maciel

[Jens Thoms Toerring]

Post by Rui Maciel
I've just started looking into flex+bison and unfortunately I've stumbled
on a problem that I'm not able to figure out. I have a grammar definition
which I believe is sane and a lexer which works fine but when I try to
feed the parser a valid entry, the parser complains about a syntax error.
Could anyone please point out to me what is wrong with this code? Thanks
in advance.
%{
#include <stdio.h>
#include <string.h>
#include "definition.h"
extern char* yylval;
%}
char [A-Za-z]
num [0-9]
eq [=]
name {char}+
age {num}+
%%
{name} { yylval = strdup(yytext); return NAME; }
{eq} { return EQ; }
{age} { yylval = strdup(yytext); return AGE; }
[ \t] /* ignore */
. {yyerror("invalid character\n");}
%%
int yywrap()
{
return 1;
}
%{
#include <stdio.h>
typedef char* string;
#define YYSTYPE string
%}
%token NAME EQ AGE
%start file
%%
file: record file
| record
;
record: '\n'
| NAME EQ AGE { printf("%s is %s years old!!!\n", $1, $3); }
;
%%
int main(void)
{
yyparse();
return 0;
}
int yyerror(char *msg)
{
yydebug=1;
printf("Error encountered: %s \n", msg);
}

It's rather unclear what you consider to be not working.
The "grammar" you have here is rather simple, all it
accepts as correct syntax is something like

[ \t]?[A-Za-z]+[ \t]?=[ \t]?[0-9]+

and repetititons of this, e.g. "xyz = 23 abc = 123". Everything
else is a syntax error - that's how the grammar is specified.

I also would recomment to change the whole thing a bit.
NAME looks like it is supposed to be string, but AGE
rather likely should be an (unsigned) integer value (and
you're probably going to use that value as in integer
sometime later). So I would recommend to change the
files a bit:

scanner.l
----------8<--------------------------
%option noyywrap nounput

%{
#include <stdio.h>
#include <string.h>
#include "definition.h"
extern void yyerror( const char *msg );
%}

char [A-Za-z]
num [0-9]
eq [=]
name {char}+
age {num}+

%%
{name} { yylval.sptr = strdup( yytext );
return NAME; }
{eq} { return EQ; }
{age} { yylval.ulval = strtoul( yytext, NULL, 10 );
return AGE; }
[ \t] /* ignore */
. { yyerror( "invalid character\n" ); }
%%
----------8<--------------------------

definition.y:
----------8<--------------------------
%{
#include <stdio.h>

void yyerror( const char *msg );
extern int yylex( void );
%}

%union {
char * sptr;
unsigned long int ulval;
}

%token <sptr> NAME
%token <ulval> AGE
%token EQ

%start file

%%
file: record file
| record
;
record: '\n'
| NAME EQ AGE { printf( "%s is %lu years old!!!\n", $1, $3 );
free( $1 ); }
;
%%

int
main( void )
{
yyparse( );
return 0;
}

void
yyerror( const char *msg )
{
yydebug = 1;
printf( "Error encountered: %s \n", msg );
}
----------8<--------------------------

Create the program with

Post by Rui Maciel
bison -d -v -t -o definition.c definition.y
flex -B -o scanner.c scanner.l
gcc -W -Wall -o my_grammar scanner.c definition.c

This should compiler cleanly and create a file 'definition.h'
to be included in scanner.l.
Regards, Jens

[Rui Maciel]

It's rather unclear what you consider to be not working. The "grammar"
you have here is rather simple, all it accepts as correct syntax is
something like
[ \t]?[A-Za-z]+[ \t]?=[ \t]?[0-9]+
and repetititons of this, e.g. "xyz = 23 abc = 123". Everything else is
a syntax error - that's how the grammar is specified.

Indeed. I was expecting to get the behavior that you described but the
parser simply wasn't cooperating. The code that I presented, when
compiled without specifying any flags (flex lex.l, bison -t grammar.y),
produced this result:

***@kubuntu:flex$ ./main
joe=10
Error encountered: syntax error
Cleanup: discarding lookahead token $undefined ()
Stack now 0

On the other hand, your code appears to work exactly as expected. Do you
happen to know what went wrong with my code so that it could compile
flawlessly but nonetheless produced broken code?


Thanks for the help,
Rui Maciel

[Jens Thoms Toerring]

Post by Rui Maciel

It's rather unclear what you consider to be not working. The "grammar"
you have here is rather simple, all it accepts as correct syntax is
something like
[ \t]?[A-Za-z]+[ \t]?=[ \t]?[0-9]+
and repetititons of this, e.g. "xyz = 23 abc = 123". Everything else is
a syntax error - that's how the grammar is specified.

Indeed. I was expecting to get the behavior that you described but the
parser simply wasn't cooperating. The code that I presented, when
compiled without specifying any flags (flex lex.l, bison -t grammar.y),
joe=10
Error encountered: syntax error
Cleanup: discarding lookahead token $undefined ()
Stack now 0
On the other hand, your code appears to work exactly as expected. Do you
happen to know what went wrong with my code so that it could compile
flawlessly

Flawlessly is a bit of a msinomer, the compiler complained quite
a bit about several things;-)

Post by Rui Maciel
but nonetheless produced broken code?

Well, the content of the header file you posted in your reply
to "fjblurt"

#ifndef DEFINITION
#define DEFINITION

#define NAME 257
#define EQ 258
#define AGE 259

#endif

doesn't look correct when compared to what I get when I run bison
on your .y file. In the resulting .tab.c file the definintions
look a bit different

#define NAME 258
#define EQ 259
#define AGE 260

And that could explain the problem you're getting since that makes
the tokenizer and lexer misunderstanding each other. If I use the
definitions from the .tab.c file everything starts to work again.
It tends to be a good idea to use the header file created by bison
when the '-d' option is specified because that will help to avoid
such problems.
Regards, Jens