Parsing Allowed Token Lists Using Boost Spirit Qi
Using boost :: spirit :: qi I am trying to parse strings consisting of a label followed by a variable amount of delimited tokens. I call the grammar a_parse clause and using the provided empty parser, how do I skip the parser to preserve the newlines as I need to make sure the label is the first element on each line.
Simple base case:
label token, token, token
You can analyze the grammar:
line = label >> (token % ',') >> eol;
The problem I'm running into is that the grammar must accept zero or more tokens and that the tokens can be empty. The grammar should accept the following lines:
label
label ,
label , token
label token, , token,
I was unable to create a grammar that accepts all of the above examples. Any suggestions on how to solve this?
Edit:
Thanks for all the input on the above issue. Now for the fun part I forgot to include ... The grammar should also accept empty lines and split lines. (tokens without label) When I try to make the label optional, I get an infinite loop matching an empty string.
label
label token
token
source to share
You can accept an empty list with
line = label >> -(token % ',') >> eol;
Note that eol
won't work if your skipper also skips eol (so don't use for this purpose qi::space
, but for example qi::blank
)
Also, depending on the definition, token
you have to change it to accept the "empty" token as well
In response to comment: Fully working sample Live On Coliru
#include <boost/spirit/include/qi.hpp>
namespace qi = boost::spirit::qi;
int main()
{
using namespace qi;
using It = std::string::const_iterator;
using Token = std::string;
using Tokens = std::vector<Token>;
rule<It, blank_type> label
= lexeme[+~char_(":")] >> ':'
;
rule<It, Token(), blank_type> token
= lexeme[*~char_(",\n")];
;
rule<It, Tokens(), blank_type> line
= label >> -(token % ',') >> eol
;
for (std::string const input : {
"my first label: 123, 234, 345 with spaces\n",
"1:\n",
"2: \n",
"3: ,,,\n",
"4: , \t ,,\n",
"5: , \t , something something,\n",
})
{
std::cout << std::string(40, '=') << "\nparsing: '" << input << "'\n";
Tokens parsed;
auto f = input.begin(), l = input.end();
bool ok = phrase_parse(f, l, line, blank, parsed);
if (ok)
{
std::cout << "Tokens parsed successfully, number parsed: " << parsed.size() << "\n";
for (auto token : parsed)
std::cout << "token value '" << token << "'\n";
}
else
std::cout << "Parse failed\n";
if (f != l)
std::cout << "Remaining input: '" << std::string(f, l) << "'\n";
}
}
Output:
========================================
parsing: 'my first label: 123, 234, 345 with spaces
'
Tokens parsed successfully, number parsed: 3
token value '123'
token value '234'
token value '345 with spaces'
========================================
parsing: '1:
'
Tokens parsed successfully, number parsed: 1
token value ''
========================================
parsing: '2:
'
Tokens parsed successfully, number parsed: 1
token value ''
========================================
parsing: '3: ,,,
'
Tokens parsed successfully, number parsed: 4
token value ''
token value ''
token value ''
token value ''
========================================
parsing: '4: , ,,
'
Tokens parsed successfully, number parsed: 4
token value ''
token value ''
token value ''
token value ''
========================================
parsing: '5: , , something something,
'
Tokens parsed successfully, number parsed: 4
token value ''
token value ''
token value 'something something'
token value ''
source to share