Scala Parser Token Delimiter Problem

bnf, ebnf, parser-combinators, parsing, scala

Solution

It is really simple. When you use `~`, you have to understand that there's no backtracking on individual parsers which have completed succesfully.

So, for instance, `message` got everything up to before the colon, as all of that is an acceptable pattern. Next, `properties` is a `rep` of `property`, which requires `propertyName`, but it only finds the colon (the first char not gobbled by `message`). So `propertyName` fails, and `property` fails. Now, `properties`, as mentioned, is a `rep`, so it finishes succesfully with 0 repetitions, which then makes `command` finish succesfully.

So, back to `parseAll`. The `command` parser returned succesfully, having consumed everything before the colon. It then asks the question: are we at the end of the input (`\z`)? No, because there is a colon right next. So, it expected end-of-input, but got a colon.

You'll have to change the regex so it won't consume the last identifier before a colon. For example:

def message = """[\w\d\s\.]+(?![:\w])""".r

By the way, when you use `def` you force the expression to be reevaluated. In other words, each of these defs create a parser every time each one is called. The regular expressions are instantiated every time the parsers they belong to are processed. If you change everything to `val`, you'll get much better performance.

Remember, these things define the parser, they do not run it. It is `parseAll` which runs a parser.

Problem

I'm trying to define a grammar for the commands below. ``` object ParserWorkshop { def main(args: Array[String]) = { ChoiceParser("todo link todo to database") ChoiceParser("todo link todo to database deadline: next tuesday context: app.model") } } ``` The second command should be tokenized as: ``` action = todo message = link todo to database properties = [deadline: next tuesday, context: app.model] ``` When I run this input on the grammar defined below, I receive the following error message: ``` [1.27] parsed: Command(todo,link todo to database,List()) [1.36] failure: string matching regex `\z' expected but `:' found todo link todo to database deadline: next tuesday context: app.model ^ ``` As far as I can see it fails because the pattern for matching the words of the message is nearly identical to the pattern for the key of the property key:value pair, so the parser cannot tell where the message ends and the property starts. I can solve this by insisting that start token be used for each property like so: ``` todo link todo to database :deadline: next tuesday :context: app.model ``` But i would prefer to keep the command as close natural language as possible. I have two questions: What does the error message actually mean? And how would I modify the existing grammar to work for the given input strings? ``` import scala.util.parsing.combinator._ case class Command(action: String, message: String, properties: List[Property]) case class Property(name: String, value: String) object ChoiceParser extends JavaTokenParsers { def apply(input: String) = println(parseAll(command, input)) def command = action~message~properties ^^ {case a~m~p => new Command(a, m, p)} def action = ident def message = """[\w\d\s\.]+""".r def properties = rep(property) def property = propertyName~":"~propertyValue ^^ { case n~":"~v => new Property(n, v) } def propertyName: Parser[String] = ident def propertyValue: Parser[String] = """[\w\d\s\.]+""".r } ```

Original source