Haskell Parsec - error messages are less helpful while using custom tokens
haskell, lexical-analysis, parsec, parsing
Solution
A beginning of solution can be to define your choice function in the Parser, use a specific unexpected function to override unexpected error and finally use the `<?>` operator to override the expecting message:
mychoice [] = mzero
mychoice (x:[]) = (tok x <|> myUnexpected) <?> show x
mychoice (x:xs) = ((tok x <|> mychoice xs) <|> myUnexpected) <?> show (x:xs)
myUnexpected = do
input <- getInput
unexpected $ (id $ first input )
where
first [] = "eof"
first (x:xs) = show $ fst x
and call your parser like that :
ghci> Parser.parseTest (mychoice [Ide "ok", Ide "nop"]) "asdf "
parse error at (line 1, column 1):
unexpected Ide "asdf"
expecting [Ide "ok",Ide "nop"]
Problem
I'm working on seperating lexing and parsing stages of a parser. After some tests, I realized error messages are less helpful when I'm using some tokens other than Parsec's Char tokens. Here are some examples of Parsec's error messages while using Char tokens: ``` ghci> P.parseTest (string "asdf" >> spaces >> string "ok") "asdf wrong" parse error at (line 1, column 7): unexpected "w" expecting space or "ok" ghci> P.parseTest (choice [string "ok", string "nop"]) "wrong" parse error at (line 1, column 1): unexpected "w" expecting "ok" or "nop" ``` So, string parser shows what string is expected when found an unexpected string, and choice parser shows what are alternatives. But when I use same combinators with my tokens: ``` ghci> Parser.parseTest ((tok $ Ide "asdf") >> (tok $ Ide "ok")) "asdf " parse error at "test" (line 1, column 1): unexpected end of input ``` In this case, it doesn't print what was expected. ``` ghci> Parser.parseTest (choice [tok $ Ide "ok", tok $ Ide "nop"]) "asdf " parse error at (line 1, column 1): unexpected (Ide "asdf","test" (line 1, column 1)) ``` And when I use `choice`, it doesn't print alternatives. I expect this behavior to be related with combinator functions, and not with tokens, but seems like I'm wrong. How can I fix this? Here's the full lexer + parser code: Lexer: ``` module Lexer ( Token(..) , TokenPos(..) , tokenize ) where import Text.ParserCombinators.Parsec hiding (token, tokens) import Control.Applicative ((<*), (*>), (<$>), (<*>)) data Token = Ide String | Number String | Bool String | LBrack | RBrack | LBrace | RBrace | Keyword String deriving (Show, Eq) type TokenPos = (Token, SourcePos) ide :: Parser TokenPos ide = do pos <- getPosition fc <- oneOf firstChar r <- optionMaybe (many $ oneOf rest) spaces return $ flip (,) pos $ case r of Nothing -> Ide [fc] Just s -> Ide $ [fc] ++ s where firstChar = ['A'..'Z'] ++ ['a'..'z'] ++ "_" rest = firstChar ++ ['0'..'9'] parsePos p = (,) <$> p <*> getPosition lbrack = parsePos $ char '[' >> return LBrack rbrack = parsePos $ char ']' >> return RBrack lbrace = parsePos $ char '{' >> return LBrace rbrace = parsePos $ char '}' >> return RBrace token = choice [ ide , lbrack , rbrack , lbrace , rbrace ] tokens = spaces *> many (token <* spaces) tokenize :: SourceName -> String -> Either ParseError [TokenPos] tokenize = runParser tokens () ``` Parser: ``` module Parser where import Text.Parsec as P import Control.Monad.Identity import Lexer parseTest :: Show a => Parsec [TokenPos] () a -> String -> IO () parseTest p s = case tokenize "test" s of Left e -> putStrLn $ show e Right ts' -> P.parseTest p ts' tok :: Token -> ParsecT [TokenPos] () Identity Token tok t = token show snd test where test (t', _) = case t == t' of False -> Nothing True -> Just t ``` SOLUTION: Ok, after fp4me's answer and reading Parsec's Char source more carefully, I ended up with this: ``` {-# LANGUAGE FlexibleContexts #-} module Parser where import Text.Parsec as P import Control.Monad.Identity import Lexer parseTest :: Show a => Parsec [TokenPos] () a -> String -> IO () parseTest p s = case tokenize "test" s of Left e -> putStrLn $ show e Right ts' -> P.parseTest p ts' type Parser a = Parsec [TokenPos] () a advance :: SourcePos -> t -> [TokenPos] -> SourcePos advance _ _ ((_, pos) : _) = pos advance pos _ [] = pos satisfy :: (TokenPos -> Bool) -> Parser Token satisfy f = tokenPrim show advance (\c -> if f c then Just (fst c) else Nothing) tok :: Token -> ParsecT [TokenPos] () Identity Token tok t = (Parser.satisfy $ (== t) . fst) <?> show t ``` Now I'm getting same error messages: ghci> Parser.parseTest (choice [tok $ Ide "ok", tok $ Ide "nop"]) " asdf" parse error at (line 1, column 1): unexpected (Ide "asdf","test" (line 1, column 3)) expecting Ide "ok" or Ide "nop"