How many hands is this over? I would definitely let this go, but you cant rely that much on his raising range without a good amount of hands simply because a lot of possible hands may not have even been dealt to him. In a situation like this I would fold against an unknown, against this specific player I would fold, against a maniac who was way out of line I would call.