Unless this guy is a total maniac he has a big pair, doesn't put you on a made hand and wants to protect.

You don't have the pot odds to draw to the flush and your pair is most likely no good. There is a chance he's bluffing but they're slim IMO unless you have reads.

This decision comes down to whether you want to gamble or not.