I don't think he's ever bluffing here because, as you point out, your odds to call are relatively good and you're likely to call with marginal hands as well as good ones. You have 3-bet pre-flop and bet the flop, it's fairly likely you have some kind of hand here. Unless you have a strong reason to believe he's bullshitting, this seems to be a very easy fold.