Exactly the spot where "I had a pair!" fits. It all depends on image, like you say I'm not going to insult someone who thinks I'm decent by pretending otherwise, but if he's convinced I'm a donkey I'm happy to reinforce that.