The reason I thought he did not have a set is that I thought he would push allin to protect against the flush draws instead of his minraise of your flop bet. I would have probably done the same thing as you here. Hopefully more advanced players can give you their (more valuable) advice on this situation. Anyway, you still had 11 outs after the flop, which is a pretty decent .40 to improve to the winning hand here. Sorry it didn't work out.