Hmm you are drawing for a diamond if he has a made flush, a diamond with no pair on the board if he has a set and you want to avoid diamonds and aces if he has the Ace of diamonds).

None of these cases I think gives you odds to call for your stack. Only 1 scenario puts you ahead (the AdX) and only by a very small margin (like 55/45 I think if you "only" have to dodge aces and diamonds).

Unless you have a really good read that this guy is capable of making this play with any other hand (top pair and the Qd or Jd/a cold out bluff) I think this is an easy fold.