I don't think you're putting villain on the correct range of hands. I agree that his flop line looks like a draw, but once he pushes turn I think you should at best limit the draws in his range to 6dXd and AdKd (which of course the fish play completely different from any other AdXd). Especially since you said he didn't seem too aggro. Also you excluded 45 for the flopped nuts, and my personal favourite AA.

I would fold this against passive/fishy player and don't worry too much about it.