I think I would fold this. I also think I would have bet the pot (6$) on the flop to signal strength. If I was him I would read your betting as weakness and put you on a AK not hitting. So it's possible that he is bluffing or betting a TT or something thinking you have no pair. Still, if he puts you on AK the turn 5 gave you a straight draw and he would then maybe like to protect a set as well. If he holds A4 he would have hit the straight, but would he raise that much then with no flush on he table? Probably not, but possible.

So I think your play early in the hand is too weak but that the fold was ok.