Thanks for the comments guys. I figured it was the right call...sets at my level tend to slowplay, and most people would at least think that I might have AK and not try to throw all their money in when behind with just two pair.

The pot (after his allin) was $7.90 (his calling my $1 plus plopping down the rest of his stack $4.75) so I was risking 4.75 to win $7.90 which is a little less than 2:1, when it was in fact a coinflip, but I have to bump up a 20% chance or so that he's bluffing with KQ/KJ/KT, A9/9T etc. which would have me ahead and make it a call all day for sure.