weak fold?

two hands before, i won a 60BB pot from villain because i induced a bluff on the river and called his busted flushdraw, so i don't think he's bluffing again. on the other hand, this fold feels VERY weak.

PokerStars No-Limit Hold'em, $1.00 BB (5 handed) Hand History Converter Tool from FlopTurnRiver.com (Format: FlopTurnRiver)

saw flop|saw showdown

Hero ($235.05)
UTG ($106.10)
MP ($139.75)
Button ($115.30)
SB ($66.25)

Preflop: Hero is BB with Jhttp://www.flopturnriver.com/phpBB2/...es/diamond.gif, Jhttp://www.flopturnriver.com/phpBB2/...iles/heart.gif.
2 folds, Button calls $1, SB completes, Hero raises to $4, Button calls $3, SB folds.

Flop: ($9) 6http://www.flopturnriver.com/phpBB2/...miles/club.gif, Thttp://www.flopturnriver.com/phpBB2/...miles/club.gif, 4http://www.flopturnriver.com/phpBB2/...iles/spade.gif (2 players)
Hero bets $6, Button raises to $20, Hero folds.

Final Pot: $35

Results in white below:
No showdown. Button wins $35.

if hes too aggro, why couldnt he be doing this with just top pair thinking hes good? i think i would shove it in

I personnally dont hate a fold here, but keep in mind im of the conservative type OOP