Since he is very loose PF, is it possible he had 76? that would give him a straight draw on the flop and turn and in his mind a gut-shot might be worth making those calls. Then the 4 would give him the straight on the river which explains why he called your shove.