The math breaks down with the singularity. It's not the infinity that's a real problem, it's that our ability to describe the real problem is lacking.
If the maths breaks down, and the singularity exists in the physical world, why do you not see this as evidence of infinity in the physical world?