Thinking about it more, I think the term "it's all downhill from here" should be used to describe a situation where all the hard work has been done and now it's time to chill and reap the benefits.
It shouldn't describe something that started off good and then went shit. I'd be more inclined to describe that as uphill.