Yeah well "fit for office" is not a legal phrase that I'm aware of, so if it doesn't mean "in my opinion" then it can only mean "in the public's opinion".

In Nixon's case, the public were wrong. If you want to argue the public are wrong with Trump too, fine.

But that is your opinion, not the public's. The only way we find out the public's opinion is by holding an election.