As I write this, I have some debilitating respiratory infection. It's absolutely a disability at the moment - even if only temporary (I hope lol). I can barely do anything without assistance but most people wouldn't consider it a disability like some others.

Now sickness might be a different case than what you're considering, but I don't consider it ableism for "healthy" people to not want my company right now. I would probably avoid them, too, if the roles were reversed.