Focal length and aperture both affect how much focus matters.

Also with most regular cameras, from what I've heard they actually have their max focus slightly past infinity to accound for thermal issues so the general advice for doing astrophotography is to focus on the subject, not just whacking it to infinity.