Yeah but (blind) IP spoofing over the Internet is infeasible nowadays. Maybe 15 years ago when ISN randomization was not the rule (successful attack described in http://web.textfiles.com/hacking/shimomur.txt)
But that's correct, in recent openssh versions, it seems that you can add specific-host-only rules for authentication etc.
Of course this leaves you a vulnerable to ip spoofing, but adds a ton of convenience and could be a good trade-off.