I think an important added security is to disallow root login to ssh, and preferably to only allow access to 1 special user with a complicated name. This way the attacker has to guess the correct username as well. Just saying.
The passphrase you set on SSH keys is used to encrypt your key with. This makes it hard for people that have physical access to your key (admins on your system, etc.) to use the key to impersonate you. So for that reason you should have a passphrase set.
It has no influence on the security of the login to other machines though. That only involves the key. Having a passphrase will not make it harder to brute-force the key (which is close to impossible;-), and it is not part of the authentication itself. So this is no two factor authentication: The only factor is the key, the passphrase is just required to retrieve that key.
Don’t forget using hosts.allow. While IPs can be spoofed, it’s another barrier to entry and a very good one.
I think an important added security is to disallow root login to ssh, and preferably to only allow access to 1 special user with a complicated name. This way the attacker has to guess the correct username as well. Just saying.
I have a nit to pick:
The passphrase you set on SSH keys is used to encrypt your key with. This makes it hard for people that have physical access to your key (admins on your system, etc.) to use the key to impersonate you. So for that reason you should have a passphrase set.
It has no influence on the security of the login to other machines though. That only involves the key. Having a passphrase will not make it harder to brute-force the key (which is close to impossible;-), and it is not part of the authentication itself. So this is no two factor authentication: The only factor is the key, the passphrase is just required to retrieve that key.