Many guides for securing a SSH server will mention changing the SSH port you are using away from the default (TCP/22). while this is not bad advise what security are you really gaining from this?
Changing the default port does create one more thing that you have to remember (or teach to new personal). you can mitigate this by using a standard port such as TCP/2222 but then you are also than lowering the security gained. But having unique ports across multiple servers would add even more management.
Firewalls will block certain ports so using any port carries along some baggage. Both good and bad. so if you are not on a network you control you could be unable to access your server if the local admins do not allow the port you are trying to use pass the firewall. But this holds true for the standard ports as well as non-standard.
The background bots attacks will take up resources on your server. probably not a lot but you are paying to have these bots attack you.
You will ether ignore the SSH logs or waste time reading about these bots. This is probably the most compelling argument to move the SSH port. By moving the port off you will only see logs from attackers specifically going after you. Using something like fail-to-ban will mitigate this but is still a consideration.
For me right now I am sticking with port 22. My SSH server is well secured with only Key logon, no root logon, fail to ban, etc. But I am really doing this to keep an eye on the logon names that bots are trying to use.