You are not telling me that you believe they are only trying to access your system at the default SSH port, do you? It is a false sense of security as they will most likely discover the new assigned port as well.
1-I didn't say they only scan port 22 but the (large) majority of ssh hacking attempts do occur on port 22. Let's do the math, there are some 65,000 ports and a hacker can either spend all his time scanning my IP address up to 65,000 times to try to see IF I have an ssh port open OR he can use the same amount of resources and scan 65,000 IP addresses to see if someone has port 22 open.
2-I also never said it would make the server more secure, you made that assumption. What I said is if you change the port, it will reduce all that log noise and that was part of the OP's original question.
But since we are on the subject lets make a comparison.
To prove my point about #1 above, early last year I set up a new server, enabled ssh to the public (using keys) for some configurations and accidentaly left it on port 22 for about a week before I discovered my mistake. In my logs, there were between 500-1000 auth failures per DAY (thank goodness for keys and strong passwords).
I changed the port and ever since (its been over a year now), I have NEVER had another unauthorized attempt, EVER. This is an "odds" game and my odds just got a whole lot better.
Now, does that make my server more secure...You be the judge. I would say that a 1000/day reduction in hacking attempt is more secure, wouldn't you?
If nothing else, it made my ssh log file tiny and reduced the amount of work my server was doing.
This can be debated forever but as far as I'm concerned, reducing the number of hacking attempts (by hundreds or even thousands per day) is most definitely added security.
There is no perfect solution (other than disabling ssh altogether) but if you must have ssh access to your server then using private/public keys AND changing the port is (in my opinion) better than just one or the other and that's why I suggested that the OP ALSO change the port. I never suggested that he change the port as the only thing he should be doing.
Changing the port is only one piece of the security puzzle, it's not a fix-all and I never suggested that it was.