pssl
Have you even tried diagnosing what's wrong with your sme6.5 server.
That may be the quickest way to resolve your immediate problem.
Did you run top -i or htop to see how much & which system resouces and memory are being used ? Have you looked at the messages & other eg squid log files in the View log files panel in server manager. They should give you very good indications of what's going on. You need to diagnose the problem rather than make wild guesses like the squid log file is too big. What about disk space check with df -h
If you upgrade now or backup & restore you may carry over the problem (depending on what the problem is of course).
If your server had been hacked (which is only a wild possibility at present with no proof), then you would want to clean up the system first and remove all traces of the hacking before transferring data & config to a new server.
In my opinion the simplest way to restore your sme6.5 to a newly installed sme7.4 is to use the restore from disk method outlined here
http://wiki.contribs.org/UpgradeDiskYou must follow all instructions to the letter as every step is critical. The beauty of the method is that it leaves your old disk intact, and if something goes wrong you can revert to your old sme6.5 system just by reconnecting the disk as it originally was. You also do not need to perform a seperate backup before doing the restore.
Having said that you should make a full backup before doing anything, just to be sure all your data & config is safe in case of accidental erasure from the disk or whatever.
Also see
http://wiki.contribs.org/Updating_from_SME_7.1.x_or_earlierNow you could just do a new install of sme7.4 and manually add users and ibays etc and then transfer selected data & config over using the tips outlined here
http://wiki.contribs.org/Backup_server_configbut that method also requires some skill and understanding. Also the naming convention of user mail folders changed so you will run into problems that will need manual intervention to fix, ie skills you say you don't have. The renaming mentioned happened automatically during an upgrade.