Steve,
My understanding is that there is no 2GB file limitation for a file residing on SME 5.6 hard drive. Also if you are backing up to a tape, the file size is not an issue. With the file size and the backups to conflict is when you use flexbackup to backup to a disk not physically installed in the SME box (ie network share).
Since SME is basically RedHat you might want to check out thier hardware compatibility list to sus out which tape drives are compatible with 5.6.
I dont have any clue on high performance hardware nor dell servers, but good luck!
Cyrus Bharda