Any ideas or suggestions?
How many ISPs are there between your server and your user? Do you get the same slow download speed if you go to a nearby network (friend's house, wifi hotspot) and download the same file?
Your ISP or your user's ISP may be throttling bandwidth for multimedia files. I don't know how you would prove this - maybe you could give him several files of different sizes, and see if the overall speed changes dramatically when the file exceeds a certain size (is the overall throughput the same on 10Mb, 50Mb, 100Mb and 700Mb files?) or a certain file name structure (maybe .mpg or .avi files download slower than .dbf files?)
The download may be using an inefficient protocol - can you give the user a different download method, in order to test? (If he's using FTP now, give him an HTTP address; if he's using HTTP, give him an FTP link; if he's using webdav, give him a "plain" HTTP URL; if he's using a VPN, give him a straight HTTP or FTP address; if he's using a webapp like owncloud, give him a direct HTTP or FTP link).
There could be a problem with network settings - if you have an ADSL modem enforcing a packet size of 1492 bytes, you might get better performance by hard-coding your SME server packet size to match as described
here (I have no idea if this would affect overall download speed -- it just seems like a possibility).
If none of the above steps suggests a cause or solution, you could setup an alternative host for the file in question (an appliance with a built-in FTP server, a windows box running an FTP server, or another non-SME linux box) and see if the download speed improves.