Just so you understand why no one has answered you:
I too have this problem, it appears to be a problem with qmail or the smtp handler itself not generating a bounce message. Here's the kicker, I've configured obtuse-smtpd and qmail to deny messages over 8MB.
You would be lead to believe this is a good idea, with respsect to your box. However, here's the Catch-22:
Becauese the messages don't bounce, the remote mailer daemon attemps to keep retrying until eventually it times out. It's a HUGE bandwidth waste. It will continue to transmit the message/attachment and only cut it off after it has exceeded the defined quota. No bounce messages will be generated and the remote MTA may try again, up to X times (Real-world, I've seen about 6, it depends on how the remote MTA is configured).
I'd love to see a response where someone could generate a bounce. I'm even open to source-code patching.