Generally, the people who are nay-sayers of greylisting haven't used it much or at all.
Yes, "delays" in email are annoying to people who confuse email for being an "instant messenger service" and don't appreciate SMTP for the protocol that it is. (This is "most users" as it turns out) And for some, a bit of tuning is needed when weird uses of email protocols are in use.
To address the "delays" problem, it is easy to tell users "oh, it must have gotten stuck in our spam filter. ask them to send it again and it should go through." This works for most users who will happily bounce away and ask the remote party to send again... happily because they usually love that there is strong spam filtering in place and that it is working.
As for the tuning, there are different ways this is done, but in nearly all cases, it is done either initially or early on in greylisting configuration and then never again. One nice thing about greylisting is that there are "white lists" of known misbehaving mail servers and multi-server clusters that operate from multiple IPs (like yahoo). For your particular application, some people like to also white-list common contact email servers... that's usually okay, but not necessary if patient.
One thing about greylisting is that it only works when there is direct server-to-server communications. Your SME server can't have a spam filter appliance in front of it. This is because greylisting tracks the IP address of the sender and if there is a filtering appliance or server processing the mail and relaying it to your SME server, all incoming mail appears to be coming in from the same IP address.
Greylisting confuses people because there is a bit more to know and understand about how SMTP [is supposed to] works. It also annoys people because inappropriate implementations of mail services can more confusion and annoyance. (Rather like MSIE's broken implementation of HTML standards makes people believe things that aren't true.)
For people who can stick to their guns on these issues, greylisting is probably the most important facet of their spam blocking solution since it stops bad email from being scanned and processed and that sort of thing. Greylisting reduces server load in a very big way. But with all that said, I don't recommend implementing greylisting on your SME server.
"Why?! If you are such a fan and supporter of greylisting, why do you not recommend it for SME server?!"
Simple: I recommend ESVA which is a free/open source spam filter virtual appliance that really, really works. It employs greylisting and other techniques and also provides useful functions like quarantining and report generation.