It's Linux.
You can install just about anything you want on it. RPMs just make it easier to take care of dependant libraries, etc.
For instructions on how to install a non-RPM "thing", look at the instructions on how to install the "thing" in question! Most software comes with a README or INSTALL file. If it's source code, the steps are usually:
$ ./configure
$ make
$ su
$ make install
But it really depends on the software in question. Note that SME Server doesn't come with any compilers, etc installed. This is a *good* thing, as nothing's worse than putting development tools on a firewall. Assuming your system was breached, it would be like giving a hacker a workshop to play in.

So, you may have to build/compile whatever it is you want to install on a separate machine. Since SME Server is based on RedHat 7.0 (and some 7.1 if I'm not mistaken), then a 7.0/7.1 Redhat machine would be a good choice to use as a compile/build machine.