Yes - My firewall/smoothwall is a direct replacement for your hardware firewall.
It's an old PC with 2 LAN cards - one is connected to the Router, and the other to my network switch, where the E-Smith, and PC's are connected
Smoothwall/IPCOP can be run in various configurations, some of which include a DMZ Network, but this is not neccesary.
Standard Microsoft NETBIOS runs using the NetBui Network Protocol (in the older operating Systems - Win98 and below) - This was not routable, however, it was/is possible to run NETBIOS over TCP/IP, and remove this restriction
However, to put the E-smith server in the DMZ will cause more problems - by default, there would be issues with accessing Samba File Shares, and domain logins as you say.
These can be overcome by opening pinholes for the relevant network Protocols from the DMZ to the internal Network. However this kind of defeats the purpose of putting the server in the DMZ
In the vast majority of home networks, there is no need for a DMZ
I have used various Firewalls - both GPL and corporate using DMZ's
These tend to be used separate dedicated servers that provide resources to external users - eg Web portals to corporate applications, Mail Scanners, etc.
In these cases, there are other servers providing SAMBA, domain controllers etc inside the firewall