X-Windows _is_ a security risk... due to the protocols it uses. X was developed for a LAN environment where network security is much tighter. So it is advisable to X behind a firewall to protect it from the flames of the internet.
People who run X-Windows on their machine directly connected to the internet commonly (should) use a host based firewall. This is a firewall running on the same computer, filtering packets before they reach X-Windows. Running a firewall on a stand alone machine ideally provides a higher level of security than a host based firewall. The e-smith gateway is designed to allow the user/system administrator to easily and securely create such a stand alone gateway/firewall.
(see any good Unix networking text for information regarding the protocols of X-Windows, their advantages and potential problems) - Though recent version of X have implemented changes that provide much higher security. It should still be remembered that X is a very complex system, so there are many places where security breaches could occur.
==================================================
If you just want a file manager there are a number console based file managers, that give a "graphical" display of the file system without the need for X-Windows. Some even mimic file managers of old such a Xtree Gold or Nortons Commander, indeed may people find the Linux version Midnight Commander a friendly and and powerful tool.
http://linuxberg.tucows.com/conhtml/fil_managers.html"...it is simplicity that is difficult to make."
- Bertholdt Brecht