I agree that even if VirtualBox can be used headless, libvirt+KVM will give better performance, and is much more flexible (but does require a bit of work to learn how to use it). If you have another Linux desktop, I'd just recommend not to install virt-manager (a GUI for libvirt) directly on the server, so you won't have to install all the graphic stack. Just deploy libvirt+KVM on the headless server, and access it remotely (through SSH for example) from another linux desktop