Koozali.org: home of the SME Server

Server crashed

Offline beast

  • *
  • 254
  • +0/-0
Server crashed
« on: March 27, 2026, 04:30:38 PM »
Hi.

My server is unable to boot and load the OS. It ask for configuration as during installation. What is the correct procedure to handle this. The server has 2 disk in raid 1 and I have a backup some weeks old.

Thank you in advance
Benny

Offline ReetP

  • *
  • 4,049
  • +6/-0
Re: Server crashed
« Reply #1 on: March 27, 2026, 08:30:53 PM »
The server has 2 disk in raid 1

OK good.

Quote
and I have a backup some weeks old.

Ah not so good :-(

OK, so please start at the beginning and tell us the history.

We need to understand how you got to this point before we can tell you how to get out of it.

The server was working OK. Then what? Power outage/server crash? Hard disk failure?

Have you checked both disks are working? No errors?

What did you do once it restarted? Did it have any menus?

Try and describe the sequence of events as accurately as possible - don't leave anything out - even if it is embarrassing ;-)

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #2 on: March 28, 2026, 05:52:45 AM »

What I did before was to try and get dkim working as my mails was rejected at gmail

yum install smeserver-opendkim

Then I may have made

signal-event post-install (this may be the error)

signal-event reboot

Then the server ask for kernel (recovery mode) in a list and later ask for root password and also the possibility to recover from backup. Looks like standard behavior after fresh install.

I have not checked the disk as I belive I have to take them out of the server for this?

Offline ReetP

  • *
  • 4,049
  • +6/-0
Re: Server crashed
« Reply #3 on: March 28, 2026, 08:12:35 AM »
Oh.

You should have read the docs first:

https://wiki.koozali.org/Email#DKIM_Setup_-_qpsmtpd_version_.3E.3D_0.96

No idea where you got smeserver-opendkim from as the above method has been used for many, many years.

Quote
signal-event post-install (this may be the error)

Most likely. You have ignored the docs and standard methods.

I'm not sure what the solution to this is. I'll ask if someone will take a look.

I suggest you do no more harm for now and wait for a response - not sure if it can be repaired or you need to restore from backup.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #4 on: March 28, 2026, 09:05:02 AM »
I think I trusted an AI too much :-(

The backup is made by a cron script to a Synology network storage

I think all files are still on the disk, but the configuration is broken.

Take out one disk as a backup and sync later. Mount the other on another PC

Is it then not possible to move the config from the backup to the server somehow (know it is in a db). Think I have only changed a couple passwords in the time period.

This is the cron script for backup (removed some security related info)

<?php         
      
         
<?php               
         
include_once($folder."phpmailer/class.phpmailer.php");         
         
function isCli()         
{         
$sapiname = php_sapi_name();         
         
if((substr($sapiname , 0, 3) == 'cgi') || (substr($sapiname , 0, 3) == 'cli')) // PHP 4 and 5 version         
if(empty($_SERVER['REMOTE_ADDR']))         
{         
return true;         
}         
         
return false;         
}         
         
function mailoutput($to, $mailbody)         
{         
   $mail = new PHPMailer();      
   $mail->IsMail();      
   $mail->SetLanguage("en");      
   $mail->IsHTML(true);      
   $mail->FromName = "noreply@beast.dk";      
   $mail->From = $mail->FromName;      
   $mail->Subject = "Beastserver backup log";      
   $mail->WordWrap = 50;      
         
   $mail->MsgHTML($mailbody);      
   $mail->AltBody = strip_tags(preg_replace("/<br>/", "\r\n", $mailbody));      
   $mail->AddAddress($to);      
   if(!$mail->Send())      
   {      
      die("Mail Error: " . $mail->ErrorInfo);   
   }      
}         
         
$parameters = $_SERVER['argv'];         
         
$mailbody = "<h2>Beastserver backup server log</h2>";         
         
$starttime = time();         
         
$returnstr = shell_exec("signal-event pre-backup");         
      
**** this below is removed for security issues
   
//$returnstr .= shell_exec("rsync -az -e \"ssh -p xxxx\" /home/e-smith/* admin@x.x.x.x::NetBackup/beastserver/home/e-smith/");         

/*         
signal-event pre-backup         
home/e-smith         
etc/e-smith/templates-custom         
etc/e-smith/templates-user-custom         
etc/ssh         
root         
etc/sudoers         
etc/passwd         
etc/shadow         
etc/group         
etc/gshadow         
/etc/dehydrated         
etc/samba/secrets.tdb         
etc/samba/smbpasswd         
signal-event post-backup         
*/      

**** until here      
         
$returnstr .= shell_exec("signal-event post-backup");         
            
         
$execmin = (time() - $starttime)/60;         
         
if(strlen($returnstr))         
$mailbody .= $returnstr;         
else         
$mailbody .= "<br>Backup performed to NAS in ".$execmin." minutes";         
               
if(isCli())         
{         
   foreach($parameters as $key => $value)      
   {      
      if($key > 0)   
         mailoutput($value, $mailbody);
   }      
}         
else         
{         
   if(isset($_GET['email']))      
      mailoutput($_GET['email'], $mailbody);   
}         
         
?>         
« Last Edit: March 28, 2026, 09:08:26 AM by beast »

Offline TerryF

  • grumpy old man
  • *
  • 1,876
  • +6/-0
Re: Server crashed
« Reply #5 on: March 28, 2026, 09:06:59 AM »
Feel the pain, no solutions sorry, personaly only seen the same scenario once befoe, asking for root pass and backup etc when the servers mysql users table was corrupted, new install and full restore was only solution.

Hopefully others may have a better idea
--
qui scribit bis legit

Offline ReetP

  • *
  • 4,049
  • +6/-0
Re: Server crashed
« Reply #6 on: March 28, 2026, 09:23:02 AM »
Quote
I think I trusted an AI too much :-(

Yup. FAFO.

We have said this before. Don't AT ALL with SME. Never, ever, trust it with AI.

Quote
The backup is made by a cron script to a Synology network storage

Joy. That'll make it harder to restore, assuming it will. Much easier with a tgz backup file. It has all the bits you need for easy restore.

You may need to look at the migratehelper script on how to resolve that.

Quote
I think all files are still on the disk, but the configuration is broken.

Most likely. The question is how to fix it.

JP may know and will likely be about later.

Quote
Take out one disk as a backup and sync later. Mount the other on another PC

I said earlier - do no harm. Don't break your RAID as well. Have some patience - hopefully you may get help later today.

Quote
Is it then not possible to move the config from the backup to the server somehow (know it is in a db).

It's not JUST config dbs. Ldap, sql etc. It's a complicated dance.

Again, rushing aimlessly got you into this mess, but won't get you out of it.

Patience, and see comments later.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mophilly

  • *
  • 385
  • +0/-0
    • Mophilly
Re: Server crashed
« Reply #7 on: March 28, 2026, 04:21:44 PM »
Take ReetP’s advice: don’t break the raid. That will confound an already bad situation.

Walking another person through a problem often reveals a new option.

What kind of machine is your server?
What steps did you take leading up to the corruption?

Without that information, it is all but impossible to offer useful advice from afar. Troubleshooting via public forum has its challenges. Contact me privately if you wish. I may not be able to help but I will try.

If you can boot from a USB stick, one loaded with a minimal OS, you may be able to copy files off the drive. That can be an enourmous amount of work. Consider whether the climb is worth the view at the end.

Email reputation is a thorny business, as you know. Not impossible but there are a few key details that must be set correctly; server config and DNS zone record in particular. SME makes the server side config easier, if you follow the SME way. The documentation is good and getting better. Getting to know it is time well spent.

A last thought: AI is a truly amazing tool. However, it is not all-knowing or even “smart.” It must be guided to proper sources and given well considered guidelines and specific tasks. For this example, I would build a prompt that includes links to relevant pages in the Koozali SME documentation set as well as the configuration details of the target server. I would include statements like “list options that are applicable to Koozali SME only”, “do not include examples from other Linux distributions, except for CentOS 7”, and “list sources for each finding”.

If your AI tool offers a “project” concept that retains knowledge of prior queries, use it. Then, review the output and validate the result by hand (not on your production system.) Amend the prompt with questions and new info as you find it, and run it again. This approach will likely provide more useful results.

Ok. Time for more coffee.
« Last Edit: March 28, 2026, 04:25:21 PM by mophilly »
- Mark

Offline ReetP

  • *
  • 4,049
  • +6/-0
Re: Server crashed
« Reply #8 on: March 28, 2026, 04:44:05 PM »
Note a lot of AI traffic is blocked from our docs, bugs etc as it was overloading our server.

Hence information will be scant, and old. So expect major hallucinations.

Best answer is always to read the docs, and ask here first.

Love to know where you found that contrib. That is truly ancient.

Note you do have a Rocket login so you can ask there too.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,978
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Server crashed
« Reply #9 on: March 28, 2026, 06:52:52 PM »
never heard of smeserver-opendkim rpm. have you a link?


signal-event post-instal is only for install. should never be run after, but i am not sure this is your issue


do not break your raid, you might enter in a situation where you might not boit latter on the one you think and lost some data.

use the install disk or supergrub disk to repear your boot/ reinstall last kernel. 

you might also take the time to chexk all disks with smartcl and fsck.


forget AI for anything related to SME, it will really break your server as we prevent any AI scrapper to play with our servers and shitting down access to regular users.

I have test most of the current AI chat and none is even close to be helpfull for SME, without saying mostly give commands that will break your install. 


Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #10 on: March 28, 2026, 07:27:10 PM »
After reading all your comment I think this is the way to go.

  • Make a USB OS and mount the drive. 
  • Copy important files out
  • Use the install disk or supergrub disk to repair the boot or reinstall last kernel. 
  • Check all disks with smartcl and fsck.

Do you think this will fix it? Does not recreate the configuration in my view?

/Benny

Offline Jean-Philippe Pialasse

  • *
  • 2,978
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Server crashed
« Reply #11 on: March 28, 2026, 09:53:29 PM »
when booted with rescue disk i would first check disk before doing any modification in it including repairing the boot.


from what you report this really only seems an issue with boot and/or disk

Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #12 on: March 28, 2026, 10:16:22 PM »
This was the latest commands:

[root@beastserver ~]# ps aux | grep opendkim
root      8119  0.0  0.0 112812   980 pts/0    S+   15:00   0:00 grep --color=auto opendkim
[root@beastserver ~]# yum install smeserver-opendkim
Loaded plugins: fastestmirror, post-transaction-actions, priorities, smeserver
Loading mirror speeds from cached hostfile
 * smeaddons: mirror.pialasse.com
 * smeextras: mirror.pialasse.com
 * smeos: mirror.pialasse.com
 * smeupdates: mirror.pialasse.com
base                                                                                                                                                             | 3.6 kB  00:00:00     
remi-safe                                                                                                                                                        | 3.0 kB  00:00:00     
smeaddons                                                                                                                                                        | 2.9 kB  00:00:00     
smeextras                                                                                                                                                        | 3.1 kB  00:00:00     
smeos                                                                                                                                                            | 3.1 kB  00:00:00     
smeupdates                                                                                                                                                       | 2.9 kB  00:00:00     
updates                                                                                                                                                          | 2.9 kB  00:00:00     
No package smeserver-opendkim available.
Error: Nothing to do
[root@beastserver ~]# signal-event post-install
 
[root@beastserver ~]# sign
-bash: sign: command not found
[root@beastserver ~]# signal-event reboot
Connection to 192.168.11.11 closed by remote host.
Connection to 192.168.11.11 closed.
benny@benny-pc:~$ ping 192.168.11.11
PING 192.168.11.11 (192.168.11.11) 56(84) bytes of data.
From 192.168.11.200 icmp_seq=1 Destination Host Unreachable
From 192.168.11.200 icmp_seq=2 Destination Host Unreachable
From 192.168.11.200 icmp_seq=3 Destination Host Unreachable
From 192.168.11.200 icmp_seq=4 Destination Host Unreachable

Offline Jean-Philippe Pialasse

  • *
  • 2,978
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Server crashed
« Reply #13 on: March 29, 2026, 11:50:57 AM »
Quote
Then the server ask for kernel (recovery mode) in a list and later ask for root password and also the possibility to recover from backup. Looks like standard behavior after fresh install.

at kernel selection edit the line and add what is needed to enter single user mode.

select your Kernel version from the grub menu item and press e key to edit the first boot option. Now use the Down arrow key to find the kernel line (starts with “linux16“), then change the argument
ro
to
rw init=/sysroot/bin/sh

then do ctrl X

alternatively boot with rescue disk and chroot on your sme.
Code: [Select]
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
mount --bind /run /mnt/sysimage/run
chroot /mnt/sysimage

then run this:
Code: [Select]
/sbin/e-smith/config setprop bootstrap-console Run no
/sbin/e-smith/config set PasswordSet yes
usermod --unlock admin
usermod --unlock root

Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #14 on: March 29, 2026, 12:15:25 PM »
I will try your suggestions. Right now I have a USB disk with SystemRescue running and like to mount the SME disk.

Any suggestions about how I mount the raid disk?

Is it LVM?

vgchange -ay
lvdisplay (ls /dev/mapper)
mount /dev/mapper/XXXXX /mnt/XXX

Or shall I use SME as rescue disk?

/Benny
« Last Edit: March 29, 2026, 12:39:46 PM by beast »

Offline Jean-Philippe Pialasse

  • *
  • 2,978
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Server crashed
« Reply #15 on: March 29, 2026, 04:51:54 PM »
if you use an install disk from centos/ sme/ fedora and select rescue mode it will :
- enable the raid
- enable lvm
- mount your system to /mnt/sysimage


otherwise you are on your own to do all those (non trivial)  steps

Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #16 on: March 29, 2026, 09:12:00 PM »
if you use an install disk from centos/ sme/ fedora and select rescue mode it will :
- enable the raid
- enable lvm
- mount your system to /mnt/sysimage

I am unable to find out how to enter rescue mode on the SME disk after boot (I can only see install and test mode)

Offline TerryF

  • grumpy old man
  • *
  • 1,876
  • +6/-0
Re: Server crashed
« Reply #17 on: March 30, 2026, 08:14:59 AM »
When you boot the iso from USB or such at first screen select the Troubleshooting item
Then at next screen select
Rescue ........
--
qui scribit bis legit

Offline beast

  • *
  • 254
  • +0/-0
Re: Server crashed
« Reply #18 on: March 31, 2026, 09:13:37 PM »
at kernel selection edit the line and add what is needed to enter single user mode.

select your Kernel version from the grub menu item and press e key to edit the first boot option. Now use the Down arrow key to find the kernel line (starts with “linux16“), then change the argument
ro
to
rw init=/sysroot/bin/sh

then do ctrl X


I was unable to find that line

alternatively boot with rescue disk and chroot on your sme.
Code: [Select]
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
mount --bind /run /mnt/sysimage/run
chroot /mnt/sysimage

then run this:
Code: [Select]
/sbin/e-smith/config setprop bootstrap-console Run no
/sbin/e-smith/config set PasswordSet yes
usermod --unlock admin
usermod --unlock root

After a lot of trouble I have the server running again with the above solution. But I need to replace the servers (I have 2 identical more than 10 years old) as I also expect that there are hardware errors with the motherboard. Monitor some times was black, DVD not loading and other small issues.

Thank you all for the assistance - makes my life better :-)