Topic: Problem on backup process / LDAP Backup

[dadou]

Hello,

My backup process is returning Backup terminated: pre-backup failed - status: 256.

When I look @logs in /var/log/messages I've got a pb on this command of the backup process :

Code: [Select]

[root@server ~]# /etc/e-smith/events/pre-backup/S30ldap-dump
64590ce5 bdb(dc=xxxx,dc=xxx): BDB0060 PANIC: fatal region error detected; run recovery
64590ce5 bdb_db_open: database "dc=xxxx,dc=xxx" cannot be opened, err -30973. Restore from backup!
64590ce5 backend_startup_one (type=bdb, suffix="dc=xxxx,dc=xxx"): bi_db_open failed! (-30973)
slap_startup failed

This seems to kill the backup process.
I tried to launch a signal-event ldap-update but didn't change the issue.
Is there a way to rebuild the database ?

Thanks

David

[ReetP]

First thing is give us some debug information please.

What version of Koozali SME?

When did this start? What had you done before to get to this point? History of server etc.

Anything in the logs to show when this started or any other errors?

See how to looks for errors here:

https://wiki.koozali.org/Talk:Useful_Commands

Then either run the server-manager 'Report a Bug' or use these commands

Code: [Select]

/sbin/e-smith/audittools/newrpms
/sbin/e-smith/audittools/repositories
/sbin/e-smith/audittools/templates

I don't believe there is an easy way to get your LDAP DB back as it is synced with various other things.

However, before rushing off lets see if we can determine what broke, how and when.

[dadou]

Hello Reet,

Thanks for your answer.

I'm ok Koozali 10.1 updated regularly.
On 26th Apr backup was ok.

I made yum update on 27th (update of php packages I think) and I think I had a powerloss that made a reboot uncleaned of the system. Probably while backup was running. the problem appeared since this reboot.

At reboot logs are pretty clean exept these (but I already had some of these sometimes before the problem) : 

Code: [Select]

Apr 27 11:24:35 server kernel: [   35.730817] db_recover[2802]: segfault at 7fe7b29dadc0 ip 00007fe7b34c14a1 sp 00007fffe6337e38 error 4 in libdb-5.3.so[7fe7b339c000+1b5000]
Apr 27 11:24:36 server kernel: [   37.254804] db_recover[3339]: segfault at 7f201faeddc0 ip 00007f20205d44a1 sp 00007ffecc6d97d8 error 4 in libdb-5.3.so[7f20204af000+1b5000]
Apr 27 11:24:38 server kernel: [   38.711661] db_recover[3880]: segfault at 7f3af92b7dc0 ip 00007f3af9d9e4a1 sp 00007ffe1e7af028 error 4 in libdb-5.3.so[7f3af9c79000+1b5000]
Apr 27 11:24:39 server kernel: [   39.965502] db_recover[4439]: segfault at 7f0315b80dc0 ip 00007f03166674a1 sp 00007ffe61479968 error 4 in libdb-5.3.so[7f0316542000+1b5000]
Apr 27 11:24:41 server kernel: [   41.339104] db_recover[4905]: segfault at 7f2472e9bdc0 ip 00007f24739824a1 sp 00007ffe76cef138 error 4 in libdb-5.3.so[7f247385d000+1b5000]
Apr 27 11:26:21 server kernel: [  141.406682] cvm-unix[748]: segfault at 0 ip 00007f708d07b037 sp 00007ffd4b8180a8 error 4 in libc-2.17.so[7f708cf3c000+1c4000]

And here is my configuration report.

Code: [Select]

Configuration report created lun 08 mai 2023 23:45:24 CEST

==================
Base configuration
==================

SME server version:   10.1
SME server mode:      serveronly
SME server previous mode: serveronly
Running Kernel:        3.10.0-1160.88.1.el7.x86_64



===========================
New RPMs not in base system
===========================
       
Modules complémentaires chargés : fastestmirror, post-transaction-actions,
                                : priorities, smeserver
Loading mirror speeds from cached hostfile
 * base: centos.mirrors.proxad.net
 * smeaddons: mirror.pialasse.com
 * smeos: mirror.pialasse.com
 * smeupdates: mirror.pialasse.com
 * updates: centos.mirrors.proxad.net
Paquets supplémentaires
GeoIP.x86_64                            1.6.12-9.el7.sme            @smecontribs
GeoIP-GeoLite-data.noarch               2018.06-7.el7.sme           @smecontribs
GeoIP-GeoLite-data-extra.noarch         2018.06-7.el7.sme           @smecontribs
fail2ban-sendmail.noarch                0.11.2-3.el7                @smecontribs
fail2ban-server.noarch                  0.11.2-3.el7                @smecontribs
fping.x86_64                            3.10-4.el7                  @smecontribs
perl-Data-Validate-IP.noarch            0.27-13.el7                 @smecontribs
smeserver-fail2ban.noarch               9:0.1.18-30.el7.sme         @smecontribs
smeserver-sysmon.noarch                 6.5-5.el7.sme               @smecontribs
 



===========================
Custom and modified templates
===========================
/etc/e-smith/templates-custom/etc/hosts.allow/sshd: MANUALLY_ADDED, OVERRIDE
/etc/e-smith/templates-custom/home/e-smith/ssl.crt: MANUALLY_ADDED, OVERRIDE




===========================
Modified events
===========================




=======================
Additional repositories
=======================

base: enabled
centosplus: disabled
epel: disabled
extras: disabled
fasttrack: disabled
fws: disabled
remi-safe: enabled
smeaddons: enabled
smecontribs: disabled
smedev: disabled
smeextras: enabled
smeos: enabled
smetest: disabled
smeupdates: enabled
smeupdates-testing: disabled
updates: enabled

If I cannot rebuild the ldap db with an easy way, I suppose it will be more easy for me to reinstall full server.
But if I could do a degraded backup in the meantime it would be great. Is there a way to remove the backup of the ldap db from the backup script ?

Thanks again,

David

[ReetP]

Thanks for your answer.

NP

I'm ok Koozali 10.1 updated regularly.

Cool. And thank you for providing the requested information - it makes life SOOOOOOO much easier! At least you have kept the box pretty clean which normally helps.

On 26th Apr backup was ok.

I made yum update on 27th (update of php packages I think) and I think I had a powerloss that made a reboot uncleaned of the system. Probably while backup was running. the problem appeared since this reboot.

Ah that's a tad fugly then

Dare I ask what happened to your UPS? That would normally force an orderly shutdown?

If I cannot rebuild the ldap db with an easy way, I suppose it will be more easy for me to reinstall full server.
But if I could do a degraded backup in the meantime it would be great. Is there a way to remove the backup of the ldap db from the backup script ?

So some discussion amongst us on RocketChat about this.

Verbatim paste with a couple of small <edits>

Gary:

I've been there before with ldap, kept some notes, not reviewed, looks like sme9, will paste here
SME Server pre-backup fails LDAP;

ALSO; https://bugs.contribs.org/show_bug.cgi?id=6921

Shutdown the ldap service

Code: [Select]

sv d /service/ldap
<Edit: Possibly now: >

Code: [Select]

systemctl stop ldap
Put your correct ldif in /home/e-smith/db/ldap/$domainname.ldif

Wipe your ldap DB

Code: [Select]

find /var/lib/ldap/ -type f -exec rm -f "{}" \;
And start ldap again

Code: [Select]

sv u /service/ldap
<Edit: Possibly now: >

Code: [Select]

systemctl stop ldap

Jean Philippe:

Out of sync will be fixed while reloading the db using the different script
However corrupted ldap will prevent anything to get sync or backup. so Garys approach above is good
Just need to change the way to start and stop ldap <edit systemctl stop/start ldap> and I would suggest a cp -a before deleting
Also I am just unsure if the password will be sync for new users this way - I mean those created between the ldif and now

The simple answer is this is not a great scenario mainly due to the fact that the server tries to synchronise databases - passwd, ldap and possibly something else I forget - groups?

When one goes you are in a world of pain as it is nigh on impossible to properly sync them again AFAIAA.

This may get you going for a while, but a full restore will be required ASAP.

[dadou]

Hello Reet,

Thanks again.

I just deleted the unclean ldap db and restarted ldap who recreated new db files as suggested.
That solved the pb of backups for the moment (that was the immediate need). I'll fully reinstall ASAP.

Have a good day.

David

[ReetP]

NP. Glad you are running.

Smb was the other DB!

So Unix passwd/group, smbpasswd and ldap.

It is a very complicated dance.

Ironic as I have a corruption on one of my old boxes but it's of no consequence in my instance. Annoyance rather than hindrance.

I'll do a restore at some point to tidy it up.

B. Rgds
John

[rossbcan]

Saved my bacon.