Check the db logs in /var/log/mariadb
Checking the db logs in /var/log/mariadb, I get lots of these types of errors on the tikiwiki database:
230930 1:00:47 [ERROR] mysqld: Can't open file: './tikiwiki/tiki_rss_modules.frm' (errno: 24)
230930 1:00:47 [ERROR] mysqld: Can't open file: './tikiwiki/tiki_search_stats.frm' (errno: 24)
... and a bit later on the same with horde, although it's always the same table:
230930 1:25:01 [ERROR] mysqld: Can't open file: './horde/horde_alarms.frm' (errno: 24)
230930 1:30:02 [ERROR] mysqld: Can't open file: './horde/horde_alarms.frm' (errno: 24)
If you go back further in time, there's a log from a month ago with these entries:
230825 15:59:16 InnoDB: The InnoDB memory heap is disabled
230825 15:59:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins
230825 15:59:16 InnoDB: Compressed tables use zlib 1.2.7
230825 15:59:16 InnoDB: Using Linux native AIO
230825 15:59:16 InnoDB: Initializing buffer pool, size = 128.0M
230825 15:59:16 InnoDB: Completed initialization of buffer pool
230825 15:59:16 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
230825 15:59:33 InnoDB: Waiting for the background threads to start
230825 15:59:34 Percona XtraDB (http://www.percona.com) 5.5.61-MariaDB-38.13 started; log sequence number 508399285
230825 15:59:35 [Note] Plugin 'FEEDBACK' is disabled.
230825 15:59:40 InnoDB: Starting shutdown...
230825 15:59:43 InnoDB: Shutdown completed; log sequence number 508399285
230825 15:59:43 InnoDB: The InnoDB memory heap is disabled
230825 15:59:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
230825 15:59:43 InnoDB: Compressed tables use zlib 1.2.7
230825 15:59:43 InnoDB: Using Linux native AIO
230825 15:59:43 InnoDB: Initializing buffer pool, size = 128.0M
230825 15:59:43 InnoDB: Completed initialization of buffer pool
230825 15:59:43 InnoDB: highest supported file format is Barracuda.
230825 15:59:43 InnoDB: Waiting for the background threads to start
230825 15:59:44 Percona XtraDB (http://www.percona.com) 5.5.61-MariaDB-38.13 started; log sequence number 508399285
230825 15:59:44 [Note] Plugin 'FEEDBACK' is disabled.
230825 15:59:44 [Note] Server socket created on IP: '0.0.0.0'.
230825 15:59:45 [Note] Event Scheduler: Loaded 0 events
230825 15:59:45 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.68-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
However I'm not sure if that's a problem or not because the recurring error only happened a few days ago, not a month ago.
There was a power failure about a month ago, long enough for the UPS to fail as well. No issues until recently.
Check /var/log/messages for occurred prior to.
/var/log/messages contains these messages:
Sep 25 01:29:08 fileserver smartd[799]: Device: /dev/sdb [SAT], SMART Usage Attribute: 7 Seek_Error_Rate changed from 100 to 200
Sep 25 03:29:08 fileserver smartd[799]: Device: /dev/sda [SAT], SMART Usage Attribute: 7 Seek_Error_Rate changed from 100 to 200
Sep 25 18:29:08 fileserver smartd[799]: Device: /dev/sdb [SAT], SMART Usage Attribute: 9 Power_On_Hours changed from 87 to 86
Oct 1 01:20:49 fileserver smartd[801]: Device: /dev/sda [SAT], SMART Usage Attribute: 7 Seek_Error_Rate changed from 200 to 100
Oct 1 01:20:49 fileserver smartd[801]: Device: /dev/sdb [SAT], SMART Usage Attribute: 7 Seek_Error_Rate changed from 200 to 100
Is that bad?
Then there's entries like this which point to a corrupt database:
Sep 28 01:00:01 fileserver esmith::event[14688]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Sep 28 01:00:14 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'SHOW TRIGGERS LIKE 'tiki\_referer\_stats'': Can't read dir of './tikiwiki/' (errno: 24) (1018)
Sep 28 01:00:14 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'SHOW TRIGGERS LIKE 'galaxia\_activities'': Can't read dir of './tikiwiki/' (errno: 24) (1018)
Sep 28 01:00:14 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'SHOW TRIGGERS LIKE 'galaxia\_activity\_roles'': Can't read dir of './tikiwiki/' (errno: 24) (1018)
Sep 28 01:00:14 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'SHOW TRIGGERS LIKE 'galaxia\_instance\_activities'': Can't read dir of './tikiwiki/' (errno: 24) (1018)
Sep 28 01:00:14 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'SHOW TRIGGERS LIKE 'galaxia\_instance\_comments'': Can't read dir of './tikiwiki/' (errno: 24) (1018)
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'show fields from `tiki_related_categories`': Out of resources when opening file '/var/tmp/#sql_882_0.MAI' (Errcode: 24) (23)
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'SHOW TRIGGERS LIKE 'tiki\_related\_categories'': Out of resources when opening file '/var/tmp/#sql_882_0.MAI' (Errcode: 24)
(23)
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'show table status like 'tiki\_rss\_modules'': Can't read dir of './tikiwiki/' (errno: 24) (1018)
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Couldn't execute 'show create table `tiki_rss_modules`': Can't open file: './tikiwiki/tiki_rss_modules.frm' (errno: 24) (1016)
Sep 28 01:00:15 fileserver esmith::event[14688]: \nThere was an error trying to dump database tikiwiki, please check for table errors in this db. Forcing a backup of the corrupted DB. \nFailed to force
backup of corrupted db tikiwiki as tikiwiki-failed.dump
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Error: 'Out of resources when opening file '/var/tmp/#sql_882_2.MAI' (Errcode: 24)' when trying to dump tablespaces
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Error: 'Out of resources when opening file '/var/tmp/#sql_882_2.MAI' (Errcode: 24)' when trying to dump tablespaces
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Error: 'Out of resources when opening file '/var/tmp/#sql_882_2.MAI' (Errcode: 24)' when trying to dump tablespaces
Sep 28 01:00:15 fileserver esmith::event[14688]: mysqldump: Error: 'Out of resources when opening file '/var/tmp/#sql_882_2.MAI' (Errcode: 24)' when trying to dump tablespaces
Sep 28 01:00:15 fileserver esmith::event[14688]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1695834001 584707|End|1695834015 896282|Elapsed|14.311575
Also
journalctl -xe
# journalctl -xe
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 1046 has been created for the user pmulroney.
--
-- The leading process of the session is 20062.
Oct 01 06:38:59 fileserver.logicaldevelopments.com.au systemd[1]: Started Session 1046 of user pmulroney.
-- Subject: Unit session-1046.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-1046.scope has finished starting up.
--
-- The start-up result is done.
Oct 01 06:38:59 fileserver.logicaldevelopments.com.au sshd[20062]: pam_unix(sshd:session): session opened for user pmulroney by (uid=0)
Oct 01 06:38:59 fileserver.logicaldevelopments.com.au kernel: usb 1-1.6: new low-speed USB device number 51 using ehci-pci
Oct 01 06:38:59 fileserver.logicaldevelopments.com.au kernel: usb 1-1.6: device descriptor read/64, error -32
Oct 01 06:39:00 fileserver.logicaldevelopments.com.au kernel: usb 1-1.6: device descriptor read/64, error -32
Oct 01 06:39:00 fileserver.logicaldevelopments.com.au kernel: usb 1-1-port6: attempt power cycle
There are lots of the last 4 lines repeated over and over. I have a KVM that connects to three servers, and I get these errors a lot so I think it's background noise.
Get a list of installed contribs & template mods too.
# /sbin/e-smith/audittools/newrpms
Loaded plugins: fastestmirror, post-transaction-actions, priorities, smeserver
Loading mirror speeds from cached hostfile
* base: mirror.internode.on.net
* smeaddons: smeserver.de-labrusse.fr
* smeos: smeserver.de-labrusse.fr
* smeupdates: smeserver.de-labrusse.fr
* updates: mirror.internode.on.net
Extra Packages
GeoIP.x86_64 1.6.12-9.el7.sme @smecontribs
GeoIP-GeoLite-data.noarch 2018.06-7.el7.sme @smecontribs
GeoIP-GeoLite-data-extra.noarch 2018.06-7.el7.sme @smecontribs
php-php-gettext.noarch 1.0.12-1.el7 @smecontribs
php-tcpdf.noarch 6.2.26-1.el7 @smecontribs
php-tcpdf-dejavu-sans-fonts.noarch 6.2.26-1.el7 @smecontribs
phpMyAdmin.noarch 5.1.0-1.el7.sme @smecontribs
smeserver-git.noarch 1.2.0-11.el7.sme @smecontribs
smeserver-gitweb.noarch 1.1.0-14.el7.sme @smetest
smeserver-gitweb-theme.noarch 1.1.0-1.el7.sme @smetest
smeserver-mod_dav.noarch 1.1-8.el7.sme @smecontribs
smeserver-nfs.noarch 1.2.0-21.el7.sme @smecontribs
smeserver-phpmyadmin.noarch 4.0.10.2-13.el7.sme @smecontribs
smeserver-userpanel.noarch 1.4-6.el7.sme @smeupdates
smeserver-webhosting.noarch 0.0.9-16.el7.sme @smecontribs
smeserver-wordpress.noarch 1.2-12.el7.sme @smecontribs
smeserver-wsdd.noarch 0.2-5.el7.sme @smecontribs
wsdd.noarch 0.7.0-1.el7 @smecontribs
# /sbin/e-smith/audittools/templates
/etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf/45prune: MANUALLY_ADDED, OVERRIDE
It may require a db restore from backup.
Agreed. The tikiwiki database is a leftover from many years ago - we moved to dokuwiki for our intranet. Therefore this database could just be removed. I reckon I can restore horde-alarms from another SME server since I'm fairly sure it's just an empty database.