Topic: RAID failure: strange error at startup

[Jáder]

I have a RAID1 disk failing... no problem...more or less.
When I rebooted with just one disk I can see this on messages:

Code: [Select]

Dec  8 22:05:46 leopardo mysql.init: ERROR 1060 (42S21) at line 423: Duplicate column name 'Routine_type'
Dec  8 22:05:46 leopardo mysql.init: @hadGrantPriv:=1
Dec  8 22:05:46 leopardo mysql.init: 1
Dec  8 22:05:46 leopardo last message repeated 19 times
Dec  8 22:05:46 leopardo mysql.init: @hadShowDbPriv:=1
Dec  8 22:05:46 leopardo mysql.init: 1
Dec  8 22:05:46 leopardo last message repeated 19 times
Dec  8 22:05:46 leopardo mysql.init: @hadCreateViewPriv:=1
Dec  8 22:05:46 leopardo mysql.init: 1
Dec  8 22:05:46 leopardo last message repeated 19 times
Dec  8 22:05:46 leopardo mysql.init: @hadCreateRoutinePriv:=1
Dec  8 22:05:46 leopardo mysql.init: 1
Dec  8 22:05:46 leopardo last message repeated 19 times
Dec  8 22:05:46 leopardo mysql.init: @hadCreateUserPriv:=1
Dec  8 22:05:46 leopardo mysql.init: 1
Dec  8 22:05:46 leopardo last message repeated 19 times
Dec  8 22:05:46 leopardo mysql.init: waiting for mysqld to restart
Dec  8 22:06:06 leopardo last message repeated 19 times
Dec  8 22:06:07 leopardo mysql.init: mysqld failed to restart
Dec  8 22:06:07 leopardo mysql.init: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Dec  8 22:06:07 leopardo last message repeated 3 times
There are thousands of errors on messages.
And I can see FAILURE on last stage of startup : that Loading Fix_.... Loading...

Right now I disable as many services as possible from my server... to avoid second disk failure.
But I´m afraid I have another problem... should I start over on a new pair of HDDs ?

Regards,

Jáder

[janet]

jader

If I were you I would be running a disk diagnostic test on both drives eg download the UBCD (Ultimate boot CD) & boot up to that & run the drive brand manufacturers test on the drives.

Ascertain what is really wrong before trying to fix it.

If you have one drive failed & a second drive partly faulty, then you may want to consider copying all existing data from the faulty drive to another known good drive, to safeguard the data that is still on the drive eg using dd comand, (it will be slow).

Further advice depends on what you discover.

You could also boot up to the SME server CD/DVD, & mount the "good " drive to see what is on it.

[Jáder]

Janet

Server appears to be working ok on single drive (RAID1 broken).
Just can see that error on startup...no other side effects so far (thanks GOD!).

I´d like to understand why that error happens and how can I trace it to source of problem.
I´m not sure that error wasn´t there before... this server run headless for years.

I´m looking for a new HDD to replace old one and starting backup things as space permits.

Thank you by your tips.

Jáder

[Stefano]

jader, first of all buy a new disk and add it to the array..

then, take the last good backup, and try to restore it on a test machine.. you'll see if there's something wrong/corrupt..

[TerryF]

Some reading for you to do:
http://forums.contribs.org/index.php?topic=48013.0
http://bugs.contribs.org/show_bug.cgi?id=5486

Not all the answers but a start on the path.

[CharlieBrady]

When I rebooted with just one disk I can see this on messages:

You have at least two problems here. One is that you have a file in /etc/e-smith/sql/init, which indicates a failed script the last time you did a system upgrade. The second is that mysql is not starting up.

[CharlieBrady]

Server appears to be working ok on single drive (RAID1 broken).

But which disk did it boot from/is it working from? Sometimes people don't notice when RAID1 is broken, and they don't discover the problem until the second drive dies, and they end up booting from a very out of date system on the first drive.

[Jáder]

Charlie,

Disk1 (the bad one) is disconected. Linux do not startup with it connected (kernel Panic).

Jáder

[Jáder]

You have at least two problems here. One is that you have a file in /etc/e-smith/sql/init, which indicates a failed script the last time you did a system upgrade. The second is that mysql is not starting up.

Hum... so what I can do now?
I'm sure Mysql is running:

Code: [Select]

[root@leopardo ~]# /etc/init.d/mysqld status
mysqld (pid 19602) is running...
[root@leopardo ~]#

Can I just clean up some dir to skip this problem or should I revert to back stage and debug the scripts problem ?
Remember, this is a headless server, I must upgraded it hundred times without watch console screen during startup. :$
And I never watch for errors anywhere :$ if nothing force me to do it! :$:$

[CharlieBrady]

Can I just clean up some dir to skip this problem or should I revert to back stage and debug the scripts problem ?

See what you see when you do:

/etc/rc.d/init.d/mysql.init start

You can delete files in /etc/e-smith/sql/init if you are certain that the errors aren't really an issue.

But why are you seeing this:

Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

?

[CharlieBrady]

Have a look at this too:

http://bugs.contribs.org/show_bug.cgi?id=7467

[Jáder]

See what you see when you do:

/etc/rc.d/init.d/mysql.init start

It's these:
[root@leopardo ~]# /etc/rc.d/init.d/mysql.init start
Loading fix_privilege_tables into mysql                    [FAILED]
Loading mysql_migrate_horde into mysql                     [  OK  ]
Loading horde.mysql_set_password into mysql                [  OK  ]
Loading horde_mysql_create_tables into mysql               [  OK  ]
Loading sogo_mysql_create_database into mysql              [  OK  ]
Loading horde_upgrade into mysql                           [  OK  ]
Loading sogo_mysql_update_privileges into mysql            [  OK  ]
Loading sogo_upgrade into mysql                            [  OK  ]
Loading imp_mysql_create_tables into mysql                 [  OK  ]
Loading imp_upgrade into mysql                             [  OK  ]
Loading mysql.create.turba into mysql                      [  OK  ]
Loading turba_upgrade into mysql                           [  OK  ]
Loading turba_mysql_reset_addressbook_pref into mysql      [  OK  ]
Loading horde_mysql_update_privs into mysql                [  OK  ]
Loading mysql.create.ingo into mysql                       [  OK  ]
Loading ingo_upgrade into mysql                            [  OK  ]
Loading zabbix-server into mysql                           [FAILED]
Loading pydio into mysql                                   [  OK  ]

You can delete files in /etc/e-smith/sql/init if you are certain that the errors aren't really an issue.

Hum... I'll move the to another dir... just for be at safe side... but most of them went imported!
 

But why are you seeing this:

Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

That's the 1 million dollar question!
I'll restart server later to find out if this message still shows!

?
[/quote]

for now I:

Code: [Select]

[root@leopardo ~]# ls -la  /etc/e-smith/sql/init
total 16
drwxr-xr-x 2 root root 4096 Dec 10 12:55 .
drwxr-xr-x 3 root root 4096 Apr 27  2014 ..
-r-xr----- 1 root root  348 Nov  1  2014 10fix_privilege_tables
-rwxr-x--- 1 root root 1832 Dec  8 22:01 80zabbix-server
[root@leopardo ~]#
So I'm in right direction!!

[CharlieBrady]

That's the 1 million dollar question!
I'll restart server later to find out if this message still shows!

Don't do that. Investigate if mysqld is running, and if not, why not, and if so, then why mysql.init says that it cannot connect.

[Jáder]

Hi Charlie,

So let's debug!
If I do:

Code: [Select]

tail -100 /var/log/mysqld.log

I can see this:

Code: [Select]

140519 11:29:31  InnoDB: Error: cannot allocate 3145744384 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 287552336 bytes. Operating system errno: 12

So I search for Mysql.cnf file and found this:

Code: [Select]

[mysqld]
pid-file=/var/run/mysqld/mysqld.pid
basedir=/usr
datadir=/var/lib/mysql
innodb_data_home_dir = /var/lib/mysql/
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /var/lib/mysql/
innodb_log_arch_dir = /var/lib/mysql/
innodb_buffer_pool_size = 3000M
innodb_additional_mem_pool_size = 256M
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

socket=/var/lib/mysql/mysql.sock
# networking is enabled
max_allowed_packet=16M
user=mysql


[mysqld_safe]

And I have 8 GB on this machine:

Code: [Select]

[root@leopardo ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          8085       3325       4760          0        447       1478
-/+ buffers/cache:       1398       6687
Swap:         5503          0       5503
[root@leopardo ~]#

So I'm thinking about  double the "innodb_additional_mem_pool_size = 256M", restart mysql , and run mysql.init
What do you think ?

Regards,

Jáder

EDIT1: I found that my.cnf is customized by this discussion (http://forums.contribs.org/index.php?topic=50268.0) with Daniel B. about Zabbix.
The error message say about cannot allocate 3GB ... even this settings are effective from octobre 2013!
I've tryed to change: innodb_buffer_pool_size = 3000M
to lower values... till 1000M and nothing changed.
After change values I restart mysql (/etc/init.d/mysqld restart) and try to run mysql.init start

[CharlieBrady]

After change values I restart mysql (/etc/init.d/mysqld restart) ...

Don't do it that way. Do:

sv restart /service/mysqld

or

service mysqld restart