Topic: Last two updates causing this? (solved)

[tviles]

I can putty into server. I can VPN into server. Users cannot get out on the internet, test internet functions fails at the su admin panel. Users cannot get into the ibays.
===
=== yum reports available updates:
===

mysql.i386                               4.1.22-2.el4_8.4       updates         
mysql-server.i386                        4.1.22-2.el4_8.4       updates         



View log files
   /var/log/boot.log: Viewed at Sat 06 Nov 2010 09:08:56 PM CDT.
Nov  6 18:29:34 shpdserver syslog: syslogd startup succeeded
Nov  6 18:29:34 shpdserver syslog: klogd startup succeeded
Nov  6 18:29:04 shpdserver rc.sysinit: -e
Nov  6 18:29:04 shpdserver date: Sat Nov  6 18:29:04 CDT 2010
Nov  6 18:29:04 shpdserver rc.sysinit: Setting clock  (utc): Sat Nov  6 18:29:04 CDT 2010 succeeded
Nov  6 18:29:06 shpdserver start_udev: Starting udev:  succeeded
Nov  6 18:29:21 shpdserver rc.sysinit: -e
Nov  6 18:29:22 shpdserver sysctl: kernel.sysrq = 0
Nov  6 18:29:22 shpdserver sysctl: net.ipv4.conf.all.rp_filter = 1
Nov  6 18:29:22 shpdserver sysctl: net.ipv4.conf.default.rp_filter = 1
Nov  6 18:29:22 shpdserver sysctl: net.ipv4.ip_dynaddr = 1
Nov  6 18:29:22 shpdserver sysctl: net.ipv4.ip_forward = 1
Nov  6 18:29:22 shpdserver sysctl: net.ipv4.tcp_keepalive_time = 300
Nov  6 18:29:22 shpdserver sysctl: net.ipv4.tcp_syncookies = 1
Nov  6 18:29:22 shpdserver rc.sysinit: Configuring kernel parameters:  succeeded
Nov  6 18:29:22 shpdserver rc.sysinit: Loading default keymap succeeded
Nov  6 18:29:22 shpdserver rc.sysinit: Setting hostname shpdserver:  succeeded
Nov  6 18:29:23 shpdserver fsck: /dev/main/root: clean, 79103/17465344 files, 34873634/34930688 blocks
Nov  6 18:29:23 shpdserver rc.sysinit: Checking root filesystem succeeded
Nov  6 18:29:23 shpdserver rc.sysinit: Remounting root filesystem in read-write mode:  succeeded
Nov  6 18:29:26 shpdserver lvm.static:   2 logical volume(s) in volume group main now active
Nov  6 18:29:26 shpdserver rc.sysinit: Setting up Logical Volume Management: succeeded
Nov  6 18:29:28 shpdserver lvm.static:   2 logical volume(s) in volume group main now active
Nov  6 18:29:28 shpdserver rc.sysinit: Setting up Logical Volume Management: succeeded
Nov  6 18:29:28 shpdserver fsck: /dev/md1: clean, 54/26104 files, 27114/104320 blocks
Nov  6 18:29:28 shpdserver rc.sysinit: Checking filesystems succeeded
Nov  6 18:29:28 shpdserver rc.sysinit: Mounting local filesystems:  succeeded
Nov  6 18:29:28 shpdserver rc.sysinit: Enabling local filesystem quotas:  succeeded
Nov  6 18:29:29 shpdserver rc.sysinit: Enabling swap space:  succeeded
Nov  6 18:29:34 shpdserver microcode_ctl: microcode_ctl startup succeeded
Nov  6 18:31:15 shpdserver syslog: klogd shutdown succeeded
Nov  6 18:31:19 shpdserver syslog: syslogd startup succeeded
Nov  6 18:31:19 shpdserver syslog: klogd startup succeeded
Nov  6 18:31:19 shpdserver raidmonitor: Starting raidmonitor succeeded
Nov  6 18:31:19 shpdserver syslog: syslogd shutdown succeeded
Nov  6 18:31:22 shpdserver sysctl: kernel.sysrq = 0
Nov  6 18:31:22 shpdserver sysctl: net.ipv4.conf.all.rp_filter = 1
Nov  6 18:31:22 shpdserver sysctl: net.ipv4.conf.default.rp_filter = 1
Nov  6 18:31:22 shpdserver sysctl: net.ipv4.ip_dynaddr = 1
Nov  6 18:31:22 shpdserver sysctl: net.ipv4.ip_forward = 1
Nov  6 18:31:22 shpdserver sysctl: net.ipv4.tcp_keepalive_time = 300
Nov  6 18:31:22 shpdserver sysctl: net.ipv4.tcp_syncookies = 1
Nov  6 18:31:22 shpdserver network: Setting network parameters:  succeeded
Nov  6 18:31:22 shpdserver network: Bringing up loopback interface:  succeeded
Nov  6 18:31:27 shpdserver network: Bringing up interface eth0:  succeeded
Nov  6 18:31:31 shpdserver network: Bringing up interface eth1:  succeeded
Nov  6 18:31:31 shpdserver wan: Starting wan succeeded
Nov  6 18:31:32 shpdserver irqbalance: irqbalance startup succeeded
Nov  6 18:31:32 shpdserver crond: crond startup succeeded
Nov  6 18:31:32 shpdserver acpid: acpid startup succeeded
Nov  6 18:31:32 shpdserver mysqld: Starting mysqld succeeded
Nov  6 18:31:32 shpdserver dnscache: Starting dnscache succeeded
Nov  6 18:31:33 shpdserver imap: Starting imap succeeded
Nov  6 18:31:33 shpdserver imaps: Starting imaps succeeded
Nov  6 18:31:33 shpdserver pop3: Starting pop3 succeeded
Nov  6 18:31:33 shpdserver pop3s: Starting pop3s succeeded
Nov  6 18:31:33 shpdserver tinydns: Starting tinydns succeeded
Nov  6 18:31:34 shpdserver lpd: Starting lpd succeeded
Nov  6 18:31:34 shpdserver dhcpd: Starting dhcpd succeeded
Nov  6 18:31:34 shpdserver clamd: Starting clamd succeeded
Nov  6 18:31:34 shpdserver freshclam: Starting freshclam succeeded
Nov  6 18:31:35 shpdserver ldap: Starting ldap succeeded
Nov  6 18:31:35 shpdserver ntpd: Starting ntpd succeeded
Nov  6 18:31:35 shpdserver qmail: Starting qmail succeeded
Nov  6 18:31:35 shpdserver qpsmtpd: Starting qpsmtpd succeeded
Nov  6 18:31:36 shpdserver sqpsmtpd: Starting sqpsmtpd succeeded
Nov  6 18:31:36 shpdserver sshd: Starting sshd succeeded
Nov  6 18:31:37 shpdserver httpd-admin: Starting httpd-admin succeeded
Nov  6 18:31:37 shpdserver httpd-e-smith: Starting httpd-e-smith succeeded
Nov  6 18:31:37 shpdserver pptpd: Starting pptpd succeeded
Nov  6 18:31:38 shpdserver radiusd: Starting radiusd succeeded
Nov  6 18:31:38 shpdserver smolt:  succeeded
Nov  6 18:31:38 shpdserver spamd: Starting spamd succeeded
Nov  6 18:31:39 shpdserver squid: Starting squid succeeded
Nov  6 18:31:39 shpdserver nmbd: Starting nmbd succeeded
Nov  6 18:31:39 shpdserver smbd: Starting smbd succeeded
Nov  6 18:31:40 shpdserver messagebus: messagebus startup succeeded
Nov  6 18:31:41 shpdserver haldaemon: haldaemon startup succeeded
Nov  6 18:31:41 shpdserver 10fix_privilege_tables: ERROR
Nov  6 18:31:41 shpdserver 10fix_privilege_tables:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:42 shpdserver 10fix_privilege_tables: kill: usage: kill [-s sigspec | -n signum | -sigspec] [pid | job]... or kill -l [sigspec]
Nov  6 18:31:42 shpdserver 10fix_privilege_tables: waiting for mysqld to restart
Nov  6 18:31:43 shpdserver mysql.init: Loading 10fix_privilege_tables into mysql succeeded
Nov  6 18:31:43 shpdserver 20mysql_migrate_horde: ERROR
Nov  6 18:31:43 shpdserver 20mysql_migrate_horde:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:43 shpdserver mysql.init: Loading 20mysql_migrate_horde into mysql failed
Nov  6 18:31:43 shpdserver 21horde.mysql_set_password: ERROR
Nov  6 18:31:43 shpdserver 21horde.mysql_set_password:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:43 shpdserver mysql.init: Loading 21horde.mysql_set_password into mysql failed
Nov  6 18:31:43 shpdserver 30horde_mysql_create_tables: ERROR
Nov  6 18:31:43 shpdserver 30horde_mysql_create_tables:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:43 shpdserver mysql.init: Loading 30horde_mysql_create_tables into mysql failed
Nov  6 18:31:43 shpdserver 31horde_upgrade: DBI connect('horde','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /etc/e-smith/events/actions/horde_upgrade line 41
Nov  6 18:31:43 shpdserver mysql.init: Loading 31horde_upgrade into mysql failed
Nov  6 18:31:43 shpdserver 35imp_mysql_create_tables: ERROR
Nov  6 18:31:43 shpdserver 35imp_mysql_create_tables:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:43 shpdserver mysql.init: Loading 35imp_mysql_create_tables into mysql failed
Nov  6 18:31:44 shpdserver 36imp_upgrade: DBI connect('horde','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /etc/e-smith/events/actions/imp_upgrade line 41
Nov  6 18:31:44 shpdserver mysql.init: Loading 36imp_upgrade into mysql failed
Nov  6 18:31:44 shpdserver 40mysql.create.turba: ERROR
Nov  6 18:31:44 shpdserver 40mysql.create.turba:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:44 shpdserver mysql.init: Loading 40mysql.create.turba into mysql failed
Nov  6 18:31:44 shpdserver 50turba_upgrade: DBI connect('horde','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /etc/e-smith/events/actions/turba_upgrade line 44
Nov  6 18:31:44 shpdserver mysql.init: Loading 50turba_upgrade into mysql failed
Nov  6 18:31:44 shpdserver 57turba_mysql_reset_addressbook_pref: ERROR
Nov  6 18:31:44 shpdserver 57turba_mysql_reset_addressbook_pref:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:44 shpdserver mysql.init: Loading 57turba_mysql_reset_addressbook_pref into mysql failed
Nov  6 18:31:45 shpdserver mysql.init: Loading 60migrate-imp-to-turba into mysql succeeded
Nov  6 18:31:45 shpdserver 77horde_mysql_update_privs: ERROR
Nov  6 18:31:45 shpdserver 77horde_mysql_update_privs:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:45 shpdserver mysql.init: Loading 77horde_mysql_update_privs into mysql failed
Nov  6 18:31:45 shpdserver 78mysql.create.ingo: ERROR
Nov  6 18:31:45 shpdserver 78mysql.create.ingo:  2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
Nov  6 18:31:45 shpdserver mysql.init: Loading 78mysql.create.ingo into mysql failed
Nov  6 18:31:46 shpdserver 79ingo_upgrade: DBI connect('horde','root',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) at /etc/e-smith/events/actions/ingo_upgrade line 41
Nov  6 18:31:46 shpdserver mysql.init: Loading 79ingo_upgrade into mysql failed

[cactus]

I guess there is something wrong, but MySQL should be unrelated to the fact that the internet test is failing or that users can not access your ibays. You should investigate further what is causing your network issue as well as what is causing MySQL not to start.
Have a  look at /var/log/mysqld.log to find out what is happening with MySQL.

[tviles]

I am up working on this and have found the error stating no space to write file. My /dev/mapper/main-root is full due to me starting up a backup is the only thing I can think of. I have gone into server-manager and removed backup job and am waiting for server to reboot. It appears the backup job is written to hard drive first before writing it to usbdrive?

[cactus]

I am up working on this and have found the error stating no space to write file. My /dev/mapper/main-root is full due to me starting up a backup is the only thing I can think of. I have gone into server-manager and removed backup job and am waiting for server to reboot. It appears the backup job is written to hard drive first before writing it to usbdrive?

Since you do not provide details on the backup it is hard to judge if it is a configuration issue. If you are using the SME Server backup methods and you configured them properly the backup should be written to the location you specified and not to the local disks of the server IIRC.

[tviles]

Disk usage
     

Sun Nov 7 04:46:30 CST 2010


Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                      132G  131G     0 100% /
/dev/md1               99M   24M   71M  25% /boot
none                 1014M     0 1014M   0% /dev/shm
/dev/sda1             917G  149G  722G  18% /home/e-smith/files/ibays/police/files
 

[tviles]

Can I remove these to help free help some space? They are in /etc/dar2

[root@shpdserver dar2]# dir -l
total 8
-rw-r--r--  1 root root 803 Nov  6 05:03 11-2010pdserver
-rw-r--r--  1 root root 951 Nov  4 00:37 pdserver2

[tviles]

101106 18:05:56 [Note] /usr/libexec/mysqld: Shutdown complete

101106 18:11:06 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:06 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:07 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:07 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:08 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:08 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:10 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:10 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:11 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:11 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:12 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:12 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:14 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:14 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:15 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101106 18:11:15 [ERROR] Can't start server: can't create PID file: No space left on device
101106 18:11:16 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)

[janet]

tviles

-rw-r--r--  1 root root 803 Nov  6 05:03 11-2010pdserver
-rw-r--r--  1 root root 951 Nov  4 00:37 pdserver2

They are just the dar2 backup job config files and deleting them will not gain you any significant space.

Look in /mnt and /media & subfolders
I suspect you may have backup files saved there inadvertantly if the attached USB backup drive was not mounted correctly.
If so delete them as it appears most of your problems are related to the main-root partition being full.

Then sort out why your USB drive is not mounting properly.

[tviles]

hill is an ibay looks like I need to do a rm filename is the dar2 dir?

Last login: Sun Nov  7 05:05:47 2010 from 173.159.204.92
[root@shpdserver ~]# cd /mnt
[root@shpdserver mnt]# dir
dar2  floppy  hill  smb
[root@shpdserver mnt]# cd dar2
[root@shpdserver dar2]# dir
11-2010pdserver  pdserver2
[root@shpdserver dar2]# dir -l
total 8
drwxrwxrwx  2 root admin 4096 Nov  4 15:07 11-2010pdserver
drwxrwxrwx  2 root admin 4096 Nov  4 00:30 pdserver2
[root@shpdserver dar2]# cd media
-bash: cd: media: No such file or directory
[root@shpdserver dar2]# cd /media
[root@shpdserver media]# dir
cdrom  floppy  usbdisk  usbdisk1
[root@shpdserver media]# cd usbdisk
[root@shpdserver usbdisk]# dir
[root@shpdserver usbdisk]# cd ..
[root@shpdserver media]# cd usbdisk1
[root@shpdserver usbdisk1]# dir
[root@shpdserver usbdisk1]# cd ..
[root@shpdserver media]# dir
cdrom  floppy  usbdisk  usbdisk1
[root@shpdserver media]#

[tviles]

Nothing there.

Is there somewhere else I can look?

Last login: Sun Nov  7 06:07:50 2010 from 173.159.204.92
[root@shpdserver ~]# cd /media
[root@shpdserver media]# dir
cdrom  floppy  usbdisk  usbdisk1
[root@shpdserver media]# cd usbdisk
[root@shpdserver usbdisk]# dir
[root@shpdserver usbdisk]# cd ..
[root@shpdserver media]# cd usbdisk1
[root@shpdserver usbdisk1]# dir
[root@shpdserver usbdisk1]# cd /mnt
[root@shpdserver mnt]# dir
dar2  floppy  hill  smb
[root@shpdserver mnt]# cd dar2
[root@shpdserver dar2]# dir
11-2010pdserver  pdserver2
[root@shpdserver dar2]# cd pdserver2
[root@shpdserver pdserver2]# dir
[root@shpdserver pdserver2]# cd ..
[root@shpdserver dar2]# cd 11-2010pdserver
[root@shpdserver 11-2010pdserver]# dir
[root@shpdserver 11-2010pdserver]#

[janet]

tviles

Try looking in /tmp
I have seen mount points created there and backup files saved there when USB drives are not mounted correctly.

[janet]

tviles

Also use du to find where the space usage is
cd /
du --max-depth 1

then say
cd /home
or
cd /tmp
then do
du --max-depth 1
du --max-depth 2
du --max-depth 3
and so on until you find where the most usage of space is

[tviles]

Found it.
A backup with no specified location will end up here.

The dir of shpwserver.shpw.local had the DAR files in it.

Same thing showed up on the shpdserver.

Thank you for your help.
They do end up on the usbdisk though after writing first to the hard drive. I will be more careful with next attempt of backup to the usbdrive. I have some reading to do first.



[root@shpwserver /]# dir
?             command  lib         package  service                tmp
aquota.group  dev      lost+found  proc     shpwserver.shpw.local  usr
aquota.user   etc      media       root     srv                    var
bin           home     mnt         sbin     sys
boot          initrd   opt         selinux  temp
[root@shpwserver /]#

[janet]

tviles

A backup with no specified location will end up here.
They do end up on the usbdisk though after writing first to the hard drive. I will be more careful with next attempt of backup to the usbdrive. I have some reading to do first.

The temporary copy AFAIK is copied to the backup disk in a /tmp/xxxx folder and then when the backup is complete the archive is moved to the final location on the backup disk.
As you state, if not setup correctly the temp copy may not be saved to the /tmp folder on the USB disk but to the /tmp folder of the local disk.
It is really a user setting error.

[tviles]

It is really a user setting error

Yes my bad.

I was just glad to have this forum to help me out. Having cops call you out at in the middle of the night stating the server is down is no fun.