Topic: webserver stopped working

[firedrake55]

My webserver suddenly stopped working.  There have been no changes to it, I just tried to access the webpage and it didnt respond.  I started with a reboot but it didnt help.  I can log into the system without any issues and I can still browse ibays and user folders through windows explorer.  As far as i can tell, the email server is working fine, just not webmail since that uses the webserver.  I tried ps aux and it looks as though httpd is running.  I had a similar problem about a year ago that I was able to work through, but I dont remember exactly what I did.  I recall just deleting a few files and doing a reboot and everything being ok.  I know this is a vague post, but Im not sure what I should be looking for.  If anyone has had this problem or if you need some more info let me know. Thanks.

[Stefano]

hi

starting from /var/log/messages, /var/log/httpd/error_log and /var/log/httpd/admin_error_log could be a good start

[firedrake55]

ok, looking the logs I was able to figure out what the problem is.  Im not not sure what caused it or how to fix it.  It seems that the server is upset because my "RSA server certificate CommonName does not match my server name".  I got by this by just renaming the system, but I dont want a different system name.  This is very reminiscent of the problem I had last year.  The server was upset about my IP address.  I was able to just remove a few files to fix this but I dont recall how I tracked them down.  Im wondering if this is similar and I can just delete those files again?  Any help?  Here are the logs for those interested....


MESSAGES:

Oct 14 17:26:51 www kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Down
Oct 14 17:26:52 www kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
Oct 15 01:12:03 www esmith::event[24073]: Processing event: logrotate 
Oct 15 01:12:03 www esmith::event[24073]: Running event handler: /etc/e-smith/events/actions/generic_template_expand
Oct 15 01:12:03 www esmith::event[24073]: expanding /etc/mime.types 
Oct 15 01:12:04 www esmith::event[24073]: expanding /etc/crontab 
Oct 15 01:12:04 www esmith::event[24073]: expanding /etc/syslog.conf 
Oct 15 01:12:04 www esmith::event[24073]: expanding /etc/e-smith/web/common/foot.tmpl 
Oct 15 01:12:04 www esmith::event[24073]: expanding /etc/e-smith/web/common/head.tmpl 
Oct 15 01:12:04 www esmith::event[24073]: expanding /etc/e-smith/web/panels/manager/html/header.htm 
Oct 15 01:12:05 www esmith::event[24073]: expanding /etc/logrotate.d/syslog 
Oct 15 01:12:05 www esmith::event[24073]: expanding /etc/sysconfig/syslog 
Oct 15 01:12:05 www esmith::event[24073]: expanding /etc/httpd/admin-conf/httpd.conf 
Oct 15 01:12:05 www esmith::event[24073]: generic_template_expand=action|Event|logrotate|Action|generic_template_expand|Start|1255583523 278467|End|1255583525 444157|Elapsed|2.16569
Oct 15 01:12:05 www esmith::event[24073]: Running event handler: /etc/e-smith/events/logrotate/S75purge-old-logs
Oct 15 01:12:05 www esmith::event[24073]: S75purge-old-logs=action|Event|logrotate|Action|S75purge-old-logs|Start|1255583525 444662|End|1255583525 812389|Elapsed|0.367727
Oct 15 01:12:05 www esmith::event[24073]: Running event handler: /etc/e-smith/events/actions/adjust-services
Oct 15 01:12:06 www esmith::event[24073]: adjusting supervised httpd-e-smith (sigusr1) 
Oct 15 01:12:06 www esmith::event[24073]: adjusting supervised httpd-e-smith (up) 
Oct 15 01:12:06 www esmith::event[24073]: adjusting supervised httpd-admin (sigusr1) 
Oct 15 01:12:06 www esmith::event[24073]: adjusting supervised httpd-admin (up) 
Oct 15 01:12:06 www esmith::event[24073]: adjusting non-supervised syslog (reload) 
Oct 15 01:12:07 www kernel: Kernel logging (proc) stopped.
Oct 15 01:12:07 www kernel: Kernel log daemon terminating.
Oct 15 01:12:09 www syslog: klogd shutdown succeeded
Oct 15 01:12:11 www exiting on signal 15

new MESSAGES file:

Oct 15 01:12:15 www syslogd 1.4.1: restart.
Oct 15 01:12:15 www syslog: syslogd startup succeeded
Oct 15 01:12:15 www esmith::event[24073]: Starting system logger: [  OK  ]
 
Oct 15 01:12:15 www kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 15 01:12:15 www syslog: klogd startup succeeded
Oct 15 01:12:16 www esmith::event[24073]: Starting kernel logger: [  OK  ]
 
Oct 15 01:12:16 www esmith::event[24073]: adjust-services=action|Event|logrotate|Action|adjust-services|Start|1255583525 812950|End|1255583535 551698|Elapsed|9.738748
Oct 15 01:12:15 www kernel: Inspecting /boot/System.map-2.6.9-42.0.3.ELsmp
Oct 15 01:12:16 www kernel: Loaded 24512 symbols from /boot/System.map-2.6.9-42.0.3.ELsmp.
Oct 15 01:12:16 www kernel: Symbols match kernel version 2.6.9.
Oct 15 01:12:16 www kernel: No module symbols loaded - kernel modules not enabled.
Oct 15 01:12:14 www syslog: syslogd shutdown succeeded
Oct 15 06:25:49 www kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Down
Oct 15 06:25:51 www kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
Oct 15 19:07:16 www shutdown: shutting down for system halt

at this point I was already aware of the problem and took the system down.

ADMIN_ERROR_LOG:

[Thu Oct 15 01:12:06 2009] [notice] Graceful restart requested, doing restart

and in a new ADMIN_ERROR_LOG file:

[Thu Oct 15 01:12:16 2009] [notice] Digest: generating secret for digest authentication ...
[Thu Oct 15 01:12:16 2009] [notice] Digest: done
[Thu Oct 15 01:12:17 2009] [notice] Apache configured -- resuming normal operations
[Thu Oct 15 19:56:10 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Oct 15 19:56:11 2009] [notice] Digest: generating secret for digest authentication ...
[Thu Oct 15 19:56:11 2009] [notice] Digest: done
[Thu Oct 15 19:56:11 2009] [notice] Apache configured -- resuming normal operations

took the system down and didnt get to it again until the 18th and then again the 24th:

[Sun Oct 18 17:38:24 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Sun Oct 18 17:38:24 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Oct 18 17:38:24 2009] [notice] Digest: done
[Sun Oct 18 17:38:25 2009] [notice] Apache configured -- resuming normal operations
[Sun Oct 18 17:42:04 2009] [notice] caught SIGTERM, shutting down
[Sat Oct 24 11:53:09 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Sat Oct 24 11:53:09 2009] [notice] Digest: generating secret for digest authentication ...
[Sat Oct 24 11:53:09 2009] [notice] Digest: done
[Sat Oct 24 11:53:10 2009] [notice] Apache configured -- resuming normal operations
[Sat Oct 24 12:01:03 2009] [notice] caught SIGTERM, shutting down
[Sat Oct 24 12:03:10 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Sat Oct 24 12:03:09 2009] [notice] Digest: generating secret for digest authentication ...
[Sat Oct 24 12:03:09 2009] [notice] Digest: done
[Sat Oct 24 12:03:10 2009] [notice] Apache configured -- resuming normal operations
[Sat Oct 24 13:48:00 2009] [notice] caught SIGTERM, shutting down

ERROR_LOG

[Tue Oct 13 16:49:48 2009] [error] [client 208.43.133.130] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 16:52:04 2009] [error] [client 74.63.192.66] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 16:55:30 2009] [error] [client 208.43.133.130] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 17:03:04 2009] [error] [client 74.63.192.66] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 17:07:27 2009] [error] [client 208.43.133.130] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 17:08:27 2009] [error] [client 74.63.192.66] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 17:26:15 2009] [error] [client 74.63.192.66] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 17:26:41 2009] [error] [client 208.43.133.130] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 17:34:14 2009] [error] [client 208.43.133.130] File does not exist: /home/e-smith/files/ibays/Primary/html/intl
[Tue Oct 13 19:07:17 2009] [error] [client 65.55.105.149] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[Tue Oct 13 19:09:12 2009] [error] [client 65.55.105.149] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[client 61.160.216.63] script '/home/e-smith/files/ibays/Primary/html/prx2.php' not found or unable to stat
[client 221.195.73.68] script '/home/e-smith/files/ibays/Primary/html/prx2.php' not found or unable to stat
[client 221.195.73.68] script '/home/e-smith/files/ibays/Primary/html/prx2.php' not found or unable to stat
[Wed Oct 14 06:56:22 2009] [error] [client 65.55.105.149] File does not exist: /home/e-smith/files/ibays/Primary/files/VPN/VPN_terms.pdf
[Wed Oct 14 10:15:53 2009] [error] [client 65.55.105.146] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[Wed Oct 14 13:24:12 2009] [error] [client 65.55.105.148] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[Wed Oct 14 13:25:22 2009] [error] [client 65.55.105.148] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[Wed Oct 14 14:56:40 2009] [error] [client 65.55.105.136] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[Wed Oct 14 16:12:31 2009] [error] [client 66.249.71.53] File does not exist: /home/e-smith/files/ibays/Primary/html/robots.txt
[client 61.160.216.63] script '/home/e-smith/files/ibays/Primary/html/prx2.php' not found or unable to stat
[client 61.160.216.63] script '/home/e-smith/files/ibays/Primary/html/prx2.php' not found or unable to stat
[Thu Oct 15 01:12:06 2009] [notice] Graceful restart requested, doing restart

new ERROR_LOG file:

[Thu Oct 15 01:12:18 2009] [notice] Digest: generating secret for digest authentication ...
[Thu Oct 15 01:12:18 2009] [notice] Digest: done
[Thu Oct 15 01:12:19 2009] [warn] RSA server certificate CommonName (CN) `www.xxx.net' does NOT match server name!?
[Thu Oct 15 01:12:19 2009] [error] Unable to configure RSA server private key
[Thu Oct 15 01:12:19 2009] [error] SSL Library Error: 185073780 error:0B080074:x509 certificate routines:func(128):reason(116)
[Thu Oct 15 01:12:20 2009] [warn] RSA server certificate CommonName (CN) `www.xxx.net' does NOT match server name!?
[Thu Oct 15 01:12:20 2009] [error] Unable to configure RSA server private key
[Thu Oct 15 01:12:20 2009] [error] SSL Library Error: 185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch
[Thu Oct 15 01:12:21 2009] [warn] RSA server certificate CommonName (CN) `www.xxx.net' does NOT match server name!?
[Thu Oct 15 01:12:21 2009] [error] Unable to configure RSA server private key
[Thu Oct 15 01:12:21 2009] [error] SSL Library Error: 185073780 error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch

This message continues every 1-3 sec until:

[Thu Oct 15 01:15:22 2009] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
[Thu Oct 15 01:15:23 2009] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
[Thu Oct 15 01:15:24 2009] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

This error message continues indefinetly every sec.

[mmccarn]

[Thu Oct 15 01:15:22 2009] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
[Thu Oct 15 01:15:23 2009] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
[Thu Oct 15 01:15:24 2009] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

See if your disk has filled up using

Code: [Select]

df -h
The important line always wraps onto two lines (at least on my system).  Check to make sure that the root filesystem has space available and that "Use%" is not "100%":

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                       36G  6.8G   27G  21% /
/dev/md1               99M   26M   69M  27% /boot
none                  248M     0  248M   0% /dev/shm

[firedrake55]

Only have about 60% in use.  Still plenty of room.

Thanks for the idea though.

[Stefano]

please search the forums and the wiki.. I'm sure it's a issue we already faced in the past

[mmccarn]

Here's an old post from when I had this problem: http://forums.contribs.org/index.php/topic,33171.msg144080.html#msg144080 (Here are the instructions I used: http://forums.contribs.org/index.php/topic,33171.msg141328.html#msg141328)

[CharlieBrady]

Your problem is due to your SSL key not matching the SSL certificate. Apache/openssl then make the matter worse by having an IPC memory leak.

Please open a bug report in the bug tracker, and show the output of:

ls -lt /home/e-smith/ssl/*