Topic: httpd-admin is broken, how can I fix SME's guts?

[FreakWent]

On SME Server release 7.5.1, I'm having a lot of trouble after an fsck that made changes.

https://server/server-manager gives a 500, see:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

It persisted despite a reboot.

Now *usually* we look in /var/log/httpd-admin/error.log, but all the files there are 0 bytes, probably from the fsck.  After some other messing about, in desperation I deleted all the 0byte files, and restarted the services.

Now I have current, lock and state, all 0 bytes.

So what's happening? So we bump up LogLevel to debug and log to /tmp/templog and we see:

[Wed Jun 08 17:48:58 2011] [error] [client 127.0.0.1] fileparse(): need a valid pathname at /usr/lib/perl5/vendor_perl/5.8.5/CGI/Persistent.pm line 30
[Wed Jun 08 17:48:58 2011] [error] [client 127.0.0.1] Premature end of script headers: index.cgi

Wow, *information* at last! Checking /usr/lib/perl5/vendor_perl/5.8.5/CGI/Persistent.pm, lines 28-30:

  unless ( $fn ) {
        my $po = new Persistence::Object::Simple ( __Dope => $dope );
        $fn = fileparse $po->{ __Fn };

Oh God, so what's dope? Is it straight from the source?

Anyway....

So something's messed up deep(?) in my server.

How can I rebuild the core parts of SME server without losing i-bays or user email or anything else I love?

Should I try and upgrade my (broken?) server to 8?  What could possibly go wrong?!?!

Please help me!

[Stefano]

Please help me!

first of all tell us what did you do recently on your server.. did you make an update with external repo enabled?
did you install something? if so, what? how?

thank you

P.S. your signature sounds bad to me

[FreakWent]

There was an fsck that made changes, orphaned inodes and so on.

I think perhaps some necessary files got clobbered, but that's only an assumption at this stage.

Other than that, there have been no changes in ages.

Signature fixed!

[Stefano]

There was an fsck that made changes, orphaned inodes and so on.

mmmhh... abnormal shutdown? a little more history of the server, please

Signature fixed!

thank you

[FreakWent]

I'm reluctant because I don't want to provide info that might mislead you -- on the other hand, I'm the one asking for help.

I don't keep a server log of every change, it's just for home use.  Perhaps I should start.

It's got DansGuardian on it, and I (think) I triggered the fsck by doing a tune2fs -m 1 because disk space was tight.  Apparently it's 'okay' to do this live, especially if you're *reducing* the amount of reserved space.  Of course, this could just be a coincidence.

Other than those though, there's no significant/relevant changes I can think of, just forwarded ports and a backup cron, stuff like that.

[CharlieBrady]

Now I have current, lock and state, all 0 bytes.

So what's happening?

Perhaps your disk is full.

I (think) I triggered the fsck by doing a tune2fs -m 1 because disk space was tight.

That's probably an unwise way to deal with the situation. It will make your system more likely to be unusable - it allows unimportant programs to hog disk space, rather than reserve space for important ones.

[FreakWent]

Perhaps your disk is full.

Disk space was tight, but it's not now so that doesn't explain why the perl module is failing.

it allows unimportant programs to hog disk space, rather than reserve space for important ones.

Yes and no.  Lots of stuff breaks that doesn't run as root -- like smelog -- so this improves the situation for them.  Its a 500Gb drive so 5Gb of 'emergency' space really is enough for a low load server.  Reserving 25Gb is overkill for this system.

I think I'll just bite the bullet tonight and do yum upgrade, since there are no other helpful answers yet.  My worst case outcome there is to rebuild and restore from backup, which is, AFAICT, my next option anyway.

[cactus]

I think I'll just bite the bullet tonight and do yum upgrade, since there are no other helpful answers yet.

That will most likely help you nothing as this will only update configuration files and the files from the packages that are contained in the update, it might very well be that that is not all that is modified, missing or corrupt.

My worst case outcome there is to rebuild and restore from backup, which is, AFAICT, my next option anyway.

IMHO that is your best bet and the quickest solution to your problem. I suggest you also plug in a bigger drive while you are at it

[CharlieBrady]

So what's happening? So we bump up LogLevel to debug and log to /tmp/templog and we see:

[Wed Jun 08 17:48:58 2011] [error] [client 127.0.0.1] fileparse(): need a valid pathname at /usr/lib/perl5/vendor_perl/5.8.5/CGI/Persistent.pm line 30
[Wed Jun 08 17:48:58 2011] [error] [client 127.0.0.1] Premature end of script headers: index.cgi

A similar problem was reported once before - you might find some clues in the investigation of that problem:

http://bugs.contribs.org/show_bug.cgi?id=4074

[FreakWent]

I found that too, thanks for the link!

As they say there, "I'm going to mark this bug as INVALID there are signs of manual changes made outside the scope of the SME Server."

Since the detail of the fsck changes are unkown, then I guess that applies to mine too.

What I'd find useful is an analog to Windows' repair routine that can run from the install CD.  Running and "upgrade" over the top to the same version didn't help.

I tried to force reinstallation of some RPMs from the CD, with no luck, and when I removed some from the server to allow reinstallaion, I found a dependency (server-admin-images) which I could not find a matching RPM for.  Thus, some essential RPMs could not be installed and I could not proceed, the server was broken and wouldn't function properly any more.

So, I erased and reinstalled.  It's a pain because it takes me a while to copy back all the little bits and pieces, but it gives me a chance to review and improve my docco as I go....

So thanks all, but that's about it!

[cactus]

I found a dependency (server-admin-images) which I could not find a matching RPM for.  Thus, some essential RPMs could not be installed and I could not proceed, the server was broken and wouldn't function properly any more.

That is not a SME Server core package, not sure where you installed that from, but certainly not from SME Server CD or repositories.

[CharlieBrady]

That is not a SME Server core package, not sure where you installed that from, but certainly not from SME Server CD or repositories.

server-admin-images is not a package, it's a metapackage. It's provided by smeserver-support:

rpm -q --whatprovides server-admin-images

I don't know why yum would not be finding and satisfying that dependency. But FreakWent could have solved the problem by doing:

yum install smeserver-support

[FreakWent]

FreakWent could have solved the problem by doing:

yum install smeserver-support

Ah well, at least now I know for next time.  I should not have been so impatient!

Thanks everyone!