Topic: Sigh, end users.... Moved large email folder now tree seems corrupt

[Drifting]

Any help with this one really appreciated.

A close friend uses an SME server that I setup for him to run his home business. He had created a number of folders below inbox. He uses Imap and outlook. For some reason best know to him he moved one of these large folders (Actually by mistake) to another location in the inbox tree that he created.

Both thuderbird and outlook flatly refuse to show these sub folders, and say that the mailbox does not exist. However using Webmail they are there, and so are the contents? Any idea how I can convince IMAP to display them properly on the client?

Did not really know where to start, had a look at the Imap & Imapd logs, could not see any obvious errors.

Regards P

[guest22]

Did he subscribe to the new folder within Thunderbird?

[Drifting]

Yes HF he did, I can see the folders in Outlook and Thunderbird, but cannot display the contents, just pops up a message about mailbox not existing. Did wonder if it was something to do with Dovecot.index files , having read a similar thread in a bug report, but this was a long while ago.

Thanks P

[Stefano]

you could try tihs:

- close all your email clients
- stop dovecot
- rename all dovecot.index files
- reboot your server

let us know

in dovecot 2.1.X there's a index flag in doveadm utility to reindex emails..

[Drifting]

Not managed to do the dovecot.index files yet, trying to find a quick way to do it all in one, not that proficient in Linux yet


Also noticed this in the log, bit worrying:-

Mar 19 13:54:35 ins1 HORDE[18459]: [turba] Could not instantiate the localldap source: Bind failed: (-1) Can't contact LDAP server [pid 1845

[Drifting]

Further to the above, I just fired up Webmail to another users mailbox that outlook and thunderbird says is empty, and sure enough it shows empty, yet across the top it says 6 emails in the mailbox?

Beginning to not see the wood for the trees! Or is this related to the index files?

P

[Drifting]

All starts to become clear.

Followed Stefano's suggestion on index files, sadly no change.

This may help someone as a way of finding the index files and renaming them .backup

find /home/e-smith/files/users/ -name 'dovecot.index' -print 2> /dev/null | sed -e 's;^\(.*\)/\([^/][^/]*\)$;mv "\1/\2" "\1/dovecot.backup";' | /bin/sh

I have just found out that the server was restored from a previous backup, this was done by mistake when they were trying to recover a file! Nice of them to drop that one on me. Hope someone can advise on the next move as things have got quite serious.
Perhaps I should list the problems in order.
Few users missing mail in their inboxes, yet subfolders off the inboxes seem fine. New mail seems to be coming in to them fine.
One user has all their subfolders off inbox fine, however they have nothing in their inbox, and no new mail arriving.

I am wondering if I should restore the mailboxes, but would really like to know how to keep any email that has come in. I know I could do this with outlook and a pst file, but the user who has no visible email, would that be stored anywhere recoverable?

Final idea would be to just totally re install clean and then do a restore.

Any advice welcome.

Paul

[janet]

Paul

Doing a full restore to a server with existing data on it will most likely result in a server with many inconsistencies, which appears to be what you are seeing.

The restored system data has one set of folders, subfolders & email messages etc etc, but there are many orphaned folders & messages left behind from the old system that the newly restored system does not know about.

One answer:
Attach another drive, use command line to copy all user data, folders & email messages, databases etc & anything else you think you may want to retrieve, to the attached mounted drive (formatted with ext 3 or 4).
Disconnect the drive.

Wipe the server using dd & then reinstall SME OS & restore from the last known good backup (ie before the errant restore happened).

Reattach the drive with latest data & then manually copy messages & whatever else you want to the server.
Depending how much data & how many users it may take a long time, & you need to compare restored against data/messages on the extra drive to see what is missing so you know what to copy over.

Another method:
Using command line I assume you can see the folders & messages that SME server & mail clients are not seeing, so then copy those messages to known good folders.
Again a slow process.

Blending data from different "backup sets" is not easy, so think think think carefully before doing this.

Using the second method may still leave many orphaned files & folders etc, depending what has happened (changed) to the server since the errant restore was done.
You will still need to look hard.

[Drifting]

Thank you so much for the reply Janet.

I will take your advice, I have been doing this all remotely thus far, but looks like a long drive tomorrow, and a clean install and restore a knows good backup.

Couple of questions as I copy the using home dirs and email folders.... Restoring email ? Which folders? all of the Maildir per user? How does IMAP / Dovecot deal with me adding those folder? Have I understood what you were telling me on email copying? (Doubt it)

Paul.

[janet]

Drifting

Just copying folders across will still probably leave you with inconsistencies & orphaned folders & messages.
Only copy email messages to known good folders.
Create the folders each user needs (that are missing) using an IMAP email client.

Remember the procedure is to Restore from known good data, & then manually add in missing or changed data that has happened since the restore.
This hopefully avoids lost data & inconsistent data.

I suggest opening 2 windows of mc & tag & copy groups of changed/new email messages on a folder by folder basis.
Reset ownership on each users Maildir something like (add full path)
cd/ .../username/Maildir
chown -R user:user

Also I would be making a full backup of the existing server before erasing & reinstalling
or
remove a hard disk from the RAID1 array & set it aside for safekeeping for a few weeks/months until sure of the integrity of the rebuilt server.

Remember to THINK about all the consequences of changed data & manually adding changed data.
What about SME db's & mysql db's & other configuration settings that may have changed, or newly installed apps or web pages  or files users have changed or added etc etc etc.

THINK  !!!

Remember I have not seen your server or looked at the files to verify my approach is correct, but I have had to do (successfully) similar things in the past, unfortunately.

When desperate, desperate actions are necessary.

[Drifting]

Thank you HF, Stefano & finally Janet for your help. Without it the pain would have been much worse.

After following your advice, I managed to do a clean install, restored the last know backup, then sort the remaining problems.
The last one was particularly time consuming. This is what was happening. Client was using outlook, all new email was coming in fine, however none of the existing restored email would show. Read numerous googles on how crap Outlook was with IMAP, chased numerous wild geese. Installed Thunderbird and all the email showed up? Did a clean account with Outlook, and the problem was still there.
Not sure who it was said check the logs, but I should have listened. There it was staring me in the face, access denied to blah,blah,blah in imap log.
Set chown to each user on their Maildir and away it all went!

Thank you all so much again, was really suffering from wood and trees!

Paul.