Koozali.org: home of the SME Server

Backing up SME

Offline k0r54

  • ****
  • 111
  • +0/-0
Backing up SME
« on: February 01, 2007, 12:32:28 PM »
Hi,

I have done some research on backups for SME and come across a few contribs.

I have tried using a couple but was wondering what the easiest and quickest way is to do a full backup. I am worried that I am not backing up sme fully.

I need it so in the event of a crash I can just reinstall SME and then perform a restore and it is then back EXACTLY as it was. profiles, emails(using fetchmail), password, ibays, domains, sme settings the entire lot.

It there a good way someone could recommend?

Thanks
k0r54

Offline Agent86

  • *****
  • 592
  • +0/-0
    • http://www.iclbiz.com
Re: Backing up SME
« Reply #1 on: February 01, 2007, 02:31:24 PM »
Quote from: "k0r54"
Hi,

I have done some research on backups for SME and come across a few contribs.

I have tried using a couple but was wondering what the easiest and quickest way is to do a full backup. I am worried that I am not backing up sme fully.

I need it so in the event of a crash I can just reinstall SME and then perform a restore and it is then back EXACTLY as it was. profiles, emails(using fetchmail), password, ibays, domains, sme settings the entire lot.

It there a good way someone could recommend?

Thanks
k0r54


I'm in the same boat, I know about the backup to desktop, then install fresh SME then restore is simple enough, however that has size limitations and I'm not sure about it backing up the ibays and things I believe it backs up the configuration and user folders all settings, but I'm not sure about the actual ibays folders etc.

The flex backup feature which is included in SME is good for full tape backup, and will backup the whole hard drive.

And the raid feature will mirror the drive so that if one drive fails the other identical drive will pickup and keep going, thats the one I like best.

I'm not sure if that is considered backup though I with I could be of more help, but I'm really not sure which is the best SME full backup myself.

I suggest if you don't hear back from someone who knows to keep searching the how to's and search the forums there must be something here that describes a full backup
Sorry I can't be more help but I'm sure someone will post about this

Offline k0r54

  • ****
  • 111
  • +0/-0
Backing up SME
« Reply #2 on: February 01, 2007, 02:46:37 PM »
hey,

yeah I am trying to do the backup to desktop but I have 14GB and it even says its not recommended but its taking a VERY long time.

k0r54

Offline byte

  • *
  • 2,183
  • +2/-0
Backing up SME
« Reply #3 on: February 01, 2007, 03:02:29 PM »
Quote from: "k0r54"
but I have 14GB and it even says its not recommended but its taking a VERY long time.


That is too big and will not restore properly just search forums re: 2GB limit using backup to desktop.

Try in the contribs either DAR from Darrell or JPL's great work on another backup tool
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline Agent86

  • *****
  • 592
  • +0/-0
    • http://www.iclbiz.com
Long time on desktop backups
« Reply #4 on: February 01, 2007, 11:19:23 PM »
I installed fresh SME

I'm currently doing a restore from desktop

It seems to be taking a long time ?

My question is that I've installed fresh install, however the backup copy was backed up and has some installations of newer horder RPM's and things

Will my restore give me back what I've had ? and I only had a 500 meg backup file so I'm not sure why it's taking so long ?

Please advise

P.S also does it matter if I'm restoring to larger hard drives ?

Thanks

Offline bpivk

  • *
  • 908
  • +0/-0
    • http://www.bezigrad.com
Backing up SME
« Reply #5 on: February 02, 2007, 12:06:54 AM »
You'll have back your ibays, sql database and mails. Horde won't be a problem. Well the computer has to write a lot of data so be patient.
"It should just work" if it doesn't report it. Thanks!

Offline Agent86

  • *****
  • 592
  • +0/-0
    • http://www.iclbiz.com
Thanks
« Reply #6 on: February 02, 2007, 12:13:31 AM »
Thanks and yep

Well It appears to have backed up the ibays, and mail etc.

However not the RPM's of horde there was not install of the task,gollem, and horde modules that I've installed I reconfigured and restarted but not there.

So I'm going to look into the contribs DAR program or something
I need to figured out how to upgrade my hard drive,

Really it would be nice if I was more eduacated on the commands subjects.

I read the manual and it says when adding a larger hard drive then to enter some commands and (FIXME) but I don't really know wher to begin I've been searching everywhere and can't seem to grasp the concept of adding a larger hard drive

Basically I want to add the larger drive, and sync mirror with raid, then take out the smaller drive and add another identical larger drive and then sync up that drive, but now I'm starting to thing that It may be better to drive copy to larger drive first.

Need suggestions and direction ?

Offline raem

  • *
  • 3,972
  • +4/-0
Re: Thanks
« Reply #7 on: February 02, 2007, 11:22:52 AM »
Agent86

Your backup questions have been answered many times in these forums so I suggest you do a search on backup.

Here it is again as succinctly as I can put it.

The sme backup & restore concept is that only certain data & config files are backed up. These are sufficient to rebuild your server to the way it was ie from a data integrity point of view eg users, pseudonyms, ibays, mail, mysql dbs, custom templates, & other system data etc, but this does not include add on contrib rpms. Note that the config files for contribs will be backed up and restored but not the contrib rpms themselves.

When you restore a backup file it should ONLY ever be to a fresh install of the operating system, which means that after the restore is complete you should then reinstall the ad on contrib rpms. Then your server will be the same as the original.

Other backup contribs such as backup2 or backupdar use the same concept, so they do not backup and restore contribs either.

The default backup to desktop also does the same thing BUT it is limited in usefullness due to the 4Gb backup file creation limits and even worse 2Gb restore file size limits. So any backup you create with that feature, you must ensure the resultant backup file is smaller than 2Gb if you wish to be able to restore from it.
ALWAYS check the integrity of the backup file after you perform a backup to desktop, the routine is in the server manager panel, that ensures your backup file is good (but it still must be less than 2Gb to be restorable).

If you have a "small" server with very limited amount of data on it, the backup to desktop function is useful & quite good.
For many servers with tens or hundreds of Gb's of data is is NOT usable. You then need to use tape or backup2 or backupdar or some other  method.

Tape backups do backup all the files on the server, but they are not all restored, only the limited set mentioned above are restored, so essentially you still need to reinstall all contribs after a tape backup & restore. Search the FAQ for details of what is backed up and restored.

The only way to backup your whole fully configured hard disk including the installed contribs is to do a disk clone using appropriate tools.
The linux dd command will copy the whole hard disk bit by bit to another connected disk but it will take many hours.
for usage
man dd
There are cloning software eg Ghost, there is the Mondo contrib and there are GPL clone solutions too. Make sure they handle RAID drives correctly.

A practical way to achieve a "full system backup" of your fully configured sme server is to use software RAID1 with two drives & swap out one of the RAID drives.
You need a third drive of identical size to those in your sme server RAID1 array. Shutdown the system and physically remove one of the drives.
Reconnect a clean drive and rebuild the array (menu selectable).
In the event of a major disk failure this "spare drive with data on it" can be connected to the system (as a single drive in degraded RAID mode) and you can start up the system using only this drive. Then you can shutdown and connect a second clean (identically sized drive) and rebuild the array (selectable from the console menu when you log in as admin).
This usually take a couple of hours to rebuild but the sme server is usable the whole time.

If you want to increase the size of your drive, see the restore from disk procedure that is explained in the sticky post re upgrading from sme6 to sme7.
http://forums.contribs.org/index.php?topic=30745.0

The same concept applies to upgrades/restores from sme7 to sme7 from smaller to larger disks.

You can't add a larger drive to a smaller drive in RAID1 array. The array will be created to the size of the smaller drive.

If you want to increase the disk size the simplest approach is backup, rebuild server with larger drive(s) preferably in software RAID1, install clean OS, restore, install contribs.
Note the restore can be done by using one of the backup contribs (backup2 or backupdar from Darrell May) & then restore.
You can save some backup time by using the restore from disk routine, its the same thing but just bypasses the backup step.
Follow the steps in the link provided.

You will need to reinstall contribs after any restore is complete but any reconfiguration should be minimal as the contribs will see & use the restored config files and databases & mysql dbs.

Then when your server is fully rebuilt with larger drives in software RAID1, you can remove one of the drives, replace it with a empty partitionless drive of the identical size, and select the option to rebuild the array from the admin console menu.
Always issue the command
cat /proc/mdstat
to check the array is fully synchronised before removing a drive.

Always erase partitions and data from a drive before you add it to an array, it must be totally EMPTY & partitionless. You can issue commands in Linux to erase the disk
eg (replacing hda & hdc with the correct details ie sda sdb as applicable)
dd if=/dev/zero of=/dev/hda
dd if=/dev/zero of=/dev/hdc

or use the Delpart.exe utility on a floppy or CD boot disk, search Internet for it. You would do this in a different box before doing the swap.

The drive to be removed (swapped) would ideally be mounted in a removable front panel caddie to allow easy regular swapping without opening the box. Get a good brand of caddie to ensure the little fans inside keep working after a few years.

I would still run regular full backups using backup2 or backupdar as these backup your system in between disk swaps, so you have the best of both worlds. You can quickly be up and running to the last good disk swap (although the data may be old), and then rebuild a machine at a more leisurely pace using restore from the backup & reinstall contribs, or otherwise just selectively restore data to your swapped in drive. It depends on your circumstances, and most likely which way will bring about the quickest resolution of your disk failure problem.

Now go back to the beginning of this post and read it all again SLOWLY.
...

Offline cactus

  • *
  • 4,880
  • +3/-0
    • http://www.snetram.nl
Re: Thanks
« Reply #8 on: February 02, 2007, 12:31:15 PM »
Quote from: "RayMitchell"
Now go back to the beginning of this post and read it all again SLOWLY.

Compile error: Indefinite loop ;-)
Be careful whose advice you buy, but be patient with those who supply it. Advice is a form of nostalgia, dispensing it is a way of fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than its worth ~ Baz Luhrmann - Everybody's Free (To Wear Sunscreen)

Offline Agent86

  • *****
  • 592
  • +0/-0
    • http://www.iclbiz.com
Done that
« Reply #9 on: February 02, 2007, 08:43:01 PM »
Thanks for the help

I've read all this already and have now re-read it and even again several more times to see what I'm missing.

Conclusion
This seems to have branched into 2 separate subjects

1. upgrading SME versions to higher versions(this seems to somewhat overlap upgrading hard drives and backup restore topics but not necessarily)

2.upgrading to larger hard drive capacity and the reference to the manual
http://smeserver.sourceforge.net/sme7/tech/Chapter8?v=2vh

Since I could not really find a freebie solution that I could get working (note that I could get working LOL) that would duplicate the drive or drive copy for linux etc. I was considering purchasing one.
But while researching this subject I decided to also consider the backup/restore method either backup to desktop and/or backup2 or dar method etc..

With all the various topics and apparent complications of doing the backup/restore and then re-installing RPM's.;and other concerns for possibly doing this again in the near future,I resorted back to the manual for upgrading hard drives.

Since I initially started in the direction of backup restore to fresh installation I basically had now 2 larger drives that synced , however I did not bother installing the contribs as this was not what I was really after.

So, not knowing how the drive would react I followed the manual on this subject. I essentially installed the old smaller drive and took out all drives to start on a resynce of the larger drive

I then booted with the one drive in the system, then added the larger drive to the system following the manuals instructions. This was not the way to go as I did not even think of removing the partitions;and thus I had the larger drive with the earlier attempt at a fresh install and restore on it;These drives seem to be fighting with each other and the larger drive won and so I had 2 synced drives that were working fine, however it was a sync of the fresh installed version that was on the larger drive and not the sync of the older drive.

So back to the drawing board, I then removed all partitions from the larger drives and the grabbed the last smaller drive that was the last smaller drive with the original data on it. I then put in the system as Master1 and started up to check for problems.
Then put the new larger drive in as Secondary Master
Then did a sync from the console panel.
all went well,
I then took out the smaller drive and the moved the larger drive from Secondary Master to Primary Master
booted up to check things out
Then put in another identical larger drive in the Secondary Master, then did a sync

All is fine and clean Raid

The question now is regarding the manual in trying to expand the partitions, Or do I need to expand anything ? I've read the forums that the partitions would still be small ?

I've used Gparted and thought I had this solved and expanded the partition, but when I booted up the system I got GRUB command prompt and nothing else happened ?

So I reviewed this again in Gparted and noticed that that partitions are a boot:raid of about 200MB capacity not GB but MB and also the unused is about 57GB which is the proper size. I noticed the old smaller drive is basically the same configuration of about 200GB boot:raid and about 18GB of unused and unkown file type also which is the proper size

So anyhow, I did the process all over again and now have 2 synced drives clean raid

I'm assuming I still need to expand the partitions, but I'm not sure how ?

Do I need to expand the partitions ?
And will Gparted do this ? or do I need some other tool ?
Will SME expand the partitions with some commands ?

Please advise thanks

Offline raem

  • *
  • 3,972
  • +4/-0
Re: Done that
« Reply #10 on: February 03, 2007, 02:54:36 AM »
Agent86

> Do I need to expand the partitions ?

As I understand it, you only need to expand the data partition to make use of the rest of the unused space on the larger disk.


> Will SME expand the partitions with some commands ?

Not sme itself as such, but by using standard Linux tools, see below.


There have been some good posts re using various hard drive tools, search on LVM and mdadm. as these are appropriate for sme7

Here's an older post about using g4u to clone the drive and gparted to increase the partition size (might not be applicable any more with sme7).
http://forums.contribs.org/index.php?topic=34533.msg149308#msg149308

Here's a post that specifically (although briefly) describes a method to do what you want for sme7 (using Ubuntu).
http://forums.contribs.org/index.php?topic=34533.msg149308#msg149308

Also try these for starters/general info
http://forums.contribs.org/index.php?topic=30745.0

http://www.centos.org/modules/smartfaq/faq.php?faqid=45

http://www.redhat.com/magazine/009jul05/features/lvm2/

http://www.tldp.org/HOWTO/LVM-HOWTO/


Those instructions in the maiual re using FIXME are brief, but the crux of it for you, is to learn how to use FIXME (assuming the process works OK as I have not tried it). See

http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=FIXME+partitioning+tool

http://www.gnu.org/software/grub/manual/grub.html


This "plain english" lvm introduction may help

http://no.longer.valid/phpwiki/index.php/lvm
...

Offline stephen noble

  • *
  • 607
  • +1/-0
    • Dungog
Backing up SME
« Reply #11 on: February 03, 2007, 01:32:54 PM »
Shad has updated the manual, count your blessings

http://smeserver.sourceforge.net/sme7/tech/Chapter8

Offline Agent86

  • *****
  • 592
  • +0/-0
    • http://www.iclbiz.com
Awsome
« Reply #12 on: February 03, 2007, 03:16:04 PM »
Quote from: "snoble"
Shad has updated the manual, count your blessings

http://smeserver.sourceforge.net/sme7/tech/Chapter8


This a awsome news

I really wished I had learned linux before all i've learned about dos,and windows.
I would have most likely already known how to do this.

This really helps alot thanks
This is my new favorite post
Thanks again

Here is what I get:

[root@hostname ~]# mdadm --grow /dev/md2 --size=max
[root@hostname ~]# pvresize /dev/md2
  Physical volume "/dev/md2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
[root@hostname ~]# lvresize -l $(vgdisplay -c main | cut -d: -f16) main/root
  Extending logical volume root to 38.66 GB
  Logical volume root successfully resized
[root@hostname ~]# ext2online -C0 /dev/main/root
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
Added 5414912/5414912...
[root@auction ~]#

Not sure what this means:
[root@hostname ~]# pvresize /dev/md2
  Physical volume "/dev/md2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Is this expanded or not ? only (1)volume resized ?

main/root
  Extending logical volume root to 38.66 GB
  Logical volume root successfully resized

This seems a bit small I am using 2)60GB mirrored drives.
Does this seem correct ?

Offline stephen noble

  • *
  • 607
  • +1/-0
    • Dungog
Backing up SME
« Reply #13 on: February 03, 2007, 03:35:13 PM »
df -h

Offline raem

  • *
  • 3,972
  • +4/-0
Backing up SME
« Reply #14 on: February 03, 2007, 03:46:40 PM »
>Shad has updated the manual, count your blessings

Thank you Shad, saves wading through the mdadm doco's.
...