Topic: Backing up SME

[k0r54]

Hi,

I have done some research on backups for SME and come across a few contribs.

I have tried using a couple but was wondering what the easiest and quickest way is to do a full backup. I am worried that I am not backing up sme fully.

I need it so in the event of a crash I can just reinstall SME and then perform a restore and it is then back EXACTLY as it was. profiles, emails(using fetchmail), password, ibays, domains, sme settings the entire lot.

It there a good way someone could recommend?

Thanks
k0r54

[Agent86]

Hi,

I have done some research on backups for SME and come across a few contribs.

I have tried using a couple but was wondering what the easiest and quickest way is to do a full backup. I am worried that I am not backing up sme fully.

I need it so in the event of a crash I can just reinstall SME and then perform a restore and it is then back EXACTLY as it was. profiles, emails(using fetchmail), password, ibays, domains, sme settings the entire lot.

It there a good way someone could recommend?

Thanks
k0r54

I'm in the same boat, I know about the backup to desktop, then install fresh SME then restore is simple enough, however that has size limitations and I'm not sure about it backing up the ibays and things I believe it backs up the configuration and user folders all settings, but I'm not sure about the actual ibays folders etc.

The flex backup feature which is included in SME is good for full tape backup, and will backup the whole hard drive.

And the raid feature will mirror the drive so that if one drive fails the other identical drive will pickup and keep going, thats the one I like best.

I'm not sure if that is considered backup though I with I could be of more help, but I'm really not sure which is the best SME full backup myself.

I suggest if you don't hear back from someone who knows to keep searching the how to's and search the forums there must be something here that describes a full backup
Sorry I can't be more help but I'm sure someone will post about this

[k0r54]

hey,

yeah I am trying to do the backup to desktop but I have 14GB and it even says its not recommended but its taking a VERY long time.

k0r54

[byte]

but I have 14GB and it even says its not recommended but its taking a VERY long time.

That is too big and will not restore properly just search forums re: 2GB limit using backup to desktop.

Try in the contribs either DAR from Darrell or JPL's great work on another backup tool

[Agent86]

I installed fresh SME

I'm currently doing a restore from desktop

It seems to be taking a long time ?

My question is that I've installed fresh install, however the backup copy was backed up and has some installations of newer horder RPM's and things

Will my restore give me back what I've had ? and I only had a 500 meg backup file so I'm not sure why it's taking so long ?

Please advise

P.S also does it matter if I'm restoring to larger hard drives ?

Thanks

[bpivk]

You'll have back your ibays, sql database and mails. Horde won't be a problem. Well the computer has to write a lot of data so be patient.

[Agent86]

Thanks and yep

Well It appears to have backed up the ibays, and mail etc.

However not the RPM's of horde there was not install of the task,gollem, and horde modules that I've installed I reconfigured and restarted but not there.

So I'm going to look into the contribs DAR program or something
I need to figured out how to upgrade my hard drive,

Really it would be nice if I was more eduacated on the commands subjects.

I read the manual and it says when adding a larger hard drive then to enter some commands and (FIXME) but I don't really know wher to begin I've been searching everywhere and can't seem to grasp the concept of adding a larger hard drive

Basically I want to add the larger drive, and sync mirror with raid, then take out the smaller drive and add another identical larger drive and then sync up that drive, but now I'm starting to thing that It may be better to drive copy to larger drive first.

Need suggestions and direction ?

[raem]

Agent86

Your backup questions have been answered many times in these forums so I suggest you do a search on backup.

Here it is again as succinctly as I can put it.

The sme backup & restore concept is that only certain data & config files are backed up. These are sufficient to rebuild your server to the way it was ie from a data integrity point of view eg users, pseudonyms, ibays, mail, mysql dbs, custom templates, & other system data etc, but this does not include add on contrib rpms. Note that the config files for contribs will be backed up and restored but not the contrib rpms themselves.

When you restore a backup file it should ONLY ever be to a fresh install of the operating system, which means that after the restore is complete you should then reinstall the ad on contrib rpms. Then your server will be the same as the original.

Other backup contribs such as backup2 or backupdar use the same concept, so they do not backup and restore contribs either.

The default backup to desktop also does the same thing BUT it is limited in usefullness due to the 4Gb backup file creation limits and even worse 2Gb restore file size limits. So any backup you create with that feature, you must ensure the resultant backup file is smaller than 2Gb if you wish to be able to restore from it.
ALWAYS check the integrity of the backup file after you perform a backup to desktop, the routine is in the server manager panel, that ensures your backup file is good (but it still must be less than 2Gb to be restorable).

If you have a "small" server with very limited amount of data on it, the backup to desktop function is useful & quite good.
For many servers with tens or hundreds of Gb's of data is is NOT usable. You then need to use tape or backup2 or backupdar or some other  method.

Tape backups do backup all the files on the server, but they are not all restored, only the limited set mentioned above are restored, so essentially you still need to reinstall all contribs after a tape backup & restore. Search the FAQ for details of what is backed up and restored.

The only way to backup your whole fully configured hard disk including the installed contribs is to do a disk clone using appropriate tools.
The linux dd command will copy the whole hard disk bit by bit to another connected disk but it will take many hours.
for usage
man dd
There are cloning software eg Ghost, there is the Mondo contrib and there are GPL clone solutions too. Make sure they handle RAID drives correctly.

A practical way to achieve a "full system backup" of your fully configured sme server is to use software RAID1 with two drives & swap out one of the RAID drives.
You need a third drive of identical size to those in your sme server RAID1 array. Shutdown the system and physically remove one of the drives.
Reconnect a clean drive and rebuild the array (menu selectable).
In the event of a major disk failure this "spare drive with data on it" can be connected to the system (as a single drive in degraded RAID mode) and you can start up the system using only this drive. Then you can shutdown and connect a second clean (identically sized drive) and rebuild the array (selectable from the console menu when you log in as admin).
This usually take a couple of hours to rebuild but the sme server is usable the whole time.

If you want to increase the size of your drive, see the restore from disk procedure that is explained in the sticky post re upgrading from sme6 to sme7.
http://forums.contribs.org/index.php?topic=30745.0

The same concept applies to upgrades/restores from sme7 to sme7 from smaller to larger disks.

You can't add a larger drive to a smaller drive in RAID1 array. The array will be created to the size of the smaller drive.

If you want to increase the disk size the simplest approach is backup, rebuild server with larger drive(s) preferably in software RAID1, install clean OS, restore, install contribs.
Note the restore can be done by using one of the backup contribs (backup2 or backupdar from Darrell May) & then restore.
You can save some backup time by using the restore from disk routine, its the same thing but just bypasses the backup step.
Follow the steps in the link provided.

You will need to reinstall contribs after any restore is complete but any reconfiguration should be minimal as the contribs will see & use the restored config files and databases & mysql dbs.

Then when your server is fully rebuilt with larger drives in software RAID1, you can remove one of the drives, replace it with a empty partitionless drive of the identical size, and select the option to rebuild the array from the admin console menu.
Always issue the command
cat /proc/mdstat
to check the array is fully synchronised before removing a drive.

Always erase partitions and data from a drive before you add it to an array, it must be totally EMPTY & partitionless. You can issue commands in Linux to erase the disk
eg (replacing hda & hdc with the correct details ie sda sdb as applicable)
dd if=/dev/zero of=/dev/hda
dd if=/dev/zero of=/dev/hdc

or use the Delpart.exe utility on a floppy or CD boot disk, search Internet for it. You would do this in a different box before doing the swap.

The drive to be removed (swapped) would ideally be mounted in a removable front panel caddie to allow easy regular swapping without opening the box. Get a good brand of caddie to ensure the little fans inside keep working after a few years.

I would still run regular full backups using backup2 or backupdar as these backup your system in between disk swaps, so you have the best of both worlds. You can quickly be up and running to the last good disk swap (although the data may be old), and then rebuild a machine at a more leisurely pace using restore from the backup & reinstall contribs, or otherwise just selectively restore data to your swapped in drive. It depends on your circumstances, and most likely which way will bring about the quickest resolution of your disk failure problem.

Now go back to the beginning of this post and read it all again SLOWLY.

[cactus]

Now go back to the beginning of this post and read it all again SLOWLY.

Compile error: Indefinite loop

[Agent86]

Thanks for the help

I've read all this already and have now re-read it and even again several more times to see what I'm missing.

Conclusion
This seems to have branched into 2 separate subjects

1. upgrading SME versions to higher versions(this seems to somewhat overlap upgrading hard drives and backup restore topics but not necessarily)

2.upgrading to larger hard drive capacity and the reference to the manual
http://smeserver.sourceforge.net/sme7/tech/Chapter8?v=2vh

Since I could not really find a freebie solution that I could get working (note that I could get working LOL) that would duplicate the drive or drive copy for linux etc. I was considering purchasing one.
But while researching this subject I decided to also consider the backup/restore method either backup to desktop and/or backup2 or dar method etc..

With all the various topics and apparent complications of doing the backup/restore and then re-installing RPM's.;and other concerns for possibly doing this again in the near future,I resorted back to the manual for upgrading hard drives.

Since I initially started in the direction of backup restore to fresh installation I basically had now 2 larger drives that synced , however I did not bother installing the contribs as this was not what I was really after.

So, not knowing how the drive would react I followed the manual on this subject. I essentially installed the old smaller drive and took out all drives to start on a resynce of the larger drive

I then booted with the one drive in the system, then added the larger drive to the system following the manuals instructions. This was not the way to go as I did not even think of removing the partitions;and thus I had the larger drive with the earlier attempt at a fresh install and restore on it;These drives seem to be fighting with each other and the larger drive won and so I had 2 synced drives that were working fine, however it was a sync of the fresh installed version that was on the larger drive and not the sync of the older drive.

So back to the drawing board, I then removed all partitions from the larger drives and the grabbed the last smaller drive that was the last smaller drive with the original data on it. I then put in the system as Master1 and started up to check for problems.
Then put the new larger drive in as Secondary Master
Then did a sync from the console panel.
all went well,
I then took out the smaller drive and the moved the larger drive from Secondary Master to Primary Master
booted up to check things out
Then put in another identical larger drive in the Secondary Master, then did a sync

All is fine and clean Raid

The question now is regarding the manual in trying to expand the partitions, Or do I need to expand anything ? I've read the forums that the partitions would still be small ?

I've used Gparted and thought I had this solved and expanded the partition, but when I booted up the system I got GRUB command prompt and nothing else happened ?

So I reviewed this again in Gparted and noticed that that partitions are a boot:raid of about 200MB capacity not GB but MB and also the unused is about 57GB which is the proper size. I noticed the old smaller drive is basically the same configuration of about 200GB boot:raid and about 18GB of unused and unkown file type also which is the proper size

So anyhow, I did the process all over again and now have 2 synced drives clean raid

I'm assuming I still need to expand the partitions, but I'm not sure how ?

Do I need to expand the partitions ?
And will Gparted do this ? or do I need some other tool ?
Will SME expand the partitions with some commands ?

Please advise thanks

[raem]

Agent86

> Do I need to expand the partitions ?

As I understand it, you only need to expand the data partition to make use of the rest of the unused space on the larger disk.


> Will SME expand the partitions with some commands ?

Not sme itself as such, but by using standard Linux tools, see below.


There have been some good posts re using various hard drive tools, search on LVM and mdadm. as these are appropriate for sme7

Here's an older post about using g4u to clone the drive and gparted to increase the partition size (might not be applicable any more with sme7).
http://forums.contribs.org/index.php?topic=34533.msg149308#msg149308

Here's a post that specifically (although briefly) describes a method to do what you want for sme7 (using Ubuntu).
http://forums.contribs.org/index.php?topic=34533.msg149308#msg149308

Also try these for starters/general info
http://forums.contribs.org/index.php?topic=30745.0

http://www.centos.org/modules/smartfaq/faq.php?faqid=45

http://www.redhat.com/magazine/009jul05/features/lvm2/

http://www.tldp.org/HOWTO/LVM-HOWTO/


Those instructions in the maiual re using FIXME are brief, but the crux of it for you, is to learn how to use FIXME (assuming the process works OK as I have not tried it). See

http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=FIXME+partitioning+tool

http://www.gnu.org/software/grub/manual/grub.html


This "plain english" lvm introduction may help

http://no.longer.valid/phpwiki/index.php/lvm

[stephen noble]

Shad has updated the manual, count your blessings

http://smeserver.sourceforge.net/sme7/tech/Chapter8

[Agent86]

Shad has updated the manual, count your blessings

http://smeserver.sourceforge.net/sme7/tech/Chapter8

This a awsome news

I really wished I had learned linux before all i've learned about dos,and windows.
I would have most likely already known how to do this.

This really helps alot thanks
This is my new favorite post
Thanks again

Here is what I get:

[root@hostname ~]# mdadm --grow /dev/md2 --size=max
[root@hostname ~]# pvresize /dev/md2
  Physical volume "/dev/md2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
[root@hostname ~]# lvresize -l $(vgdisplay -c main | cut -d: -f16) main/root
  Extending logical volume root to 38.66 GB
  Logical volume root successfully resized
[root@hostname ~]# ext2online -C0 /dev/main/root
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
Added 5414912/5414912...
[root@auction ~]#

Not sure what this means:
[root@hostname ~]# pvresize /dev/md2
  Physical volume "/dev/md2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Is this expanded or not ? only (1)volume resized ?

main/root
  Extending logical volume root to 38.66 GB
  Logical volume root successfully resized

This seems a bit small I am using 2)60GB mirrored drives.
Does this seem correct ?

[stephen noble]

df -h

[raem]

>Shad has updated the manual, count your blessings

Thank you Shad, saves wading through the mdadm doco's.

[Agent86]

df -h

[root@hostname ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                       39G  2.0G   35G   6% /
/dev/md1               99M   13M   82M  14% /boot
none                  125M     0  125M   0% /dev/shm

Forgive my ignorance please
Why is the size 39G ?
How do I tell if the partition has been expanded all the way properly ?

I have 2)60G drives should it not be somewhere in the 57G range or so ?

Thanks

[k0r54]

Hey,

Ok I have installed dar-2.3.1-1 but am a little unsure of how to run the backup.

I want it so that if SME goes wrong I can format reinstall SME run add my domain and stuff (in the activation thing that comes up) reinstall all my rpm's and then run the backup and all users, ibays, email (from fetchmail) all my fetchmail settings and pretty much everything else is back on through the one restore.

i need it sliced into 4400MB sizes so I can put it to a disk

What command line would I need to run to do that?

Thanks
k0r54

[raem]

k0r54

Install
dar-2.3.2-1.i386.rpm
and the server manager panel/interface to configure it with
smeserver-dar2-0.0.1-1dmay.noarch.rpm  
from
http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/

The max file size to burn is 2Gb, so configure it to create parts of approx 1.529Gb, 3 parts will fit on a DVD.

[k0r54]

Hey,

Brill contrib,

for it to backup everything i said what would i need to put in the backup selections?

Thanks
k0r54

[raem]

k0r54

Create a backup job and by default it already includes all the "necessary' folders.
Search forums on backup if you want more info.

I suggest you fully test a backup and restore procedure to prove it really works for your situation and methods.

With respect to restores, it will restore ALL your configuration including domain & server setup etc, so you only need to configure the newly installed OS with enough parameters to get it running on your system in order to do a restore (after reinstalling the backup dar contrib of course).

[k0r54]

Hi I have set that all up but am gettin the following error

Code: [Select]


Feb  5 13:05:17 compunetsrv01 /etc/e-smith/web/panels/manager/cgi-bin/dar2[7491]: /home/e-smith/db/dar2: NEW Manual_Backup=dar2||manual|12.00|enabled|Backup|etc/e-smith/templates-custom,etc/e-smith/templates-user-custom,etc/group,etc/gshadow,etc/passwd,etc/samba/secrets.tdb,etc/samba/smbpasswd,etc/shadow,etc/smbpasswd,etc/ssh,etc/sudoers,home/e-smith,root|Compression|0|Description|Manual Backup Full|Exclude||Expiry|0|Mount|/mnt/dar2|Password|xxxxxx|Prune||Slice|1500|Target|\\apc-lp-xp-001-s\SMEBackup|Username|admin|VFSType|smbfs|Verbose|off|admin|VFSType|backupDay|manual|backupTime|12.00|enabled||ext3|Verbose|manual|backupTime|off|admin|status|enabled



Feb  5 13:06:07 compunetsrv01 e-smith-bg: Connecting /mnt/dar2/Manual_Backup to \\apc-lp-xp-001-s\SMEBackup
Feb  5 13:06:18 compunetsrv01 e-smith-bg: Error connecting to 192.168.0.177 (No route to host)
Feb  5 13:06:18 compunetsrv01 e-smith-bg: 7514: Connection to apc-lp-xp-001-s failed
Feb  5 13:06:18 compunetsrv01 e-smith-bg: SMB connection failed

I have set up the share on my windows machine and permision is given to the domain admin, the location username and password is all correct.

This is pulled from the /var/log/messages but i have been using the web panel to activate the backup.

Any ideas?

k0r54

[raem]

k0r54

> Connecting /mnt/dar2/Manual_Backup to \\apc-lp-xp-001-s\SMEBackup

Check the backup destination in the server manager panel.
It should be in the form //workstationname/sharename (with forward slashes).

If it is correct, can you connect to that share using another workstation ?

[k0r54]

Hi,

Ok have tested and yes i can access the share from another windows machine via \\apc-lp-xp-001-s\SMEBackup  but not by // obviously but no username or password crops up.

The problem still occurs with the sme though

Thanks
k0r54

[raem]

k0r54

....But do you have the Target entered as
//apc-lp-xp-001-s/SMEBackup
in server manager panel ?

[raem]

k0r54

db dar2 show
will show you what is actually in the database

eg
test=dar2
    Backup=home/e-smith/files/ibays/test,root
    Compression=2
    Description=test ibay backup
    Exclude=
    Expiry=5
    Mount=/mnt/dar2
    Password=xxxxxxxxxxxx
    Prune=
    Slice=650
    Target=//station9/backup-server
    Username=admin
    VFSType=smbfs
    Verbose=on
    backupDay=manual
    backupTime=23:30
    status=enabled

[k0r54]

Hi mine reads as

Code: [Select]


Manual_Backup=dar2
    =manual
    12.00=enabled
    Backup=etc/e-smith/templates-custom,etc/e-smith/templates-user-custom,etc/group,etc/gshadow,etc/passwd,etc/samba/secrets.tdb,etc/samba/smbpasswd,etc/shadow,etc/smbpasswd,etc/ssh,etc/sudoers,home/e-smith,root
    Compression=3
    Description=Manual Backup Full
    Exclude=
    Expiry=0
    Mount=/mnt/dar2
    Password=xxx
    Prune=
    Slice=1500
    Target=//apc-lp-xp-001-s/SMEBackup
    Username=admin
    VFSType=smbfs
    Verbose=off
    admin=VFSType
    backupDay=manual
    backupTime=12.00
    enabled=
    ext3=Verbose
    manual=backupTime
    off=admin
    status=enabled

[k0r54]

ok figured out for some reason i changed my pc name to apc001 and it worked.

My pc name must have just not been valid, any reason why because my pc name means things for my netlogon.

cheers
k0r54

[k0r54]

Hey,

Just a quick question, if i want to do a backup every night what would be the best way to do it. Incremental? Will that just do the changes made that day?

Do I have to do a full backup and then set the incremental up?

Then for a restore run the full and then the incremental?

Thanks
k0r54

[k0r54]

Hi,

As well as the above questions I also have another.

I done a successfull backup and then tried to restore after formating.

Unfortunatly I ran into an error where it would not let me restore from the //apc001/SMEBackup because it was not apart of the domain. (obviously after a format).

I therefore had to run the sme backup 2 desktop I done. After i ran that i could then run the dar restore. Any idea's why this happened and how to get around it?

Cheers
k0r54

[raem]

k0r54

> Do I have to do a full backup and then set the incremental up?

There is no incremental option in dar2 that I'm aware of, so you need to do a full backup every day.
Perhaps you are referring to partial backups of data etc only, using other backup jobs with a limited set of data to be backed up ?


>...tried to restore after formating.

I assume you mean after reinstalling the operating system & the backup dar2 contrib ?


>...I ran into an error where it would not let me restore
> .... because it was not apart of the domain

What is the error that makes you conclude not being part of the domain was your problem.

I'm guessing that by doing the desktop restore you configured the server with basic networking configuration required for your network.
Usually you would do some minimal reconfiguration of your server with it's newly installed operating system prior to a restore, so that it worked on your network eg IP, workgroup name, domain name etc.

Did you do basic reconfiguration before attempting the first restore ?

[k0r54]

Hi,

No there is not so a full backup everynight it is, bit annoying but its pretty good.

Did you do basic reconfiguration before attempting the first restore ?

No I didn't I installed the dar contrib and tired a restore but it was saying the the pc (apc001) was not registered on the domain.

I then set the basics up but still nothing because i think it wanted to to readd the pc to the domain. So when I then run the sme restore from desktop util it sorted everything out for me.

Anyway everything is working ok now so quite happy with it.

Just wondering if i run the backup every night will it constantly overwrite the previous data or will it just keep makin new files without removin the old ones?

cheers
k0r54

[wjhobbs]

Just wondering if i run the backup every night will it constantly overwrite the previous data or will it just keep makin new files without removin the old ones?

Modify the dar job so that Expiry is not zero. Zero says 'overwrite each time' but if you enter a positive number here it is supposed to keep backups for n+1 days.

[Agent86]

df -h

[root@hostname ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                       39G  2.0G   35G   6% /
/dev/md1               99M   13M   82M  14% /boot
none                  125M     0  125M   0% /dev/shm

Forgive my ignorance please
Why is the size 39G ?
How do I tell if the partition has been expanded all the way properly ?

I have 2)60G drives should it not be somewhere in the 57G range or so ?

Thanks

Re posting this question

[k0r54]

Hi,

If i want to put the backups onto a USB device how do I go about it?

I have plugged in the USB so what do I put the location as in the dar admin panel and do I need to mount it at all?

Thanks
k0r54

[wjhobbs]

A search for 'dar2 howto' would have led you here http://forums.contribs.org/index.php?topic=34398.0