Topic: Backup systems that dont work

[steve1084]

[edit]
Stupid post by stupid user withdrawn.
[/edit]

Hi  Im sorry if my comments about affa box offended and caused you to remove your post.  I only want to comment that SME is in theory a community based project that is I hope attempting to cater to both the home user and the SMB markets.  The use of an AFFA box is perfectly correct in the right environment but for the home user that only has $100 to spend on a backup solution is probably not going to use this method.  They are more likely to want to use either an add-on usb drive for local storage or a cheap lan server that they can setup at there mates house and FTP there backup files to so they have there backups off site.

ie its not a mission critical situation for the smaller type user.  But backup is still something they want to have.

For many people the ability to send files off site via ftp is the most sensible, cost effective and easy to manage option to use.  Its up to the creator of the backup system to ensure that the files being sent are reasonably secure.  either they should be encrypted or at least zip/password protected.

[raem]

steve1084

I'm not sure I understand your demands re "must have ftp".

sftp is supported, which is a secure protocol, why would you use insecure ftp when sftp is available.

As far as a offsite backup solution is concerned, a ready to go answer does exist in the dungog commercial product.
Apart from that and other current sme server backup solutions, that is what is available now, no one else has developed the exact product you want (for free).
You have a number of choices, develop it yourself, pay someone to develop it and it then gets donated to the community, buy the commercial product.
You could also lodge a New Feature Request (NFR) at bugzilla and if the developers consider your representations worthy enough, then someone may develop it. Even better if some code (or procedural steps) is posted that can be used as a basis for further development & creation of a suitable rpm (by those who know how to package such stuff into an rpm).
I'm pretty sure it won't use ftp though.

You could also put together/hack current offerings (e-smith backup with dar) and create a script that securely transfers the backup files, using say ssh with public private keys, and the system would do it all automatically.
If you go back a long time and read old forum posts (search), you will find the basis of how to do something like that already published.

[stephen noble]

ftp passes usernames and passwords over the internet in plain text;
how can you protect the data when the hacker has your login details

When I moved to dungog.net/wiki I removed mention of the ftp transfer (weex).
I'd forgotten it was there, rsync over ssh is the way to remotely backup your server

[CharlieBrady]

I doesn't mean to be cynical but cum on people this is pathetic.

What's pathetic is people like you who complain so vehemently about a gift. Do you complain to your grandmother at Christmas that the socks you receive are the wrong colour?

If you don't like what is offered freely to you, you are free to make whatever improvements you think are necessary. Or to pay someone to implement exactly what you want.

[Daniel B.]

Hi. I'm the packager of backuppc rpm for sme. I'd just like to share my opinion on this.
I haven't integrated a tool to export the backups via FTP because it's not secure, but there are some other tools to export it on
- a remote host via rsync over ssh
- a remouvable media such as USB disk
- a local directory

Backups are part of the security of a system, but if you send it over the net via an insecure connexion, you expose all your system (well, it depends on the data you backup, but with the default configuration, backuppc and most of other backup contribs, all the accounts informations, including passwords are saved)

If you really need your backups to be sent via FTP, you can create archives in a local directory, and then push them on your FTP with a little script, it won't be too hard.


Then, for the FTP server on SME, I think having FTP as an option is convinient, because some people still use it, but removing it will "force" users to use more secure tools, like sftp. FTP shouldn't be used anymore nowadays, it's so easy to catch username and passwords. If we don't remove FTP from the base one day or another, of course, some people will continue using it, but if we remove it, users will learn to use secure connexions, it'll be a little harder at the begining, but with some time, it'll become the standard.

The only thing that should be added before removing FTP, is an option to chroot SFTP connexions (without it, any user will be able to browse the entire file system, of course, they'll be limited by the fs permissions). I've read a little about it, and it seems harder than FTP chroot, but it's possible.

[steve1084]

Firstly there are two very different issues being talked about here

One is that if a person wants a backup system with ftp abilities then its going to cost $1000 from dungog.  Or buy another computer for $600 or more and use rsync or affa.   Thats fine for a business but is not an option for many home users.

Due to the way SME is built its not easy to just go out and find any old backup system for Linux and implement it either, so your stuck to some degree with whats on offer and for the home user your stuck with the available tools and equipment at hand.

I don't disagree with there being some risk's involved with the use of plain ftp but it is used extensively by people and most web hosting uses ftp.  This is slowly being replaced by sftp and other options.  I dont have access to a second server for rsync or affa but I do have access to several ftp solutions even if thats not a perfect solution.  I believe these risks to be very small, but you are free to disagree.

Its about providing options to people from all quarters not just those that can afford to spend $1000 or $3000 or whatever on a backup solution.  BackupPC works great and so does sme but when the available options become elitist due to the high cost and don't cater to everyone then thats something I find I must speak up about.

Two When people mentioned that ftp would be removed completely from sme they did not mention that sftp would still be available (please confirm this) so maybe I was led a little astray but I would still rather have ftp at the moment then not have it.  As it allows me to use the equipment I already have available to me.

There is not a chance that I am the only person in this position so what I'm saying is that the available options are very limiting and It would be nice if more options and choices were available.

This is the first forum I have joined that is positively Anti FTP and yet there is every chance that every person on this forum uses or has used ftp.  Forcing people to spend Thousands of Dollars in not I think a good healthy attitude.  Whats better is to provide options that suite a greater number of people and where needed place the appropriate warnings to encourage best practice.  To be so anti FTP is bordering on paranoia to the extreme.

Ive never met anyone who was the victim of a man in the middle attach using ftp.  I have however met and read a lot about people who get bots on there computers that steel passwords etc.  Currently the most common method to have your server hacked is to keep your passwords etc on your windows box and it gets hacked.  Somebody gaining passwords etc via ftp man in the middle attacks is I believe very very low.

Generally there has to be a bot on the machine your connecting to or on the machine your using so either way encryption or not they will get something from you.


I believe everyone has a right to speak about there experiences even if some don't like it.  This is my experience and my frustrations with trying to get a solution that works with the equipment available to me.  Whats pathetic (Charlie) is when people cant accept that there are some that have not been able to find a solution to a problem that they feel should not exist. I'm sorry Charlie but I don't celebrate Christmas so your on your own with that one.

Thanks
Steve

[CharlieBrady]

Forcing people to spend Thousands of Dollars in not I think a good healthy attitude.

Nobody here forces anybody to do anything.

[Stefano]

BackupPC works great and so does sme but when the available options become elitist due to the high cost and don't cater to everyone then thats something I find I must speak up about.

so, where's the problem?
use backuppc, then wtite down a 5 rows bash script to export backups to tgz files an then ftp them..

google is your friend.. you'll find many, many examples..

just a question.. have you already made a donation to sme?

ciao

Stefano

[byte]

Two When people mentioned that ftp would be removed completely from sme they did not mention that sftp would still be available (please confirm this) so maybe I was led a little astray but I would still rather have ftp at the moment then not have it.  As it allows me to use the equipment I already have available to me.

No where did I say it "would" be removed I said in "future" releases it could be, reason I didn't mention sftp is because it would be unaffected as ftp and sftp are different and provided by different packages, therefore "if" ftp was to be removed then sftp would remain due to being different to ftp.

This is the first forum I have joined that is positively Anti FTP and yet there is every chance that every person on this forum uses or has used ftp. 

Not Anti-FTP, just trying to educate/show the more secure approach.

I'm sorry Charlie but I don't celebrate Christmas so your on your own with that one.

Everyone should celebrate Christmas

[steever]

One is that if a person wants a backup system with ftp abilities then its going to cost $1000 from dungog.

Dungog sells their standard life time update version for AUD 299 which is a good price considering you get access to all of Stephen's great software for use on one server.  $299 is a lot cheaper than $1000.

If Steve1084 needs this functionality on multiple servers because he has a number of customers then he can buy separate licenses for each server (at $299) and bill the client for "online backup".  But in this case, getting the AUD 999 license makes better sense anyway.

[raem]

steve1084

If you read the Affa Wiki carefully, you can also backup to a mounted filesystem or USB etc.
See the topic "Use Affa to backup to a NFS-mounted NAS or a local attached USB drive"

From there it should be a relatively simple matter to rsync/ssh to a remote server.

[imcintyre]

Steve1084;

I didn't want to stick my nose into this "discussion" but, in your very first post you mention that;

Dar2 doesnt work properly....

but I did not see this expanded upon or a bug # mentioned. Is there a bug already raised for the problem you speak of or a specific issue that should be known about?

[steve1084]

Steve1084;

I didn't want to stick my nose into this "discussion" but, in your very first post you mention that;but I did not see this expanded upon or a bug # mentioned. Is there a bug already raised for the problem you speak of or a specific issue that should be known about?

Myself and a friend tested all available backup solutions and found several deficiencies according to our environment.

We created a test server environment with several users and several joomla test sites.  Installed and tested dar dar2 backupPC etc etc

What we found was that after trying a full restore from what was supposedly a full backup of each of the systems was that they mostly failed to restore the databases properly or didn't backup all the folders files and we could not determine why.  I probably need to look into it further.  The standard backup doesn't do full backups so its not really usable.  BackupPC was the only one that truly worked properly.

I believe a bug report was posted

As to the issue of the price for access to the dungog rpms yes it is only $300 for a "single server".  But as in time I would like to have more than one server the real cost is $1000.  Its not like he sells each of the rmps for $20 or has a club yearly subscription for $30.  It makes what is supposedly a free system quite expensive if you choose to go down that path.  Admittedly its still a hell of a lot cheaper that a Microsoft alternative.  I just cant afford or Justify the cost when I am still in the testing phase with sme.

The only off site storage available to me ATM is a plain FTP option.

The samba option would not login to the "ritmo lan server" so that option didn't work. and the local windows machines are not suitable for accepting the backups as they are not always on.  The older test machine we used for testing dosnt fully support usb but that would probably have worked on a slightly better machine. at least for locally stored backups.

So the option I have left is to use backupPC to create the backup files and to send them off site with another tool or script of some sort.  It simply would have been easier and less frustrating if plain FTP had of been available as part of the backup programs even if it was an extra install with lots of warnings.  This is a bit messy and will probably entail hand removal of the older files from time to time? ie its not an integrated solution.

At first I thought I would be able to use cobian backup to do the offsite transfers but It turned out that It doesnt have ssh abilities only ssl so it couldn't login to the sme server. so I'm still hunting around for something like cobian backup that has both plain ftp and ssh abilities to run either on the sever or on a windows machine.

I have now deployed one sme server for a local business and when I get paid a percentage will be donated to sme.

My apologies if I have managed to misinterpret any persons comments or have coursed offense with anyone.  When your really trying to get something to work and at every step it fails to do the job or falls down in some way its very frustrating and yes it makes you grumpy when you find out that the one element you need has deliberately not been included in any of the available options.  Well lets just say it didn't help my grumpiness.

When I can I will setup a dedicated backup server but that will not be in the very near future.

When it comes to Christmas? I don't know its all a bit over rated for me. maybe your right?

I have made a lot of comments based on my short experience with sme.  Previously I used a (debian etch ispconfig) setup for pure web hosting but wanted something more.  SME offered the something more but at (From my point of view) a cost to the ease of management for websites and backup.  Its strange to use ibays and then the rpms don't use ibays and email setups are a bit strange from whats normally found on web servers so its a learning curve.

I can only hope that something of what I have said is helpful and once again I am sorry is I upset anyone, especially the developers.

Thanks
Steve

[raem]

steve1084

...after trying a full restore from what was supposedly a full backup of each of the systems was that they mostly failed to restore the databases properly or didn't backup all the folders files and we could not determine why.....The standard backup doesn't do full backups so its not really usable.

Are you aware of the backup & restore concept for sme server ?

The typical meaning of a sme "full backup", is a backup of all config files, data files & mysql db's, but not the installed applications and system files. Enough information is included in the "full" backup to allow the server to be rebuilt from the backup. The requirement is that the backup be restored to a fresh installation of the sme server operating system, and then any add on contribs need to be reinstalled.

Only a tape backup backs up every file on the server, but a tape restore does not restore every file, it only restores the files mentioned above. Again the requirement is to restore to a freshly installed OS and then reinstall add on contribs.
You can manually restore any file from a tape backup that is not part of the automatic restore, if you wish.

Only a bare metal clone image type backup will backup everything (every file) on the server.

If using the standard sme backup, or dmays dar2 backup or even the e-smith with dar backup, these will all include a mysql dump (as part of the pre backup event), and that mysql dump will be included in the backup. If your databases are stored in mysql tables, then they should be backed up.
If you have installed stuff in /opt then you may need to tweak settings to include that folder in the backup.
dmays dar2 is fully configurable (from the server manager panel) as far as selecting which folders to include or exclude from the backup). It is NOT advisable to select every file/folder on your server to be backed up as you are likely to have  problems running the backup job.
The default selection set in a new backup job for dar2 is usually sufficient.

As far as your claims re databases and folders not being restored, then I can only say you must be using the programs incorrectly as I have used all three and they work correctly as designed, which also assumes you are using them correctly (in keeping with the concepts mentioned above).

The e-smith backup with dar would be a good candidate for your requirements as it creates an inital full backup and then much smaller daily incrementals. The incrementals would be much easier/smaller to transfer across the Internet on a regular basis by ftp/sftp/rsync/ssh or whichever way you prefer.

You didn't mention trying this, so perhaps you better give that a go.
http://wiki.contribs.org/Backup_with_dar
The contrib is currently being tested for inclusion into the base release, but for now it is an add on.

BackupPC was the only one that truly worked properly.

I don't use that, but my understanding was that it was for backing up PC workstations to the server, I don't see how that would be useful for backing up the server ?

I believe a bug report was posted

Please provide a link to this bug report.

[byte]

The contrib is currently being tested for inclusion into the base release, but for now it is an add on.

Ray,

The contrib JPL releases is now available via smetest, will have possible bugs, so any bugs please open a bug ticket, at the moment it's under testing conditions and you can find it in the smetest repo's the more testing done with the rpm in smetest the quicker this can be released for inclusion in to the base release.