Topic: Backup systems that dont work

[steve1084]

Have now tried all the freely available backup options that I can find for sme and not one of them can do a full backup and send it off site via FTP.  This is very frustrating.

Dar2 doesnt work properly, cant send backup offsite via ftp and wont login to local work-group computers or my local ritmo workgroup server.

Plain dar doesnt have ftp ability and wont login to my local ritmo work-group server. also doesn't do full backup of all databases or allow any options of what to backup

BackupPC is the only one that makes a full and complete backup but it also misses the mark by not having any ability to send the backup off site via ftp and also wont sent files off via the local work-group or ritmo lan server.

The standard built in backup also has NO FTP and wont login to my local work-group computers or the ritmo lan server and simply doesn't do a full backup.

It seems that dungog has cornered the market in that he is the only person who had the brains to include ftp in his backup options.  Unfortunately at $300 or $1000 I cant afford to purchase anything he has to offer and that realy sux.

I doesn't mean to be cynical but cum on people this is pathetic.  FTP should be a standard option on all backup software.

Why call it backup when it doesn't do as the name suggests.  Backup is for data recovery when everything goes stupid. Its an emergency procedure and if the backup doesn't capture all thats needed to fully restore the server back to how it was prior to the disaster then plain and simple it ain't worth having.  If all its doing is backing up the config then its just stupid and worthless.

After two weeks of trial and error with all the available backup options I am very disappointed as I have not achieved a reasonable integrated solution.

My only current solution is to use BackupPC to create the files and to then use something like "cobian backup 8" to ftp the files off site on a regular schedule.  At least cobian can do differentials and incremental on the backed up files.

To All developers

Please include FTP as part of any backup options for all future versions of SME server and associated backup systems.  It may not be the (technically) best way to backup but I guarantee that it will always work where other options fall down.

PS I also have issues with USB not working on older hardware and it doesn't solve the issue of storing backups OFF SITE.

Thanks
Steve

[psoren]

Have now tried all the freely available backup options that I can find for sme and not one of them can do a full backup and send it off site via FTP.  This is very frustrating.

I think they don't support FTP since FTP is not secure.
Use rsync instead, it's encrypted via SSH and can send only the files that differs at each backup so it also saves bandwith and time. I have used it for several years and works fine.
For MYSQL database you can make a dump file and just add it to your backup, if its not already done by the backup solution you will use.

Per

[mmccarn]

[edit]
Stupid post by stupid user withdrawn.
[/edit]

[steve1084]

Discouraging ftp is just plain stupid.  Its not the most secure but its not an open book either.  Simply allow the use of SFTP if its that big an issue.  I would rather have a backup that works than not have one just because somebody doesn't like FTP because its not the most secure method around.  Whats better, no backup or an ftp backup.  Ill choose ftp backup any day.

I'm not a coder but I am lead to believe that adding ftp to the backup systems should be very easy for coders and given that its better to have something that works than something that doesn't work I can only ask that FTP or SFTP be included as an OPTION to send backup files OFF SITE or to a local FTP server.

Most people don't want to setup an Affa box or don't have a spare box just to overcome the short comings of the current Backup systems and if the option is not built in then restore can then become a pain so it needs to be one system, not two three or four different things all strung together via command line, thats stupid and not very user friendly.

Im not opening up my sme server to ftp so where is the issue. I have learned from the howtoforge.com website how to make ftp as secure as I want it to be.

Stop this paranoia and make the backup systems user friendly and above all easy and integrated and usable.

FTP is the workhorse of the internet so PLEASE include FTP or SFTP options into ALL backup systems

[byte]

FTP is the workhorse of the internet so PLEASE include FTP or SFTP options into ALL backup systems

There may not be a FTP server in the future release(s) in case anyone may try to build a backup contrib/script.

[steve1084]

Please explain why as this sounds like complete stupidity to me

[byte]

Please explain why as this sounds like complete stupidity to me

http://bugs.contribs.org/show_bug.cgi?id=2835

Why do you think its stupid to remove it ?

[steve1084]

Because anyone running there own server uses ftp on a daily basis.  As I have stated it is the stock standard daily workhorse for any Admin or client managing or using a server environment.

If you remove it from the standard install of sme the first thing that will happen is an rpm will be created to install it and it is then more likely than ever that it will become insecure as it will be less integrated with the sme system and the setup parameters will no longer be under the control of the sme development team who would otherwise have the ability to make it secure from the get go.

[byte]

Because anyone running there own server uses ftp on a daily basis.

I don't use FTP. And I doubt many others do as it really isn't best practice. Why do you think contribs.org doesn't offer contrib users ftp access anymore!?

[p-jones]

steve1084, what other mainstream server solutions/backup solutions do have software to send backups off site via FTP ??

or don't have a spare box just to overcome the short comings of the current Backup

This being the case, a FTP option would be of little use anyway...

[benny]

Hi I agree that fading out ftp is not good. I have sme server running with approx 50 sites on it, I cant run some of my customers site on here because they manage their own sites and use ftp to do so. I have had to resort to using a cms system for those who want to manage their site. This is all well and good but they are resource hungry and I have little time to educate the users how to administer the sites using cms

[cactus]

Hi I agree that fading out ftp is not good. I have sme server running with approx 50 sites on it, I cant run some of my customers site on here because they manage their own sites and use ftp to do so. I have had to resort to using a cms system for those who want to manage their site. This is all well and good but they are resource hungry and I have little time to educate the users how to administer the sites using cms

Have you read this part of the manual about FTP: http://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Chapter11#FTP ? It says it is possible... if not a bug should be filed against the manual.

[benny]

Yes I have read that but the problem is setting users and passwords to access the ibay, setting ftp that way only gets them into the home dir. the ony way they can access their ibay is if they have the admin credentials which will expose all the ibays.
Im not proficient with sme and hesitant to install mods

[steve1084]

When it comes to using SME there are I think 2 very different types of users
-------------------------------------------------------------------------------------------
Type 1

SMB Business user who has sme server in the office with a bunch of employees and 1 or 2 websites that are all maintained by one person from the local environment or vpn.

They use shared folders and a few extras add-ons, all very minimalistic.  For this person FTP is not so important but is nevertheless still used from time to time.  Each person has workgroup and vpn access so they can download there update files to their desktop and then simply drag and drop into the ibay folder to update there cms systems.

--------------------------------------------------------------------------------------------

Type 2

Can also be a business but is likely to be a home user has 10 to 100 websites all owned and maintained by different people that need to maintain their own sites.  They all use ftp as their main tool of choice to upload and download files from their Ibay or /opt/folder.  Why because it works and it’s easy to use and fairly secure.
There may also be work-groups and other add-ons for the local users but not for the website owners who are all remote users.  There only access is Via FTP.
These people don’t have vpn, or workgroup or root etc access so please tell me how they are going to maintain their websites without FTP.
-----------------------------------------------------------------------------------------------
The removal of ftp is outright lunacy, this is I am led to believe a community project where the users can have input into its development to some extent.

For me at least, if FTP is removed I would find continuing with the use of sme difficult if not impossible.
The use of CMS systems that have their own built in FTP is not a good choice and creates many problems.  Mostly this option allows only one to 10 files at a time so when it comes to the next upgrade of the cms you’re expecting people to upload 200 plus files one at a time.  That’s Complete lunacy, and totally impractical.  Not to mention that many cms systems just don’t have this option available.  Are we going to be forced to use only one cms system? There are more than 1,000 different cms systems out there so who gets to choose what a client can run as there cms.
If you’re going to cater ONLY to the type 1 user then please tell me now so that I can immediately drop all my plans to use and implement SME personally and for my clients.
I can only reiterate my comments that FTP is the workhorse of the website administrator even if it’s not used so much for server administration.  To remove FTP is a mistake that will cost sme dearly and lose patronage from many quarters.
You need to cater to the person who wants to use sme primarily as a web server environment with many websites maintained by individual owners just as much as you cater to the office user environment.  It’s what makes SME a great tool.  Take away the tools that make it a user friendly and ultra usable option and it will lose much of its appeal and support base.


Thanks
Steve

PS there are over 1,000 Linux based backup systems available from source forge that all offer ftp as an option.  Ill say that again  ITS AN OPTION.  For those that dont want to use it then they don't have to.

[steve1084]

Hi I agree that fading out ftp is not good. I have sme server running with approx 50 sites on it, I cant run some of my customers site on here because they manage their own sites and use ftp to do so. I have had to resort to using a cms system for those who want to manage their site. This is all well and good but they are resource hungry and I have little time to educate the users how to administer the sites using cms

Hi Benny

You might want to try this contrib from dungog.  It works really well. And if used correctly locks the user to the ibay.

http://wiki.contribs.org/FTP_Access_to_Ibays

Thanks
Steve

[steve1084]

[edit]
Stupid post by stupid user withdrawn.
[/edit]

Hi  Im sorry if my comments about affa box offended and caused you to remove your post.  I only want to comment that SME is in theory a community based project that is I hope attempting to cater to both the home user and the SMB markets.  The use of an AFFA box is perfectly correct in the right environment but for the home user that only has $100 to spend on a backup solution is probably not going to use this method.  They are more likely to want to use either an add-on usb drive for local storage or a cheap lan server that they can setup at there mates house and FTP there backup files to so they have there backups off site.

ie its not a mission critical situation for the smaller type user.  But backup is still something they want to have.

For many people the ability to send files off site via ftp is the most sensible, cost effective and easy to manage option to use.  Its up to the creator of the backup system to ensure that the files being sent are reasonably secure.  either they should be encrypted or at least zip/password protected.

[raem]

steve1084

I'm not sure I understand your demands re "must have ftp".

sftp is supported, which is a secure protocol, why would you use insecure ftp when sftp is available.

As far as a offsite backup solution is concerned, a ready to go answer does exist in the dungog commercial product.
Apart from that and other current sme server backup solutions, that is what is available now, no one else has developed the exact product you want (for free).
You have a number of choices, develop it yourself, pay someone to develop it and it then gets donated to the community, buy the commercial product.
You could also lodge a New Feature Request (NFR) at bugzilla and if the developers consider your representations worthy enough, then someone may develop it. Even better if some code (or procedural steps) is posted that can be used as a basis for further development & creation of a suitable rpm (by those who know how to package such stuff into an rpm).
I'm pretty sure it won't use ftp though.

You could also put together/hack current offerings (e-smith backup with dar) and create a script that securely transfers the backup files, using say ssh with public private keys, and the system would do it all automatically.
If you go back a long time and read old forum posts (search), you will find the basis of how to do something like that already published.

[stephen noble]

ftp passes usernames and passwords over the internet in plain text;
how can you protect the data when the hacker has your login details

When I moved to dungog.net/wiki I removed mention of the ftp transfer (weex).
I'd forgotten it was there, rsync over ssh is the way to remotely backup your server

[CharlieBrady]

I doesn't mean to be cynical but cum on people this is pathetic.

What's pathetic is people like you who complain so vehemently about a gift. Do you complain to your grandmother at Christmas that the socks you receive are the wrong colour?

If you don't like what is offered freely to you, you are free to make whatever improvements you think are necessary. Or to pay someone to implement exactly what you want.

[Daniel B.]

Hi. I'm the packager of backuppc rpm for sme. I'd just like to share my opinion on this.
I haven't integrated a tool to export the backups via FTP because it's not secure, but there are some other tools to export it on
- a remote host via rsync over ssh
- a remouvable media such as USB disk
- a local directory

Backups are part of the security of a system, but if you send it over the net via an insecure connexion, you expose all your system (well, it depends on the data you backup, but with the default configuration, backuppc and most of other backup contribs, all the accounts informations, including passwords are saved)

If you really need your backups to be sent via FTP, you can create archives in a local directory, and then push them on your FTP with a little script, it won't be too hard.


Then, for the FTP server on SME, I think having FTP as an option is convinient, because some people still use it, but removing it will "force" users to use more secure tools, like sftp. FTP shouldn't be used anymore nowadays, it's so easy to catch username and passwords. If we don't remove FTP from the base one day or another, of course, some people will continue using it, but if we remove it, users will learn to use secure connexions, it'll be a little harder at the begining, but with some time, it'll become the standard.

The only thing that should be added before removing FTP, is an option to chroot SFTP connexions (without it, any user will be able to browse the entire file system, of course, they'll be limited by the fs permissions). I've read a little about it, and it seems harder than FTP chroot, but it's possible.

[steve1084]

Firstly there are two very different issues being talked about here

One is that if a person wants a backup system with ftp abilities then its going to cost $1000 from dungog.  Or buy another computer for $600 or more and use rsync or affa.   Thats fine for a business but is not an option for many home users.

Due to the way SME is built its not easy to just go out and find any old backup system for Linux and implement it either, so your stuck to some degree with whats on offer and for the home user your stuck with the available tools and equipment at hand.

I don't disagree with there being some risk's involved with the use of plain ftp but it is used extensively by people and most web hosting uses ftp.  This is slowly being replaced by sftp and other options.  I dont have access to a second server for rsync or affa but I do have access to several ftp solutions even if thats not a perfect solution.  I believe these risks to be very small, but you are free to disagree.

Its about providing options to people from all quarters not just those that can afford to spend $1000 or $3000 or whatever on a backup solution.  BackupPC works great and so does sme but when the available options become elitist due to the high cost and don't cater to everyone then thats something I find I must speak up about.

Two When people mentioned that ftp would be removed completely from sme they did not mention that sftp would still be available (please confirm this) so maybe I was led a little astray but I would still rather have ftp at the moment then not have it.  As it allows me to use the equipment I already have available to me.

There is not a chance that I am the only person in this position so what I'm saying is that the available options are very limiting and It would be nice if more options and choices were available.

This is the first forum I have joined that is positively Anti FTP and yet there is every chance that every person on this forum uses or has used ftp.  Forcing people to spend Thousands of Dollars in not I think a good healthy attitude.  Whats better is to provide options that suite a greater number of people and where needed place the appropriate warnings to encourage best practice.  To be so anti FTP is bordering on paranoia to the extreme.

Ive never met anyone who was the victim of a man in the middle attach using ftp.  I have however met and read a lot about people who get bots on there computers that steel passwords etc.  Currently the most common method to have your server hacked is to keep your passwords etc on your windows box and it gets hacked.  Somebody gaining passwords etc via ftp man in the middle attacks is I believe very very low.

Generally there has to be a bot on the machine your connecting to or on the machine your using so either way encryption or not they will get something from you.


I believe everyone has a right to speak about there experiences even if some don't like it.  This is my experience and my frustrations with trying to get a solution that works with the equipment available to me.  Whats pathetic (Charlie) is when people cant accept that there are some that have not been able to find a solution to a problem that they feel should not exist. I'm sorry Charlie but I don't celebrate Christmas so your on your own with that one.

Thanks
Steve

[CharlieBrady]

Forcing people to spend Thousands of Dollars in not I think a good healthy attitude.

Nobody here forces anybody to do anything.

[Stefano]

BackupPC works great and so does sme but when the available options become elitist due to the high cost and don't cater to everyone then thats something I find I must speak up about.

so, where's the problem?
use backuppc, then wtite down a 5 rows bash script to export backups to tgz files an then ftp them..

google is your friend.. you'll find many, many examples..

just a question.. have you already made a donation to sme?

ciao

Stefano

[byte]

Two When people mentioned that ftp would be removed completely from sme they did not mention that sftp would still be available (please confirm this) so maybe I was led a little astray but I would still rather have ftp at the moment then not have it.  As it allows me to use the equipment I already have available to me.

No where did I say it "would" be removed I said in "future" releases it could be, reason I didn't mention sftp is because it would be unaffected as ftp and sftp are different and provided by different packages, therefore "if" ftp was to be removed then sftp would remain due to being different to ftp.

This is the first forum I have joined that is positively Anti FTP and yet there is every chance that every person on this forum uses or has used ftp. 

Not Anti-FTP, just trying to educate/show the more secure approach.

I'm sorry Charlie but I don't celebrate Christmas so your on your own with that one.

Everyone should celebrate Christmas

[steever]

One is that if a person wants a backup system with ftp abilities then its going to cost $1000 from dungog.

Dungog sells their standard life time update version for AUD 299 which is a good price considering you get access to all of Stephen's great software for use on one server.  $299 is a lot cheaper than $1000.

If Steve1084 needs this functionality on multiple servers because he has a number of customers then he can buy separate licenses for each server (at $299) and bill the client for "online backup".  But in this case, getting the AUD 999 license makes better sense anyway.

[raem]

steve1084

If you read the Affa Wiki carefully, you can also backup to a mounted filesystem or USB etc.
See the topic "Use Affa to backup to a NFS-mounted NAS or a local attached USB drive"

From there it should be a relatively simple matter to rsync/ssh to a remote server.

[imcintyre]

Steve1084;

I didn't want to stick my nose into this "discussion" but, in your very first post you mention that;

Dar2 doesnt work properly....

but I did not see this expanded upon or a bug # mentioned. Is there a bug already raised for the problem you speak of or a specific issue that should be known about?

[steve1084]

Steve1084;

I didn't want to stick my nose into this "discussion" but, in your very first post you mention that;but I did not see this expanded upon or a bug # mentioned. Is there a bug already raised for the problem you speak of or a specific issue that should be known about?

Myself and a friend tested all available backup solutions and found several deficiencies according to our environment.

We created a test server environment with several users and several joomla test sites.  Installed and tested dar dar2 backupPC etc etc

What we found was that after trying a full restore from what was supposedly a full backup of each of the systems was that they mostly failed to restore the databases properly or didn't backup all the folders files and we could not determine why.  I probably need to look into it further.  The standard backup doesn't do full backups so its not really usable.  BackupPC was the only one that truly worked properly.

I believe a bug report was posted

As to the issue of the price for access to the dungog rpms yes it is only $300 for a "single server".  But as in time I would like to have more than one server the real cost is $1000.  Its not like he sells each of the rmps for $20 or has a club yearly subscription for $30.  It makes what is supposedly a free system quite expensive if you choose to go down that path.  Admittedly its still a hell of a lot cheaper that a Microsoft alternative.  I just cant afford or Justify the cost when I am still in the testing phase with sme.

The only off site storage available to me ATM is a plain FTP option.

The samba option would not login to the "ritmo lan server" so that option didn't work. and the local windows machines are not suitable for accepting the backups as they are not always on.  The older test machine we used for testing dosnt fully support usb but that would probably have worked on a slightly better machine. at least for locally stored backups.

So the option I have left is to use backupPC to create the backup files and to send them off site with another tool or script of some sort.  It simply would have been easier and less frustrating if plain FTP had of been available as part of the backup programs even if it was an extra install with lots of warnings.  This is a bit messy and will probably entail hand removal of the older files from time to time? ie its not an integrated solution.

At first I thought I would be able to use cobian backup to do the offsite transfers but It turned out that It doesnt have ssh abilities only ssl so it couldn't login to the sme server. so I'm still hunting around for something like cobian backup that has both plain ftp and ssh abilities to run either on the sever or on a windows machine.

I have now deployed one sme server for a local business and when I get paid a percentage will be donated to sme.

My apologies if I have managed to misinterpret any persons comments or have coursed offense with anyone.  When your really trying to get something to work and at every step it fails to do the job or falls down in some way its very frustrating and yes it makes you grumpy when you find out that the one element you need has deliberately not been included in any of the available options.  Well lets just say it didn't help my grumpiness.

When I can I will setup a dedicated backup server but that will not be in the very near future.

When it comes to Christmas? I don't know its all a bit over rated for me. maybe your right?

I have made a lot of comments based on my short experience with sme.  Previously I used a (debian etch ispconfig) setup for pure web hosting but wanted something more.  SME offered the something more but at (From my point of view) a cost to the ease of management for websites and backup.  Its strange to use ibays and then the rpms don't use ibays and email setups are a bit strange from whats normally found on web servers so its a learning curve.

I can only hope that something of what I have said is helpful and once again I am sorry is I upset anyone, especially the developers.

Thanks
Steve

[raem]

steve1084

...after trying a full restore from what was supposedly a full backup of each of the systems was that they mostly failed to restore the databases properly or didn't backup all the folders files and we could not determine why.....The standard backup doesn't do full backups so its not really usable.

Are you aware of the backup & restore concept for sme server ?

The typical meaning of a sme "full backup", is a backup of all config files, data files & mysql db's, but not the installed applications and system files. Enough information is included in the "full" backup to allow the server to be rebuilt from the backup. The requirement is that the backup be restored to a fresh installation of the sme server operating system, and then any add on contribs need to be reinstalled.

Only a tape backup backs up every file on the server, but a tape restore does not restore every file, it only restores the files mentioned above. Again the requirement is to restore to a freshly installed OS and then reinstall add on contribs.
You can manually restore any file from a tape backup that is not part of the automatic restore, if you wish.

Only a bare metal clone image type backup will backup everything (every file) on the server.

If using the standard sme backup, or dmays dar2 backup or even the e-smith with dar backup, these will all include a mysql dump (as part of the pre backup event), and that mysql dump will be included in the backup. If your databases are stored in mysql tables, then they should be backed up.
If you have installed stuff in /opt then you may need to tweak settings to include that folder in the backup.
dmays dar2 is fully configurable (from the server manager panel) as far as selecting which folders to include or exclude from the backup). It is NOT advisable to select every file/folder on your server to be backed up as you are likely to have  problems running the backup job.
The default selection set in a new backup job for dar2 is usually sufficient.

As far as your claims re databases and folders not being restored, then I can only say you must be using the programs incorrectly as I have used all three and they work correctly as designed, which also assumes you are using them correctly (in keeping with the concepts mentioned above).

The e-smith backup with dar would be a good candidate for your requirements as it creates an inital full backup and then much smaller daily incrementals. The incrementals would be much easier/smaller to transfer across the Internet on a regular basis by ftp/sftp/rsync/ssh or whichever way you prefer.

You didn't mention trying this, so perhaps you better give that a go.
http://wiki.contribs.org/Backup_with_dar
The contrib is currently being tested for inclusion into the base release, but for now it is an add on.

BackupPC was the only one that truly worked properly.

I don't use that, but my understanding was that it was for backing up PC workstations to the server, I don't see how that would be useful for backing up the server ?

I believe a bug report was posted

Please provide a link to this bug report.

[byte]

The contrib is currently being tested for inclusion into the base release, but for now it is an add on.

Ray,

The contrib JPL releases is now available via smetest, will have possible bugs, so any bugs please open a bug ticket, at the moment it's under testing conditions and you can find it in the smetest repo's the more testing done with the rpm in smetest the quicker this can be released for inclusion in to the base release.

[steve1084]

The problems with dar are being discussed http://forums.contribs.org/index.php?topic=38935.0  Im assuming he placed a bug report but may not have gotten round to it or still testing?? 

backupPC seems to be more comprehensive in all areas of backup so thats what I have chosen to go with, and yes I am aware that it can do more than just backup the localhost, just another reason to use it I suppose?

backupPC gives me almost all the options I want and was only tried because of issues with restoring the dar backups.  Biggest issues were with the restore of the databases for the joomla websites and the files from the joomla websites.  you may be right in that it could have been a human error issue but either way I am sticking with backupPC and its overall more comprehensive abilities as my main line of defence.

I might still do a bit more testing with dar but only if I get time.


Thanks
Steve

[CharlieBrady]

The only off site storage available to me ATM is a plain FTP option.

Both http://rsync.net/ and http://ibackup.com/ (for example) offer rsync backup.

[stephen noble]

> I just cant afford or Justify the cost when I am still in the testing phase with sme.

It's still not recommended but for you information.
If the application you were looking at was smeserver-weex it's been GPL since Jan 07....

http://sme.dungog.net/packages/smeserver/7.0/i386/html/index_dungog.html