Koozali.org: home of the SME Server

Affa is broken - while I fix it, how do you use it?

Offline ReetP

  • *
  • 3,871
  • +5/-0
Affa is broken - while I fix it, how do you use it?
« on: September 16, 2022, 02:53:14 PM »
So following on from the thread opened by Gary:

https://forums.koozali.org/index.php/topic,54879.0.html

A recent update to rsync that fixed a number of security holes in it broke Affa backup.

If you have recently updated your Affa server your backups of remote servers are now failing - only your logs will tell you.

I have rewritten some of Affa - which is a great big sprawling monolithic perl script - to cope with the changes, and whilst there I found a few other bugs too. The scripts have been added to the bug.

https://bugs.koozali.org/show_bug.cgi?id=12165

PLEASE help test.

Onwards.

One thing Affa can do is back up generic linux servers - SME was an add-on originally.

In rewriting the script I have essentially left out generic linux servers and just focused on SME.

By question was whether anyone still uses it to back up generic linux boxes?

If so I need to go back and think about how to manage that.

A bit more explanation.

Before you affa could run rsync in this format:

Code: [Select]
rsync ip.address.of.target: /etc/templates /etc/templates-custom /home/e-smith
You can no longer do that and have to make each directory of file specific like this:

Code: [Select]
"rsync ip.address.of.target:/etc/templates" "ip.address.of.target:/etc/templates-custom" "ip.address.of.target:/home/e-smith"
Now, SME has some backup routines which were used by Affa, but they used the LOCAL directories on the Affa server as a reference for directories to back up on the REMOTE server. Not great and a bit of a bug really, but it got away with things.

However, a recent change to the SME Backup routines means that it now includes directories for contribs as well, and they may not be the same on the remote server as the local server.

So part of the rewrite was to run the Backup routine on the remote sever, find the directories it wanted backed up, and then made a note of that with the backup so it would correctly restore them as well - you have to specify each directory to restore.

So I have done all that.

However, I have have modified the usage of the Included/Excluded directories as well. rsync uses them for pattern matching but for SME I just use them as literal include/exclude on top of directories that the Backup routine automatically suggests. (I still need to add the exclude part but that is trivial).

And this brings me to the main point. Standard Linux boxes.

It has no backup routines.

Do I drop the old Include/Exclude meaning like SME and say use / as default and then Include/Exclude as per the configuration file?

Or something else? Does anyone even care?

Any suggestions appreciated before I go out and break all your installs.

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline jameswilson

  • *
  • 790
  • +0/-0
    • Security Warehouse, trade security equipment
Re: Affa is broken - while I fix it, how do you use it?
« Reply #1 on: September 23, 2022, 12:46:22 AM »
Quote
And this brings me to the main point. Standard Linux boxes.
It has no backup routines.
Do I drop the old Include/Exclude meaning like SME and say use / as default and then Include/Exclude as per the configuration file?
Or something else? Does anyone even care?
Any suggestions appreciated before I go out and break all your installs.

I care
I only use affa to backup sme boxes, so i personally dont use the non sme options

Id be quite happy for it to be sme only in this case

Offline JohnG

  • ***
  • 88
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #2 on: September 23, 2022, 10:19:33 PM »
Thanks so much for your efforts!

And I care. At the moment I'm only using Affa with SME boxes.

At one point Affa was backing up a standard centos box and it worked well but it got retired. And I've always wanted to try Affa with a windows machine but never got further than installing rsync on windows then I got sidetracked.

So yeah, if it's easier to modify the sprawling perl to only do SME boxes then I'm OK with that, but it would be nice in the future to be able to Affa other generic operating systems as well.

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #3 on: September 24, 2022, 02:53:12 AM »
Ok.

I'll try and make sure some generic linux works.

Windows - theoretically should but I believe the original notes sad it needed cygwin.

Needs a tester....

Talk to me on Rocket if you want to help test the latest.

Remember, if your affa server is a v10, and you have installed the latest rsync update on that affa server, you have no backup....

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline jameswilson

  • *
  • 790
  • +0/-0
    • Security Warehouse, trade security equipment
Re: Affa is broken - while I fix it, how do you use it?
« Reply #4 on: September 24, 2022, 11:13:05 PM »
Quote
Remember, if your affa server is a v10, and you have installed the latest rsync update on that affa server, you have no backup....
That is the concern!

Thanks for all you do Reet and sme team

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #5 on: September 24, 2022, 11:52:44 PM »
That is the concern!

It is. Check your logs. You can force a downgrade of rsync as a temporary fix.

Quote
Thanks for all you do Reet and sme team

Thanks. We try. Please jump in on Rocket and help test.

New version is close to working - it basically does - but needs a lot of testing.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline nicolatiana

  • *
  • 724
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #6 on: November 30, 2022, 07:30:00 PM »
SME Boxes only.
Consulente di Smeserver.it -  Soluzioni e supporto su Sme server in Italia.

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #7 on: November 30, 2022, 10:38:35 PM »
SME Boxes only.

Link to 4.0-9 is on Rocket. Please test.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline david000

  • ****
  • 200
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #8 on: December 17, 2022, 01:05:56 PM »
I would like to install Affa but I'm technically not experienced enough to comment on the original question.

I can say that I would only use it on my dedicated SME10 box and run a second box with affa to back up daily.

Offline JohnG

  • ***
  • 88
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #9 on: March 09, 2023, 06:33:13 PM »
Quick Affa status; upgraded about a month ago on 10.1 from Affa 3.3.1-5 to 4.0.0-7, all backups from 3 SME boxes look good, haven't tried any restores or rises but the directories and files all look good, logs look fine, nice work!

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #10 on: March 09, 2023, 06:46:26 PM »
That version has probably got a load of bugs and probably won't rise or restore.

It is for testing only. Do not rely on it.

Unless you have followed on our Rocket.Chat instance you won't have latest which still has two bits I need to fix.

Ask if you want an account to test.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline JohnG

  • ***
  • 88
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #11 on: March 09, 2023, 07:39:16 PM »
Yep this is for testing. Haven't followed on rocket, i'd like an account thanks.

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #12 on: March 09, 2023, 09:07:48 PM »
Yep this is for testing. Haven't followed on rocket, i'd like an account thanks.

Cool.

Account - PM your proper nane & email address for notifcations.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #13 on: March 10, 2023, 10:27:06 AM »
And done!
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #14 on: March 12, 2023, 12:38:20 PM »
FYI I have attached the latest test version to bugzilla. V4.0.0-12

I think it cures most of the problems.

It needs a LOT of testing.

Please do NOT use it with a v3 backup. It needs a fresh start.

However, I have fixed issues with restore & rise but they need testing.

I have also properly enabled rising or restoring a specific backup eg weekly.2

By default they restore scheduled.0 but you can select another.

This should work.

Code: [Select]
affa --rise mybackup daily.3
If our tests show it's vaguely ok I'll build a rpm.

Let us know via Rocket or the bug.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline vassili

  • *
  • 9
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #15 on: July 03, 2023, 08:32:38 AM »
Hi, team, any news or updates on affa development ?

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #16 on: July 03, 2023, 10:15:24 AM »
Slow as I keep getting interrupted.

I think there is a v4.01 now in smedev but we know it still has some issues.

I think basic backup/restore works but some of the trickier stuff like restore a specific set - which was broken - still has some issues.

Please get a copy and test.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline vassili

  • *
  • 9
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #17 on: July 03, 2023, 10:18:36 AM »
Sure, I will set up some test systems, test and report back the result

Slow as I keep getting interrupted.

I think there is a v4.01 now in smedev but we know it still has some issues.

I think basic backup/restore works but some of the trickier stuff like restore a specific set - which was broken - still has some issues.

Please get a copy and test.

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #18 on: July 03, 2023, 11:12:05 AM »
If you DM or email me your name and an email address I can set you up on my Rocket instance and you can chat to us directly for testing.......

We know you'll find a few bugs and it will be easier to explain to you there how and what to test methodically.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Gary Douglas

  • *
  • 79
  • +1/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #19 on: April 26, 2024, 08:02:58 AM »
Way back on rocket one of my first comments was 'affa --rise is god'. I have tested and used AFFA4 extensively since then and this is still true. The RPM affa-4.0-1 in SMETEST has one small issue that affa --rise does not use hardlinks. It will still RISE if you have sufficient disk space. An earlier version 4.0-1a does RISE using hardlinks. The difference between 4.0-1a and SMETEST 4.0-1 is 4.0-1 has line 3852 commented, replaced with line 3853. 4.0-1a uses "--link-dest=$archive$src",
3852            # "--link-dest=$archive$src",
3853            ( $linkDest ? "--link-dest='$job{'RootDir'}/$jobname/$linkDest'" : '' ),
There may be other issues with affa4, there are many variables, undo RISE fails, restore old archive works but not tested recently.
My use and testing AFFA4 now includes RISE of several production servers, and other various purposes, i.e. saving my bacon.
An important Include if using i.e. MariaDB105 is; Include=/var/opt/rh/rh-mariadb105/lib/mysql
My sincere thanks to the dev team... my other gods!
« Last Edit: April 26, 2024, 08:20:30 AM by Gary Douglas »

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #20 on: April 26, 2024, 10:17:03 AM »
It's my fault - just overwhelmed with life   :cry:

I know where the issue is but it's linked to something else.

Just need to get my head back into it.

I haven't forgotten. Just time......
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Gary Douglas

  • *
  • 79
  • +1/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #21 on: April 26, 2024, 11:28:01 AM »
not your fault !!! Affa just saved my bacon to fix a 4month old problem, and now going to look at another nasty one which can't be fixed on the live box. Sync a copy minus data to a VM here, rise, then play.
« Last Edit: April 26, 2024, 11:34:07 AM by Gary Douglas »

Offline ReetP

  • *
  • 3,871
  • +5/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #22 on: April 26, 2024, 05:31:23 PM »
Thanks but I still feel guilty.

I haven't forgotten it........
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Gary Douglas

  • *
  • 79
  • +1/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #23 on: April 27, 2024, 10:10:02 AM »
thanks John. my other issue now fixed thanks to rise and play, in fact issue had disappeared on VM so updated live box

Offline bugzilla

  • 3
  • +0/-0
Re: Affa is broken - while I fix it, how do you use it?
« Reply #24 on: August 31, 2024, 05:15:44 AM »
So following on from the thread opened by Gary:

https://forums.koozali.org/index.php/topic,54879.0.html

A recent update to rsync that fixed a number of security holes in it broke Affa backup.


For anyone in ~Aug 2024, with SMEServer 10, who accidentally ran "yum update" ...


I test this by having two terminals open - one ssh'd to the affa server, the other ssh'd to the production server.
On the production server;
Code: [Select]
watch -n1 "ps aux|grep rsync"

On the affa server;
Code: [Select]
affa --run my_server_job

If the rsync appears on the production server, affa at least started. If it goes away in about 4 seconds, something crashed it, check your logs to see more info

To see if its the bad rsync version that is tripping up affa ...

First, confirm you have the bad rsync version;
Code: [Select]
rpm -qa|grep rsync
rsync-3.1.2-12.el7.x86_64

We are looking at the number after "3.1.2-"
"12" is bad. "11" is bad. "10" is good.

If you have 12 ...
Code: [Select]
yum downgrade rsync
This gets  you back to 11 ( rsync-3.1.2-11.el7_9.x86_64 )

Then downgrade again;
Code: [Select]
yum downgrade rsync

This should get you back to v10; confirm this;
Code: [Select]
rpm -qa|grep rsync
rsync-3.1.2-10.el7.x86_64

Now you should be good to go.

Now ... to stop it happening again;
Code: [Select]
yum install yum-plugin-versionlock
yum versionlock rsync-*

# check this at any time ...
yum versionlock status
# or
cat /etc/yum/pluginconf.d/versionlock.list

"yum update" will also warn you each time you run it that versionlock is holding a/some packages, and also remind you of the command to see those packages. Very handy!
D