Koozali.org: home of the SME Server

Pre upgrade ponderings

Offline david000

  • ****
  • 193
  • +0/-0
Pre upgrade ponderings
« on: August 06, 2021, 04:28:05 PM »
I'm looking at migrating to SME10 but have a couple of questions first.

I have server manager backups scheduled everyday to network storage. According to the documentation I should be able to install SME 10 and restore from one of these.  Are there any pit falls in this approach ?

I currently have 2 identical machines, one of which is an Affa backup to supplement manage back up. My loose plan is to take the backup machine and use it to upgrade to 10 and if all looks good use it as the production machine.  I can then hopefully set up an affa backup when I've upgraded the second machine.

My concern is that the machines in question are somewhat old - 2007 spec Dells (Athlon 64 X2 5000+ 2.6 GHz) but with additional memory and additional HDD for Raid.  The system is currently using 65GB of disk space across 5 users.     

Too old - or OK on the basis it still works ?

Any comments \ thoughts appreciated.
 

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #1 on: August 06, 2021, 05:36:26 PM »
globally this is a good plan. test before going live.


you will have to test to see how the hardware is supported. The installer is likely to be the pitfall. it tends to support less hardware than what would support the final software. 

Network backup could be used to restore from the manager. I would  however suggest to make a console backup as it should be faster to restore from local rather than network. 

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #2 on: August 06, 2021, 05:49:35 PM »
Network backup could be used to restore from the manager. I would  however suggest to make a console backup as it should be faster to restore from local rather than network.

ok - I guess plugging in a removable USB hard drive should facilitate that ?  Taking a quick look now, the console asks you plug in removable media before scanning for it.

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #3 on: August 06, 2021, 10:21:19 PM »
if already plugged just ignore the message and proceed. 


use usb2 at least or it will be really looooooooooooong. 


also try to update to last available version before backuping. we did few e-smith-backup upgrade in the last year that could potentially be helpful.

if you have yum erroring not able to find valid repos. disable all repos except smeupdates and update smeserver-yum first.

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #4 on: August 07, 2021, 05:51:01 PM »
Thank you. Good point on the usb2.

I'll give it a go and see how I get on.

Offline sages

  • *
  • 182
  • +0/-0
    • http://www.sages.com.au
Re: Pre upgrade ponderings
« Reply #5 on: August 08, 2021, 07:38:28 AM »
My 2c worth.
Seriously consider separating the OS and data onto different drives. The install and update was a fairly quick process, backing and restoring the data was painful and I had about  1TB of data.
I did a server upgrade at the same time. Due to the time to backup data I decided to leave my SME9 running until I had my new sever online. And I wanted to reuse the drives in my SME9 which was set up with raid 1/2 drives only. So I had to migrate my sme9 to another drive, recover the drives, build a new sme10 and restore data.
Pretty much used up a week as my backups (tried USB(old hardware only supported usb2), nfs, aoe, took best part of a day for a backup then another day for a restore) with hindsight if adding another drive to the server had been an option then I'd have used that.
Given that I only do a server upgrade every 4-5 years  :P it isn't the end of the world but I'm heading down the path of an ~120GB ssd for the operating system and two rust drives in raid 1 for data mounted at /home/e-smith/files/shares (or ibays) depending upon where you keep your data.
Any upgrade or OS rebuild should just require a relatively quick process as the main data drives would be untouched. I believe that there is a few other install out there with this setup. ymmv
...

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #6 on: August 08, 2021, 09:28:36 AM »
And what you describe is my setup, was on old sme9 system made the move simple, sans raid on data drives

I removed data drives from fstab, restart and backed up that, restore to new sme10, add fstab entry restart reinstall contribs.

Does the job
« Last Edit: August 08, 2021, 09:32:11 AM by TerryF »
--
qui scribit bis legit

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #7 on: August 08, 2021, 04:57:43 PM »
I second the advice.  ;)

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #8 on: August 09, 2021, 06:06:48 PM »
Thanks Sages, Terry.

It seems that the PC boxes only have old slow USB ports so as you say,  backing up to a portable drive could be a slow process.

I have raid 1 configured on the boxes so do have a 'spare' drive on the second box.  In the console screen they are described as md0 and md1.   

This could give the option of separating out the data and the OS and ditching the raid configuration until I upgrade the box.   

As mentioned, I do have daily Grandfather, farther, son backups sent to a network drive.

I'm not super technical with Linux so please excuse my inability to 'join the dots' on this (let alone type meaningful command line stuff !).

Terry, are you saying that because you were running separate OS and data drives, you basically backed up the data drive 'just in case' but were able to use it directly once the OS drive was updated to 10 ?

If so, it seems to make sense to work out how to separate out the data from the OS onto the separate drives I have ?

Given the second box I have is an 'Affa' back up, perhaps one option would be to 'raise' it and try the modification and update on that box ? 

One potential issue should it all go poo poo is that I have some ibay data I'd like to keep.





Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #9 on: August 11, 2021, 02:12:00 AM »
Terry, are you saying that because you were running separate OS and data drives, you basically backed up the data drive 'just in case' but were able to use it directly once the OS drive was updated to 10 ?

If so, it seems to make sense to work out how to separate out the data from the OS onto the separate drives I have ?

Given the second box I have is an 'Affa' back up, perhaps one option would be to 'raise' it and try the modification and update on that box ? 

One potential issue should it all go poo poo is that I have some ibay data I'd like to keep.

Didn't even backup the data drive, just disable the fstab entry for it, did the backup of the system, remove all HDs from box
new ssd - install sme10 and at prompt during first reboot offer the sme9 backup, all good, once sme10 is rebooted and updated, put data drive back in box

edit fstab and enable the data drive, reboot, reinstall contribs..

Probably dont need to remove the data drive, just being careful  All up an hour or so, OS backup was about 45mb, data drive is almost 700gb :-)
« Last Edit: August 11, 2021, 02:14:02 AM by TerryF »
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #10 on: August 11, 2021, 02:21:23 AM »
Thanks Sages, Terry.

If so, it seems to make sense to work out how to separate out the data from the OS onto the separate drives I have ?

Given the second box I have is an 'Affa' back up, perhaps one option would be to 'raise' it and try the modification and update on that box ? 

One potential issue should it all go poo poo is that I have some ibay data I'd like to keep.

When I setup the data HD and setup fstab, I followed the wiki, once I rebooted I simpley copied all of the data from the sme9 system from the following as that was what I had setup in fstab  /home/e-smith/files/shares  I used cli and switches to also copy perms/dirs etc, then deleted the original data location, backup was then the 45mb, of course made sure data on data drive was all there before the big D :-) and accessable

this is my fstab entry, I use the UUID instead of vol name etc, going back probably would do ext4 or xfs now

UUID=566cf197-313c-4109-95f7-155d4913a747 /home/e-smith/files/shares  ext3    usrquota,grpquota,,noatime,acl        0 0

--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #11 on: August 11, 2021, 09:49:30 AM »
Didn't even backup the data drive, just disable the fstab entry for it, did the backup of the system, remove all HDs from box, new ssd - install sme10 and at prompt during first reboot offer the sme9 backup, all good, once sme10 is rebooted and updated, put data drive back in box

Got it.

I'll have a read of the wike and see how it looks.

Offline sages

  • *
  • 182
  • +0/-0
    • http://www.sages.com.au
Re: Pre upgrade ponderings
« Reply #12 on: August 11, 2021, 10:09:43 AM »
Thought I'd finally change the config on my server to separate the data onto the raid 1 drives and load the OS onto a ssd.
Bit of a hardware saga. I nutted out a procedure to hopefully allow moving data and reconfiguring things and just degrading the original raid array and then building a new degraded array, copy data from drive to drive and finally rebuilding adding the remaining original raid disk into a new array. Basically avoiding data backup to external storage.
Best laid plans.
I knew my motherboard only had two sata ports (sata 2 as it turns out) so added a dual pcie sata 3 card. Debated where the new ssd should be fitted. ie onto one of the motherboard sata 2 ports or onto the sata 3 add on ports. Finally decided that the hdd should stay on the sata 2 ports as they weren't fast enough to benefit from sata 3 and the ssd would maybe benefit from it. Turns out the motherboard won't let me select the add on sata 3 card as a boot option.
It all seemed much easier before I pulled the hardware apart.  :?
Now I need to move some cables around to reconfigure the drives and see what limitations (if any) moving the ssd onto a sata 2 port will have.
...

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #13 on: August 11, 2021, 10:29:04 AM »
Now I need to move some cables around to reconfigure the drives and see what limitations (if any) moving the ssd onto a sata 2 port will have.

I had hells own trouble recently when I tried to use  a new samsung 883 DCT and then a 870 EVO to upgrade to SME10 on my little N54L microserver, fail, could install but then suffered with sata errors, poor little thing was to slow :-)

Went back to a Seagate Barracude SSD, no probs.

It was just the age of my hardware
--
qui scribit bis legit

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #14 on: August 11, 2021, 01:56:09 PM »
good advice from Terry to remove or at least unplug the data drive to avoid the installer to erase it, and plug it back after the install is successful. 

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #15 on: August 20, 2021, 05:05:31 PM »
I'm still plucking up the courage to have a go at this.   Given the age of the boxes I'm using,  does anyone have a suggestion for a cheap new box to install too ?

I usually pick up cheap Dells for office stuff but it seems you can no longer buy without an OS, although I could live with that. Maybe something like this:

https://www.dell.com/en-uk/work/shop/desktop-computers-for-businesses/vostro-desktop-3681/spd/vostro-3681-desktop/s304vd3681bts01_2101

Edit - in an effort to actually make a start I've started a test install on an old PC I had sitting about. The installer thew an error during the initial selections a couple of times but 3rd time it went to the actual install.

Or so it seemed.  On reboot it drops into an endless loop quickly coming back to the start menu over again. If I select the recovery option it makes no difference but I can get to the command line.

I'll give it another go and see how I get on.   The hardware on this box may be out of the ordinary as it was from a smll UK maker called 'Mesh'. Maybe different hardware will be better.

« Last Edit: August 20, 2021, 08:00:02 PM by david000 »

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #16 on: August 20, 2021, 08:20:12 PM »
installer is python. let it settle down on nee screen. do not rush to click on anything before screen stop to refresh. 

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #17 on: August 20, 2021, 10:47:22 PM »
I'm still plucking up the courage to have a go at this.   Given the age of the boxes I'm using,  does anyone have a suggestion for a cheap new box to install too ?

Go to ebay and pick up a proper secondhand server.

Old rack servers are 2 a penny.

Get one with plenty of RAM and a few drives and run Proxmox.

HP - look for say a tower ML310 or 350, or in racks a DL380G7 or Dell equivalent.

You can buy a ton of 2/h enterprise drives for messing about.

You will never go back to using an old desktop as a server....
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #18 on: August 21, 2021, 05:21:14 PM »
Thanks both.

I'll have a look at a s/h server.  A friend runs an IT recycling company so I expect they should have some kicking about.

The test install failed. The install goes through ok but in restart it gets into an endless reboot loop.  I'll need to try and work out how to extract the log to a USB drive (my command line skills are lacking) but for info the screen output mentions a 'Firmaware bug' AMD v1 IOAPCIO not in IVRS Table and No southbridge IOAPICfound.

It's not a problem as I'll try with something else.


Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #19 on: August 24, 2021, 05:09:05 PM »
I apologize for asking but google has failed to to provide.

I've a 1tb usb drive which thought I'd try and back up to from the console. I've formatted it on my PC but the SME admin console doesn't recognise the USB file system as valid.

I can see the drive listed as   sdc - sdc1:

Quote
[root@backup-sme ~]# lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                      8:0    0 931.5G  0 disk
├─sda1                   8:1    0   250M  0 part
│ └─md0                  9:0    0   250M  0 raid1 /boot
└─sda2                   8:2    0 931.3G  0 part
  └─md1                  9:1    0 931.1G  0 raid1
    ├─main-root (dm-0) 253:0    0 929.3G  0 lvm   /
    └─main-swap (dm-1) 253:1    0   1.9G  0 lvm   [SWAP]
sdb                      8:16   0 931.5G  0 disk
├─sdb1                   8:17   0   250M  0 part
│ └─md0                  9:0    0   250M  0 raid1 /boot
└─sdb2                   8:18   0 931.3G  0 part
  └─md1                  9:1    0 931.1G  0 raid1
    ├─main-root (dm-0) 253:0    0 929.3G  0 lvm   /
    └─main-swap (dm-1) 253:1    0   1.9G  0 lvm   [SWAP]
sr0                     11:0    1  1024M  0 rom
sdc                      8:32   0 931.5G  0 disk
└─sdc1                   8:33   0 931.5G  0 part

I assumed that reformatting the disk would fix the format issue but I'm struggling to find the correct commend or access the "sdc1"

If anyone has a suggestion or can point me to a an online resource I'd be grateful. I've tried the #sudo mkfs.vfat /sdc1  command but I suspect mkfs isn't part of the system ? 

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #20 on: August 24, 2021, 07:23:49 PM »
mkfs is part of SME.

Offline warren

  • *
  • 291
  • +0/-0
Re: Pre upgrade ponderings
« Reply #21 on: August 24, 2021, 08:29:48 PM »
I apologize for asking but google has failed to to provide.

I've a 1tb usb drive which thought I'd try and back up to from the console. I've formatted it on my PC but the SME admin console doesn't recognise the USB file system as valid.

I can see the drive listed as   sdc - sdc1:

I assumed that reformatting the disk would fix the format issue but I'm struggling to find the correct commend or access the "sdc1"

If anyone has a suggestion or can point me to a an online resource I'd be grateful. I've tried the #sudo mkfs.vfat /sdc1  command but I suspect mkfs isn't part of the system ?

You dont need to use sudo

Code: [Select]
# mkfs.vfat  /dev/sdc1

Look here for USB disks : https://wiki.koozali.org/USBDisks

I would rather use/do format it as ext3:
first check what existing partitions are on it
Code: [Select]
# fdisk /dev/sdc

Delete any existing dos vfat partitions see => https://wiki.koozali.org/AddExtraHardDisk#Partitioning_the_disk
Partitioning the disk.

then do :
Code: [Select]
# mkfs ext3 -L MyLabel /dev/sdc1

« Last Edit: August 24, 2021, 08:31:50 PM by warren »

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #22 on: August 25, 2021, 02:36:01 AM »
On any used disk I always clear it using  # dd if=/dev/zero of=/dev/sd? bs=512 count=1 before setting up and formating..most of my testing now is is VM but when I was using bar metal above was essential when swapping disks about.

Just makle sure you know which drive you are nuking :-)

--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #23 on: August 27, 2021, 12:14:33 PM »
Thanks all, I'll take another look over the weekend and see how I get on.

Best


David

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #24 on: September 03, 2021, 04:33:34 PM »
You dont need to use sudo

Code: [Select]
# mkfs.vfat  /dev/sdc1
....

Thanks. That all helped a lot.

I managed to format the USB and it's now trying to back up 55Gb over a basic USB port so I'll see how that goes.  I've ordered a s/h newer box to install the V10 on so I'll have a go next week.

Edit - All backed up from the admin console taking about 2.5 hrs.  Fingers crossed it will restore to the V10 once I have that set up.

 
« Last Edit: September 03, 2021, 07:28:24 PM by david000 »

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #25 on: September 07, 2021, 05:06:58 PM »
Update -

I've successfully install v10 on a new Del box using a USB stick and that went easily.

I backed up my live production server via the console to a USB drive (Putty in as admin - not server manager). The machine said that the backup would be around 55Gb and it backed up in about 3 hours.

On installing V10, I restored from the USB drive when prompted, but it only found a 33Gb backup file.  That seems to restore OK and now I've clicked the correct date sort,  I think all the email is there ok. 

I've yet to try and access the ibays

I've connected the new server to the network now with email collection disabled so I'll see if I can get roundcube and the rest installed.

I think, so far, so good....



« Last Edit: September 07, 2021, 05:43:35 PM by david000 »

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #26 on: September 07, 2021, 07:17:09 PM »
Did you actually check the file size?

If compressed it could be wrong.

No idea how Roundcube does things but think it is still just an Imap client like Horde.

So next take a look in your user mail directories.

/home/e-smith/users or similar. Use 'mc' to have a browse around and see if you can see newer mail files.

Could be the indexes have got trashed so worth a look.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #27 on: September 07, 2021, 11:25:50 PM »
Update -
I've successfully install v10 on a new Del box using a USB stick and that went easily.
---------snip------
I think, so far, so good....

 :pint:
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #28 on: September 08, 2021, 01:34:31 PM »
Did you actually check the file size?

If compressed it could be wrong.

I'm sure you are right and it's the difference between uncompressed and the compressed backup files.  Looking at the 'Disk Usage' I've installed to the server manager,  on both systems it's about the same.

I confess I managed to confuse myself when opening the webmail and found the email sorted oldest first..... /blush/

Testing continues.  The backup in the server manager put up a fight (cifs issue) but resetting the NAS to SMB2 minimum fixed that.

 

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #29 on: September 08, 2021, 03:47:12 PM »
A little more progress. The daily backup is up and running and seemingly backing up to a local NAS.

I'm trying to understand why email isn't being collected from my POP box under the "email reception settings".  Is there a log which would show if the current settings are being rejected, and why ?



« Last Edit: September 08, 2021, 04:15:29 PM by david000 »

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #30 on: September 08, 2021, 04:42:37 PM »
I'm trying to understand why email isn't being collected from my POP box under the "email reception settings".  Is there a log which would show if the current settings are being rejected, and why ?

Check the difference between servers - is this the same on both?

Code: [Select]
config show fetchmail
Logs for POP collection from your server:

Code: [Select]
tail -f /var/log/maillog
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #31 on: September 08, 2021, 05:45:17 PM »
without more explanation logs to check would be

dovecot
qpsmtp
fetchmail

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #32 on: September 09, 2021, 03:27:38 PM »
Thank you ReetP and Jean-Philippe,

I've spent a bit of time looking at this and rebuilt the server so that I could try accessing the 'live' pop box without making any modifications. However no change.

I am seeing a Cron email from Cron <root@sme10> /etc/startmail:

Code: [Select]
/var/lock/fetchmail/fetchmail.pid: No such file or directory
fetchmail: lock creation failed.

The maillog on the rebuild server has a repeated entry regarding spam checking so presumably no an issue. I've tried turning off spam checking just in case with no change:

Code: [Select]
Sep  9 12:43:12 sme10 spamd[5275]: config: configuration file "/usr/share/spamassassin/73_sandbox_manual_scores.cf" requires version 3.004004 of SpamAssassin, but this is code version 3.004005. Maybe you need to use the -C switch, or remove the old config files? Skipping this file
I've tried to check the other logs but I assume they are empty as the tail command doesn't open them, but they are in the log folder.

When I list the log directory there is no fetchmail log.   
I'm unable to see if the server is trying to make a connection or what the error code might be.


On this fresh build I've not installed any contribs,  although on the last build and test I did install:

DiskUsage
Fetchmail (to enable the 'Mail Retrieval' button in server manage and allow collection of mail from 2nd pop box.
using the command in the documentation:
Code: [Select]
yum --enablerepo=smecontribs install smeserver-fetchmail
signal-event console-save

In addition:
Code: [Select]
# /sbin/e-smith/audittools/newrpms
Returns the following for my working production server. 

Code: [Select]
Loaded plugins: fastestmirror, post-transaction-actions, smeserver
Loading mirror speeds from cached hostfile
 * smeaddons: www.mirrorservice.org
 * smeos: www.mirrorservice.org
 * smeupdates: www.mirrorservice.org
Extra Packages
imapsync.noarch                          1.684-1.el6                @epel
libmcrypt.x86_64                         2.5.8-9.el6                @smecontribs
perl-Data-Uniqid.noarch                  0.12-2.el6                 @epel
perl-IO-Socket-INET6.noarch              2.72-1.of.el6              @smecontribs
perl-IO-Tee.noarch                       0.64-1.el6                 @epel
perl-Mail-IMAPClient.noarch              3.37-1.el6                 @epel
perl-NTLM.noarch                         1.09-1.el6                 @epel
perl-Parse-RecDescent.noarch             1.965-1.el6                @epel
php-kolab-net-ldap3.noarch               1.0.7-2.el6                @smecontribs
php-pear-Mail-mimeDecode.noarch          1.5.5-1.el6                @smecontribs
php-pear-Net-IDNA2.noarch                0.1.1-3.el6                @smecontribs
php-pear-Net-LDAP2.noarch                2.1.0-1.el6                @smecontribs
php-pear-Net-Sieve.noarch                1.3.4-1.el6                @smecontribs
roundcubemail.noarch                     1.1.3-101.el6.sme          @smecontribs
roundcubemail_plugins.noarch             1.2-2.el6.sme              @smecontribs
smeserver-diskusage.noarch               0.2.0-2.el6.sme            @smecontribs
smeserver-dovecot-extras.noarch          0.1.1-2.el6.sme            @smecontribs
smeserver-fetchmail.noarch               1.6-15.el6.sme             @smecontribs
smeserver-roundcube.noarch               1.2-9.el6.sme              @smecontribs
sqlite.x86_64                            3.7.17-9.el6.sme           @smecontribs

The server is connecting to the internet as I'm able to send outgoing mail to the pop box I'm trying to collect from, but oddly not to email accounts outside of my 'reseller' hosting server. [Fixed by setting a valid 'from' email address in Horde !]

Could it be that a missing contrib is the issue ?  It seems that imapsync was installed to try and move some mail some time ago and I wonder of Dovecot-extras was related as we don't use IMAP.

I've since istalled fetchmail and diskusage but no change.  However the cron email message above is  is repeated 3 times, which matches the number of pop accounts to be checked as set in "email>email reception settings" and "mail retrieval".



 



« Last Edit: September 09, 2021, 05:49:58 PM by david000 »

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #33 on: September 09, 2021, 06:08:37 PM »
Remove the important bits from this and then paste here:

Code: [Select]
cat /etc/fetchmail
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #34 on: September 09, 2021, 06:10:49 PM »
Ooooh and actually this too:

Code: [Select]
cat /etc/startmail
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #35 on: September 09, 2021, 06:12:44 PM »
HMmm.

i can see this

[root@files startmail]# ll /var/lock

lrwxrwxrwx. 1 root root 11 Oct 16  2020 /var/lock -> ../run/lock

[root@files startmail]# ll /var/lock/fetchmail

ls: cannot access /var/lock/fetchmail: No such file or directory

That might be the issue. Not directory.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #36 on: September 09, 2021, 06:26:42 PM »
Ooooh and actually this too:

Code: [Select]
cat /etc/startmail

Thank you. I've removed all the pw and domain info:

Code: [Select]
[root@sme10 ~]# cat /etc/fetchmail
#!/bin/sh
#------------------------------------------------------------
#              !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------



# multi-drop configuration; fetchmail retrieves all mail from a remote POP mailbox:

/usr/bin/fetchmail --syslog --silent --auth password --fetchmailrc - <<EOF

set postmaster "postmaster@xxxxxxxx.local"
set bouncemail
set properties ""

poll xxxxxxxxxxxxxxxxx.com
envelope "Deliver-To:"
    protocol POP3
    no dns
    localdomains xxxxxxx.co.uk xxxxxxx.co.uk xxxxxxx.local xxxxxxx.co.uk
       username "test@xxxx.co.uk" password "xxxxxxxxxx" to * here
           smtphost 127.0.0.200
           fetchall
           forcecr

EOF
#
#   Mail N1 Account for : stuff
/usr/bin/fetchmail -l15000000 -d0 --silent --syslog --fetchmailrc - <<EOF
set postmaster "postmaster@xxxxxxx.local"
set bouncemail
set no softbounce
set properties ""
poll 172.16.16.74 with proto POP3 timeout 60 and options uidl no dns
  user 'admin' smtpaddress xxxxxxx.local with password 'xxxxxxxxxxxxxxxxx' is stuff here keep
  smtphost 127.0.0.200/25
  antispam 550
EOF
#
#   Mail N2 Account for : stuff
/usr/bin/fetchmail -l15000000 -d0 --silent --syslog --fetchmailrc - <<EOF
set postmaster "postmaster@xxxxxxx.local"
set bouncemail
set no softbounce
set properties ""
poll xxxxxxxxx.com with proto POP3 timeout 60 and options uidl no dns
  user 'djt\@xxxxxxxx.co.uk' smtpaddress mediacopy.local with password 'xxxxxxxxxxxxxxxxxx' is stuff here keep
  smtphost 127.0.0.200/25
  antispam 550
EOF
#
#

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #37 on: September 09, 2021, 06:30:27 PM »
HMmm.

i can see this

[root@files startmail]# ll /var/lock

lrwxrwxrwx. 1 root root 11 Oct 16  2020 /var/lock -> ../run/lock

[root@files startmail]# ll /var/lock/fetchmail

ls: cannot access /var/lock/fetchmail: No such file or directory

That might be the issue. Not directory.

I get the same:

Code: [Select]
[root@sme10 ~]# ll /var/lock
lrwxrwxrwx. 1 root root 11 Sep  9 09:48 /var/lock -> ../run/lock

[root@sme10 ~]# ll /var/lock/fetchmail
ls: cannot access /var/lock/fetchmail: No such file or directory
[root@sme10 ~]#



Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #38 on: September 09, 2021, 08:41:21 PM »
Hmmm. Odd. It should be there.

In the rpm spec file used to build e-smith-email we have:

Code: [Select]
%setup
mkdir -p root/var/lock/fetchmail

And then:

Code: [Select]
/sbin/e-smith/genfilelist $RPM_BUILD_ROOT \
    --dir /var/lock/fetchmail 'attr(755,qmailr,qmail)' \

Wonder if it should be actually created in /run/lock ??

I can see on a fairly standard v10 the same thing occurs.

You need to open a bug against e-smith/email and we can take a look at it.

Something like "fetchmail lock directory does not exist"
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #39 on: September 10, 2021, 06:14:19 AM »
recurrent bug on SME10 CentOS7 and newer

/var/lock/ is linked to /run/var/lock/

all content is deleted on reboot, even if rpm owned, except if declared somewhere ...

already opened bug for this particular one https://bugs.koozali.org/show_bug.cgi?id=11634

so is someone answer the question to get if fixed : in the contrib or in core ?

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #40 on: September 10, 2021, 06:25:32 AM »
fixed

solution was given by Terry

need testers  ;)

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #41 on: September 10, 2021, 08:15:08 AM »
fixed
solution was given by Terry
need testers  ;)

Yeah well way to much credited to moi..initial triage interupted by our big almost three week power outage, my old grey cells moved the bug into a side dark room, out of sight out of mind.. :-) bug is on the verificatuion table
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #42 on: September 10, 2021, 10:00:56 AM »
fixed

solution was given by Terry

need testers  ;)

Thanks all.  I may be able to help :D

I see the current download has notes from June, so will the fix appear via a server manager update ?


Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #43 on: September 10, 2021, 11:10:21 AM »
update is in /smetest, easiets and quickest way is to login to server with say putty and from the terminal

# yum update e-smith-email --enablerepo=smetest

Added: never ever do a straight yum update --enablerepo=smetest
« Last Edit: September 10, 2021, 01:11:51 PM by TerryF »
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #44 on: September 10, 2021, 11:26:04 AM »
update is in /smetest, easiets and quickest way is to login to server with say putty and from the terminal

# yum update e-smith-email --enablerepo=smetest

Cool, that seems to have installed. I'll update later on how it works :)

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #45 on: September 10, 2021, 01:39:27 PM »
Cool, that seems to have installed. I'll update later on how it works :)

Let us know if that fixes it please.

You can help by doing a proper verification:

https://wiki.koozali.org/SME_Server:Documentation:QA:Verification
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #46 on: September 10, 2021, 01:52:07 PM »
Let us know if that fixes it please.

You can help by doing a proper verification:

https://wiki.koozali.org/SME_Server:Documentation:QA:Verification

I'm just looking at it now.

While it appeared to have worked directly after updating and fetched a batch of mail from the 'catch all' email it appears to have been a 1 time collection and then stopped.

That seems to be confirmed by the cron email time stamps which skip 10 mins (I'm on a 5min collection) at that point before resuming as before with this message repeated 3 times as before:   

Code: [Select]
/var/lock/fetchmail/fetchmail.pid: Permission denied
fetchmail: lock creation failed. 

So I think still broken. I could try the yum again ?

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #47 on: September 10, 2021, 01:54:38 PM »
what gives

ll -d /var/lock/fetchmail
ll /var/lock/fetchmail


also paste here what you did to update.

i do not think fetchmail is restarted on e-smith-email update

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #48 on: September 10, 2021, 02:34:29 PM »
what gives

ll -d /var/lock/fetchmail
ll /var/lock/fetchmail

First attempt it asked for a pw but then locked me out. I restarted the Putty session and:
Code: [Select]
[root@sme10 ~]# ll -d /var/lock/fetchmail

drwxr-xr-x 2 root root 40 Sep 10 09:25 /var/lock/fetchmail
[root@sme10 ~]# ll /var/lock/fetchmail
total 0
[root@sme10 ~]#

Quote
also paste here what you did to update.

Code: [Select]
yum update e-smith-email --enablerepo=smetest
followed by

Code: [Select]
signal-event post-upgrade; signal-event reboot
I ran the yum command a second time and get a "No packages marked for update" response at the end.

Quote
i do not think fetchmail is restarted on e-smith-email update

The machine did reboot. I've rebooted again just to see but no change.
« Last Edit: September 10, 2021, 02:53:48 PM by david000 »

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #49 on: September 10, 2021, 03:15:44 PM »
my lock dir better yet

[root@fagehome ~]# ll /var/lock/
total 0
drwxr-xr-x 2 root   root   40 Sep 10 21:40 dmraid
drwxr-xr-x 2 qmailr qmail  40 Sep 10 23:15 fetchmail
drwx------ 2 root   root   60 Sep 10 21:39 iscsi
drwxrwxr-x 2 root   lock   40 Sep 10 21:39 lockdev
-rw------- 1 root   root    0 Sep 10 21:39 masq.adjust
drwxr-xr-x 2 root   root   40 Sep 10 21:39 ppp
drwxr-xr-x 2 root   root  100 Sep 10 21:40 subsys

so this is with changes to ownership
« Last Edit: September 10, 2021, 03:19:27 PM by TerryF »
--
qui scribit bis legit

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #50 on: September 10, 2021, 03:23:43 PM »
as a quick test, change contents of
/usr/lib/tmpfiles.d/fetchmail.conf

to
d /var/lock/fetchmail 0755 qmailr qmail

restart and let have it

+Added: new update does that
« Last Edit: September 10, 2021, 03:30:31 PM by TerryF »
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #51 on: September 10, 2021, 03:58:33 PM »
as a quick test, change contents of
/usr/lib/tmpfiles.d/fetchmail.conf

to
d /var/lock/fetchmail 0755 qmailr qmail

restart and let have it

+Added: new update does that

I did the edit with nano.

All seems OK on the restart. No cron emails, the catch all is collecting. Email appears to be sorted to the correct accounts. The other pop collection in 'mail retrieval' seems to be OK (but just testing again).

I think that would appear all to be working.

I'll let this run for a while before rebuilding it again and going live. Will this change pick up automatically now with the new update ?

 
 

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #52 on: September 10, 2021, 04:01:13 PM »
Good result, yes latest update in /smetest e-smith-email-5.6.0-13.el7.sme.noarch.rpm   has this change

Please post your results to the Bug
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #53 on: September 10, 2021, 04:26:39 PM »
Good result, yes latest update in /smetest e-smith-email-5.6.0-13.el7.sme.noarch.rpm   has this change

Please post your results to the Bug

Do I need to log a new bug or is it already logged ? I see one from mid Summer but I'm not sure it's the same as marked as fixed.  That said reading back, I think it must be this:

https://bugs.koozali.org/show_bug.cgi?id=11634
« Last Edit: September 10, 2021, 04:36:23 PM by david000 »

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #54 on: September 10, 2021, 04:35:54 PM »

Do I need to log a new bug or is it already logged ? I see one from mid Summer but I'm not sure it's the same as marked as fixed.
as stated in previous page :


already opened bug for this particular one https://bugs.koozali.org/show_bug.cgi?id=11634

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #55 on: September 10, 2021, 04:45:27 PM »
K - I've added some notes to the page. Let me know if I need to add more detail ?

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Pre upgrade ponderings
« Reply #56 on: September 10, 2021, 07:58:00 PM »
Nice work, thanks for the patience, and verification.

We need more 'doers' !!

If you want to get a bit more involved just ask DM me for a Rocket account here.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #57 on: September 10, 2021, 09:17:07 PM »
Nice work, thanks for the patience, and verification.

We need more 'doers' !!

If you want to get a bit more involved just ask DM me for a Rocket account here.

Glad to be able to help. I do have a rocket account and am glad to help if my limited skills (and time) can be applied. My linux is super basic but I was excited to be able to make the nano edit !!!!

All that said, I'm happy to test and assist to get roundcube up and running plus affa.  They are the missing links from my current v9 set up. 

I'll ping a hello back on rocket to make contact again.



Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #58 on: September 12, 2021, 04:50:01 PM »
Moving on.

The server has been 'up' for about 48hrs.  However the /webmail/ returned a 'service not available message when I tried to look at it this afternoon.  I thought the PC may have just gone into sleep mode but I think not - server manager was available. That said I did press the space bar on the keyboard so maybe that awoke it, but /webmail/ stayed off line.

On reboot /webmail/ is back.  I'll leave it overnight and report back.
 

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Pre upgrade ponderings
« Reply #59 on: September 13, 2021, 01:25:28 AM »
next time, check httpd/error_log

might encounter php74 fpm not reachable

if it is this a simple
systemctl restart php74-php-fpm.service



but still this should not happen.

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #60 on: September 16, 2021, 04:51:53 PM »
next time, check httpd/error_log

might encounter php74 fpm not reachable

if it is this a simple
systemctl restart php74-php-fpm.service

but still this should not happen.

I've left it up and running for a couple of days and all seems OK. I've not been able to reproduce the problem so fingers crossed it's was an anomaly.

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #61 on: October 13, 2021, 12:35:10 PM »
A quick update. Nothing to see here :)

I took the test box off line for a couple of weeks and brought it back up this morning. All seems fine still, the server manager update went without a hitch and I'm struggling to find a reason not to go 'live' with it. I'll keep it up for another week to keep an eye on it and find some time to set it up again as a production box.


Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #62 on: October 13, 2021, 03:15:48 PM »
always a nice feeling when it all goes as it should :-)
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #63 on: November 27, 2021, 12:34:00 PM »
After much procrastination I've built my production SME10, restored, added a couple of packages and checked what I can.

So far, so good as far as I can tell.  The work station backup is scheduled as before and will presumably continue to backup, eventually replacing the old SME9's.  (I do a rolling 3 time full backup)

Is there anything else I should quickly check before starting to collect mail ?

I can then turn off the old SME9 machine(s) ? (I run affa on a second machine, but that will have to wait).


Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Pre upgrade ponderings
« Reply #64 on: November 27, 2021, 10:11:57 PM »
Only thing of late as in last week or so has been the upstream changes made to dehydrated :-( caught a few by surprise with letsencrypt certs etc easily updated once it was known, would have been nice to have had a warning and leadtime etc, a bit MS like :-) have also been a couple of other updates to core packages of late but they were specific.

Other than that safety first, set old sme9 aside in cotton wool until happy :-)
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Pre upgrade ponderings
« Reply #65 on: November 28, 2021, 09:52:39 PM »
Other than that safety first, set old sme9 aside in cotton wool until happy :-)

Sounds a plan. All seems OK so far....