Koozali.org: home of the SME Server

Fetchmail retrieves all email each time I reboot server

Offline Stefano

  • *
  • 10,836
  • +2/-0
Fetchmail retrieves all email each time I reboot server
« on: May 27, 2022, 08:38:20 AM »
Hi all, first post since a quite long time :-)

I already opened a bug (see https://bugs.koozali.org/show_bug.cgi?id=11979)

I have a SME10 fully updated with smeserver-fetchmail which retrieves emails from a bunch of remote mailboxes via pop3s protocol.. so far, so good.
All accounts are configured in ghost mode and all emails are forwarded to another (remote, O365) server.. and, again, everything is working as aspected; iow, I have no local mailbox nor email stored on my server.


But every time I reboot my server, fetchmail starts to retrieve all emails from the beginning and I'm talking about 15k emails.. users are not happy about it :-)

this is an example of user configuration:
Code: [Select]



[root@srvsmtp users]# db accounts show fm-comcli
fm-comcli=user
    City=Ottawa
    Company=XYZ Corporation
    Dept=Main
    EmailForward=local
    FM-Accounts=mbox.cert.legalmail.it,SPOP3,xyz,zxy,
    FM-DebugOption=YES
    FM-Ghost=YES
    FM-KeepOption=YES
    FM-MailCopyTo=
    FM-NumBox=1
    FM-TransName=target_email@externaldomain.tld
    FirstName=pec
    ForwardAddress=
    LastName=comunicazioni clienti
    PasswordSet=no
    Phone=555-5555
    Street=123 Main Street
    Uid=5008
    VPNClientAccess=no
is there anything I am missing? is it the expected behaviour?Thank you in advance

Offline mmccarn

  • *
  • 2,626
  • +10/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #1 on: May 27, 2022, 12:44:40 PM »
It looks like the default template supports 'fetchall', 'keep' or 'no fetchall' based on the value of "FM-KeepOption" for each user.

Quote from: /etc/e-smith/templates/etc/fetchmail/90fetchmail
my $keepopt   = $account->prop( 'FM-KeepOption' )  || '' ;
...
my $keep='fetchall';

if($keepopt eq "YES") { $keep='keep'; }
elsif($keepopt eq "NEW") { $keep='no fetchall'; }
 


If so, setting FM-KeepOption to NEW will delete mail from the remote server after downloading.


fetchall   -a      Fetch all messages whether seen or not
keep   -k      Do not delete seen messages from server (for POP3, uidl is recommended)
no keep   -K      Delete seen messages from server (default)
« Last Edit: May 27, 2022, 12:53:34 PM by mmccarn »

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #2 on: May 27, 2022, 03:15:32 PM »
thank you for your answer.
Unfortunately I can't delete emails from remote server.

since we (me and the customer) are talking about auto archiviation of remote mailbox, maybe I can enable such a feature in the (hopefully near) future.

Just a question more: if I use IMAPS protocol and use the NEW parameter, will emails just be flagged as deleted or purged?

finally, when it says
Quote
(for POP3, uidl is recommended)
is there anything I can check?

Thank you

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #3 on: July 26, 2022, 05:49:14 PM »
ok, revamping this old one.
did some tests..
fetchmail stores the ids of fetched emails in /var/lock/fetchmail/.fetchids file (/var/lock/fetcmail is defined in /etc/startmail file as FETCHMAILHOME)

setting an account as follow
Code: [Select]
FM-Ghost=YES
FM-KeepOption=YES

works as desired, but if I reboot the server, the .fetchids file is deleted and recreated on the frist run, so that all the mail are fetched again.

tried editing /etc/fetchmail file using the -i flag, i.e from
Code: [Select]
/usr/bin/fetchmail -l75000000 -d0 --verbose --syslog --fetchmailrc - <<

to
Code: [Select]
/usr/bin/fetchmail -l75000000 -d0 --verbose --syslog -i /mynewpath/.fetchids --fetchmailrc - <<

where mynewpath is something like (example) /var/lib/fetchmail, with qmailr:qmail owner
moved .fetchids to the new location, tried running startmail.. everything is ok
then rebooted the server and checked the file is still here

so, here are my questions:
- I dind't find where the file is deleted.. is it something related to /var/lock?
- am I missing anything?

Will write everything on bugzilla too

TIA

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #4 on: July 27, 2022, 10:42:46 AM »
ok, at the end I had my answers :-)

/var/lock is a symlink to /run/lock, which is a tmpfs filesisystem:

Code: [Select]
[root@svrticket var]# pwd
/var
[root@svrticket var]# ls -la lock
lrwxrwxrwx. 1 root root 11 29 giu 10.03 lock -> ../run/lock
[root@svrticket var]# mount | grep run
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=404508k,mode=700)

and it's not a good place where to store files that should really survive to reboot.
according to fetchmail's manpage, usually .fetchids is stored into the user's home dir

I see also that /var/lock/fetchmail is defined in /usr/lib/tmpfiles.d/fetchmail.conf
Code: [Select]
[root@svrticket tmpfiles.d]# more fetchmail.conf
d /var/lock/fetchmail 0755 qmailr qmail

which is owned by e-smith-email rpm
Code: [Select]
[root@svrticket tmpfiles.d]# rpm -qf fetchmail.conf
e-smith-email-5.6.0-17.el7.sme.noarch

so, should I create another dir with same owner/group just to store the .fetchids file? if yes, which is the best place?

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Fetchmail retrieves all email each time I reboot server
« Reply #5 on: July 27, 2022, 04:15:39 PM »
/run is runtime folder.  all inside is deleted including folders except the one in defined in the tmp.d files. 


could you bug that against e-smith-mail and also against fetchmail contrib ?

good job digging that one Stefano.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #6 on: July 27, 2022, 04:29:42 PM »
/run is runtime folder.  all inside is deleted including folders except the one in defined in the tmp.d files. 

could you bug that against e-smith-mail and also against fetchmail contrib ?

good job digging that one Stefano.

Hi JPP.. bug is https://bugs.koozali.org/show_bug.cgi?id=11979 and it's against smeserver-fetchmail.
I think that the /run path is ok for the pid file and so, IMVHO, there's no need to change e-smith package.



Offline david000

  • ****
  • 196
  • +0/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #7 on: August 06, 2022, 11:47:40 AM »
But every time I reboot my server, fetchmail starts to retrieve all emails from the beginning and I'm talking about 15k emails.. users are not happy about it :-)

I see this as well, since starting with SME8 or 9.  It's only one mailbox for me so I just try and remember to move the mail to a sub folder in the pop box, but usually forget.

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Fetchmail retrieves all email each time I reboot server
« Reply #8 on: August 06, 2022, 03:53:18 PM »
Hi David, thanks to Stefano investigations and fix proposal we have. a fix published both for contribs and contrib. 

you can take the time to test the core fix and contrib they are respectively in smeupdates-testing and smetest

yum update e-smith-email smeserver-fetchmail —enablerepo=smetest,smeupdates-testing

then you can report on both bugs.

this is the last bug awaiting verification before release of sme10.1

Offline david000

  • ****
  • 196
  • +0/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #9 on: August 08, 2022, 10:43:23 AM »
Hi David, thanks to Stefano investigations and fix proposal we have. a fix published both for contribs and contrib. 

you can take the time to test the core fix and contrib they are respectively in smeupdates-testing and smetest

yum update e-smith-email smeserver-fetchmail —enablerepo=smetest,smeupdates-testing

then you can report on both bugs.

this is the last bug awaiting verification before release of sme10.1

Thanks Jean-Philippe,   I've tried to install the fix by following your command:

Code: [Select]
yum update e-smith-email smeserver-fetchmail —enablerepo=smetest,smeupdates-testing
but i get the response

Code: [Select]
No Match for argument: —enablerepo=smetest,smeupdates-testing
No packages marked for update

My apologies if I'm missing the obvious  ?


Offline bunkobugsy

  • *
  • 274
  • +4/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #10 on: August 08, 2022, 01:10:49 PM »
that's a double minus --enablerepo=
relevant bugs are 12131 and 11979

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Fetchmail retrieves all email each time I reboot server
« Reply #11 on: August 08, 2022, 07:35:17 PM »
yes long options start with double minus sign.

unfortunatly my phone took the liberty to format the -- as a —

Offline david000

  • ****
  • 196
  • +0/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #12 on: August 11, 2022, 02:11:03 PM »
yes long options start with double minus sign.

unfortunatly my phone took the liberty to format the -- as a —

I've just installed that and rebooted\reconfigured. All Good, no unwanted mail collected. I've added a note to the bug report as above :)

Offline david000

  • ****
  • 196
  • +0/-0
Re: Fetchmail retrieves all email each time I reboot server
« Reply #13 on: August 11, 2022, 02:43:35 PM »
just reading another thread which says:

Quote
Warning - Be aware you should never perform a full yum update with the /smetest and/or /smedev repos enabled it will almost certainly put your server in a state that may not be recoverable from..

I usually update using #yum update     as the server manager update wasn't working.

Is it safe to say that the reference to test and dev repos used to install the fix above isn't permanent ?

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Fetchmail retrieves all email each time I reboot server
« Reply #14 on: August 11, 2022, 05:08:45 PM »
all you install is permanent.


so yes you should not do a full yum udpate without saying what packages you want with dev and test repo. 

smedev will break your install.  there are few abandoned rpm there because they did not pass the dev stage. 

smetest, can break your install. upstream update and freshly baked rpm of released rpm are there. 

smeupdates-testing should be safer, some of us run our server with it enabled, but packages need to be confirmed safe before releasing. and to test rule is one or two at atime if there are dependencies.