Koozali.org: home of the SME Server
Contribs.org Forums => Koozali SME Server 10.x => Topic started by: pmulroney on November 22, 2021, 02:19:32 AM
-
Hi there,
We're trying to get some of our guys to work from home, and they need to access our server remotely. We setup an iBay with public access via web set as "Enter internet (password required outside local network).
When he goes to use the web-based system, fail2ban kicks in and blocks him. There's a new entry in the http-auth jail.
IP BanTime UnbanTime Jail
xx.xx.xx.xx 2021-11-22 09:01:47 2021-11-23 09:01:47 [http-auth]
looking at the /var/log/httpd/error_log I see entries like the following:
[Mon Nov 22 09:01:47.276096 2021] [access_compat:error] [pid 16149] [client xx.xx.xx.xx:52435] AH01797: client denied by server configuration: /home/e-smith/files/ibays/ld_external/html/timekeeper/TaskList.php, referer: https://mail.logicaldevelopments.com.au/ld_external/timekeeper/
I don't think he's getting the domain login dialog box when he goes to access the pages.
This worked flawlessly for many many years under previous versions of SME, and i'm fairly sure it worked when we first upgraded to SME 10.
Where do I start looking to solve this problem?
Regards,
Paul.
-
get his ip, see the hit he did to understand where it triggered fail2ban
you can explore
/var/log/httdp/error_log
/var/log/httdp/access_log
/var/log/fail2ban.log
if you do not understand why and he has a fixed IP whitelist his ip (i think the panel is broken for that, so you will have to do it command line)
if he is always ont he same provider, you can at try whitelist the provider's block, with some risks
alternativelly, a vpn, like wireguard...
-
Hi Jean-Philippe
I checked with him, and he IS getting the domain login dialog box when he goes to access the pages.
/var/log/httpd/error_log has 17 entries similar to this one:
[Mon Nov 22 09:01:47.276096 2021] [access_compat:error] [pid 16149] [client xx.xx.xx.xx:52435] AH01797: client denied by server configuration: /home/e-smith/files/ibays/ld_external/html/timekeeper/TaskList.php, referer: https://mail.logicaldevelopments.com.au/ld_external/timekeeper/
/var/log/httpd/access_log has entries similar to this one:
logicaldevelopments.com.au xx.xx.xx.xx - ld_external [22/Nov/2021:09:01:47 +0800] "GET /ld_external/timekeeper/TaskList.php?usSeq=xx HTTP/1.1" 200 25124 "https://mail.logicaldevelopments.com.au/ld_external/timekeeper/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0"
/var/log/fail2ban/daemon.log has entries similar to this:
2021-11-22 09:01:42,888 fail2ban.filter [55395]: INFO [http-auth] Found xx.xx.xx.xx - 2021-11-22 09:01:42
2021-11-22 09:01:42,893 fail2ban.filter [55395]: INFO [http-auth] Found xx.xx.xx.xx - 2021-11-22 09:01:42
2021-11-22 09:01:47,237 fail2ban.filter [55395]: INFO [http-auth] Found xx.xx.xx.xx - 2021-11-22 09:01:47
2021-11-22 09:01:47,279 fail2ban.filter [55395]: INFO [http-auth] Found xx.xx.xx.xx - 2021-11-22 09:01:47
2021-11-22 09:01:47,538 fail2ban.actions [55395]: NOTICE [http-auth] Ban xx.xx.xx.xx
2021-11-22 09:01:47,565 fail2ban.filter [55395]: INFO [recidive] Found xx.xx.xx.xx - 2021-11-22 09:01:47
For now, he logs in via VPN so he can access the services, but we have a number of clients who also access our issue tracking system in a similar way in the same iBay - they haven't reported any issues yet but that could also mean that they haven't tried.
-
So he does have the ibay auth box and try to auth and fails ?
is he using :
- its own username password as SME user,
- the ibay name and dedicated password or
- the username/password of the protectect app ?
the rule triggered is
^client (?:denied by server configuration|used wrong authentication scheme)\b
I would check the actual settings of the ibay in the httpd.conf
-
He gets the ibay auth box, and he authorises, and then he says that it displays, but then a few minutes later he can't get access anymore.
He's using an ibay name and dedicated password
In httpd.conf is the following:
<Directory /home/e-smith/files/ibays/ld_external/html>
SSLRequireSSL
Options None
Options +Indexes
Options +Includes
AllowOverride None
order deny,allow
deny from all
allow from 127.0.0.1 192.168.X.0/255.255.255.0
AuthName "External functions"
AuthType Basic
AuthBasicProvider external
AuthExternal pwauth
require user ld_external
Satisfy any
</Directory>
... and a bit later on ...
<Directory /home/e-smith/files/ibays/ld_external/html>
SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
<FilesMatch \.(phar|php)$>
<If "-f %{SCRIPT_FILENAME}">
SetHandler "proxy:unix:/var/run/php-fpm/php-ld_external.sock|fcgi://localhost"
</If>
</FilesMatch>
</Directory>
-
can you check rights on this file and subfolders. www user need to be able to access it.
/home/e-smith/files/ibays/ld_external/html/timekeeper/TaskList.php
-
can you check rights on this file and subfolders. www user need to be able to access it.
/home/e-smith/files/ibays/ld_external/html/timekeeper/TaskList.php
ls -la TaskList.php
-rwxrw-r-- 1 pmulroney ld_team 19271 Aug 17 10:18 TaskList.php
All the files in that folder are the same permissions and ownership.
Note that on the internal network, the whole team uses this app constantly throughout the day with no issues. It's only when we try to access it from an external network that we have issues,
-
and what are the ownership / right for those files? exact same as folder ?
is www part of ld_team group?
is fail2ban set to whitelist the lan ip? that would explain why not ban when accessing from inside
is there any error poping in log when browsing same page from the lan ?
ok seems this is important https://client.ashdowntech.com/2016/03/ah01797-client-denied-by-server-configuration/
so might be the situation where the lan is whitelisted. or perhaps the warning only pops when logged in
-
and what are the ownership / right for those files? exact same as folder ?
is www part of ld_team group?
www is not part of the ld_team group - the ld_team group was setup using the SME Server Manager interface, so www isn't an option to include in the group.
is fail2ban set to whitelist the lan ip? that would explain why not ban when accessing from inside
yes.
is there any error poping in log when browsing same page from the lan ?
No errors when accessed from the lan.
ok seems this is important https://client.ashdowntech.com/2016/03/ah01797-client-denied-by-server-configuration/
so might be the situation where the lan is whitelisted. or perhaps the warning only pops when logged in
I setup this iBay many years ago when SME Server was called eSmith, and the config has rolled through several upgrades, the latest being SME 10. Maybe there's something in the combination of fail2ban and domain login and the current version of apache? Or maybe it's just time to retire the domain login, and rely on fail2ban alone stopping hackers from trying to access our support services.
Thanks for your help, much appreciated!
Regards
Paul.
-
ok. this is a bug.
using old httpd syntax without fail2ban does work, but when using the login way outside of the network a compat warning is added to error log.
this triggers the fail2ban regex.
this needs a bug.
2 ways
- against fail2ban to add an exclusion pattern. i think it is possible. with the compat part of the line.
- against core to use the new syntax.
will probably need also to fix few contribs the same way. thinking to the mod_dav one as example
-
the temp workaround would be to create an apache-auth.local file with an. ignore regex in it that catches the compat line
i just meed to test that actually failing to login still is logged and catch byf2b.
-
ok. this is a bug.
using old httpd syntax without fail2ban does work, but when using the login way outside of the network a compat warning is added to error log.
this triggers the fail2ban regex.
this needs a bug.
Thank you for this - I don't know how writeup the bug report, is that something that you would do?
-
post here https://bugs.koozali.org/enter_bug.cgi?product=SME%20Server%2010.X
and report the same way you did there with all the elements we discovered.
nothing magical, just repporting what you did, what you expected, and what happened.
-
Bug is done, https://bugs.koozali.org/show_bug.cgi?id=11774
Just added all notes from here...
-
Bug is done, https://bugs.koozali.org/show_bug.cgi?id=11774
Just added all notes from here...
Thank you Terry - much appreciated!
-
I am hit by the same bug and was wondering what is the best workaround? Should I try to patch the httpd.conf file with the correct syntax, or is there another way of doing it? JPP mentions creating an "apache-auth.conf.local file with an ignore regex", but I do not see how that can be done.
-
best way would be to update the fragments with the incriminated elements
mkdir -p /etc/e-smith/templates-custom /etc/httpd/conf/httpd.conf
starting with
90e-smithAccess40ibays
cp /etc/e-smith/templates/etc/httpd/conf/httpd.conf/90e-smithAccess40ibays /etc/e-smith/templates-custom /etc/httpd/conf/httpd.conf
then edit
vim /etc/e-smith/templates-custom /etc/httpd/conf/httpd.conf/90e-smithAccess40ibays
reading this https://httpd.apache.org/docs/2.4/upgrading.html
and make all needed changes suggested
test for obvious errors
expand-template /etc/httpd/conf/httpd.conf
httpd -t
if this does not any error you can do a live test to see you are prevented to what you should be
systemctl restart httpd-e-smith
and when ok post here : https://bugs.koozali.org/show_bug.cgi?id=11774
if you have extra motivation you can also do :
./VirtualHosts/27ManagerProxyPass
./VirtualHosts/28phpkiProxyPass
./85HordeAccess
./96ProxyConf
./85ImpAccess
./85IngoAccess
./85TurbaAccess
./86MnemoAccess
./35ProxyPass
./86NagAccess
./98nextcloud
./85DefaultAccess
./85ServerResourcesAccess
./90e-smithAccess10icons
./95ProxyService
./87KronolithAccess
./80DenyHTAccess
-
best way would be to update the fragments with the incriminated elements
Thanks a lot for the fast response and detailed description of a solution! I will look into it as soon as possible and feedback my findings to the forum and update the bug report.
-
best way would be to update the fragments with the incriminated elements
Thanks again for the instructions JPP!
I did the following:
cp /etc/e-smith/templates/etc/httpd/conf/httpd.conf/90e-smithAccess40ibays /etc/e-smith/templates-custom/etc/httpd/conf/httpd.conf
vim /etc/e-smith/templates-custom /etc/httpd/conf/httpd.conf/90e-smithAccess40ibays
expand-template /etc/httpd/conf/httpd.conf
httpd -t
systemctl restart httpd-e-smith
The 90e-smithAccess40ibays has been changed in such a way that what used to be this fragment in httpd.conf
<Directory /home/e-smith/files/ibays/familiealbum/html>
SSLRequireSSL
Options None
Options +Indexes
Options +Includes
AllowOverride None
order deny,allow
deny from all
allow from 127.0.0.1 192.168.185.0/255.255.255.0
AuthName "Familie bilder"
AuthType Basic
AuthBasicProvider external
AuthExternal pwauth
require user familiealbum
Satisfy any
</Directory>
Now looks like this:
<Directory /home/e-smith/files/ibays/familiealbum/html>
SSLRequireSSL
Options None
Options +Indexes
Options +Includes
AllowOverride None
Allow from 127.0.0.1 192.168.185.0/255.255.255.0
AuthName "Familie bilder"
AuthType Basic
AuthBasicProvider external
AuthExternal pwauth
<RequireAny>
Require user familiealbum
Require ip 127.0.0.1 192.168.185.0/255.255.255.0
</RequireAny>
</Directory>
The require block might be RequireAll depending on the config of the ibay.
By changing to the new syntax there is no more access_compat error in the httpd error log and fail2ban is no longer blocking the ibay.
I will update the bug report with this text and a version of my 90e-smithAccess40ibays file.
-
great !
update with the fragment content, not the output of the fragment ;)
-
great !
update with the fragment content, not the output of the fragment ;)
If you by that mean the code fragment, I uploaded the 90e-smithAccess40ibays file as an attachment to the bug report, see https://bugs.koozali.org/attachment.cgi?id=6655.