Topic: An intruder!

[Ed Form]

I located a problem with an e-smith 4.1.2 installation today that appears to be the result of an intruder. Nothing was lost or stolen but the machine in question has run up a dial-up access bill of several hundred pounds over the few weeks since it was installed, of which about twenty pounds can be accounted for by client activity. The symptoms were only reported to me when the first of the phone bills came in, so the problem has been around for a while.

The server is connected to the Internet by an external serial modem and is set to gather mail every hour. The logs show that this has in fact been happening. But some other entity is causing the machine to dial constantly and to stay on line for long periods. In all over 1000 spurious calls have been made in a short period of time. A test with all the workstations shut down showed that these calls are not triggered by requests from the Windows 98 workstations but are originating at the server itself. With the modem switched off the server triggers diald at very frequent intervals and seeks the IP address 195.8.167.18 which is not known to Whois or any of the other registrar libraries. The ISP uses addresses in the range 195.8.179... so the fact that the spurious address is in the sector 195.8... is suspicious. The ISP is Business Serve who are usually ultra reliable so I guess the spurious address is an attempt to disguise the bad guys by hiding them in the trees.

Here are fragment from two diald logs...

Friday 11th January.

Fri Jan 11 14:00:00 2002 GMT: Calling site 195.8.167.18
Fri Jan 11 14:00:33 2002 GMT: Connected to site 195.8.167.18.
Fri Jan 11 14:01:43 2002 GMT: Disconnected. Call duration 70 seconds.
      IP transmitted 642 bytes and received 876 bytes.
Fri Jan 11 14:26:48 2002 GMT: Calling site 195.8.167.18.
Fri Jan 11 14:27:19 2002 GMT: Connected to site 195.8.167.18.
Fri Jan 11 14:27:53 2002 GMT: Disconnected. Call duration 34 seconds.
      IP transmitted 410 bytes and received 480 bytes.
Fri Jan 11 14:28:04 2002 GMT: Calling site 195.8.167.18.
Fri Jan 11 14:28:32 2002 GMT: Connected to site 195.8.167.18.
Fri Jan 11 14:29:27 2002 GMT: Disconnected. Call duration 55 seconds.
      IP transmitted 0 bytes and received 0 bytes....

Today - Wednesday 6th February.

Wed Feb  6 11:26:31 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:27:28 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:28:25 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:29:22 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:30:18 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:31:16 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:32:13 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:33:10 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:34:08 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:35:05 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:36:02 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:36:59 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:37:56 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:38:53 2002 GMT: Calling site 195.8.167.18.
Wed Feb  6 11:39:51 2002 GMT: Calling site 195.8.167.18...

The Squid store log is interesting. It consists of dozens of entries of the form...

1012999142.726 SWAPOUT 00000000  200 1012999142 964804111        -1 image/gif 227/227 GET http://e-smith.myclientsdomain.com:3128/squid-internal-static/icons/anthony-image.gif

And includes files GET requests for...

anthony-text.gif, anthony-dirup.gif, anthony-dir.gif, anthony-link.gif, anthony-sound.gif, anthony-movie.gif, anthony-portal.gif, anthony-box.gif, anthony-unknown.gif, anthony-ps.gif, anthony-compressed.gif, anthony-tar.gif, anthony-script.gif, anthony-dvi.gif, anthony-tex.gif, anthony-xbm.gif, anthony-xpm.gif, anthony-c.gif, anthony-binhex.gif, anthony-bomb.gif, anthony-image.gif,

Does anyone know what is going on here? In particular, does anyone know what process is trigerring the dialup? If it is a control program inserted from outside the system then this would seem to be a serious vulnerability in the e-smith setup.

I asked some weeks back if the firewall functions of e-smith protect against attack via a serial modem and was told that they do. It is certainly true that nothing has penetrated onto the local network, and when the server is on line asking for 195.8.179.18 no traffic ever moves in or out except during the first split second of the connection, which is the system talking CHAP to the ISP's registration server. But something in the software currently present on the server is triggering dialups and costing my client a fortune.

We originally had a dial-up router on this installation connected to Eth1 but problems with line quality confused us into thinking that it was faulty. I will be reinstalling the router tomorrow and cleaning up the mess, so I'm confident I can get rid of the parasite and keep it out.

Any comments? If this turns out to be as serious as it looks after you have had a chance to tell me I'm and idiot and it is all my fault, I'll report it as a bug and supply the rest of the details.

Ed Form

[Ed Form]

OK,

I've done some homework and found out that the anthony-X.gif files are actually part of SQUID. So that bit was a red herring, But what has got into the server to trigger these constant dialups?

I've checked the mail configuration very carefully and the IP that's being searched does not come from anything we've put in.

I could really do with some help on this one.

Ed Form

[robert]

Couldn't 195.8.167.18 simply be the ISP's end of the PPP connection? What is the dial-up policy you set up when you configured the server?

[guestHH]

Traceroute 195.8.167.18  takes you to the UK

[Garret]

Connecting to whois.arin.net...

Deferred to specific whois server: whois.ripe.net...

% This is the RIPE Whois server.
% The objects are in RPSL format.
% Please visit http://www.ripe.net/rpsl for more information.
% Rights restricted by copyright.
% See http://www.ripe.net/ripencc/pub-services/db/copyright.html

inetnum:      195.8.167.0 - 195.8.167.255
netname:      NC-INTERFACES
descr:        Customer Interfaces
country:      GB
admin-c:      DH6545-RIPE
tech-c:       DH6545-RIPE
status:       ASSIGNED PA
notify:       david.haworth@norwebtelecom.com
mnt-by:       NC-NOC
changed:      david.haworth@norwebtelecom.com 20010123
source:       RIPE

person:       David Haworth
address:      Your Communications
address:      Hathersage Road
address:      Manchester
address:      Lancs
address:      M13 0EH
address:      UK
phone:        +44 161 609 7307
fax-no:       +44 161 609 7300
e-mail:       david.haworth@yourcommunications.co.uk
nic-hdl:      DH6545-RIPE
changed:      david.haworth@yourcommunications.co.uk 20010926
source:       RIPE

[Charlie Brady]

Ed Form wrote:

> I located a problem with an e-smith 4.1.2 installation today
> that appears to be the result of an intruder.

If you believe that there is a security vulnerability in the server, then you should be sending a detailed report to security@e-smith.com, rather than shouting it from the rooftops. Please do so.

Thanks

Charlie

[JonB]

Have a read of this thread in the development mail list

http://www.mail-archive.com/devinfo%40lists.e-smith.org/msg07128.html

Jon

[Ed Form]

Charlie Brady wrote:
>
> Ed Form wrote:
>
> > I located a problem with an e-smith 4.1.2 installation today
> > that appears to be the result of an intruder.
>
> If you believe that there is a security vulnerability in the
> server, then you should be sending a detailed report to
> security@e-smith.com, rather than shouting it from the
> rooftops. Please do so.

I looked at the thread...

http://www.mail-archive.com/devinfo%40lists.e-smith.org/msg07128.html

...which was pointed out to me by Jon Blakely, and found that it appears to discuss a very similar problem. I note that you did not rattle Kees Blokland's cage about reporting that matter in the open forum. I also note that there was no follow up from any member of your team advising the community of a general fix for the problem, either in the devinfo forum or on your websites.

I don't think it a very good idea to waste the time of your bug or security folk with what might well be nothing more than a stupid error on my part, so asking here first, giving no information that could possibly aid the intruder fraternity, seemed like the best, first step. This is particularly the case because I am not using a purchased, supported, copy of the software, so I will not get any help from those I report the problem to - see the recent moans about this here..

I've asked here a number of times about another, non security related, aspect of the dial-up configuration of essg - the fact that it sends all mail immediately - which produces big phonebills for those who have no option than to use paid-per-call dialup access. None of your team bothered to indicate that a workround is available, and since, as I indicated, that aspect of the product is preventing me from buying supported copies for the 25 or so new installations I do each year, the neglect isn't particularly commercially sensible. I have two test installations with clients who are willing guinea pigs, and a third in my own office. Because of the country location in which we work, the vast majority of our customers are, and will always be, dialup users, which means that problems which produce a proliferation of unnecessary calls are very bad news. I'd be happier with a mild ticking off in the forum if one of you had bothered to respond to those earlier enquiries, even if you had only said: 'There is an easy fix for this Ed. Take a support contract and we'll sort it out.'

I now have two dial-up problems and one of them appears to be a resident call-home agent on the server itself. How much nicer if you had said: 'This looks like a genuine security issue. Let's take it off air and report it to security...' or 'This is a known user-misconfiguration issue, look at xxx for a fix.' Either of those responses, and some response, of any kind, to my earlier enquiry, would have helped me to make the important commercial decision to major on essg. Being treated like a painful wart in public tends to make EngardeLinux Workgroup look more and more attractive. Sadly that would cut off other commercially attractive propositions I've heard about in this community, so I really would like to go with essg. A lighter tone from Mitel folk would help.

Notwithstanding my bristling hackles, I will report this problem to the security folk today, and will be happy to give any further help needed for their investigation.

Ed Form

[Ed Form]

In article <624c29c12a3a1a7ee271085669011fd1.phorum-owner@e-smith.net>, admin@ruffdogs.com (Garret) wrote:

> This message was sent from: Experienced User Forum.
> http://forums.contribs.org/index.php?topic=12741.msg47894#msg47894
> ----------------------------------------------------------------
>
> Connecting to whois.arin.net...
>
> Deferred to specific whois server: whois.ripe.net...
>
> % This is the RIPE Whois server.
> % The objects are in RPSL format.
> % Please visit http://www.ripe.net/rpsl for more information.
> % Rights restricted by copyright.
> % See http://www.ripe.net/ripencc/pub-services/db/copyright.html
>
> inetnum:      195.8.167.0 - 195.8.167.255
> netname:      NC-INTERFACES
> descr:        Customer Interfaces
> country:      GB

Snip.

Thanks for that. It enabled me to contact the people who provide my clients ISP with comms facilities to report the problem. It turns out that the device at the end of 195.8.167.18 is not a computer at all and no one I have been able to speak to can imagine how our server could have obtained and be calling that address.

Ed Form

[Garret]

Glad to help

[Ed Form]

Thanks very much. I've decided to blow the entire server away and replace it with a different system in order to get rid of the problem

If I understand the thread you pointed me to correctly the basic problem was a spyware product on a Windoze workstation. Even if this problem had a similar cause, it looks odd that the triggering could continue on a newly rebooted server with no workstations connected to it at all.

Ed Form

[Rich Lafferty]

Ed Form wrote:
>
> I don't think it a very good idea to waste the time of your bug
> or security folk with what might well be nothing more than a
> stupid error on my part, so asking here first, giving no
> information that could possibly aid the intruder fraternity,
> seemed like the best, first step.

While I appreciate the sentiment -- please, waste my time.
I'm more than happy to spend time following up misplaced
security concerns. We know that investigation and reassurance
is a necessary component of producing a secure product and
have resources to dedicate to it.

I agree that this particular instance didn't reveal any
vulnerabilities, but it's really something we prefer to
evaluate ourselves, if only because we have resources beyond
those of a typical user that can be applied to tracking down
the origin of the unusual behavior. The potential damage
of an observation reported to us as a vulnerability
is much lower than that of a vulnerability posted here as
an observation.

As for tone, we pride ourselves on the security of our
product, which is one of the features we promote heavily;
I trust you understand why our reaction to public announcements
which inaccurately suggest that our claims of security
are false might be negative.

Lastly, if you're considering purchasing the product, I'd
encourage you to get in touch with one of our Authorized
Partners, listed at

    http://www.e-smith.com/partners/

Your Authorized Partner will be able to give you personal
attention and assist you in your evaluation of our product
and services.

Cheers,

Rich Lafferty
Network Server Solutions Group
Mitel Networks

[Ed Form]

Rich Lafferty wrote:
>
> Ed Form wrote:
>
> > I don't think it a very good idea to waste the time of your
> > bug or security folk with what might well be nothing more
> > than a stupid error on my part, so asking here first, giving
> > no information that could possibly aid the intruder fraternity,
> > seemed like the best, first step.
>
> While I appreciate the sentiment -- please, waste my time.
> I'm more than happy to spend time following up misplaced
> security concerns. We know that investigation and reassurance
> is a necessary component of producing a secure product and
> have resources to dedicate to it.

I have mailed the details to the security@e-smith.com address.

> I agree that this particular instance didn't reveal any
> vulnerabilities, but it's really something we prefer to
> evaluate ourselves, if only because we have resources beyond
> those of a typical user that can be applied to tracking down
> the origin of the unusual behavior.

I don't think its possible to say that this didn't indicate some kind of vulnerability. Unlike Kees Blokland, I did not find any evidence of triggering by a spyware product resident on the Windows workstations or by the network print server. The triggering continued to occur when the only machine switched on in the building was the server itself. It looks from the logs as though it has been going on for some weeks.

> The potential damage of an observation reported to us as a vulnerability
> is much lower than that of a vulnerability posted here as an observation.

The problem with that idea is that I need to try to locate and understand the culprit problem because I am deciding whether to use this product for something like 30 installations each year. As I am not currently running a supported copy I am probably not going to get that answer from the security crew. But I take your nicely stated point.

> As for tone, we pride ourselves on the security of our
> product, which is one of the features we promote heavily;
> I trust you understand why our reaction to public announcements
> which inaccurately suggest that our claims of security
> are false might be negative.

I made no such claim. I reported some very odd, and *highly* unnaceptable behaviour. My attitude as I typed the message might have been better represented had I used the title: 'Surely I can't have had an intruder?'

> Lastly, if you're considering purchasing the product, I'd
> encourage you to get in touch with one of our Authorized
> Partners, listed at
>
>     http://www.e-smith.com/partners/
>
> Your Authorized Partner will be able to give you personal
> attention and assist you in your evaluation of our product
> and services.

I am in touch with them. I was on one of their courses two weeks ago, and I'm looking to purchase quite a bit of the product.

Ed Form

[Ed Form]

A follow up to my 'security' problem...

> Rich Lafferty said:
>
> > While I appreciate the sentiment -- please, waste my time.
> > I'm more than happy to spend time following up misplaced
> > security concerns. We know that investigation and reassurance
> > is a necessary component of producing a secure product and
> > have resources to dedicate to it.

I've been very pleased with the response of the Mitel guys to this problem
and with their help I've managed to track down the cause of the trouble.

It was a faulty modem!

If a dial-up mail event fails, for whatever reason, the system appears to
retry the call at short intervals until it succeeds. With a faulty modem,
that can ring out, but cannot properly negotiate a session with the ISP's
equipment, the result is scores of calls, each costing the minimum call
charge [about 4 pence during the business day in the UK].

In the case on which I was reporting the phone bills showed that the
system had made 1024 calls in a short period of time. Analysis of the logs
suggests that quite a small percentage of these were simple scheduled mail
calls or workstation triggered internet activity.

The unknown IP number that turned up was the address of a router owned by
the telecoms provider and sitting between our system and the ISP. The logs
contain a number of first attempts by the mail system - it calls 0.0.0.0
and then remembers the actual address that it finished up talking to and
calls it direct next time, and from then on until something changes.

So there was no intruder, and no external cause of any kind. My face is
currently the colour of a smacked bottom with shame.

I've commented in email to Charlie Brady that the potential to go on
repeating calls in the event of bad line conditions or a faulty modem is
not a very good idea. It can produce big phone bills for anyone who pays
per call. Having some ability to control the number of repeats after a
failure would be a big help - as would the ability to hold outgoing mail
to be sent with the scheduled mail sweeps - as I've noted before.

Ed Form