Topic: Asterisk dropping out every so often

[David Harper]

Every so often, seemingly at random, SARK stops processing, and we cannot call in our out, both over the TDM card and over the VOIP (PennyTel).

The only fix seems to be to stop and then restart sail. In addition, sometimes we have to issue a "load chan_zap.so" command from the Asterisk CLI.

Very weird. Note that this is SAIL 688. I am waiting for SAIL 715 to come out before upgrading to the 7xx series.

[SARK devs]

What hardware are you using?

Have you searched the Asterisk BB's?


S

[David Harper]

It's a genuine Digium TDM413E. 3x FXS, 1x FXO.

I'm unsure what I would search for. It's unclear what exactly is causing the issue. The fact that it is losing its configuration suggested a SAIL issue to me, though I could be wrong.

[SARK devs]

We have no reported issues of freezing and we have recorded just under 2000 downloads.

When you say "losing configuration"  can you explain what you mean please?  The asterisk configs, as you know, are held on flat files in /etc/asterisk.  Which ones are being lost?

S

[David Harper]

Basically the analog channels disappear - also one of the handsets stops communicating altogether. The first time, I had to reload the chan_zap.so driver from the Asterisk console. The second time, I am unsure what happened.

It's unclear what exactly is going on, as there have been workpeople on site and lots of other stuff going on. I will see what happens tomorrow and report back.

[SARK devs]

Most references I can find are analogue card related. In preparation fro the next failure you might want to do the following:---
 
You might want to turn on full logging to see what is happening just before the freeze... In /etc/asterisk/logger, uncomment the line which reads

;full => notice,warning,error,debug,verbose

Also, when it freezes, capture the output from dmesg and publish the last few lines of dmesg and the last couple hundred lines of the full log (var/log/asterisk).

S

[David Harper]

Well this is an Analogue card, so that would make sense

Thanks for the info -- will grab the above information if/when this happens again.

[d6hq]

Did you ever get to the bottom of this. We have the same issue with two separate servers. For unknown reasons Asterisk simply stops running at some point overnight. Both have analogue cards (OpenVox) and both are connected to BT lines. The dropout time is varied. If we restart Sail then they come back onstream. A similar server without analogue card does not exhibit the same problem. Any pointers would be most welcome.

[David Harper]

A similar server without analogue card does not exhibit the same problem.

What is different about the "similar" server? The BT lines?

[d6hq]

Thanks David for the quick response. All three servers are Dell SC430 or 440. All running SME 7.3 with Sail 716 or above. 2 have analogue cards using BT lines. The third has no card and no connection to the PSTN. The 2 with cards show the drop out the one without does not.

[David Harper]

Is there anything in the logs that might help us? Particularly just before the line drops out?

[d6hq]

There is nothing in the Asterisk message log that gives any indication of the problem or indeed the exact time it occurs. We can only estimate when by the last successful call. At the moment we have bodged a fix by restarting all the underlying services on schedule first thing in the morning but this does not always sort out the problem fully as sometimes the system does not re-register one of the SIP trunklines.

[David Harper]

I'm not sure what to suggest to help narrow down what might be going wrong. Have you tried Selintra's earlier advice:

You might want to turn on full logging to see what is happening just before the freeze... In /etc/asterisk/logger, uncomment the line which reads

;full => notice,warning,error,debug,verbose

Also, when it freezes, capture the output from dmesg and publish the last few lines of dmesg and the last couple hundred lines of the full log (var/log/asterisk).

If the worst comes to the worst, perhaps you need to call in or out every few minutes after hours until Asterisk stops working, so you know what time frame we are looking at.

[d6hq]

We will need to do some more digging. I have just realised that there is another common factor between the two failing servers which is that their broadband lines are provided by BT Opensore. The one without the analogue card sits on a Zen line. Hence the problem may be nothing to do with the existence of the analogue card at all.

I shall report back in due course.

[d6hq]

Another strange thing about these servers is that the "State" indicators in both Trunklines and Extensions show the red cross even though all are registered. After a full reboot the indicators are Ok for a while and then are quickly replaced by the red cross. Clicking on the trunkline indicator gives the following message:

"Unable to connect to remote asterisk (does /var/run/asterisk/asterisk.ctl exist?)"

Have you seen this annoyance before ?

[David Harper]

What version of Asterisk are you using?

rpm -q asterisk

[d6hq]

asterisk-1.4.18.1-59.el4

All servers are SME7.3 loaded from the Sark 7.3 iso then updated via yum localinstall to Sail-2.2.1-716 or higher. The servers are connected on broadband lines and all other services - very few actually being used - are working without issue. Confguration changes have been minimal. The two analogue cards are from OpenVox.

[David Harper]

I would upgrade Asterisk (& addons etc.) to the version in my SARK archive. This is the last version that works with Zaptel.

http://sme.david-harper.com/data/sark/

See if that fixes your issue.

[d6hq]

Thanks David - we will give it a go this evening.

Is the zaptel driver in there suitable for 7.4 ?

[David Harper]

Only 7.4 Gold (-78.0.8 ). The driver for the updated kernel that was released a few days ago (-78.0.13) is in the smecontribs and/or atrpms repository.

[d6hq]

We did the update as suggested and are now monitoring the situation to establish whether or not we still get the dropouts. I will report back.

[SARK devs]

What does 'sip show peers' give if you type it at the asterisk console?

S

[d6hq]

Returning to this issue. We have yummed the test box to 7.4 with 7.8.0.13 kernel. Upgraded Asterisk and Zaptel turned on full logging and left the thing to run untouched since Monday afternoon. Asterisk has failed at some stage overnight on Tuesday / Wesdnesday. It ran for 24 + hours OK but is displaying exactly the same symptoms as before. A sudden unexplained stop of Asterisk for no apparent reason.

Log extract.

[Mar 24 09:58:28] WARNING[4350] chan_sip.c: Got 200 OK on REGISTER that isn't a register
[Mar 24 10:23:18] NOTICE[4350] chan_sip.c: Peer '8232498' is now UNREACHABLE!  Last qualify: 39
[Mar 24 10:23:45] NOTICE[4350] chan_sip.c: Peer '8232498' is now Reachable. (1517ms / 3000ms)
[Mar 24 11:04:53] NOTICE[4350] chan_sip.c: Peer '8232498' is now UNREACHABLE!  Last qualify: 38
[Mar 24 11:05:03] NOTICE[4350] chan_sip.c: Peer '8232498' is now Reachable. (62ms / 3000ms)
[Mar 24 11:42:09] NOTICE[4350] chan_sip.c: Peer '8232498' is now Lagged. (4059ms / 3000ms)
[Mar 24 11:42:19] NOTICE[4350] chan_sip.c: Peer '8232498' is now Reachable. (45ms / 3000ms)
[Mar 24 13:07:39] NOTICE[4350] chan_sip.c: Peer '8232498' is now UNREACHABLE!  Last qualify: 36
[Mar 24 13:07:49] NOTICE[4350] chan_sip.c: Peer '8232498' is now Reachable. (36ms / 3000ms)
[Mar 24 13:55:42] WARNING[4350] chan_sip.c: Got 200 OK on REGISTER that isn't a register
[Mar 24 14:39:19] WARNING[4350] chan_sip.c: Got 200 OK on REGISTER that isn't a register
[Mar 24 14:53:57] WARNING[4350] chan_sip.c: Got 200 OK on REGISTER that isn't a register
[Mar 24 15:03:44] WARNING[4350] chan_sip.c: Got 200 OK on REGISTER that isn't a register
[Mar 24 15:18:15] WARNING[4350] chan_sip.c: Got 200 OK on REGISTER that isn't a register
[Mar 24 16:20:22] NOTICE[4350] chan_sip.c: Peer '8232498' is now Lagged. (4396ms / 3000ms)
[Mar 24 16:20:32] NOTICE[4350] chan_sip.c: Peer '8232498' is now Reachable. (37ms / 3000ms)
[Mar 24 21:13:14] VERBOSE[4278] logger.c:   == Starting post polarity CID detection on channel 1
[Mar 24 21:13:15] NOTICE[5539] chan_zap.c: Got event 17 (Polarity Reversal)...


Test box hardware is as follows.
Dell SC600 - 640mb RAM, 2 x 80GB HD.
Openvox 400P card with 2 x FXO modules from Voipon.co.uk

My money is on the PCI card as the box was originally built without and did not start failing until the card and the necessary directives for UK caller ID were added.

[David Harper]

So let's confirm that suspicion by removing the card (over the weekend perhaps) and seeing how Asterisk copes.

[d6hq]

Thanks David - I have added the log extract from asterisk/full above. As you will see the last event is a polarity reversal on the card then nothing. I shall remove the card this afternoon and run it overnight without.

Can anyone advise whether they have experienced similar issues with true Digium cards in UK with BT caller ID on.

The zapata.conf modifications as advised by Selintra are follows

usecallerid=yes
callerid=asreceived
cidsignalling=v23 ; Added for UK CLI detection
cidstart=polarity ; Added for UK CLI detection
;internationalprefix: 00
;nationalprefix:0

[SARK devs]

We will check the various BB's to see what we can see.  We ourselves have only one customer who uses analog (4 sites), but they don't use CLIP and we've never had a failure of this nature with them (their oldest site is just coming up for its second birthday).

For us, analog isn't worth the hassel so we only ever install BRI and PRI (mostly PRI).   Digium Analog is very Ma Bell orientated and it shows in the zaptel code, which is bodged up to hell and back for BT and other networks.

Sorry we can't help more but this really does look like an Asterisk or Digium card issue.

Best

S

[d6hq]

Selintra

Unfortunately ISDN in any form isn't an option here. We have two SME customers where they wanted to keep their BT line and number but expand their ability to make outbound calls.

In one case we have managed to work around the drop out with a simple reload of sail at 4am whether the system is down or not. This works but has an annoying bi-product of making the extensions and trunklines appear to be unregistered in the GUI even though they are properly registered. We can live with that Ok.

In the other case the drop outs are more frequent and more severe in their effect. We applied a reload at 4am but found further drop outs before they open for business at 9am so changed to 8am only to find them occurring between 8am and 9am. They make the system too unreliable. In this case they need to keep the analogue BT circuit as the number is in circulation. They do however have 4 x Siemens S450's on a single base station so I guess I could remove the OpenVox card and reimplement the Siemen's landline support but then they don't get the voicemail etc etc etc.

Your customer with analogue cards - do you recall what manufacturer ?

[SARK devs]

All Digium - we don't use anything else (except for single and twin BRI, which Digium don't do so we use OpenVox). 

Try running without CLIP and see if it cures it.  It looks to be failing on the polarity reversal.

Best

S

[d6hq]

We will try without CLIP as this does look like it could be the cause.

Have you any experience of the Cisco SPA3102 used simply as an FXO gateway ?

[SARK devs]

Yes, we have a site in Scotland which we upgraded from Tbox and they have either 2 or 3 of them (can't remember which).  They seem to work OK but I've heard a lot of horror stories regarding echo and such like.

Best

S 

[d6hq]

I have ordered one so I'll let you know if there are any issues. I just need a solution that allows the Caller ID to be displayed.

Once we have proved its the CLIP I may also try our test box with card and CLIP enabled behind an old cheap but brilliant little ISDN PBX (Agfeo) on one of it FXS analogue ports to see whether it behaves the same way in that scenario. My guess is that the implementation of CLIP in the ISDN box will be perfect.

Had ISDN in the business since the days when it was the only reliable method of getting an internet connection. Kept it for telephony - first Agfeo PBX, then a fax server, then moved from the Agfeo to Epygi and use an ISDN failover for the ADSL Router. Could save a load of money by dumping it but ........

[SARK devs]

Hope you got it from our Distributor ProVu .   

Why not use the BRI for the SAIL Box?   

S

[d6hq]

Sorry no I bought it from Broadbandbuyer - quite happy to use ProVu in future though.

We do use BRI ourselves with the Epygi (bought about three years ago before Asterisk was stable enough to be considered suitable). It is a fiddly beast but very customisable once you get your head around it. There are half a dozen extensions on the LAN of varying sorts and a couple of remote extensions in our satellite offices. It is also connected to a GSM gateway which routes calls to mobiles and shares our contract minutes. Our preference is for Snom phones due to the ease with which they can be configured for CTI support. A right click out of our central Lotus Notes Contact Manager application fires up the users Snom. We haven't at this point implemented LDAP lookup but it should be a fairly easy exercise.

The Agfeo ISDN PBX is also plugged in and usuable - it runs 3 extensions, a phone, an internal fax machine and an analogue modem attached to our fax server (i.e. zero cost internal faxing or scanning to email). The fax server has a Diva ISDN card in it and we utilise it for fax to email based on DDI routing for a number of purposes including faxing in of signed Rugby match results forms for one of our customers. Why waste paper?

The testbox has an analogue card only at this stage simply to replicate the situation at a couple of customer sites. We might consider letting Sail replace our Epygi at some point in the future but there would be a lot to configure.

Hope that clarifies.

[d6hq]

Switching off UK CID detection on the OpenVox card seems to have stopped the drop outs. We have not had to remove the card.

[David Harper]

Great! Thanks for reporting back. Hopefully Selintra will be able to figure out why the CID was causing the dropouts.

[SARK devs]

It looks to be occasionally missing the line reversal, so it just sits there with its thumb up its bum waiting for something to happen.  BT CLIP reversal is <= 100ms - see SIN 227.  It may help if you set your card up for UK capacitance.  In /etc/init.d/sark find the modprobe for the card (wctdm) - it should be at line 150(ish)...

Code: [Select]

        if [ $WCTDM = 'YES' ]; then
#a          COUNTRY=`/sbin/e-smith/db selintra  getprop global IMPEDANCE`
            /sbin/modprobe wctdm #opermode=$COUNTRY
        fi

change it to

Code: [Select]

        if [ $WCTDM = 'YES' ]; then
#a          COUNTRY=`/sbin/e-smith/db selintra  getprop global IMPEDANCE`
            /sbin/modprobe wctdm opermode=uk
        fi


With a bit of luck that might sharpen it up enough to catch the reversals.

Best

S

[markleman]

Did the modification to /etc/init.d/sark fix your crashes? I have done it and still get crashes on average once a day on my home system which is SME 7.4 with the following installed:
 
 kernel-2.6.9-78.0.13.EL
 sail-2.2.1-714
 asterisk-1.4.21.1-65.el4
 asterisk-addons-1.4.7-21.el4
 zaptel-kmdl-2.6.9-78.0.13.ELsmp-1.4.12.1-54.99.el4
 zaptel-1.4.12.1-54.99.el4

I get the same event in /var/log/asterisk/messages reported above:

Code: [Select]

[Apr 12 20:47:21] NOTICE[17151] chan_zap.c: Got event 17 (Polarity Reversal)...
[Apr 13 09:25:24] NOTICE[8488] loader.c: 1 modules will be loaded.

zapata.conf has the following in it:

Code: [Select]

usecallerid=yes
ukcallerid=yes
cidsignalling=v23
cidstart=polarity

I have an OpenVOX A400 card with 1xFXO and 2xFXS modules.

Regards, Mark Leman.

[d6hq]

Mark - Haven't had a chance to test the mod. We had already turned off CLID on the card and the drop outs ceased. I concur with Selintra's view that it is the way BT do polarity reversals that causes the problem so if you ar in the UK you may experience the same.

[markleman]

Thanks for the reply - Based on my experience it probably won't fix it . The odd thing is I don't get this on our system at work which was supplied by Selintra. However there are a couple of factors here 1) I don't ever fiddle with it (Selintra supplied it and it' has just worked for 2years+ so why fiddle? ) so the kernel,SAIL,asterisk and zaptel versions are all much older 2) the analogue line is mainly there for ADSL and is only wired to the SARK box as a backup in case ISDN goes down, therefore it rarely receives incoming calls and so is less likely to encounter this "polarity reversal" problem.

As for BT's polarity reversals causing the problem, IIRC Selintra did mention some patches a while back which IIRC they had not tried due to lack of demand. I find it a shame that Digium's analog code does not support BT's well documented and IMHO quite sensible Caller ID system very well and I am slightly surprised that more UK asterisk users have not complained / corrected the code.

Mark Leman

[d6hq]

Mark - I agree. Although I would add that all of the serious traditional PBX's I have encountered have all been ISDN based so perhaps the driver (no pun intended) was not really there. However I think that the market for an analogue PBX will probably grow due to the capability of VOIP. Why pay for an ISDN circuit when you don't really need it if you route all of your calls via the net and just need the PSTN for failover ? Is your server a Dell by any chance ?

[markleman]

My home server was thrown together by me from an old Tyan dual Athlon MP motherboard (S2469 IIRC) a couple of 2G CPUs, a 3ware raid card, 6 hard disks and an OpenVOX A400 card. It serves all sorts of stuff up to the house

At work we have the server Selintra supplied, a very small 'embedded' PC style motherboard, a PCI riser, 2 BRI cards and a Digium card with 1xFXO for the ADSL POTS line and 1xFXS for the Fax machine.

[d6hq]

Right probabaly not relevant then. It was just that all of the servers where we had experienced problems with drop out were Dell's and had OpenVox cards. We also have some issues with Dell's and USB disks used for backup. It may be SME or it may be the Dell hardware but sometimes the server doesn't see a newly plugged USB disk. Very annoying and not clear why.