Technical Helpweb

- more articles

IMG resets unexpectedly when configured as a peer for SS7 redundancy

Symptom 

In a setup where two Dialogic® IMG 1010 Integrated Media Gateways (IMG 1010) or two Dialogic® IMG 2020 Integrated Media Gateways (IMG 2020) are configured for SS7 signalling redundancy, the active SS7 node may unexpectedly reset.

Reason for the issue 

In the scenario described in the documentation for Configuring SS7 Redundancy [1] there are two nodes configured for SS7 signalling. One is designated the primary and one the secondary. Only one of these SS7 nodes will be "active" (ie. controlling the SS7 signalling) at a given point in time. The two nodes communicate using RCOMM signalling over the CTRL Ethernet port(s).

The RCOMM signalling is extremely sensitive to latency and packet loss because we want to switch over from the primary to secondary (or vice-versa) as soon as possible should something happen to one of the peer nodes to avoid losing active calls. This means that should the active node lose communication (RCOMM) with its peer and the GCEMS for even a fraction of a second, the SS7 signalling will switch to its peer and as described in the documentation [1] the active node will reboot to ensure that we never get into a situation where both SS7 nodes are active.

Usually in this case the IMG call trace / telnet (IMG 1010) or ssh (IMG 2020) debug output will be similar to this:

01:36:30.099 SYS(I)    (sysCtl) RCOMM Event, LINK_UNRESPONSIVE (9) Received for
                        remote node 5
01:36:30.099 SS7C(W)   STATE CHANGE 8
01:36:30.099 ALOG(SYS) SS7 State Change: ACTIVE WITH PEER -> ACTIVE LOST PEER
01:36:30.109 SCFG(F)   SYSstACTIVE_LOST_PEER Commanded to SYScondGO_OFFLINE Res
                       et Machine
01:36:30.109 ALOG(SYS) -- COMP [SS7] 18 STATE MACHINE RESET --  
01:36:30.109 ALOG(NET) HOST CLOSED:  LinkID 0 on x3142.
01:36:30.109 SS7C(W)   STATE CHANGE 8
01:36:30.109 ALOG(SYS) SS7 State Change:        RESET -> ACTIVE LOST PEER

Here we can see that the RCOMM link for node 5 was dropped for some reason and then reset itself.

Fix 

Given that the RCOMM signalling mechanism is so sensitive it would make sense to ensure that the CTRL ports of the SS7 nodes configured for this kind of redundancy should be connected to the same network switch or at least on a very low latency network connection.

This being the case, a common cause, and the reason for this article, is to ensure that the link between the CTRL port of both IMGs and the network switch is full duplex. Running in half duplex mode can cause serious network performance issues related to packet collision. In half duplex mode there is a period of time where both devices can think it's OK to transmit. When they transmit at the same time a collision occurs and both sides stop transmitting and wait for a random amount of time and try again. When the link runs in full duplex mode these issues do not occur.

Check the Ethernet port status from the telnet (IMG 1010) or ssh (IMG 2020) interface of the IMG. Log in to the "Advanced Debug" option with "d" and from there enter the zNet menu with "n" and dump the network configuration with "@":

Advanced Debug> n
zNet> N
MAC Devices:
   0)      mgi0: 00:20:1c:13:10:a3 (Boot Device - hst:192.168.185.248)
   1)      mgi1: 00:20:1c:13:10:a4
   2)      mgi2: 00:20:1c:13:10:a5
   3)     vmod2: 00:20:1c:13:10:a6
   4)     vmod2: 00:20:1c:13:10:a7
   5)     vmod2: 00:20:1c:13:10:a8
   6)     vmod2: 00:20:1c:13:10:a9

External Port Status:
   CTRL_0:  100 Mbps Half Duplex ( Enabled)   CTRL_1: Down
   DATA_0:  100 Mbps Half Duplex ( Enabled)   DATA_1: Down
   SGNL_0: Down                        SGNL_1: Down

Public Network Interfaces:
  CTRL: act - CTRL_0, stby - CTRL_1, evlan - x0000, cpu if - mgi0
    0) CPU   0 - 192.168.185.33:ffffff00 gw-192.168.185.5 evlan-x0000 Up ARP
  DATA: act - DATA_0, stby - DATA_1, evlan - x0000, cpu if - mgi2
    1) VMOD0_0 - 192.168.185.34:ffffff00 gw-192.168.185.5 evlan-x0000 Up ARP
  SGNL: act - SGNL_0, stby - SGNL_1, evlan - x0000, cpu if - mgi2

Note the external port status for CTRL_0 and DATA_0.

By default, the Ethernet ports on the IMG are set to auto negotiate. On the IMG 1010, the DATA/SIG ports can be set explicitly to 100M Full Duplex [2] however this option is not available for the CTRL ports (always auto-negotiate) that are used for RCOMM. On the IMG 2020, all Ethernet ports are always auto-negotiate. You will need to set this on your network switch if the link is half duplex.

On configuring this, you should see the following on the IMG:

Advanced Debug> n
zNet> N
MAC Devices:
   0)      mgi0: 00:20:1c:13:10:a3 (Boot Device - hst:192.168.185.248)
   1)      mgi1: 00:20:1c:13:10:a4
   2)      mgi2: 00:20:1c:13:10:a5
   3)     vmod2: 00:20:1c:13:10:a6
   4)     vmod2: 00:20:1c:13:10:a7
   5)     vmod2: 00:20:1c:13:10:a8
   6)     vmod2: 00:20:1c:13:10:a9

External Port Status:
   CTRL_0:  100 Mbps Full ( Enabled)   CTRL_1: Down
   DATA_0:  100 Mbps Full ( Enabled)   DATA_1: Down
   SGNL_0: Down                        SGNL_1: Down

Public Network Interfaces:
  CTRL: act - CTRL_0, stby - CTRL_1, evlan - x0000, cpu if - mgi0
    0) CPU   0 - 192.168.185.33:ffffff00 gw-192.168.185.5 evlan-x0000 Up ARP
  DATA: act - DATA_0, stby - DATA_1, evlan - x0000, cpu if - mgi2
    1) VMOD0_0 - 192.168.185.34:ffffff00 gw-192.168.185.5 evlan-x0000 Up ARP
  SGNL: act - SGNL_0, stby - SGNL_1, evlan - x0000, cpu if - mgi2

Be sure to configure the above for both peer IMGs.

Assuming that the IMG(s) were previously running in 100M half duplex mode and are now running in 100M full duplex mode, this should improve the stability of the RCOMM signalling link and prevent future random resets.

Product List

Dialogic® IMG 1010 Integrated Media Gateway
Dialogic® IMG 2020 Integrated Media Gateway (IMG 2020), formerly referred to as Dialogic® BorderNet™ 2020 Session Border Controller

Related Documentation 

[1] Configuring SS7 Redundancy

IMG 1010: http://www.dialogic.com/webhelp/IMG1010/10.5.3/WebHelp/IMG.htm#conf_ss7_red_10.5.1.htm
IMG 2020: http://www.dialogic.com/webhelp/BorderNet2020/2.2.0/WebHelp/config_ss7_redundant_nodes.htm

[2] Physical Port Configuration

IMG 1010: http://www.dialogic.com/webhelp/IMG1010/10.5.3/WebHelp/IMG.htm#physportconfig.htm

IMG 2020: not available (all Ethernet ports are auto-negotiate)