16. Platform Security, Performance and Troubleshooting

Once Sipwise C5 is in production, security and maintenance becomes really important. In this chapter, we’ll go through a set of best practices for any production system.

16.1. Sipwise SSH access to Sipwise C5

The Sipwise C5 provides SSH access to the system for Sipwise operational team for debugging and final tuning. Operational team uses user sipwise which can be logged in through SSH key only (password access is disabled) from dedicated access server jump.sipwise.com only.

To completely remove Sipwise access to your system, please execute as user root:

root@myserver:~# ngcp-support-access --disable && apt-get install ngcp-support-noaccess

you have to execute the command above on each node of your Sipwise C5 system!


please ensure that the script complete successfully:

  * Support access successfully disabled.

If you need to restore Sipwise access to the system, please execute as user root:

root@myserver:~# apt-get install ngcp-support-access && ngcp-support-access --enable

please ensure that the script complete successfully:

  * Support access successfully enabled.

16.2. Firewalling

16.2.1. Firewall framework

The Sipwise C5 runs a wide range of services. In order to secure the platform while allowing access to Sipwise C5 , Sipwise C5 configuration framework provides a set of predefined network zones. Services are aggregated into appropriate zones by default. Zones are assigned to network interfaces (and VLANs if applicable) in /etc/ngcp-config/network.yml.


Though the default firewall setup provided by Sipwise C5 configuration framework provides a safe setup for Sipwise C5, security audits of the platform performed by qualified engineers before commissioning the platform into service are strongly recommended. Customization of the setup requires in-depth knowledge of firewalling principles in general and the netfilter facility in particular.

Table 21. Sipwise C5 network zones

Zone nameDescription


Internal HA (High Availability) communication interface between the services


Interface to connect external monitoring appliances (SNMP)


Interface for external RTP media relay between Sipwise C5 and endpoints (e.g. user agents, peers)


Interface for external SIP signalling between Sipwise C5 and endpoints (e.g. user agents, peers)


Interface for internal signalling, e.g. between load-balancers, proxies and applications servers


Interface providing external access to Sipwise C5 command line interface


Interface providing internal access to Sipwise C5 command line interface (necessary for ngcp-installer)


Interface providing access to the customers' self-care Web panel


Interface for access to the administrative Web panel, its REST APIs and internal API communications


Additional custom zones may be configured, but will not be automatically integrated into the firewall configuration.

To facilitate firewall functionality, Sipwise C5 uses the Kernel’s netfilter facility and iptables-persistent as an interface to netfilter. Netfilter is using tables and within that chains to store rules in this hierarchy: tablechainrule. Default firewall setups of Sipwise C5 do not use netfilter tables nat and raw, but only default table filter.


Custom nat rules for IPv4 and IPv6 may be added in file /etc/ngcp-config/config.yml in sections security→firewall→nat_rules4 and security→firewall→nat_rules6.

Each chain deploys a default policy handling packets which did not trigger and rule in a prticular chain.

Table 22. Sipwise C5 netfilter default policies

ChainDefault policyDescription



Handling all packets directly destined for a Sipwise C5 node (only packets matching a rule are allowed)



Handling all packets received by a Sipwise C5 node and destined for another, non-local IP destination (no default rules added)



Handling all packets originating on a Sipwise C5 node (no default rules added)



Container for rptengine rule to allow the rule to persist even when the Kernel module is unloaded (e.g. during upgrades)

The default firewall setup provided by Sipwise C5:

  • adds rules to INPUT to secure access to platform and services
  • blocks all traffic from and to FORWARD
  • allows all OUTPUT traffic

16.2.2. Sipwise C5 firewall configuration

The Sipwise C5 comes with a preconfigured set of firewall rules, which can be enabled and configured in /etc/ngcp-config/config.yml in section security→firewall. Refer to Section 1.34, “security” for available configuration options.

Firewall configuration is applied by running ngcpcfg apply. However, this will not activate new rules automatically to avoid inadvertent self-lockout. To finally activate new firewall rules run iptables-apply. This will prompt for another system logon to verify access remains available. If the prompt is not confirmed, firewall rules will automatically be reverted to the previous state re-enabling access to the command line.


The Sipwise C5 firewall subsystem by default is disabled in /etc/ngcp-config/config.yml key security.firewall.enable: no. This is to avoid blocking any traffic inadvertently during installation. After the firewall subsystem has been configured appropriately, it needs to be enabled by setting security.firewall.enable: yes in /etc/ngcp-config/config.yml.

16.2.3. IPv4 System rules

The following set of rules is added by the system upon activation of the firewall subsystem. Individual system rules are configured in /etc/ngcp-config/templates/etc/iptables/rules.v4.tt2 and /etc/ngcp-config/templates/etc/iptables/rules.v6.tt2

Table 23. Firewall system rules





-p udp -j rtpengine

Redirects all incoming UDP packets to chain rtpengine (putting RTPENGINE rule into a dedicated chain allows for the rule to persist even when the Kernel module gets unloaded, e.g. during upgrades)




-p udp -j RTPENGINE --id 0

Feeds all RTP packets to RTPENGINE Kernel module




-i lo -j ACCEPT

Accept all packets received by local loopback interface





Accept all incoming packets tied to related or established connections




-p icmp -m icmp --icmp-type 8 -j ACCEPT

Accept all ICMP echo messages




-p icmp -m icmp --icmp-type 0 -j ACCEPT

Accept all ICMP echo reply messages




-A INPUT -p ipv6-icmp -j ACCEPT

Accept all ICMPv6 messages




-j cluster

Divert all incoming packets to the cluster chain




-s <node_ip> -j ACCEPT

Set of rules white-listing all IP-addresses owned by Sipwise C5 platform for incoming traffic




-p tcp --dport <ossbss.port> -j ACCEPT

Set of rules for all api_int interfaces accepting all incoming packets for API port defined in /etc/ngcp-config/config.yml with key ossbss.port




+-p udp -s <snmpclient_ip> --dport 161 -j ACCEPT

Set of rules for all mon_ext interfaces based on a list of IPs for all SNMP communities configured in snmpd.communities




-p udp --dport <rtpproxy.minport>:'<rtpproxy.maxport>' -j ACCEPT/name

Set of rules for all rtp_ext interfaces accepting all incoming packets for RTP port range defined in /etc/ngcp-config/config.yml with keys rtpproxy.minport and rtpproxy.maxport (see note below for custom options)




-p udp --dport <kamailio.lb.port> -j ACCEPT

 Set of rules for all sip_ext interfaces accepting all packets on the loda balancer’s SIP signalling port defined in /etc/ngcp-config/config.yml with key kamailio.lb.port (UDP)




-p tcp --dport <kamailio.lb.port> -j ACCEPT

 Set of rules for all sip_ext interfaces accepting all packets on the loda balancer’s SIP signalling port defined in /etc/ngcp-config/config.yml with key kamailio.lb.port (TCP)




-p tcp --dport <kamailio.lb.tls.port> -j ACCEPT

 Set of rules for all sip_ext interfaces accepting all packets on the loda balancer’s SIP signalling port defined in /etc/ngcp-config/config.yml with key kamailio.lb.tls.port (TCP/TLS)




-p tcp --dport 5222 -j ACCEPT

Set of rules for all sip_ext interfaces accepting all packets on TCP port 5222 (XMPP client)




-p tcp --dport 5269 -j ACCEPT

Set of rules for all sip_ext interfaces accepting all packets on TCP port 5269 (XMPP server)




-p tcp --dport <pushd.port> -j ACCEPT

Set of rules for all sip_ext interfaces accepting all packets incoming for the pushd server port configured in /etc/ngcp-config/config.yml with key pushd.port




-A INPUT -i <ssh_ext_interface> -p tcp -s <sshd.permit_support_from> --dport sshd.port -j ACCEPT

List of rules to accept incoming packets for SSH on all ssh_ext interfaces from hosts configured in /etc/ngcp-config/config.yml with key sshd.permit_support_from




-p tcp --dport <www_admin.http_csc.port> -j ACCEPT

List of rules to accept incoming packets for the Customer Self Care interface defined in /etc/ngcp-config/config.yml with key www_admin.http_csc.port on all web_ext interfaces




-p tcp --dport <www_admin.http_admin.port> -j ACCEPT

List of rules to accept incoming packets for the Admin Panel interface defined in /etc/ngcp-config/config.yml with key www_admin.http_admin.port on all web_int interfaces


To function correctly, the rtpengine requires an additional iptables rule installed. This rule (with a target of RTPENGINE) is automatically installed and removed when the rtpengine starts and stops, so normally you don’t need to worry about it. However, any 3rd party firewall solution can potentially flush out all existing iptables rules before installing its own, which would leave the system without the required RTPENGINE rule and this would lead to decreased performance. It is imperative that any 3rd party firewall solution either leaves this rule untouched, or installs it back into place after flushing all rules out. The complete parameters to install this rule (which needs to go into the INPUT chain of the filter table) are: -p udp -j RTPENGINE --id 0


Some of the parameters used to populate the firewall rules automatically may contain hostnames instead of IP addresses. Since firewall rules need to be configured based on IP addresses by design, Sipwise C5 configuration framework will lookup such hostnames during ngcpcfg apply and expand them to the IP addresses as returned by gethostbyname. If DNS resolving changes for such hostnames due to changes to DNS the rules will not update automatically. Another run of ngcpcfg apply will be needed to reperform the lookup and update the rules to reflect changes in DNS. If this step is omitted, clients may be locked out of the system.


By default, the rules for the rtp_ext zone are created with a target of ACCEPT. It is optionally possible to create these rules with another iptables chain as target, and instruct the RTP proxy to dynamically manage individual rules for each running call in this chain. If this is enabled, the chain with the name given in the /etc/ngcp-config/config.yml key rtpproxy→firewall_iptables_chain will be created as empty, leaving the effective target for UDP packets within the RTP port range as the table’s default policy (normally DROP). The RTP proxy will then dynamically created one ACCEPT rule for each open RTP media port in the given chain when a call starts, and delete it when the call is finished. It should be noted that dynamically creating and deleting iptables rules can incur a singificant performance overhead, especially in scenarios with high call volumes, and it is therefore not recommended to enable this feature in such cases.

16.2.4. Custom rules

The Sipwise C5 configuration framework makes it possible to add custom rules to the firewall setup in /etc/ngcp-config/config.yml. The custom rules are added after the system rules. Hence, they apply for packets not matched by the systems rules only.

Example custom rule to whitelist all IPv4 traffic from network interface eth1.301 effectively making VLAN 301 a trusted network:

  - '-A INPUT -i eth1.301 -j ACCEPT'

Example custom rule to accept incoming traffic from monitoring station for an optionally installed check_mk agent:

  - '-A INPUT -p tcp -s --dport 6556 -j ACCEPT'

To add hosts or networks to the SSH whitelist they can be either added to key sshd.permit_support_from in /etc/ngcp-config/config.yml or a custom rule may be used:

  - '-A INPUT -s --dport 22 - j ACCEPT'
  - '-A INPUT -s --dport 22 -j ACCEPT'

In custom rules keys from /etc/ngcp-config/config.yml cannot be referenced. Thus, the values need to be manually looked up, hard coded, and kept in sync manually. This is by design of YAML.

16.2.5. Example firewall configuration section

An example for Sipwise C5 firewall configuration in /etc/ngcp-config/config.yml enabling both the firewall subsystem and the logging facility may look like:

    enable: yes
      enable: yes
      file: '/var/log/firewall.log'
      tag: 'NGCPFW'
      input: 'DROP'
      forward: 'DROP'
      output: 'ACCEPT'
      - '-A INPUT -i eth0 -j ACCEPT'

16.3. Password management

The Sipwise C5 comes with some default passwords the user should change during the deployment of the system. They have been explained in the previous chapters of this handbook.


Many Sipwise C5 services use MySQL backend. Users and passwords for these services are created during the installation. These passwords are unique for each installation, and the connections are restricted to localhost. You should not change these users and passwords.

16.3.1. The "root" account

The Sipwise C5’s super-user account comes with a preconfigured password. It is imperative that this password is changed by the operator immediately after Sipwise C5 is shipped and before it is connected to any potentially unsecure public or private network using a secure password in compliance with existing password policies of the operator. The "root" password must not be shared outside of the operator’s organization including Sipwise engineers. The "root" password must not be shared in any publicly accessible communications including e-mail or ticketing systems.

To change the root password log into the freshly deployed system as "root" using the preconfigured password and execute:

root@myserver:~# passwd

Then follow the prompts to change the password.

16.3.2. The "administrator" account

The Sipwise C5 Web-interface comes with a preconfigured "administrator" account deployed with a default password. This account can be considered Sipwise C5 application super-user and has far-reaching access to application specific settings via the Web-interface. It is imperative that the password for this account is changed by the operator immediately after Sipwise C5 is shipped and before it is connected to any potentially unsecure public or private network using a secure password in compliance with existing password policies of the operator. The "administrator" password must not be shared outside of the operator’s organization including Sipwise engineers. The "administrator" password must not be shared in any publicly accessible communications including e-mail or ticketing systems.

The password for the "administrator" account can be changed via the Web-interface.

16.3.3. The "cdrexport" account

The login for the system account cdrexport is disabled by default. Although this is a jailed account, it has access to sensitive information, namely the Call Detail Records of all calls. SSH keys should be used to login this user, or alternatively a really strong password should be used when setting the password via passwd cdrexport.

16.3.4. The MySQL "root" user

The root user in MySQL has no default password. A password should be set using the mysqladmin password command.

16.3.5. The "ngcpsoap" account

Generate new password for user ngcpsoap to access the provisioning interfaces, see the details in Section 10, “Provisioning REST API Interface”.

16.4. Remote root logins via SSH

To mitigate possible system intrusions from the outside, it is commonly held best practise to disable remote root (administrator) logins. With remote root logins disabled, it’s still possible to log into the system via SSH to a regular, non-privileged user account, and then use sudo or su to elevate account privileges to root administrator level.


For system setup purposes, the default setting on the Sipwise C5 is to permit remote root logins via SSH. It is strongly recommended to change this default setting as soon as possible.

Once you’ve created a user account for yourself that can be used to gain root privileges, remote root logins can be disabled via the config switch sshdpermit_root_login in /etc/ngcp-config/config.yml.

  permit_root_login: yes

Valid options are:

  • yes - permit remote root logins via SSH. Not recommended.
  • no - prohibit all remote root logins via SSH.
  • prohibit-password - permit remote root logins, except when password authentication is used. This is a good setting if you still wish to access the root account directly using public-key authentication, but also want to prohibit remote brute-force attacks against the root account password.
  • forced-commands-only - identical to prohibit-password but with the additional restriction that only SSH config sections that have a forced command (via command= or ForceCommand) are permitted to log in. For advanced users.

If either no or forced-commands-only is chosen, the Sipwise C5 will generate special SSH config sections that still allow root logins (using prohibit-password) from IP addresses that are internal to the Sipwise C5. This is necessary to ensure that the scripts handling the Sipwise C5 config framework continue to function and does not compromise security.

16.5. SSL certificates

The Sipwise C5 provides default, self-signed SSL certificates for SSL connections. These certificates are common for every installation. Before going to production state, the system administrator should provide SSL certificates for the web services. These certificates can either be shared by all web interfaces (provisioning, administrator interface and customer self care interface), or separate ones for each them can be used.

  • Generate the certificates. The customer self care interface certificate should be signed by a certification authority to avoid browser warnings.
  • Upload the certificates to the system
  • Set the path to the new certificates in /etc/ngcp-config/config.yml:

    • ossbssapacheautoprovsslcertfile and ossbssapacheautoprovsslcertkeyfile for the provisioning interface.
    • ossbssapacherestapisslcertfile and ossbssapacherestapisslcertkeyfile for the REST interface.
    • www_adminhttp_adminsslcertfile and www_adminhttp_adminsslcertkeyfile for the admin interface.
    • www_adminhttp_cscsslcertfile and www_adminhttp_cscsslcertkeyfile for the customer self care interface.
  • Apply the configuration changes with ngcpcfg apply 'added web ssl certs'.

The Sipwise C5 also provides the self-signed SSL certificates for SIP over TLS services. The system administrator should replace them with certificates signed by a trusted certificate authority if he is going to enable it for the production usage (kamailiolbtlsenable (disabled by default)).

  • Generate the certificates.
  • Upload the certificates to the system
  • Set the path to the new certificates in /etc/ngcp-config/config.yml:

    • kamailiolbtlssslcertfile and kamailiolbtlssslcertkeyfile .
  • Apply the configuration changes with ngcpcfg apply 'added kamailio certs'.

16.6. Securing your Sipwise C5 against SIP attacks

The Sipwise C5 allows you to protect your VoIP system against SIP attacks, in particular Denial of Service and brute-force attacks. Let’s go through each of those attacks and let’s see how to configure your system in order to face such situations and react against them.

16.6.1. Denial of Service

As soon as you have packets arriving on your Sipwise C5 server, it will require a bit of time of your CPU. Denial of Service attacks are aimed to break down your system by sending floods of SIP messages in a very short period of time and keep your system busy to handle such huge amount of requests. Sipwise C5 allows you to block such kind of attacks quite easily, by configuring the following section in your /etc/ngcp-config/config.yml :

  dos_ban_enable: yes
  dos_ban_time: 3600
  dos_reqs_density_per_unit: 50
  dos_sampling_time_unit: 2
  dos_whitelisted_ips: []
  dos_whitelisted_subnets: []

Basically, as soon as Sipwise C5 receives more than 50 messages from the same IP in a time window of 2 seconds, that IP will be blocked for 3600 sec, and you will see in the kamailio-lb.log a line saying:

Nov 9 00:11:53 sp1 lb[41958]: WARNING: <script>: IP '' is blocked and banned - R=<null> ID=304153-3624477113-19168@tedadg.testlab.local

The banned IP will be stored in kamailio memory, you can check the list via web interface or via the following command:

# ngcp-kamctl lb fifo htable.dump ipban

You have to run this command on the active load balancer node.

Excluding SIP endpoints from banning

There may be some SIP endpoints that send a huge traffic towards Sipwise C5 from a specific IP address. A typical example is a SIP Peering Server.


Sipwise C5 supports handling such situations by excluding all defined SIP Peering Servers from DoS protection mechanism.

The Sipwise C5 platform administrator may also add whitelisted IP addresses manually in /etc/ngcp-config/config.yml at kamailio.lb.security.dos_whitelisted_ips and kamailio.lb.security.dos_whitelisted_subnets parameters.

16.6.2. Bruteforcing SIP credentials

This is a very common attack you can easily detect checking your /var/log/ngcp/kamailio-proxy.log. You will see INVITE/REGISTER messages coming in with strange usernames. Attackers is trying to spoof/guess subscriber’s credentials, which allow them to call out. The very first protection against these attacks is: ALWAYS USE STRONG PASSWORD. Nevertheless Sipwise C5 allow you to detect and block such attacks quite easily, by configuring the following /etc/ngcp-config/config.yml section:

  failed_auth_attempts: 3
  failed_auth_ban_enable: yes
  failed_auth_ban_time: 3600

You may increase the number of failed attempt if you want (in same cases it’s better to be safed, some users can be banned accidentally because they are not writing the right password) and adjust the ban time. If a user try to authenticate an INVITE (or REGISTER) for example and it fails more then 3 times, the "user@domain" (not the IP as for Denial of Service attack) will be block for 3600 seconds. In this case you will see in your /var/log/ngcp/kamailio-lb.log the following lines:

Nov 9 13:31:56 sp1 lb[41952]: WARNING: <script>: Consecutive Authentication Failure for 'sipvicous@mydomain.com' UA='sipvicous-client' IP='' - R=<null> ID=313793-3624525116-589163@testlab.local

Both the banned IPs and banned users are shown in the Admin web interface, you can check them by accessing the Security Bans section in the main menu. You can check the banned user as well by retrieving the same info directly from kamailio memory, using the following commands:

# ngcp-kamctl lb fifo htable.dump auth

You have to run this command on the active load balancer node.

16.7. Topology Hiding

16.7.1. Introduction to Topology Hiding on NGCP

The term "topology hiding" in SIP is used to describe the measures taken by typically an SBC (Session Border Controller) to hide detailed information of the internal network at the border of which it is located. Pieces of information such as IP addresses and port numbers used by SIP endpoints and intermediaries within the network are considered sensitive, as these can give some hints to potential attackers about the topology of the network.

In a typical SIP session the mandatory headers may carry that sensitive information, for example: Contact, Via, Record-Route, To, From, Call-ID. An SBC applying topology hiding will mangle the content of those headers.

16.7.2. Topology Masking Mechanism

Concealment of sensitive information using this mechanism is achieved through encoding the original content of selected SIP headers. Then Sipwise C5 will create a new SIP URI using a preselected IP address and the encoded content as URI parameter, finally re-assembling the SIP header.

Examples for encoded SIP headers:

Record-Route: <sip:;line=sr-NvaAlWtecghucEhu6WtAcu...>
Contact: <sip:;line=sr-NvaAli-1VeL.kRxLcbN86W...>

The load-balancer element of Sipwise C5 has an SBC role, from the SIP peers point of view. The LB offers topology masking function that can be simply activated through a configuration change. By default the function is disabled. Configuration of Topology Masking

Activating topology masking function is possible through the modification of the following configuration parameters in /etc/ngcp-config/config.yml file (shown below with default values of parameters):

        enable: no
        mask_callid: no

Meaning of the configuration parameters:

  • enable: if set to yes, the topology mask will be activated
  • mask_callid: if set to yes, the SIP Call-ID header will also be encoded
  • mask_ip: an IP address that will be used to create valid SIP URIs, after encoding the real/original header content.


    Any valid, preferably private network address can be used. The suggestion is however to use an address that is not used by any other SIP endpoint or intermediary element in the network. Considerations for Topology Masking

Although masking sensitive information about a VoIP provider’s network is desired, there are some potential side effects caused by topology masking.

The most common example is the consequence that SIP message size may grow when applying topology masking. The fact that SIP messages become larger may even prevent Sipwise C5 from communicating successfully with another SIP entity (a peer SBC, for example). This can be expected under following circumstances:

  • SIP transport protocol is UDP
  • SIP messages have more Via and Record-Route headers
  • IP packets of SIP messages without the topology masking feature already have a size close to the MTU

In such a case the IP packets carrying SIP messages with encoded headers will have a size exceeding the MTU, that will cause loss of data in some networks.

The recommended solution in such a case is to use TCP transport for SIP messages.

16.7.3. Topology Hiding Mechanism

This mechanism achieves topology hiding by stripping the SIP routing headers that show topology details and storing those data in the associative data structure (hash) in the Redis DB so that it can look it up when a reply or in-dialog SIP message comes in. From the signaling perspective it simulates a SBC (Session Border Controller) on the LB. Considerations for Topology Hiding

This mechanism offers some benefits over the older topology masking approach:

  • It enables the Sipwise C5 to interconnect with SIP endpoints that are not capable of operating through a SIP proxy.
  • The message size is decreased because of stripping the SIP Record-Route, Route and Via header fields.
  • It solves the interoperability issues with SIP ALG in some cases.
  • It retains also the lightweight nature and the efficient operation.

The module uses the auto-expiration of the Redis keys so it can cause temporary spikes in the memory usage and redis keys count until produced data is cleaned up by redis. Configuration of Topology Hiding

Activation of the topology hiding function is done through the modification of the following configuration parameters in /etc/ngcp-config/config.yml file (shown below with default values of parameters):

        enable: no
        redis_db: 24

In order to activate the function, you should set enable: 'yes' in /etc/ngcp-config/config.yml and leave the Redis DB number unchanged, then execute ngcpcfg apply "activated topos".

16.8. System Requirements and Performance

The Sipwise C5 is a very flexible system, capable of serving from hundreds to several tens of thousands of subscribers in a single node. The system comes with a default configuration, capable of serving up to 50.000 subscribers in a normal environment. But there is no such thing as a normal environment. And Sipwise C5 has sometimes to be tuned for special environments, special hardware requirements or just growing traffic.


If you have performance issues with regards to disk I/O please consider enabling the noatime mount option for the root filesystem. Sipwise recommends the usage of noatime, though remove it if you use software which conflicts with its presence.

In this section some parameters will be explained to allow Sipwise C5 administrator tune the system requirements for optimum performance.

Table 24. Requirement_options

OptionDefault valueRequirement impact



Heavy impact on the harddisk storage needed for mysql logs. It can help to restore the database from backups or restore broken replication.


1/2 * Total system RAM

The installer will calculate the total system RAM and dedicate 50% to the mysql innodb buffer. This value won’t be changed in case the system RAM changes so it’s up to the administrator to adjust it. For test systems or low RAM systems, lowering this setting is one of the most effective ways of releasing RAM. The administrator can check the innodb buffer hit rate on production systems; a hit rate over 99% is desired to avoid bottlenecks.



This setting affects the amount of RAM the system will use. Each kamailio-lb worker will have this amount of RAM reserved. Lowering this setting up to 8 will help to release some memory depending on the number of kamailio-lb workers running. This can be a dangerous setting as the lb process could run out of memory. Use with caution.


1/16 * Total System RAM

The installer will set this value to 1/16 of the total system RAM. This setting does not change even if the system RAM does so it’s up to the administrator to tune it. It has been calculated that 1024 (1GB) is a good value for 50K subscriber environment. For a test environment, setting the value to 64 should be enough. "Out of memory" messages in the kamailio log can indicate that this value needs to be raised.



Number of TCP workers kamailio-lb will spawn per listening socket. The value should be fine for a mixed UDP-TCP 50K subscriber system. Lowering this setting can free some RAM as the number of kamailio processes would decrease. For a test system or a pure UDP subscriber system 2 is a good value. 1 or 2 TCP workers are always needed.



Enable or not TLS signaling on the system. Setting this value to "no" will prevent kamailio to spawn TLS listening workers and free some RAM.



See kamailio→lb→tcp_children explanation



See kamailio→lb→tcp_children explanation. In this case the proxy only listens udp so these children should be enough to handle all the traffic. It could be set to 2 for test systems to lower the requirements.


Set the default and the max and min registration interval. The lower it is more REGISTER requests will be handled by the lb and the proxy. It can impact in the network traffic, RAM and CPU usage.



Interval for the proxy to send a NAT keepalive OPTIONS message to the nated subscriber. If decreased, this setting will increase the number of OPTIONS requests the proxy needs to send and can impact in the network traffic and the number of natping processes the system needs to run. See kamailio→proxy→natping_processes explanation.



Kamailio-proxy will spawn this number of processes to send keepalive OPTIONS to the nated subscribers. Each worker can handle about 250 messages/second (depends on the hardware). Depending the number of nated subscribers and the kamailio→proxy→natping_interval parameter the number of workers may need to be adjusted. The number can be calculated like nated_subscribers/natping_interval/pings_per_second_per_process. For the default options, assuming 50K nated subscribers in the system the parameter value would be 50.000/30/250 = (6,66) 7 workers. 7 is the maximum number of processes kamailio will accept. Raising this value will cause kamailio not to start.


1/16 * Total System RAM

See kamailio→lb→shm_mem explanation.



Set this to no if the system shouldn’t perform rating on the CDRs. This will save CPU usage.



If enabled, the system will send the log messages to an external server. Depending on the rsyslog→external_loglevel parameter this can increase dramatically the network traffic.



This setting will set the number of days ngcp logs under /var/log/ngcp will be kept in disk. Lowering this setting will free a high amount of disk space.


In case of using virtualized environment with limited amount of hardware resources, you can use the script ngcp-toggle-performance-config to adjust Sipwise C5 configuration for high/low performance:

root@spce:~# /usr/sbin/ngcp-toggle-performance-config
/usr/sbin/ngcp-toggle-performance-config - tool to adjust Sipwise C5  configuration for low/high performance

  --help                Display this usage information
  --high-performance    Adjust configuration for system with normal/high performance
  --low-performance     Adjust configuration for system with low performance (e.g. VMs)


16.9. Troubleshooting

The Sipwise C5 platform provides detailed logging and log files for each component included in the system via rsyslog. The main folder for log files is /var/log/ngcp/, it contains a list of self explanatory log files named by component name.

The Sipwise C5 is a high performance system which requires compromise between traceability (maximum amount of debug information being written to hard drive) and productivity (minimum load on IO subsystem). This is the reason why different log levels are configured for the provided components by default.

Most log files are designed for debugging Sipwise C5 by Sipwise operational team while main log files for daily routine usage are:

Log fileContentEstimated size


API logs providing type and content of API requests and responses as well as potential errors


/var/log/ngcp/panel.log /var/log/ngcp/panel-debug.log

Admin Web UI logs when performing operational tasks on the ngcp-panel



mediation and rating logs, e.g. how many CDRs have been generated and potential errors in case of CDR generation or rating fails for particular accounting data



fail-over related logs in case a node in a pair loses connection to the other side, when a standby node takes over or an active node goes standby due to intra-node communication issues or external ping node connection issues



Overview of SIP requests and replies between lb, proxy and sems processes. It’s the main log file for SIP overview



Overview of SIP requests and replies along with network source and destination information flowing through the platform



Overview of SIP requests and replies between lb, proxy and sems processes



rtpengine related log, showing information about RTP communication



it is highly NOT recommended to change default log levels as it can cause system IO overloading which will affect call processing.


the exact size of log files depend on system type, system load, system health status and system configuration, so cannot be estimated with high precision. Additionally operational network parameters like ASR and ALOC may impact the log files' size significantly.

16.9.1. Collecting call information from logs

The easiest way to fetch information about a single call among the log files is the search for the SIP CallID (a unique identifier for a SIP dialog). The call ID is used as call marker in almost all the VoIP related log file, such as /var/log/ngcp/kamailio-lb.log, /var/log/ngcp/kamailio-proxy.log, /var/log/ngcp/sems.log or /var/log/ngcp/rtp.log. Example of kamailio-proxy.log line:

Nov 19 00:35:56 sp1 proxy[7475]: NOTICE: <script>: New request on proxy - M=REGISTER R=sip:sipwise.local
F=sip:jdoe@sipwise.local T=sip:jdoe@sipwise.local IP= ( ID=364e4676776621034977934e055d19ea@ UA='SIP-UA'

The above line shows the SIP information you can find in a general line contained in /var/log/ngcp/kamailio-*:

  • M=REGISTER : The SIP Method
  • R=sip:sipwise.local : The SIP Request URI
  • F=sip:jdoe@sipwise.local : The SIP From header
  • T=sip:jdoe@sipwise.local : The SIP To header
  • IP= ( : The source IP where the message is coming from. Between brackets it is shown the local internal IP where the message come from (in this case Load Balancer)
  • ID=364e4676776621034977934e055d19ea@ : The SIP CallID.
  • UAIP= : The User Agent source IP
  • UA=SIP-UA : The SIP User Agent header

In order to collect the full log related to a single call, it’s necessary to "grep" the /var/log/ngcp/kamailio-proxy.log using the ID= string, for example:

# grep "364e4676776621034977934e055d19ea@" /var/log/ngcp/kamailio-proxy.log

16.9.2. Collecting SIP traces

The Sipwise C5 platform provides several tools to collect SIP traces. It can be used Sipwise C5 ngrep-sip tool to collect SIP traces, for example to fetch traffic in text format from outbound and among load balancer, proxy and sems :

# ngrep-sip b

see the manual to know all the options:

# man ngrep-sip

The ngrep debian tool can be used in order to make a SIP trace and save it into a .pcap file :

# ngrep -s0 -Wbyline -d any -O /tmp/SIP_trace_file_name.pcap port 5062 or port 5060

The sngrep debian graphic tool as well can be used to visualize SIP trace and save them in a .pcap file :

# sngrep

The Sipwise C5 PRO platform provides also the native VoIP sniffer, called ngcp-voisniff, which provide a graphic view of all the calls passing through the platform. It can be enabled via __/etc/ngcp-config/config.yml:

  admin_panel: yes
    custom_bpf: ''
      - active: '0'
        case_insensitive: '1'
        pattern: '\ncseq: *\d+ +(register|notify|options)'
      include: []
      - 5060
      - 5062
      extra: []
      - sip_int
      - sip_ext
      - rtp_ext
        - _pbx\-1(?:_[0-9]{1,10})?$
        - _b2b\-1(?:_[0-9]{1,10})?$
        - _xfer\-1(?:_[0-9]{1,10})?$
        cin_max: '3000'
        cin_min: '0'
          threads: 20
      client_certificate: ''
      enable: no
      fix_checksums: no
      fragmented: no
        excludes: []
      local_name: sipwise
        port: '18090'
        protocol: sipwise
      enable: yes
      max_query_len: 67108864
      num_threads: '4'
    rtp_filter: yes
    start: yes
    threads_per_interface: '2'
    increment: '700000'
    keep: '10'

admin_panel should be set to yes as well as start and mysql_dump.enable. Also filter.exclude.active should be set to 1 in order to avoid to sniff REGISTER, NOTIFY, OPTIONS and SUBSCRIBE messages. Then run:

ngcpcfg apply 'enable voisniff' && ngcpcfg push

Please notice that enabling voisniff, specially under a huge amount of traffic, may affect the system performance due to the fact that voisniff needs to save all the traffic into the database.

16.10. Log file obfuscation

As many of the log files produced by Sipwise C5 contain sensitive and private data, and as various jurisdictions around the world have placed restrictions on who can view whose private data (e.g. GDPR in the EU), Sipwise C5 provides a mechanism to safely view log files in a partially obfuscated and anonymised (pseudonymised) fashion.

This obfuscated view is provided by the system service ngcp-logfs and is enabled by default. This service provides a read-only, partially obfuscated view of the Sipwise C5 log files in a separate folder, which by default is /var/log/mirror-ngcp/. The actual log files in /var/log/ngcp/ are normally readable only by the system administrator (root), while the obfuscated view of them in /var/log/mirror-ngcp/ is readable even by non-administrator system users by default.

Log files produced by Sipwise C5 contain special markers that identify data fields that correspond to private data belonging to 3rd parties. When accessing the log files through ngcp-logfs, the data contained in these fields will be replaced by other, seemingly random strings. However, this replacement is deterministic, meaning that the same original string will always be replaced with the same obfuscated string, making it still possible to correlate log lines belonging to the same entity, even across log files from different applications. Examples of such obfuscated data fields are user names, phone numbers, IP addresses, and other uniquely identifiable data fields.

The ngcp-logfs service also provides the same kind of access to archived (rotated) log files contained in /var/log/ngcp/old/. While these log files are compressed (.gz) on disk, they appear as uncompressed, plain text files when viewed through ngcp-logfs, as the service decompresses them in the background on demand.

16.10.1. Configuration

The service can be configured via its respective section in /etc/ngcp-config/config.yml:

  cache_db: /usr/lib/ngcp-logfs/cache.db
  chmod_dirs: '0555'
  chmod_files: '0444'
  disk_retention_timeout: 365
  enable: yes
  file_cache_timeout: 2
  gid: 0
  log_dir: /var/log/ngcp
  max_mem_usage: 500
  mem_cache_timeout: 24
  mountpoint: /var/log/mirror-ngcp
  obfuscation_prefix: GDPR
  suffix: \.\d+$|-\d{8}$|-\d{8}-\d+$
  uid: 0
  • cache_db: path and file name of the on-disk cache for obfuscated strings and their replacements. Does not normally need to be changed.
  • chmod_dirs: Unix file mode for directories visible through ngcp-logfs in octal. Defaults to octal 0555 (world readable).
  • chmod_files Unix file mode for log files visible through ngcp-logfs in octal. Defaults to octal 0444 (world readable).
  • disk_retention_timeout: how long to store obfuscated strings in the on-disk cache before they get deleted, in days. Defaults to 365 (one year).
  • enable: master switch for the service itself, yes or no.
  • file_cache_timeout: how long to cache obfuscated files (portions or entirely) in memory, in hours. Defaults to 2 hours.
  • gid: numeric Unix group ID for presented files and directories. Defaults to 0 (root).
  • log_dir: root directory of the log files that should be mirrored. Does not normally need to be changed.
  • max_mem_usage: upper limit in megabytes for the in-memory cache for obfuscated files. If this limit is hit, the in-memory cache will start to get aggressively emptied, even if file_cache_timeout isn’t yet reached. Defaults to 500 MB.
  • mem_cache_timeout: how long to cache obfuscated strings and their replacements in memory, in hours. Defaults to 24 hours.
  • mountpoint: where to make obfuscated log files visible in the file system.
  • obfuscation_prefix: optional prefix that obfuscated strings are guaranteed to have, provided the string is at least twice as long as the prefix. The prefix therefore should be kept short. Can be empty to disable this feature.
  • suffix: regular expression to match the file name suffix for archived (rotated) log files. Does not normally need to be changed.
  • uid: numeric Unix user ID for presented files and directories. Defaults to 0 (root).

Access to obfuscated log files can be further restricted by setting chmod_files, chmod_dirs and uid and/or gid. For example, to make log files accessible only to users belonging to the system group adm with group ID 4, set gid to 4, set chmod_files to 0440, and chmod_dirs to 0550, followed by executing ngcpfg apply.

16.10.2. Forward and reverse lookup

In some cases, for example for troubleshooting purposes, it can be necessary to determine the underlying unobfuscated plain text string from its obfuscated version, or perhaps even vice versa. For this purpose, the tool ngcp-lookup-obfuscated is provided, which can only be used by the system administrator (root). To perform a forward lookup (obfuscated string to unobfuscated), simply call it with the obfuscated string as its first argument. To perform a reverse lookup (unobfuscated to obfuscated), add the -r option.

For example, take the following sample log line provided by ngcp-logfs:

Mar 28 06:00:27 sp1 proxy[2544]: NOTICE: <script>: Sending reply S=200 Alive fs='' du='' - R=«GDPRcarPrUHeRvvWF2JjGei2Lu0bUjQIgI» ID=«GDPRqOB2kvuAm0JC7zQN6E1w4K» UA='sipsak 0.9.7pre'

The following commands would be used to perform both forward and reverse lookups on the call ID:

root@sp1:~# ngcp-lookup-obfuscated GDPRqOB2kvuAm0JC7zQN6E1w4K
root@sp1:~# ngcp-lookup-obfuscated -r 1899127565@

These lookups, in particular the reverse lookup, only work on strings that were actually processed by ngcp-logfs. You cannot use the reverse lookup procedure to obfuscate any arbitrary string that wasn’t previously provided by ngcp-logfs. The lookup must also be performed on the same host on which ngcp-logfs performed the obfuscation. Lookups don’t necessarily work on other hosts.

16.11. NGCP Panel passwords encryption

Encryption is used in case of administrator users passwords. Starting with mr8.4 release, subscribers web passwords are also encrypted the same way administrator passwords are by default.

In case you have upgraded from an earlier version where the passwords where plain text, the ngcp-bcrypt-webpassword can be used, which will encrypt all subscribers web passwords.