Software Upgrade

The maintenance release mr9.5 and newer uses the new upgrade approach commonly known as 'A/B Upgrade'

ngcp-upgrade requires a special partitioning schema of disk subsystem. The server has 3 separate partitions:

ngcp-data - it stores data files, like databases files, logs, etc.

ngcp-root and ngcp-fallback - they are equal size and contain the software, OS files and NGCP packages. One of them is the current root '/' partition, the another one is mounted as /ngcp-fallback directory.

See more details in The default disk partitions

During the upgrade the second partition is formatted and the target version is installed into it. After the reboot the system will be started from this partition and previous one becomes the /ngcp-fallback directory.

Please, pay particular attention that this partitioning schema is mandatory and if your system doesn’t have it - create it beforehand.
This is the only supported upgrade schema. The old, in-place upgrade is not supported and technically not possible.

Release Notes

The Sipwise C5 version mr10.5.1 has the following important changes:

  • IMPORTANT: [PRO-Carrier] The 'ngcp-sems-modules' package has been deprecated and all the code/libraries provided by that (.so plug-in’s) are from now on provided by the 'ngcp-sems-pbx' package. From the NGCP users perspective nothing has really changed, apart that 'ngcp-sems-modules' package does not exist anymore. The 'ngcp-sems-pbx' package provides .so plug-in’s into the '/usr/lib/sems-pbx/plug-in/' directory of the file system. [TT#174554]

  • Added support for g729, g722 and Opus codec on SEMS. This allows SEMS to play announcements and music-on-hold using these new codecs [TT#169904].

  • Two new options are added to 'config.yml': 'b2b.sbc.codec_list' and 'tpcc.codec_list'. The first indicates the list of codecs enabled on SEMS, while the second indicates the codecs used on TPCC’s generated calls. For both parameters the available options are 'PCMA' , 'PCMU', 'g729', 'g722' and 'opus'. The codecs have to be listed in priority order with no spaces between the elementes, for example "PCMA,PCMU,g729,g722,opus". [TT#169766].

  • The 'tags_header' concept for templating of sems-b2b configurations has been introduced. Now all the templating concerning sems-b2b is being done using it. In case of having any customtt files applied above the existing sems-b2b configurations, check them carefully to actualize them. And if your customtt files are covering a work on the templating scope of sems-b2b configurations, then rework your existing customtt files so, that they are applied above the 'tags_header' instead. [TT#168500]

  • For security reasons TLS protocol versions have been restricted to versions 1.2 and 1.3 where applicable. If support for older TLS versions (up to and including 1.1) is needed for certain services, then this support can be explicitly re-enabled through the respective configuration keys in 'config.yml' (under 'nginx.ssl_*' for HTTP interfaces, under 'prosody.tls.' for XMMP, and under 'kamailio.lb.tls.' for SIP). Not affected by this change is the interface for device auto-provisioning as such a change is likely to cause problems with existing devices. [TT#175102]

  • Removed obsolete RTC Engine, Janus and ComX components. [TT#175000]

  • New config.yml option kamailio.lb.filter_content_type. This option helps to filter from requests, reply or both some specific body content-type. This parameter substitutes the already existing kamailio.lb.remove_isup_body_from_replies, extending its functionality to other content types. The migration from the previous setting to the new one is automatically done during the upgrade. For additional info please see Message Body Filtering. [TT#68165]

  • [PRO-Carrier] The option kamailio.proxy.xfer_other_party_pai has been extended for both attended and unattended call transfers. If this option is set to "yes", then newly generated P-Asserted-Identity SIP Header sent to the "Transfer Target" will contain data relative to the "Transferee" and vice versa. This will cause in the phones to update their display with data relative to the parties in conversation. [TT#165301] [TT#165302]

  • Removed Heartbeat v2 support, which was deprecated in mr10.0. Systems should switch the High Availability subsystem to Corosync/Pacemaker before upgrading. [TT#174750]

  • [PRO-Carrier] Add support for LNP lookup via an external 302 SIP redirect server. NGCP LNP Daemon now supports the possibility to query an external server that provides the redirecting number and lnp code in the Contact and X-LNP (it can be customized) headers of a plain 302 SIP reply message External LNP via LNP API. [TT#169903]

  • SIP Loop Detection option is now enabled at default inside config.yml,more info about loop detection here Sip Loop Detection. [TT#181350]

  • New config.yml options kamailio.proxy.presence.notifier_poll_rate and kamailio.proxy.presence.notifier_processes. These options help in tuning kamailio presence module behavior. In particular, the first parameter defines the number of times per second that the notifier processes should check for work, the latter controls the number of notifier processes that should be started. The default values are '3' and '1' respectively and are valid for system with low resources or systems that doesn’t use 'presence' feature too much. In systems where the 'presence' feature is highly used, the seggestion is to configure them to '10' and '3' respectivelly. [TT#183550]

  • [CE] Header Manipulations and Phonebook features are now hidden on CE and not available on the UI and via the API anymore. This is for better consistency as while the features was shown on the UI/API the actual functionalty of these features was not availalbe for the CE users. [TT#182053] [TT#182101]

Some important technical points for those interested:

  • Handling of the transport protocol has been added for SIP peering outbound registrations (ngcp-panel). Now the main transport used in SIP peering configurations, is the one to be used for registering. [TT#167300]

  • [PRO-Carrier] 'outbound_proxy_set' and 'outbound_proxy_set_port' parameters has been added to the db_reg_agent.conf. Now if the SIP peering has a specific outbound_socket selected and the setup has multi-active LB architecture, outbound registrations will be sent via the correct LB only (the one owning this outbound_socket). [TT#167300]

  • The config.yml 'sems' settings are now migrated to the "b2b" section. The config.yml 'sems' section is from now on deprecated. The constants.yml 'sems' section is renamed to 'b2b'. [TT#169602]

Deprecation notice:

  • [PRO-Carrier] The 'libinewrate' package and all the code/libraries provided by that will be deprecated starting from mr11.0.1. [TT#179901]

  • [PRO-Carrier] The config.yml parameter 'b2b.prepaid.inew' will be deprecated starting from mr11.0.1. [TT#179901]

  • API /api/journals endpoint is now managed by default by the new ngcp-rest-api server instead of ngcp-panel, it has got an enhanced security and visibility scope. [TT#180255]

  • the following new columns are available:

    • "reseller_id" (only visible for the "system" and "admin" roles)

    • "tx_id" (transaction id, possible to search the logs in /var/log/ngcp/ by the transaction id to backtrace the request)

    • "role" (admin user role name)

  • incompatible changes:

    • "timestamp" column is now returned as a unix timestamp (decimal 13,3) instead of the datetime string, there will be a config.yml option rest_api.legacy.inflate_timestamp that enables all timestamp values to be automatically returned as a datetime string

  • Some PBX devices are deprecated starting from mr10.5.1+ version. Those are phones for which the device manufacturer has dropped hardware and software support entirely. These are still available for autoprovisioning for now, but will no longer be supported and potentially removed in a future versions of NGCP. See the updated list at available pre-configured devices.

Please find the complete changelog in our release notes on our WEB site.

Overview

The Sipwise C5 software upgrade procedure to mr10.5.1 will perform several fundamental tasks:

  • format fallback partition

  • install Debian to fallback partition

  • install NGCP packages to fallback partition

  • copy current configuration to fallback partition

  • upgrade the NGCP configuration schema in fallback partition

  • update grub configuration so after reboot the current fallback partition becomes the active one

  • reboot to new partition. This steps needs to be taken care manually during maintenance window

  • upgrade the NGCP database schema

Please make sure to have remote access to the system via out-of-band management (like IPMI, iLO, IMM, iDRAC, KVM, etc)

Pre-upgrade checks

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

Log into the NGCP server

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

Check the overall system status

Check the overall system status:

ngcp-status

Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

Check system integrity

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Run "apt-get update" and ensure that you do not have any warnings and errors in the output.

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

Pre-upgrade steps

Sipwise C5 can be upgraded to mr10.5.1 from previous LTS release, or any non-LTS release since the previous LTS-release. The script ngcp-upgrade will find all the possible destination releases for the upgrade and makes it possible to choose the proper one.
If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

ngcp-upgrade options

The following options in ngcp-upgrade can be specially useful in some instances of upgrade:

  • --step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).

  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:

    • to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or right before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option --step-by-step), or continue without stop until the end

    • another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.

Preparing for maintenance mode

Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Enable maintenance mode:

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
  • Apply configuration changes by executing:

ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr10.5.1'

Set the proper software repositories

Ensure you are using the Sipwise APT repositories.

Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.

The custom modification handling (optional)

During the execution of ngcp-upgrade in step place_customtt_files, you will be asked to place customtt/patchtt files for the new system to /ngcp-fallback/etc/ngcp-config.

Upgrading Sipwise C5 CE

Execute the following commands as root:

ngcp-prepare-upgrade mr10.5.1
ngcp-upgrade

There is "stop" step in upgrade scenario, upgrader stops there. At this time the new system in fallback partition is ready so you can reboot the server to boot from mr10.5.1 system.

After reboot run ngcp-upgrade again with the same parameters so upgrade will continue with post-upgrade steps.

Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.

Post-upgrade steps

Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Disable the maintenance mode:

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
  • Apply the changes to configuration templates:

ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr10.5.1'

Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr10.5.1-\* (where \* is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/ngcp-upgrade/$FROM-mr10.5.1/logs/.

Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest applying the latest hotfixes and critical bug fixes.

Execute all steps as described in Pre-upgrade checks. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.

Apply hotfixes

ngcp-update

Recheck or update the custom configuration templates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'

Execute the final checks as described in the Post-upgrade checks section.