Sipwise C5 can be upgraded to mr9.5.6 from previous LTS release mr8.5, any non-LTS release since the previous LTS-release (mr8.*) or from LTS release mr7.5.
The mr9.5.6 maintenance release uses the new upgrade approach commonly known as 'A/B Upgrade'
ngcp-upgrade requires a special partitioning schema of disk subsystem. The server has 3 separate partitions:
ngcp-data - it stores data files, like databases files, logs, etc.
ngcp-root and ngcp-fallback - they are equal size and contain the software, OS files and NGCP packages. One of them is the current root '/' partition, the another one is mounted as /ngcp-fallback directory.
See more details in The default disk partitions
During the upgrade the second partition is formatted and the target version is installed into it. After the reboot the system will be started from this partition and previous one becomes the /ngcp-fallback directory.
|Please, pay particular attention that this partitioning schema is mandatory and if your system doesn’t have it - create it beforehand.|
|This is the only supported upgrade schema. The old, in-place upgrade is not supported and technically not possible.|
The Sipwise C5 version mr9.5.6 has the following important changes:
Network interfaces were renamed from eth* to neth* [TT#105151]
Add possibility to play to the caller an emulated ringback tone after pre-call announcement. See Emulated Ringback Tone for more details [TT#120551]
Add possibility to play an announcement in Auto-Attendant before terminating the call in case of selection timeout [TT#119455]
[PRO/Carrier] Enable 'kamailio.proxy.skip_pbx_loop' config.yml option by default [TT#105874]
Add support of 'SO_REUSEPORT' socket option (TCP/IP stack). Reusing socket descriptors already bound to listening at LB, for initiating new TCP connections from behalf of Sipwise C5. Applies also to TLS. [TT#124801]
Add support of TCP transport, as well as TLS for outbound registrations. [TT#124801]
Extend DTX jitter buffer to all other codecs supported for transcoding. [TT#122401]
Add option to use generic comfort noise generator instead of the native AMR SID noise generator to handle AMR SID/DTX. [TT#122401]
Providing Kamailio’s secsipid modules, which implement the IETF extensions for Secure Telephone Identity (RFC8224, RFC 8588), known as STIR and SHAKEN [TT#105600]
A/B Upgrade schema, ngcp-upgrade installs a new release to the second 'ngcp-fallback' partition [TT#104381]
New alarm based SNMP traps (in addition to the existing event based ones). See External Monitoring Using SNMP for more details. [TT#108956]
Some important technical points for those interested:
Removes support for obsolete legacy binary Apple APNS protocol [TT#120601]
CSC mixed mode to use Perl and JS CSC interfaces in parallel in mr9.5 LTS [TT#122026]
The soundset announcements announce_before_call_setup, announce_before_cf, announce_before_recording and announce_to_callee have been moved from 'early_rejects' section to the new 'early_media' section [TT#120551]
User preference "force_inbound_calls_to_peer" has now always priority over CFNA [TT#83601] The new priority only applies if CALLEE has "force_inbound_calls_to_peer" set to "force_offline*" or CALLER has "force_outbound_calls_to_peer" set to "force_offline*"
[PRO/Carrier] 'ngcp-approx-snapshots' is a new tool to sync Approx caches between LAB and PROD systems [TT#116404]
To avoid MAC address changes on virtual interfaces (e.g. with network bonding), the system is running with 'MACAddressPolicy=none' (see
Applied further systemd services security hardenings (haveged + ngcp-rate-o-mat) [TT#76552]
Automatic manual page database updates have been disabled for speed reasons (if you need 'apropos(1)' or 'whatis(1)' tools, invoke
systemctl restart man-db.service) [TT#122950]
[PRO/Carrier] The git remote origin of
/etc/ngcp-configis set to
shared_name:/mnt/glusterfs/ngcpcfg-shareon all nodes. Shared_name is
spfor PRO and
web01for Carrier. No longer relevant
/mnt/glusterfs/ngcpcfg-sharedirectory on non-mgmt Carrier nodes get automatically deleted during upgrades [TT#123200]
[PRO/Carrier] As part of upgrade, the new GlusterFS v9.2 will be installed to current Debian/buster system on both nodes in pair (active and standby) prior the upgrade to Debian/bullseye [TT#104381]
Some major software related changes/updates [TT#118659]:
Debian/bullseye v11 (updated from Debian/buster, v10)
Linux kernel v5.10 (updated from v4.19)
GCC v10 (updated from v8)
MariaDB v10.5 (updated from v10.3)
[PRO/Carrier] GlusterFS v9.2 (updated from v5.5)
Perl v5.32 (updated from v5.28)
Python v3.9 (updated from v3.7)
Python2 is no longer shipped ('python' binary is no longer available, use 'python3' in custom scripts instead)
OpenSSL v1.1.1k (updated from v1.1.1d)
OpenSSH v8.4p1 (updated from v7.9p1)
Kamailio v5.5.1 (updated from v5.4.5)
Perl-based CSC (Customer Self-Care) interface has been deprecated in mr6.5 LTS and will be removed in mr10.x. Please ensure JS-based CSC is in use [TT#122032]
With the introduction of a new ngcp-rest-api service that re-uses port 2443, that was previously used by the legacy SOAP interface, the ossbss section in config.yml is going to be removed in mr10.x as most of the parameters there are not used any longer, and ones that are still in use will be moved to the corresponding existing or new sections [TT#124400]
The InfluxDB-based monitoring backend is deprecated and is fully replaced with the Prometheus-based backend. InfluxDB will be removed in mr10.0 [TT#129100]
The feature 'SIP over Websockets' is deprecated and will be removed in mr10.0 [TT#122032]
Please find the complete changelog in our release notes on our WEB site.
The Sipwise C5 software upgrade procedure to mr9.5.6 will perform several fundamental tasks, and it’s split into 2 stages:
do pre-upgrade checks
format fallback partition
install Debian to fallback partition
install NGCP packages to fallback partition
copy current configuration to fallback partition
upgrade the NGCP configuration schema in fallback partition
update grub configuration so after reboot the current fallback partition becomes the active one
reboot to new partition. This steps needs to be taken care manually during maintenance window
upgrade the NGCP database schema
The 1st stage is safe to run outside the maintenance window - it changes nothing on currently running system, only prepares the fallback partition.
Also it can be done in parallel on all the nodes.
|It still can affect the system in term of CPU, network and disk load, to download, unpack and install the packages.|
|Grub configuration is changed so in case of any reboot (expected or not) system will be booted to mr9.5.6 partition. It won’t affect the installation though as no NGCP services are run there. So if it was unintentionally you can reboot it back to previous partition via Rollback procedure.|
The 2nd stage should be run during maintenance window with enabled maintenance mode in configuration.
|Please make sure to have remote access to the system via out-of-band management (like IPMI, iLO, IMM, iDRAC, KVM, etc)|
It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.
Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):
screen -S my_screen_name_for_ngcp_upgrade
Become root inside your screen session:
For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).
|If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.|
Find the local changes to the template files:
The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:
In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:
Find all available script options with the "-h" parameter.
Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:
Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:
Run "apt-get update" and ensure that you do not have any warnings and errors in the output.
|If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!|
For the upgrade to this version it is important that these packages are installed at their latest available version, to not cause problems during the upgrade.
To achieve this, execute the following commands:
apt-get update apt-get install ngcp-system-tools-ce ngcp-ngcpcfg
|Ensure you are using the Sipwise APT repositories.|
Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.
Run the following command node to install the package responsible for upgrading Sipwise C5 to a newer release:
|Don’t worry, ngcp-upgrade-carrier does not exist, ngcp-upgrade-pro is used in Carrier too.|
There is a list of checks before the actual upgrade so it’s wise to run them beforehand to detect and fix all the issues.
If there is an error during the upgrade, the ngcp-upgrade script will
request you to solve it. Once you’ve fixed the problem, execute
The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.
The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).
The following options in
ngcp-upgrade can be specially useful in some
instances of upgrade:
--step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).
--pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:
to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or right before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option
--step-by-step), or continue without stop until the end
another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end
--skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.
During the execution of ngcp-upgrade in step place_customtt_files, you will be asked to place customtt/patchtt files for the new system to /ngcp-fallback/etc/ngcp-config.
You can run 1st stage outside the maintenance window.
To do this run the upgrade script as root:
ngcp-upgrade --target mr9.5.6
There is "stop" step in upgrade scenario, ngcp-upgrade stops there. At this time the new system in fallback partition is ready.
The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.
Enable maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"
Apply configuration changes by executing:
ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr9.5.6'
Before reboot switch boot record, run as root:
Reboot the standby node and run as root:
ngcp-upgrade --target mr9.5.6
Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.
In order to disable the maintenance mode, do the following:
Disable the maintenance mode:
ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"
Apply the changes to configuration templates:
ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr9.5.6'
When everything has finished successfully, check that replication is running.
Finally, do a basic functionality test.
Check the web interface, register two test subscribers and perform a test call
between them to ensure call routing works.
|You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr9.5.6-\* (where \* is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/ngcp-upgrade/$FROM-mr9.5.6/logs/.|
If your current release is already the latest or you prefer to be on the LTS release, we still suggest applying the latest hotfixes and critical bug fixes.
Execute all steps as described in Pre-upgrade checks. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.
Merge/add the custom configuration templates if needed.
Apply the changes to configuration templates:
ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'
Execute the final checks as described in the Post-upgrade checks section.