Software Upgrade

Sipwise C5 can be upgraded to mr11.5.1 from previous LTS release, or any non-LTS release since the previous LTS-release.

The mr11.5.1 maintenance release uses the new upgrade approach commonly known as 'A/B Upgrade'

ngcp-upgrade requires a special partitioning schema of disk subsystem. The server has 3 separate partitions:

ngcp-data - it stores data files, like databases files, logs, etc.

ngcp-root and ngcp-fallback - they are equal size and contain the software, OS files and NGCP packages. One of them is the current root '/' partition, the another one is mounted as /ngcp-fallback directory.

See more details in The default disk partitions

During the upgrade the second partition is formatted and the target version is installed into it. After the reboot the system will be started from this partition and previous one becomes the /ngcp-fallback directory.

Please, pay particular attention that this partitioning schema is mandatory and if your system doesn’t have it - create it beforehand.
This is the only supported upgrade schema. The old, in-place upgrade is not supported and technically not possible.

Release Notes

Starting from Sipwise C5 version mr11.3.1 the Soundset inheritance feature has been added. Every Sound_Set can have a parent Sound_Set: if the parent is configured, all the announcements not defined in the current Sound_Set will be inherited from the parent, if any.

Additionally the Contract_Sound_Set has been extended to contain all the announcements available in the standard Sound_Set. In case both of the sound set types are assigned to a subscriber, the Contract_Sound_Set takes precedence. Sipwise C5 systems without the CloudPBX module enabled or where Contract_Sound_Set are not assigned to Subscribers/Customers will not be affected. This change could instead have an impact on all the Subscribers/Customers with Contract_Sound_Set assigned. This is because right after the upgrade Contract_Sound_Set contain many handles without an associated announcement (for example all the Early Media announcements). In this case Sipwise C5 will not play any message back to the caller.

To solve this, a new script called 'ngcp-upgrade-contract-sound-sets' has been created. The script automatically runs through all Contract_Sound_Set assigned to subscribers and if those subscribers all use the same Sound_Set, the Sound_Set is set as a parent of the respective Contract_Sound_Set. For subscribers using a different Sound_Set, a copy of the Contract_Sound_Set is created as 'Contract_Sound_Set copy #1' for each discrepant Sound_Set (the following copies of the same Contract_Sound_Set are created as '…​ copy #2' and so on). The script can be called anytime after the upgrade to version mr11.3.1. If no parameter is specified, the script will simply print in the standard output the list of operations it is going to perform, without actually executing them. When all the reported changes has been acknowledged by the system administrator, the script has to be run again with '--apply' parameter. In this case the operations are committed to the database. Further invocations of the script will not have any effect on the system.

If you don’t want any automated changes to the provisioned Contract_Sound_Set, you can manually apply one of the following solutions through the web or api interfaces:

  1. Extend all the used Contract_Sound_Set by uploading the missing sound files. This solution should be avoided because it will increase the number of uploaded announcement in the database with a consistent grow of the used disk space.

  2. For all the used Contract_Sound_Set set the Sound_Set as a parent. This step enables the Contract_Sound_Set to inherit from the parent all the missing announcements. In case if more than one subscriber uses the same Contract_Sound_Set but a different Sound_Set, then it will be necessary to clone the Contract_Sound_Set, in order to have one for each such Subscriber. This solution is the preferred one for clients who decide to manually perform the necessary changes.

Please find the complete release notes and changelog of Sipwise C5 version mr11.5.1 on our WEB site.

Overview

The Sipwise C5 software upgrade procedure to mr11.5.1 will perform several fundamental tasks, and it’s split into 2 stages:

First stage:

  • do pre-upgrade checks

  • format fallback partition

  • install Debian to fallback partition

  • install NGCP packages to fallback partition

  • copy current configuration to fallback partition

  • upgrade the NGCP configuration schema in fallback partition

  • update grub configuration so after reboot the current fallback partition becomes the active one

Second stage:

  • reboot to new partition. This steps needs to be taken care manually during maintenance window

  • upgrade the NGCP database schema

The 1st stage is safe to run outside the maintenance window - it changes nothing on currently running system, only prepares the fallback partition.

Also it can be done in parallel on all the nodes.

It still can affect the system in term of CPU, network and disk load, to download, unpack and install the packages.
Grub configuration is changed so in case of any reboot (expected or not) system will be booted to mr11.5.1 partition. It won’t affect the installation though as no NGCP services are run there. So if it was unintentionally you can reboot it back to previous partition via Rollback procedure.

The 2nd stage should be run during maintenance window with enabled maintenance mode in configuration.

Please make sure to have remote access to the system via out-of-band management (like IPMI, iLO, IMM, iDRAC, KVM, etc)

Pre-upgrade checks

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

Log into the NGCP server

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

Check the overall system status

Check the overall system status:

ngcp-status

Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

Check system integrity

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Run "apt-get update" and ensure that you do not have any warnings and errors in the output.

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

RedisDB to KeyDB Migration

If not done already after the migration to mr10.5.x version, please migrate the key-value database from RedisDB to KeyDB. See the Appendix chapter for detailed information about the steps to execute.

Pre-upgrade steps

Ensure you are using the latest hot-fix of ngcp-upgrade-ce package from current repository before proceeding.

From node run:

apt update
apt install ngcp-upgrade-ce

Set the proper software repositories

Ensure you are using the Sipwise APT repositories.

Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.

Prepare the upgrade (on all the nodes)

Run the following command on all the nodes to install the package responsible for upgrading Sipwise C5 to a newer release:

ngcp-prepare-upgrade mr11.5.1
Don’t worry, ngcp-upgrade-carrier does not exist, ngcp-upgrade-pro is used in Carrier too.

Run upgrade checks (on all the nodes)

There is a list of checks before the actual upgrade so it’s wise to run them beforehand to detect and fix all the issues.

ngcp-upgrade-pre-checks mr11.5.1

ngcp-upgrade options

The following options in ngcp-upgrade can be specially useful in some instances of upgrade:

  • --step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).

  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:

    • to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or right before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option --step-by-step), or continue without stop until the end

    • another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.

Upgrading Sipwise C5 CE

Make sure you are prepared to spend about one hour upgrading the system. Note that a short service downtime is possible during the system reboot to the upgraded partition.

If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The custom modification handling (optional)

During the execution of ngcp-upgrade in step place_customtt_files, you will be asked to place customtt/patchtt files for the new system to /ngcp-fallback/etc/ngcp-config.

First stage of upgrade

You can run 1st stage outside the maintenance window.

To do this run the upgrade script as root:

ngcp-upgrade --target mr11.5.1

There is "stop" step in upgrade scenario, ngcp-upgrade stops there. At this time the new system in fallback partition is ready.

Enable Maintenance Mode

The maintenance mode of the Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Enable maintenance mode:

If you upgrade from version mr11.1.1 or lower use:

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"

Otherwise use:

ngcpcfg set /etc/ngcp-config/maintenance.yml "general.maintenance=yes"

  • Apply configuration changes by executing:

ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr11.5.1'

Second stage of upgrade

Before reboot switch boot record, run as root:

/usr/share/ngcp-upgrade/ngcp-switch-root-partition

Reboot the standby node and run as root:

ngcp-upgrade --target mr11.5.1

Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.

Post-upgrade steps

RedisDB to KeyDB Migration

If not done already before the upgrade, please migrate the key-value database from RedisDB to KeyDB. See the Appendix chapter for detailed information about the steps to execute.

Soundset migration

As anticipaed in the release notes, starting from Sipwise C5 version mr11.3.1 the Soundset inheritance feature has been added. Every Sound_Set can have a parent Sound_Set: if the parent is configured, all the announcements not defined in the current Sound_Set will be inherited from the parent, if any. Additionally the Contract_Sound_Set has been extended to contain all the announcements available in the standard Sound_Set. In case both of the sound set types are assigned to a subscriber, the Contract_Sound_Set takes precedence. This change could have an impact on systems with the CloudPBX module enabled and a mix of Sound_Set and Contract_Sound_Set are used.

To proceed with the migration, please follow the steps described in the release notes.

Migrate ncos and ncos_set preferences

The exposition of the ncos and ncos_set preferences to subscribers may have an impact on NGCP systems where system administrators used to setup ncos instead of adm_nos to apply barrings on subscribers/domains. That change might potentially have an unintended consequence where subscribers modify the previously assigned by administrators 'ncos' preference.

remember that to apply administrative ncos that subscribers cannot edit, the system administrator have to use the preferences adm_ncos, adm_cf_ncos, adm_ncos_set, adm_cf_ncos_set.

In case if the ncos levels/sets were wrongly used in past, those have to be migrated to the corresponding admin level/set. This process can be easily done and automated using API or the Web interface. Alternatively, to help customer with this migration, a dedicated script called 'ngcp-migrate-adm-ncos' has been created. The script has to be run ONLY in systems where ALL the ncos and ncos_set have to be migrated to adm_ncos and adm_ncos_set preferences. In case of a mixed environment (some preferences have to be migrated and others not) then the migration has to be done manually. The script automatically runs through all subscribers, profiles, customer and domain preferences, check for any ncos or ncos_set occurrence and migrate it to the corresponding administrative preference. The script can be called anytime after the upgrade. If no parameter is specified, the script will simply print to the standard output the list of potential conflicts in the system: in cases where both the ncos and the adm_ncos (or ncos_set and adm_ncos_set) are configured. When all the reported changes has been acknowledged by the system administrator, the script should be run again with '--apply' parameter. Further invocations of the script will not have any effect on the system.

Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Disable the maintenance mode:

ngcpcfg set /etc/ngcp-config/maintenance.yml "general.maintenance=no"

  • Apply the changes to configuration templates:

ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr11.5.1'

Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr11.5.1-\* (where \* is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/ngcp-upgrade/$FROM-mr11.5.1/logs/.

Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest applying the latest hotfixes and critical bug fixes.

Execute all steps as described in Pre-upgrade checks. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.

Apply hotfixes

ngcp-update

If there are custom configuration templates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'

Execute the final checks as described in the Post-upgrade checks section.