Software Upgrade

Sipwise C5 can be upgraded to mr11.2.1 from previous LTS release, or any non-LTS release since the previous LTS-release.

The mr11.2.1 maintenance release uses the new upgrade approach commonly known as 'A/B Upgrade'

ngcp-upgrade requires a special partitioning schema of disk subsystem. The server has 3 separate partitions:

ngcp-data - it stores data files, like databases files, logs, etc.

ngcp-root and ngcp-fallback - they are equal size and contain the software, OS files and NGCP packages. One of them is the current root '/' partition, the another one is mounted as /ngcp-fallback directory.

See more details in The default disk partitions

During the upgrade the second partition is formatted and the target version is installed into it. After the reboot the system will be started from this partition and previous one becomes the /ngcp-fallback directory.

Please, pay particular attention that this partitioning schema is mandatory and if your system doesn’t have it - create it beforehand.
This is the only supported upgrade schema. The old, in-place upgrade is not supported and technically not possible.

Release Notes

The Sipwise C5 version mr11.2.1 has the following important changes:

  • [TT#178602] [PRO/Carrier] TPCC: Add a support of preferred audio codecs list for MakeCall and ConsultationCall APIs.

  • [MT#55636] [PRO/Carrier] TPCC: Add a support of Early leg termination with a specified SIP code/reason, ClearConnection API.

  • [MT#56078] Voicemail: Add the new asterisk.voicemail.transmit_silence parameter in config.yml, to keep sending RTP packets even during voicemail message recording.

  • [TT#55447] Add optional support for a third-party provided implementation of the EVS audio codec. Support for EVS is not directly included as EVS requires an additional third-party license, but support can be enabled if an implementation is provided.

  • [MT#55795] Added kamailio.proxy.permissions_max_subnets in config.yml. Now it is possible to specify in config.yml the max number of allowed_ips (ip or subnets) entries the system can accept. Default value is 512.

  • [MT#55858] Unified 302 call flow with internal CFB/CFT behavior.

  • [MT#56234] Subscriber’s preference 'ringtimeout' is now hidden from the preferences list and available only when configuring a new call forward on timeout. This to avoid misunderstanding and unwanted setup.

  • [MT#53595] Subscriber’s profiles have been reworked in order to simplify its usage. Main changes are the following:

    • the flags displayed in the Create/Edit page control only if those setting are shown in the CSC portal of the subscribers assigned to that profile. They don’t have any relation with the profile’s preferences.

    • the number of preferences that can be configured under the subscriber profiles has been extended: now all the preferences are always available regardless of what was configured in the above mentioned flags. For more details, please check Subscriber Profile chapter.

  • [MT#32998] [PRO/Carrier] The PBX Seats and Groups configuration in the CSC of Subscriber’s Administrators are now opened in a dedicated page. This simplify the configuration and let the possibility to extend it in the future with additional parameters.

  • [MT#33000] [PRO/Carrier] PBX Subscriber’s Administrators can now configure, using the CSC, the CLI of the other subscribers belonging to the same Customer PBX. The selection is restricted to the following values: primay number and aliases associated to the configured subscriber, primary number of the other subscribers of the PBX.

  • [MT#32999] [PRO/Carrier] PBX Subscriber’s Administrators can now configure, using the CSC, the Call forward of the other subscribers belonging to the same Customer PBX.

  • [MT#55932] [PRO/Carrier] New device preference 'user_conf_priority' is available. If set it avoids that the settings done by users manually on the phone devices are overwritten by the automated provisioning coming from the server. It is currently only supported by ALE and Yealink devices.

  • [MT#55450] [PRO/Carrier] Device provisioning feature has been extended adding new supported types of softkeys: Speeddial, Forward and Transfer. The first one allows to configure a number that can be quickly dialed, the second allows to easily redirect an incoming call to a predefined number, the third allows to transfer an active call to another number. It is currently only supported by ALE, Yealink and Snom devices.

  • [MT#56090] The configuration option general.maintenance was moved from config.yml to a new maintenance.yml file.

  • [MT#54588] [PRO/Carrier] Add support for faxserver instances

  • [MT#55680] New NCOS Set feature is now available! It allows to create sets of NCOS Levels and assign them to domains/subscribers useing the new preferences 'adm_ncos_set_id', 'ncos_set_id' and 'adm_cf_ncos_set_id'.

  • [MT#55680] Now the system allows to assign a timeset to NCOS Levels. This allows to restrict the limitation of the level to some specific time condition.

  • [MT#55349] New Customer Speed Dial feature. Customers can now create a list of 1000 speed dial numbers each (from 000 to 999) that can be used by all the subscribers belonging to it. The VSC code to use them is the same already in place for subscriber speeddial (by default *1) followed by the slot number. In this case the slot number has 3 digits instead of 1.

  • [MT#55258] [PRO/Carrier] PBX applications: improve internal flow to not use CF profile. An internal processing has stopped using CF profile for the PBX applications, such as: auto-attendant, conference, fax2mail. But only for cases with CFU, which is the only way now to enable these services for a subscriber. Instead of using the CF profile, start using direct call to the requested PBX application (such as we do for the callqueues). This behavior is controlled via the newly added config.yml option: kamailio.proxy.skip_cf_loop = 'yes' / 'no'. By default is disabled. In the closest future rest of applications will be moved to this new flow, such as: voicemail, calling card, office hours, custom announcement, manager secretary.

  • [MT#56208] Subscribers have access to call forward sets that do not belong to them but used in one of call forwards that are assigned to them. Those destination sets are visible on the UI (have '(inherited)' suffix) and also obtainable via the API but they cannot be modified/deleted. Once the call forward where they are present is removed from the subscriber, the related inherited sets are not obtainable anymore (not visible in the UI, nor accessbile via the API).

Some important technical points for those interested:

  • [MT#33046] [PRO/Carrier] Checks against the license availability for emergency calls has been disabled.

  • [TT#186701] Add/update Allow header in 200OK, if wasn’t empty in previously received request. It is necessary to keep the actual 'Allow:' header in the request, if the incoming request has had it. One of the use cases for that is for example the INVITE coming to the AA, when we must respond with the 200OK, but there is no 'Allow:' header inside the 200OK. Certain systems can be confused by that, even though the RFCs are not forcing us to do so. Which can even lead to an impossibility to use a normal call clearing with the BYE method, from behalf of the caller.

  • [TT#120250] [PRO/Carrier] Sems DSM: announce_dbprompt application is now fetching and storing audio in cache. The system now better caches audio prompts used by the announce_dbprompt to reduce the usage of memory on the system. A cached audio will be stored in the filesystem, default folder: /ngcp-data/cache/sems-b2b

  • [TT#187351] [PRO/Carrier] Be able to embed early media for a caller, when the caller is in the established session state (e.g. MoH). This is a useful feature, which allows to keep the Early media generated by the Sems DSM applications, and be able to carry this to the caller, even though the caller is already in the Connected state of the dialog (due to the call via AA, or due to MoH etc.) .

  • [TT#184101] Behavior of the B2B in regards of treating 183 with no SDP body has changed. System will re-use existing local sdp, in case there is a sequential 183 coming, but it has no SDP body. This fixes cases, when:

    • first 183 provisional response has been sent, and it had provided to the system the SDP

    • second/third etc. 183 is coming, but has no SDP body To not confuse the recipient (end subscriber) of these 183, the media description in the scenario with 183 having no SDP body, mustn’t be changed (session id, media and attributes). Previously the behavior was to generate local SDP with a new session id and add sems-b2b into the media processing by that (new media port, codecs and other attributes). However, this change doesn’t have an influence on very first 183 sent with no SDP body, because there was essentially no local SDP saved before. So in this situation behavior is not changed.

  • [MT#55878] [PRO/Carrier] Callqueue: add support of early reject playback. If the call failed while trying to reach the owner of the callqueue, the early reject playback requested from the DSM must be supported.

  • [MT#56025] [PRO/Carrier] DSM: voucher, added support of encrypted code lookup (AES128 CBC, in base64 formatting). It encrypts the value of the entered voucher code, when selecting it for matching/comparison reasons from the database backend.

  • [MT#56058] [PRO/Carrier] DSM: new option: 'dsm_cp_write_mode' has been introduced. This option controls the mode, in which DSM call session replication acts in terms of inserting the data into the NoSQL backend. Two modes of behavior introduced: 'dsm_cp_write_mode' = 'strict' / 'soft'. Default is 'soft'. For more information read the description added to this option as commentaries in the 'sems-b2b/sbc_replication.conf'.

  • [MT#55934] Implement 'recvonly' beginning on-hold. This is an improvement of the way to detect, whether or not a received SDP offer is the 'on-hold' request. Before this, such an SDP offer wasn’t considered as the 'on-hold' offer (only 'inactive', 'sendonly' or "zeroed" on-hold types were). This change assumes, that the end recipient of such offer returns the 'sendonly' or worst case the 'sendrecv' SDP answer.

API Changes:

  • [MT#55680] New v2 NCOS Sets related endpoints: /api/v2/ncos, /api/v2/ncos/sets, /api/v2/ncos/sets/levels

  • [MT#55680] Add time_set_id support to /api/ncoslevels v1 endpoint.

  • [MT#55737] Bulk provisioning support for POST operations:

    • e.g.: curl -X POST …​ /api/v2/endpoint --binary-data '[{item1},{item2}]'.

    • the current POST format that creates one item is supported.

    • new 'Prefer' header values:

    • return=link - returns a collection of links to the created items.

    • return=id - returns a collection of ids, of the created items.

    • POST 201 response always returns an array of items, even if there is only 1 created, for consistency.

  • [MT#46388] API v2 add Location header support for POST operations.

  • [MT#56068] API '/api/resellerbrandings/:id', the :id is now the branding’s 'id' instead of reseller_id. That was a bug as full collection with GET '/api/resellerbrandings' returns 'id' of the branding and it’s now consistent with the rest of the endpoints.

Deprecation notice:

  • [PRO/Carrier] Usage of 'P-Early-Announce' header is deprecated for custom applications based on DSM [MT#55431]

  • The ngcp-clish CISCO-like interface has been removed [MT#56273]

Please find the complete changelog in our release notes on our WEB site.

Overview

The Sipwise C5 software upgrade procedure to mr11.2.1 will perform several fundamental tasks, and it’s split into 2 stages:

First stage:

  • do pre-upgrade checks

  • format fallback partition

  • install Debian to fallback partition

  • install NGCP packages to fallback partition

  • copy current configuration to fallback partition

  • upgrade the NGCP configuration schema in fallback partition

  • update grub configuration so after reboot the current fallback partition becomes the active one

Second stage:

  • reboot to new partition. This steps needs to be taken care manually during maintenance window

  • upgrade the NGCP database schema

The 1st stage is safe to run outside the maintenance window - it changes nothing on currently running system, only prepares the fallback partition.

Also it can be done in parallel on all the nodes.

It still can affect the system in term of CPU, network and disk load, to download, unpack and install the packages.
Grub configuration is changed so in case of any reboot (expected or not) system will be booted to mr11.2.1 partition. It won’t affect the installation though as no NGCP services are run there. So if it was unintentionally you can reboot it back to previous partition via Rollback procedure.

The 2nd stage should be run during maintenance window with enabled maintenance mode in configuration.

Please make sure to have remote access to the system via out-of-band management (like IPMI, iLO, IMM, iDRAC, KVM, etc)

Pre-upgrade checks

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

Log into the NGCP server

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

Check the overall system status

Check the overall system status:

ngcp-status

Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

Check system integrity

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

Run "apt-get update" and ensure that you do not have any warnings and errors in the output.

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

Pre-upgrade steps

Set the proper software repositories

Ensure you are using the Sipwise APT repositories.

Public Debian mirrors may not provide packages for old Debian releases anymore. Also, they might be outdated. Consider using Sipwise repositories for the time of the upgrade.

Run the following command node to install the package responsible for upgrading Sipwise C5 to a newer release:

ngcp-prepare-upgrade mr11.2.1
Don’t worry, ngcp-upgrade-carrier does not exist, ngcp-upgrade-pro is used in Carrier too.

Run upgrade checks

There is a list of checks before the actual upgrade so it’s wise to run them beforehand to detect and fix all the issues.

ngcp-upgrade-pre-checks mr11.2.1
If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

ngcp-upgrade options

The following options in ngcp-upgrade can be specially useful in some instances of upgrade:

  • --step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).

  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:

    • to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or right before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option --step-by-step), or continue without stop until the end

    • another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.

The custom modification handling (optional)

During the execution of ngcp-upgrade in step place_customtt_files, you will be asked to place customtt/patchtt files for the new system to /ngcp-fallback/etc/ngcp-config.

First stage of upgrade

You can run 1st stage outside the maintenance window.

To do this run the upgrade script as root:

ngcp-upgrade --target mr11.2.1

There is "stop" step in upgrade scenario, ngcp-upgrade stops there. At this time the new system in fallback partition is ready.

Enable Maintenance Mode

The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Enable maintenance mode:

If you upgrade from version mr11.1.1 or lower use:

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"

Otherwise use:

ngcpcfg set /etc/ngcp-config/maintenance.yml "general.maintenance=yes"

  • Apply configuration changes by executing:

ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr11.2.1'

Second stage of upgrade

Before reboot switch boot record, run as root:

/usr/share/ngcp-upgrade/ngcp-switch-root-partition

Reboot the standby node and run as root:

ngcp-upgrade --target mr11.2.1

Once up again, double-check your config file /etc/ngcp-config/config.yml (sections will be rearranged now and will contain more parameters) and your domain/subscriber/peer configuration and test the setup.

Post-upgrade steps

Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Disable the maintenance mode:

ngcpcfg set /etc/ngcp-config/maintenance.yml "general.maintenance=no"

  • Apply the changes to configuration templates:

ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr11.2.1'

Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr11.2.1-\* (where \* is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/ngcp-upgrade/$FROM-mr11.2.1/logs/.

Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest applying the latest hotfixes and critical bug fixes.

Execute all steps as described in Pre-upgrade checks. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.

Apply hotfixes

ngcp-update

Recheck or update the custom configuration templates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'

Execute the final checks as described in the Post-upgrade checks section.