Switch to the terminal multiplexer under the user sipwise (to reuse Sipwise .screenrc settings which are user-friendly for handling upgrade in multiple windows):

screen -S ngcp-upgrade

Become a root inside your screen session:

sudo -s

Check the system overall status:

ngcp-status --all

Ensure that all proxy nodes replicate read-only DB (MySQL on 127.0.0.1:3308) from the node db01a. Otherwise, inform your manager about the special state here.

Try to find local changes to the template files by issuing:

find /etc/ngcp-config -name \*customtt.tt2

You will also need to find the dpkg-dist files under the templates files because people sometimes forget about creating customtt files and edit tt2 files directly. That makes upgrades not to replace the tt2 files. If so, you need to treat the tt2 files as if they were customtt files and make sure you merge the new templates with the changes of the old ones.

find /etc/ngcp-config -name \*.tt2.dpkg-dist

Also, please check/clean old dpkg backup files (just in case if another engineer did the previous step not carefully enough).

Normally the list should be empty:

find /etc/ngcp-config -name \*.tt2.dpkg\*

You will have to understand why the changes are there and if they are still needed after the upgrade.

You must create a ticket in the bug tracker to include customtt changes in the following releases (to remove customtt one day).

Open separate windows for all the servers inside your screen session. (Press Ctrl+a + c to open new window, Ctrl+a+a or Ctrl+a + [0-9] to change the window. Ctrl+a + " can open list of all windows for you. C+a + A can be used to change the screen name, so you can mark hosts here).

Check the system for locally modified files (move them to appropriate customtt.tt2 files if necessary) on all servers:

ngcp-status --integrity

Make sure the cluster status is ok - on all nodes issue manually:

Can be checked on all nodes in parallel, using clish and parallel-ssh:

root@web01b:~# ngcp-clish
Entering 'clish-enable' view (press Ctrl+Z to exit)...
# exit
root@web01b:~#

Run "apt-get update", ensure you have no warnings/errors here.

A cluster failover could be a good idea to see if everything works on the second node too. On the standby node issue:

ngcp-make-active

Afterwards, again check ngcp-status --all.

Create two test subscribers or retrieve the credentials for two of them. Register a client to the platform and perform a test call between the two to ensure call routing works.

Sipwise NGCP introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of NGCP will disable some background services (for instance: mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

Enable maintenance mode:

ngcpcfg pull

general:
  maintenance: yes

Disabling background services. These tasks are for systems currently running NGCP release earlier than mr5.4.1. The maintenance mode of NGCP is not available in your system yet, before the software upgrade. Therefore it is necessary to manually disable the following background services to avoid potential inconsistency in system state:

To do:

Please don’t forget to reverse the manual changes after the upgrade, as described in Enabling background services section of the handbook!

For all NGCP systems, regardless of their version:

Apply configuration changes, execute commands:

ngcpcfg apply 'Enable maintenance mode before the upgrade to mr5.4.2'
ngcpcfg push all

The main goal here is to fill the approx cache with new version of packages. So all the other nodes will get identical version of packages as the first one.

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.4.2/" /etc/apt/sources.list.d/sipwise.list
ngcp-approx-cache-helper --auto --node localhost
apt-get update
apt-get install ngcp-upgrade-pro

Execute ngcp-upgrade in inactive node as root:

ngcp-upgrade

Merge/add the customtt configuration templates if needed. Apply the changes to configuration templates if any:

ngcpcfg apply 'applying customtt for new release mrX.X on node xxx01a'

Send new templates to the shared storage and the other nodes

ngcpcfg push --nobuild --noapply all

Run the following commands to upgrade inactive DB node "A" (choose the same release version as above and follow on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.4.2/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade

Run the following commands here (choose the same release version and follow on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.4.2/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade

Run on all "A" nodes:

ngcp-make-active

Ensure "A" node became active, feel free to reuse 'ngcp-status' and 'ngcp-clish' commands described above.

Ensure ALL "B" nodes are inactive now!

Run the following commands here (choose the same release version and follow on-screen recommendations):

NGCP_CURRENT_VERSION=$(cat /etc/ngcp_version)
sed -i "s/$NGCP_CURRENT_VERSION/mr5.4.2/" /etc/apt/sources.list.d/sipwise.list
apt-get update
apt-get install ngcp-upgrade-pro
ngcp-upgrade

In order to disable maintenance mode do the following:

ngcpcfg pull

general:
  maintenance: no

Enabling background services. If you upgraded from NGCP release earlier than mr5.4.1 you have to manually reverse the changes you applied before the upgrade. (Disabling background services)

For all NGCP systems, regardless of their previous version:

ngcpcfg apply 'Disable maintenance mode after the upgrade to mr5.4.2'
ngcpcfg push all

When all finishes successfully check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check web interface, register two test subscribers and perform a test call between them to ensure call routing works.