Configuration Framework

The Sipwise C5 provides a configuration framework for consistent and easy to use low level settings management. A basic usage of the configuration framework only needs two actions already used in previous chapters:

  • Edit /etc/ngcp-config/config.yml file.

  • Execute ngcpcfg apply 'my commit message' command.

Low level management of the configuration framework might be required by advanced users though. This chapter explains the architecture and usage of Sipwise C5 configuration framework. If the basic usage explained above fits your needs, feel free to skip this chapter and return to it when your requirements change.

A more detailed workflow of the configuration framework for creating a configuration file consists of 7 steps:

  • Generation or editing of configuration templates and/or configuration values.

  • Generation of the configuration files based on configuration templates and configuration values defined in config.yml, constants.yml and network.yml files.

  • Execution of prebuild commands if defined for a particular configuration file or configuration directory.

  • Placement of the generated configuration file in the target directory. This step is called build in the configuration framework.

  • Execution of postbuild commands if defined for that configuration file or configuration directory.

  • Execution of services commands if defined for that configuration file or configuration directory. This step is called services in the configuration framework.

  • Saving of the generated changes. This step is called commit in the configuration framework.

Configuration templates

The Sipwise C5 provides configuration file templates for most of the services it runs. These templates are stored in the directory /etc/ngcp-config/templates.

Example: Template files for /etc/kamailio/proxy/kamailio.cfg are stored in /etc/ngcp-config/templates/etc/kamailio/proxy/.

There are different types of files in this template framework, which are described below.

.tt2, .customtt.tt2 and .patchtt.tt2 files

These files are the main template files that will be used to generate the final configuration file for the running service. They contain all the configuration options needed for a running Sipwise C5 system. The configuration framework will combine these files with the values provided by config.yml, constants.yml and network.yml to generate the appropriate configuration file.

Example: Let’s say we are changing the IP used by kamailio load balancer on interface eth0 to IP 1.2.3.4. This will change kamailio’s listen IP address, when the configuration file is generated. A quick look to the template file under /etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.tt2 will show a line like this:

listen=udp:[% ip %]:[% kamailio.lb.port %]

After applying the changes with the ngcpcfg apply 'my commit message' command, a new configuration file will be created under /etc/kamailio/lb/kamailio.cfg with the proper values taken from the main configuration files (in this case network.yml):

listen=udp:1.2.3.4:5060

All the low-level configuration is provided by these .tt2 template files and the corresponding config.yml file. Anyway, advanced users might require a more particular configuration.

Instead of editing .tt2 files, the configuration framework recognises .customtt.tt2 files. These files are the same as .tt2, but they have higher priority when the configuration framework creates the final configuration files. If you need to introduce changes in a template, you must always copy the required .tt2 file to .customtt.tt2, make changes in the latter file one and leave the .tt2 file untouched. This way, the system will use the new custom configuration allowing you to switch back to the original one quickly.

Example: We’ll create /etc/ngcp-config/templates/etc/lb/kamailio.cfg.customtt.tt2 and use it for our customized configuration. In this example, we’ll append a comment at the end of the template.

cd /etc/ngcp-config/templates/etc/kamailio/lb
cp kamailio.cfg.tt2 kamailio.cfg.customtt.tt2
echo '# This is my last line comment' >> kamailio.cfg.customtt.tt2
ngcpcfg apply 'my commit message'

The ngcpcfg command will generate /etc/kamailio/lb/kamailio.cfg from our custom template instead of the general one:

tail -1 /etc/kamailio/lb/kamailio.cfg
# This is my last line comment
users have to upgrade all .customtt.tt2 manually every time .tt2 is upgraded, as ngcpcfg completely ignores new code in .tt2 received from new package version.

The huge drawback of .customtt.tt2 files are necessity to keep them up-to-date manually. Keeping them outdated will cause the system misbehaviour as different components will use different code version (as new .tt2 version will be overwritten by old .customtt.tt2).

The .patchtt.tt2 concept should help users here. It will minimise the manual efforts by using linux "patch" utility. The ngcpcfg tool is searching for .patchtt.tt2 files every time 'ngcpcfg build' has been called. If .patchtt.tt2 is detected, the ngcpcfg tool will try to apply .patchtt.tt2 on .tt2 and store result in .customtt.tt2 if no conflicts noticed during patching. Further building process happens in a common way. Example:

root@spce:~# ngcpcfg build /etc/kamailio/lb/kamailio.cfg
spce: yml configs were validated successfully
spce: configs were checked successfully
spce: Validating patch '/etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.patchtt.tt2'
spce: Applying patch '/etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.patchtt.tt2'
spce: Successfully created '/etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.customtt.tt2'
spce: Requested patchtt operation has finished successfully.
Loading /etc/ngcp-config/config.yml in memory: OK
Loading /etc/ngcp-config/network.yml in memory: OK
Loading /etc/ngcp-config/constants.yml in memory: OK
spce: Generating /etc/kamailio/lb/kamailio.cfg: OK
spce: Executing postbuild for /etc/kamailio/lb/kamailio.cfg
root@spce:~#

To convert some/all the current .customtt.tt2 users can use command ngcpcfg patch --from-customtt [<customtt_file>]:

root@spce:~# ngcpcfg patch --from-customtt /etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.customtt.tt2
spce: Validating customtt '/etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.customtt.tt2'
spce: Creating patchtt file '/etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.patchtt.tt2'
spce: Requested customtt operation has finished successfully.
root@spce:~#

Here is the example of newly created .patchtt.tt2 file:

root@spce:~# cat /etc/ngcp-config/templates/etc/kamailio/lb/kamailio.cfg.patchtt.tt2
@@ -1799,3 +1799,4 @@
 }

 # vim: ft=cfg
+# This is my last line comment
root@spce:~#

See more details about .patchtt.tt2 files below in patchtt section.

The .tt2 files use the Template Toolkit language. Therefore you can use all the feature this excellent toolkit provides within ngcpcfg’s template files (all the ones with the .tt2 suffix).

Using patchtt for generation of a relevant customtt file

Keeping custom modifications directly in the .customtt.tt2 templates is NOT recommended as templates become outdated with every software upgrade.

A better way is to handle custom modifications using .patchtt.tt2 files (e.g. /etc/ngcp-config/templates/etc/cron.d/cleanup-tools.patchtt.tt2). In this case, on every "ngcpcfg patch", a .patchtt.tt2 file will be applied on top of the .tt2 file and the result will be saved into the customtt file and used commonly as described in the previous section. "ngcpcfg patch" is the first step on "ngcpcfg build" that guarantees the latest upstream templates with the availability of the necessary local changes on every configuration apply.

The patch to be applied to the corresponding .tt2 template file is selected in the following order (highest to lowest): *.patchtt.tt2.$HOSTNAME *.patchtt.tt2.$PAIRNAME *.patchtt.tt2.$HA_NODE *.patchtt.tt2
If a suitable patchtt file is found for a template, then the ngcpcfg patch command will overwrite the corresponding customtt file, if any.

Creating a patchtt file

Let us see how to introduce custom changes into a template through a patchtt file. For example, we need to change the accounting records cleanup time, which is defined in cleanup-tools.tt2. Here is how to do this:

  • Go to the corresponding templates directory:

cd /etc/ngcp-config/templates/etc/cron.d/
  • Duplicate the required .tt2 file to .customtt.tt2

cp ./cleanup-tools.tt2 ./cleanup-tools.customtt.tt2
  • Introduce the necessary changes to the duplicated file:

vim ./cleanup-tools.customtt.tt2
  • Create the patchtt file from your customtt file and recheck it:

ngcpcfg patch --from-customtt ./cleanup-tools.customtt.tt2
cat ./cleanup-tools.patchtt.tt2
  • Apply and push the changes

ngcpcfg apply "Change acc-cleanup time from 00 to 02 hours"
ngcpcfg push all

You will notice that the "ngcpcfg apply" command has generated the customtt file for the corresponding template:

root@web01a:/etc/ngcp-config/templates/etc/cron.d# ls -l ./cleanup-tools*
-rw------- 1 root root 932 Jan  4 11:11 ./cleanup-tools.customtt.tt2
-rw-r--r-- 1 root root 630 Jan  4 11:08 ./cleanup-tools.patchtt.tt2
-rw-r--r-- 1 root root 932 Dec 18 15:09 ./cleanup-tools.tt2

Now, even if cleanup-tools.tt2 slightly changes after a software upgrade, "ngcpcfg apply" will still preserve your custom changes.

If in a new release the .tt2 file gets changed in the same lines where you had introduced custom changes (e.g. your changes were temporary until a feature is implemented properly in a new software release), the apply process will fail and ask you to review the corresponding .patchtt.tt2 file. Then, check it and either correct if it is still required or remove it.
To convert all existing customtt files to patchtt files use the command: ngcpcfg patch --from-customtt

.prebuild and .postbuild files

After creating the configuration files, the configuration framework can execute some commands before and after placing that file in its target directory. These commands usually are used for changing the file’s owner, groups, or any other attributes. There are some rules these commands need to match:

  • They have to be placed in a .prebuild or .postbuild file in the same path as the original .tt2 file.

  • The file name must be the same as the configuration file, but having the mentioned suffixes.

  • The commands must be bash compatible.

  • The commands must return 0 if successful.

  • The target configuration file is matched by the environment variable output_file.

Example: We need www-data as owner of the configuration file /etc/ngcp-ossbss/provisioning.conf. The configuration framework will by default create the configuration files with root:root as owner:group and with the same permissions (rwx) as the original template. For this particular example, we will change the owner of the generated file using the .postbuild mechanism.

echo 'chgrp www-data ${output_file}' \
  > /etc/ngcp-config/templates/etc/ngcp-ossbss/provisioning.conf.postbuild

.services files

.services files are pretty similar and might contain commands that will be executed after the build process. There are two types of .services files:

  • The particular one, with the same name as the configuration file it is associated to.
    Example: /etc/ngcp-config/templates/etc/asterisk/sip.conf.services is associated to /etc/asterisk/sip.conf

  • The general one, named ngcpcfg.services that is associated to every file in its target directory.
    Example: /etc/ngcp-config/templates/etc/asterisk/ngcpcfg.services is associated to every file under /etc/asterisk/

When the services step is triggered all .services files associated to a changed configuration file will be executed. In case of the general file, any change to any of the configuration files in the directory will trigger the execution of the commands.

If the service script has the execute flags set (chmod +x $file) it will be invoked directly. If it doesn’t have execute flags set it will be invoked under bash. Make sure the script is bash compatible if you do not set execute permissions on the service file.

These commands are usually service reload/restarts to ensure the new configuration has been loaded by running services.

The configuration files mentioned in the following example usually already exist on the platform. Please make sure you don’t overwrite any existing files if following this example.

Example:

echo 'ngcp-service restart mariadb' \
  > /etc/ngcpcfg-config/templates/etc/mysql/my.cnf.services
echo 'ngcp-service restart asterisk' \
  > /etc/ngcpcfg-config/templates/etc/asterisk/ngcpcfg.services

In this example we created two .services files. Now, each time we trigger a change to /etc/mysql.my.cnf or to /etc/asterisk/* we’ll see that MySQL or Asterisk services will be restarted by the ngcpcfg system.

config.yml, constants.yml and network.yml files

The /etc/ngcp-config/config.yml file contains all the user-configurable options, using the YAML (YAML Ain’t Markup Language) syntax.

The /etc/ngcp-config/constants.yml file provides configuration options for the platform that aren’t supposed to be edited by the user. Do not manually edit this file unless you really know what you’re doing.

The /etc/ngcp-config/network.yml file provides configuration options for all interfaces and IP addresses on those interfaces. You can use the ngcp-network tool for conveniently change settings without having to manually edit this file.

The /etc/ngcp-ngcpcfg/ngcpcfg.cfg file is the main configuration file for ngcpcfg itself. Do not manually edit this file unless you really know what you are doing.

ngcpcfg and its command line options

The ngcpcfg utility supports the following command line options:

apply

The apply option is a short-cut for the options "check && build && services && commit" and also executes etckeeper to record any modified files inside /etc. It is the recommended option to use the ngcpcfg framework unless you want to execute any specific commands as documented below.

build

The build option generates (and therefore also updates) configuration files based on their configuration (config.yml) and template files (.tt2). Before the configuration file is generated a present .prebuild will be executed, after generation of the configuration file the according .postbuild script (if present) will be executed. If a file or directory is specified as argument the build will generate only the specified configuration file/directory instead of running through all present templates.

Example: to generate only the file /etc/nginx/sites-available/ngcp-panel you can execute:

ngcpcfg build /etc/nginx/sites-available/ngcp-panel

Example: to generate all the files located inside the directory /etc/nginx/ you can execute:

ngcpcfg build /etc/nginx/

commit

The commit option records any changes done to the configuration tree inside /etc/ngcp-config. The commit option should be executed when you’ve modified anything inside the configuration tree.

decrypt

Decrypt /etc/ngcp-config-crypted.tgz.gpg and restore configuration files, doing the reverse operation of the encrypt option. Note: This feature is only available if the ngcp-ngcpcfg-locker package is installed.

diff

Show uncommitted changes between ngcpcfg’s Git repository and the working tree inside /etc/ngcp-config. Iff the tool doesn’t report anything it means that there are no uncommitted changes. If the --addremove option is specified then new and removed files (iff present) that are not yet (un)registered to the repository will be reported, no further diff actions will be executed then. Note: This option is available since ngcp-ngcpcfg version 0.11.0.

encrypt

Encrypt /etc/ngcp-config and all resulting configuration files with a user defined password and save the result as /etc/ngcp-config-crypted.tgz.gpg. Note: This feature is only available if the ngcp-ngcpcfg-locker package is installed.

help

The help options displays ngcpcfg’s help screen and then exits without any further actions.

initialise

The initialise option sets up the ngcpcfg framework. This option is automatically executed by the installer for you, so you shouldn’t have to use this option in normal operations mode.

services

The services option executes the service handlers for any modified configuration file(s)/directory.

status

The status option provides a human readable interface to check the state of the configuration tree. If you are unsure what should be done as next step or if want to check the current state of the configuration tree, invoke ngcpcfg status.

If everything is OK and nothing needs to be done the output should look like:

# ngcpcfg status
Checking state of ngcpcfg:
OK:   has been initialised already (without shared storage)
Checking state of configuration files:
OK:   nothing to commit.
Checking state of /etc files
OK:   nothing to commit.

If the output doesn’t say "OK", follow the instructions provided by the output of 'ngcpcfg status'.

Further details regarding the ngcpcfg tool are available through 'man ngcpcfg' on the Sipwise Next Generation Platform.