Configure backups with Archivist

The Archivist utility allows you to configure and create backups to make sure that the most critical data on your installation is preserved. You can back up the following kinds of data:

• MySQL database (hot backups performed while the database is running will not interfere with normal database operations).
• Cassandra database (storage for invoices, xDRs, reports, etc.).
• Recorded phone conversations.
• The general configuration of a server’s operating system (the “/etc” folder).
• Applied custom modifications (added to the deposit files).
• Custom RPM files.
• Billing statistics.
• Voice prompt files.
• Radacc files.
• Elasticsearch database (storage for log files and CDR collections).

You can access Archivist from the Configuration server web interface:

1. Log in to the Configuration server web interface.
2. Click the  Home button in the right-hand corner of the page.
3. Click the Archivist button.

   

You can use preset settings to configure a backup procedure. In this case you specify a time period during which the system should perform a backup. The system selects the kinds of data that it’s possible to back up within this period and will back up these kinds of data to the Configuration server from all the servers that contain them. However, backup parameters can be adjusted later as described in this section and in the Configure a backup procedure task by task section.

1. At the top of the Tasks panel, click the  Default settings button.

   

2. Specify the time period during which the backup procedure should be executed. Time period should be specified in the format “hr{hh:mm-hh:mm}.”

    

   For example, if you specify hr{00:00-03:30}, it means that the backup procedure will start after 00:00 and finish before 03:30. If the backup procedure is not completed before 03:30, the system terminates the procedure.

3. Select Remove all current tasks to confirm that you know that the default settings will override the current ones.
4. Click Next.

   

5. You can review and adjust the list of backup tasks to be executed.

   

   • To adjust the scheduled time and frequency of backup, click on the Schedule (in cron format) column and type the required value in the cron format, i.e., “Minute(0,..,59) Hour(0,..,23) Day(1,..,31) Month(1,..,12 or Jan,..,Dec) Weekday(0,1,..,7 or Sun,Mon,..,Sun).” Use a hyphen to indicate a range of values, a comma to indicate a list of values and an asterisk * to indicate every possible value.

      

     For example, if you specify Schedule (in cron format): 30 2 5,25 * 5-7, then your backup procedure will be scheduled to be performed at 02:30 on the 5th and 25th of every month as well as on every Friday, Saturday and Sunday.

     

   • To adjust the timeout, click on the Timeout (min.) field and specify the required value. If the backup task is not completed within the timeout period, the system terminates the task.

      

     For example, if you specify Timeout (min.): 180, then the backup process should finish within 3 hours from starting.

     Make sure that you give your system enough time to complete the backup task. Many kinds of data require no more than 10 minutes, but for example, a MySQL database backup procedure can take more than 2 hours.

     
     

     

   • To adjust how many of the most recent backups the system should keep, click on the Backup count field and specify the required value.
   • To remove the task click on the  Remove button.
6. Click Apply.
7. If necessary, now you can adjust settings for every backup task as described in steps 1–7 of the Configure a backup procedure task by task section.

A backup task is a combination of a type of data that you want to back up, a backup source and backup destinations. For example, one can define such a backup task as, “Backup Elasticsearch data from PortaBilling Web server to the Configuration server and to the remote FTP server.” To configure a backup procedure task by task, please complete the following steps.

1. In the Tasks panel, click on the kind of data that you want to back up (e.g., “Backup Elasticsearch”).

   

2. In the Backup Source panel, select the checkboxes against the servers where this data is located (e.g., select your PortaSwitch Web servers if your logs are located on them).

   

    If you want to review which instances are configured on which server, click the  Expand/Collapse button at the Backup Source panel top.

   

3. To adjust backup parameters, in the Backup Source panel, click on the first of the backup sources selected in the previous step.
   Be careful to click away from the checkbox, because when you click on the checkbox and clear it, you remove the backup source.
4. In the Schedule panel, specify when the backup should be performed, how long the backup process can be active for and how many of the most recent backups the system must keep.
   • Minute – type minutes in diapason from 0 to 59 or *, where * corresponds to every minute.
   • Hour – type hours in diapason from 0 to 23 or *, where * corresponds to every hour.
   • Day – type days of the month in diapason from 1 to 31 or *, where * corresponds to every day.
   • Month – type months in diapason from 1 to 12 or *, where * corresponds to every month.
   • Weekday – type weekday numbers in diapason from 0 to 6 or *, where 0 (and 7) correspond to Sunday, 6 to Saturday and so on and where * sets the weekday to undefined.
   • Timeout – type the maximum time in minutes that you want to allocate for the backup process. If the process is not completed within this period from its start, the system terminates it.

     Make sure that you give your system enough time to complete the backup task. Many kinds of data require no more than 10 minutes, but for example, a MySQL database backup procedure can take more than 2 hours.

   • Allowed indices – specify the Elasticsearch indices to back up in the following format: <index1>*,<index2>*,<index3>*, etc., where indexN is one of the Elasticsearch indices. For example, to configure Elasticsearch backup for SIP and BE logs only, specify “siplogs*,belogs*” value. To configure Elasticsearch backup for all indices, specify *.
   • Backup count – type the number of the most recent backups to keep.
You can use a hyphen to indicate a range and a comma to indicate a list of values, e.g., specify 1-6 in the Day field and 1,3,5 in the Month field. It means that the backup process is taken each day from the 1st till the 6th of January, March, and May.

For example, if you specify Minute: 30; Hour: 2; Day: 5,25; Month: *; Weekday: 5-7; Timeout: 180; Request timeout: 90; Allowed indices: siplogs*,belogs*; Backup count: 5; then your backup procedure will be scheduled to be performed at 02:30 on the 5th and 25th of every month as well as on every Friday, Saturday and Sunday. The backup process should finish within 3 hours from its start, and the system will keep the most recent 5 backups of this data.

You can review the descriptions of configured schedule settings and see exactly when the next backup will be performed on the bottom of the Schedule panel.



 In addition, after several backup runs, you can see how long it takes to back up selected types of data. Check the following performance characteristics:

• Avg execution time shows how long it takes to copy data to the final destination (if single destination is used) or to the Configuration server (used as an intermediary between a backup source and final destinations when you select more than one or an external destination). This value is of higher importance because the source server is actively engaged in the backup procedure during this time.
• Avg upload time shows how long it takes to copy data from the Configuration server to final destinations.

   

  Utilize this information to adjust the time frames for backup procedures if required.

  Note that to provide the best performance and to avoid mistakes it is recommended that backup procedures be scheduled for off-peak periods. Make sure they don’t overlap with resource-intensive calculations such as creating statistics/invoices.

5. In the Backup Destination panel, select servers where files with backup data will be placed. If you want to add a new remote destination, please refer to the Create a new remote destination section of this handbook.

   Note that when you select more than one destination or if you select an external destination, the configurator backup destination is automatically selected too. The reason for this is that sending backups to external or several destinations creates an additional load on the server, therefore it’s safer to perform this operation from the server that is not actively involved in performing critical real-time services – the Configuration server. So the system first copies backup files to the Configuration server and only then copies them from the Configuration server to specified destinations. (Note that the backup copy on the Configuration server won’t be deleted after the backup task finishes.)

    It’s recommended to choose more than one destination per source.

   Please make sure that there is enough free space on the destination servers.

    

   

6. If you have selected more than one backup source in step 2, repeat steps 4–5 for the rest of them.
7. Click the Save button at the bottom of the page.
8. If you want to configure the backup procedure for any other kinds of data, repeat steps 1–7.

1. At the top of the Backup Destination panel, click the  Add button.

   

2. In the Add custom destination dialog box, specify the following options:
   • Server name – type the name under which you want this destination to be shown in the list.
   • Protocol – select the protocol to be used to send files to the remote destination. Available options are “ftp,” “ssh (scp),” and “rsync over ssh.”
   • Path – specify the folder on your server where the system will save your backup files, use the following format: “/folder_name”. The system creates two additional nested subfolders:  “/IP_of_the_server_it_saves_data_from/A_kind_of_data_you_are_backing_up.”

      

     For example, if you specify “/backup” for this option and you back up statistics data from the server with IP “10.10.10.10,” the system will save your data in:

     /home/backup/10.10.10.10/statistics.

     If you want to edit local destination, the Path option is the only one you can modify. By default the Path option is set to “/porta_var/backup” for every local server.

   • Hostname – type the IP address of your destination server.
   • Port – type the port for connection.
   • Login – type the login for authorization on the destination server.
   • Password – type the password for authorization on the destination server.
   • Login Script – if required, for ssh (scp) and rsync over ssh protocols you may specify the path to the login script file. The login script includes a set of commands that are executed each time a user logs into the remote destination via an ssh connection. The script is mostly used to allow password-less user authentication. The Login parameter is still mandatory here to explicitly define the user who connects to the remote destination.

      

     The login script can also be used to add additional ssh connection options such as connect request timeout, number of connection attempts, etc.

3. Click Save.

1. In the Task panel, click on the kinds of data that you no longer want to back up.
2. In the Backup Source panel, clear the checkboxes against the servers where this data is located.
3. Click the Save button at the bottom of the page.
4. If you don’t want to back up any other kinds of data, repeat steps 1–3.
   You need to repeat all three steps for each kind of data that you no longer want to back up. If you clear the checkboxes against the servers where this data is located for one kind of data and then move to another without clicking Save, then all the changes that you made will be lost.

The main purpose of this option is to give you a possibility to check whether a backup task configuration is operable: that the chosen backup source contains the required data and that the connection to the backup destination works properly. We recommend that you always double-check a backup task configuration when you select remote destinations.

1. In the Tasks panel, click on the kind of data that you configured the backup task for (e.g., “Backup statistics”).
2. In the Backup Source panel, click on the required backup source.
   Be careful to click away from the checkbox, because when you click on the checkbox and clear it, you remove the backup source.
3. At the top of the Schedule panel, click the  Run Now button.

Use the Calendar view to get an overview of forthcoming and finished backup tasks by month, week or day.

To open the Calendar view, click the  Show Calendar button at the top of the Schedule panel.

The system prompts you to open the Logs Viewer every time you run the backup procedure manually (for how to perform this, see the Run a backup task manually section). However, you can check your backup logs with the Logs Viewer anytime you need to.

To open the Logs Viewer click the  Logs button at the top of the Schedule panel.

The Logs Viewer shows the logs at the run time of the backup process. So you can always check what stage the backup procedure is working on.

• To review a log, select the backup task from the list in the left-hand panel. You can review log details in the right-hand panel.
• To terminate the currently running backup procedure, select it in the left-hand panel and click the  Kill task button.
• To delete a backup and all the logged information about it, select the backup in the left-hand panel and click the  Delete button.
• To refresh a log immediately click the  Refresh button or set it to refresh automatically by clicking the arrow on the right-hand side of the Refresh button and selecting the required interval from the list.
• To close the Log Viewer, click the  Close button in the top right-hand corner of the page.

The backup tasks in the left-hand panel are grouped by backup sources and kinds of backup data. The icons near the backup task items indicate information about the status of the backup tasks. Backup tasks can have the following status:

•  Started – the backup task is waiting for processing.
•  Running – the backup task is being executed.
•  Done – the data was copied to the backup destination (when only one backup destination was selected) or to the Configuration server (when more than one destination was selected), but the synchronization script is still being executed.
•  Synced – the backup task has successfully finished.
•  Killed – the backup task was manually terminated.
•  Fail – the backup task wasn’t completed because of errors.
•  Unknown status – the system cannot identify the backup task status.

The recovery procedure can be manually executed from the CLI (command-line interface). Please contact support@portaone.com for more information on recovery procedures.

If your backup is located on another server you can perform rsync command remotely.

Executed on a restoration host.

 sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

  <ip_address>:/remote/path/to/file /local/path/to/file

Executed on a host with backups.

 sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

  /local/path/to/file <ip_address>:/remote/path/to/file

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/etc_dir/<date_time>/_etc/ /etc/

Note that the process must be executed on all nodes in the cluster, otherwise nodes that did not get the restored data will “update” the restored nodes with the newer, bad data.

1. Shut down the Cassandra node.

   Executed on a restoration host.

   sudo systemctl stop cassandra

2. Clear all files in /var/lib/cassandra/commitlog/

   Executed on a restoration host.

   sudo rm -rf /porta_var/<ip_address>/cassandra/commitlog/*.log

   Delete all .db files in the cassandra directory /porta_var/<ip_address>/cassandra/data/ <keyspace_name>/<table_name>/ Executed on a restoration host.

    sudo rm -rf /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/*

3. Find the most recent snapshot folder in this directory.

   /porta_var/backup/<ip_address>/cassandra/<date_time>/porta_var_<cassandra_node>_cassandra_data_<keyspace_name>_<table_name>_snapshots_incremental

4. Copy its contents into this directory /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/

   Executed on a restoration host.

    sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \
   
     --rsh=/home/porta-one/scripts/rsh_porta.sh \
   
   <remote_ip_address>:/porta_var/backup/<ip_address>/cassandra/<date_time>/_porta_var_<cassandra_node>_cassandra_data_<keyspace_name>_<table_name>_snapshots_incremental/ \
   
   /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/

5. Restart the Cassandra node.

   Executed on a restoration host.

    sudo systemctl start cassandra

6. Restarting causes a temporary burst of I/O activity and consumes a large amount of CPU resources. Run nodetool repair.

   Executed on a restoration host.

   sudo nodetool -h localhost -p 7199 repair

1. Create a new directory on restoration host where you will copy the backup file with the mysql database.

   Executed on a restoration host.

   mkdir /porta_var/tmp/mysql_restore_$(date -I)

2. Copy the backup file with the mysql database locally.

   Executed on a restoration host.

   cd /porta_var/tmp/mysql_restore_$(date -I)
   
   rsync -v --rsh=/home/porta-one/scripts/rsh_porta.sh \
   
   <ip_address>:/porta_var/backup/<ip_address>/mysql/<date_time>/mysql.xbstream .
   
   rsync -v --rsh=/home/porta-one/scripts/rsh_porta.sh \
   
   <ip_address>:/porta_var/backup/<ip_address>/mysql/<date_time>/my.cnf .

3. Untar the backup file.

   Executed on a restoration host.

   xbstream -x < ./mysql.xbstream -C ./

4. Stop the mysql service.

   Executed on a restoration host.

   sudo systemctl stop porta-mysqld

5. Delete or move the old mysql database to some place (be very careful!).

   Executed on a restoration host.

   sudo mv /porta_var/<ip_address>/mysql /porta_var/tmp/mysql_backup_$(date -I)

6. Create a mysql directory.

   Executed on a restoration host.

   sudo mkdir /porta_var/<ip_address>/mysql

7. Run the restoration command.

   Executed on a restoration host.

   sudo innobackupex --use-memory=4G --apply-log ./
   
   sudo innobackupex --copy-back ./

8. Copy the my.cnf file to the mysql directory.

   Executed on a restoration host.

   sudo cp ./my.cnf  /porta_var/<ip_address>/mysql/my.cnf

9.  Set correct user rights.>

   Executed on a restoration host.

    sudo chown -R mysql:mysql /porta_var/<ip_address>/mysql/

10. Start the mysql service.

    Executed on a restoration host.

    sudo systemctl start porta-mysqld

Please note that the commands should be executed depending on the availability of directories.

Executed on a restoration host.

 sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_home_porta-um_apache_prompts/ \

/home/porta-um/apache/prompts/

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_asterisk_sounds/ \

/var/lib/asterisk/sounds/

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-sip_sounds/ \

/var/lib/porta-sip/sounds/

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-um_prompts/ \

/var/lib/porta-um/prompts/

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-um_video_brands/ \

/var/lib/porta-um/video/brands/

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_porta_var_psmsc_custom-files_prompts/ \

/porta_var/psmsc/custom-files/prompts/

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_psmsc_prompts/ \

/var/lib/psmsc/prompts/

Deposit files are placed into the directories which names are identical to the file patch + file name. So they also can be copied with rsync command in the following way:

• file

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/deposits/<date_time>/<directory_name>/<file_name> \

<local_path_to_file>

•  directory

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/deposits/<date_time>/<directory_name>/ \

<local_path_to_directory>/

Please note that paths on your system might be different. For example /porta_var/porta-call-recording-1/ can be /porta_var/porta-call-recording-2/ or something like that.

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/call_recording/<date_time>/_porta_var_porta-call-recording-1_call-recording_wav/

/porta_var/porta-call-recording-1/call-recording/wav/

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/statistics/<date_time>/_porta_var_porta-billing-web-1_statistics/ \

/porta_var/porta-billing-web-1/statistics/

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_patches/<date_time>/_porta_var_custom_patches/ \

/porta_var/custom_patches/

The following steps should be performed to restore data from Elasticsearch snapshot.

Executed on a restoration host.

1. Find the IP address which is used by Elasticsearch and its http port.

    export elasticip=`grep 'network.host:' /etc/elasticsearch/elasticsearch.yml| cut -d ' ' -f2`
   
   export elasticport=`grep 'http.port:' /etc/elasticsearch/elasticsearch.yml| cut -d ' ' -f2`

2. Verify that snapshots repository “archivist” exists.

   curl -XGET "http://${elasticip}:${elasticport}/_cat/repositories?v"

3.  Check for available snapshots in the “archivist” repository.

    curl -XGET "http://${elasticip}:${elasticport}/_snapshot/archivist/_all?pretty"

4.  Check the state of all available indices.

    curl -XGET "http://${elasticip}:${elasticport}/_cat/indices/_all?v&pretty"

5. Close all indices before restoring the snapshot.

    curl -XPOST "http://${elasticip}:${elasticport}/_all/_close/?v&pretty"

6. Check that all indices are closed.

    curl -XGET "http://${elasticip}:${elasticport}/_cat/indices/_all?v&pretty"

7. Restore the snapshot (e.g., the “snapshot-1501664714” snapshot).

    curl -XPOST "http://${elasticip}:${elasticport}/_snapshot/archivist/snapshot-1501664714/_restore?wait_for_completion=true&pretty"

8. Check that all indices are open and their “health” is marked as “green”.

    curl -XGET "http://${elasticip}:${elasticport}/_cat/indices/_all?v&pretty"