Release

Search

Configure backups with Archivist

Link copied to clipboard

Introduction

Link copied to clipboard

The Archivist utility allows you to configure and create backups to make sure that the most critical data on your installation is preserved. You can back up the following kinds of data:

  • MySQL database (hot backups performed while the database is running will not interfere with normal database operations).
  • Cassandra database (storage for invoices, xDRs, reports, billing statistics, etc.).
  • Recorded phone conversations.
  • The general configuration of a server’s operating system (the “/etc” folder).
  • Applied custom modifications (added to the deposit files).
  • Custom RPM files.
  • Voice prompt files.
  • Radacc files.
  • Opensearch database (storage for log files and CDR collections).

How to access Archivist

Link copied to clipboard

You can access Archivist from the Configuration server web interface:

  1. Log in to the Configuration server web interface.
  2. Click the house Home button in the right-hand corner of the page.
  3. Click the Archivist button.

    Access Archivist

Configure a backup procedure using the default settings

Link copied to clipboard

timesaverYou can use preset settings to configure a backup procedure. In this case you specify a time period during which the system should perform a backup. The system selects the kinds of data that it’s possible to back up within this period and will back up these kinds of data to the Configuration server from all the servers that contain them. However, backup parameters can be adjusted later as described in this section and in the Configure a backup procedure task by task section.

When you apply default settings all currently configured backup tasks are removed.
  1. At the top of the Tasks panel, click the add Default settings button.

    Input time period for backup tasks

  2. Specify the time period during which the backup procedure should be executed. Time period should be specified in the format “hr{hh:mm-hh:mm}.”

     

    For example, if you specify hr{00:00-03:30}, it means that the backup procedure will start after 00:00 and finish before 03:30. If the backup procedure is not completed before 03:30, the system terminates the procedure.

  3. Select Remove all current tasks to confirm that you know that the default settings will override the current ones.
  4. Click Next.

    Review and adjust the list of backup tasks to be executed.

  5. You can review and adjust the list of backup tasks to be executed.

    Adjust the scheduled time and frequency of backup

    • To adjust the scheduled time and frequency of backup, click on the Schedule (in cron format) column and type the required value in the cron format, i.e., “Minute(0,..,59) Hour(0,..,23) Day(1,..,31) Month(1,..,12 or Jan,..,Dec) Weekday(0,1,..,7 or Sun,Mon,..,Sun).” Use a hyphen to indicate a range of values, a comma to indicate a list of values and an asterisk * to indicate every possible value.

       

      For example, if you specify Schedule (in cron format): 30 2 5,25 * 5-7, then your backup procedure will be scheduled to be performed at 02:30 on the 5th and 25th of every month as well as on every Friday, Saturday and Sunday.

      Adjust the timeout

    • To adjust the timeout, click on the Timeout (min.) field and specify the required value. If the backup task is not completed within the timeout period, the system terminates the task.

       

      For example, if you specify Timeout (min.): 180, then the backup process should finish within 3 hours from starting.

      Make sure that you give your system enough time to complete the backup task. Many kinds of data require no more than 10 minutes, but for example, a MySQL database backup procedure can take more than 2 hours.

       

      Adjust the number of recent backups to be kept

    • To adjust how many of the most recent backups the system should keep, click on the Backup count field and specify the required value.
    • To remove the task click on the remove Remove button.
  6. Click Apply.
  7. If necessary, now you can adjust settings for every backup task as described in steps 1–7 of the Configure a backup procedure task by task section.

Configure a backup procedure task by task

Link copied to clipboard

A backup task is a combination of a type of data that you want to back up, a backup source and backup destinations. For example, one can define such a backup task as, “Backup Opensearch data from PortaBilling Web server to the Configuration server and to the remote FTP server.” To configure a backup procedure task by task, please complete the following steps.

Configure a backup procedure task by task

  1. In the Tasks panel, click on the kind of data that you want to back up (e.g., “Backup Opensearch”).

    Select Backup Opensearch

  2. In the Backup Source panel, select the checkboxes against the servers where this data is located (e.g., select your PortaSwitch Web servers if your logs are located on them).

    Select servers

    tips If you want to review which instances are configured on which server, click the expand-collapse Expand/Collapse button at the Backup Source panel top.

    Expand-Collapse section

  3. To adjust backup parameters, in the Backup Source panel, click on the first of the backup sources selected in the previous step.
    Be careful to click away from the checkbox, because when you click on the checkbox and clear it, you remove the backup source.
  4. In the Schedule panel, specify when the backup should be performed, how long the backup process can be active for and how many of the most recent backups the system must keep.
    • Minute – type minutes in diapason from 0 to 59 or *, where * corresponds to every minute.
    • Hour – type hours in diapason from 0 to 23 or *, where * corresponds to every hour.
    • Day – type days of the month in diapason from 1 to 31 or *, where * corresponds to every day.
    • Month – type months in diapason from 1 to 12 or *, where * corresponds to every month. You can use the format Jan / JAN as well.
    • Weekday – type weekday numbers in diapason from 0 to 6 or *, where 0 (and 7) correspond to Sunday, 6 to Saturday and so on and where * sets the weekday to undefined. You can use the format Mon / MON as well.
    • Timeout – type the maximum time in minutes that you want to allocate for the backup process. If the process is not completed within this period from its start, the system terminates it.

      Make sure that you give your system enough time to complete the backup task. Many kinds of data require no more than 10 minutes, but for example, a MySQL database backup procedure can take more than 2 hours.

    • Allowed indices – specify the Opensearch indices to back up in the following format: <index1>*,<index2>*,<index3>*, etc., where indexN is one of the Opensearch indices. For example, to configure Opensearch backup for SIP and BE logs only, specify “siplogs*,belogs*” value. To configure Opensearch backup for all indices, specify *.
    • Backup count – type the number of the most recent backups to keep.
You can use a hyphen to indicate a range and a comma to indicate a list of values, e.g., specify 1-6 in the Day field and 1,3,5 in the Month field and and mon-wed in the Weekday field. It means that the backup process is taken each day from the 1st till the 6th of January, March, and May, and also from Monday to Wednesday each week.

For example, if you specify Minute: 30; Hour: 2; Day: 5,25; Month: *; Weekday: 5-7; Timeout: 180; Request timeout: 90; Allowed indices: siplogs*,belogs*; Backup count: 5; then your backup procedure will be scheduled to be performed at 02:30 on the 5th and 25th of every month as well as on every Friday, Saturday and Sunday. The backup process should finish within 3 hours from its start, and the system will keep the most recent 5 backups of this data.

You can review the descriptions of configured schedule settings and see exactly when the next backup will be performed on the bottom of the Schedule panel.

configure Opensearch backup for SIP and BE logs only

tips In addition, after several backup runs, you can see how long it takes to back up selected types of data. Check the following performance characteristics:

  • Avg execution time shows how long it takes to copy data to the final destination (if single destination is used) or to the Configuration server (used as an intermediary between a backup source and final destinations when you select more than one or an external destination). This value is of higher importance because the source server is actively engaged in the backup procedure during this time.
  • Avg upload time shows how long it takes to copy data from the Configuration server to final destinations.

     

    Utilize this information to adjust the time frames for backup procedures if required.

    Note that to provide the best performance and to avoid mistakes it is recommended that backup procedures be scheduled for off-peak periods. Make sure they don’t overlap with resource-intensive calculations such as creating statistics/invoices.

  1. In the Backup Destination panel, select servers where files with backup data will be placed. If you want to add a new remote destination, please refer to the Create a new remote destination section of this handbook.

    Note that when you select more than one destination or if you select an external destination, the configurator backup destination is automatically selected too. The reason for this is that sending backups to external or several destinations creates an additional load on the server, therefore it’s safer to perform this operation from the server that is not actively involved in performing critical real-time services – the Configuration server. So the system first copies backup files to the Configuration server and only then copies them from the Configuration server to specified destinations. (Note that the backup copy on the Configuration server won’t be deleted after the backup task finishes.)

    tips It’s recommended to choose more than one destination per source.

    Please make sure that there is enough free space on the destination servers.

     

    Schedule the backup procedure

  2. If you have selected more than one backup source in step 2, repeat steps 4–5 for the rest of them.
  3. Click the Save button at the bottom of the page.
  4. If you want to configure the backup procedure for any other kinds of data, repeat steps 1–7.

Create a new remote destination

Link copied to clipboard
  1. At the top of the Backup Destination panel, click the add Add button.

    Create a new remote destination

  2. In the Add custom destination dialog box, specify the following options:
    • Server name – type the name under which you want this destination to be shown in the list.
    • Protocol – select the protocol to be used to send files to the remote destination. Available options are “FTP,” “SSH (SCP),” and “RSync over SSH”.
    • Path – specify the folder on your server where the system will save your backup files, use the following format: “/folder_name”. The system creates two additional nested subfolders:  “/IP_of_the_server_it_saves_data_from/A_kind_of_data_you_are_backing_up.”

       

      For example, if you specify “/backup” for this option and you back up data in the Cassandra database from the server with IP “10.10.10.10,” the system will save your data in:

      /home/backup/10.10.10.10/cassandra.

      If you want to edit local destination, the Path option is the only one you can modify. By default the Path option is set to "/porta_var/backup" for every local server.

    • Hostname – type the IP address of your destination server.
    • Port – type the port for connection.
    • Login – type the login for authorization on the destination server.
    • Password – type the password for authorization on the destination server.
    • Login Script – if required, for SSH (SCP) and RSync over SSH protocols you may specify the path to the login script file. The login script includes a set of commands that are executed each time a user logs into the remote destination via an ssh connection. The script is mostly used to allow password-less user authentication. The Login parameter is still mandatory here to explicitly define the user who connects to the remote destination.
    • Compression level – set the level of backup files compression, when files are sent via SSH (SCP) and FTP protocols, from 0 to 9. If you set “0”, it means that the compression of files is the lowest (but takes less time). If you set “9”, the compression level is the highest, but it takes more time. The default level is “3”.
    • Compression threads – set the number of threads to compress files that are sent via SSH (SCP) and FTP protocols. The default value is “2” which means that the processor (CPU) cores will use 2 threads to compress the files. If you set “0”, it means that all available CPU cores will be used for compression. For example, 4 threads will be used for compression on the quad-core CPU.

       

      The login script can also be used to add additional ssh connection options such as connect request timeout, number of connection attempts, etc.

  3. Click Save.

Remove scheduled backup task

Link copied to clipboard
  1. In the Task panel, click on the kinds of data that you no longer want to back up.
  2. In the Backup Source panel, clear the checkboxes against the servers where this data is located.
  3. Click the Save button at the bottom of the page.
  4. If you don’t want to back up any other kinds of data, repeat steps 1–3.
    You need to repeat all three steps for each kind of data that you no longer want to back up. If you clear the checkboxes against the servers where this data is located for one kind of data and then move to another without clicking Save, then all the changes that you made will be lost.

Run a backup task manually

Link copied to clipboard

The main purpose of this option is to give you a possibility to check whether a backup task configuration is operable: that the chosen backup source contains the required data and that the connection to the backup destination works properly. We recommend that you always double-check a backup task configuration when you select remote destinations.

Run a backup task manually

  1. In the Tasks panel, click on the kind of data that you configured the backup task for (e.g., “Backup Cassandra”).
  2. In the Backup Source panel, click on the required backup source.
    Be careful to click away from the checkbox, because when you click on the checkbox and clear it, you remove the backup source.
  3. At the top of the Schedule panel, click the runnow Run Now button.

Check backup tasks in the calendar view

Link copied to clipboard

Use the Calendar view to get an overview of forthcoming and finished backup tasks by month, week or day.

Open the calendar

To open the Calendar view, click the calendar Show Calendar button at the top of the Schedule panel.

Check backup tasks in the calendar view

Check backup logs, stop running backup tasks and remove backup files in the logs viewer

Link copied to clipboard

The system prompts you to open the Logs Viewer every time you run the backup procedure manually (for how to perform this, see the Run a backup task manually section). However, you can check your backup logs with the Logs Viewer anytime you need to.

Open the log viewer

To open the Logs Viewer click the log Logs button at the top of the Schedule panel.

Check the stage of the backup procedure

The Logs Viewer shows the logs at the run time of the backup process. So you can always check what stage the backup procedure is working on.

  • To review a log, select the backup task from the list in the left-hand panel. You can review log details in the right-hand panel.
  • To terminate the currently running backup procedure, select it in the left-hand panel and click the killtask Kill task button.
  • To delete a backup and all the logged information about it, select the backup in the left-hand panel and click the delete Delete button.
  • To refresh a log immediately click the refresh Refresh button or set it to refresh automatically by clicking the arrow on the right-hand side of the Refresh button and selecting the required interval from the list.
  • To close the Log Viewer, click the closelogviewer Close button in the top right-hand corner of the page.

The backup tasks in the left-hand panel are grouped by backup sources and kinds of backup data. The icons near the backup task items indicate information about the status of the backup tasks. Backup tasks can have the following status:

  • startedtask Started – the backup task is waiting for processing.
  • running Running – the backup task is being executed.
  • donetask Done – the data was copied to the backup destination (when only one backup destination was selected) or to the Configuration server (when more than one destination was selected), but the synchronization script is still being executed.
  • successtask Synced – the backup task has successfully finished.
  • killedtask Killed – the backup task was manually terminated.
  • failedtask Fail – the backup task wasn’t completed because of errors.
  • umknowntask Unknown status – the system cannot identify the backup task status.

Data recovery

Link copied to clipboard

The recovery procedure can be manually executed from the CLI (command-line interface). Please contact support@portaone.com for more information on recovery procedures.

How to use rsync for remote hosts

Link copied to clipboard

If your backup is located on another server you can perform rsync command remotely.

To copy something from the remote host
Link copied to clipboard

Executed on a restoration host.

 sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

  <ip_address>:/remote/path/to/file /local/path/to/file
To copy something from the remote host
Link copied to clipboard

Executed on a host with backups.

 sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

  /local/path/to/file <ip_address>:/remote/path/to/file

How to restore /etc

Link copied to clipboard

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

  --rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/etc_dir/<date_time>/_etc/ /etc/

How to restore Cassandra

Link copied to clipboard
If restoring a single node, you must first shutdown the node. If restoring an entire cluster, you must shut down all nodes, restore the snapshot data, and then start all nodes again.

Note that the process must be executed on all nodes in the cluster, otherwise nodes that did not get the restored data will “update” the restored nodes with the newer, bad data.

  1. Shut down the Cassandra node.

    Executed on a restoration host.

    sudo systemctl stop cassandra
  2. Clear all files in /var/lib/cassandra/commitlog/

    Executed on a restoration host.

    sudo rm -rf /porta_var/<ip_address>/cassandra/commitlog/*.log

    Delete all .db files in the cassandra directory /porta_var/<ip_address>/cassandra/data/ <keyspace_name>/<table_name>/ Executed on a restoration host.

     sudo rm -rf /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/*
  3. Find the most recent snapshot folder in this directory.
    /porta_var/backup/<ip_address>/cassandra/<date_time>/porta_var_<cassandra_node>_cassandra_data_<keyspace_name>_<table_name>_snapshots_incremental
  4. Copy its contents into this directory /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/

    Executed on a restoration host.

     sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \
    
      --rsh=/home/porta-one/scripts/rsh_porta.sh \
    
    <remote_ip_address>:/porta_var/backup/<ip_address>/cassandra/<date_time>/_porta_var_<cassandra_node>_cassandra_data_<keyspace_name>_<table_name>_snapshots_incremental/ \
    
    /porta_var/<ip_address>/cassandra/data/<keyspace_name>/<table_name>/
  5. Restart the Cassandra node.

    Executed on a restoration host.

     sudo systemctl start cassandra
  6. Restarting causes a temporary burst of I/O activity and consumes a large amount of CPU resources. Run nodetool repair.

    Executed on a restoration host.

    sudo nodetool -h localhost -p 7199 repair

How to restore mysql

Link copied to clipboard
  1. Create a new directory on restoration host where you will copy the backup file with the mysql database.

    Executed on a restoration host.

    mkdir /porta_var/tmp/mysql_restore_$(date -I)
  2. Copy the backup file with the mysql database locally.

    Executed on a restoration host.

    cd /porta_var/tmp/mysql_restore_$(date -I)
    
    rsync -v --rsh=/home/porta-one/scripts/rsh_porta.sh \
    
    <ip_address>:/porta_var/backup/<ip_address>/mysql/<date_time>/mysql.xbstream .
    
    rsync -v --rsh=/home/porta-one/scripts/rsh_porta.sh \
    
    <ip_address>:/porta_var/backup/<ip_address>/mysql/<date_time>/my.cnf .
  3. Untar the backup file.

    Executed on a restoration host.

    xbstream -x < ./mysql.xbstream -C ./
  4. Stop the mysql service.

    Executed on a restoration host.

    sudo systemctl stop porta-mysqld
  5. Delete or move the old mysql database to some place (be very careful!).

    Executed on a restoration host.

    sudo mv /porta_var/<ip_address>/mysql /porta_var/tmp/mysql_backup_$(date -I)
  6. Create a mysql directory.

    Executed on a restoration host.

    sudo mkdir /porta_var/<ip_address>/mysql
  7. Run the restoration command.

    Executed on a restoration host.

    sudo innobackupex --use-memory=4G --apply-log ./
    
    sudo innobackupex --copy-back ./
  8. Copy the my.cnf file to the mysql directory.

    Executed on a restoration host.

    sudo cp ./my.cnf  /porta_var/<ip_address>/mysql/my.cnf
  9.  Set correct user rights.>

    Executed on a restoration host.

     sudo chown -R mysql:mysql /porta_var/<ip_address>/mysql/
  10. Start the mysql service.

    Executed on a restoration host.

    sudo systemctl start porta-mysqld

How to restore prompts

Link copied to clipboard

Please note that the commands should be executed depending on the availability of directories.

Executed on a restoration host.

 sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_home_porta-um_apache_prompts/ \

/home/porta-um/apache/prompts/
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_asterisk_sounds/ \

/var/lib/asterisk/sounds/
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-sip_sounds/ \

/var/lib/porta-sip/sounds/
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-um_prompts/ \

/var/lib/porta-um/prompts/
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_porta-um_video_brands/ \

/var/lib/porta-um/video/brands/
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_porta_var_psmsc_custom-files_prompts/ \

/porta_var/psmsc/custom-files/prompts/
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_prompts/<date_time>/_var_lib_psmsc_prompts/ \

/var/lib/psmsc/prompts/

 

How to restore Deposit files

Link copied to clipboard

Deposit files are placed into the directories which names are identical to the file patch + file name. So they also can be copied with rsync command in the following way:

  • file

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/deposits/<date_time>/<directory_name>/<file_name> \

<local_path_to_file>
  •  directory
sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/deposits/<date_time>/<directory_name>/ \

<local_path_to_directory>/

 

How to restore callrecording

Link copied to clipboard

Please note that paths on your system might be different. For example /porta_var/porta-call-recording-1/ can be /porta_var/porta-call-recording-2/ or something like that.

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/call_recording/<date_time>/_porta_var_porta-call-recording-1_call-recording_wav/

/porta_var/porta-call-recording-1/call-recording/wav/

How to restore custom RPMs

Link copied to clipboard

Executed on a restoration host.

sudo rsync --numeric-ids --delete -aHSv --rsync-path='sudo rsync' \

--rsh=/home/porta-one/scripts/rsh_porta.sh \

<remote_ip_address>:/porta_var/backup/<ip_address>/custom_patches/<date_time>/_porta_var_custom_patches/ \

/porta_var/custom_patches/

How to restore Opensearch

Link copied to clipboard

The following steps should be performed to restore data from Opensearch snapshot.

Executed on a restoration host.

  1. Find the IP address which is used by Opensearch and its http port.
     export opensearchip=`grep 'network.host:' /etc/opensearch/opensearch.yml| cut -d ' ' -f2`
    
    export opensearchport=`grep 'http.port:' /etc/opensearch/opensearch.yml| cut -d ' ' -f2`
  2. Verify that snapshots repository “archivist” exists.
    curl -XGET "http://${opensearchip}:${opensearchport}/_cat/repositories?v"
  3.  Check for available snapshots in the “archivist” repository.
     curl -XGET "http://${opensearchip}:${opensearchport}/_snapshot/archivist/_all?pretty"
  4.  Check the state of all available indices.
     curl -XGET "http://${opensearchip}:${opensearchport}/_cat/indices/_all?v&pretty"
  5. Close all indices before restoring the snapshot.
     curl -XPOST "http://${opensearchip}:${opensearchport}/_all/_close/?v&pretty"
  6. Check that all indices are closed.
     curl -XGET "http://${opensearchip}:${opensearchport}/_cat/indices/_all?v&pretty"
  7. Restore the snapshot (e.g., the “snapshot-1501664714” snapshot).
     curl -XPOST "http://${opensearchip}:${opensearchport}/_snapshot/archivist/snapshot-1501664714/_restore?wait_for_completion=true&pretty"
  8. Check that all indices are open and their “health” is marked as “green”.
     curl -XGET "http://${opensearchip}:${opensearchport}/_cat/indices/_all?v&pretty"

On this page

Release
What's new
Admin manuals
Handbooks
Developers documentation
UI help
Search