Recipe "Easy System Cloning (DSESC) Backup"

From SMW CindyKate - Main
Component 947746
Jump to: navigation, search

Content


Understanding

For background and reasoning see https://dataspects.com/easy-system-cloning and for a conceptual overview see Concept "Easy System Cloning (DSESC)".

  • You can run multiple backup configurations which use different sources, destinations, options and encryptions keys.
  • For each backup configuration you have to choose a BACKUP_CONFIGURATION_NAME.
  • A specific BACKUP_CONFIGURATION_NAME backs up ONE specific source to ONE specific destination.

Setup

All the following steps must be carried out as the user that is going to run the backup jobs! In case you need to set up a new user for this on Ubuntu, see how-to-create-a-sudo-user-on-ubuntu-quickstart.

Clone the Git repository

git clone [email protected]:dataspects/EasySystemCloning_DSESC_Backup.git BACKUP_CONFIGURATION_NAME

Check the environment

Fundamental software

BACKUP_CONFIGURATION_NAME$ chmod +x check_environment.sh && ./check_environment.sh

You should get an output like this one:

DSESC check_environment.sh
---
GnuPG: gpg --version
gpg (GnuPG) 1.4.16
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2
# In case this is missing you need to install https://gnupg.org.

---
MYSQLDUMP: mysqldump --version
mysqldump  Ver 10.14 Distrib 5.5.46-MariaDB, for debian-linux-gnu (x86_64)
# In case this is missing you need to install https://dev.mysql.com/doc/refman/5.5/en/mysqldump.html.

---
DUPLICITY: duplicity --version
duplicity 0.6.23
# In case this is missing you need to install http://duplicity.nongnu.org.

---
GnuPG Keys: gpg --list-keys
gpg: directory `/home/vagrant/.gnupg' created
gpg: new configuration file `/home/vagrant/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/vagrant/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/vagrant/.gnupg/pubring.gpg' created
gpg: /home/vagrant/.gnupg/trustdb.gpg: trustdb created

The last 6 lines are shown in case you have not used GnuPG on this server before. Otherwise gpg --list-keys is either empty or listing the keys installed like this:

/home/user/.gnupg/pubring.gpg
----------------------------
pub   4096R/023D23A1 2016-09-06
uid                  User Name (Description) <[email protected]>
sub   4096R/2G738E12 2016-09-06

Interfaces

Depending on your choice of backup destination, you might need to install necessary packages providing interfaces to e.g. Amazon Web Services S3 (e.g. python-boto).

Provide or create a GnuPG keypair

See Recipe "How to create, configure, export and import GnuPG keys"

Set up the following structure based on the SMW Factory Components below

[email protected]:/home/user/duplicity_backup_main$ tree
.
├── backup.sh                        # Contains the duplicity command execution for remote backup storage
├── backup_local.sh                  # Contains the duplicity command execution for local backup storage
├── duplicity_configuration          # Contains the duplicity command configuration for remote backup storage
├── duplicity_configuration_local    # Contains the duplicity command configuration for local backup storage
├── mysqldump.sh                     # Contains the mysqldump command execution
└── smw_configuration                # Contains the SMW database credentials and dump configuration

SMW Factory Components

smw_configuration


SMW_MYSQL_HOST=
SMW_MYSQL_DATABASE=
SMW_MYSQL_USER=
SMW_MYSQL_PASSWORD=
SMW_DUMPFILE=/var/www/html/m/smw_dump.sql

referred by

mysqldump.sh


#!/bin/sh

. /home/user/duplicity_backup_main/smw_configuration

mysqldump \
--password=${SMW_MYSQL_PASSWORD} \
--user=${SMW_MYSQL_USER} \
--host=${SMW_MYSQL_HOST} \
--complete-insert \
--add-drop-database \
--add-drop-table \
--compact \
--create-options \
--no-create-db \
--result-file=${SMW_DUMPFILE} \
${SMW_MYSQL_DATABASE}

referred by

Duplicity Configuration


duplicity_configuration

export ENCRYPT_KEY=
export PASSPHRASE=

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

export SOURCE=/var/www/html/m/
export DESTINATION="s3://s3.amazonaws.com/"

export DAILYLOGFILE=/var/log/duplicity/backup.daily.log

duplicity_configuration_local

export ENCRYPT_KEY=
export PASSPHRASE=




export SOURCE=/var/www/html/m/
export DESTINATION="file://"

export DAILYLOGFILE=/var/log/duplicity/backup.daily.log


referred by

Backup Shell Scripts


backup.sh

#!/bin/bash

/home/user/duplicity_backup_main/mysqldump.sh

. /home/user/duplicity_backup_main/duplicity_configuration

echo ${ENCRYPT_KEY}
echo ${SOURCE}
echo ${DESTINATION}

duplicity --full-if-older-than 1W -v9 --encrypt-key ${ENCRYPT_KEY} ${SOURCE} ${DESTINATION} >> ${DAILYLOGFILE}

backup_local.sh

#!/bin/bash

/home/user/duplicity_backup_main/mysqldump.sh

. /home/user/duplicity_backup_main/duplicity_configuration_local

echo ${ENCRYPT_KEY}
echo ${SOURCE}
echo ${DESTINATION}

duplicity --full-if-older-than 1W -v9 --encrypt-key ${ENCRYPT_KEY} ${SOURCE} ${DESTINATION} >> ${DAILYLOGFILE}


referred by

Scheduled (See Ubuntu CronHowto)

30 02 * * * /home/user/duplicity_backup_main/backup.sh

Ad hoc

/home/user/duplicity_backup_main/backup.sh
/home/user/duplicity_backup_main/backup_local.sh