Backup & Recovery Procedures

Comprehensive backup and disaster recovery procedures for MDHosting Ltd infrastructure, ensuring business continuity and data protection.

Overview

MDHosting's backup strategy is designed to protect client data and ensure rapid recovery in the event of data loss, hardware failure, or security incidents. All backups are stored off-site at Hetzner's dedicated backup storage facility for maximum resilience.

Backup Philosophy: - Defence in Depth: Multiple backup retention periods (daily, weekly, monthly) - Off-Site Storage: Backups stored separately from production servers - Automated Operations: Daily automated backups with error notification - GDPR Compliance: Client data backups follow data retention policies - Tested Recovery: Regular testing ensures backups are functional

Key Metrics: - RPO (Recovery Point Objective): 24 hours (daily backups) - RTO (Recovery Time Objective): <4 hours for critical systems, <24 hours for full server - Backup Success Rate Target: 99%+ - Off-Site Storage: Hetzner BX11 Storage Box (separate infrastructure)

Current Backup Infrastructure

Backup Architecture

graph TB
    subgraph "Production Servers - Hetzner Germany"
        EU1[eu1.cp<br/>CPX31 Hosting Server<br/>~30 Client Accounts]
        NS1[ns1.mdhosting.co.uk<br/>DNS Server 1]
        NS2[ns2.mdhosting.co.uk<br/>DNS Server 2]
    end

    subgraph "Backup Storage - Hetzner Germany"
        BX11[Hetzner BX11<br/>Storage Box<br/>Off-Site Backup Storage]
    end

    subgraph "Backup Schedule"
        DAILY[Daily Backups<br/>Automated via cPanel]
        WEEKLY[Weekly Retention]
        MONTHLY[Monthly Retention]
    end

    EU1 -->|SSH/SFTP Transfer| BX11
    NS1 -->|Configuration Backups| BX11
    NS2 -->|Configuration Backups| BX11

    DAILY --> WEEKLY
    WEEKLY --> MONTHLY

    BX11 -->|Recovery Operations| EU1

    classDef production fill:#3498db,stroke:#2c3e50,stroke-width:2px,color:#fff
    classDef backup fill:#27ae60,stroke:#2c3e50,stroke-width:2px,color:#fff
    classDef schedule fill:#f39c12,stroke:#2c3e50,stroke-width:2px,color:#fff

    class EU1,NS1,NS2 production
    class BX11 backup
    class DAILY,WEEKLY,MONTHLY schedule

Backup Solution

Primary Backup System: cPanel/WHM Backup Configuration

Technology Stack: - Backup Engine: cPanel Backup System (built-in) - Transfer Method: SSH/SFTP to remote storage - Storage Location: Hetzner BX11 Storage Box (off-site) - Scheduling: cron-based daily execution - Notification: Email alerts on backup failures

Storage Box Specifications: - Provider: Hetzner Storage Box BX11 - Location: Hetzner Germany datacenter (separate from hosting) - Capacity: 1 TB - Access: SSH/SFTP, CIFS/Samba, FTP - Redundancy: Hetzner-managed RAID storage

What Gets Backed Up

eu1.cp (Primary Hosting Server)

User Account Data: - ✅ Home directories (/home/username/) - Website files (public_html, subdomains) - Email data (mail directories) - Private files (.ssh, logs, temporary files) - Configuration files (.htaccess, .user.ini)

Databases: - ✅ MySQL/MariaDB databases - All client databases - Database users and permissions - Full database dumps in SQL format

Email Accounts: - ✅ Mailboxes (IMAP/POP3 mail storage) - ✅ Email filters and forwarding rules - ✅ Autoresponders configuration - ✅ Mailing lists

DNS Zones: - ✅ All DNS zone files - ✅ Zone configuration and serial numbers - ✅ Custom DNS records

Server Configuration: - ✅ cPanel account configurations - ✅ Apache/nginx virtual host configurations - ✅ PHP-FPM pool configurations - ✅ SSL/TLS certificates (Let's Encrypt) - ✅ Email server configuration (Exim, Dovecot) - ✅ FTP account configurations - ✅ Cron jobs

NOT Backed Up (By Design): - ❌ Operating system files (reinstall from ISO) - ❌ cPanel/WHM software (reinstall from repo) - ❌ System packages (yum/dnf reinstall) - ❌ Logs older than retention period - ❌ Temporary files and caches

DNS Servers (ns1, ns2)

DNS Configuration: - ✅ cPanel DNS zone files - ✅ PowerDNS configuration (when migrated) - ✅ Server configuration files

Note: DNS zones are primarily backed up from eu1.cp as the DNS master. DNS servers act as secondaries.

Backup Schedule & Retention

Backup Timing

Daily Backups: - Frequency: Every 24 hours - Execution Time: Automated via cron (exact time configured in cPanel) - Type: Full backups (entire account data) - Duration: 1-3 hours depending on data volume

Backup Window

Backups run during off-peak hours to minimise performance impact. Check WHM > Backup Configuration for exact schedule.

Retention Policy

Standard Retention Schedule:

Retention Period	Number of Copies	Description
Daily	7 days	Last 7 daily backups
Weekly	4 weeks	One backup per week for 4 weeks
Monthly	3 months	One backup per month for 3 months

Retention Example:

Daily:   [Day 1] [Day 2] [Day 3] [Day 4] [Day 5] [Day 6] [Day 7]
Weekly:  [Week 1] [Week 2] [Week 3] [Week 4]
Monthly: [Month 1] [Month 2] [Month 3]

Total Storage: Approximately 14 backup snapshots at any time (7 daily + 4 weekly + 3 monthly)

Automatic Rotation: - Oldest daily backup deleted after 7 days - Oldest weekly backup deleted after 4 weeks - Oldest monthly backup deleted after 3 months - Rotation handled automatically by cPanel backup system

Storage Management

Local Storage (Production Server): - ⚠️ Backups are removed from production server after transfer to off-site storage - Minimises disk usage on hosting server (160GB NVMe) - Reduces risk of simultaneous server and backup loss

Remote Storage (Hetzner BX11): - ✅ All backups stored on Hetzner Storage Box - 1 TB capacity for backup retention - Separate physical infrastructure from hosting servers - Accessible via SSH/SFTP for recovery operations

Current Usage Monitoring:

# Check storage box usage (from local machine with SSH access)
ssh [storage-box-user]@[storage-box-host] df -h

# List backups
ssh [storage-box-user]@[storage-box-host] ls -lh /path/to/backups/

Backup Verification

Automated Monitoring

cPanel Backup Notifications: - ✅ Email alerts sent to admin@mdhosting.co.uk - ✅ Success notifications (daily summary) - ✅ Failure notifications (immediate alert) - ✅ Backup log files in WHM

Email Alert Examples:

Success Notification:

Subject: Backup Success - eu1.cp - [Date]

Daily backup completed successfully.
- Accounts backed up: 30
- Databases backed up: 75
- Total size: 45 GB
- Duration: 2 hours 15 minutes
- Destination: Hetzner BX11 Storage Box

Failure Notification:

Subject: ⚠️ BACKUP FAILURE - eu1.cp - [Date]

Backup failed for the following accounts:
- clientdomain.com: Database dump failed (connection timeout)

Action Required: Investigate and retry manually.

Manual Verification

Monthly Backup Audit (Recommended):

Access Storage Box:

sftp [storage-box-user]@[storage-box-host]

Verify Recent Backups Exist:

ls -lh /backups/daily/
ls -lh /backups/weekly/
ls -lh /backups/monthly/

Check Backup Sizes:
Compare recent backup sizes for anomalies
Sudden size changes may indicate issues
Document baseline sizes for comparison
Test Backup Integrity:
Download a small account backup
Extract and verify archive integrity
Check database dumps are readable

Backup Integrity Test Procedure:

# Download a small backup
scp [storage-box]:/backups/daily/[account]-[date].tar.gz /tmp/

# Extract and verify
cd /tmp
tar -tzf [account]-[date].tar.gz | head -20

# Check database dump
tar -xzf [account]-[date].tar.gz mysql/[database].sql
head -50 mysql/[database].sql

Recovery Procedures

Recovery Scenarios

MDHosting recovery procedures cover four primary scenarios:

File Recovery - Restore individual files or directories
Account Recovery - Restore complete client account
Server Recovery - Rebuild failed server from backups
Disaster Recovery - Complete infrastructure failure

1. File Recovery (Individual Files/Folders)

Use Case: Client accidentally deleted files, restore specific files from backup.

Recovery Time: 15-30 minutes

Procedure:

Identify Required Backup:
Determine when file was last known to exist
Select appropriate backup date (daily/weekly/monthly)

Access Backup via WHM:

WHM → Backup → Backup Restoration
→ Select Account
→ Select Backup Date
→ Browse Files

Download Specific Files:
Navigate to required directory in backup
Download files to local machine
Verify file integrity
Restore to Client Account:
Upload files via cPanel File Manager or SFTP
Set correct ownership and permissions
Verify file functionality
Notify Client:
Confirm files restored successfully
Document restoration in support ticket

Alternative Method (Command Line):

# SSH to server
ssh root@eu1.cp

# Download backup from storage box
scp [storage-box]:/backups/daily/[account]-[date].tar.gz /tmp/

# Extract specific files
cd /tmp
tar -xzf [account]-[date].tar.gz home/[user]/public_html/[specific-file]

# Copy to client account
cp -a home/[user]/public_html/[specific-file] /home/[user]/public_html/

# Fix ownership
chown [user]:[user] /home/[user]/public_html/[specific-file]

# Clean up
rm -rf /tmp/[account]-[date].tar.gz /tmp/home

2. Database Recovery

Use Case: Database corruption, accidental table drop, restore to previous state.

Recovery Time: 15-45 minutes

Procedure:

Access Backup:

WHM → Backup → Backup Restoration
→ Select Account
→ Select Backup Date
→ MySQL Databases section

Download Database Dump:
Download .sql file for required database
Verify SQL dump is readable (check with text editor)
Prepare for Restoration:
CRITICAL: Backup current database state first (if any data exists)
Drop existing database (if replacing completely)
Or restore to new database name for comparison

Restore Database:

# Via command line
mysql -u root -p
CREATE DATABASE [database_name];
GRANT ALL ON [database_name].* TO '[user]'@'localhost';
exit

mysql -u root -p [database_name] < /tmp/database_backup.sql

Or via cPanel:

cPanel → phpMyAdmin
→ Select Database
→ Import tab
→ Choose SQL file
→ Execute

Verify Restoration:
Check table count and record counts
Test application functionality
Verify client website/application works
Clean Up:
Remove temporary SQL dump files
Document restoration

3. Full Account Recovery

Use Case: Complete account restoration after data loss, migration, or client request.

Recovery Time: 1-4 hours (depending on account size)

Procedure:

Preparation:
Identify backup date to restore from
Verify account doesn't currently exist (or will be overwritten)
Notify client of maintenance window if live

Download Backup:

# SSH to server
ssh root@eu1.cp

# Download backup from storage box
scp [storage-box]:/backups/[period]/cpmove-[account].tar.gz /home/

Restore via WHM:

WHM → Transfer Tool → Restore a Full Backup/cpmove File
→ Upload backup file (or select from /home/)
→ Click "Restore"
→ Wait for completion (monitor progress)

Verify Restoration:
✅ Website files present and accessible
✅ Databases restored and functional
✅ Email accounts operational (send/receive test)
✅ DNS records correct
✅ SSL certificate active
✅ Cron jobs configured
✅ Subdomain/addon domains working

Post-Restoration Checks:

# Check account exists
cat /etc/userdomains | grep [domain]

# Check disk usage
du -sh /home/[username]

# Check databases
mysql -e "SHOW DATABASES LIKE '[username]_%';"

# Check email
ls -lh /home/[username]/mail/[domain]/

# Test website
curl -I https://[domain].com

Client Notification:
Confirm restoration complete
Request client verify functionality
Document restoration details

4. Full Server Recovery (Disaster Recovery)

Use Case: Complete server failure, hardware replacement, catastrophic data loss.

Recovery Time: 4-24 hours (depending on data volume)

RTO Target: 24 hours RPO: 24 hours (last daily backup)

Prerequisites: - Access to Hetzner Storage Box with backups - New server provisioned (or existing server reinstalled) - Root SSH access to new server - cPanel/WHM license transferred or reactivated

Procedure:

Phase 1: Server Preparation (1-2 hours)

Provision New Server:
Deploy new Hetzner CPX31 (or equivalent)
Install AlmaLinux 8 (or AlmaLinux 10 for ApisCP migration)
Configure network and hostname
Update system packages

Install cPanel/WHM:

# Download and install cPanel
cd /home
curl -o latest -L https://securedownloads.cpanel.net/latest
sh latest
# Installation takes 30-60 minutes

Basic Server Configuration:
Configure hostname
Set up CSF firewall rules
Configure SSH (non-standard port, key auth)
Install additional required software
Configure Backup Destination:
Reconnect to Hetzner Storage Box
Configure SSH keys for backup access
Test connectivity to storage box

Phase 2: Backup Restoration (2-8 hours)

Download Backups from Storage Box:

# Create temporary directory
mkdir -p /backup_restore
cd /backup_restore

# Download all account backups from most recent daily backup
scp [storage-box]:/backups/daily/cpmove-*.tar.gz ./

# Or download specific accounts
scp [storage-box]:/backups/daily/cpmove-[account].tar.gz ./

Restore Accounts:

# Method 1: Via WHM (recommended for small batches)
WHM → Transfer Tool → Restore a Full Backup/cpmove File
→ Select each backup file
→ Restore one by one

# Method 2: Command line batch restore (faster for many accounts)
cd /backup_restore
for backup in cpmove-*.tar.gz; do
    /scripts/restorepkg $backup
    echo "Restored: $backup"
done

Monitor Restoration Progress:
Check /usr/local/cpanel/logs/cpbackup/ for logs
Monitor server load and disk space
Verify each account after restoration

Phase 3: Service Verification (1-2 hours)

DNS Configuration:
Update DNS records if server IP changed
Configure nameservers (ns1, ns2)
Wait for DNS propagation (up to 48 hours)
Use hosts file overrides for immediate testing

Verify All Services:

# Check web services
systemctl status httpd
systemctl status nginx

# Check email services
systemctl status exim
systemctl status dovecot

# Check databases
systemctl status mysql
mysql -e "SHOW DATABASES;"

# Check cPanel services
/scripts/restartsrv_httpd
/scripts/restartsrv_cpanel

Account-by-Account Verification:
- Test website access for each domain
- Send test email (SMTP)
- Receive test email (IMAP/POP3)
- Verify database connectivity
- Check SSL certificates (reissue if needed)
- Test cPanel access for sample accounts

Phase 4: Post-Recovery (1-2 hours)

Security Hardening:
- Reconfigure CSF firewall
- Enable Fail2Ban
- Configure SSH restrictions
- Review and lock down unnecessary services
- Run ClamAV malware scan
Monitoring Setup:
- Configure backup system to Storage Box
- Run initial backup to verify
- Set up monitoring alerts
- Configure Wazuh agent (if deployed)
Client Communication:
- Notify all clients of recovery completion
- Request clients verify their services
- Provide new IP address if changed
- Update support documentation
Documentation:
- Document cause of failure
- Record recovery timeline
- Note any issues encountered
- Update disaster recovery procedures
- Calculate RTO/RPO achieved

Disaster Recovery Checklist

Client Self-Service Restore

cPanel Backup Interface

Clients can restore their own files and databases without administrator intervention:

Access Method:

cPanel → Files → Backup Wizard

Available Restore Options:

Home Directory Restore:
Download full home directory backup
Restore via File Manager or FTP
Self-service file recovery
Database Restore:
Download database backups (.sql.gz files)
Restore via phpMyAdmin
Self-service database recovery
Email Forwarders and Filters:
Included in home directory backup
Restore via Email section

Limitations: - Only most recent backup available (if enabled) - May not include all retention periods - Administrator has access to full retention (daily/weekly/monthly)

Client Guidance:

Client Self-Service Restore Guide

To restore your own files: 1. Log in to cPanel 2. Navigate to Files → Backup Wizard 3. Click "Restore" 4. Select "Home Directory" or "MySQL Database" 5. Download the backup file 6. Extract and upload files as needed

If you need backups older than the most recent backup, contact support.

Backup Configuration

cPanel/WHM Backup Settings

Access Backup Configuration:

WHM → Backup → Backup Configuration

Current Configuration Settings:

Backup Type: - ✅ Full backups (complete account data) - Frequency: Daily - Retention: See retention policy above

Backup Destinations: - ✅ Remote destination: Hetzner BX11 Storage Box - Transfer type: SSH/SFTP - Compression: gzip (tar.gz format) - ❌ Local retention: Disabled (backups removed after upload)

Backup Scope: - ✅ User accounts (home directories) - ✅ Databases (MySQL dumps) - ✅ DNS zones - ✅ Server configuration - ✅ Email accounts and data

Notification Settings: - ✅ Email on backup completion: admin@mdhosting.co.uk - ✅ Email on backup failure: admin@mdhosting.co.uk - Log location: /usr/local/cpanel/logs/cpbackup/

Storage Box Configuration

Hetzner Storage Box Details: - Product: BX11 (1 TB capacity) - Access: SSH/SFTP (primary), FTP, CIFS/Samba (available) - Location: Hetzner datacentre (separate from hosting servers) - Authentication: SSH key (recommended) + password backup

Access Configuration:

# Test storage box connectivity
ssh -p 23 [storage-box-user]@[storage-box-host]

# SFTP access
sftp -P 23 [storage-box-user]@[storage-box-host]

# Mount as filesystem (optional, for manual operations)
sshfs [storage-box-user]@[storage-box-host]:/ /mnt/backups -p 23

Directory Structure:

/backups/
├── daily/
│   ├── cpmove-account1-[date].tar.gz
│   ├── cpmove-account2-[date].tar.gz
│   └── ...
├── weekly/
│   └── [weekly backups]
└── monthly/
    └── [monthly backups]

Backup Testing Schedule

Regular backup testing is essential to ensure recovery capabilities. MDHosting follows a structured testing schedule:

Monthly Testing

When: First Monday of each month Duration: 30-45 minutes Responsibility: System Administrator

Test Procedure:

Verify Backup Completion:
Check email notifications for last 7 days
Review backup logs in WHM
Confirm no failed backups
Storage Box Audit:
SSH to storage box
List all backup files
Check storage capacity usage
Verify retention policy is working
Test File Restore:
Select one client account at random
Download latest backup
Extract and verify 5-10 random files
Verify file integrity
Document Results:
Record test date and outcomes
Note any issues discovered
Update this documentation if needed

Quarterly Testing

When: First Monday of January, April, July, October Duration: 1-2 hours Responsibility: System Administrator

Test Procedure:

All Monthly Tests (as above)
Database Restore Test:
Select one database from backup
Restore to test database name
Verify table structure and record counts
Delete test database
Full Account Restore Test:
Select small client account (or create test account)
Perform full account restoration to test account name
Verify all components (files, databases, email, DNS)
Delete test account after verification
Document Results:
Calculate restoration time (actual RTO)
Record any issues or delays
Update disaster recovery estimates

Annual Testing

When: January (planning year) Duration: 4-8 hours Responsibility: System Administrator + Team Review

Test Procedure:

All Monthly and Quarterly Tests (as above)
Simulated Disaster Recovery:
Provision test server
Attempt full server recovery from backups
Time the entire process
Document every step
Backup Strategy Review:
Review retention policy adequacy
Assess storage capacity requirements
Evaluate backup technology (alternatives to cPanel backup)
Consider additional off-site locations
Update Documentation:
Update disaster recovery procedures based on simulation
Revise RTO/RPO targets if needed
Document lessons learned

Annual Report: - Backup success rate for the year - Number of restores performed - Average restoration time - Storage usage trends - Recommendations for improvements

Troubleshooting

Common Backup Issues

1. Backup Fails with "Disk Space Full"

Symptoms: - Backup error: "No space left on device" - Backup incomplete or failed

Diagnosis:

# Check server disk usage
df -h

# Check storage box usage
ssh [storage-box] df -h

# Check largest directories
du -h /home/ | sort -rh | head -20

Resolution: - If server disk full: Clear temporary files, old logs - If storage box full: Adjust retention policy or upgrade storage - Remove old backups manually if needed - Consider excluding log directories from backups

2. Backup Transfer Fails to Storage Box

Symptoms: - Backup completes locally but doesn't transfer - Email notification: "Remote transfer failed"

Diagnosis:

# Test storage box connectivity
ssh -p 23 [storage-box-user]@[storage-box-host]

# Check WHM backup logs
tail -100 /usr/local/cpanel/logs/cpbackup/[date].log

# Test SFTP transfer manually
sftp -P 23 [storage-box-user]@[storage-box-host]
put test.txt

Resolution: - Verify SSH keys are configured correctly - Check storage box authentication - Verify storage box has capacity - Check firewall rules (port 23/SSH) - Reconfigure backup destination in WHM

3. Database Backup Fails

Symptoms: - Account backup succeeds but database missing - Error: "MySQL dump failed"

Diagnosis:

# Check MySQL service
systemctl status mysql

# Test database access
mysql -u root -p -e "SHOW DATABASES;"

# Check database size
mysql -e "SELECT table_schema AS 'Database',
           ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS 'Size (MB)'
           FROM information_schema.tables
           GROUP BY table_schema;"

Resolution: - Verify MySQL is running - Check database permissions - Repair corrupted databases: mysqlcheck --repair --all-databases - Check disk space for temp files during dump - Manually dump problem database to test

4. Backup Runs Too Long

Symptoms: - Backup still running after 6+ hours - Server performance degraded during backup

Diagnosis:

# Check backup process
ps aux | grep backup

# Check I/O wait
top
# Look for high %wa (I/O wait)

# Check backup logs for slow accounts
tail -f /usr/local/cpanel/logs/cpbackup/[date].log

Resolution: - Identify large accounts causing delays - Consider excluding large log directories - Adjust backup compression level - Split backups across different times - Upgrade storage/network bandwidth

5. Backup Restoration Fails

Symptoms: - Restore process hangs or fails - Error: "Invalid archive" or "Corrupted backup"

Diagnosis:

# Test backup file integrity
tar -tzf /path/to/backup.tar.gz | head -20

# Check file size (compare to expected)
ls -lh /path/to/backup.tar.gz

# Try extracting to temp location
mkdir /tmp/test_restore
cd /tmp/test_restore
tar -xzf /path/to/backup.tar.gz

Resolution: - Download backup again (may be transfer corruption) - Try older backup if recent backup corrupted - Extract backup manually and restore components individually - Check storage box for backup file integrity - Report to Hetzner if storage box issue

Emergency Contact

Hetzner Support: - Web: https://www.hetzner.com/support - Emergency: Available 24/7 via support portal - Storage Box Issues: Create support ticket

cPanel Support: - Technical Support: Via WHM support portal - Documentation: https://docs.cpanel.net

Internal Escalation: - Primary: System Administrator - Backup: See Contacts

Security Considerations

Backup Data Protection

Encryption: - ⚠️ Backups are not encrypted in transit (SSH/SFTP provides transport encryption) - ⚠️ Backups are not encrypted at rest on storage box - 🔐 Storage box access protected by SSH keys + password - 🔐 Storage box access restricted to MDHosting infrastructure

Future Enhancement: - Consider implementing backup encryption (GPG) for sensitive client data - Evaluate encrypted storage solutions

Access Control: - ✅ Storage box accessible only via SSH key authentication - ✅ SSH keys stored securely in password manager - ✅ Storage box credentials not shared - ✅ Regular review of storage box access logs

GDPR Compliance: - Backups contain client personal data - Retention periods align with data retention policies - Right to erasure: Client account deletion includes backup removal - See GDPR Compliance for data subject rights

Backup Retention and Data Deletion

Account Deletion Process: 1. Remove account from production server 2. Remove account backups from storage box (all retention periods) 3. Document deletion in client records 4. Confirm deletion to client if requested (GDPR right to erasure)

Command to Delete Account Backups:

# SSH to storage box
ssh [storage-box]

# Remove all backups for deleted account
rm /backups/daily/cpmove-[account]-*.tar.gz
rm /backups/weekly/cpmove-[account]-*.tar.gz
rm /backups/monthly/cpmove-[account]-*.tar.gz

Disaster Recovery Planning

Business Continuity Scenarios

Scenario 1: Single Server Failure

Event: EU1 hosting server hardware failure

Impact: - All client websites offline - Email services disrupted - Client data at risk

Recovery Plan: - Provision new Hetzner server (1-2 hours) - Install cPanel (1 hour) - Restore all accounts from storage box (4-8 hours) - Update DNS if IP changed (24-48 hours for propagation) - Total RTO: 8-12 hours (24-48 hours for full DNS propagation)

Mitigation: - Regular backup testing - Documented recovery procedures - Pre-tested server provisioning process

Scenario 2: Storage Box Failure

Event: Hetzner Storage Box failure or data loss

Impact: - Loss of backup history - Cannot restore from backups - Production data still intact

Recovery Plan: - Contact Hetzner support immediately - Trigger immediate full backup to alternative location - Provision alternative backup storage (AWS S3, Backblaze B2) - Configure dual backup destinations - Total RTO: N/A (production not affected)

Mitigation: - Hetzner Storage Box has built-in redundancy - Consider secondary backup location for critical data - Regular backup testing ensures early detection

Scenario 3: Complete Datacentre Failure

Event: Hetzner Germany datacentre unavailable

Impact: - All servers offline (hosting + DNS + backups) - Complete service outage - Maximum business impact

Recovery Plan: - Provision servers at alternative provider (2-4 hours) - Access storage box backups (if accessible) - If storage box unavailable, download from recent local backups - Restore all accounts (8-12 hours) - Update nameserver records at internet.bs (1-24 hours) - Total RTO: 16-24 hours

Mitigation: - Future: Consider geographic redundancy (UK backup location) - Document alternative provider options (AWS, DigitalOcean) - Maintain offline copy of critical documentation - Regular RTO/RPO testing

Scenario 4: Ransomware Attack

Event: Ransomware encrypts production server data

Impact: - Client data encrypted and inaccessible - Potential backup infection if ransomware spreads - Ransom demand

Recovery Plan: - DO NOT pay ransom - Immediately isolate affected servers (network disconnect) - Verify backups not encrypted (storage box immutable) - Provision clean server - Restore from most recent clean backup - Conduct malware analysis to identify entry point - Total RTO: 8-12 hours

Mitigation: - Regular security updates - Fail2Ban and CSF firewall protection - Wazuh SIEM monitoring (planned) - ClamAV malware scanning - Regular backup testing ensures clean restore capability

RTO/RPO Summary

Recovery Scenario	RPO (Data Loss)	RTO (Recovery Time)	Priority
Single file restore	24 hours	15-30 minutes	High
Database restore	24 hours	15-45 minutes	High
Full account restore	24 hours	1-4 hours	High
Single server failure	24 hours	8-12 hours	Critical
Complete disaster recovery	24 hours	16-24 hours	Critical

Recovery Priority Order: 1. eu1.cp (client hosting) - Critical 2. ns1/ns2 (DNS) - High 3. Non-critical services - Medium

Future Improvements

Short-Term (Next 6 Months)

Implement quarterly backup restore testing
Document exact backup timing (identify cron schedule)
Create backup monitoring dashboard
Test backup encryption (GPG)
Verify storage box capacity planning for 1-2 year growth

Medium-Term (6-12 Months)

Implement secondary backup location (AWS S3 or Backblaze B2)
Automate backup integrity testing
Create client-facing backup status portal
Document ApisCP backup migration plan
Implement backup reporting metrics

Long-Term (12+ Months)

Evaluate continuous replication solutions
Consider geographic redundancy (UK backup storage)
Implement immutable backups (ransomware protection)
Automate disaster recovery testing
Evaluate backup solutions for post-ApisCP migration

Appendices

A. Quick Reference Commands

Check Backup Status:

# View recent backup logs
tail -100 /usr/local/cpanel/logs/cpbackup/$(date +%Y-%m-%d).log

# Check running backup processes
ps aux | grep backup

# View backup configuration
/usr/local/cpanel/bin/backup_config_get

Manual Backup Trigger:

# Trigger full backup manually (use with caution)
/scripts/cpbackup --force

# Backup single account
/scripts/pkgacct [username]

Storage Box Access:

# SSH to storage box
ssh -p 23 [storage-box-user]@[storage-box-host]

# SFTP to storage box
sftp -P 23 [storage-box-user]@[storage-box-host]

# List backups
ssh -p 23 [storage-box-user]@[storage-box-host] ls -lh /backups/daily/

Restore Commands:

# Restore account from backup
/scripts/restorepkg /path/to/cpmove-[account].tar.gz

# Restore account with new username
/scripts/restorepkg --newuser=[newname] /path/to/backup.tar.gz

B. Backup Testing Checklist

Monthly Backup Test: - [ ] Review backup success emails (last 7 days) - [ ] Check WHM backup logs for errors - [ ] SSH to storage box and verify backups exist - [ ] Check storage box capacity usage - [ ] Download and test one random account backup - [ ] Verify 5-10 random files from backup - [ ] Document test results

Quarterly Backup Test: - [ ] Complete all monthly tests - [ ] Restore one database to test database - [ ] Verify database integrity - [ ] Perform full test account restoration - [ ] Verify all account components - [ ] Time the restoration process - [ ] Document RTO achieved

Annual Disaster Recovery Test: - [ ] Complete all monthly and quarterly tests - [ ] Provision test server - [ ] Perform simulated full server recovery - [ ] Time entire recovery process - [ ] Test all services on recovered server - [ ] Document lessons learned - [ ] Update disaster recovery procedures - [ ] Generate annual backup report

C. Backup Retention Examples

Example Account Timeline:

Account: exampleclient.com
Created: 2024-01-15
Current Date: 2025-01-07

Available Backups:
Daily (Last 7 days):
  - 2025-01-07 (today)
  - 2025-01-06
  - 2025-01-05
  - 2025-01-04
  - 2025-01-03
  - 2025-01-02
  - 2025-01-01

Weekly (Last 4 weeks):
  - 2024-12-31 (Week 1)
  - 2024-12-24 (Week 2)
  - 2024-12-17 (Week 3)
  - 2024-12-10 (Week 4)

Monthly (Last 3 months):
  - 2024-12-01 (Month 1)
  - 2024-11-01 (Month 2)
  - 2024-10-01 (Month 3)

Total Available Restore Points: 14 backups
Earliest Restore Point: 2024-10-01 (3 months ago)

D. Contact Information for Backup Issues

Internal: - System Administrator: (See Contacts) - Email Alerts: admin@mdhosting.co.uk

External Vendors: - Hetzner Support: https://www.hetzner.com/support (Storage Box issues) - cPanel Support: https://docs.cpanel.net (Backup system issues)

Emergency Procedures: - See Incident Response for escalation

Last updated: January 2025 Next review: Before ApisCP migration commencement