PTS/Church-Music
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
Church-Music/legacy-site/documentation/md-files/SYSTEMD_PRODUCTION_GUIDE.md

11 KiB
Raw Permalink Blame History

Church Music Database - Systemd Production Deployment

Overview

This project is configured to run automatically on Ubuntu server reboot using systemd services. The deployment consists of two services:

  1. Backend Service (church-music-backend.service)

    • Flask API served by Gunicorn
    • Binds to 127.0.0.1:8080
    • 2 worker processes
    • Auto-restart on failure

  2. Frontend Service (church-music-frontend.service)

    • React static files served by serve
    • Listens on port 5100
    • Single-page application mode
    • Auto-restart on failure

Quick Start

Initial Setup

# Make setup script executable
chmod +x /media/pts/Website/Church_HOP_MusicData/systemd-setup.sh

# Run the setup script (installs and starts services)
sudo /media/pts/Website/Church_HOP_MusicData/systemd-setup.sh

Service Management (Easy Way)

# Make management script executable
chmod +x /media/pts/Website/Church_HOP_MusicData/manage-services.sh

# Check service status
./manage-services.sh status

# Check health (HTTP endpoints)
./manage-services.sh health

# Restart services
./manage-services.sh restart

# View live logs
./manage-services.sh logs         # Backend logs
./manage-services.sh logs-fe      # Frontend logs

# Stop services
./manage-services.sh stop

# Start services
./manage-services.sh start

Manual systemd Commands

Service Status

# Check backend status
sudo systemctl status church-music-backend

# Check frontend status
sudo systemctl status church-music-frontend

# Check if enabled for auto-start
sudo systemctl is-enabled church-music-backend
sudo systemctl is-enabled church-music-frontend

Start/Stop/Restart

# Start services
sudo systemctl start church-music-backend
sudo systemctl start church-music-frontend

# Stop services
sudo systemctl stop church-music-backend
sudo systemctl stop church-music-frontend

# Restart services
sudo systemctl restart church-music-backend
sudo systemctl restart church-music-frontend

# Reload backend without downtime (graceful reload)
sudo systemctl reload church-music-backend

Enable/Disable Auto-start

# Enable auto-start on boot
sudo systemctl enable church-music-backend
sudo systemctl enable church-music-frontend

# Disable auto-start
sudo systemctl disable church-music-backend
sudo systemctl disable church-music-frontend

View Logs

# Backend logs (live tail)
sudo journalctl -u church-music-backend -f

# Frontend logs (live tail)
sudo journalctl -u church-music-frontend -f

# View last 100 lines
sudo journalctl -u church-music-backend -n 100

# View logs since boot
sudo journalctl -u church-music-backend -b

# View logs for specific date
sudo journalctl -u church-music-backend --since "2025-12-15"

# Backend application logs (Gunicorn)
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/service.log
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/error.log
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/access.log

Service Details

Backend Service Configuration

File: /etc/systemd/system/church-music-backend.service

Key Features:

  • Runs as user pts (non-root)
  • Working directory: /media/pts/Website/Church_HOP_MusicData/backend
  • Uses Python virtual environment
  • Environment variables loaded from .env file
  • Gunicorn with 2 workers
  • Memory limit: 512MB
  • CPU quota: 50%
  • Auto-restart with 10s delay
  • Protected filesystem (read-only system, private /tmp)

Frontend Service Configuration

File: /etc/systemd/system/church-music-frontend.service

Key Features:

  • Runs as user pts (non-root)
  • Serves static files from build/ directory
  • Single-page application routing (--single)
  • Memory limit: 256MB
  • CPU quota: 25%
  • Auto-restart with 10s delay
  • Protected filesystem

Security Features

Both services implement security best practices:

  • ✅ Run as non-root user (pts)
  • ✅ NoNewPrivileges=true - Cannot escalate privileges
  • ✅ PrivateTmp=true - Isolated /tmp directory
  • ✅ ProtectSystem=strict - Read-only system directories
  • ✅ ProtectHome=read-only - Limited home directory access
  • ✅ Resource limits (CPU & Memory)
  • ✅ Automatic restart on crash
  • ✅ Non-blocking boot (Type=notify for backend, Type=simple for frontend)

Resource Limits

Current configuration:

Service Memory Limit CPU Quota Workers
Backend 512MB 50% 2
Frontend 256MB 25% 1

To adjust limits, edit the service files in /etc/systemd/system/ and run:

sudo systemctl daemon-reload
sudo systemctl restart church-music-backend
sudo systemctl restart church-music-frontend

Troubleshooting

Service Won't Start

# Check detailed status
sudo systemctl status church-music-backend -l

# View recent errors
sudo journalctl -u church-music-backend -n 50 --no-pager

# Check if port is already in use
sudo netstat -tulpn | grep 8080
sudo netstat -tulpn | grep 5100

# Test backend directly
cd /media/pts/Website/Church_HOP_MusicData/backend
source venv/bin/activate
gunicorn --config gunicorn_config.py app:app

Database Connection Issues

# Check PostgreSQL is running
sudo systemctl status postgresql

# Test database connection
psql -h 192.168.10.130 -U songlyric_user -d church_songlyric

# Check backend environment variables
sudo systemctl show church-music-backend --property=Environment

Service Crashes Repeatedly

# View crash logs
sudo journalctl -u church-music-backend --since "10 minutes ago"

# Check system resources
free -h
df -h
top

# Increase restart delay if needed (edit service file)
# RestartSec=30

Frontend Not Loading

# Check if build exists
ls -la /media/pts/Website/Church_HOP_MusicData/frontend/build/

# Rebuild if needed
cd /media/pts/Website/Church_HOP_MusicData/frontend
npm run build

# Restart service
sudo systemctl restart church-music-frontend

Testing Auto-Start on Reboot

# Method 1: Safe reboot
sudo reboot

# After reboot, check services started automatically
sudo systemctl status church-music-backend
sudo systemctl status church-music-frontend

# Method 2: Simulate reboot (without rebooting)
sudo systemctl isolate default.target

Updating the Application

# 1. Stop services
./manage-services.sh stop

# 2. Pull latest code
cd /media/pts/Website/Church_HOP_MusicData
git pull  # or update files manually

# 3. Update backend dependencies (if needed)
cd backend
source venv/bin/activate
pip install -r requirements.txt

# 4. Rebuild frontend
cd ../frontend
npm install  # if package.json changed
npm run build

# 5. Run database migrations (if needed)
cd ../backend
source venv/bin/activate
python migrate_database.py

# 6. Start services
cd ..
./manage-services.sh start

# 7. Verify
./manage-services.sh health

Uninstalling Services

# Stop and disable services
sudo systemctl stop church-music-backend church-music-frontend
sudo systemctl disable church-music-backend church-music-frontend

# Remove service files
sudo rm /etc/systemd/system/church-music-backend.service
sudo rm /etc/systemd/system/church-music-frontend.service

# Reload systemd
sudo systemctl daemon-reload
sudo systemctl reset-failed

Production Checklist

  • Backend runs on Gunicorn (production WSGI server)
  • Frontend served as static files (not dev server)
  • Services run as non-root user
  • Auto-restart on failure enabled
  • Resource limits configured
  • Security hardening applied
  • Logs properly configured
  • Environment variables secured
  • Database connection pooling (SQLAlchemy)
  • SSL/TLS certificate installed (nginx-ssl.conf ready)
  • Database migrations applied
  • Firewall rules configured
  • Regular backups scheduled

Architecture

┌─────────────────────────────────────────────────┐
│           Ubuntu Server (Systemd)               │
├─────────────────────────────────────────────────┤
│                                                 │
│  ┌──────────────────────────────────────────┐  │
│  │   church-music-frontend.service          │  │
│  │   (serve @ port 5100)                    │  │
│  │   - Static React files                   │  │
│  │   - Single-page app routing              │  │
│  └──────────────────────────────────────────┘  │
│                      │                          │
│                      │ HTTP API calls           │
│                      ↓                          │
│  ┌──────────────────────────────────────────┐  │
│  │   church-music-backend.service           │  │
│  │   (Gunicorn @ 127.0.0.1:8080)           │  │
│  │   - Flask REST API                       │  │
│  │   - 2 worker processes                   │  │
│  └──────────────────────────────────────────┘  │
│                      │                          │
│                      │ PostgreSQL protocol      │
│                      ↓                          │
│  ┌──────────────────────────────────────────┐  │
│  │   PostgreSQL Database                    │  │
│  │   192.168.10.130:5432                   │  │
│  │   church_songlyric                      │  │
│  └──────────────────────────────────────────┘  │
│                                                 │
└─────────────────────────────────────────────────┘

  Nginx (Optional SSL Termination)
  ├─ HTTPS :443 → Frontend :5100
  └─ /api/* → Backend :8080

Monitoring

# Watch service status in real-time
watch -n 2 'systemctl status church-music-backend church-music-frontend'

# Monitor resource usage
sudo systemctl status church-music-backend | grep -E "Memory|CPU"

# Check service uptime
systemctl show church-music-backend --property=ActiveEnterTimestamp

Support

For issues or questions:

  1. Check logs: ./manage-services.sh logs
  2. Verify health: ./manage-services.sh health
  3. Review service status: ./manage-services.sh status
  4. Check this documentation: /media/pts/Website/Church_HOP_MusicData/SYSTEMD_PRODUCTION_GUIDE.md
Reference in New Issue View Git Blame Copy Permalink