Church-Music/legacy-site/documentation/md-files/SYSTEMD_PRODUCTION_GUIDE.md

# Church Music Database - Systemd Production Deployment

## Overview

This project is configured to run automatically on Ubuntu server reboot using **systemd** services. The deployment consists of two services:

1. **Backend Service** (`church-music-backend.service`)
   - Flask API served by Gunicorn
   - Binds to `127.0.0.1:8080`
   - 2 worker processes
   - Auto-restart on failure

2. **Frontend Service** (`church-music-frontend.service`)
   - React static files served by `serve`
   - Listens on port `5100`
   - Single-page application mode
   - Auto-restart on failure

## Quick Start

### Initial Setup

```bash
# Make setup script executable
chmod +x /media/pts/Website/Church_HOP_MusicData/systemd-setup.sh

# Run the setup script (installs and starts services)
sudo /media/pts/Website/Church_HOP_MusicData/systemd-setup.sh
```

### Service Management (Easy Way)

```bash
# Make management script executable
chmod +x /media/pts/Website/Church_HOP_MusicData/manage-services.sh

# Check service status
./manage-services.sh status

# Check health (HTTP endpoints)
./manage-services.sh health

# Restart services
./manage-services.sh restart

# View live logs
./manage-services.sh logs         # Backend logs
./manage-services.sh logs-fe      # Frontend logs

# Stop services
./manage-services.sh stop

# Start services
./manage-services.sh start
```

## Manual systemd Commands

### Service Status

```bash
# Check backend status
sudo systemctl status church-music-backend

# Check frontend status
sudo systemctl status church-music-frontend

# Check if enabled for auto-start
sudo systemctl is-enabled church-music-backend
sudo systemctl is-enabled church-music-frontend
```

### Start/Stop/Restart

```bash
# Start services
sudo systemctl start church-music-backend
sudo systemctl start church-music-frontend

# Stop services
sudo systemctl stop church-music-backend
sudo systemctl stop church-music-frontend

# Restart services
sudo systemctl restart church-music-backend
sudo systemctl restart church-music-frontend

# Reload backend without downtime (graceful reload)
sudo systemctl reload church-music-backend
```

### Enable/Disable Auto-start

```bash
# Enable auto-start on boot
sudo systemctl enable church-music-backend
sudo systemctl enable church-music-frontend

# Disable auto-start
sudo systemctl disable church-music-backend
sudo systemctl disable church-music-frontend
```

### View Logs

```bash
# Backend logs (live tail)
sudo journalctl -u church-music-backend -f

# Frontend logs (live tail)
sudo journalctl -u church-music-frontend -f

# View last 100 lines
sudo journalctl -u church-music-backend -n 100

# View logs since boot
sudo journalctl -u church-music-backend -b

# View logs for specific date
sudo journalctl -u church-music-backend --since "2025-12-15"

# Backend application logs (Gunicorn)
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/service.log
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/error.log
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/access.log
```

## Service Details

### Backend Service Configuration

**File:** `/etc/systemd/system/church-music-backend.service`

**Key Features:**

- Runs as user `pts` (non-root)
- Working directory: `/media/pts/Website/Church_HOP_MusicData/backend`
- Uses Python virtual environment
- Environment variables loaded from `.env` file
- Gunicorn with 2 workers
- Memory limit: 512MB
- CPU quota: 50%
- Auto-restart with 10s delay
- Protected filesystem (read-only system, private /tmp)

### Frontend Service Configuration

**File:** `/etc/systemd/system/church-music-frontend.service`

**Key Features:**

- Runs as user `pts` (non-root)
- Serves static files from `build/` directory
- Single-page application routing (`--single`)
- Memory limit: 256MB
- CPU quota: 25%
- Auto-restart with 10s delay
- Protected filesystem

## Security Features

Both services implement security best practices:

- ✅ Run as non-root user (`pts`)
- ✅ `NoNewPrivileges=true` - Cannot escalate privileges
- ✅ `PrivateTmp=true` - Isolated /tmp directory
- ✅ `ProtectSystem=strict` - Read-only system directories
- ✅ `ProtectHome=read-only` - Limited home directory access
- ✅ Resource limits (CPU & Memory)
- ✅ Automatic restart on crash
- ✅ Non-blocking boot (Type=notify for backend, Type=simple for frontend)

## Resource Limits

Current configuration:

| Service  | Memory Limit | CPU Quota | Workers |
|----------|--------------|-----------|---------|
| Backend  | 512MB        | 50%       | 2       |
| Frontend | 256MB        | 25%       | 1       |

To adjust limits, edit the service files in `/etc/systemd/system/` and run:

```bash
sudo systemctl daemon-reload
sudo systemctl restart church-music-backend
sudo systemctl restart church-music-frontend
```

## Troubleshooting

### Service Won't Start

```bash
# Check detailed status
sudo systemctl status church-music-backend -l

# View recent errors
sudo journalctl -u church-music-backend -n 50 --no-pager

# Check if port is already in use
sudo netstat -tulpn | grep 8080
sudo netstat -tulpn | grep 5100

# Test backend directly
cd /media/pts/Website/Church_HOP_MusicData/backend
source venv/bin/activate
gunicorn --config gunicorn_config.py app:app
```

### Database Connection Issues

```bash
# Check PostgreSQL is running
sudo systemctl status postgresql

# Test database connection
psql -h 192.168.10.130 -U songlyric_user -d church_songlyric

# Check backend environment variables
sudo systemctl show church-music-backend --property=Environment
```

### Service Crashes Repeatedly

```bash
# View crash logs
sudo journalctl -u church-music-backend --since "10 minutes ago"

# Check system resources
free -h
df -h
top

# Increase restart delay if needed (edit service file)
# RestartSec=30
```

### Frontend Not Loading

```bash
# Check if build exists
ls -la /media/pts/Website/Church_HOP_MusicData/frontend/build/

# Rebuild if needed
cd /media/pts/Website/Church_HOP_MusicData/frontend
npm run build

# Restart service
sudo systemctl restart church-music-frontend
```

## Testing Auto-Start on Reboot

```bash
# Method 1: Safe reboot
sudo reboot

# After reboot, check services started automatically
sudo systemctl status church-music-backend
sudo systemctl status church-music-frontend

# Method 2: Simulate reboot (without rebooting)
sudo systemctl isolate default.target
```

## Updating the Application

```bash
# 1. Stop services
./manage-services.sh stop

# 2. Pull latest code
cd /media/pts/Website/Church_HOP_MusicData
git pull  # or update files manually

# 3. Update backend dependencies (if needed)
cd backend
source venv/bin/activate
pip install -r requirements.txt

# 4. Rebuild frontend
cd ../frontend
npm install  # if package.json changed
npm run build

# 5. Run database migrations (if needed)
cd ../backend
source venv/bin/activate
python migrate_database.py

# 6. Start services
cd ..
./manage-services.sh start

# 7. Verify
./manage-services.sh health
```

## Uninstalling Services

```bash
# Stop and disable services
sudo systemctl stop church-music-backend church-music-frontend
sudo systemctl disable church-music-backend church-music-frontend

# Remove service files
sudo rm /etc/systemd/system/church-music-backend.service
sudo rm /etc/systemd/system/church-music-frontend.service

# Reload systemd
sudo systemctl daemon-reload
sudo systemctl reset-failed
```

## Production Checklist

- [x] Backend runs on Gunicorn (production WSGI server)
- [x] Frontend served as static files (not dev server)
- [x] Services run as non-root user
- [x] Auto-restart on failure enabled
- [x] Resource limits configured
- [x] Security hardening applied
- [x] Logs properly configured
- [x] Environment variables secured
- [x] Database connection pooling (SQLAlchemy)
- [ ] SSL/TLS certificate installed (nginx-ssl.conf ready)
- [ ] Database migrations applied
- [ ] Firewall rules configured
- [ ] Regular backups scheduled

## Architecture

```
┌─────────────────────────────────────────────────┐
│           Ubuntu Server (Systemd)               │
├─────────────────────────────────────────────────┤
│                                                 │
│  ┌──────────────────────────────────────────┐  │
│  │   church-music-frontend.service          │  │
│  │   (serve @ port 5100)                    │  │
│  │   - Static React files                   │  │
│  │   - Single-page app routing              │  │
│  └──────────────────────────────────────────┘  │
│                      │                          │
│                      │ HTTP API calls           │
│                      ↓                          │
│  ┌──────────────────────────────────────────┐  │
│  │   church-music-backend.service           │  │
│  │   (Gunicorn @ 127.0.0.1:8080)           │  │
│  │   - Flask REST API                       │  │
│  │   - 2 worker processes                   │  │
│  └──────────────────────────────────────────┘  │
│                      │                          │
│                      │ PostgreSQL protocol      │
│                      ↓                          │
│  ┌──────────────────────────────────────────┐  │
│  │   PostgreSQL Database                    │  │
│  │   192.168.10.130:5432                   │  │
│  │   church_songlyric                      │  │
│  └──────────────────────────────────────────┘  │
│                                                 │
└─────────────────────────────────────────────────┘

  Nginx (Optional SSL Termination)
  ├─ HTTPS :443 → Frontend :5100
  └─ /api/* → Backend :8080
```

## Monitoring

```bash
# Watch service status in real-time
watch -n 2 'systemctl status church-music-backend church-music-frontend'

# Monitor resource usage
sudo systemctl status church-music-backend | grep -E "Memory|CPU"

# Check service uptime
systemctl show church-music-backend --property=ActiveEnterTimestamp
```

## Support

For issues or questions:

1. Check logs: `./manage-services.sh logs`
2. Verify health: `./manage-services.sh health`
3. Review service status: `./manage-services.sh status`
4. Check this documentation: `/media/pts/Website/Church_HOP_MusicData/SYSTEMD_PRODUCTION_GUIDE.md`