Initial commit - Church Music Database
This commit is contained in:
390
legacy-site/documentation/md-files/SYSTEMD_PRODUCTION_GUIDE.md
Normal file
390
legacy-site/documentation/md-files/SYSTEMD_PRODUCTION_GUIDE.md
Normal file
@@ -0,0 +1,390 @@
|
||||
# Church Music Database - Systemd Production Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
This project is configured to run automatically on Ubuntu server reboot using **systemd** services. The deployment consists of two services:
|
||||
|
||||
1. **Backend Service** (`church-music-backend.service`)
|
||||
- Flask API served by Gunicorn
|
||||
- Binds to `127.0.0.1:8080`
|
||||
- 2 worker processes
|
||||
- Auto-restart on failure
|
||||
|
||||
2. **Frontend Service** (`church-music-frontend.service`)
|
||||
- React static files served by `serve`
|
||||
- Listens on port `5100`
|
||||
- Single-page application mode
|
||||
- Auto-restart on failure
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Initial Setup
|
||||
|
||||
```bash
|
||||
# Make setup script executable
|
||||
chmod +x /media/pts/Website/Church_HOP_MusicData/systemd-setup.sh
|
||||
|
||||
# Run the setup script (installs and starts services)
|
||||
sudo /media/pts/Website/Church_HOP_MusicData/systemd-setup.sh
|
||||
```
|
||||
|
||||
### Service Management (Easy Way)
|
||||
|
||||
```bash
|
||||
# Make management script executable
|
||||
chmod +x /media/pts/Website/Church_HOP_MusicData/manage-services.sh
|
||||
|
||||
# Check service status
|
||||
./manage-services.sh status
|
||||
|
||||
# Check health (HTTP endpoints)
|
||||
./manage-services.sh health
|
||||
|
||||
# Restart services
|
||||
./manage-services.sh restart
|
||||
|
||||
# View live logs
|
||||
./manage-services.sh logs # Backend logs
|
||||
./manage-services.sh logs-fe # Frontend logs
|
||||
|
||||
# Stop services
|
||||
./manage-services.sh stop
|
||||
|
||||
# Start services
|
||||
./manage-services.sh start
|
||||
```
|
||||
|
||||
## Manual systemd Commands
|
||||
|
||||
### Service Status
|
||||
|
||||
```bash
|
||||
# Check backend status
|
||||
sudo systemctl status church-music-backend
|
||||
|
||||
# Check frontend status
|
||||
sudo systemctl status church-music-frontend
|
||||
|
||||
# Check if enabled for auto-start
|
||||
sudo systemctl is-enabled church-music-backend
|
||||
sudo systemctl is-enabled church-music-frontend
|
||||
```
|
||||
|
||||
### Start/Stop/Restart
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
sudo systemctl start church-music-backend
|
||||
sudo systemctl start church-music-frontend
|
||||
|
||||
# Stop services
|
||||
sudo systemctl stop church-music-backend
|
||||
sudo systemctl stop church-music-frontend
|
||||
|
||||
# Restart services
|
||||
sudo systemctl restart church-music-backend
|
||||
sudo systemctl restart church-music-frontend
|
||||
|
||||
# Reload backend without downtime (graceful reload)
|
||||
sudo systemctl reload church-music-backend
|
||||
```
|
||||
|
||||
### Enable/Disable Auto-start
|
||||
|
||||
```bash
|
||||
# Enable auto-start on boot
|
||||
sudo systemctl enable church-music-backend
|
||||
sudo systemctl enable church-music-frontend
|
||||
|
||||
# Disable auto-start
|
||||
sudo systemctl disable church-music-backend
|
||||
sudo systemctl disable church-music-frontend
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# Backend logs (live tail)
|
||||
sudo journalctl -u church-music-backend -f
|
||||
|
||||
# Frontend logs (live tail)
|
||||
sudo journalctl -u church-music-frontend -f
|
||||
|
||||
# View last 100 lines
|
||||
sudo journalctl -u church-music-backend -n 100
|
||||
|
||||
# View logs since boot
|
||||
sudo journalctl -u church-music-backend -b
|
||||
|
||||
# View logs for specific date
|
||||
sudo journalctl -u church-music-backend --since "2025-12-15"
|
||||
|
||||
# Backend application logs (Gunicorn)
|
||||
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/service.log
|
||||
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/error.log
|
||||
tail -f /media/pts/Website/Church_HOP_MusicData/backend/logs/access.log
|
||||
```
|
||||
|
||||
## Service Details
|
||||
|
||||
### Backend Service Configuration
|
||||
|
||||
**File:** `/etc/systemd/system/church-music-backend.service`
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- Runs as user `pts` (non-root)
|
||||
- Working directory: `/media/pts/Website/Church_HOP_MusicData/backend`
|
||||
- Uses Python virtual environment
|
||||
- Environment variables loaded from `.env` file
|
||||
- Gunicorn with 2 workers
|
||||
- Memory limit: 512MB
|
||||
- CPU quota: 50%
|
||||
- Auto-restart with 10s delay
|
||||
- Protected filesystem (read-only system, private /tmp)
|
||||
|
||||
### Frontend Service Configuration
|
||||
|
||||
**File:** `/etc/systemd/system/church-music-frontend.service`
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- Runs as user `pts` (non-root)
|
||||
- Serves static files from `build/` directory
|
||||
- Single-page application routing (`--single`)
|
||||
- Memory limit: 256MB
|
||||
- CPU quota: 25%
|
||||
- Auto-restart with 10s delay
|
||||
- Protected filesystem
|
||||
|
||||
## Security Features
|
||||
|
||||
Both services implement security best practices:
|
||||
|
||||
- ✅ Run as non-root user (`pts`)
|
||||
- ✅ `NoNewPrivileges=true` - Cannot escalate privileges
|
||||
- ✅ `PrivateTmp=true` - Isolated /tmp directory
|
||||
- ✅ `ProtectSystem=strict` - Read-only system directories
|
||||
- ✅ `ProtectHome=read-only` - Limited home directory access
|
||||
- ✅ Resource limits (CPU & Memory)
|
||||
- ✅ Automatic restart on crash
|
||||
- ✅ Non-blocking boot (Type=notify for backend, Type=simple for frontend)
|
||||
|
||||
## Resource Limits
|
||||
|
||||
Current configuration:
|
||||
|
||||
| Service | Memory Limit | CPU Quota | Workers |
|
||||
|----------|--------------|-----------|---------|
|
||||
| Backend | 512MB | 50% | 2 |
|
||||
| Frontend | 256MB | 25% | 1 |
|
||||
|
||||
To adjust limits, edit the service files in `/etc/systemd/system/` and run:
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart church-music-backend
|
||||
sudo systemctl restart church-music-frontend
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
```bash
|
||||
# Check detailed status
|
||||
sudo systemctl status church-music-backend -l
|
||||
|
||||
# View recent errors
|
||||
sudo journalctl -u church-music-backend -n 50 --no-pager
|
||||
|
||||
# Check if port is already in use
|
||||
sudo netstat -tulpn | grep 8080
|
||||
sudo netstat -tulpn | grep 5100
|
||||
|
||||
# Test backend directly
|
||||
cd /media/pts/Website/Church_HOP_MusicData/backend
|
||||
source venv/bin/activate
|
||||
gunicorn --config gunicorn_config.py app:app
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
```bash
|
||||
# Check PostgreSQL is running
|
||||
sudo systemctl status postgresql
|
||||
|
||||
# Test database connection
|
||||
psql -h 192.168.10.130 -U songlyric_user -d church_songlyric
|
||||
|
||||
# Check backend environment variables
|
||||
sudo systemctl show church-music-backend --property=Environment
|
||||
```
|
||||
|
||||
### Service Crashes Repeatedly
|
||||
|
||||
```bash
|
||||
# View crash logs
|
||||
sudo journalctl -u church-music-backend --since "10 minutes ago"
|
||||
|
||||
# Check system resources
|
||||
free -h
|
||||
df -h
|
||||
top
|
||||
|
||||
# Increase restart delay if needed (edit service file)
|
||||
# RestartSec=30
|
||||
```
|
||||
|
||||
### Frontend Not Loading
|
||||
|
||||
```bash
|
||||
# Check if build exists
|
||||
ls -la /media/pts/Website/Church_HOP_MusicData/frontend/build/
|
||||
|
||||
# Rebuild if needed
|
||||
cd /media/pts/Website/Church_HOP_MusicData/frontend
|
||||
npm run build
|
||||
|
||||
# Restart service
|
||||
sudo systemctl restart church-music-frontend
|
||||
```
|
||||
|
||||
## Testing Auto-Start on Reboot
|
||||
|
||||
```bash
|
||||
# Method 1: Safe reboot
|
||||
sudo reboot
|
||||
|
||||
# After reboot, check services started automatically
|
||||
sudo systemctl status church-music-backend
|
||||
sudo systemctl status church-music-frontend
|
||||
|
||||
# Method 2: Simulate reboot (without rebooting)
|
||||
sudo systemctl isolate default.target
|
||||
```
|
||||
|
||||
## Updating the Application
|
||||
|
||||
```bash
|
||||
# 1. Stop services
|
||||
./manage-services.sh stop
|
||||
|
||||
# 2. Pull latest code
|
||||
cd /media/pts/Website/Church_HOP_MusicData
|
||||
git pull # or update files manually
|
||||
|
||||
# 3. Update backend dependencies (if needed)
|
||||
cd backend
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 4. Rebuild frontend
|
||||
cd ../frontend
|
||||
npm install # if package.json changed
|
||||
npm run build
|
||||
|
||||
# 5. Run database migrations (if needed)
|
||||
cd ../backend
|
||||
source venv/bin/activate
|
||||
python migrate_database.py
|
||||
|
||||
# 6. Start services
|
||||
cd ..
|
||||
./manage-services.sh start
|
||||
|
||||
# 7. Verify
|
||||
./manage-services.sh health
|
||||
```
|
||||
|
||||
## Uninstalling Services
|
||||
|
||||
```bash
|
||||
# Stop and disable services
|
||||
sudo systemctl stop church-music-backend church-music-frontend
|
||||
sudo systemctl disable church-music-backend church-music-frontend
|
||||
|
||||
# Remove service files
|
||||
sudo rm /etc/systemd/system/church-music-backend.service
|
||||
sudo rm /etc/systemd/system/church-music-frontend.service
|
||||
|
||||
# Reload systemd
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl reset-failed
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [x] Backend runs on Gunicorn (production WSGI server)
|
||||
- [x] Frontend served as static files (not dev server)
|
||||
- [x] Services run as non-root user
|
||||
- [x] Auto-restart on failure enabled
|
||||
- [x] Resource limits configured
|
||||
- [x] Security hardening applied
|
||||
- [x] Logs properly configured
|
||||
- [x] Environment variables secured
|
||||
- [x] Database connection pooling (SQLAlchemy)
|
||||
- [ ] SSL/TLS certificate installed (nginx-ssl.conf ready)
|
||||
- [ ] Database migrations applied
|
||||
- [ ] Firewall rules configured
|
||||
- [ ] Regular backups scheduled
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Ubuntu Server (Systemd) │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────┐ │
|
||||
│ │ church-music-frontend.service │ │
|
||||
│ │ (serve @ port 5100) │ │
|
||||
│ │ - Static React files │ │
|
||||
│ │ - Single-page app routing │ │
|
||||
│ └──────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ HTTP API calls │
|
||||
│ ↓ │
|
||||
│ ┌──────────────────────────────────────────┐ │
|
||||
│ │ church-music-backend.service │ │
|
||||
│ │ (Gunicorn @ 127.0.0.1:8080) │ │
|
||||
│ │ - Flask REST API │ │
|
||||
│ │ - 2 worker processes │ │
|
||||
│ └──────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ PostgreSQL protocol │
|
||||
│ ↓ │
|
||||
│ ┌──────────────────────────────────────────┐ │
|
||||
│ │ PostgreSQL Database │ │
|
||||
│ │ 192.168.10.130:5432 │ │
|
||||
│ │ church_songlyric │ │
|
||||
│ └──────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
Nginx (Optional SSL Termination)
|
||||
├─ HTTPS :443 → Frontend :5100
|
||||
└─ /api/* → Backend :8080
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Watch service status in real-time
|
||||
watch -n 2 'systemctl status church-music-backend church-music-frontend'
|
||||
|
||||
# Monitor resource usage
|
||||
sudo systemctl status church-music-backend | grep -E "Memory|CPU"
|
||||
|
||||
# Check service uptime
|
||||
systemctl show church-music-backend --property=ActiveEnterTimestamp
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
|
||||
1. Check logs: `./manage-services.sh logs`
|
||||
2. Verify health: `./manage-services.sh health`
|
||||
3. Review service status: `./manage-services.sh status`
|
||||
4. Check this documentation: `/media/pts/Website/Church_HOP_MusicData/SYSTEMD_PRODUCTION_GUIDE.md`
|
||||
Reference in New Issue
Block a user