# Zoom S3 Pipeline — Staging/Production Deployment Guide

## Prerequisites

- SSH access to staging server (`ubuntu@ip-10-0-4-197`)
- Access to Moodle CLI on server
- AWS S3 credentials (already on server at `/home/ubuntu/.aws/credentials`)
- Python 3 installed on server

---

## 1. Deploy Moodle Code Changes

Files to deploy:

```
local/zooms3/version.php              (version bump to 2026051800)
local/zooms3/payload_zoom_updated.php (new webhook handler — for staging test)
local/zooms3/db/install.xml           (DB schema)
local/zooms3/db/upgrade.php           (DB upgrade script)
local/zooms3/cli/get_download_url.php (token recovery CLI)
```

**NOTE:** For staging testing, deploy `payload_zoom_updated.php` alongside the existing `payload_zoom_temp.php`. This keeps the current production flow untouched. A second Zoom webhook app points to `payload_zoom_updated.php`.

**IMPORTANT:** Make sure `array_change_key_case()` fix is present in the endpoint (works on both Apache and nginx):

```php
$rawHeaders = getallheaders();
$headers = array_change_key_case($rawHeaders, CASE_LOWER);
```

Push code to staging branch and deploy via your normal process (CodeCommit > staging server).

### Staging Test Isolation

| Component | Current Prod (untouched) | Staging Test |
|-----------|-------------------------|-------------|
| Webhook endpoint | `payload_zoom_temp.php` | `payload_zoom_updated.php` |
| Zoom app secret | `noETtCC0QuaF6iIQgY97mg` | `KHWlYouTQ7GRM-lRZ66IeQ` |
| Temp folder | `zoom_webhook/` | `zoom_recordings_webhook/` |
| S3 path | `s3://zoom-recodings/<id>/` | `s3://zoom-recodings/test-recordings/<id>/` |
| Sync script | `sync_and_delete.sh` | `zoom_s3_sync.py` |

Both Zoom apps fire on the same recording events. Both flows run independently without interference.

## 2. Create DB Table

```bash
# SSH to server
ssh ubuntu@ip-10-0-4-197

# Navigate to Moodle root
cd /home/vlearn/vlearn.paradisolms.net/public_html

# Run upgrade
php admin/cli/upgrade.php

# Verify table created
mysql -e "DESCRIBE mdl_local_zooms3_recording_sync;"
# Expected: 21 columns including id, meeting_id, status,
# download_url, download_token, error_log, etc.

# Verify version
mysql -e "SELECT version FROM mdl_config_plugins
          WHERE plugin='local_zooms3' AND name='version';"
# Expected: 2026051800
```

## 3. Set Up Second Zoom Webhook App (for staging test)

1. Go to https://marketplace.zoom.us/develop/apps
2. Click **Develop** > **Build App** > **Webhook Only**
3. Name: "VLearn Staging Test"
4. Under **Feature** > **Event Subscriptions**, add subscription:
   - Endpoint URL: `https://iomod-lms.herovired.com/local/zooms3/payload_zoom_updated.php`
   - Event: **Recording** > **All Recordings have completed**
5. Copy the **Secret Token** and update it in `payload_zoom_updated.php`
6. Click **Validate** — should pass

## 4. Test Webhook Endpoint

```bash
# Quick sanity check - GET request
curl -s https://iomod-lms.herovired.com/local/zooms3/payload_zoom_updated.php
# Expected: "Zoom Webhook sample successfully running..."

# Wait for a real recording.completed event (next Zoom meeting)
# Both webhook apps will fire — prod saves to zoom_webhook/, test saves to zoom_recordings_webhook/

# After webhook fires, check DB:
mysql -e "SELECT id, meeting_id, topic, status, recording_type
          FROM mdl_local_zooms3_recording_sync
          ORDER BY id DESC LIMIT 5;"
# Expected: Row with status = pending
```

## 5. Deploy Python Script

```bash
# From your local machine, copy script files to server
# Do NOT copy .env (has local credentials)
# Do NOT copy venv/ (needs server-specific install)
scp server_scripts/zoom_s3_sync/zoom_s3_sync.py ubuntu@ip-10-0-4-197:~/zoom_s3_sync/
scp server_scripts/zoom_s3_sync/requirements.txt ubuntu@ip-10-0-4-197:~/zoom_s3_sync/
scp server_scripts/zoom_s3_sync/.env.example ubuntu@ip-10-0-4-197:~/zoom_s3_sync/

# SSH to server
ssh ubuntu@ip-10-0-4-197
cd ~/zoom_s3_sync
```

## 6. Install Python Dependencies

```bash
# Check Python version
python3 --version
pip3 --version

# Install dependencies
# On Ubuntu, global pip install usually works (no venv needed)
pip3 install -r requirements.txt

# If you get "externally-managed-environment" error (Ubuntu 23+):
pip3 install --break-system-packages -r requirements.txt
# OR use venv:
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

## 7. Configure .env

```bash
cp .env.example .env
nano .env
```

Fill in values for **staging test**:

```bash
# Database
DB_HOST=localhost
DB_PORT=3306
DB_USER=<moodle db user>
DB_PASSWORD=<moodle db password>
DB_NAME=<moodle db name>
DB_TABLE_PREFIX=mdl_

# AWS S3
S3_BUCKET=zoom-recodings
S3_REGION=ap-south-1
S3_PREFIX=test-recordings/
# Staging test uploads to s3://zoom-recodings/test-recordings/<meeting_id>/
# After successful testing, change to: S3_PREFIX=
AWS_ACCESS_KEY_ID=<from /home/ubuntu/.aws/credentials>
AWS_SECRET_ACCESS_KEY=<from /home/ubuntu/.aws/credentials>

# Paths
MOODLE_DATA_DIR=/home/vlearn/vlearn.paradisolms.net/moodledata
MOODLE_DIR=/home/vlearn/vlearn.paradisolms.net/public_html
PHP_BIN=/usr/bin/php

# Processing
MAX_WORKERS=3
TOKEN_SAFETY_BUFFER_HOURS=20
LOCK_FILE=/tmp/zoom_s3_sync.lock
LOG_FILE=/var/log/zoom_s3_sync.log
WEBHOOK_TEMP_DIR=zoom_recordings_webhook
# Staging test reads from zoom_recordings_webhook/
# After successful testing, change to: WEBHOOK_TEMP_DIR=zoom_webhook
```

To find DB credentials:

```bash
grep -E 'dbhost|dbname|dbuser|dbpass' \
  /home/vlearn/vlearn.paradisolms.net/public_html/config.php
```

To find AWS credentials:

```bash
cat /home/ubuntu/.aws/credentials
```

Create log file with correct permissions:

```bash
sudo touch /var/log/zoom_s3_sync.log
sudo chown ubuntu:ubuntu /var/log/zoom_s3_sync.log
```

## 8. Test Python Script Manually

```bash
# First run — see if it picks up pending records
python3 ~/zoom_s3_sync/zoom_s3_sync.py

# If using venv:
source ~/zoom_s3_sync/venv/bin/activate
python3 ~/zoom_s3_sync/zoom_s3_sync.py

# Expected output:
#   Found N records to process.
#   [Meeting XXXXX] Processing started...
#   [Meeting XXXXX] Starting S3 upload: test-recordings/XXXXX/original.mp4
#   [Meeting XXXXX] S3 upload completed: test-recordings/XXXXX/original.mp4
#   [Meeting XXXXX] Uploaded webhook.json to S3: test-recordings/XXXXX/webhook.json
#   [Meeting XXXXX] Deleted temp folder...
#   [Meeting XXXXX] Completed successfully.
#   [Meeting XXXXX] Result: SUCCESS

# Verify S3
aws s3 ls s3://zoom-recodings/test-recordings/<meeting_id>/ --region ap-south-1
# Expected: original.mp4 + webhook.json

# Verify DB
mysql -e "SELECT id, meeting_id, status, s3_path
          FROM mdl_local_zooms3_recording_sync
          ORDER BY id DESC LIMIT 5;"
# Expected: status = completed, s3_path = test-recordings/<meeting_id>/original.mp4

# Verify temp folder cleaned
ls /home/vlearn/vlearn.paradisolms.net/moodledata/temp/zoom_recordings_webhook/
# Meeting folder should be gone

# Check error_log for full step-by-step trace
mysql -e "SELECT meeting_id, error_log
          FROM mdl_local_zooms3_recording_sync
          ORDER BY id DESC LIMIT 1\G"
```

## 9. Set Up Cron (after manual test succeeds)

```bash
crontab -e

# Keep old script running (don't disable yet — testing in parallel):
# */5 * * * * /home/ubuntu/sync_and_delete.sh

# Add new script alongside it:
# If using global python3:
*/5 * * * * /usr/bin/python3 /home/ubuntu/zoom_s3_sync/zoom_s3_sync.py

# If using venv:
*/5 * * * * /home/ubuntu/zoom_s3_sync/venv/bin/python3 /home/ubuntu/zoom_s3_sync/zoom_s3_sync.py

# Verify cron is set
crontab -l | grep zoom
```

## 10. Verify Cron Is Working

```bash
# Wait 5-10 minutes, then check the log
tail -20 /var/log/zoom_s3_sync.log

# If no recordings pending, you'll see:
#   "No pending records to process."

# Trigger a real recording:
#   Start Zoom meeting > Record > End meeting > Wait 2 min
#   Both webhook apps fire:
#     - Prod: payload_zoom_temp.php downloads to zoom_webhook/ (old flow)
#     - Test: payload_zoom_updated.php saves to zoom_recordings_webhook/ + DB (new flow)
#   Cron picks up within 5 min > streams to S3 test-recordings/ > completed

# Check DB for the flow
mysql -e "SELECT id, meeting_id, status, s3_path, completed_at
          FROM mdl_local_zooms3_recording_sync
          ORDER BY id DESC LIMIT 5;"
```

## 11. Verify Recovery Script (optional)

```bash
# Test the PHP CLI recovery script with a known meeting ID
cd /home/vlearn/vlearn.paradisolms.net/public_html
php local/zooms3/cli/get_download_url.php --meetingid=<real_meeting_id>

# Expected: JSON output
# {
#   "success": true,
#   "meeting_id": "XXXXX",
#   "best_recording": {
#     "download_url": "https://herovired.zoom.us/rec/download/...",
#     "recording_type": "shared_screen_with_speaker_view",
#     ...
#   },
#   "access_token": "eyJ...",
#   "all_mp4s": [...]
# }
```

## 12. Switch to Production (after staging test is validated)

Once staging test is confirmed working:

```bash
# 1. Update .env — remove test prefix and switch temp dir
nano ~/zoom_s3_sync/.env
# Change:
#   S3_PREFIX=
#   WEBHOOK_TEMP_DIR=zoom_webhook

# 2. Move new endpoint code into payload_zoom_temp.php
#    (replace old synchronous download code with new async code)
#    Update webhookSecretToken back to production app's token

# 3. Disable old sync script
crontab -e
# Comment out: # */5 * * * * /home/ubuntu/sync_and_delete.sh

# 4. Delete test Zoom webhook app from marketplace

# 5. Verify next recording goes through the new flow
mysql -e "SELECT id, meeting_id, status, s3_path
          FROM mdl_local_zooms3_recording_sync
          ORDER BY id DESC LIMIT 5;"
```

## 13. Monitoring (ongoing)

```bash
# Dashboard: recording counts by status
mysql -e "SELECT status, COUNT(*) as count
          FROM mdl_local_zooms3_recording_sync
          GROUP BY status;"

# Failed recordings needing attention
mysql -e "SELECT id, meeting_id, topic, status, retry_count, error_log
          FROM mdl_local_zooms3_recording_sync
          WHERE status IN ('failed','token_expired','unrecoverable')
          ORDER BY id DESC;"

# Recent activity
mysql -e "SELECT id, meeting_id, topic, status,
                 FROM_UNIXTIME(timecreated) as created,
                 FROM_UNIXTIME(completed_at) as completed
          FROM mdl_local_zooms3_recording_sync
          ORDER BY id DESC LIMIT 10;"

# Live log monitoring
tail -f /var/log/zoom_s3_sync.log

# Cron running check
crontab -l | grep zoom

# S3 bucket size/contents
aws s3 ls s3://zoom-recodings/ --recursive --summarize

# Disk space check (temp dir shouldn't be growing)
du -sh /home/vlearn/vlearn.paradisolms.net/moodledata/temp/zoom_recordings_webhook/
```

## Troubleshooting

| Problem | Check |
|---------|-------|
| Webhook not firing (no DB rows) | Check Zoom app logs at https://marketplace.zoom.us/develop/apps. Verify secret token matches. Check Apache error log for PHP errors. |
| DB row created but stays pending | Cron not running: `crontab -l \| grep zoom`. Python script error: run manually. Check .env credentials. |
| Status = failed | Check `error_log` field in DB. Common: network timeout, S3 permissions, invalid download URL. Script auto-retries on next cron run. |
| Status = token_expired | Script auto-recovers using CLI recovery script. Verify Zoom OAuth credentials in Moodle admin: Site admin > Plugins > Activity modules > Zoom > accountid/clientid/clientsecret. |
| Status = unrecoverable | Recording deleted from Zoom (past retention period). Nothing can be done. Check Zoom account retention settings to prevent future loss. |
| Script lock conflict ("Another instance is already running") | Check: `ls -la /tmp/zoom_s3_sync.lock`. If stale: `rm /tmp/zoom_s3_sync.lock`. |
| PHP_BIN not found | On macOS: `/opt/homebrew/bin/php`. On Linux: `/usr/bin/php`. Check with `which php`. |