Cloudflare R2 Storage Setup

Status: Active Last Updated: 2025-01-06 Related Docs: System Overview, EPG Generation Project: Architecture & Workflow Redesign (Phase 1)

Overview

EPGOAT uses Cloudflare R2 for EPG file storage. R2 is S3-compatible object storage with zero egress fees (no cost for downloads).

Why R2? - ✅ Zero egress fees (unlimited downloads at no cost) - ✅ S3-compatible API (works with boto3, AWS SDKs) - ✅ Global CDN built-in (fast worldwide delivery) - ✅ Free tier: 10GB storage, 1M Class A ops, 10M Class B ops/month - ✅ Perfect for serving EPG XML files to users

Cost Savings: At 1,000 users with 1M EPG downloads/month: - R2: $0.15/month (storage only, egress FREE) - AWS S3 + CloudFront: $90/month (egress charges) - Savings: $1,080/year

Bucket Details

Bucket Configuration: - Name: epgoat-epg-files - Region: Automatic (Cloudflare globally distributes) - Purpose: Store generated EPG XML files - Access: Private (authenticated access via API tokens)

Dashboard Access: - https://dash.cloudflare.com/[ACCOUNT_ID]/r2/overview - Direct bucket: https://dash.cloudflare.com/[ACCOUNT_ID]/r2/epgoat-epg-files

Environment Configuration

Required Environment Variables

All credentials stored in .env.supabase (NOT committed to git):

# Cloudflare R2 Configuration
R2_ACCOUNT_ID=e7cdd1acc0daca237f0b1e9aa80fc2da
R2_ACCESS_KEY_ID=fe3b39f0b6d9bf32c6b52407f22c365b
R2_SECRET_ACCESS_KEY=e307c1cc6ac107a14e374c0f38c376130b993fca3e179571e37c1f8b70de573b
R2_BUCKET_NAME=epgoat-epg-files
R2_ENDPOINT_URL=https://e7cdd1acc0daca237f0b1e9aa80fc2da.r2.cloudflarestorage.com
R2_PUBLIC_URL=https://epgoat-epg-files.e7cdd1acc0daca237f0b1e9aa80fc2da.r2.cloudflarestorage.com

API Token Permissions

Current token: epgoat-epg-generator - Permissions: Object Read & Write - Scope: epgoat-epg-files bucket only - Created: 2025-01-06 - Expires: Never (or as configured)

Security: Token has minimal permissions (read/write only to specific bucket)

Python Usage

Basic Upload/Download

import boto3
import os
from dotenv import load_dotenv

load_dotenv('.env.supabase')

# Create R2 client (S3-compatible)
s3 = boto3.client(
    's3',
    endpoint_url=os.getenv('R2_ENDPOINT_URL'),
    aws_access_key_id=os.getenv('R2_ACCESS_KEY_ID'),
    aws_secret_access_key=os.getenv('R2_SECRET_ACCESS_KEY'),
    region_name='auto'
)

bucket_name = os.getenv('R2_BUCKET_NAME')

# Upload file
with open('epg.xml', 'rb') as f:
    s3.put_object(
        Bucket=bucket_name,
        Key='tps/tier-top/america-new_york.xml',
        Body=f,
        ContentType='application/xml'
    )

# Download file
obj = s3.get_object(Bucket=bucket_name, Key='tps/tier-top/america-new_york.xml')
content = obj['Body'].read()

# List files
response = s3.list_objects_v2(Bucket=bucket_name, Prefix='tps/')
for obj in response.get('Contents', []):
    print(f"{obj['Key']} - {obj['Size']} bytes")

EPG File Management

def upload_epg_file(provider_slug, tier, timezone, xml_content):
    """Upload EPG file to R2 with standard naming"""
    key = f"{provider_slug}/tier-{tier}/tz-{timezone.replace('/', '-')}.xml"

    s3.put_object(
        Bucket=bucket_name,
        Key=key,
        Body=xml_content.encode('utf-8'),
        ContentType='application/xml',
        CacheControl='public, max-age=3600',  # Cache for 1 hour
        Metadata={
            'provider': provider_slug,
            'tier': tier,
            'timezone': timezone,
            'generated_at': datetime.now(UTC).isoformat()
        }
    )

    return key

def delete_old_epg_files(provider_slug, tier):
    """Clean up old EPG files for a provider/tier"""
    prefix = f"{provider_slug}/tier-{tier}/"
    response = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)

    for obj in response.get('Contents', []):
        s3.delete_object(Bucket=bucket_name, Key=obj['Key'])

File Organization

Directory Structure

epgoat-epg-files/
├── tps/                          # Provider: TPS
│   ├── tier-basic/
│   │   ├── tz-america-new_york.xml
│   │   ├── tz-america-chicago.xml
│   │   └── tz-europe-london.xml
│   ├── tier-mid/
│   │   └── ...
│   └── tier-top/
│       └── ...
├── necro/                        # Provider: Necro
│   └── ...
├── test/                         # Test files (temporary)
│   └── connection-test.txt
└── archive/                      # Old files (manual cleanup)
    └── ...

File Naming Convention

Format: {provider_slug}/tier-{tier}/tz-{timezone}.xml

Examples: - tps/tier-basic/tz-america-new_york.xml - tps/tier-top/tz-europe-london.xml - necro/tier-mid/tz-asia-tokyo.xml

Timezone Encoding: - Replace / with - in timezone names - Examples: - America/New_York → tz-america-new_york.xml - Europe/London → tz-europe-london.xml - Asia/Tokyo → tz-asia-tokyo.xml

Free Tier Limits

Current Plan: Free - Storage: 10GB/month (first 10GB free) - Class A Operations (uploads, PUT, LIST): 1 million/month free - Class B Operations (downloads, GET): 10 million/month free - Egress: $0 (unlimited, always free)

Current Usage (as of 2025-01-06): - Storage: ~0 MB (bucket just created) - Operations: ~0 (testing phase)

Projected Usage (at 1,000 users): - Storage: ~500MB (450 pre-generated files × ~1MB avg + custom configs) - Class A ops: ~50k/month (uploads during regeneration) - Class B ops: ~100k/month (EPG downloads) - Cost: $0.007/month (well within free tier)

Cost Analysis

Pricing Breakdown

Storage: - First 10GB: FREE - After 10GB: $0.015/GB/month

Class A Operations (uploads, PUT, LIST): - First 1M/month: FREE - After 1M: $4.50/million

Class B Operations (downloads, GET): - First 10M/month: FREE - After 10M: $0.36/million

Egress (data transfer out): - $0 (always free, unlimited)

Scaling Projections

Note: EPGOAT will stay within free tier until ~100k users!

Cloudflare Workers Integration

Serving EPG Files

R2 integrates perfectly with Cloudflare Workers for URL routing:

// workers/epg-router.js
export default {
  async fetch(request, env) {
    const url = new URL(request.url);

    // Extract access key from URL: https://epgo.at/G7dK4r
    const accessKey = url.pathname.slice(1);

    // Look up key in Supabase (get provider, tier, timezone)
    const keyInfo = await lookupAccessKey(accessKey);

    if (!keyInfo) {
      return new Response('Invalid key', { status: 403 });
    }

    // Construct R2 object key
    const r2Key = `${keyInfo.provider}/tier-${keyInfo.tier}/tz-${keyInfo.timezone}.xml`;

    // Fetch from R2
    const object = await env.R2_BUCKET.get(r2Key);

    if (!object) {
      return new Response('EPG not found', { status: 404 });
    }

    // Return EPG file
    return new Response(object.body, {
      headers: {
        'Content-Type': 'application/xml',
        'Cache-Control': 'public, max-age=3600'
      }
    });
  }
};

Monitoring & Analytics

Check Bucket Usage

Via Dashboard: 1. Go to Cloudflare Dashboard → R2 2. Click on epgoat-epg-files bucket 3. View Metrics tab 4. See storage usage, operation counts, bandwidth

Via Python:

# List all objects and calculate total size
response = s3.list_objects_v2(Bucket=bucket_name)
total_size = sum(obj['Size'] for obj in response.get('Contents', []))
print(f"Total storage: {total_size / 1024 / 1024:.2f} MB")

Set Up Alerts

In Cloudflare Dashboard: 1. Go to Notifications 2. Create alert for: - Storage approaching 9GB (90% of free tier) - Class A ops approaching 900k/month - Class B ops approaching 9M/month

Backup & Disaster Recovery

Backup Strategy

R2 provides: - Automatic replication across Cloudflare's global network - 99.999999999% (11 9's) durability - No single point of failure

Additional backups (recommended):

# Daily backup to S3 or local storage
def backup_r2_to_local():
    """Download all EPG files for backup"""
    response = s3.list_objects_v2(Bucket=bucket_name)

    for obj in response.get('Contents', []):
        key = obj['Key']
        local_path = f"backups/r2/{key}"

        os.makedirs(os.path.dirname(local_path), exist_ok=True)

        s3.download_file(bucket_name, key, local_path)

    print("Backup complete!")

Disaster Recovery

If R2 becomes unavailable: 1. EPG files can be regenerated from database (epg_data table) 2. Regeneration takes ~5-20 seconds per file 3. Cloudflare has 99.9%+ uptime SLA 4. Use backup from local storage if needed

Troubleshooting

Access Denied Errors

Problem: AccessDenied when uploading/downloading - Cause: Incorrect API credentials or insufficient permissions - Solution: - Verify R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY in .env.supabase - Check token permissions in Cloudflare dashboard - Ensure token has "Object Read & Write" for the bucket

Bucket Not Found

Problem: NoSuchBucket error - Cause: Bucket doesn't exist or incorrect bucket name - Solution: - Verify bucket exists in R2 dashboard - Check R2_BUCKET_NAME matches exactly (case-sensitive) - Ensure account ID in R2_ENDPOINT_URL is correct

Connection Timeout

Problem: Timeout connecting to R2 - Cause: Network issues or firewall blocking - Solution: - Check internet connection - Try different network - Verify R2_ENDPOINT_URL format is correct

Security Best Practices

API Credentials: - ✅ Never commit .env.supabase to git - ✅ Use minimum required permissions (read/write only) - ✅ Rotate API keys every 90 days - ❌ Never expose keys in client-side code - ❌ Never share keys publicly

Bucket Security: - ✅ Keep bucket private (not publicly accessible) - ✅ Use signed URLs for temporary public access if needed - ✅ Enable access logging for audit trail - ❌ Never make bucket publicly writable

Data Security: - EPG files are not sensitive (no PII, no credentials) - Files are served publicly via Workers (authenticated via access keys) - Consider encryption at rest for highly sensitive data (not needed for EPG)

Migration from Other Storage

From AWS S3

# Copy files from S3 to R2
import boto3

# S3 client
s3_source = boto3.client('s3')

# R2 client
s3_dest = boto3.client(
    's3',
    endpoint_url=os.getenv('R2_ENDPOINT_URL'),
    aws_access_key_id=os.getenv('R2_ACCESS_KEY_ID'),
    aws_secret_access_key=os.getenv('R2_SECRET_ACCESS_KEY'),
    region_name='auto'
)

# Copy all files
response = s3_source.list_objects_v2(Bucket='old-epg-bucket')
for obj in response.get('Contents', []):
    copy_source = {'Bucket': 'old-epg-bucket', 'Key': obj['Key']}
    s3_dest.copy(copy_source, os.getenv('R2_BUCKET_NAME'), obj['Key'])

Related Documentation

• EPG Generation V2 - How EPG files are generated
• Architecture Redesign - Why R2?
• System Overview - Storage architecture
• Core Principles - Cost optimization

Support

Cloudflare Resources: - Dashboard: https://dash.cloudflare.com - R2 Docs: https://developers.cloudflare.com/r2/ - Support: https://dash.cloudflare.com/?to=/:account/support - Status: https://www.cloudflarestatus.com/

EPGOAT Internal: - Questions? Check #infrastructure channel - Issues? Create ticket with "r2" or "storage" label - Credentials lost? Check password manager