Project: Provider Onboarding Service

📍 Current Context

Last Updated: 2025-11-09 Current Task: Design document for FUTURE enhancements (LLM verification now implemented) Breadcrumbs: Basic implementation exists and is production-ready → LLM verification added Status: Provider onboarding service in production with LLM pattern verification (added 2025-11-09). This design covers remaining future enhancements. Implementation: See backend/epgoat/services/provider_onboarding_service.py for current working implementation Recent Addition: LLM verification using Claude Haiku 3.5 (80% confidence threshold, ~$0.0008 per pattern) Next Step: This design document outlines remaining improvements (discovery package, advanced pattern analysis, etc.)

✅ Progress Tracker

• [ ] Phase 1: Core Components (Week 1)
• [ ] Task 1.1: Create discovery/ package
  • [ ] Create backend/epgoat/discovery/__init__.py
  • [ ] Set up package structure
• [ ] Task 1.2: Implement M3UAnalyzer
  • [ ] Create parsers/m3u_analyzer.py
  • [ ] Implement ChannelInfo and M3UAnalysis dataclasses
  • [ ] Implement analyze() method (reuse existing M3U parser)
  • [ ] Implement _extract_url_patterns()
  • [ ] Implement _calculate_tvg_coverage()
  • [ ] Write unit tests with sample M3U files
• [ ] Task 1.3: Implement PatternDiscoverer
  • [ ] Create discovery/pattern_discoverer.py
  • [ ] Implement DiscoveredPattern dataclass
  • [ ] Implement _extract_prefixes() method
  • [ ] Implement _count_prefix_frequency() method
  • [ ] Implement _generate_regex() method
  • [ ] Implement _map_to_sport() method (load sport_emojis.yml)
  • [ ] Implement _assign_priority() method
  • [ ] Implement main discover() method
  • [ ] Write unit tests with TPS data samples

• [ ] Task 1.4: Implement VODPatternDiscoverer

  • [ ] Create discovery/vod_pattern_discoverer.py
  • [ ] Implement VODPattern dataclass
  • [ ] Implement _analyze_group_titles() method
  • [ ] Implement _analyze_url_paths() method
  • [ ] Implement _analyze_channel_names() method
  • [ ] Implement main discover() method
  • [ ] Write unit tests with known VOD/live channel samples

• [ ] Phase 2: Service Integration (Week 1)

• [ ] Task 2.1: Implement IdentificationMethodDetector
  • [ ] Create discovery/identification_detector.py
  • [ ] Implement IdentificationAnalysis dataclass
  • [ ] Implement detect() method
  • [ ] Implement _analyze_tvg_patterns() method
  • [ ] Implement _generate_reasoning() method
  • [ ] Write unit tests for all three scenarios (name_prefix, tvg_id, hybrid)

• [ ] Task 2.2: Implement ProviderOnboardingService

  • [ ] Create backend/epgoat/services/provider_onboarding.py
  • [ ] Implement OnboardingResult dataclass
  • [ ] Implement _create_provider() method (D1 insert)
  • [ ] Implement _fetch_m3u() method (URL/file fetch)
  • [ ] Implement _persist_patterns() method (bulk D1 insert)
  • [ ] Implement _persist_vod_patterns() method (bulk D1 insert)
  • [ ] Implement _persist_tvg_mappings() method (bulk D1 insert)
  • [ ] Implement _generate_recommendations() method
  • [ ] Implement main onboard() method
  • [ ] Define custom exceptions (ProviderAlreadyExistsError, etc.)
  • [ ] Write unit tests with mocked D1 connection

• [ ] Phase 3: CLI Tool (Week 2)

• [ ] Task 3.1: Implement CLI
  • [ ] Create cli/onboard_provider.py
  • [ ] Implement argument parsing (argparse)
  • [ ] Implement progress output with rich library
  • [ ] Implement --verbose flag for detailed output
  • [ ] Implement --dry-run flag
  • [ ] Implement JSON report generation
  • [ ] Add error handling and user-friendly error messages
• [ ] Task 3.2: Integration Testing
  • [ ] Create test fixtures (sample_tps.m3u, sample_tvg_provider.m3u)
  • [ ] Write end-to-end test: full onboarding workflow
  • [ ] Test against local Supabase database
  • [ ] Verify YAML cache generation
  • [ ] Test error scenarios (bad URL, malformed M3U, slug collision)

• [ ] Task 3.3: Documentation

  • [ ] Update Documentation/04-Guides/Command-Reference.md
  • [ ] Update Documentation/04-Guides/Quick-Start.md
  • [ ] Update Documentation/03-Architecture/System-Overview.md
  • [ ] Create Documentation/04-Guides/Provider-Onboarding.md

• [ ] Phase 4: Testing & Refinement (Week 2)

• [ ] Task 4.1: Test with TPS (validation)
  • [ ] Run onboarding against existing TPS M3U
  • [ ] Compare discovered patterns with current TPS patterns
  • [ ] Verify 90%+ pattern match
  • [ ] Adjust thresholds if needed
• [ ] Task 4.2: Test with second provider
  • [ ] Find test M3U from different provider structure
  • [ ] Run full onboarding workflow
  • [ ] Verify EPG generation works
  • [ ] Document any issues/edge cases
• [ ] Task 4.3: Performance optimization
  • [ ] Profile pattern discovery with large playlists (100k+ channels)
  • [ ] Optimize regex compilation (cache compiled patterns)
  • [ ] Optimize database bulk inserts (batch size tuning)
  • [ ] Target: <30 seconds for 100k channel playlist
• [ ] Task 4.4: Error handling improvements
  • [ ] Add retry logic for M3U fetch (network failures)
  • [ ] Add validation for generated patterns (test against sample channels)
  • [ ] Add rollback logic if database persistence fails
  • [ ] Add detailed error messages with fix suggestions

🔀 Tangent Log

Active Tangents

[No active tangents yet]

Completed Tangents

[No completed tangents yet]

📋 Original Plan

Provider Onboarding Service - Design Document

Status: Active Started: 2025-11-02 Target Completion: 2025-11-16 Priority: High Category: Architecture Design Related: Multi-Provider Architecture (Phase 2 & 3)

Table of Contents

• Overview
• Design Decisions
• Architecture
• Component Details
• Implementation Plan
• Testing Strategy
• Success Criteria

Overview

Purpose

Create a ProviderOnboardingService that enables quick, automated onboarding of new IPTV providers with minimal manual configuration. The service will analyze M3U playlists, discover channel patterns, detect VOD content, and determine the optimal identification method.

Goals

• Speed to Production: Get new providers live quickly (target: <5 minutes from M3U to working EPG)
• Automatic Discovery: Discover 70-80% of patterns automatically
• Smart Defaults: Auto-detect identification method and VOD patterns
• Data-Driven: Use frequency analysis for reliable pattern generation
• Refinement Ready: Support manual refinement via web admin later

Non-Goals

• 100% pattern coverage (diminishing returns, better done manually)
• Complex ML/clustering algorithms (YAGNI for Quick Bootstrap)
• Interactive wizard UI (CLI-first approach)
• Real-time M3U monitoring/updates

Design Decisions

Q1: Primary Use Case → Quick Bootstrap

Decision: Prioritize speed to production over comprehensive analysis.

Rationale: - Fast initial setup (70-80% coverage) is more valuable than slow perfect setup - Remaining 20-30% can be refined manually via web admin - Aligns with business goal of rapid provider expansion

Q2: Onboarding Workflow → CLI Command with M3U URL

Decision: Single command with automatic M3U fetch and analysis.

python3 onboard_provider.py \
  --name "New Provider" \
  --slug "newprovider" \
  --m3u-url "https://provider.com/playlist.m3u" \
  --analyze

Rationale: - One command = fastest workflow - No multi-step coordination needed - Easy to script/automate - Optional flags support advanced cases

Q3: Pattern Discovery Strategy → Frequency-Based (No Limits)

Decision: Frequency-based prefix analysis capturing ALL patterns with 5+ occurrences.

Rationale: - Fast execution (~10 seconds for large playlists) - High accuracy (low false positive rate) - No arbitrary limits on pattern quantity - Simple, maintainable logic - No external ML dependencies

Key Change: Original suggestion had "20-40 patterns" limit. Removed - capture all patterns meeting threshold regardless of count.

Q4: VOD Filtering Discovery → Provider-Specific Discovery

Decision: Always discover provider-specific VOD patterns during onboarding.

Rationale: - Each provider structures VOD differently (group-titles, paths, naming) - Global patterns (26 existing) won't catch provider-specific conventions - Analysis of group-titles and URL paths is fast and reliable - Critical for accurate channel filtering (TPS: 91.7% VOD filtered)

Focus Areas: 1. Group-title attributes - e.g., "Movies", "Series", "VOD", "4K Movies" 2. URL path patterns - e.g., /movies/, /series/, /vod/ 3. Channel name patterns - provider-specific indicators

Q5: Identification Method Detection → Automatic with Override

Decision: Auto-detect based on tvg-id coverage with optional --id-method flag.

Algorithm: - tvg-id coverage ≥90% → tvg_id - tvg-id coverage 10-89% → hybrid - tvg-id coverage <10% → name_prefix

Rationale: - Smart defaults = less user decision-making - Show reasoning in CLI output for transparency - Override flag supports edge cases - Hybrid is safe fallback for uncertain cases

Architecture

[... Rest of original design document content ...]

[Complete content from line 120 onwards of the original file would go here - truncated for brevity in this response, but would be included in full]

Related Documentation

• System Overview - Multi-provider architecture
• Command Reference - CLI commands
• Provider Config Manager - YAML caching
• VOD Detector - VOD filtering logic
• Migration 006 - Database schema

Changelog