2026 04 05 Managed Infra Backend 10yr Roadmap

10-Year Roadmap: Managed Infrastructure Backend

Context#

Assistance is a DevOps services platform with a marketing site, documentation, training academy, and SaaS admin dashboard. The current backend consists of 3 Go microservices (auth-api, billing-api, learn-api) running on a single Linode VPS via Podman Compose, with frontends on Cloudflare Pages/Workers.

The goal: Transform this into a managed Kubernetes + CI/CD platform serving thousands of enterprise customers over 10 years, with a managed service retainer revenue model.

Approach: Open-source assembly (Cluster API, ArgoCD, Prometheus, Backstage) + phased productization. Each phase delivers revenue while building toward full automation.

Cloud strategy: Single cloud (Linode/Akamai) + bare metal for cost optimization. Team: Growing from 4-5 to 30+ engineers.

Current State#

Phase 1: Foundation (Months 0-12)#

Goal: Production-ready internal platform. HA, observable, automated, reproducible. Team: 4-5 engineers (3 platform, 1 SRE, 1 founder/product) Revenue: $0 (internal investment)

1.1 PostgreSQL Backup (Month 1) — P0#

• pgBackRest for incremental backups + WAL archiving to Linode Object Storage
• Full backup: nightly. Differential: every 6 hours. WAL: continuous
• PITR retention: 7 days. Full backup retention: 7 copies
• Prometheus metric: pgbackrest_last_backup_age_seconds — alert if > 86400
• Monthly restore test (automated via mise task)

1.2 Monitoring Stack (Months 1-3) — P0#

• Prometheus 2.x — metrics scraping, 15-day local retention
• Grafana 11+ — dashboards, alerting UI
• Alertmanager — routing to PagerDuty/Slack
• node_exporter — host metrics
• cadvisor — container metrics
• postgres_exporter — database metrics
• blackbox_exporter — HTTP uptime probes

Go services instrumented with prometheus/client_golang via shared Gin middleware.

Initial alerts: service down >2m, PG connections >80%, disk >80%, API error rate >5%, cert expiry <14d.

1.3 CI/CD Pipeline (Months 2-4) — P1#

• GitHub Actions with self-hosted runner on Linode (LAN access to managed Artifactory)
• Managed Artifactory provides repository hosting for Docker/OCI images, Helm charts, and build artifacts, with vulnerability governance workflows where scoped
• ArgoCD 2.13+ for GitOps deployment to Kubernetes (Phase 1.7)

Backend pipeline: lint (golangci-lint) → test (go test -race) → build (Buildah) → publish image through Artifactory Docker/OCI repository → scan workflow → ArgoCD sync. Frontend pipeline: existing Cloudflare deploys triggered from GitHub Actions on merge.

Image tagging: <service>:<git-sha>-<build-number> — no latest tag.

1.4 Centralized Logging (Months 2-4) — P1#

• Loki 3.x + Promtail (not ELK — 10-50x less storage, same Grafana UI)
• Migrate Go services from unstructured logging to slog (stdlib) JSON output
• Retention: 14 days local, 90 days in Linode Object Storage

1.5 Secrets Management (Months 3-5) — P1#

• Vault OSS 1.17+ — dynamic secrets, PKI, encryption as service
• Raft storage backend (3-node), AppRole auth for Go services
• SOPS + age for static secrets in git (decryption key in Vault)
• Dynamic PostgreSQL credentials per service via Vault database engine
• Internal mTLS via Vault PKI

1.6 Infrastructure as Code (Months 2-5) — P1#

• Terraform with Linode provider — import all existing resources
• Ansible for host configuration (hardening, packages, users)
• State backend: Terraform Cloud free tier or Linode Object Storage (S3-compatible)
• Module structure: infra/terraform/modules/{compute,database,networking,monitoring}
• terraform plan on PR, terraform apply on merge (environment gate)

1.7 Internal Kubernetes Cluster (Months 4-8) — P2#

• Talos Linux 1.8+ on Linode VMs — immutable, API-driven, no SSH
• Cilium 1.16+ — CNI + network policy + eBPF observability + service mesh (no Istio)
• CloudNativePG — PostgreSQL operator (replaces Patroni, K8s-native backup/failover)
• Longhorn — distributed storage (alongside Linode Block Storage)

Cluster topology: 3x control plane (Dedicated 4GB), 3x workers (Dedicated 16GB). Namespaces: assistance-prod, assistance-staging, monitoring, argocd, vault, harbor.

Migration: deploy K8s alongside VPS → deploy services to K8s → cut DNS → decommission VPS.

Why Talos over LKE: Full control over node OS, same tooling for bare metal in Phase 3, API-only management, immutable. Why Cilium over alternatives: One component for networking + policy + mesh + observability. eBPF-based, no sidecars.

1.8 New Backend Services (Months 4-10) — P2#

• tenant-api — organization lifecycle, cluster metadata, environment management
• provisioning-api — Terraform-driven cluster provisioning, state machine with worker pool
• gateway-api — thin Go reverse proxy for auth, rate limiting, request tracing

New services use pgx/v5 + sqlc (not GORM). Existing services keep GORM for now. Replace GORM AutoMigrate with golang-migrate for all services. Separate logical databases per service domain on shared PostgreSQL.

1.9 Event-Driven Foundation (Months 6-10) — P2#

• NATS JetStream — Go-native, low ops overhead, sufficient at <10K events/sec
• Events: cluster lifecycle, billing triggers, audit trail
• Consumer pattern: at-least-once delivery, idempotent handlers

1.10 Developer Portal (Months 6-10) — P3#

• Backstage 1.30+ — service catalog, scaffolding, TechDocs
• Internal only in Phase 1. Customer-facing in Phase 3.
• Plugins: kubernetes, argocd, grafana, cost-insights, github-actions, techdocs

Phase 1 Infrastructure Cost: ~$822/month#

Phase 2: Customer Platform (Years 1-3)#

Goal: Provision and operate K8s clusters for paying customers. First 10-50 customers. Team: 7-9 engineers (platform 4, customer eng 2-3, SRE 2) Revenue: ~$100K-$600K ARR

2.1 Cluster API Provisioning (Months 12-18)#

• Cluster API 1.8+ with Linode infrastructure provider (CAPL)
• Talos Bootstrap + Control Plane providers — same OS as internal cluster
• ClusterClass — standardized cluster templates per tier
• ArgoCD ApplicationSets — one ArgoCD on mgmt cluster targeting all tenant clusters
• Provisioning time target: <15 minutes from request to running cluster

Customer cluster tiers:

Architecture: Management cluster runs CAPI controllers. Each customer gets a dedicated K8s cluster (not shared). Security boundary = VM, not namespace.

2.2 Multi-Tenancy & Isolation (Months 12-16)#

• Dedicated clusters per tenant (strongest isolation)
• Kyverno 1.12+ — admission policies deployed to all clusters via ArgoCD
• Vault PKI — per-cluster certificate authority
• OpenCost — per-tenant cost metering
• Kubernetes Audit Logs → Loki — tenant activity audit trail

2.3 Customer Networking (Months 14-20)#

• Envoy Gateway per-cluster — L7 traffic routing
• cert-manager 1.16+ — automated TLS via Let's Encrypt + Vault PKI
• ExternalDNS → Linode DNS + Cloudflare — automatic DNS for customer domains
• Cloudflare proxy — DDoS/WAF for customer domains
• WireGuard — customer on-prem VPN connectivity

2.4 Monitoring as a Service (Months 16-22)#

• Prometheus Agent (per-cluster) → Thanos Receive (central, multi-tenant)
• Promtail (per-cluster) → Loki (central, multi-tenant)
• OTel Collector (per-cluster) → Tempo (central)
• Grafana multi-org — per-tenant dashboards
• Sloth — SLO generator, burn-rate alerts
• Default dashboards per tenant + custom dashboard capability

2.5 Customer Onboarding Automation (Months 18-24)#

Onboarding = Git commit (cluster manifest in GitOps repo). ArgoCD reconciles.

Flow: Contract signed → Stripe subscription → CAPI manifest from ClusterClass → cluster provisioned → add-ons deployed → Grafana org created → VPN configured → kubeconfig + portal access delivered.

Standard tiers: fully automated, no engineer intervention. Enterprise: manual network configuration documented as runbooks.

2.6 Disaster Recovery (Months 14-20)#

• Velero 1.14+ — K8s resource + PV backup
• CloudNativePG backup to Linode Object Storage (WAL archiving + scheduled base backups)
• Cross-region backup for Enterprise tier
• Litmus 3.x — monthly automated DR drills

DR targets per tier:

2.7 Incident & SLA Management (Months 18-24)#

• Grafana OnCall (OSS) — replaces PagerDuty free tier at 10+ engineers
• Rundeck — runbook automation (auto-trigger from alerts)
• Gatus — per-customer status pages at status.customer-domain.com
• SLO dashboards with burn-rate alerting (multi-window from SRE workbook)
• Automated monthly SLA reports emailed to customers

2.8 Backend Evolution (Years 1-3)#

• monitoring-api — TimescaleDB for time-series customer metrics
• notification-svc — email, Slack, webhook notifications
• audit-svc — immutable audit log
• Multi-tenancy via org_id on all tables + PostgreSQL RLS
• Database: add read replicas, deploy PgBouncer, begin sqlc migration
• Billing: org-level billing + usage tracking + Stripe integration

Phase 2 Platform Cost: ~$1,400/month (excluding customer clusters)#

Phase 3: Scale (Years 3-5)#

Goal: 100-500 customers. Automation replaces manual operations. Bare metal enters fleet. Team: 10-14 engineers (platform 5, customer eng 3, SRE 3, product 2-3) Revenue: ~$2.5M-$7M ARR

3.1 Bare Metal Integration (Months 24-36)#

• Tinkerbell — PXE boot, OS install, hardware lifecycle
• CAPT (Cluster API Provider Tinkerbell) — unified lifecycle across cloud + bare metal
• Talos Linux on bare metal (same as cloud — identical management)
• Initial: 10 servers at Hetzner (dedicated, cost-effective)

Cost advantage: 77-83% savings vs cloud VMs at scale. Customer-transparent: same ClusterClass templates work for both.

3.2 Fleet Management (Months 24-30)#

• ArgoCD ApplicationSets (generator: cluster) — GitOps across all clusters
• Kyverno PolicySets — fleet-wide policy enforcement
• CAPI rolling upgrades — coordinated K8s version upgrades (7-day customer notification)
• Crossplane 1.17+ — managed infrastructure CRDs (PostgreSQLInstance, RedisCluster)
• Custom ClusterHealth operator — health scoring + automated remediation

3.3 Self-Service Portal (Months 26-34)#

• React portal (extend apps/manager, reuse packages/ui + packages/saas-ui)
• Go portal API — cluster CRUD, billing, support (wraps CAPI)
• CLI tool (assistance-cli) — developer-friendly cluster management
• Custom Terraform provider — IaC for customer infrastructure
• Dex (OIDC) — enterprise SSO integration

Self-service: provision clusters, scale nodes, schedule upgrades, install add-ons, view costs, manage backups, create support tickets.

All portal actions → Git commits in GitOps repo. Portal is UI over GitOps, not bypass.

3.4 Compliance Automation (Months 30-40)#

• kube-bench — CIS benchmark scanning (CronJob per cluster)
• Trivy Operator — runtime vulnerability scanning
• Syft + Grype — SBOM generation
• Falco — runtime security (syscall monitoring)
• OPA Gatekeeper — admission policies (no root, resource limits, pinned images)
• SOC 2 Type II certification targeted by end of Year 3
• Vanta or Drata for automated evidence collection

3.5 Cost Management (Months 24-30)#

• OpenCost — per-customer K8s cost allocation
• VPA (recommendation mode) — weekly right-sizing PRs
• HPA — auto-scale services (CPU + custom Prometheus metrics)
• KEDA — event-driven autoscaling (queue depth)
• Chargeback: OpenCost data → billing-api → customer invoices

3.6 Chaos Engineering (Months 28-34)#

• Chaos Mesh — pod, network, IO, HTTP, stress chaos
• Monthly gamedays: kill pods, inject latency, fail replicas, stress namespaces
• Steady-state hypothesis before each experiment
• Staging first, then production with on-call watching

3.7 Backend Decomposition (Years 3-5)#

• provisioning splits into orchestrator + worker
• tenant-api splits into org-api + cluster-api
• monitoring-api splits into ingest + query
• billing-api splits into core + metering
• gRPC internally (connect-go), REST stays external
• River for durable job processing (replaces ad-hoc goroutine workers)
• OpenTelemetry for distributed tracing
• Complete sqlc migration, retire GORM
• PostgreSQL HA via CloudNativePG (3-node clusters)

Phase 3 Platform Cost: ~$3,850/month#

Phase 4: Enterprise (Years 5-10)#

Goal: 500-5,000 customers. Multi-region. Compliance-certified. Partner ecosystem. Team: 20-30+ engineers (platform 8-10, SRE 5-6, customer eng 5-8, product 3-5) Revenue: ~$7M-$50M+ ARR

4.1 Multi-Region Architecture (Years 5-7)#

• 3 regions: EU-West (Frankfurt), US-East (Dallas), APAC (Singapore)
• Per-region management cluster — no single global bottleneck
• Global control plane: lightweight metadata sync (cluster inventory, policy distribution)
• Cross-region failover for Enterprise tier
• Akamai edge for edge K8s (K3s at edge POPs)

4.2 Advanced Compliance (Years 5-8)#

Compliance is a platform feature — customers inherit posture. Per-tenant immutable audit logs (WORM, 1-year retention).

4.3 SLA Engine (Years 5-7)#

Custom Go service for SLA calculation, breach detection, automated credit application. 99.99% requires multi-region active-active with automated failover.

4.4 Add-on Marketplace (Years 7-10)#

• Managed add-ons: PostgreSQL, MySQL, MongoDB, Redis, Dragonfly, NATS, RabbitMQ
• Partner Helm chart publishing with approval workflow
• White-label offering for MSPs and system integrators

4.5 AI/ML Infrastructure (Years 6-10)#

• NVIDIA GPU Operator + MIG — GPU workload support
• KServe — model serving
• Bare metal GPU nodes as specialized node pools
• "AI-ready Kubernetes" positioning

4.6 Backend End State (Year 10)#

• ~15 services, ~150-200K LOC
• Multi-region federation with independent regional stacks + global control plane
• Plugin architecture via configuration-driven extensions
• Advanced billing: usage-based metering, committed-use discounts, multi-currency
• Enterprise auth: SAML SSO, SCIM provisioning, custom roles

Technology Radar#

Adopt (use now)#

Talos Linux, Cilium, ArgoCD, Prometheus/Thanos, Loki, Grafana, CloudNativePG, Vault, Terraform, Cluster API, Kyverno, cert-manager, Velero, Artifactory, NATS JetStream, pgx/v5, sqlc, golang-migrate

Trial (next phase)#

Crossplane, Backstage, Tinkerbell, Envoy Gateway, River (job queue), connect-go (gRPC)

Assess (watch)#

Liqo (federation), KServe (ML), Gateway API, SpinKube (WASM), CockroachDB

Hold (avoid)#

Istio (Cilium covers it), Docker Swarm, Rancher (build own), managed K8s services (we ARE the managed service), Kafka (NATS is simpler for this scale)

Team Growth#

Key hires: Phase 1 (Senior SRE), Phase 2 (Networking + Security engineers), Phase 3 (Bare metal + Product manager), Phase 4 (Regional leads + Compliance officer).

Risk Register#

Epic & Ticket Breakdown#

Epic 1: Database Safety Net (Phase 1, Month 1)#

Epic 2: Observability Stack (Phase 1, Months 1-3)#

Epic 3: Centralized Logging (Phase 1, Months 2-4)#

Epic 4: CI/CD Pipeline (Phase 1, Months 2-4)#

Epic 5: Secrets Management (Phase 1, Months 3-5)#

Epic 6: Infrastructure as Code (Phase 1, Months 2-5)#

Epic 7: Internal Kubernetes Cluster (Phase 1, Months 4-8)#

Epic 8: New Backend Services (Phase 1, Months 4-10)#

Epic 9: Developer Portal (Phase 1, Months 6-10)#

Epic 10: Cluster API & Customer Provisioning (Phase 2, Months 12-18)#

Epic 11: Multi-Tenant Monitoring (Phase 2, Months 16-22)#

Epic 12: Incident & Operations (Phase 2, Months 18-24)#

Epic 13: Bare Metal Integration (Phase 3, Months 24-36)#

Epic 14: Self-Service Portal (Phase 3, Months 26-34)#

Epic 15: Compliance Automation (Phase 3, Months 30-40)#

Epic 16: Multi-Region (Phase 4, Years 5-7)#

Epic 17: Enterprise Features (Phase 4, Years 5-10)#

Verification#

How to validate Phase 1 completion#

1. pgbackrest verify passes — backups restorable
2. Grafana dashboards show all services healthy with 7+ days history
3. Merge to main triggers automated build → deploy with no manual steps
4. vault status shows sealed/unsealed state, all secrets migrated
5. terraform plan shows no drift from actual infrastructure
6. kubectl get nodes shows healthy Talos cluster
7. All 3 existing services + 3 new services running in K8s
8. Loki shows structured JSON logs from all services

How to validate Phase 2 completion#

1. CAPI provisions a customer cluster in <15 minutes from manifest apply
2. ArgoCD ApplicationSet deploys add-ons to new cluster automatically
3. Customer Grafana org shows their cluster metrics/logs/traces only
4. Velero backup + restore tested for customer cluster
5. Gatus status page shows uptime for all customers
6. SLO dashboard shows burn-rate for each customer's SLA tier