Unified Security Control Plane (USCP) — Implementation Guide

The single, authoritative, step-by-step guide to deploying USCP on your own cloud or on-prem estate.

Audience: an IT / DevOps engineer who has never seen this product before. This guide assumes no prior knowledge. Every step is explicit, copy-pasteable, and explained (what it does and why). You should be able to go from an empty machine to a verified, production-grade deployment without contacting support.

How to use this guide. Read Sections 1–3 once (Overview, Prerequisites, Choosing a path). Then jump to the one sub-section of Section 4 that matches your chosen deployment path and follow it top to bottom. Sections 5–10 (env-var reference, post-install, verification, upgrades, troubleshooting) apply to every path — keep them open in a second tab.

This guide is the index that ties together the detailed per-environment guides in docs/install/. Where a per-environment guide has more depth (e.g. cloud-specific IAM), this guide links to it and does not duplicate it.

Table of contents

1. Overview — what USCP is
2. Prerequisites
3. Choose your deployment path
4. Deployment walkthroughs
• 4.1 Single-VM one-command installer
• 4.2 AWS
• 4.3 Azure
• 4.4 GCP
• 4.5 On-prem (VMware vSphere / Nutanix / any hypervisor / bare metal)
• 4.6 Kubernetes / Helm
• 4.7 Air-gapped (no internet)
• 4.8 Local pilot / dry-run
5. Complete environment-variable reference
6. Post-install configuration
7. Verification & acceptance
8. Upgrades, backup/restore, and rollback
9. Troubleshooting
10. Where to go next

1. Overview

1.1 What USCP is

The Unified Security Control Plane (USCP) is a self-hostable security platform delivered as:

• One statically-linked Go binary called controlplane. It contains the REST/gRPC API, background workers, and the React admin console embedded inside the binary (via Go embed). There is no separate frontend to build, host, or serve — the binary serves the console itself.
• A PostgreSQL 16 database — the only stateful component. All tenant data, audit chains, licensing state, and configuration live here. The binary itself is stateless: back up Postgres and you can rebuild everything.
• Optionally, an Identity Provider (IdP) for corporate single sign-on (SSO) — any OIDC or SAML provider (Google, Microsoft Entra ID, Okta, Keycloak, PingFederate, ADFS…). SSO is optional; USCP ships a passwordless local login too.

"Self-deploy" means: you run it entirely on infrastructure you control — your cloud account, your VM, your Kubernetes cluster, or a fully air-gapped network. In the default (airgap licensing) mode there is no dependency on any DoubleLogic-hosted service at runtime: no phone-home, no external control plane.

1.2 Current build facts

The build this guide describes ships:

1.3 Architecture (ASCII diagram)

                     ┌─────────────────────────────────────────────────────────┐
                     │                   Your infrastructure                    │
                     │                                                          │
  ┌──────────┐  443  │  ┌──────────────┐  8080  ┌───────────────────────────┐   │  5432  ┌──────────────┐
  │ Browser  │──────────▶│ TLS front    │───────▶│  controlplane (Go binary) │───────────▶│ PostgreSQL 16│
  │ (admin)  │  HTTPS │  │ Caddy / LB / │  HTTP  │  ┌─────────────────────┐  │   │  TCP   │ per-tenant   │
  └──────────┘        │  │ Ingress      │        │  │ embedded React UI    │  │   │        │ RLS (FORCE)  │
       │              │  │ (Let's       │        │  ├─────────────────────┤  │   │        └──────────────┘
       │ SSO redirect │  │  Encrypt or  │        │  │ REST + gRPC API      │  │   │
       ▼              │  │  your cert)  │        │  ├─────────────────────┤  │   │
  ┌──────────┐        │  └──────────────┘        │  │ background workers   │  │   │
  │  IdP     │◀──── OIDC/SAML ───────────────────│  └─────────────────────┘  │   │
  │ (OIDC/   │        │                          └───────────────────────────┘   │
  │  SAML)   │        │                                     │                     │
  └──────────┘        │                                     │ verifies at boot    │
   optional           │                          ┌──────────▼──────────┐          │
                      │                          │ Licence (signed)     │          │
                      │                          │ airgap bundle + JWKS │          │
                      │                          │  or online install   │          │
                      │                          │  key                 │          │
                      │                          └─────────────────────┘          │
                      └─────────────────────────────────────────────────────────┘

Traffic flow: a browser hits https://<your-domain> on 443. A TLS terminator (Caddy on a VM, or an Ingress/Load Balancer on Kubernetes) decrypts and forwards to the controlplane binary on 8080 (HTTP, loopback / cluster-internal only). The binary talks to PostgreSQL 16 on 5432. For SSO, the browser is redirected to your IdP and back to https://<your-domain>/auth/callback.

1.4 Components created by an install (VM path)

When you run the one-command installer on a VM it provisions all of this for you:

2. Prerequisites

Work through these once, regardless of path. Checklists are at the end of this section.

2.1 Hardware sizing

Reference cloud instance types at minimum sizing: AWS t3.medium, Azure Standard_B2s, GCP e2-medium. If you run Postgres inside the VM (Docker mode) add ~1 GB RAM; use an external managed database for production.

2.2 Operating system

• Linux only for the server. Ubuntu 22.04 LTS (or 24.04) is the primary target and what the cloud-init/Terraform paths provision. RHEL 9 / Rocky / Alma / Amazon Linux 2023 / SUSE are supported by the installer's package-manager detection (apt/dnf/yum/zypper).
• The binary is CGO_ENABLED=0 static — it runs on any modern Linux with no runtime dependencies.

2.3 Software (per path)

Install only what your chosen path needs:

Check what a host already has with:

make tools-check    # from codes/ — reports go, docker, cosign, syft, etc. as ok/MISSING

2.4 Network, ports, and firewall

2.5 DNS + TLS certificate

• Choose a hostname, e.g. uscp.example.com, and create an A record (or AAAA) pointing at the server's public IP. Create it early — DNS propagation can take minutes to hours, and both TLS issuance and the OIDC redirect depend on the name resolving.
• Public TLS: Caddy obtains a Let's Encrypt certificate automatically (needs port 80 reachable and the name resolving to this host).
• Internal / air-gapped TLS: a non-public name cannot be validated by public Let's Encrypt. Either front USCP with an internal load balancer holding a certificate from your internal CA (run USCP with USCP_TLS=none), or give the binary your own cert via USCP_TLS_CERT / USCP_TLS_KEY.

2.6 An Identity Provider (optional)

SSO is optional (the default login is a passwordless email one-time code or authenticator app). To use SSO, register an application ("client" / "app registration") with your IdP and collect:

• Issuer URL → USCP_OIDC_ISSUER (e.g. https://accounts.google.com, https://login.microsoftonline.com/<tenant-id>/v2.0, your Okta/Keycloak issuer).
• Client ID → USCP_OIDC_CLIENT_ID
• Client secret → USCP_OIDC_CLIENT_SECRET
• Redirect URI — set it exactly to https://<your-domain>/auth/callback. A mismatch here is the #1 cause of a failed first login.

Per-IdP walkthroughs (Google, Entra ID, Okta, Keycloak) are in docs/install/post-install.md. SAML providers are registered in-app after first login.

2.7 SMTP relay (optional but recommended)

The default login method is a passwordless email one-time code, which needs a working SMTP relay. Collect host:port and (if authenticated) a username + password. The installer verifies SMTP actually works before proceeding (connect + EHLO + STARTTLS + AUTH, no mail sent). If you have no SMTP, you can use TOTP (authenticator app) — no external dependency — or SSO.

2.8 A licence

USCP verifies a signed entitlement before it will serve. Two modes:

• airgap (the binary's default) — a signed entitlement bundle (compact JWS) plus a trusted JWKS, both files on disk. Self-contained, no internet. Best for evaluation, on-prem, and air-gapped installs. For production you should use a vendor-signed, host-bound bundle (request one from DoubleLogic using this host's fingerprint — see §6.6). For non-production you can self-sign one with the bundled devlicgen tool.
• online — the binary calls the DoubleLogic licensing service (licensing.doublelogic.org) at boot using an install key (USCP_INSTALL_KEY). Requires outbound HTTPS.

Note on the installer default. The deploy/install.sh one-command installer defaults LICENSING_MODE=online (production default) and requires USCP_INSTALL_KEY. If you have no install key, pass LICENSING_MODE=airgap and supply a bundle, or (non-production only) USCP_DEV_LICENSE=1 to self-sign. The binary itself defaults to airgap.

2.9 Data-at-rest encryption key

USCP encrypts sensitive fields (vaulted credentials, sealed secrets) at rest with a 32-byte AES key supplied via USCP_DATA_KEY (64 hex characters). You must set this and preserve it — if it changes, previously sealed secrets can no longer be decrypted. The VM installer generates one automatically into /etc/uscp/controlplane.env and preserves it across re-runs. On Kubernetes/manual paths you must set it yourself:

openssl rand -hex 32     # → a 64-char hex string; store it in your secret manager

For enterprise key management, USCP supports BYOK/HYOK via USCP_KMS_PROVIDER and related variables (see §5).

2.10 Pre-flight checklists

Common (every path):

• [ ] Server meets sizing (2 vCPU / 4 GB / 20 GB minimum).
• [ ] PostgreSQL 16 plan chosen (in-box container, native, or external managed — recommended for prod).
• [ ] A non-superuser DB role is available for production (superusers bypass row-level security — see §6.5).
• [ ] DNS A record created (public paths).
• [ ] Licence plan chosen (airgap bundle vs online install key).
• [ ] USCP_DATA_KEY generated and stored (auto for the VM installer).
• [ ] Login method decided: email OTP (needs SMTP) / TOTP / SSO.

If using SSO:

• [ ] IdP app registered with redirect https://<host>/auth/callback.
• [ ] Issuer, client ID, client secret collected.

If air-gapped:

• [ ] Connected build host available to build the binary + licence bundle.
• [ ] Approved transfer path into the enclave (one-way diode / vetted USB / DMZ).
• [ ] Internal CA cert and internal IdP ready.

3. Choose your deployment path

Pick one path. If unsure, use 4.1 (single-VM installer) — it is the fastest way to a fully-verified production deployment on a single trusted host.

Decision shortcuts:

• Single trusted box, cloud or on-prem → the one-command installer (4.1). The AWS/Azure/GCP/on-prem guides all drive it.
• Infrastructure-as-Code → the Terraform module for your platform provisions the VM and runs the installer in one terraform apply (AWS, Azure, GCP, vSphere, Nutanix).
• HA across 3+ replicas with an external database → Kubernetes/Helm (4.6).
• No internet at all → air-gap (4.7).

Always get the code first. Every VM/source path starts from a checkout:

git clone https://github.com/cfssay/doublelogic_apex uscp && cd uscp/codes

All installer commands below are run from this codes/ directory unless noted.

4. Deployment walkthroughs

Every path lands on the same verified boot + acceptance flow. After any path, run the acceptance suite (§7) and complete post-install (§6).

4.1 Single-VM one-command installer (the recommended default)

deploy/install.sh is an idempotent (safe to re-run) production installer. In one command it: builds (or installs) the binary, provisions Postgres, generates/installs the licence, writes /etc/uscp/controlplane.env, installs the hardened systemd unit, applies all migrations, sets up Caddy auto-TLS, opens the firewall, health-checks, and runs the acceptance suite.

Step 1 — Provision the VM and DNS

Create a VM (2 vCPU / 4 GB / 20 GB min) with a static/public IP. Open inbound 22 (your IP), 80, 443. Create A uscp.example.com → <public IP> and confirm it resolves:

dig +short uscp.example.com     # must return your server's IP

Expected: the command prints your VM's public IP. If empty, wait for DNS to propagate before continuing (Caddy will retry ACME, but resolving first avoids a failed cert).

Step 2 — Install prerequisites

sudo apt-get update && sudo apt-get install -y git curl
# For the default Docker database mode:
sudo apt-get install -y docker.io && sudo systemctl enable --now docker

(Go is installed by the installer when building from source; skip that by passing USCP_BINARY=.)

Step 3 — Get the code

git clone https://github.com/cfssay/doublelogic_apex uscp
cd uscp/codes

Step 4 — Run the installer

Choose the invocation matching your licence + database choices.

A) Simplest — self-terminating TLS, Docker Postgres, self-signed air-gap licence, SSO login:

sudo \
  USCP_DOMAIN=uscp.example.com \
  USCP_ACME_EMAIL=ops@example.com \
  USCP_ADMIN_EMAIL=admin@example.com \
  LICENSING_MODE=airgap \
  USCP_DEV_LICENSE=1 \
  USCP_LOGIN_METHOD=sso \
  USCP_OIDC_ISSUER=https://accounts.google.com \
  USCP_OIDC_CLIENT_ID=<client-id> \
  USCP_OIDC_CLIENT_SECRET=<client-secret> \
  ./deploy/install.sh

USCP_DEV_LICENSE=1 self-signs a non-production host-bound bundle. For production, omit it and instead supply a vendor-signed bundle (USCP_LICENSE_BUNDLE=/path/entitlement.lic USCP_LICENSE_JWKS_IN=/path/jwks.json), or use LICENSING_MODE=online USCP_INSTALL_KEY=<key>.

B) Default email one-time-code login (needs SMTP):

sudo \
  USCP_DOMAIN=uscp.example.com \
  USCP_ACME_EMAIL=ops@example.com \
  USCP_ADMIN_EMAIL=admin@example.com \
  USCP_SMTP_HOST=smtp.example.com:587 \
  USCP_SMTP_FROM=no-reply@example.com \
  USCP_SMTP_USERNAME=apikey \
  USCP_SMTP_PASSWORD='<smtp-password>' \
  LICENSING_MODE=online USCP_INSTALL_KEY=<your-install-key> \
  ./deploy/install.sh

C) Authenticator (TOTP) login — no SMTP, no SSO:

sudo \
  USCP_DOMAIN=uscp.example.com \
  USCP_ACME_EMAIL=ops@example.com \
  USCP_LOGIN_METHOD=totp \
  LICENSING_MODE=airgap USCP_DEV_LICENSE=1 \
  ./deploy/install.sh

D) External managed Postgres (production):

sudo \
  USCP_DOMAIN=uscp.example.com \
  USCP_ACME_EMAIL=ops@example.com USCP_ADMIN_EMAIL=admin@example.com \
  USCP_OIDC_ISSUER=https://accounts.google.com \
  USCP_OIDC_CLIENT_ID=<id> USCP_OIDC_CLIENT_SECRET=<secret> \
  USCP_LOGIN_METHOD=sso \
  LICENSING_MODE=online USCP_INSTALL_KEY=<key> \
  USCP_DB_MODE=external \
  DATABASE_URL='postgres://uscp_app:PASSWORD@db.internal:5432/uscp?sslmode=require' \
  ./deploy/install.sh

Behind your own load balancer (no Caddy): add USCP_TLS=none; USCP serves plain HTTP on :8080 and your LB fronts it. Also set USCP_PUBLIC_URL=https://uscp.example.com.

Interactive mode. Running sudo ./deploy/install.sh with no USCP_* variables on a real terminal launches a short guided wizard (domain, admin email, SMTP, optional OIDC). Automated runs (cloud-init/Terraform/CI, or USCP_NONINTERACTIVE=1) skip prompts.

Step 5 — Watch it run

The installer prints progress phases: Preflight → System user + directories → Control-plane binary → login method (verifies SMTP if chosen) → PostgreSQL → Licensing (preflight-verified on this host) → Runtime config → systemd service (applies migrations) → Caddy + TLS → Health check → Acceptance suite → summary.

Expected final output includes:

==> Install complete.
  Console GUI: https://uscp.example.com/
  Front door : https://uscp.example.com/auth/login
  ...
  ✓ acceptance suite PASSED

Step 6 — Verify

sudo systemctl status controlplane            # active (running)
curl -fsS https://uscp.example.com/healthz     # {"status":"ok"}
curl -fsS https://uscp.example.com/readyz      # {"status":"ready"}  (DB connected + migrations applied)

Then complete the one manual step (add the redirect URI at your IdP, if using SSO) and log in — see §6. If anything is off, jump to §9.

4.2 AWS

Three sub-paths; full detail in docs/install/aws.md.

Path A — Terraform (recommended)

The module in codes/deploy/terraform/ provisions an EC2 instance, a locked-down security group (22/80/443), and cloud-init user_data that runs deploy/install.sh unattended.

1. cd uscp/codes/deploy/terraform
2. Create terraform.tfvars:

   region             = "us-east-1"
   instance_type      = "t3.medium"                 # 2 vCPU / 4 GB minimum
   key_name           = "my-ec2-keypair"            # an existing EC2 key pair in this region
   ssh_ingress_cidr   = "203.0.113.4/32"            # YOUR ip/32 — never 0.0.0.0/0
   domain             = "uscp.example.com"
   oidc_issuer        = "https://accounts.google.com"
   oidc_client_id     = "1234567890-abcdef.apps.googleusercontent.com"
   oidc_client_secret = "GOCSPX-xxxxxxxxxxxxxxxxxxxx"
   acme_email         = "ops@example.com"

1. Apply, then point DNS at the printed IP:

   terraform init
   terraform plan          # review: 1 instance, 1 security group
   terraform apply         # type 'yes'  → Outputs: public_ip = "54.x.x.x"
   # create A record uscp.example.com → 54.x.x.x, then:
   dig +short uscp.example.com

1. Watch the install finish and verify:

   ssh -i ~/.ssh/my-ec2-keypair.pem ubuntu@54.x.x.x
   sudo tail -f /var/log/cloud-init-output.log        # build → migrate → license → verify
   curl -fsS https://uscp.example.com/readyz

If ACME failed because DNS wasn't ready, sudo systemctl restart caddy after it resolves.

Keep terraform.tfvars out of version control (it holds the OIDC secret); prefer TF_VAR_oidc_client_secret=... from a secrets manager. terraform destroy tears it down (dump Postgres first).

Path B — Manual EC2 + install.sh

Launch Ubuntu 22.04 (t3.medium, 20 GB gp3), SG inbound 22/your-IP + 80 + 443 (never 8080/5432), create DNS, then follow §4.1 from Step 2. For production use RDS for PostgreSQL 16 with USCP_DB_MODE=external + a non-superuser uscp_app role and sslmode=require.

Path C — EKS + Helm

See §4.6 and docs/install/aws.md for ECR image build and RDS specifics.

4.3 Azure

Full detail in docs/install/azure.md. Microsoft Entra ID is the natural OIDC provider — issuer https://login.microsoftonline.com/<tenant-id>/v2.0, redirect https://<domain>/auth/callback.

Path A — Terraform (recommended)

Module codes/deploy/terraform/azure/ provisions RG, VNet/subnet, NSG (SSH to your CIDR; 80/443 public), public IP, and an Ubuntu 24.04 VM whose cloud-init runs the installer.

cd uscp/codes/deploy/terraform/azure
terraform init
terraform apply \
  -var="domain=uscp.example.com" \
  -var="ssh_public_key=$(cat ~/.ssh/id_ed25519.pub)" \
  -var="oidc_issuer=https://login.microsoftonline.com/<tenant-id>/v2.0" \
  -var="oidc_client_id=<application-client-id>" \
  -var="oidc_client_secret=<client-secret-value>" \
  -var="acme_email=ops@example.com" \
  -var="ssh_ingress_cidr=203.0.113.4/32"

Then: terraform output public_ip → create the DNS A record → confirm the redirect URI on the Entra app → ssh azureuser@<ip> sudo tail -f /var/log/uscp-bootstrap.log → verify https://<domain>/readyz.

Path B — Azure VM + install.sh (cloud-init or manual)

Use deploy/cloud-init.yaml as Azure custom data (edit the install.env block), or SSH in and run the installer by hand (§4.1). For production use Azure Database for PostgreSQL – Flexible Server (v16) with USCP_DB_MODE=external + a non-superuser uscp_app role; never use the server admin account (it bypasses RLS).

Path C — AKS + Helm

See §4.6 and docs/install/azure.md (ACR build, Azure DB).

4.4 GCP

Full detail in docs/install/gcp.md. Google is the natural OIDC provider — issuer https://accounts.google.com.

Path A — Terraform (recommended)

Module codes/deploy/terraform/gcp/ provisions a static IP, firewall rules (SSH to your CIDR; 80/443 scoped by the uscp network tag), and an Ubuntu 24.04 VM running the installer via cloud-init.

cd uscp/codes/deploy/terraform/gcp
terraform init
terraform apply \
  -var="project=my-gcp-project" \
  -var="domain=uscp.example.com" \
  -var="oidc_client_id=...apps.googleusercontent.com" \
  -var="oidc_client_secret=GOCSPX-..." \
  -var="acme_email=ops@example.com" \
  -var="ssh_ingress_cidr=203.0.113.4/32" \
  -var="ssh_public_key=$(cat ~/.ssh/id_ed25519.pub)"

Then: terraform output public_ip → DNS A record → confirm the redirect URI on the Google OAuth client → gcloud compute ssh uscp-control-plane --zone <zone> --command 'sudo tail -f /var/log/uscp-bootstrap.log' → verify /readyz.

Path B — Compute Engine VM + startup script

Reserve a static IP, open firewall (tag uscp), create the instance with a startup-script that clones the repo and runs deploy/install.sh (see the guide). Delete the startup-script metadata after first boot since it contains the OIDC secret, or use Secret Manager. For production use Cloud SQL for PostgreSQL 16 via the Cloud SQL Auth Proxy on 127.0.0.1:5432 (sslmode=disable is safe only because the proxy tunnels), with a non-superuser role.

Path C — GKE + Helm

See §4.6 and docs/install/gcp.md.

4.5 On-prem (VMware vSphere / Nutanix / any hypervisor / bare metal)

Full detail in docs/install/onprem-vm.md.

Automated (vSphere / Nutanix) via Terraform

Terraform provisions the Ubuntu 24.04 image itself (no manual golden image) and runs the installer via cloud-init.

vSphere / vCenter:

cd uscp/codes/deploy/terraform/vsphere
terraform init
terraform apply \
  -var="vsphere_server=vcenter.corp.local" \
  -var="vsphere_user=administrator@vsphere.local" -var="vsphere_password=..." \
  -var="datacenter=DC1" -var="cluster=Cluster1" -var="datastore=vsanDatastore" \
  -var="network=VM Network" -var="esxi_host=esxi01.corp.local" \
  -var="domain=uscp.corp.local" \
  -var="oidc_issuer=https://keycloak.corp.local/realms/uscp" \
  -var="oidc_client_id=uscp" -var="oidc_client_secret=..." \
  -var="acme_email=ops@corp.local"
# air-gap / static IP: -var="create_from_ovf=false" -var="vm_template=ubuntu-2404-cloudinit" \
#                      -var="ip_address=10.0.0.20" -var="ip_prefix=24" -var="gateway=10.0.0.1"

Nutanix AHV (Prism Central):

cd uscp/codes/deploy/terraform/nutanix
terraform init
terraform apply \
  -var="prism_endpoint=prismcentral.corp.local" \
  -var="prism_username=admin" -var="prism_password=..." \
  -var="cluster_name=AHV-Cluster-1" -var="subnet_name=vlan0-managed" \
  -var="domain=uscp.corp.local" \
  -var="oidc_issuer=https://keycloak.corp.local/realms/uscp" \
  -var="oidc_client_id=uscp" -var="oidc_client_secret=..." \
  -var="acme_email=ops@corp.local"
# air-gap: -var="create_image=false" -var="image_name=<your-uploaded-image>"

terraform output ip_address → create the DNS record → verify https://<domain>/readyz. Optional zero-touch internal DNS via RFC 2136 (-var="dns_zone=..." -var="dns_server=..." -var="dns_tsig_key_name=..." -var="dns_tsig_secret=...").

Manual (any hypervisor / bare metal)

1. Provision a VM (2 vCPU / 4 GB / 20 GB; static IP + resolvable hostname).
2. Decide three things: TLS (USCP_TLS=auto self-terminate, or USCP_TLS=none behind an LB), database (docker / native / external), licence (airgap / online).
3. Create DNS (or /etc/hosts for internal-only).
4. Follow §4.1 (Steps 2–6). For an internal-only name, use USCP_TLS=none behind an internal LB holding your corporate cert, or replace the Let's Encrypt directive in /etc/caddy/Caddyfile with tls /path/cert.pem /path/key.pem.

Snapshots are not backups — take logical Postgres backups (deploy/backup/pg-backup.sh). Keep the guest clock synced (chrony/VMware Tools) — token and licence validation are time-sensitive.

4.6 Kubernetes / Helm

Full detail in docs/install/kubernetes.md. The chart in codes/deploy/helm/uscp/ deploys HA defaults: 3 replicas, PDB, HPA (3→20 @ 65% CPU), hardened pod security context, Prometheus scrape annotations. The database is external (the chart does not deploy Postgres). Probes: readiness → /readyz, liveness → /livez.

Two presets:

Both have identical security, SSO+MFA, audit, and RLS posture — the single preset is not a stripped SKU.

Step 1 — Build and push the image

The chart's default image (ghcr.io/doublelogic/uscp/controlplane) is a placeholder — build from the repo Dockerfile (multi-stage Go build → distroless nonroot, EXPOSE 8080) and push to your registry:

cd uscp/codes
docker build -t <registry>/uscp/controlplane:v1 .
docker push <registry>/uscp/controlplane:v1

Step 2 — Provision the database

Managed Postgres 16 with a dedicated database and a non-superuser role (uscp_app) — critical, because USCP enforces FORCE ROW LEVEL SECURITY and a superuser bypasses it (§6.5). Reachable from the pod subnet, sslmode=require. Migrations run automatically at boot.

Step 3 — Create the uscp-secrets Secret

Every key becomes an env var in the pods.

Online licensing (clustered default):

kubectl create namespace uscp
kubectl -n uscp create secret generic uscp-secrets \
  --from-literal=DATABASE_URL='postgres://uscp_app:PASSWORD@db-host:5432/uscp?sslmode=require' \
  --from-literal=USCP_OIDC_ISSUER='https://accounts.google.com' \
  --from-literal=USCP_OIDC_CLIENT_ID='...' \
  --from-literal=USCP_OIDC_CLIENT_SECRET='...' \
  --from-literal=USCP_DATA_KEY="$(openssl rand -hex 32)" \
  --from-literal=USCP_INSTALL_KEY='<your-install-key>'

(Online mode needs outbound HTTPS from pods to licensing.doublelogic.org.) For air-gap on K8s, see the guide — you bake the bundle into the image or mount it from a Secret, and use values-single.yaml.

Step 4 — Install the chart

Clustered (HA):

helm install uscp deploy/helm/uscp -n uscp \
  --set image.repository=<registry>/uscp/controlplane \
  --set image.tag=v1 \
  --set env.USCP_REGION=us-east-1 \
  --set env.USCP_PUBLIC_URL=https://uscp.example.com

Single-instance:

helm install uscp deploy/helm/uscp -n uscp -f deploy/helm/uscp/values-single.yaml \
  --set image.repository=<registry>/uscp/controlplane --set image.tag=v1

Step 5 — Ingress, TLS, DNS

Zero-touch (with external-dns + cert-manager) — the chart creates the Ingress and both the DNS record and TLS cert are auto-provisioned:

helm upgrade uscp deploy/helm/uscp -n uscp --reuse-values \
  --set ingress.enabled=true \
  --set ingress.host=uscp.example.com \
  --set ingress.className=nginx \
  --set ingress.clusterIssuer=letsencrypt-prod \
  --set env.USCP_PUBLIC_URL=https://uscp.example.com

Otherwise bring your own Ingress (leave ingress.enabled=false) exposing the ClusterIP Service on port 8080. Either way, the OIDC redirect must be https://uscp.example.com/auth/callback and set env.USCP_PUBLIC_URL.

Step 6 — Verify

kubectl -n uscp get pods                       # all Running, READY 1/1
kubectl -n uscp rollout status deploy/uscp
kubectl -n uscp port-forward svc/uscp 8080:8080 &
curl -s localhost:8080/readyz
curl -s localhost:8080/.well-known/api-capabilities | jq .capability_count   # → 90
curl -fsS https://uscp.example.com/readyz

Key chart values (full table in the K8s guide): image.repository/tag, existingSecret (default uscp-secrets), env.*, resources (requests 500m/512Mi, limits 2/1536Mi), autoscaling (3→20 @ 65%), podDisruptionBudget.minAvailable: 2, ingress.*, backup.*, monitoring.prometheusRule/serviceMonitor/grafanaDashboard.

4.7 Air-gapped (no internet)

Full detail in docs/install/airgap.md. Same binary and feature set as online — air-gap is a licensing mode, not a stripped SKU. Zero vendor egress; the licence is a signed bundle imported via approved transfer.

Part 1 — On a connected build host

1. Build the static binary:

   git clone https://github.com/cfssay/doublelogic_apex uscp && cd uscp/codes
   make build-bin        # CGO_ENABLED=0 → ./dist/controlplane

1. Get or generate the licence bundle. If DoubleLogic supplied a signed entitlement.lic + jwks.json, use them. Otherwise self-sign (non-production) — and for a host-locked bundle, first read the target host's fingerprint inside the enclave:

   # INSIDE the enclave, on the target host:
   ./controlplane fingerprint          # → e.g. 4f3c…a19
   # On the connected build host, mint a bound bundle:
   go run -tags devtools ./cmd/devlicgen -out ./airgap-out -tenant <tenant-slug> -bind 4f3c…a19
   #   flags: -out (dir), -tenant (sub), -installation (aud), -bind (host fingerprint), -days (default 365)

1. (Recommended) sign the binary: cosign sign-blob --bundle controlplane.bundle controlplane
2. Stage the transfer bundle:

   mkdir uscp-airgap
   cp controlplane controlplane.bundle entitlement.lic jwks.json uscp-airgap/
   cp deploy/airgap/install.sh deploy/verify-deployment.sh uscp-airgap/
   tar czf uscp-airgap.tgz uscp-airgap && sha256sum uscp-airgap.tgz   # record the hash

Move uscp-airgap.tgz into the enclave via your approved path.

Part 2 — Inside the enclave

1. Prepare the host, PostgreSQL 16 (non-superuser role), and an internal OIDC provider (a ready-to-import Keycloak realm is at deploy/keycloak/realm-export.json).
2. Verify signatures and install:

   sha256sum -c <(echo "<recorded-hash>  uscp-airgap.tgz")
   tar xzf uscp-airgap.tgz && cd uscp-airgap
   DATABASE_URL='postgres://uscp_app:PASSWORD@db.internal:5432/uscp?sslmode=require' \
     ./install.sh

The deploy/airgap/install.sh script: verifies the cosign signature (aborts on failure), exports LICENSING_MODE=airgap + USCP_AIRGAP_BUNDLE=./entitlement.lic + USCP_LICENSE_JWKS=./jwks.json, runs controlplane migrate, then controlplane serve.

1. TLS with an internal CA — front with an internal LB holding a cert from your internal CA (X-Forwarded-Proto: https), or run Caddy with tls /etc/ssl/uscp.crt /etc/ssl/uscp.key.
2. Run under systemd (production) — create /etc/systemd/system/controlplane.service (template in the airgap guide), put all exported vars into /etc/uscp/controlplane.env (0600, owner uscp), then:

   sudo install -m0755 controlplane /usr/local/bin/controlplane
   sudo useradd --system --home /var/lib/uscp uscp 2>/dev/null || true
   sudo systemctl daemon-reload && sudo systemctl enable --now controlplane

1. Verify:

   curl -fsS https://uscp.internal/healthz
   curl -fsS https://uscp.internal/readyz
   curl -s http://localhost:8080/.well-known/api-capabilities | jq .capability_count

Refreshing the licence (default expiry 365 days): transfer in a new signed bundle, sudo cp entitlement.lic jwks.json /etc/uscp/, sudo systemctl restart controlplane. The import is audit-logged; the runtime never reaches the network for licensing. An anti-rollback monotonic time high-water mark prevents clock-turn-back from resurrecting an expired bundle.

4.8 Local pilot / dry-run

make pilot (codes/deploy/pilot-local.sh) is a self-contained dry-run of the ENTIRE deployment path on your machine, changing nothing on the host. It stands up a throwaway Postgres in Docker, builds the static binary, generates a host-bound self-signed licence, runs migrations, boots serve, runs the acceptance suite, an authenticated CRUD workflow, and an optional k6 latency probe — then tears everything down.

cd uscp/codes
make pilot
# or with a custom port:  PILOT_PORT=8123 ./deploy/pilot-local.sh

Requires only Docker + Go (+ k6 for the latency line). Expected final line:

✓ PILOT PASSED — the full deployment path (build → license → migrate → serve → acceptance) works end-to-end.

Use it to validate a build before a real deploy, or in CI. Exit code 0 = the full path works.

5. Complete environment-variable reference

Every configuration value comes from the environment (secrets injected via vault/KMS in production — nothing is read from a committed file). On a VM the installer writes these into /etc/uscp/controlplane.env; on Kubernetes you set them in the uscp-secrets Secret and env.* chart values.

Two distinct sets exist. Installer variables (deploy/install.sh) configure build + provisioning; the installer translates them into the runtime variables the binary reads. The tables below are the runtime variables (what the binary actually reads), which is what you set directly on Kubernetes/manual paths. Installer-only variables are called out at the end.

5.1 Workload segmentation — rollout control (§111)

The workload plane ships monitor-only. It observes, discovers applications, generates and validates allow-list policy and scores posture — but cannot change a firewall, a cloud security group or a NetworkPolicy until you say so. This is enforced at the API boundary, not by convention.

Refused in monitor mode (each with a message naming the variable): POST /v1/workloads/enforcement/apply, POST /v1/workloads/cloud/sync?apply=true, POST /v1/workloads/k8s/apply, POST /v1/workloads/intent/publish, and executing an isolate/block risk response. Compile and dry-run paths are unaffected — reviewing what enforcement would do is the point of this mode. Rollback is never gated: switching back to monitor must never strand a workload under rules you can no longer remove.

Before switching to enforce: set USCP_MANAGEMENT_CIDRS; run impact analysis against the published policy and read the violation list (those are the conversations enforcement will break); enforce on a canary ring; rehearse POST /v1/workloads/enforcement/rollback?target_id=….

The active mode is logged at boot (workload enforcement mode), returned by GET /v1/workloads/enforcement/state as platform_enforcement_mode, and shown as a banner in the console at /d/workloads.

5.2 Core / HTTP

5.3 Database

5.4 Licensing

5.5 TLS (in-binary termination)

Set these to have the binary terminate TLS itself (instead of fronting it with Caddy/LB). Floor is TLS 1.2.

5.6 Authentication & SSO

5.7 Data-at-rest encryption & KMS (BYOK/HYOK)

5.8 Rate limiting & observability

5.9 Multi-tenancy / bootstrap / integrations (advanced)

5.10 Installer-only variables (deploy/install.sh)

These are consumed by the installer, which translates them into the runtime variables above. You do not set these on Kubernetes.

6. Post-install configuration

Do these after any install. Full detail in docs/install/post-install.md.

6.1 TLS / DNS

• Public (Caddy auto-TLS): confirm https://<domain>/healthz answers. If the cert didn't issue, check that DNS resolves to this host and port 80 is reachable, then sudo systemctl restart caddy and watch journalctl -u caddy.
• Behind an LB: forward https://<domain> → http://<vm-ip>:8080, preserve Host, set X-Forwarded-Proto: https, set USCP_TRUST_PROXY=true and USCP_PUBLIC_URL=https://<domain> in /etc/uscp/controlplane.env, then sudo systemctl restart controlplane. Allow 8080 only from the LB.
• Internal CA: replace the ACME directive in /etc/caddy/Caddyfile with tls /path/cert.pem /path/key.pem, or set USCP_TLS_CERT / USCP_TLS_KEY on the binary.

6.2 First admin login

1. Browse to https://<your-domain>/auth/login.
2. Sign in with your configured method:
• Email one-time code (default): enter your admin email → a 6-digit code is emailed → enter it.
• Authenticator (TOTP): enter the username (default admin) + the 6-digit code from the otpauth://… secret the installer printed.
• SSO: click Continue with SSO.
3. You land as an admin already — no SQL, no manual role grant. The USCP_ADMIN_EMAIL / USCP_BOOTSTRAP_ADMIN you set is granted admin on first login.
4. Enrol a passkey / MFA immediately.
5. Provision break-glass emergency codes now, while SSO works (you cannot create them once locked out):

   curl -fsS -X POST https://<your-domain>/admin/break-glass/setup \
     -H "Authorization: Bearer $YOUR_SESSION_TOKEN" -H 'Content-Type: application/json' \
     -d '{"threshold":2,"ttl_minutes":60}'

Returns 10 plaintext recovery codes once — split them among trusted custodians.

If you installed without an admin email (or the first admin's SSO email differs), add USCP_BOOTSTRAP_ADMIN=<email> to the env file and restart, or grant admin directly in the DB — see the post-install guide's SQL fallback.

6.3 Connect SSO (per-IdP)

The redirect URI is always https://<your-domain>/auth/callback. Register the app at your IdP (Google / Entra ID / Okta / Keycloak — step-by-step in post-install.md) and set USCP_OIDC_ISSUER / _CLIENT_ID / _CLIENT_SECRET. SAML providers are registered in-app under Access control. If login fails with invalid_redirect_uri, the registered URI does not exactly match (scheme, host, path, trailing slash all matter).

6.4 SMTP (email one-time-code login)

Set USCP_SMTP_HOST (host:port), USCP_SMTP_FROM, and (if authenticated) USCP_SMTP_USERNAME / USCP_SMTP_PASSWORD. The installer verifies SMTP before enabling email login. To change methods later, use Access control → Login methods (at least one method must stay enabled).

6.5 Database hardening (RLS) — the single most important production step

USCP isolates tenants with PostgreSQL FORCE ROW LEVEL SECURITY. A PostgreSQL superuser bypasses RLS entirely — so if DATABASE_URL points at a superuser (RDS master, Azure server admin, Cloud SQL postgres, or the default Docker postgres), per-tenant isolation is silently not enforced. Create a dedicated non-superuser role:

-- as the DB owner/admin, once:
CREATE ROLE uscp_app LOGIN PASSWORD 'STRONG_PASSWORD' NOSUPERUSER NOCREATEDB NOCREATEROLE;
CREATE DATABASE uscp OWNER uscp_app;

Then set DATABASE_URL=postgres://uscp_app:STRONG_PASSWORD@host:5432/uscp?sslmode=require. Verify isolation: connect as uscp_app, SET app.current_tenant to one tenant's id, and confirm you cannot read another tenant's rows. If you can, you are still a superuser — fix the role. Use sslmode=require (or verify-full) and restrict 5432 to the app host/subnet.

6.6 Data-at-rest key & licensing activation

• Data key: confirm USCP_DATA_KEY is set (the VM installer generates and preserves it). Never regenerate it on an install that already holds data. Rotate via USCP_DATA_KEYS_RETIRED.
• Licensing: confirm GET /v1/licensing/status shows state: active, the expected modules, and a sensible not_after. For single-deployment installs the licence is bound to this host — read the fingerprint with controlplane fingerprint and request a bound bundle. To renew air-gapped, replace entitlement.lic (+ jwks.json if the key rolled) and restart.

6.7 Production hardening checklist

• [ ] PostgreSQL with TLS + a non-superuser role (§6.5).
• [ ] All secrets from vault/KMS/Secrets — on VMs only in /etc/uscp/controlplane.env.
• [ ] Only 443 public; 8080/5432 never public; USCP_PUBLIC_URL is the https URL.
• [ ] SSO + passkey MFA enrolled for every human.
• [ ] cosign verify passed on the binary/image before running it.
• [ ] Backups scheduled and a restore drill rehearsed (§8).
• [ ] Audit-chain verification monitored: GET /v1/_audit/verify (expect chain_valid: true).
• [ ] Break-glass recovery codes provisioned and custodied.
• [ ] USCP_REQUIRE_HOST_BINDING=true for single-deployment installs.
• [ ] Time sync (chrony/NTP) active.

7. Verification & acceptance

7.1 Health endpoints (any deployment)

curl -fsS https://<domain>/healthz     # {"status":"ok"}
curl -fsS https://<domain>/readyz      # {"status":"ready"}
curl -s   https://<domain>/.well-known/api-capabilities | jq .capability_count   # 90

A /readyz that is non-200 while /healthz is 200 means the binary is up but a dependency (DB or licence) is not ready — check logs (journalctl -u controlplane on a VM, kubectl logs on K8s).

7.2 Automated acceptance suite (run after every deploy)

deploy/verify-deployment.sh https://<your-host>     # or http://127.0.0.1:8080 locally

It checks liveness/readiness, Prometheus metrics, licence active & not expired, the OpenAPI contract, capability discovery, the embedded console is served, the auth surface responds, and that every licensed route rejects anonymous callers (401/403). It must print DEPLOYMENT ACCEPTED (0 failures). The one-command installer runs this for you automatically. Resolve any FAIL; review every WARN (e.g. "configure SMTP", "front with TLS") before go-live.

7.3 Deeper verification gates (from a source checkout)

Run these from codes/ to validate a build end-to-end. Expected pass results in parentheses.

make uat / make smoke-auth require the dev stack running (make dev); make uat-full brings it up itself.

7.4 Acceptance sign-off checklist

• [ ] verify-deployment.sh passes over HTTPS (0 failures).
• [ ] Admin login works end-to-end via your chosen method; MFA enrolled.
• [ ] /v1/licensing/status → state: active, expected modules, sensible not_after (host-bound in prod).
• [ ] Database runs as a non-superuser; RLS isolation verified.
• [ ] Backups configured and a restore drill rehearsed.
• [ ] /metrics scraped by Prometheus; alert rules loaded (Helm monitoring.prometheusRule.enabled=true).
• [ ] One real workflow exercised for your use case.
• [ ] TLS certificate valid and auto-renewing; HTTP→HTTPS redirect in place.

The go-live runbook (docs/install/go-live-runbook.md) is the phased, gated path through all of the above.

8. Upgrades, backup/restore, and rollback

8.1 Migrations

Migrations are applied automatically at boot (the systemd unit runs controlplane migrate before serve; on K8s the startup probe allows ~150s for first-boot migrations). They follow expand-contract (forward + backward compatible for one release), so a one-version rollback is safe. Always back up before upgrading. You can also run migrations manually:

DATABASE_URL='postgres://…' controlplane migrate      # or: make migrate

8.2 Backup (all state is in PostgreSQL — the binary is stateless)

DATABASE_URL='postgres://uscp_app:...@host:5432/uscp?sslmode=require' \
BACKUP_DIR=/var/backups/uscp \
BACKUP_RETENTION_DAYS=14 \
BACKUP_S3_URI='s3://my-bucket/uscp' \        # optional; needs awscli
./deploy/backup/pg-backup.sh

pg-backup.sh runs pg_dump -Fc (custom format), writes a .sha256 integrity marker, prunes beyond retention, and optionally uploads to S3. Schedule it nightly via cron/systemd-timer. On Kubernetes, enable the built-in CronJob:

helm upgrade uscp deploy/helm/uscp -n uscp --reuse-values \
  --set backup.enabled=true --set backup.schedule='0 2 * * *' \
  --set backup.retentionDays=14 --set backup.s3Uri='s3://my-bucket/uscp-backups'

8.3 Restore (DR drill)

DATABASE_URL='postgres://uscp_app:...@host:5432/uscp?sslmode=require' \
  ./deploy/backup/pg-restore.sh /var/backups/uscp/uscp-<stamp>.dump

pg-restore.sh verifies the .sha256 marker, then restores (--clean --if-exists). After restore, re-verify hash chains: GET /v1/_audit/verify and GET /v1/sla/reports (expect chain_valid: true). RPO = last backup.

8.4 Upgrade & rollback matrix

Never regenerate USCP_DATA_KEY on an existing install with data — sealed secrets would become undecryptable. Re-running install.sh preserves it.

9. Troubleshooting

Logs:

# VM
sudo journalctl -u controlplane -n 200 --no-pager
sudo journalctl -u caddy -n 100 --no-pager
# Kubernetes
kubectl -n uscp logs deploy/uscp --tail=200

Open a psql shell (Docker DB mode):

sudo docker exec -it uscp-pg psql -U uscp -d uscp        # (native/external: psql "$DATABASE_URL")

10. Where to go next

• Go-Live Runbook — the phased, gated path from package to production sign-off. Follow it for every production deployment.
• Post-install guide — first login, per-IdP OIDC setup, break-glass, database hardening, backups, upgrades, troubleshooting.
• Per-environment install guides — AWS · Azure · GCP · On-prem VM · Kubernetes · Air-gap · Install index.
• Validation evidence — what has been verified about the shipped package and how to reproduce it.
• Administrator Guide & User Guide — siblings in docs/ for day-2 operations and end-user workflows (see the console's in-app help under Access control and Licensing for feature-level guidance).

You're done: the platform is installed, hardened, backed up, recoverable, and verified.