GovStack Bootstrap

Bootstrap and Management-GitOps foundation for on-demand CAPI-on-OpenStack Spokes using CAPO.

This repository is intentionally separate from the deployed GovStack workspace. It does not target or modify an existing cluster by default.

Repository	https://git.vanillacore.net/govstack/govstack-bootstrap (git clone https://git.vanillacore.net/govstack/govstack-bootstrap.git)
Dashboard	https://git.vanillacore.net/govstack-apps/govstack-dashboard
Spoke lifecycle guide	docs/spoke-lifecycle.md
Cluster bridge operator	operator/cluster-bridge/README.md
Playbooks	playbooks/README.md
Architecture	docs/architecture.md

Single source of truth: everything, including the Spoke declarations in gitops/spokes/<name>/, lives on main. There is no separate Spoke branch. Argo CD follows main for Spokes (gitops.spoke_revision: main); the Management manifests are pinned to an exact main commit (gitops.management_revision, applied by playbook 61).

Quick start: day-2 operations

Task	How
List Spokes	kubectl -n govstack-clusters get gss or the dashboard card "Spokes (GovStackSpoke)"
Scale workers / control plane	Dashboard +/−, or scripts/spoke-lifecycle.py scale --spoke gitops/spokes/<name> --workers N, then commit to main
Replace Machines	Dashboard menu, or scripts/spoke-lifecycle.py replace --spoke gitops/spokes/<name> --pool workers
New Spoke	scripts/render-spoke-manifests.py, then commit gitops/spokes/<name>/ to main
Delete a Spoke	Remove the directory (PR), scripts/delete-spoke-resources.sh --confirm <name>, playbook 95
Roll out a Management change	Merge the PR, set gitops.management_revision to the merge SHA, run playbook 61 (and 62 for the dashboard)

Minimum target

1. One OpenTofu-created public gateway VM, configured by Ansible for floating-IP termination, routing/NAT, and WireGuard.
2. One OpenTofu-created single-node Kubernetes Management seed, configured by Ansible behind the gateway.
3. CAPI, kubeadm providers, CAPO, and required CAPO dependencies on the seed.
4. Argo CD on the Management seed as the declarative delivery layer.
5. A LoadBalancer Kubernetes cluster created and lifecycle-managed by CAPO.
6. Foundational services on every CAPO cluster: Calico, kube-vip, cert-manager, Contour/Envoy ingress, Longhorn storage, monitoring, and cloud object-storage integration.
7. A digest-pinned GovStack HTTP/SNI Envoy routing stack on the LoadBalancer cluster, reconciled by Argo CD from an exact Git revision.
8. Additional Kubernetes clusters created by the same CAPO cluster playbook and converged through the same foundation workflow.

The transitional seed bootstrap still uses Ansible for host preparation and initial Management control-plane installation. After that boundary, the intended operator interface is Git: CAPO owns child Kubernetes machines and lifecycle, while Argo CD discovers and reconciles on-demand Spokes and their foundations.

On-demand Spoke workflow

A Spoke is not a fixed inventory entry. Add a declarative cluster input and render its non-secret CAPI and foundation resources into gitops/spokes/<name>/:

cp config/clusters/spoke.example.yml config/clusters/<name>.yml
# Replace every placeholder, then render from immutable state:
python3 scripts/render-spoke-manifests.py \
  --cluster config/clusters/<name>.yml \
  --environment config/environments/<environment>.yml \
  --infrastructure-outputs state/<environment>/infrastructure-outputs.json \
  --image state/<environment>/capo-image.json \
  --force

Usage guide: docs/spoke-lifecycle.md (create, scale, replace, status, delete, troubleshooting).

Node lifecycle is CAPI-only and GitOps-driven; no tool deletes Nodes or VMs directly. Each Spoke has a GovStackSpoke (govstack.itu.int/v1alpha1, kubectl -n govstack-clusters get gss) committed in gitops/spokes/<name>/cluster.yaml: its spec is the lifecycle API (replica counts, rolloutAfter), and the cluster bridge applies it to the CAPI Cluster and reports phase, replicas, foundations and conditions in its status. Re-rendering keeps the committed lifecycle (--reset-lifecycle <current workers> destructively adopts the declaration instead). After a commit Argo CD syncs the GovStackSpoke, the bridge applies it and CAPI acts:

• scripts/spoke-lifecycle.py scale --spoke gitops/spokes/<name> --workers N adds or removes workers (scale-down drains and deletes the oldest Machines first); --control-plane 1|3|5 changes the KubeadmControlPlane size one step per commit (1<->3, 3<->5).
• scripts/spoke-lifecycle.py replace --spoke gitops/spokes/<name> --pool workers|control-plane|all sets topology rollout.after, so CAPI replaces every Machine one at a time (new VM, then drain and delete the old one). This is also the recovery for Nodes with obsolete provider IDs.
• scripts/spoke-lifecycle.py upgrade --cluster ... --kubernetes-version vX.Y.Z --image ... rolls to a new release.

With gitops.spoke_revision: main (instead of a pinned SHA) the Spoke ApplicationSets follow it and a lifecycle commit applies without re-running playbook 61. scale and replace need no environment, state or credentials beyond Git write access, so the Dashboard, API and agents perform the identical operation: edit the GovStackSpoke in cluster.yaml, commit, push.

Install playbooks/61-bootstrap-spoke-management.yml once on the Management seed. It installs the CAPI-to-Argo bridge and separate Git directory ApplicationSets for CAPI resources, foundations, and bridge RBAC. Argo then discovers each gitops/spokes/* directory, applies the narrowly scoped resources, registers provisioned children, and removes stale registrations. Adding a Spoke directory creates or reconciles it; removing a Spoke directory alone is intentionally not a deletion request because the ApplicationSet preserves CAPI resources. Use the reviewed explicit teardown workflow in gitops/spokes/README.md and scripts/delete-spoke-resources.sh.

Each generated Spoke may declare a fixed gateway address and WireGuard contract. CAPO creates the tainted gateway MachineDeployment, Kubeadm installs WireGuard with a node-local generated private key, and foundation reconciliation publishes the declared external UDP port through the shared gateway. OpenStack network/subnet and credential references remain environment/state inputs; no backend-specific values or secrets are written to the Spoke directory.

Existing automation dependency

upstream/rollout/ pins the existing govstack-openstack-rollout repository. Bootstrap implementation must extract or reuse its existing OpenStack, gateway, WireGuard, kubeadm, state, and CAPI/CAPO tasks rather than creating a parallel framework.

Initialize only the top-level dependency:

git submodule update --init

Do not initialize submodules recursively.

Planned operator flow

ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/00-test-environment-access.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/01-preflight-readonly.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/20-bootstrap-infrastructure.yml -e bootstrap_apply=true
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/30-bootstrap-gateway.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/40-bootstrap-management-seed.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/50-bootstrap-capi-capo.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/15-ciab-service-endpoint-bridge.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/16-hypervisor-capacity.yml  # optional single-host capacity guard
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/17-lab-flavors.yml          # optional right-sized lab flavors (after 20)
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/60-bootstrap-argocd.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/61-bootstrap-spoke-management.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/70-build-capo-image.yml
# Transitional CIAB verification path only:
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/80-deploy-loadbalancer-cluster.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/90-reconcile-loadbalancer-foundations.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/91-configure-object-storage.yml
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/62-deploy-dashboard-management.yml  # optional dashboard
# Spoke retirement, after scripts/delete-spoke-resources.sh:
ansible-playbook -i inventories/oslab-ciab/hosts.yml playbooks/95-revoke-spoke-identity.yml -e cluster_config_file_override=$PWD/config/clusters/<name>.yml

The component entry points will be implemented incrementally. They must be independently rerunnable and fail clearly when a prerequisite component is absent.

Configuration

make config-init
# Then review the ignored config/site.yml, config/environments/oslab-ciab.yml,
# and config/clusters/loadbalancer.yml files before running components.

Never commit OpenStack credentials, WireGuard private keys, kubeconfigs, generated state, or rendered cloud-init containing secrets.

Development checks

make init
make config-init
make check

Checks are local/static only. They do not contact OpenStack or Kubernetes.