Skip to content

Management agent: create the product's Flux objects, so its chart installs with helm --wait on Helm 4 - #1893

Open
davidmckayv wants to merge 1 commit into
mainfrom
mgmt-agent-owns-flux-objects
Open

davidmckayv wants to merge 1 commit into
mainfrom
mgmt-agent-owns-flux-objects

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

Why

helm install --wait of the openbot-agent chart never finished on Helm 4, and neither did helm upgrade --wait. Helm 4's --wait uses kstatus and waits for every object in the release to be current. The chart rendered the product's OCIRepository and HelmRelease itself, so the agent chart's readiness was really the product's:

  • Before the management service names a version, both objects are suspended on a placeholder digest. source-controller never reconciles a suspended OCIRepository, and it only records an observed generation once the object has a condition, so the OCIRepository keeps the observedGeneration: -1 its CRD defaults to. kstatus reads that as InProgress forever.
  • Once the product is installed, a failed install or upgrade (remediation is off) leaves the HelmRelease Stalled. kstatus reads that as Failed, so every helm upgrade --wait of the agent chart failed at once. That includes the pause (--set paused=true): Helm marked the upgrade failed and never ran the post-upgrade hook that suspends the HelmRelease, so the product could not be paused at the moment it most needed to be. Reproduced on kind with Helm 4.3.2.

Helm 3's --wait ignores custom resources, so only Helm 4 showed it. Helm 4 is what customers and Microsoft's Flux extension run.

What changed

The agent now creates and keeps the product's two Flux objects. The chart no longer renders them, so helm --wait only waits for the agent and its controllers.

  • config.ts: the chart hands the agent the objects' settings: product.source (URL, first pin or none, insecure, pull secret, signature check) and product.release (deployer service account, timeout, valuesFrom in the chart's order).
  • flux.ts: productObjects() builds both objects from that config. ensure() runs at the start of every heartbeat. It creates a missing object (suspended on the placeholder until a version is named, unless the chart gave a first pin) and merge-patches an existing one to the chart's settings. It never touches an existing object's spec.ref.digest or spec.suspend, which only move by a pin, a pause or a person. It also removes Helm's release annotations that an older chart left behind.
  • The HelmRelease keeps what the off path depends on: client-side apply, no remediation, no drift correction, and release records stored in the product namespace.
  • agent.ts: a failure here is logged, and the heartbeat still goes out.

The chart change that goes with this (configuration, RBAC to create the objects, a pre-upgrade handover that keeps objects an older chart rendered instead of letting Helm delete them, and the regression test) follows in CopilotKit's management chart once this is merged.

Tests

  • A real helm install --wait --timeout 10m and helm upgrade --wait of the agent chart on kind, with Helm 4.3.2 and Helm 3.22.1 and this agent image. It covers: the agent creating both objects suspended on the placeholder; an upgrade while they wait; a pin to a product build that fails (HelmRelease Stalled); a pause with --wait that succeeds and suspends; a resume with a changed product timeout that the agent applies without moving the pin; and an uninstall through the disconnect hook. It fails on the current chart and agent and passes with this change.
  • The chart's render tests check the objects with this module's productObjects() against the rendered config. The existing agent upgrade and off-path tests run on kind against this change.

@davidmckayv
davidmckayv force-pushed the mgmt-agent-owns-flux-objects branch from e12cde9 to c0fbcd8 Compare November 19, 2026 21:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant