Prisma Cloud: production upgrade and recovery
A production security-platform upgrade, a recurring connection fault, and six Defenders brought back online. The work turned a fragile upgrade procedure into a sequence of explicit acceptance checks.
Kelmavory · Production change ownership · One Kubernetes clusterThe security platform needed maintenance of its own. Vulnerability findings in Prisma Cloud Compute (formerly Twistlock) Console and Defender images prompted an upgrade on a six-node Mirantis Kubernetes Engine cluster. Production used filtered, allowlisted egress, and the original Helm chart was unavailable. A previous connection-setting problem had also recurred during upgrades. Success required a reviewed deployment, reachable images, and Defenders visibly connected to the upgraded Console.
I owned the production change from preparation through post-upgrade checks: identified the active Helm deployment, regenerated the charts, reviewed the rendered changes, tested image access, upgraded the Console and Defender DaemonSet, diagnosed the connection failure, and rewrote the operating procedure around the findings.
Open each stage for the action and its supporting result. This is a conceptual sequence; internal topology and procedures are omitted.
Inspect the live deployment before choosing the upgrade path. Confirm the target-version twistcli executable and use the existing Console configuration to reconstruct the missing chart.
What was established
Production was Helm-managed. The checks avoided assuming that the Argo CD approach used elsewhere also controlled this environment.
Render and compare the generated manifests. The old empty-configuration instruction produced an invalid service port and duplicate port name; regenerate from the live configuration and review again.
What was established
The manifest review caught the malformed chart before deployment. The corrected configuration became the basis of the change.
Use a disposable pull-test pod to check the target image and existing pull credentials through the restricted network path. Upgrade the Console first, then check its running image and UI before moving to Defenders.
What was established
The production node could pull the target image, and the upgraded Console was checked before the Defender rollout.
The Defender pods were Running but absent from the connected-agent view. Inspect the connection logs and generated destination: the WebSocket target used the external Console address where the intended path used the in-cluster service.
What was established
Correcting the generated connection address and redeploying restored all six Defender connections on the target version. The prepared rollback was not used.
Rewrite the SOP to cover controller detection, binary checks, live configuration, manifest comparison, image-pull testing, connection-address review, fallback planning, and Console-side completion checks.
What was established
The revised procedure records the checks missing from the earlier workflow. Internal image mirroring was identified as a future improvement, not part of the completed rollout.
Use the live configuration as the starting point, then inspect the generated manifest before applying it.
The tradeoffRegeneration still requires careful comparison. A chart that renders successfully is only one checkpoint; it does not establish working runtime connections.
Check image delivery on a production node before starting the rollout, using the existing approved egress path.
The tradeoffThe extra preflight adds work, but separates registry reachability and credentials from problems introduced during the upgrade.
Require connected Defenders on the target version, alongside Kubernetes deployment checks.
The tradeoffThis exposed a connection fault after pods were already Running. Restored connectivity proves that operational outcome; vulnerability closure requires separate scan evidence.
Upgraded the Prisma Cloud Compute Console and Defender fleet in one production MKE cluster.
Restored all six Defender connections on the target version after correcting the generated WebSocket destination.
Completed the production upgrade without using the prepared rollback.
Caught malformed chart output before deployment and incorporated the findings into a revised upgrade SOP.
“Running” is a Kubernetes state. “Connected” is a separate security-platform check. A production upgrade needs both, with the generated connection settings reviewed as carefully as the version being installed.
Based on the owner-provided production upgrade account reviewed on 13 September 2026. The six-Defender outcome, fault diagnosis, and SOP changes are reported in that account. Private manifests, logs, addresses, and screenshots are not reproduced. This is a separate production change from the earlier non-production recovery case.
This repository demonstrates related engineering skills. It contains personal project work, not employer code or a reproduction of the private environment.
GitOps Supply Chain