Spinnaker – Install and Configure Spinnaker

Docs: Choose your Environment

Mon, 01 Jan 0001 00:00:00 +0000

In this step, you choose where to install Spinnaker. The recommended path is an installation into a Kubernetes cluster

Kubernetes installation

Each of Spinnaker’s microservices services are deployed separately. This is highly recommended for use in production.
Local installations of Debian packages

You can also install on a single local machine, or for Spinnaker development, making sure you have the 4 cores and 16GB in each case.

Spinnaker is deployed on a single machine. This is ok for smaller Spinnaker deployments, but Spinnaker will be unavailable when it’s being updated. Note it’s often better to use a microk8s/k3s installation to deploy rather than try to deploy a local debian.
Please see Local git installations from GitHub

This is for developers contributing to the Spinnaker project.

Kubernetes installation

Spinnaker runs in a dedicated Kubernetes namespace as a set of independently deployed microservices.

See the microservice reference architecture .

The base example is available in the monorepo here: https://github.com/spinnaker/spinnaker/tree/main/spinnaker-kustomize with more information and options.

To install spinnaker:

Make sure kubectl is installed
```
kubectl version --client
```
set a working directory WORKING_DIR="$HOME/workspace-install-spinnaker/"
```
mkdir -pv "$HOME/workspace-spinnaker-install/"
```

clone the base example github repo

git clone https://github.com/spinnaker/spinnaker.git $HOME/workspace-spinnaker-install/spinnaker

Optional: configure Kubernetes probes for your Spinnaker services in their deployment manifests files. located inside base/*/deployment.yaml/
```
ls -lha $HOME/workspace-spinnaker-install/spinnaker/spinnaker-kustomize/base/*/
```
switch to the kustomize directory in the spinnaker base repo
```
 pushd $WORKING_DIR/spinnaker/spinnaker-kustomize
```
Adjust the DNS domains. Look for any example.com reference and replace with your DNS domain.
Create a file with the spinnaker kubernetes resources and apply it
```
kubectl kustomize --output="spinnaker.yaml"
```

Apply the YAML file

kubectl apply --filename="spinnaker.yaml"

Get the ingress IP address and create DNS entries for your new spinnaker domain to this new entry
```
kubectl get ingress --namespace spinnaker
```

REMINDER: This basic setup defaults with a SIMPLE username/password auth. It’s recommended to change this from the default or better yet, use an identity provider (saml/oidc/ldap) solution. The spinnaker project does integration tests today with Keycloak as a known out of the box solution.

Local Debian

Prerequisites

We recommend at least 4 cores and 16GB of RAM.

Overview

The Local Debian installation means Spinnaker will be installed and run on a single machine. We recommend at least 4 cores and 16GB of RAM. Note this is a non-recommended installation but is simple to install. Spinnaker configuration files default to /opt/spinnaker/config. You can copy the files from the spinnaker-kustomize example repo for each service. Services load properties from in the following priority order

spinnaker.yml
spinnaker-local.yml
<service>.yml
<service>-local.yml

It’s recommended NOT to change the spinnaker.yml or service.yml and instead override their settings in the service local yml files. These base files often are “defaults” or core settings. Base settings can be found in the source code for each service similar to https://github.com/spinnaker/spinnaker/blob/main/front50/front50-web/config/front50.yml path where each service has a <service>/<service>-web/config/<service>.yml file defining defaults.

Note: Local Debian installation requires a recent ubuntu/debian and installation of components like kubectl or the aws cli depending upon deployment targets. They also need a Java 17 JRE installed.

Intended use case

The Local Debian installation is intended for smaller deployments of Spinnaker, however, since all services are on a single machine, there is potential downtime.

Install local dependencies

Ensure that the following are installed on your system:

git: sudo apt-get install git
curl: sudo apt-get install curl
redis-server: sudo apt-get install redis-server
JDK 17 - JDK (we’re building from source, so a JRE is not sufficient)
npm/node (version >=16)
yarn: npm install -g yarn or guide

Next steps

Congratulations, you have a basic Spinnaker installation. You can

Integrate with Cloud Providers
check the deploy Spinnaker references

Docs: Deploy Spinnaker and Connect

Mon, 01 Jan 0001 00:00:00 +0000

Now that we’ve enabled one or more Cloud Providers , picked a Deployment Environment , and configured Persistent Storage , we’re ready to pick a version of Spinnaker, deploy it, and connect to it.

Pick a version

See the available versions on the versions page

You can follow the links to the versions’ respective changelogs to see what features each adds.
Set the version you want to use: Set this on the kustomize image tag configuration . All spinnaker versions since the monorepo use the same image tags, e.g. 2025.3.2.

Deploy Spinnaker

kubectl kustomize -o ./spinnaker.yaml
kubectl apply -f ./spinnaker.yaml

Ingress

A default ingress is included that uses a single ingress with path based routes to gate’s API on a prefix. This bypasses CORS issues and enables a simple user experience to connect to spinnaker. The default username and passwords are also set in this installation.

For ingress specific to your environment, please adjust the default ingress as needed for your use case.

Connect to the Spinnaker UI

Once deployed, get the address for your ingress controller and connect to the ingress. Make sure you have a domain address pointing to your new endpoint. Example:

kubectl get ingress --namespace spinnaker

NAME                CLASS   HOSTS                     ADDRESS          PORTS     AGE
spinnaker-ingress   nginx   spinnaker.example.com     192.168.1.2      80, 443   644d

This ingress would include 443 or TLS as configured. It’s recommended to enable a cert provider and use HTTPS for requests but this is out of the scope of the installation. See the kubernetes networking documentation on TLS for information on modifying the ingress for certificate handling.

Once you’ve configured DNS to point to your address, go to http://spinnaker.example.com/ and login with the username and password that you’ve configured (or your SSO provider if so configured per authentication configuration

Troubleshooting

First, check that services are running. IF a service isn’t running, check the logs for configuration errors.

Next once all services are running, make sure you can talk to the gate endpoint. You should be able to check the health endpoint even externally via the gate-api/health endpoint.

Upgrade Spinnaker

To upgrade, read the release notes and change the images to the newer versions. Adjust any config that’s been removed/changed as needed.

Next steps

Now that Spinnaker is deployed and capable managing your cloud provider, you can…

Continue with additional configuration, such as your image bakery
If you’re a Spinnaker end user, read how to get started using Spinnaker
Visit the Guides pages to learn more

Remember to keep your configuration in git for both history and changes!

Docs: Cloud Providers Overview

Mon, 01 Jan 0001 00:00:00 +0000

In this section, you’ll register credentials for your Cloud platforms. Those credentials are known as Accounts in Spinnaker, and Spinnaker deploys your applications via those accounts.

Supported providers

All of Spinnaker’s abstractions and capabilities are built on top of the Cloud Providers that it supports. So, for Spinnaker to do anything you must enable at least one provider, with one Account added for it. These are all managed in clouddriver’s configuration files.

Add as many of the following providers as you need. When you’re done, return to this page.

Next steps

When you’ve finished setting up your cloud provider, you’re ready to choose an environment .

Docs: Storage

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker requires a database for persisting your Application settings and configured Pipelines as well as execution history. A database is used for a few services (clouddriver, front50, orca). Because this data can be sensitive and can be costly to lose, we recommend you use a hosted storage solution you are confident in.

Supported databases

Spinnaker recommends MySQL/MariaDB or Postgresql. It’s recommended for most users to start with MariaDB/MySQL.

Services ship with mysql-connector-java by default. You can provide additional JDBC connectors on the classpath if desired. MySQL/MariaDB is reported to work all the way up to version 10 of MariaDB and the latest Aurora MySQL engines.

Before you deploy services, you need to manually create a database and user grants for each service. These are NOT automatically created when spinnaker services start. Note that the kustomize example on the install DOES create these as part of the mariadb init script .

Repeat these creation steps for echo, orca, clouddriver, fiat and front50 (and keel if using keel) replacing the database for each service

CREATE DATABASE `front50` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW
ON `front50`.*
TO 'front50_service'@'%' IDENTIFIED BY "password";

GRANT
  SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
ON `front50`.*
TO 'front50_migrate'@'%' IDENTIFIED BY "password";

Configure the services

Configure the services (when not using the native mariadb component in kustomize) to use the SQL configuration passed above. AN example for mariadb:

sql:
  enabled: true
  connectionPools:
    default:
      default: true
      # additional connection pool parameters are available here,
      # for more detail and to view defaults, see:
      # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
      jdbcUrl: jdbc:mysql://mariadb:3306/fiat
      user: fiat_service
      password: ${FIAT_MARIADB}
  migration:
    jdbcUrl: jdbc:mysql://mariadb:3306/fiat
    user: fiat_migrate
    password: ${FIAT_MARIADB}

The default spinnaker-kustomize installation uses a component to auto inject these configurations into the targeted services.

Next steps

Now that your deployment environment is set up, , you can…

Configure everything else

Docs: Configuration

Mon, 01 Jan 0001 00:00:00 +0000

There are a number of other things you can configure in your Spinnaker installation enumerated below. At any point in time, you can apply your configuration by deploying Spinnaker services:

kubectl kustomize -o spinnaker.yml
kubectl apply -f spinnaker.yml

Deploy to more cloud accounts

Documentation

You can add as many target accounts as you want. There is nothing preventing you from deploying to two Kubernetes clusters, one Google Compute Engine project, and an App Engine application all at once.

Secure your Spinnaker installation

Documentation

You can configure SSL, setup a login page, or apply role-based authorization.~~

Setup continuous integration

Documentation

Configure Jenkins or Travis CI to trigger Pipelines or supply Spinnaker with build artifacts to build into images and deploy.

Configure notifications

Documentation

Enable notifications to be sent on Spinnaker events, and allow external events to trigger Pipelines in Spinnaker.

Monitor your Spinnaker installation

Documentation

Publish timeseries data from your Spinnaker installation to a variety of metric sources into curated dashboards. This is useful for understanding how to scale and troubleshoot your deployment of Spinnaker.

Next steps

Now that you know how to make configuration changes, it’s worth learning how to Upgrade Spinnaker .

Docs: Setup FAQ

Mon, 01 Jan 0001 00:00:00 +0000

CAUTION

Halyard is deprecated and no longer supported. MANY FAQs here around halyard no longer work or are no longer relevant. See a Spinnaker Kustomize based installation for a current deployment of spinnaker. Halyard FAQs and information will be removed on an ongoing basis

I can’t load the Applications screen

After installing Spinnaker and navigating to the Applications screen, you may see one of following issues:

The following error message is displayed:

Error fetching applications. Check that your gate endpoint is accessible. Further information on troubleshooting this error is available here

The most common cause of this error is that your browser can’t communicate with your Gate endpoint. (This endpoint defaults to http://localhost:8084, but can be customized.)

Check your browser console log and/or network for any failed requests to <gate-endpoint>/applications.

Some things to check while troubleshooting:

If you are accessing Spinnaker via the default http://localhost:9000, check that you have forwarded Gate’s port (8084 by default) to the machine where your browser is running.
If you are accessing Spinnaker via a custom URL, ensure that you have set override-base-url for both the UI (Deck) and API (Gate) services, as described in the question below . These settings will configure cross-origin resource sharing (CORS) between your Gate and Deck endpoints; if this is not properly configured, your browser will reject requests from Deck to Gate.
If you have a local deployment of Spinnaker, ensure that Redis is available at the configured address (localhost:6379 by default). If not, start redis by running sudo systemctl enable redis-server && sudo systemctl start redis-server and restart spinnaker by running sudo systemctl restart spinnaker.

I want to expose LocalDebian Spinnaker on a public IP address, but it always binds to localhost

First off, on a local deployment Spinnaker binds to localhost intentionally. Your Spinnaker instance has the ability to deploy and destroy a lot of infrastructure in whatever accounts it manages, and opening it to the public is not a good idea without authentication enabled. With that in mind, there are two solutions.

Once you enable an authentication mechanism, Spinnaker will bind the UI and API servers to 0.0.0.0 automatically. This is configurable if you prefer to bind a specify address instead. Regardless, you still need to set the API & UI baseUrls so CORS and login redirects happen correctly, this is done with the commands hal config security ui edit --override-base-url <full ui url> and hal config security api edit --override-base-url <full api url>.
If you don’t want to rely on authentication, you can follow this guide . This makes sense if you’re running Spinnaker in a private network, or have another form of authentication fronting Spinnaker.

Halyard times out during a config change

Odds are Halyard can’t connect to the configuration & version bucket (in Google Cloud Storage) it uses to determine if the configuration you’ve provided works for the version of Spinnaker you want to install. The bucket is gs://halconfig, see if you can reach it locally using the <code>gsutil</code> CLI. The remediation will depend on your local network. You can also always omit validation with the --no-validate flag.

As an alternative to gsutil, you can try querying the bucket directly using its fully-qualified URI: curl storage.googleapis.com/halconfig.

Halyard times out during a deployment

If this happens, there are one of two causes:

The services that haven’t become healthy are misconfigured. Run hal deploy collect-logs to collect service logs, which will be placed in ~/.hal/default/service-logs. Check for any obvious errors.
You do not have enough resources in your environment to run Spinnaker, and the deployer is waiting for some to become available. This varies from environment to environment.

I want to configure a service beyond what Halyard exposes

First, please read the custom configuration documentation. With that in mind, if you’re configuring any of Spinnaker’s Spring-based services (everything but deck), you’re best off providing a -local.yml profile for the service in mind. For example, say you are configuring the Halyard deployment default, and the service gate, you can write the following file:

~/.hal/default/profiles/gate-local.yml

example:
  property: value

and the properties generated by Halyard will be overwritten by those provided in that gate-local.yml file.

If the service you want to configure does not rely on Spring, you will need to wholesale overwrite the config in the ~/.hal/default/profiles directory.

I don’t want to rely on any of the configuration generated by Halyard

You have two options here:

Provide custom configuration for any services whose config you prefer to override.
Deploy Spinnaker with the --omit-config flag. In the Local installation, this will pin & download validated debian packages at their respective versions without providing any configuration. In a Distributed installation, you will need to provide your own mechanisms for loading configuration for each subservice.

I’m seeing duplicate/bad infrastructure entries in the UI after deploying config changes

This information is drastically out of date. Spinnaker has cleanup services that can be configured. Configure the unknown-agent-cleanup-agent in clouddriver to purge out of date information

Spinnaker’s cloud provider integration point (clouddriver) does not clean out cache entries that may be left around after reconfiguring existing accounts. The best way to get around this is to supply the --flush-infrastructure-caches to a hal deploy apply. This may cause jittering in the UI as the caches are repopulated.

Halyard produces a lot of ugly ANSI escape sequences making it frustrating to automate

You can supply the flags -q -log=info to any Halyard command to get more digestable log messages. -q suppresses the ANSI pretty-printing, and -log=info enables info-level logs in the CLI.

I don’t want to use a bunch of CLI commands to configure Spinnaker; I prefer my text editor

Anything in the ~/.hal/config can be edited by hand at any time. You can validate what you provide there by running hal config, and deploy it as you would normally hal deploy apply. For more service-specific edits, please read the custom configuration docs.

I only want to deploy a subset of Spinnaker’s services

You can run hal deploy apply --service-names <service1> <service2...> to deploy only the services you care about. Be careful, if you have sidecars like the monitoring-daemon or the consul-client for a VM based distributed deployment, you will have to supply those as well.

This can be coupled with --omit-config in a local installation to provide a taylored way of deploying & configuring only the services you want on the machines you care about.

I want to run Halyard behind a proxy

In the file under /opt/halyard/bin/halyard, add the necessary proxy configuration to the variable DEFAULT_JVM_OPTS as described here For example,

DEFAULT_JVM_OPTS=-Dhttp.proxyHost=my.proxy.domain.com -Dhttp.proxyPort=3128

I want to run a Spinnaker service (Clouddriver, Echo, etc) behind an HTTP proxy server

For most Spinnaker service communication, this can be accomplished by setting appropriate JVM options for the service you want to proxy. For example, if you wanted to proxy Echo communication for Slack notifications, you would add the following proxy settings to ~/.hal/default/service-settings/echo.yml

env:
  JAVA_OPTS:
    "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
    -Dhttp.proxyHost=<proxy host> -Dhttp.proxyPort=<proxy port> -Dhttps.proxyHost=<proxy host>
    -Dhttps.proxyPort=<proxy port> -Dhttp.nonProxyHosts='localhost|127.*|[::1]|*.spinnaker'"

These settings will forward all external communication through the proxy server specified while keeping internal traffic non-proxied. Additional information can be found here.

The Kubernetes provider must be handled differently. Because the Kubernetes provider uses kubectl (which uses curl), you must set environment variables if you want Kubernetes traffic to be proxied.

An example clouddriver.yml that will proxy Kubernetes traffic will look like:

env:
  HTTP_PROXY: 'proxyaddress:proxyport'
  HTTPS_PROXY: 'proxyaddress:proxyport'
  NO_PROXY: 'localhost,127.0.0.1,*.spinnaker'

What is the best way to delete a Spinnaker deployment?

Delete the namespace where spinnaker runs. This should remove all secrets/configmaps/etc NOTE external configured resources like databases will need manual cleanup OR automated cleanup via tooling like OpenTofu.

Docs: Migrating from Operator/Halyard to Kustomize Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Migrating to a Kustomize Deployment

Thank you to harness spinnaker team for the script and automation to enable migration to kustomize easily. This is a mirror from their documentation on migration off of operator and though operator specific in many aspects, this should work just as well for halyard deployments

Introduction

This document provides step-by-step instructions for migrating your Spinnaker installation from using the Armory Operator deployment method OR a halyard deployment to a native Kubernetes deployment using Kustomize. This approach gives you more direct control over your Spinnaker resources and removes the dependency on halyard

Important

Please thoroughly test this migration in a non-production environment before deploying to production.

Prerequisites

Kubectl command-line tool installed and configured to access your cluster
Basic understanding of Kubernetes resources (deployments, services, configmaps)
Access to the current Spinnaker namespace

Migration Process Overview

Download current configuration files and Kubernetes resources
Set up Kustomize structure for native deployment
Remove Operator ownership from services (Only for operator changes)
Scale down the Operator
Deploy using Kustomize
Validate the deployment
Remove Operator and CRDs (after confirming stability)

Step 1: Download Current Configuration

The script provided below will download:

All configuration files located in /opt/spinnaker/config from each service
All deployment, service, and statefulset YAML files for each service

How to Use the Download Script

Save the script at the bottom of this document to a file named download_spinnaker_configs.sh
Make the script executable: chmod +x download_spinnaker_configs.sh
Run the script with your Spinnaker namespace: ./download_spinnaker_configs.sh your-spinnaker-namespace
The script will create an operator-migration directory containing all needed files

What Gets Downloaded

Deployments: YAML files for each service deployment
Services: YAML files for all Spinnaker services
StatefulSets: YAML files for any statefulsets (like front50)
Configuration Files: All files from /opt/spinnaker/config in each pod

WARNING: THIS WILL download any secrets that are NOT using an encrypted secret. USE CAUTION. It’s recommend to move secrets to a valid secret engine as part of any spinnaker deployment.

Step 2: Set Up Kustomize Structure

Create a Kustomize directory structure for your Spinnaker deployment:

Move the downloaded deployments and services to their respective directories
Create configmaps from the downloaded configuration files
Set up the kustomization.yaml files

Tip

You can use the GitHub - https://github.com/spinnaker/spinnaker/tree/main/spinnaker-kustomize repo as a reference for Kustomize structure

Step 3: Remove Operator Ownership from Services

This step detaches the Operator’s control while keeping services running. ONLY for operator deploys. Halyard deployments should not need to execute this step.

Identify resources owned by the Operator: kubectl get all -n your-spinnaker-namespace -o json | jq '.items[] | select(.metadata.ownerReferences[]? | .apiVersion=="spinnaker.io/v1alpha2" and .kind=="SpinnakerService") | {name: .metadata.name, kind: .kind}'
Remove ownership references using patch commands: kubectl patch deployment spin-deck -n your-spinnaker-namespace --type json -p='[{"op": "remove", "path": "/metadata/ownerReferences"}]'
Repeat for all resources with Operator ownership

Note
This breaks the connection between the Operator and services but keeps everything running

Step 4: Verify Ownership Removal

ONLY for operator deployments. Confirm that no resources are still owned by the Operator: kubectl get all -n your-spinnaker-namespace -o json | jq '.items[] | select(.metadata.ownerReferences[]? | .apiVersion=="spinnaker.io/v1alpha2" and .kind=="SpinnakerService") | {name: .metadata.name, kind: .kind}' The command should return empty if all ownership references have been removed.

Step 5: Extra Precautions Before Deployment

Compare current resources with your Kustomize configurations: kubectl diff -f <(kustomize build ./overlays/prod)

Review the differences carefully. Look for:

Immutable field changes (might require special handling)
Configuration changes that could affect service behavior
Missing resources that should be included

Perform a Dry Run

Test your deployment without actually applying changes: kubectl apply --dry-run=client -f <(kustomize build ./overlays/prod)

Step 6: Scale Down the Operator (only on operator deploys)

Prevent the Operator from interfering with your deployment: kubectl scale deployment spinnaker-operator -n your-spinnaker-namespace --replicas=0

Step 7: Deploy Using Kustomize

Apply your Kustomize configurations:

kubectl apply -f <(kustomize build ./overlays/prod)

Step 8: Validate and Monitor

Check that all pods are running: kubectl get pods -n your-spinnaker-namespace
Verify Spinnaker services are accessible:
- Access the Spinnaker UI
- Test a simple pipeline
- Check integrations are working
Monitor the environment for stability over the next few days

Step 9: Remove Operator and CRDs

Once stability is confirmed (at least 24 hours later):

Remove the Operator CRDs: kubectl delete crd spinnakerservices.spinnaker.io
Remove the Operator deployment if still present: kubectl delete deployment spinnaker-operator -n your-spinnaker-namespace

Rollback Plan

If issues arise during migration:

Scale up the Operator: kubectl scale deployment spinnaker-operator -n your-spinnaker-namespace --replicas=1
Reapply the previous SpinnakerService resource: kubectl apply -f original-spinnakerservice.yaml
Allow the Operator to reconcile and restore the previous state

Download Script

#!/bin/bash

# Script to download ONLY files from /opt/spinnaker/config in Spinnaker pods

set -e

if [ -z "$1" ]; then
  echo "Please provide a namespace"
  echo "Usage: $0 <namespace>"
  exit 1
fi

NAMESPACE=$1
OUTPUT_DIR="operator-migration"

echo "Creating output directory: $OUTPUT_DIR"
rm -rf "$OUTPUT_DIR"
mkdir -p "$OUTPUT_DIR"

# 1. First download deployments and services
echo "Downloading Kubernetes deployments and services..."

# Download Deployments
echo "Downloading deployments..."
mkdir -p "$OUTPUT_DIR/deployments"
kubectl get deployments -n "$NAMESPACE" -o name | while read -r deployment; do
  deployment_name=$(echo "$deployment" | cut -d/ -f2)
  echo "Downloading deployment: $deployment_name"
  kubectl get deployment "$deployment_name" -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/deployments/$deployment_name.yaml"
done

# Download Services
echo "Downloading services..."
mkdir -p "$OUTPUT_DIR/services"
kubectl get services -n "$NAMESPACE" -o name | while read -r service; do
  service_name=$(echo "$service" | cut -d/ -f2)
  echo "Downloading service: $service_name"
  kubectl get service "$service_name" -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/services/$service_name.yaml"
done

# Download StatefulSets if any
echo "Checking for statefulsets..."
if kubectl get statefulsets -n "$NAMESPACE" 2>/dev/null | grep -q .; then
  mkdir -p "$OUTPUT_DIR/statefulsets"
  kubectl get statefulsets -n "$NAMESPACE" -o name | while read -r statefulset; do
    statefulset_name=$(echo "$statefulset" | cut -d/ -f2)
    echo "Downloading statefulset: $statefulset_name"
    kubectl get statefulset "$statefulset_name" -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/statefulsets/$statefulset_name.yaml"
  done
fi

# 2. Now download files from /opt/spinnaker/config
echo "Downloading files ONLY from /opt/spinnaker/config..."

# Get all pods
PODS=$(kubectl get pods -n "$NAMESPACE" -o name | cut -d/ -f2)
for POD in $PODS; do
  SERVICE=$(echo "$POD" | sed -E 's/([a-z-]+)-[0-9a-z-]+.*/\1/')
  
  echo "Processing pod: $POD (service: $SERVICE)"
  mkdir -p "$OUTPUT_DIR/$SERVICE"
  
  # Check ONLY for /opt/spinnaker/config
  if kubectl exec -n "$NAMESPACE" "$POD" -- ls -la /opt/spinnaker/config &>/dev/null; then
    echo "Found /opt/spinnaker/config directory in $POD"
    
    # List all files first
    CONFIG_FILES=$(kubectl exec -n "$NAMESPACE" "$POD" -- find /opt/spinnaker/config -type f 2>/dev/null)
    if [ -z "$CONFIG_FILES" ]; then
      echo "No files found in /opt/spinnaker/config for $POD"
      continue
    fi
    
    echo "Found $(echo "$CONFIG_FILES" | wc -l | tr -d ' ') files in /opt/spinnaker/config for $POD"
    
    # Download each file
    for FILE in $CONFIG_FILES; do
      FILENAME=$(basename "$FILE")
      echo "Downloading $FILENAME from $POD"
      
      FILE_CONTENT=$(kubectl exec -n "$NAMESPACE" "$POD" -- cat "$FILE" 2>/dev/null)
      if [ $? -eq 0 ] && [ -n "$FILE_CONTENT" ]; then
        echo "$FILE_CONTENT" > "$OUTPUT_DIR/$SERVICE/$FILENAME"
        echo "Saved $FILENAME to $OUTPUT_DIR/$SERVICE/$FILENAME"
      else
        echo "Failed to download $FILENAME or file is empty"
      fi
    done
    
    # Check if we downloaded any files
    if [ -z "$(ls -A "$OUTPUT_DIR/$SERVICE" 2>/dev/null)" ]; then
      echo "No files were successfully downloaded from $POD"
    else
      echo "Successfully downloaded $(ls -1 "$OUTPUT_DIR/$SERVICE" | wc -l | tr -d ' ') files from $POD"
    fi
  else
    echo "No /opt/spinnaker/config directory found in $POD"
  fi
done

echo "======================================"
echo "Download completed. Files saved to: $OUTPUT_DIR"
echo "Summary of downloaded files by service:"

# Generate summary
for DIR in $(find "$OUTPUT_DIR" -mindepth 1 -maxdepth 1 -type d | sort); do
  SERVICE=$(basename "$DIR")
  FILE_COUNT=$(find "$DIR" -type f | wc -l)
  
  echo "- $SERVICE: $FILE_COUNT files"
  if [ "$FILE_COUNT" -gt 0 ]; then
    ls -1 "$DIR" | sort | while read -r file; do
      echo "  - $file"
    done
  fi
done

echo "Script execution complete!"

Conclusion

By following these steps, you’ll successfully migrate from the Spinnaker Operator to a native Kubernetes deployment using Kustomize. This approach gives you more direct control over your Spinnaker resources and eliminates dependency on the Operator. If you encounter any issues during the migration process, please submit a support ticket for assistance.

Docs: Component Sizing

Mon, 01 Jan 0001 00:00:00 +0000

CPU & Memory requests/limits

Container requests and limits for cpu and memory can be specified in the spinnaker-kustomize/base/*/deployments.yml files. Watch for OutOfMemory exceptions and monitor JVM Memory, CPU usage, and similar metrics to keep spinnaker up and running.

Limits and requests can be set per kubernetes documentation .

Updating JAVA_OPTS

All JVM-based services have the following JAVA_OPTS that can be set to adjust things like memory or additional flags. You can add these to the deployment ENV variables:

JAVA_OPTS=-XX:MaxRAMPercentage=50.0

This sets the JVM’s heap size to half the memory allocated per container. This can be overriden by specifying your own JAVA_OPTS using the env flags. Generally, for large production systems, setting explicit -Xmx and -Xms to be the same value and sized for your environment is recommended to reduce allocation times. JVM tuning is an entire separate discussion that should be had with JVM experts or looking at documentation.

As a starting point, the -Xms can be set to 80%-90% of the requests memory allotment and -Xmx can be set to 80-90% of the limits memory allotment. It’s recommend NOT to make this too large for clouddriver or processes that fork to external operations (Rosco for bakes). This is due to those processes not using JVM memory but OS memory.

Recommendations

Optimal sizing for your components will depend upon your environment, but in general clouddriver and orca will need to be larger than the rest of the components as they’re continuously processing information. Additionally, echo may also need to be larger if using event hooks to external systems or processing large pub/sub payloads.

Replicas

All spinnaker services CAN be deployed with multiple replicas for availability. There are configurations required to enable locking or to prevent concurrent operations.

For igor, locking must be enabled to run multiple replicas. This is defaulted on in spinnaker-kustomize. When running a single pod, you can turn locking off.
For echo, make sure to use a SQL database. Echo uses a quartz scheduler and may need additional adjustment depending upon configuration. Read the readme for echo for more information
For other services, like clouddriver, using a shared redis and clouddriver should be all that’s needed to run multiple replicas of all services.

Docs: High Availability

Mon, 01 Jan 0001 00:00:00 +0000

Warning this is an advanced that most basic users never need to worry about

This page describes how you can configure a Spinnaker deployment to increase the availability/performance of specific services beyond simply horizontally scaling the services.

This is done by splitting the functionalities of a service into separate logical roles. The benefits of doing this is specific to the service that is being sharded. These deployment strategies are inspired by Netflix’s large scale experience .

When sharded, the new logical services are given new names. This means that these logical services can be configured and scaled independently of each other.

Currently, this feature is primarily for Clouddriver and/or Orca. Echo can do some sharding as well.

HA Clouddriver

graph TB clouddriver(Clouddriver) --> clouddriver-caching(Clouddriver-Caching); clouddriver --> clouddriver-rw(Clouddriver-RW); clouddriver --> clouddriver-ro(Clouddriver-RO); clouddriver --> clouddriver-ro-deck(Clouddriver-RO-Deck) classDef default fill:#d8e8ec,stroke:#39546a; linkStyle default stroke:#39546a,stroke-width:1px,fill:none; classDef split fill:#42f4c2,stroke:#39546a; class clouddriver-caching,clouddriver-ro,clouddriver-ro-deck,clouddriver-rw,echo-scheduler,echo-worker split

Clouddriver benefits greatly from isolating its operations into separate services. To split Clouddriver into multiple services, you’ll configure each service to operate on specific configurations and operations.

Caching pods - for JUST executing cache agents. These will not process any user or pipeline execution data
RW pods - for execution operations
RO pods - for UI and API operations for end users.

For more advanced configurations INCLUDING sharding clouddriver by account, region or application names, you can look at the code in clouddriver, orca, and gate around ServiceSelectors. We’d welcome additional documentations for this as needed.

Although by default the four Clouddriver services will communicate with a global Redis and Database (all Spinnaker services speak to this Redis/SQL), it is recommended that the logical Clouddriver services be configured to communicate with external solutions. To be most effective, clouddriver-ro should be configured to speak to a RO SQL read replica, clouddriver-ro-deck should be configured to speak to a different read replica, and the other two should be configured to speak to the master.

For example configurations, please look into clouddrivers codebase, the SQL configuration

More information on Redis replication can be found here .

`clouddriver-caching`

The first of the four logical Clouddriver services is the clouddriver-caching service. This service caches and retrieves cloud infrastructure data. Since this is all that clouddriver-caching is doing, there is no communication between this service and any other Spinnaker service.

This service’s name when configuring its sizing is spin-clouddriver-caching.

`clouddriver-rw`

The second logical Clouddriver service is the clouddriver-rw service. This service handles all mutating operations aside from what the clouddriver-caching service does. This service can be scaled to handle an increased number of writes.

`clouddriver-ro`

The clouddriver-ro service handles all read requests to Clouddriver. This service can be scaled to handle an increased number of reads.

This service’s name when configuring its sizing is spin-clouddriver-ro.

`clouddriver-ro-deck`

The clouddriver-ro-deck service handles all read requests to Clouddriver from Deck (through Gate). This service can be scaled to handle an increased number of reads.

This service’s name when configuring its sizing is spin-clouddriver-ro-deck.

HA Echo

graph TB echo(Echo) --> echo-scheduler(Echo-Scheduler); echo(Echo) --> echo-worker(Echo-Worker); classDef default fill:#d8e8ec,stroke:#39546a; linkStyle default stroke:#39546a,stroke-width:1px,fill:none; classDef split fill:#42f4c2,stroke:#39546a; class clouddriver-caching,clouddriver-ro,clouddriver-rw,echo-scheduler,echo-worker split

Echo can be split into two separate services that handle different operations. To split Echo for increased availability, run:

hal config deploy ha echo enable

When Spinnaker is deployed with this enabled, Echo will be deploy as two different services:

Although only the echo-worker service can be horizontally scaled, splitting the services will reduce the load on both.

`echo-scheduler`

The echo-scheduler service handles scheduled tasks, or cron-jobs. Since it performs its tasks periodically (no triggers) there is no need for communication with other Spinnaker services.

This service’s name when configuring its sizing is spin-echo-scheduler. To avoid duplicate triggering, this service must be deployed with exactly one pod.

`echo-worker`

The echo-worker service handles all operations of Echo besides the cron-jobs.

This service’s name when configuring its sizing is spin-echo-worker. This service can be scaled to more than one pod, unlike the echo-scheduler.

HA Topology

With all services enabled for high availability, the new architecture looks like this:

graph TB deck(Deck) --> gate; api(Custom Script/API Caller) --> gate(Gate); gate --> kayenta(Kayenta); gate --> orca(Orca); gate --> clouddriver-ro(Clouddriver-RO); gate --> clouddriver-ro-deck(Clouddriver-RO-Deck) orca --> clouddriver-rw(Clouddriver-RW); gate --> rosco(Rosco); orca --> front50; orca --> rosco; gate --> front50(Front50); gate --> fiat(Fiat); orca --> kayenta; clouddriver-ro --> fiat; clouddriver-ro-deck --> fiat; clouddriver-rw --> fiat; orca --> fiat; front50 --> fiat; echo-worker(Echo-Worker) --> orca; echo-worker --> front50; igor(Igor) --> echo-worker; clouddriver-caching(Clouddriver-Caching); echo-scheduler(Echo-Scheduler); classDef default fill:#d8e8ec,stroke:#39546a; linkStyle default stroke:#39546a,stroke-width:1px,fill:none; classDef external fill:#c0d89d,stroke:#39546a; class deck,api external classDef split fill:#42f4c2,stroke:#39546a; class clouddriver-caching,clouddriver-ro,clouddriver-ro-deck,clouddriver-rw,echo-scheduler,echo-worker split

Recoverability

If you’ve configured an external Redis , Spinnaker can recover from failure events. Igor is responsible for polling the state store (Redis) and recreating state. During a recovery, numerous old pipelines may be re-triggered. To protect against this scenario, Igor has a setting called pollingSafeguard.itemUpperThreshold, which is the max number of pipeline triggers to accept before recognizing the scenario of numerous old re-triggers and stopping the state update. Read the inline documentation for this setting .

Although it’s possible, it is not recommended to use spot instances on AWS or preemtible nodes to lower cost in production, as outages in your continuous deployment tool will likely cost more than any savings. Also consider hosting Spinnaker on isolated infrastructure to reduce the possibility that other applications or teams will affect your Spinnaker instance.

It’s also recommended to spread out services over multiple availability zones, as described in the Netflix implementation .

Docs: Deprecated - Halyard

Mon, 01 Jan 0001 00:00:00 +0000

This documentation is deprecated as halyard is deprecated and no longer supported. This is left here for historical purposes

Halyard is a command-line administration tool that manages the lifecycle of your Spinnaker deployment, including writing & validating your deployment’s configuration, deploying each of Spinnaker’s microservices, and updating the deployment.

All production-capable deployments of Spinnaker require Halyard in order to install, configure, and update Spinnaker. Though it’s possible to install Spinnaker without Halyard, we don’t recommend it, and if you get stuck we’re just going to tell you to use Halyard.

There are two ways you can install Halyard:

locally on Debian/Ubuntu

This can be on a desktop or laptop computer, or on a VM.
on Docker

We recommend you install Halyard on a machine with at least 12GB of RAM.

Install on Debian/Ubuntu

Halyard runs on…

Ubuntu 18.04 or higher
Debian 10 or higher

Get the latest version of Halyard:

curl -O https://raw.githubusercontent.com/spinnaker/halyard/master/install/debian/InstallHalyard.sh

Install it:

sudo bash InstallHalyard.sh

If you’re prompted for any information, default answers are usually suitable.
Check whether Halyard was installed properly:

hal -v

If this command fails, make sure hal is in your $PATH, and check the logs under /var/log/spinnaker/halyard/halyard.log.
Run . ~/.bashrc to enable command completion.

To get help with any hal command, append -h. Also, see the Halyard command Reference .

Update Halyard on Debian/Ubuntu

Note: This upgrade procedure only works with Halyard v1.45.0 and higher. If you are using a Halyard version prior to v1.45.0, follow the instructions above to Install Halyard .

sudo apt-get update
sudo apt --only-upgrade install spinnaker-halyard

Uninstall Halyard from Debian/Ubuntu

Important: uninstalling Halyard deletes the entire contents of your ~/.hal directory. Don’t do it unless you’re prepared to lose your configuration.

If you used Halyard to deploy Spinnaker, and you want to purge that deployment, run the following command:
```
hal deploy clean
```
Now you can safely uninstall Halyard:
```
sudo ~/.hal/uninstall.sh
```

Install Halyard on Docker

Note: If you install Halyard in a Docker container, you will need to manually change permissions on the mounted ~/.hal directory to ensure Halyard can read and write to it.

Make sure you have Docker CE installed .
On your current machine, make a local Halyard config directory.
```
mkdir ~/.hal
```
This will persist between runs of the Halyard docker container.
Start Halyard in a new Docker container.

The following command creates the Halyard Docker container, mounting the Halyard config directory:
```
docker run -p 8084:8084 -p 9000:9000 \
    --name halyard --rm \
    -v ~/.hal:/home/spinnaker/.hal \
    -it \
    us-docker.pkg.dev/spinnaker-community/docker/halyard:stable
```
This runs as a foreground process in your current shell. This is useful because it emits all of the Halyard daemon’s logs, which are not persisted. If you don’t care about the logs, and would rather run in detached mode, replace the -it with -d

Note: Any secrets/config you need to supply to the daemon (for example, a kubeconfig file) must be mounted in either your local ~/.hal directory, or another directory that you supply to docker run with additional -v command-line options.
In a separate shell, connect to Halyard:
```
docker exec -it halyard bash
```
You can interact with Halyard from here.
Run the following command to enable command completion:
```
source <(hal --print-bash-completion)
```

To get help with any hal command, append -h. Also, see the Halyard command Reference .

Update Halyard on Docker

Fetch the latest Halyard version.

docker pull us-docker.pkg.dev/spinnaker-community/docker/halyard:stable

Stop the running Halyard container.

docker stop halyard

Re-run the container:

docker run -p 8084:8084 -p 9000:9000 \
    --name halyard --rm \
    -v ~/.hal:/home/spinnaker/.hal \
    -it \
    us-docker.pkg.dev/spinnaker-community/docker/halyard:stable

This re-starts the container using the updated image you got in step 1.

In a separate shell, run:
```
docker exec -it halyard bash
```

Uninstall Halyard from Docker

To uninstall Halyard, just delete the container.

docker rm halyard

Next steps

Now that Halyard is running, it’s time to choose your cloud provider .

Spinnaker – Install and Configure Spinnaker

Docs: Choose your Environment

Kubernetes installation

Local Debian

Prerequisites

Overview

Intended use case

Install local dependencies

Further reading

Next steps

Docs: Deploy Spinnaker and Connect

Pick a version

Deploy Spinnaker

Ingress

Connect to the Spinnaker UI

Troubleshooting

Upgrade Spinnaker

Next steps

Docs: Cloud Providers Overview

Supported providers

Next steps

Docs: Storage

Supported databases

Configure the services

Next steps

Docs: Configuration

Deploy to more cloud accounts

Secure your Spinnaker installation

Setup continuous integration

Configure notifications

Monitor your Spinnaker installation

Next steps

Docs: Setup FAQ

CAUTION

I can’t load the Applications screen

I want to expose LocalDebian Spinnaker on a public IP address, but it always binds to localhost

Halyard times out during a config change

Halyard times out during a deployment

I want to configure a service beyond what Halyard exposes

I don’t want to rely on any of the configuration generated by Halyard

I’m seeing duplicate/bad infrastructure entries in the UI after deploying config changes

Halyard produces a lot of ugly ANSI escape sequences making it frustrating to automate

I don’t want to use a bunch of CLI commands to configure Spinnaker; I prefer my text editor

I only want to deploy a subset of Spinnaker’s services

I want to run Halyard behind a proxy

I want to run a Spinnaker service (Clouddriver, Echo, etc) behind an HTTP proxy server

What is the best way to delete a Spinnaker deployment?

Docs: Migrating from Operator/Halyard to Kustomize Deployment

Migrating to a Kustomize Deployment

Introduction

Important

Prerequisites

Migration Process Overview

Step 1: Download Current Configuration

How to Use the Download Script

What Gets Downloaded

Step 2: Set Up Kustomize Structure

Tip

Step 3: Remove Operator Ownership from Services

Note

Step 4: Verify Ownership Removal

Step 5: Extra Precautions Before Deployment

Perform a Dry Run

Step 6: Scale Down the Operator (only on operator deploys)

Step 7: Deploy Using Kustomize

Step 8: Validate and Monitor

Step 9: Remove Operator and CRDs

Rollback Plan

Download Script

Conclusion

Docs: Component Sizing

CPU & Memory requests/limits

Updating JAVA_OPTS

Recommendations

Replicas

Docs: High Availability

Warning this is an advanced that most basic users never need to worry about

HA Clouddriver

clouddriver-caching

clouddriver-rw

`clouddriver-caching`

`clouddriver-rw`

`clouddriver-ro`

`clouddriver-ro-deck`

`echo-scheduler`

`echo-worker`