Spinnaker – Configure Everything Else

Docs: Server Group Launch Settings

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Configure Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

Artifacts are remote, deployable resources in Spinnaker. Depending on their “ type ”, they will need credentials to be downloaded or uploaded. These pages take you through configuring those credentials.

Docs: Bakery

Mon, 01 Jan 0001 00:00:00 +0000

Since the concept of Immutable Infrastructure is core to Spinnaker, Spinnaker includes an image bakery powered by Hashicorp’s Packer to help you produce machine images.

For getting started, the default configuration (no changes required) provides you with enough Packer templates & base machine image options to learn how the system works. However, once you want to start customizing the bake process or use artifacts not supported by the default configuration, these documentation pages will describe how to further configure the bakery.

There are two types of configuration of the Bakery:

Packer Templates

Every time you initiate a bake using Spinnaker, Spinnaker invokes a packer template with a mix of variables provided by you, the Pipeline you are currently executing, and Spinnaker itself. All the default packer templates are versioned alongside Rosco, the image bakery service here . If want to override/include a new template, place it into a volume that replaces the /opt/rosco/config/packer folder on the rosco pods. This can be a configmap, NFS volume or any solution you choose from kubernetes volume mount support.

Any local scripts/artifacts required by that template can also be placed into that directory, and referenced relative to the configDir Packer variable that will automatically be set.

You can then use the Template File Name field under Advanced Options in the bake stage UI to refer to your custom template:

Image Provider Configuration

The following providers support configuring their image bakery:

Docs: Security

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker has multiple options for both authentication and authorization. Instead of reinventing yet-another-login system, Spinnaker hooks into a login system your organization probably already has, such as OAuth 2.0, SAML, or LDAP.

For authorization, Spinnaker similarly leverages a role-provider that your organization may already have set up, including Google Groups, GitHub Teams, SAML Roles, or LDAP groups.

For reference…

Authorization and Authentication Architecture and workflow

Next steps

Setup authentication for your Spinnaker deployment.

Docs: Set Up Triggers

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Install and Configure Spin CLI

Mon, 01 Jan 0001 00:00:00 +0000

For more information, see the spin CLI Guide

Install `spin`

One way to manage applications and pipelines as code is through spin .

To acquire spin, do the following:

On Linux

curl -LO https://storage.googleapis.com/spinnaker-artifacts/spin/$(curl -s https://storage.googleapis.com/spinnaker-artifacts/spin/latest)/linux/amd64/spin

chmod +x spin

sudo mv spin /usr/local/bin/spin

On MacOS

# Apple Silicon
curl -LO https://storage.googleapis.com/spinnaker-artifacts/spin/$(curl -s https://storage.googleapis.com/spinnaker-artifacts/spin/latest)/darwin/arm64/spin
# Intel
curl -LO https://storage.googleapis.com/spinnaker-artifacts/spin/$(curl -s https://storage.googleapis.com/spinnaker-artifacts/spin/latest)/darwin/amd64/spin

chmod +x spin

sudo mv spin /usr/local/bin/spin

On Windows

New-Item -ItemType Directory $env:LOCALAPPDATA\spin -ErrorAction SilentlyContinue

Invoke-WebRequest -OutFile $env:LOCALAPPDATA\spin\spin.exe -UseBasicParsing "https://storage.googleapis.com/spinnaker-artifacts/spin/$([System.Text.Encoding]::ASCII.GetString((Invoke-WebRequest https://storage.googleapis.com/spinnaker-artifacts/spin/latest).Content) -replace "`n")/windows/amd64/spin.exe"

Unblock-File $env:LOCALAPPDATA\spin\spin.exe

$path = [Environment]::GetEnvironmentVariable("PATH", [EnvironmentVariableTarget]::User) -split ";"
if ($path -inotcontains "$env:LOCALAPPDATA\spin") {
  $path += "$env:LOCALAPPDATA\spin"
  [Environment]::SetEnvironmentVariable("PATH", $path -join ";", [EnvironmentVariableTarget]::User)

  $env:PATH = (([Environment]::GetEnvironmentVariable("PATH", [EnvironmentVariableTarget]::Machine) -split ";") + $path) -join ";"
}

Configure `spin`

spin reads its configuration from ~/.spin/config.

This configuration file doesn’t exist yet, after you install spin. You need to create it.

Create the directory:

mkdir ~/.spin/

In that directory, create the config file.

Use example.yaml to populate it.

Currently, all configuration is for authentication mechanisms only.

X.509

spin can be configured with X.509 to authenticate calls against Spinnaker. The configuration block looks like this:

auth:
  enabled: true
  x509:
    certPath: <cert file path>
    keyPath: <key file path>

auth:
  enabled: true
  x509:
    # Pipes for multi-line strings in yaml.
    # Cert and key contents are 64 encoded pem values.
    cert: |
    <cert>
    key: |
    <key>

Follow the ssl and x509 guides to generate the X.509 certificate and key files. Refer to the example config and the README for more information about X.509 in spin.

OAuth2

Client ID and Client Secret

spin can be configured with an OAuth2 client ID and secret to authenticate calls against Spinnaker. The configuration block looks like this:

auth:
  enabled: true
  oauth2:
    authUrl: # OAuth2 provider auth url
    tokenUrl: # OAuth2 provider token url
    clientId: # OAuth2 client id
    clientSecret: # OAuth2 client secret
    scopes: # Scopes requested for the token
    - scope1
    - scope2

Read the OAuth setup instructions to see examples for acquiring a clientId/clientSecret from your provider.

This OAuth2 configuration method needs to be initialized once to authenticate with the provider before it can be used for automation. To authenticate, configure OAuth2 as shown above and execute any spin command. You will be prompted to authenticate with your OAuth2 provider and paste an access code. This process involves configuring the callback url to be http://localhost:8085 in order to view and retrieve the access code to be provided to the prompt. spin then exchanges the code for an OAuth2 access/refresh token pair, which it caches in your ~/.spin/config file for future use. All subsequent spin calls will use the cached OAuth2 token for authentication with no user input required. If an OAuth2 access token expires, spin will use the refresh token to renew the access token expiry.

Access and Refresh Token

spin can also be configured with an OAuth2 access token and refresh token in lieu of configuring an OAuth2 client ID and secret. The spin configuration looks like this:

auth:
  enabled: true
  oauth2:
    authUrl: # OAuth2 provider auth url
    tokenUrl: # OAuth2 provider token url
    # no clientId or clientSecret
    scopes: # Scopes requested for the token
    - scope1
    - scope2
    cachedToken:
      accesstoken: ${ACCESS_TOKEN} # Note the key capitalization
      refreshtoken: ${REFRESH_TOKEN} # Note the key capitalization

This method is OAuth2-provider specific since the workflow to acquire a token is different for each provider. To do so using Google OAuth2 and gcloud:

Authenticate with Google via gcloud auth login.
Use the following commands to acquire the tokens:

```
ACCESS_TOKEN=$(gcloud auth print-access-token)
REFRESH_TOKEN=$(gcloud auth print-refresh-token)
```

Google Service Account

If using Google OAuth2, spin can also be configured with a service account . Generate a service account key in JSON format. For more information, see the Google docs . The JSON file should look something like this:

{
  "type": "service_account",
  "project_id": "project-id",
  "private_key_id": "key-id",
  "private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
  "client_email": "service-account-email",
  "client_id": "client-id",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}

The spin configuration looks like this:

auth:
  GoogleServiceAccount:
    file: /path/to/key/pair.json

Basic

spin can be configured with basic authentication credentials to authenticate calls against Spinnaker. The configuration block looks like this:

auth:
  enabled: true
  basic:
    username: < username >
    password: < password >

LDAP/Active Directory

spin can be configured with LDAP to authenticate calls against Spinnaker. The configuration block looks like this:

auth:
  enabled: true
  ldap:
    username: < username >
    password: < password >

Global Flags

spin has a few helpful global flags:

Global Options:

        --gate-endpoint               Gate (API server) endpoint.
        --no-color                    Removes color from CLI output.
        --insecure=false              Ignore certificate errors during connection to endpoints.
        --quiet=false                 Squelch non-essential output.
        --output <output format>      Formats CLI output.

Output Formatting

The global --output flag allows users to manipulate spin’s output format. By default, spin will print the command output in json. Users can also specify a jsonpath in the output flag to extract nested data from the command output:

spin pipeline get --name pipelineName --application app --output jsonpath="{.stages}"

[
<stages>
]

spin leverages the kubectl jsonpath libraries to support this. Follow the kubectl documentation for more information about the possible jsonpath arguments.

Docs: CI

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker can listen to events, and collect artifacts produced by builds in external Continuous Integration (CI) systems. These events can trigger Pipelines, and the artifacts can be used by Spinnaker’s image bakery to produce machine images.

For example, the Source to Prod codelab configures a Jenkins Job to produce a Debian package as an artifact that is handed to Spinnaker to build a VM image which Spinnaker deploys.

Currently, Spinnaker supports several CI systems which can be enabled and run independently of one-another, enumerated below.

Supported CI systems

These are the CI systems currently supported by Spinnaker:

Docs: Logging

Mon, 01 Jan 0001 00:00:00 +0000

There are two logging options in Spinnaker:

basic logging - set root (default) and Java Class log levels.
advanced logging with logback - configurable log levels, structured JSON output and other features.

The JSON log format available with logback provides the following key advantages:

Logging providers index JSON formatted structured logs enabling rich query capabilities.
Multi-line Java exceptions become a single log line with the exception details stored in the stack_trace key.
JSON log events can include more information like: X-SPINNAKER-EXECUTION-ID and X-SPINNAKER-REQUEST-ID

Basic logging

Add the following per service (eg: orca-local.yml) or to a shared spinnaker-local.yml file.

logging:
  level:
    # Enable debug logging by changing level to DEBUG
    root: INFO  # default

    # Example setting per class log level output
    # Disable logging out `Evaluated 27 expression(s) - ({ json file })
    # com.netflix.spinnaker.orca.pipeline.util.ContextParameterProcessor: WARN

Advanced logging with logback

For more information see the manual: https://logback.qos.ch/manual/index.html

Installation

Create a file called logback.xml per Service or to share with below contents:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <!-- Default is JSON - https://github.com/logstash/logstash-logback-encoder#data-format -->
    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
  </appender>

  <!-- Enable debug logging by changing level to DEBUG -->
  <root level="INFO">
    <appender-ref ref="CONSOLE" />
  </root>
</configuration>

Mount logback.xml into your Spinnaker services at a known path like /opt/spinnaker/config/logback.xml.

For Kubernetes you may wish to use a ConfigMap .
Configure Spinnaker services to load logback.xml.

In each of the service files (eg: orca-local.yml) or in a shared file (spinnaker-local.yml) replace any existing logging: configuration with:
```
logging:
  config: /opt/spinnaker/config/logback.xml  # change path as necessary
```

Set log level per class

Logging for some classes can be verbose and it may be preferable to log less.

For example, if the root log level is INFO you may wish exclude INFO log output and log at WARN and below.

  <!-- Disable logging of SPeL evaluation errors due to log volume -->
  <!-- `Failed to evaluate {}` -->
  <logger name="com.netflix.spinnaker.kork.expressions.ExpressionTransform" level="WARN" />

  <!-- Disable logging out `Evaluated 27 expression(s) - ({ json file }) -->
  <logger name="com.netflix.spinnaker.orca.pipeline.util.ContextParameterProcessor" level="WARN" />

Filter log keys

Add an <excludeMdcKeyName> block inside the <encoder> block like so:

    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <!-- Exclude key from output due to long line lengths when a lot of accounts configured -->
      <excludeMdcKeyName>X-SPINNAKER-ACCOUNTS</excludeMdcKeyName>
    </encoder>

Docs: Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Each Spinnaker microservice is instrumented with numerous metrics, published through an internal Spectator API that wraps a Micrometer MeterRegistry and exposed on a built-in /spectator/metrics endpoint. You can scrape that endpoint yourself, or collect metrics and traces with OpenTelemetry (OTEL) , which is the recommended approach.

The metrics use a multi-dimensional, tag-based data model, described in Consuming metrics and the Monitoring reference .

OpenTelemetry (Recommended)

Attach the OpenTelemetry Java auto-instrumentation agent to each Spinnaker JVM process. The agent exports telemetry over OTLP, either to an OpenTelemetry Collector or directly to any OTLP-compatible vendor endpoint. A collector is the more flexible option, since it lets you send the same data to more than one backend.

Why this works with Spinnaker

Spinnaker registers its metrics as Micrometer meters on each service’s MeterRegistry (exposed externally through the Spectator API and /spectator/metrics). The OTEL Java agent includes optional Micrometer instrumentation that exports Micrometer meters through the OpenTelemetry metrics SDK over OTLP, so no Spinnaker code changes are required.

The Micrometer bridge is disabled by default in OTEL Java agent 2.0.0 and later. Enable it with OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true (or -Dotel.instrumentation.micrometer.enabled=true). Without it the agent still provides auto-instrumented traces (HTTP, JDBC, and other supported libraries) but exports no Micrometer meters.

Independently of the Micrometer bridge, the agent collects JVM runtime metrics and auto-instruments supported HTTP, RPC, database, messaging, and other libraries for traces and selected metrics. See the supported libraries for the current coverage.

The same setting also turns on the agent’s Spring Boot actuator autoconfiguration, which registers the OpenTelemetry meter registry as a Spring bean. Spring Boot combines it with the service’s existing registry into a CompositeMeterRegistry, and that composite is the registry Spinnaker’s Spectator API wraps, so Spinnaker’s own metrics are included. Without it the agent would only register with Micrometer’s static global registry, which Spinnaker does not publish to. Compare the exported series against /spectator/metrics if you depend on specific Spinnaker metrics.

Minimal configuration

How you attach the agent depends on your deployment method (Halyard, Operator, Helm, raw Kubernetes manifests, and so on). In all cases you need:

The OTEL Java agent JAR available to the process (for example via an init container that copies it into a shared volume, or baked into the image).
JVM options and environment variables similar to:

JAVA_OPTS="-javaagent:/path/to/opentelemetry-javaagent.jar -Dotel.service.name=clouddriver"

OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://opentelemetrycollector:4318

Repeat for each microservice, using a distinct otel.service.name (or OTEL_SERVICE_NAME) per service. Point OTEL_EXPORTER_OTLP_ENDPOINT at your collector or at a vendor OTLP ingest URL.

Consult the OpenTelemetry Java agent configuration reference for additional exporters, protocols, resource attributes, and sampling options.

New Relic note

The New Relic APM Java agent and the OpenTelemetry Java agent are alternative instrumentation paths. The New Relic agent does not automatically collect Spinnaker’s Spectator/Micrometer metrics.

Only one Java agent per JVM

Don’t attach both the New Relic APM Java agent and the OpenTelemetry Java agent to the same process. New Relic’s guidance is to pick one and only one APM product per process . Both agents rewrite bytecode and both emit W3C trace context headers, which can break instrumented classes and split one request across multiple trace IDs .

New Relic Java agent 9.1.0 and later can, when this support is enabled, capture signals recorded through the OpenTelemetry Tracing, Metrics, and Logs APIs . This does not turn it into the OpenTelemetry Java agent or automatically export Spinnaker’s Micrometer meters. New Relic lists separately installed OpenTelemetry library instrumentation and Java bytecode auto-instrumentation as unsupported. Its Micrometer instructions require the OpenTelemetry Micrometer bridge, SDK and exporter, and application wiring.

Choose one of these collection paths:

Keep the New Relic APM agent and collect Spinnaker’s own metrics separately. Scrape /spectator/metrics with an external collection pipeline when practical. If you need direct, in-process export to New Relic, the legacy Observability Plugin ’s New Relic registry remains a compatibility option.
Use the OpenTelemetry agent instead and send OTLP to New Relic. New Relic renders this data as an OpenTelemetry service with APM views for summary metrics, transactions and traces, databases, external services, errors, logs, and JVM runtime data. This provides broad APM coverage, but is not identical to the New Relic agent: some views are derived from sampled spans, and instrumentation coverage and names can differ. The JVM runtime view requires service.instance.id, which the current OpenTelemetry Java agent generates automatically.

Legacy: Spinnaker Observability Plugin

Legacy

The Spinnaker Observability Plugin is legacy. Use the OpenTelemetry Java agent to export Spinnaker’s Micrometer metrics when it is your APM agent. Existing users can keep running the plugin. Deployments that retain the New Relic APM agent can use the plugin’s New Relic registry as a compatibility option. Prefer external collection of /spectator/metrics where practical.

The Spinnaker Observability plugin replaced the Spinnaker monitoring daemon, which was deprecated as of Spinnaker release 1.20. See the announcement for background.

The plugin collects metrics reported by each microservice instance and reports them to a third-party monitoring system. It currently supports Prometheus , New Relic , and DataDog . The plugin is extensible so that it should be straightforward to add other systems as well.

The plugin can be configured to control which collected metrics are forwarded to the persistent metrics store. This can alleviate costs and pressure on the underlying metric stores depending on your situation.

Configuring the Spinnaker Observability Plugin

Install and configure the plugin using the armory-observability-plugin repository. Additional configuration examples:

Plugin v2.0.0 and later require Spinnaker 2025.4.0 or newer, which is the release that moved to Spring Boot 3. Use the v1.6.x line on earlier Spinnaker versions.

Once this is complete, you can optionally use the spinnaker-mixin package to deploy pre-configured Spinnaker dashboards for Grafana.

Consuming metrics

This data model applies however you collect metrics: by scraping /spectator/metrics, with the legacy Observability Plugin , or through the OpenTelemetry Micrometer bridge .

Each “metric” has a name and type. Each data point is a numeric value that is time-stamped at the time of reporting and tagged with a set of one or more “label”=“value” tags. These tag values are strings, though some may have numeric-looking values. Taken together, the set of tags convey the context for the reported measurement. Each of these contexts forms a distinct time-series data stream.

For example a metric counting we requests may be tagged with a “status” label and values indicating whether the call was successful or not. So rather than having two metrics, one for successful calls and the other for unsuccessful calls, there is a single metric, where the underlying monitoring system can filter the successful from unsuccessful as you want depending on how you wish to abstract and interpret the data. In practice the metrics have many tags providing a lot of granularity and ways in which you can aggregate and interpret them. The data model is described further in the Monitoring reference section .

In practice there are relatively few distinct metric names (hundreds). However when considering all the distinct time-series streams from the different label values there are thousands of distinct streams. Some metrics are tagged with the application or account they were used on behalf of, so the number of streams may grow as the scope of your deployment grows. Typically you will be aggregating these dimensions together while breaking out along others. The granularity can come in handy when it comes time to diagnose problems or investigate for deeper understanding of runtime behaviors but you can aggregate across dimensions (or parts of dimensions) when you dont care about that level of refinement.

Types of metrics

There are two basic types of metrics currently supported, counters and gauges.

Counters are monotonically increasing values over the lifetime of the process. The process starts out with them at 0, then increments them as appropriate. Some counters may increase by 1 each time, such as the number of calls. Other counters may increase by an arbitrary (but non-negative) amount, such as number of bytes.

Counters are scoped to the process they are in. If you have a counter in each of two different microservice replicas (including a restart), those counters will be independent of one another. Each process only knows about itself. Collection tooling typically adds a tag to each data point that identifies which instance it came from so that you can drill down into individual instances if you need. However, typically you will use your monitoring system to aggregate counters across all replicas.

Counters are useful to determine rates. Given two points in time, the counter differences will be the measurement delta and the delta divided by the time difference will be the rate. (divide by another 1000000 to convert nanoseconds to milliseconds, such as for latency-oriented metrics or by another 100000000 for seconds, such as for call-rate metrics).
- Spinnaker also has a special type of counter called a Timer.
  
  Timers are used to measure timing information. These are always in nanoseconds. When consuming metrics straight from Spinnaker, a Timer will have two complementary time series. One will have a tag “statistic” with the value “count” and the other a tag with a “statistic” with the value “totalTime”.
  
  The “count” represents the number of measurements taken. The “totalTime” represents the number of nanoseconds measured across all the calls. Dividing the “totalTime” by the “count” over some time window gives the latency over that time window.
  
  For example given a series of measurements for the pair of metrics example__count and example__totalTime, where the sum of the __count values was 5 and of the __totalTime values was 50000000, then dividing the time by count gives 10000000 as an average time per count. Since this is in nanoseconds, we can divide by another 1000000000 to get 0.1 seconds per call. (or we could divide by 1000000 to get 100 milliseconds per call)
  
  Note that in order to do this, the tag bindings for the two measurements should be the same. Dividing measurements whose count has a success=true tag by times that have success=false tags wont give you the average time of the success calls (but would give you the average cost in total time spent for each successful call outcome if that is what you wanted.)
Gauges are instantaneous value readings at a given point in time. Like counters, individual gauges are scoped to individual microservice instances. Collection tooling typically adds an instance tag to each data point so that you can identify the particular instance if you want to, but typically you will use your monitoring system to aggregate across instances.

Since gauges are instantaneous, the values between samples is unknown. Gauges are useful to determine current state, such as the size of queues. Sometimes answers to questions provided by gauges (e.g. active requests) might be answered by taking the difference in counters (e.g. completed requests - started requests).

Example

Each microservice has a controller.invocations metric used to instrument API calls into it. Since this is a timer, in practice this is broken out into two ‘controller.invocations__count’ and ‘controller.invocations__totalTime’.

These typically have the labels “controller”, “method”, “status”, “statusCode”, and “success”. Some microservices may add an additional label such as “account” or “application” depending on the nature of the microservices API.

These metrics will have several time series, such as those with the following tag bindings:

account	controller	method	status	statusCode	success
my-account	ClusterController	getForAccountAndNameAndType	2xx	200	true
my-account	ClusterController	getForAccountAndNameAndType	4xx	404	false
my-account	ClusterController	getForAccountAndNameAndType	4xx	400	false
None	ApplicationsController	list	2xx	200	true
None	ApplicationsController	get	2xx	200	true
None	ApplicationsController	get	4xx	404	false

You can aggregate over the success tag to count successful calls vs failures, perhaps breaking out by controller and/or method to see where the failures were. You can break out by statusCode to see which controller and/or method the errors are coming from and so forth.

Different metrics have different tags depending on their concept and semantics. Some of these tags may be of more interest than others. In the case above, some of the tags are at different levels of abstraction and not actually independent. For example a 2xx status will always be success=true and a non-2xx status code will always be success=false. Which to use is a matter of convenience but given the status tag (which can distinguish 4xx from 5xx errors) the success tag does not add any additional time-series permutations since its value is not actually independent.

Supplied dashboards

Each of the supplied monitoring solutions provides a set of dashboards tailored for that system. These are likely to evolve at different rates so are not completely analogous or consistent across systems and might not be completely consistent with the document. However, the gist and intent described here should still hold since the monitoring intent is the same across all the concrete systems.

As a rule of thumb, the dashboards currently prefer showing value differences (over rates) for 1-minute sliding windows. This might change in the future. Some of the caveats here are due to the choice to show values over rates, but at this time the values seem more meaningful than rates, particularly where there arent continuous streams of activity. Where latencies are shown, they are computed using the counters from the past minute.

Depending on the chart and underlying monitoring system, some charts show instantaneous value differences (between samples) while others show differences over a sliding window. The accuracy of the timeline may vary depending on the dashboard, but the underlying trends and relative signals over time will still be valid.

Types of dashboards

There are several different dashboards. Each monitoring system has its own implementation of the dashboards. See the corresponding documentation for that system for more details or caveats.

Note: Some systems might have an earlier prototype “KitchenSinkDashboard” that has not yet been broken out into the individual dashboards. Most of the information is still there, just all in the one dashboard.

<Microservice> Microservice

These dashboards are tailored for an individual microservice. As a rule of thumb they provide a system wide view of all replicas of a given microservice while also letting you isolate a particular instance. They show success/error counts and latencies for the different APIs the microservice offers as well as special metrics that are fundamental to the operation or responsibilities of that particular service.
Spinnaker <Provider> API

These dashboards are tailored for a particular cloud provider. They show a system level perspective of Spinnaker’s interaction with that provider. Depending on the provider, the dashboard details may vary. In general they offer a system wide view while also letting you isolate a particular instance and region, showing success/error counts and latencies for different resource interactions or individual operations. This provides visibility into what your deployment is doing and where any problems might be coming from.
Minimal Spinnaker

The intent of this dashboard is provide the most essential or useful metrics to quickly suggest whether there are any issues and confirm Spinnaker is behaving normally. Your needs may vary so consult each of the other dashboards and consider refining your own. If you do, also consider sharing that back!

Docs: Set up Canary Analysis Support

Mon, 01 Jan 0001 00:00:00 +0000

Setting up automated canary analysis in Spinnaker consists of configuration for Kayenta which does canary analysis by querying metrics, storing the canary configuration templates, and storage for the canary reports in a storage location. Before you can use the canary analysis service, you must configure at least one metrics service, and at least one storage service. The most common setup is to have one metrics service configured (e.g. Stackdriver, Atlas, Prometheus, Datadog or New Relic) and one storage service (e.g. S3, GCS or Minio) configured.

Quick start

If you’d prefer to just get up and running quickly a very simple example with prometheus for metrics and s3 for storing the results/reports looks like the below. Add this to your kayenta-local.yml file.

kayenta:
  ## This doesn't enable metrics, but does enable STORAGE of data
  ## using a minio server as an example.
  ## SQL IS supported, but has not been regularly tested at this time.
  aws:
    accounts:
      - accessKeyId: minio
        bucket: canaries
        endpoint: http://minio:9000
        name: minio
        rootFolder: kayenta
        secretAccessKey: secretKey
        supportedTypes:
          ## ONLY object/config stores are supported.  CloudWatch
          ## Metrics are NOT supported in any OSS distribution
          ## at this point in time.
          - CONFIGURATION_STORE
          - OBJECT_STORE
    enabled: true
  prometheus:
    accounts:
      - endpoint:
          baseUrl: http://prometheus.monitoring:9090
        name: local-prometheus
        supportedTypes:
          - METRICS_STORE
      - endpoint:
          baseUrl: http://prometheus.monitoring:9090
        name: local-prometheus-2
        supportedTypes:
          - METRICS_STORE
    enabled: true
  ## Note though AWS is enabled, you must ALSO have s3 enabled
  s3:
    enabled: true

You also need the following in deck in the settings-local.js to set some defaults depending upon which canary you choose to enable for which purpose

window.spinnakerSettings.canary.metricsAccountName = local-prometheus;
// note that this is the metricStore TYPE.  NOT an account NAME
window.spinnakerSettings.canary.metricStore = prometheus; 
// Storage account allows segmentation of reports/types as needed.
window.spinnakerSettings.canary.storageAccountName = minio;

Specify the scope of canary configs

By default, each canary configuration is visible to all pipeline canary stages in all apps. But you can change that so each canary config will be VISIBLE in the application it was created.

Add the following to your settings-local.js in deck

window.spinnakerSettings.canary.showAllCanaryConfigs = false;

Set it to true to revert to global visibility.

Set the canary judge

The current default judge is NetflixACAJudge-v1.0. The behavior of this judge is described here .

If there are any other judges available in your world, you can set Spinnaker to use it:

Add the following to your settings-local.js in deck

window.spinnakerSettings.canary.defaultJudge = 'static-baseline-judge';

Identify your metrics provider

Some options for metrics are:

atlas (see Netflix Atlas )
datadog
stackdriver
prometheus
newrelic
signalfx

Note: Setup instructions for Atlas and SignalFx are not included in this guide. Refer to the respective provider documentation and Kayenta source code for configuration details.

Provide the default metrics account

Add the account name to use for your metrics provider. This default can be overridden in canary configuration as seen above

Provide the default storage account

Add the account name for your storage provider . This default can be overridden in canary configuration .

This default can be overridden in canary configuration as seen above

Set up canary analysis to use Datadog

If your telemetry provider is Datadog, add the datadog configuration to kayenta-local.yml. You can add/remove additional accounts as needed.

kayenta:
  datadog:
    enabled: true
    accounts:
    - name: my-dd-account
      apiKey: apiKey
      applicationKey: appKey
      endpoint:
        baseUrl: https://api.datadog.com
      supportedTypes:
      - METRICS_STORE

Set up canary analysis for Google

Configure your canary analysis to work with Google, including Stackdriver and GCS . Add the following configuration to kayenta-local.yml. You can add/remove additional accounts as needed.

kayenta:
  google:
    enabled: true
    accounts: 
    - name: my-google-account
      bucket: my-gcs-storage-bucket
      bucket-location: location
      json-path: authData
      project: myproject
      root-folder: sometplace
      supportedTypes:
        - CONFIG_STORE

## Set up canary analysis to use New Relic

If your telemetry provider is New Relic, add the following to your kayenta-local.yml

kayenta:
  newrelic:
    enabled: true
    accounts:
    - name: accountName
      apiKey: nrqlApiKey
      applicationKey: appKey
      endpoint:
        baseUrl: https://insights-api.newrelic.com
      supportedTypes:
        - METRICS_STORE

More providers/configuration options are available in the kayenta codebase. NOT all providers support configuration stores. You can see configuration references for most providers in their configuration files like the newrelic configuration files These generally are found in the com/netflix/kayenta/<provider>/config/ folder for each library type.

Docs: Additional Features

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Clouddriver Account Management

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker 1.28 introduced an alpha API for loading, storing, updating, and otherwise managing Clouddriver account configurations from a database. Combined with a supported external secrets manager , account credentials can be provided by end users when setting up an account configuration.

The following table lists the Clouddriver account types that currently support the account management API. Note that the type discriminator is used as the value for the key @type in the JSON representation of an account definition.

Cloud Provider	Type Discriminator	Account Definition Class
Kubernetes	`kubernetes`	`com.netflix.spinnaker.clouddriver.kubernetes.config.KubernetesAccountProperties.ManagedAccount`
AWS	`aws`	`com.netflix.spinnaker.clouddriver.aws.security.config.AccountsConfiguration.Account`
Google	`google`

Other account providers should support dynamic accounts at this point

Enabling account management

As an alpha API and feature, this must be explicitly enabled in your clouddriver-local.yml configuration by adding:

account:
  storage:
    enabled: true

Note that credentials polling must be enabled for each cloud provider type being managed by this API. For example, to enable credentials polling for Kubernetes with a polling period of one minute, add the following configuration to your clouddriver-local.yml configuration.

credentials:
  poller:
    enabled: true
    types:
      kubernetes:
        reloadFrequencyMs: 60000

The default implementation of account storage is through SQL. This can be configured in your clouddriver-local.yml file as well.

sql:
  enabled: true
  connectionPools:
    accounts:
      dialect: Postgres
      jdbcUrl: jdbc:postgresql:clouddriver

Use of the default connection pool is also allowed when all Clouddriver SQL data is stored in the same database.

Account management API

Accounts can be created, updated, and deleted through the /credentials Gate endpoints. Any authenticated user may create an account, and write permissions are required to update, delete, or otherwise use that account. Secrets should be specified using either an encrypted:... URI for strings or an encryptedFile:... URI for files. For example, to create a new Kubernetes account, suppose we have a kubeconfig file for a Kubernetes service account stored in AWS Secrets Manager in the us-west-2 region named my-kubernetes-credentials. Then the following REST API call may be used for configuring an account by sending a POST request to GATE_URL/credentials.

{
  "@type": "kubernetes",
  "name": "my-kubernetes-account",
  "permissions": {
    "READ": [
      "my-group"
    ],
    "WRITE": [
      "my-group"
    ]
  },
  "context": "eks",
  "namespaces": [
    "default"
  ],
  "kubeconfigContents": "encrypted:secrets-manager!r:us-west-2!s:my-kubernetes-credentials"
}

Sending the same request body as a PUT request to GATE_URL/credentials will update the existing account definition. Finally, sending a DELETE request to GATE_URL/credentials/my-kubernetes-account will delete the my-kubernetes-account account definition. Both of these endpoints require write permission on the affected account.

Docs: External Account Configuration

Mon, 01 Jan 0001 00:00:00 +0000

This page describes Spinnaker’s external configuration feature. Using external configuration, you can manage account configuration (either complete account configuration, or only secrets such as account passwords) externally from Spinnaker.

External configuration uses the open-source Spring Cloud Config project, which includes a configuration server and client libraries to support centralized external configuration in a distributed system. The Config Server connects to a remote configuration repository and serves stored configuration properties to client applications.

Configuration properties can be organized by Spring profile and application name and can be stored in any of a number of backends, including GitHub, HashiCorp Vault, AWS S3, JDBC, CredHub, and others. For more information about Spring Cloud Config, see the documentation .

The following table lists the Spinnaker services that currently incorporate external configuration.

Service	Account Types	Notes
Clouddriver	Cloud provider, artifact	Automatic configuration refreshing is supported for Cloud Foundry and Kubernetes cloud provider accounts only.
Echo	Pub/Sub
Igor	CI system, SCM (e.g. Git)

For Clouddriver, support for external configuration was added in Spinnaker 1.15. For Echo and Igor, support was added in Spinnaker 1.16.

Enabling external configuration

To enable external configuration via the Config Server, add Spring Cloud Config configuration properties, which are under spring.cloud.config.server, and include settings for the backend you wish to use.

Place this configuration in a YAML file called spinnakerconfig.yml, that must be deployed as a config file in /opt/spinnaker/config/ for your services. This is typically done via a ConfigMap volume mount or similar. NOTE the file MUST be named spinnakerconfig.yml

Using a Git backend

To use a Git backend, configure the settings under spring.cloud.config.server.git. Your configuration might look like the following example:

spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/example/spinnaker-config
          basedir: /tmp/config-repo
          refresh-rate: 10

The repository will be cloned to the directory specified by the git.basedir property. If using multiple Git repositories, you must give each repository a unique value for basedir.

External SCC server support

An alternative is to use an externally managed spring-cloud-config server You can add configuration like this to igor-local.yml or a spinnaker.yml file:

spring.config.import=optional:configserver:http://myconfigserver.com

For more information, see the spring cloud config docs

Using a Vault backend

To use a HashiCorp Vault backend, configure the settings under spring.cloud.config.server.vault. Your configuration might look like the following example:

spring:
  profiles:
    include: vault
  cloud:
    config:
      server:
        vault:
          host: config-vault.example.com
          port: 8200
          backend: secret
          kvVersion: 2
          defaultKey: clouddriver
          token: [vault access token]

The above example configures authentication for Vault using using a static authentication token, via the setting of the spring.cloud.config.server.vault.token property. You can use other methods of configuring authentication for Vault. See the relevant Spring Cloud Config Server documentation for more information.

Using an S3 backend

To use an AWS S3 backend, enable the awss3 profile and configure the settings under spring.cloud.config.server.awss3. Your configuration might look like the following example:

spring:
  profiles:
    include: awss3
  cloud:
    config:
      server:
        awss3:
          region: us-west-2
          bucket: my-bucket

To configure authentication, you must use AWS’s Default Credential Provider Chain. See the AWS SDK documentation for more information.

For information about configuring other supported Config Server backends, see the Spring Cloud Config Server documentation .

Managing external configuration

You can use the Config Server to manage all of your account configuration or to manage secrets. You can also use the Config Server to retrieve configuration files or encrypted configuration properties.

Complete account configuration

If you wish to use external configuration to store complete configuration for your accounts, you can move the account configuration to files stored in a Config Server backend, such as a Git repository or HashiCorp Vault server.

To store complete account configuration in Git, create a Git repository and within it, place a file named spinnaker.yml. The file should contain the account configuration, as in the following example of Echo configuration for a Pub/Sub account:

pubsub:
  enabled: true
  google:
    enabled: true
    pubsubType: GOOGLE
    subscriptions:
    - name: my-gcs-subscription
      project: my-project
      subscriptionName: my-gcs-subscription
      jsonPath: /path/to/my-gce-account.json
      ackDeadlineSeconds: 10
      messageFormat: GCS
    publishers: []

To store complete account configuration in HashiCorp Vault, create a JSON file (for example, echo.json) containing the account configuration, as in the following example:

{
   "pubsub": {
      "enabled": true,
      "google": {
         "enabled": true,
         "pubsubType": "GOOGLE",
         "subscriptions": [
            {
               "name": "my-gcs",
               "project": "My-Project",
               "subscriptionName": "my-gcs-subscription",
               "jsonPath": "/home/spinnaker/.hal/default/staging/dependencies/my-gce-account.json",
               "ackDeadlineSeconds": 10,
               "messageFormat": "GCS"
            }
         ],
         "publishers": []
      }
   }
}

Then use the vault command-line interface tool to store this JSON in Vault:

$ vault kv put secret/echo @echo.json

Secrets

If you wish to use spinnaker local configuration to manage accounts but store secrets (such as account passwords) externally, you can move the account secrets into files stored in a Config Server backend, such as a Git repository or HashiCorp Vault server. You can then use Spring property placeholders to reference the secret values.

If you are configuring a Jenkins account in Igor, your igor-local.yml file might look like the following example:

jenkins:
  enabled: true
  masters:
  - name: my-jenkins-master
    permissions: {}
    address: http://my-jenkins.ci.example.com
    username: myuser
    password: ${ci.my-jenkins.password}

In the Config Server backend (such as a Git repository), the account secrets file might look like the following example:

ci:
  my-jenkins:
    password: 'secret'

Configuration files

Some account configuration, including configuration for the Kubernetes, Google Compute Engine, and Google App Engine cloud providers in Clouddriver and Google Cloud Pub/Sub accounts in Echo, allows credentials to be loaded from files that are separate from the YAML configuration. You can load these separate files from an external source using the Config Server’s Resource abstraction. In the service YAML configuration, prefix a file path with configserver: to indicate that the file should be retrieved by the Config Server.

External configuration files (such as kubeconfig files and GCP JSON files) can only be loaded from S3 or Git (not from Vault).

If you are configuring a Kubernetes cloud provider account and want to load an external kubeconfig.yml file using Config Server, your account configuration might look like the following example:

kubernetes:
  enabled: true
  accounts:
    - name: default
      kubeconfigFile: configserver:kubeconfig.yml
      dockerRegistries:
        - accountName: dockerhub
          namespaces: []
      context: default

Encrypted values

The Config Server can store encrypted secrets in its backend and decrypt the secret values when serving them to clients. You can use this to store encrypted Spinnaker account secrets in a Config Server backend (such as a Git repository).

To enable Config Server decryption, configure the encryption key in your spinnaker-config-local.yml, along with the Config Server backend configuration properties. You can configure the encryption key using properties under encrypt.

When using Config Server encryption or decryption, your configuration might look like the following example:

spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/example/spinnaker-config
          basedir: /tmp/config-repo
          refresh-rate: 10
encrypt:
  key: mykey

For more information about using the Config Server’s encryption and decryption features, see the Spring Cloud Config Server documentation .

External configuration refresh

If you have enabled caching , Spinnaker will refresh its external configuration automatically on the same time interval as the Clouddriver agent caches. When using the default in-memory caching, this interval defaults to 60 seconds. When using Redis caching , you can configure this interval using the redis.poll.intervalSeconds property.

Spinnaker – Configure Everything Else

Docs: Server Group Launch Settings

Docs: Configure Artifacts

Docs: Bakery

Packer Templates

Image Provider Configuration

Docs: Security

For reference…

Next steps

Docs: Set Up Triggers

Docs: Install and Configure Spin CLI

Install spin

On Linux

On MacOS

On Windows

Configure spin

X.509

OAuth2

Client ID and Client Secret

Access and Refresh Token

Google Service Account

Basic

LDAP/Active Directory

Global Flags

Output Formatting

Docs: CI

Supported CI systems

Docs: Logging

Basic logging

Advanced logging with logback

Installation

Set log level per class

Filter log keys

Docs: Monitoring

OpenTelemetry (Recommended)

Why this works with Spinnaker

Minimal configuration

New Relic note

Only one Java agent per JVM

Legacy: Spinnaker Observability Plugin

Legacy

Configuring the Spinnaker Observability Plugin

Consuming metrics

Types of metrics

Example

Supplied dashboards

Types of dashboards

Docs: Set up Canary Analysis Support

Quick start

Specify the scope of canary configs

Set the canary judge

Identify your metrics provider

Provide the default metrics account

Provide the default storage account

Set up canary analysis to use Datadog

Set up canary analysis for Google

Docs: Additional Features

Docs: Clouddriver Account Management

Enabling account management

Account management API

Docs: External Account Configuration

Enabling external configuration

Using a Git backend

External SCC server support

Using a Vault backend

Using an S3 backend

Managing external configuration

Complete account configuration

Secrets

Configuration files

Encrypted values

External configuration refresh

Install `spin`

Configure `spin`