Spinnaker – Security

Docs: Authorization (RBAC)

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Fiat (Fix it Again Travis) is the authorization (authz) microservice of Spinnaker. It can grant access to users to execute pipelines, view infrastructure, etc. It is disabled by default. Much like authentication, Spinnaker allows for a variety of pluggable authorization mechanisms.

With Fiat, you can…

Restrict access to specific accounts .
Restrict access to specific applications .
Run pipelines using automated triggers against access-controlled applications.
Use and periodically update user roles from a backing role provider .

Permissions can be attached to applications and (provider) accounts. A permission associates a role with one of these options: READ, WRITE, or EXECUTE (for apps only).

Important notes

Keep these in mind as you consider your authorization strategy:

Fiat’s authorization model is an approve list that is open by default. In other words, when a resource does not define who is allowed to access it, it is considered unrestricted. This means:
- If an account is unrestricted, any user with access to Spinnaker can deploy a new application to that account.
- If an application is unrestricted, any user with access to Spinnaker can deploy that application into a different account. They may also be able to see basic information like instance names and counts within server groups.
Every permission in Spinnaker is granted to a role. Individual users cannot be granted permissions. You also grant Super admin access to a role. You may see discussions of users in Fiat’s implementation but it’s just an optimization in the storage to not recompute user → roles → permissions.
Account and application access control can be confusing unless you understand the core relationship: accounts can contain multiple applications, and applications can span multiple accounts. Giving access to an account does not grant access to the application and vice versa. Sometimes you need both permissions to perform certain actions.

Requirements

Authentication successfully setup in Gate.
An external role provider from one of the following:
- Google Groups via a G Suite Account
  - With access to the G Suite Admin console
- GitHub Organization
- LDAP server
- SAML Identity Provider (IdP) that includes groups in the assertion
  
  SAML roles are fixed at login time, and cannot be changed until the user needs to reauthenticate.
- OAUTH2 with groups in the claims
Enable fiat in the spinnaker.yml file. This enables services to query fiat for auth information.
(Highly Suggested) All Spinnaker component microservices are either:
- Firewalled off as a collective group, or:
- Use mutual TLS authentication:

Implementation

Accounts

Because Clouddriver is the source of truth for cloud accounts, Fiat reaches out to Clouddriver to gather the list of available accounts. There are two types of access restrictions to an account: READ and WRITE. Users must have at least one READ permission of an account to view the account’s cloud resources, and at least one WRITE permission to make changes to the resources.

Every cloud provider, and many OTHER account types like those in igor for CI systems can set permissions. You’ll set these permissions by modifying the account with a permissions set of entries that contain the roles/groups that you want to grant READ/WRITE permissions.

google:
  accounts:
  - name: my-google-account
    permissions:
      READ:
      - role1
      - role2
      WRITE:
      - role1
      - role2

NOTE That READ permissions ARE NOT implicitly granted by giving WRITE permissions. AS SUCH it’s HIGHLY recommended to ALWAYS add READ to any WRITE access.

Applications

Applications have three levels of permissions: EXECUTE, READ and WRITE permissions.

By DEFAULT, READ permissions ALSO grant EXECUTE permissions. To adjust this,

Make sure you have EXECUTE set as needed for your applications
Flip the default behavior across all applications to only grant WRITE users implicit EXECUTE access by setting the following property in fiat-local.yml:

fiat:
  executeFallback: 'WRITE'

Examples of required permissions

To delete a load balancer in account Z, you need to have WRITE permission on the account.
To update a pipeline in an app, you need WRITE permission on that app.
You can run a pipeline with just the EXECUTE permission.
To successfully run a pipeline in app X that deploys to account Y, you need (at least) EXECUTE on the app X and WRITE on the account Y.

Role Providers

In Spinnaker there are a few ways you can associate a role with a user:

With a YAML file : contains user ↔ role mapping. A YAML parseable map with structure [user]: list of roles
Via GitHub Teams : roles are the teams a user belongs to in a configured Org
Via Google Groups : roles are mapped (see settings) from the Google directory
Via LDAP : roles are searched in LDAP from the user
Via SAML Groups (also covers OAuth ONLY with OIDC): The authentication method can also bring its own roles. In this case, roles are referred in Fiat as EXTERNAL. They can be used in addition to authorization roles.

In all these methods, users are referenced by a userId which is determined by the authentication method of your choice.

Effects of restrictions

Because of the new access restrictions, https://localhost:9000/#/applications should no longer list applications that have been restricted. Even navigating to the previously accessible page should be denied:

Automated pipeline triggers

A popular feature in Spinnaker is the ability to run pipelines automatically based on a triggering event, such as a git push or a Jenkins build completing. When pipelines run against accounts and applications that are protected, it is necessary to configure them with enough permissions to access those protected resources. This can be done in two ways:

Using Pipeline Permissions - this is an automatic creation of service accounts on demand.
Using a Fiat service account

Reference documentation

Deeper details on Authorization in Spinnaker

Docs: Administrator functionality

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

In Spinnaker, it is possible to define that users belonging to a certain role are considered “Administrators”. This virtually removes all READ/WRITE restrictions to accounts and applications for these users.

This feature gives admins an immense amount of power. Proceed with caution.

Enable and Configure Admin functionality

Manually add configuration in Fiat

In the fiat-local.yml config file, add the following:

fiat:
  admin:
    roles:
      - devops-admin

Docs: Authentication

Mon, 01 Jan 0001 00:00:00 +0000

There are a lot of moving parts involved with getting authentication to work. This page describes the basics and tools that make setup easier to configure and test.

Introduction

There are three basic systems involved with Spinnaker’s authentication workflow: your identity provider, Gate, and Deck. The changes will primarily be made to either your identity provider or Gate. Deck itself will not require changes or updates, but it’s useful to understand how all three parts interact.

graph LR classDef default fill:#d8e8ec,stroke:#7a8288; linkStyle default stroke:#7a8288, stroke-width:2px, fill:none; gate(Gate) idp(IdentityProvider) deck(Deck/Browser) deck-->gate gate-->deck deck-->idp idp-->deck

Deck: Spinnaker’s UI. Consists of a set of static HTML, JavaScript, and CSS files. Generally served from an Apache server, but there is nothing special about Apache that makes Deck work. Replace with your favorite HTTP(S) server if you’d like. The Javascript being an SPA Single Page Application is going to do the communication with your identity provider. Which IDP is determined by Gate.
Gate: Spinnaker’s API Gateway. All traffic (including traffic generated from Deck) flows through Gate. It is the point at which authentication is confirmed and one point (of several) where authorization is enforced.
Identity Provider: Your organization’s OAuth 2.0, SAML 2.0, or LDAP service. X.509 client certificates can be used in addition to any of these services, or used standalone.

Incognito mode

{:width=“50px”}

Getting the authentication working rarely happens on the first try. Each login attempt during configuration (or development) causes a new session to be established in Gate’s session repository. Re-using these sessions is undesirable when testing configuration changes.

We highly recommend using Google Chrome’s Incognito mode when working with configuration changes.

Open a new Incognito window.
Navigate to your Spinnaker’s Deck endpoint.
Observe behavior and make configuration change. Restart affected Spinnaker service.
Close Incognito window.
Repeat from step 1.

A common issue with Incognito windows is that they all share the same cookie jar. This means that when you want to test a new configuration change, you need to close all Incognito windows. Otherwise, the session cookie will not be deleted.

Available Options

Methods
- OAuth 2.0/OIDC - The main examples are Google & GitHub endpoints.
- SAML - Lots of examples on this with one of the most prevalent being Okta.
- LDAP - This covers Active Directory and other LDAP servers, such as OpenLDAP.
- X.509 - Often used for client or application communications. Can operate in conjunction with other authentication methods.

Next steps

Set up Authorization .

Learn how to configure Spinnaker to communicate over SSL .

Docs: SSL

Mon, 01 Jan 0001 00:00:00 +0000

This section covers communication with Spinnaker from parties external to your Spinnaker instance. That is, any requests between…

your browser and the Spinnaker UI (Deck)
Deck and Gate (the API gateway)
any other client and Gate

Network configurations

Warning: Many operators like to get authentication working before adding HTTPS, but experience bears out that the transition is not smooth. We recommend you implement at least a temporary SSL solution first.

Load Balancer-terminated SSL

A common practice is to offload SSL-related bits to outside of the server in question. This is fully supported in Spinnaker, but it does affect the authentication configuration slightly. See your authentication method for specifics.

During certain authentication workflows, Gate makes an intelligent guess on how to assemble a URI to itself, called the redirect_uri. Sometimes this guess is wrong when Spinnaker is deployed in concert with other networking components, like an NGINX ingress controller, an SSL-terminating load balancer.

To manually set the URLs which spinnaker runs, you’ll need to adjust a few files: Deck’s configuration for the API services in settings.js

var gateHost = 'http://example.com/api/v1';

and the spinnaker.yaml which defines the “UI” endpoint

services:
   ## Change to whatever domain you're using.  This enables redirect on logins post auth
   deck:
      baseUrl: http://example.com/
      enabled: true

Additionally, these are some recommended defaults to pass protocol and IP information downward. Add this to your gate-local.yml file:

server:
  tomcat:
    protocolHeader: X-Forwarded-Proto
    remoteIpHeader: X-Forwarded-For
    internalProxies: .*
    ## This may or may not be needed depending upon your environment.  
    ## See https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/catalina/filters/RemoteIpFilter.html
    httpsServerPort: X-Forwarded-Port

This is already setup by default in the example installation repository

Server-terminated SSL

Each private key, and several other of the sensitive files generated in this doc will have a password/passphrase. These are the password/passphrases bash variables used in this doc (please substitute your own passwords/passphrases):

CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY
DECK_KEY_PASSWORD=SOME_PASSWORD_FOR_DECK_KEY
GATE_KEY_PASSWORD=SOME_PASSWORD_FOR_GATE_KEY
JKS_PASSWORD=SOME_JKS_PASSWORD
GATE_EXPORT_PASSWORD=SOME_PASSWORD_FOR_GATE_P12

In addition, in many of the calls below, if you want openssl or keytool to prompt for the key rather than providing them via the CLI, you can just remove the relevant flag.

Terminating SSL within the Gate server is the de-facto way to enable SSL for Spinnaker. This works with or without a load balancer proxying traffic to this instance.

1. Generate key and self-signed certificate

We will use openssl to generate a Certificate Authority (CA) key and a server certificate. These instructions create a self-signed CA. You might want to use an external CA, to minimize browser configuration, but it’s not necessary (and can be expensive).

Use the steps below to create a certificate authority. (If you’re using an external CA, skip to the next section.)

It will produce the following items:

ca.key: a pem-formatted private key, which will have a pass phrase.
ca.crt: a pem-formatted certificate, which (with the private key) acts as a self-signed Certificate Authority.

Create the CA key. This command below references the pass phrase environment variable used to encrypt the key ca.key.
```
openssl genrsa \
  -des3 \
  -out ca.key \
  -passout pass:${CA_KEY_PASSWORD} \
  4096
```

Self-sign the CA certificate. This command below references the pass phrase environment variable used to decrypt the key ca.key.

openssl req \
  -new \
  -x509 \
  -days 365 \
  -key ca.key \
  -out ca.crt \
  -passin pass:${CA_KEY_PASSWORD}

2. Create the server certificate(s)

If you have different DNS names for your Deck and Gate endpoints, you can either create a certificate with a CN and/or SAN that covers both DNS names, or you can create two certificates. This document details creating these items, signed by the self-signed CA cert created above:

deck.key: a pem-formatted private key, which will have a pass phrase.
deck.crt: a pem-formatted certificate, which (with the private key) serves as the server certificate used by Deck.
gate.jks: a Java KeyStore (JKS) that contains the following:
- The certificate and private key for use by Gate (with alias gate)
- The certificate for the Certificate Authority created above (with alias ca)

Additionally, these intermediate files will be created:

deck.csr: a Certificate Signing Request file, generated from deck.key and used in conjunction with ca.key to sign deck.crt
gate.csr: a Certificate Signing Request file, generated from gate.key and used in conjunction with ca.key to sign gate.crt
gate.crt: a pem-formatted certificate for Gate. This will be converted to .p12 and imported into the JKS.
gate.p12: a p12-formatted certificate and private key for Gate. This will be imported into the JKS.

Create a server key for Deck. Keep this file safe!

This command below references the pass phrase environment variable used to encrypt the key deck.key.
```
openssl genrsa \
  -des3 \
  -out deck.key \
  -passout pass:${DECK_KEY_PASSWORD} \
  4096
```
Generate a certificate signing request (CSR) for Deck. Specify localhost or Deck’s eventual fully-qualified domain name (FQDN) as the Common Name (CN).

This command below references the pass phrase environment variable used to decrypt the key deck.key.
```
openssl req \
  -new \
  -key deck.key \
  -out deck.csr \
  -passin pass:${DECK_KEY_PASSWORD}
```
Use the CA to sign the server’s request and create the Deck server certificate (in pem format). If using an external CA, they will do this for you.

This command below references the pass phrase environment variable used to decrypt the key ca.key.
```
openssl x509 \
  -sha 256 \
  -req \
  -days 365 \
  -in deck.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out deck.crt \
  -passin pass:${CA_KEY_PASSWORD}
```
Create a server key for Gate. This command below references the pass phrase environment variable used to encrypt the key gate.key. Keep this file safe!
```
openssl genrsa \
  -des3 \
  -out gate.key \
  -passout pass:${GATE_KEY_PASSWORD} \
  4096
```
Generate a certificate signing request for Gate. Specify localhost or Gate’s eventual fully-qualified domain name (FQDN) as the Common Name (CN).

This command below references the pass phrase environment variable used to decrypt the key gate.key.
```
openssl req \
  -new \
  -key gate.key \
  -out gate.csr \
  -passin pass:${GATE_KEY_PASSWORD}
```
Use the CA to sign the server’s request and create the Gate server certificate (in pem format). If using an external CA, they will do this for you.

This command below references the pass phrase environment variable used to decrypt the key ca.key.
```
openssl x509 \
  -sha 256 \
  -req \
  -days 365 \
  -in gate.csr \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -out gate.crt \
  -passin pass:${CA_KEY_PASSWORD}
```
Convert the pem format Gate server certificate into a PKCS12 (p12) file, which is importable into a Java Keystore (JKS).

This command below references the pass phrase environment variable used to decrypt the key gate.key, and then for an import/export password to use to encrypt the p12 file.
```
openssl pkcs12 \
  -export \
  -clcerts \
  -in gate.crt \
  -inkey gate.key \
  -out gate.p12 \
  -name gate \
  -passin pass:${GATE_KEY_PASSWORD} \
  -password pass:${GATE_EXPORT_PASSWORD}
```
This creates a p12 keystore file with your certificate imported under the alias “gate”.

Create a new Java Keystore (JKS) containing your p12-formatted Gate server certificate.

Because Gate assumes that the keystore password and the password for the key in the keystore are the same, we must provide both via the command line. This will prompt for the import/export password used to encrypt the p12 file.

keytool -importkeystore \
  -srckeystore gate.p12 \
  -srcstoretype pkcs12 \
  -srcalias gate \
  -destkeystore gate.jks \
  -destalias gate \
  -deststoretype pkcs12 \
  -deststorepass ${JKS_PASSWORD} \
  -destkeypass ${JKS_PASSWORD} \
  -srcstorepass ${GATE_EXPORT_PASSWORD}

Import the CA certificate into the Java Keystore.

This will prompt for a password to encrypt the Keystore file.

keytool -importcert \
  -keystore gate.jks \
  -alias ca \
  -file ca.crt \
  -storepass ${JKS_PASSWORD} \
  -noprompt

Verify the Java Keystore contains the correct contents.
```
keytool \
  -list \
  -keystore gate.jks \
  -storepass ${JKS_PASSWORD}
```
It should contain two entries:
- gate as a PrivateKeyEntry
- ca as a trustedCertEntry

Voilà! You now have a Java Keystore with your certificate authority and server certificate ready to be used by Spinnaker Gate, and, separately, a pem-formatted key and server certificate ready to be used by Spinnaker Deck!

3. Configure SSL for Gate and Deck

With the above certificates and keys in hand, you can setup SSL for Gate and Deck .

For Gate, add the following to the gate-local.yml

server:
  ssl:
    enabled: true
    crlFile: <optional>
    key-store: <reference to [service].p12>
    key-store-type: PKCS12
    key-store-password: <[SERVICE]_KEY_PASS>

For Deck, you’ll want to adjust the deck apache configuration to set the cert information. This file should be mounted to /etc/apache2/sites-enabled/spinnaker.conf on the deck pod.

<VirtualHost 0.0.0.0:9000>
 <IfModule ssl_module>
      SSLEngine on
      SSLCertificateFile "/opt/spinnaker/config/ssl/deck.crt"
      SSLCertificateKeyFile "/opt/spinnaker/config/ssl/deck.key" 
  </IfModule>

  DocumentRoot /opt/deck/html
  <Directory "/opt/deck/html/">
     Require all granted
  </Directory>
</VirtualHost>

Make sure the certs are stored in a secret volume that is mounted into the /opt/spinnaker/config/ssl folder. Deck does NOT support encrypted secrets as it is a plain apache2 server, and as such, you can configure additional settings as needed. You can mount these in other locations as needed. The above

Verify your SSL setup

To verify that you’ve successfully set up SSL, try to reach one of the endpoints, like Gate or Deck, over SSL.

Troubleshooting

If you have problems…

Are you using https?
Do you have certs correct?
Try curl or openssl commands against services to validate your certificate chain

Using a custom CA for internal communications

There are a lot of places in Spinnaker which support the ability to configure custom Java trust/key stores for organizations who use internally signed certificates. In some cases, however, this isn’t supported yet but you still need to talk to a service which serves one of these certificates. This section will show you how to import your certificate into a Java trust/key store and configure a Spinnaker service with it.

Create a temporary copy of your system’s Java trust/key store and import your internal certificate. If you’re on a Mac, this will be located at /usr/libexec/java_home/)/jre/lib/security/cacerts. It will be different on Linux.

mkdir /tmp/custom-trust-store`
cp {path-to-cacerts} /tmp/custom-trust-store
keytool import -alias custom-ca -keystore /tmp/custom-trust-store/cacerts -file {your-internal-certificate}

kubectl create secret generic -n {your-spinnaker-namespace} internal-trust-store \
   --from-file=cacerts=/tmp/custom-trust-store/cacerts

Configure a Spinnaker service with the new trust/key store using a volume mount. Mount the cacerts file into a secret that’s then mounted to replace the operating system cacerts. This would be the file: /etc/ssl/certs/java/cacerts by default.

The Spinnaker component for which you configured the cacerts file should now be using the new trust/key store by default.

Next steps

Choose an authentication method .