Spinnaker – Configure Artifacts

Docs: Configure a Git Repo Artifact Account

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Each time a pipeline needs a Git repo artifact during execution, Clouddriver clones the entire repo, sends the repo artifact to the pipeline, and then deletes the cloned repo immediately.

Spinnaker includes a feature for caching a Git repo artifact. Clouddriver clones the Git repo the first time a pipeline needs it and then caches the repo for a configured retention time. Each subsequent time the pipeline needs to use that Git repo artifact, Clouddriver does a git pull to fetch updates rather than cloning the entire repo again. This behavior is especially useful if you have large repositories. Clouddriver deletes the cloned Git repo when the configured retention time expires.
This is an opt-in feature that is disabled by default. See the Enable <code>git pull</code> support section for details.

Prerequisites

You need a Git account.

Download credentials

Token auth

Generate an access token for your Git provider (eg, GitHub or GitLab ). The token requires the repo scope.

User-Password auth

Create a username-password file, with contents in the following format:
```
<username>:<password>
```

SSH key auth

Create an SSH key for your user or add to the ssh allowed keys configuration.

Store the credentials for user

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it with one of the auth options

artifacts:
  enabled: true
  gitrepo:
    enabled: true
    accounts:
    - name: my-git-account
      username: git
      password: password
      tokenFile: /mnt/someplace/tokenFile
      sshPrivateKeyFilePath: SSH_KEY_FILE
      sshPrivateKeyPassphrase: somePassPhrase
      sshPrivateKeyPassphraseCmd: <some command>
      sshKnownHostsFilePath: KNOWN_HOSTS_FILE
      sshTrustUnknownHosts: false

Enable `git pull` caching support

This feature is disabled by default. To enable git pull support, add the following section to your clouddriver-local.yml file:

artifacts:
  enabled: true
  gitrepo:
    enabled: true
    clone-retention-minutes: 60
    clone-retention-max-bytes: 104857600

clone-retention-minutes: Default: 0. How much time to keep clones. Values are:
- 0: no retention.
- -1: retain forever.
- any whole number of minutes, such as 60.
clone-retention-max-bytes: Default: 104857600 (100 MB). Maximum amount of disk space to use for clones. When the maximum amount of space is reached, Clouddriver deletes the clones after returning the artifact to the pipeline, just as if retention were disabled.

Other options

See the GitRepo Artifact Account code for additional options.

Docs: Configure a GitHub artifact account

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

You need a GitHub account.

Download credentials

Start by generating an access token for GitHub. The token requires the repo scope.

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

Add the below to clouddriver-local.yml to enable artifacts and add a github account

artifacts:
  enabled: true
  github:
    enabled: true
    accounts:
    - name: my-github-account
      tokenFile: /mnt/some/file|secretReference

There are more options available if you look at the github account definition code

Docs: Configure a GitLab artifact account

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

You need a GitLab account.

Download credentials

Start by generating an access token for GitLab.

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

All that’s required are the following configuration:

artifacts:
  enabled: true
  gitlab:
    enabled: true
    accounts:
    - name: my-gitlab-account
      token: patToken
      tokenFile: /mnt/someplace/tokenFile

There are more options including URL restrictions that can be set to allow only internal gitlab hosts. See the gitlab account definition for information.

Docs: Configure Bitbucket Artifact Credentials

Mon, 01 Jan 0001 00:00:00 +0000

If the files are hidden behind basic auth, you can configure an artifact account with the needed credentials to read your artifact.

Download credentials

Collect your basic auth $USERNAME and $PASSWORD
Pick a $USERNAME_PASSWORD_FILE location on your disk

Run:

echo ${USERNAME}:${PASSWORD} > $USERNAME_PASSWORD_FILE

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

Enable the Bitbucket artifact provider in clouddriver-local.yml and add an artifact account:

artifacts:
  enabled: true
  bitbucket: 
    enabled: true
    accounts:
    - name: my-bitbucket-account
      username-password-file: /mnt/password/file

Add the password file either to a secrets manager or to a volume mounted into the clouddriver pod by modifying the deployment.yaml for clouddriver.

more configuration for an account is available looking at the account code.

Docs: Configure Helm Artifact account

Mon, 01 Jan 0001 00:00:00 +0000

The provider can be an Artifactory like Nexus , JFrog , or Chartmuseum .

If the files are hidden behind basic auth, you can configure an artifact account with the needed credentials to read your artifacts. Basic auth is the only authentication mechanism supported for accessing a Helm artifact account.

Setup usage in pipeline executions

Download credentials

Collect your basic auth $USERNAME and $PASSWORD
Pick a $USERNAME_PASSWORD_FILE location on your disk

Run:

echo ${USERNAME}:${PASSWORD} > $USERNAME_PASSWORD_FILE

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

Add this to clouddriver-local.yml

artifacts:
  enabled: true
  helm:
    enabled: true
    accounts:
    - name: my-helm-account
      username-password-file: <secret-manager|/mnt/secret/filename>

Trigging from helm changes

To enable helm changes to TRIGGER a pipeline, you need to add some configuration to igor which then polls clouddriver for accounts and changes and tracks updates.

Add this to igor-local.yml

helm:
  enabled: true

Igor will then poll clouddriver for helm artifact accounts and look for new helm resources for that account. When it finds new helm resources, it will trigger any pipeline with a helm trigger configured, and then track completion of triggering on the new helm so it does not refire new executions. See the common polling in igors README for more information.

There are more options in the code base if you need more control over your configuration.

Docs: Configure HTTP Artifact Credentials

Mon, 01 Jan 0001 00:00:00 +0000

If the files are hidden behind basic auth, you can configure an artifact account with the needed credentials to read your artifact. If not, no further configuration is needed, Spinnaker can automatically add a no-auth-http-account for this purpose.

You can configure more than one artifact account, each with separate credentials. Specify which account to use in the configuration for the stage that reads the data. If you have only one such account configured, the stage config for this is hidden, and the single account is automatically used.

Download credentials

Collect your basic auth $USERNAME and $PASSWORD
Pick a $USERNAME_PASSWORD_FILE location on your disk

Run:

echo ${USERNAME}:${PASSWORD} > $USERNAME_PASSWORD_FILE

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

Add the following to clouddriver-local.yml

artifacts:
  enabled: true
  http:
    enabled: true
    accounts:
    - name: some-account
      username-password-file: </mnt/file|secretReference>

There are more options in the code including the ability to set restrictions on endpoints receiving these requests. IT IS HIGHLY recommended that you limit these requests to specific trusted URLs since it will send credentials to these URLs.

Docs: Configure Maven Artifact Credentials

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker can be configured to deploy artifacts stored in a Maven repository.

You can configure more than one artifact account, each with separate credentials. Specify which account to use in the configuration for the stage that reads the data.

Download credentials

Maven accounts at this time do not support credential authorization. We’d welcome PRs to enable this and get maven closer to full artifact support.

Add the account and enable it

artifacts:
  enabled: true
  maven:
    enabled: true
    accounts:
    - name:  nexus-internal
      repositoryUrl: https://my.repo.example.com

Docs: Configure S3 Artifact

Mon, 01 Jan 0001 00:00:00 +0000

All that’s required are the following values:

API_ENDPOINT=
API_REGION=
REGION=
ARTIFACT_ACCOUNT_NAME=my-s3-account

Download credentials

Create a user with an accessKey or secretKey. Otherwise, spinnaker will use the credentials of the clouddriver pod to attempt to access the s3 resource.

Add the accesskey and secret key either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

Add the following to clouddriver-local.yml

artifacts:
  enabled: true
  s3: 
    enabled: true
    accounts:
    - name: my-local-bucket
      region: region
      apiEndpoint: swift-or-ceph-s3-endpoint
      apiRegion: region
      awsAccessKeyId: accessKey
      awsSecretAccessKey:  secretAccessKey
      signerOverride: signerOverride

Note: Setting apiEndpoint & apiRegion is optional. It is needed only if you are using a S3 clone such as Minio.

Additional configuration properties are avaialble in the source code for things like timeout configurations and similar.

Docs: Configuring GCS Artifact Credentials

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker stages that read data from artifacts can consume GCS objects as artifacts.

Prerequisites

You need a Google Cloud Platform (GCP) project to host a bucket in. The next steps assume you’ve already created a project , and installed <code>gcloud</code> . You can check that gcloud is installed and authenticated by running:

gcloud info

Download credentials

Spinnaker needs a service account to authenticate as against GCP, with the roles/storage.admin role enabled. If you don’t already have such a service account with the corresponding JSON key downloaded, you can run the following commands to do so:

SERVICE_ACCOUNT_NAME=spin-gcs-artifacts-account
SERVICE_ACCOUNT_DEST=~/.gcp/gcs-artifacts-account.json

gcloud iam service-accounts create \
    $SERVICE_ACCOUNT_NAME \
    --display-name $SERVICE_ACCOUNT_NAME

SA_EMAIL=$(gcloud iam service-accounts list \
    --filter="displayName:$SERVICE_ACCOUNT_NAME" \
    --format='value(email)')

PROJECT=$(gcloud config get-value project)

gcloud projects add-iam-policy-binding $PROJECT \
    --role roles/storage.admin --member serviceAccount:$SA_EMAIL

mkdir -p $(dirname $SERVICE_ACCOUNT_DEST)

gcloud iam service-accounts keys create $SERVICE_ACCOUNT_DEST \
    --iam-account $SA_EMAIL

Once you have run these commands, your GCS JSON key is sitting in a file called $SERVICE_ACCOUNT_DEST.

Add the account and enable it

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Next, enable gcs artifacts and add an artifact account to clouddriver-local.yml:

artifacts:
  enabled: true
  gcs:
    enabled: true
    accounts:
      - name: my-gcs-artifact-account
        ## This can be a secret reference
        json-path: /mnt/secrets/service-account-file.json

Last, redeploy clouddriver with these changes

Docs: Configuring Oracle Object Storage Artifact Credentials

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker stages that read data from artifacts can consume Oracle Object Storage objects as artifacts.

Download credentials

If you have enabled Oracle Cloud provider in Spinnaker, you may use the same region, Tenancy’s OCID, user’s OCID, private key file, and fingerprint to enable Oracle Object Storage Artifact. You will need the following to enable Oracle Object Storage Artifact in Spinnaker:

A user in IAM for the person or system who will be using Spinnaker, and that user must be granted access to Object Storage or in one IAM group with permissions of Object Storage.

See Adding Users , and Object Storage Policy
The user’s home region.

See Managing Regions . (e.g. --region us-ashburn-1)
RSA key pair in PEM format (minimum 2048 bits).

See How to Generate an API Signing Key . (e.g. --ssh-private-key-file-path /home/ubuntu/.oci/myPrivateKey.pem)
Fingerprint of the public key.

See How to Get the Key’s Fingerprint . (e.g. --fingerprint 11:22:33:..:aa)
Tenancy’s OCID and user’s OCID.

See Where to Get the Tenancy’s OCID and User’s OCID . (e.g. --tenancyId ocid1.tenancy.oc1..aa... --user-id ocid1.user.oc1..aa...)
Upload the public key from the key pair in the Console.

See How to Upload the Public Key .
Namespace: this is your Tenancy name. On Oracle Cloud Console, click on the user menu. The Tenancy name is next to your user name.

See Object Storage Namespaces , and Managing Compartments . (e.g. --namespace my-tenancy)

Add the credentials either to a secrets manager for use by reference or to a volume mounted into the clouddriver pods by modifying the deployment.yaml for clouddriver.

Add the account and enable it

Add to clouddriver-local.yml the following configuration

artifacts:
  enabled: true
  oracle:
    enabled: true
    accounts:
    - name: my-dev-account
      namespace: <replaceme>
      region: region
      userId: userId
      fingerprint: fingerprint
      sshPrivateKeyFilePath: /mnt/someplace|encryptedFileReference
      privateKeyPassphrase: phrase|encryptedReference
      tenancyId:  <replaceme>

Spinnaker – Configure Artifacts

Docs: Configure a Git Repo Artifact Account

Overview

Prerequisites

Download credentials

Token auth

User-Password auth

SSH key auth

Store the credentials for user

Add the account and enable it with one of the auth options

Enable git pull caching support

Other options

Docs: Configure a GitHub artifact account

Prerequisites

Download credentials

Add the account and enable it

Docs: Configure a GitLab artifact account

Prerequisites

Download credentials

Add the account and enable it

Docs: Configure Bitbucket Artifact Credentials

Download credentials

Add the account and enable it

Docs: Configure Helm Artifact account

Setup usage in pipeline executions

Download credentials

Add the account and enable it

Trigging from helm changes

Docs: Configure HTTP Artifact Credentials

Download credentials

Add the account and enable it

Docs: Configure Maven Artifact Credentials

Download credentials

Add the account and enable it

Docs: Configure S3 Artifact

Download credentials

Add the account and enable it

Docs: Configuring GCS Artifact Credentials

Prerequisites

Download credentials

Add the account and enable it

Docs: Configuring Oracle Object Storage Artifact Credentials

Download credentials

Add the account and enable it

Enable `git pull` caching support