Spinnaker – Extending Spinnaker

Docs: Integrate your CI

Mon, 01 Jan 0001 00:00:00 +0000

Overview

At Netflix, we built CI integrations to enable features like the Builds view in Spinnaker, fetching artifact metadata and more. You can see the full list of CI features here .

Viewing CI details in Spinnaker

We recently added the option to see CI details in Spinnaker, in a new “Builds” tab. It looks like this:

This is a new tab that can be accessed from the main Spinnaker UI. The UI code (in deck) is calling the gate service to fetch information, and gate is just proxying the call to igor.

If you wish to use it, this is what you’ll need to do:

Make sure you are using gate . Here’s the CiController the UI is calling (no need to do any actions here).
Fork igor .
Create a new service in igor which implements the <code>CiBuildService</code> interface, and contains two endpoints:

ci/builds which returns the GenericBuild object.
ci/builds/{buildId}/output which returns the build’s log by providing a build number, in a map format which looks like:

{"log": "...."}

Example: your service implementation should be something like:

 public class SomeCiService implements CiBuildService[..]

These endpoints are called by the CiController .

Once both of these endpoints are implemented, it should be possible for the installations using that implementation to show CI information in Spinnaker! If you contribute a new implementation to the project, make sure to include documentation for operators explaining how to enable and configure it in their installation.

Managed Delivery artifacts

You can read about how the CI integration enriches the Managed Delivery experience with artifacts here .

Detecting a new published artifact

When a new artifact is published, keel gets notified by echo (for Debian packages) or by igor (for Docker images). Each artifact has a metadata section, which is populated by the corresponding artifact supplier implementation in keel. For example, for docker artifacts, the metadata field is populated by querying clouddriver, which stores information from the docker registries configured in the Spinnaker installation.

Here is an example of a Docker artifact event tracked by keel (this is a piece of a log statement):

Received artifact published event: ArtifactPublishedEvent(artifacts=[PublishedArtifact(name=mce/agent, type=DOCKER,
  reference=dockerRegistry:v2:testregistry:mce/agent, version=h2012.2626f5f, customKind=false, location=testregistry, artifactAccount=null, provenance=null, uuid=null, metadata={date=1607111907158, registry=testregistry, fullname=mce/agent, tag=tag, commitId=2626f5f, buildNumber=2012, branch=tags/v1.2.0^0}

In order to get additional metadata for each tracked artifact (such as build and git details), we’ve created a convention that the buildNumber and commitId will be part of the artifact metadata object, regardless of the artifact type. In this example, we will query our CI provider to get metadata for build number 2012 and commit id 2626f5f.

Getting artifact metadata

Detailed metadata (like commit message, author, timestamp) is visible in the Environments view in the UI, by clicking on an artifact version.

In order to get the metadata information, you’ll have to plug in your CI provider. keel’s buildService calls igor to fetch this information. It uses the endpoint <igorBaseURL>/ci/builds, which looks like:

  @GET("/ci/builds")
  @Headers("Accept: application/json")
  suspend fun getArtifactMetadata(
    @Query("commitId") commitId: String,
    @Query("buildNumber") buildNumber: String,
    @Query("projectKey") projectKey: String? = null,
    @Query("repoSlug") repoSlug: String? = null,
    @Query("completionStatus") completionStatus: String? = null
  ): List<Build>?

At Netflix, we pass as input the commitId and buildNumber. Those are mandatory to uniquely identify an artifact across repositories. The response contains a list of builds (we pick the first from the list), which look like:

data class Build(
  val building: Boolean = false,
  val fullDisplayName: String? = null,
  val name: String? = null,
  val number: Int = 0,
  val duration: Long? = null,
  /** String representation of time in nanoseconds since Unix epoch  */
  val timestamp: String? = null,
  val result: Result? = null,
  val url: String? = null, //url to get build details from the CI provider (like jenkins)
  val id: String? = null, //build uid
  val scm: List<GenericGitRevision>? = null,
  val properties: Map<String, Any?>? = null
)

The Build object is a mirror of GenericBuild in igor.

Note: if you already implemented the /ci/builds endpoint as described above, jump to step 3 and you are done!

If you are interested in plugging your CI provider, you’ll need to do the following:

Fork igor .
Implement the ci/builds endpoint, and return a GenericBuild response (which is converted to Build object above).

Controller endpoints
Generic build service
Example: your service implementation should be somthing like:

 public class SomeCiService implements CiBuildService[...]

Make sure the artifacts being sent to keel contains the buildNumber and commitId.

See code changes between deployments

The logic to generate a link to compare commits is done in keel: ArtifactVersionLinks .

Note: this supports only Stash/BitBucket at the moment. We’ll appreciate contributions for other SCMs!

Surface build information in the UI

A new section called “Pre-deployment” is now available in the UI. This section will surface pre-deployment steps like baking (for Debian packages only) or building. By clicking on “See details”, you’ll be taken to the CI-detailed view (see above), or a default job log which you can provide as a part of your implementation (in the url field in Build object, see above).

Docs: Kubernetes CRD Extensions

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker Extension Points for Custom Resource Definitions

At Google, we’ve built extension points for deep CRD integrations within Spinnaker.

This work has allowed us to support the following features within Spinnaker:

Custom models for representing CRDs as spinnakerKinds
Deploying CRDs with custom Spinnaker artifact types
Custom Kubernetes API versions
Custom Spinnaker naming strategies
Per-account, custom Spinnaker UIs that can run alongside the existing Kubernetes UIs

This guide is for developers who want to duplicate this functionality for their CRDs. It also exists as an explanation of certain code paths within Spinnaker which include hooks with no current corresponding open-source implementations.

Developers who want to implement these features will have to build their own layered version of Clouddriver - see Adam Jorden’s blog post - and should be familiar with the Kubernetes provider and writing code for Clouddriver.

Custom Handlers

The central extension point is the KubernetesHandler class. A subclass of KubernetesHandler - e.g., KubernetesReplicaSetHandler - defines the relationship between Spinnaker and your Kubernetes kind.

For example, if you wanted to build a Spinnaker integration for your CRD of kind MyCRDKind, you would start with the following handler:

@Component
public class MyCRDHandler extends KubernetesHandler {

  public MyCRDHandler() {
    // Hook point for registering a custom `ArtifactReplacer`
    // for your CRD. During a deploy operation,
    // if an artifact of the type specified in the replacer is present,
    // the artifact will be inserted into the manifest using the
    // strategy described in the replacer.
  }

  @Override
  public KubernetesKind kind() {
    return KubernetesKind.from("MyCRDKind");
  }

  @Override
  public boolean versioned() {
    // If the CRD resource should be versioned - i.e., assigned a sequence
    // v001, v002, etc.
    return false;
  }

  @Override
  public SpinnakerKind spinnakerKind() {
    // The Spinnaker kind that the resource will be represented as in Spinnaker's API and UI.
    return SpinnakerKind.SERVER_GROUPS;
  }

  @Override
  public Status status(KubernetesManifest manifest) {
    // Includes logic for determining whether your CRD manifest is stable.
    // A deploy manifest operation, for example, will block until this
    // method returns a stable status.
  }

  @Override
  public KubernetesV2CachingAgentFactory cachingAgentFactory() {
    return KubernetesCoreCachingAgent::new;
  }
}

Custom Spinnaker Resource Models

You may want to change how their CRD is represented in Spinnaker’s API. By default, a CRD of spinnakerKind serverGroups will be represented with the model class KubernetesV2ServerGroup .

You may want to override this representation, for example, if you want to define how your server group’s region is resolved.

To override the default model class, MyCRDHandler should implement ServerGroupHandler (or ServerGroupManagerHandler if your resource is of spinnakerKind serverGroupManagers). You will be responsible for translating raw Spinnaker cache data into a subclass of KubernetesServerGroup.

Custom Kubernetes API Versions

If you want Spinnaker to support custom Kubernetes API versions, subclass KubernetesApiVersion.

For example,

public class MyApiVersion extends KubernetesApiVersion {
  public static MY_BETA_API_VERSION = new MyApiVersion("myApiVersion/v1beta");

  public MyApiVersion(String name) {
    // Base class maintains state.
    super(name);
  }
}

Custom Spinnaker Naming Strategies

To use a custom naming strategy for your CRD, implement NamingStrategy . For example,

@Component
public class MyManifestNamer implements NamingStrategy<KubernetesManifest> {
  @Override
  public String getName() {
    return "myManifestNamingStrategy";
  }

  @Override
  public void applyMoniker(KubernetesManifest manifest, Moniker moniker) {
    // Strategy for applying a Spinnaker `Moniker` to a Kubernetes
    // manifest prior to deployment.
  }

  @Override
  public Moniker deriveMoniker(KubernetesManifest manifest) {
    // Strategy for deriving a Spinnaker `Moniker` from a Kubernetes
    // manifest.
  }
}

This naming strategy can be referenced in a Kubernetes account config. For example:

kubernetes:
  accounts:
    - name: my-kubernetes-account
      namingStrategy: myManifestNamingStrategy

Be careful - this naming strategy will be applied to all manifests manipulated by this account.

Docs: Writing a New Stage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

To create a new stage, you need to make backend changes to orca to implement the logic of the stage, and the front-end changes to Deck to implement the UI. Depending on what the stage does, you may need to implement new cloud provider-specific logic into Clouddriver as well.

This doc currently only covers the backend changes made to orca.

Backend (orca)

For the backend, you need to define:

A stage class
One or more task classes associated with the stage

Stage class

A stage class must implement the com.netflix.spinnaker.orca.pipeline.StageDefinitionBuilder interface.

For providing additional functionality, it can also implement other interfaces:

A CancellableStage can be cancelled.
A RestartableStage can be restarted.
A CloudProviderAware stage exposes information about the cloud provider.
An AuthenticatedStage can perform custom authentication.

Here’s an example based on a stage that is used internally at Netflix to integrate with the Chaos Automation Platform (ChAP).

The stage is composed of two tasks:

beginChap starts a new ChAP run
monitorChap waits for the ChAP run to finish

package com.netflix.spinnaker.orca.pipeline;

import com.google.common.collect.ImmutableMap;
import com.netflix.spinnaker.orca.CancellableStage;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.chap.tasks.BeginChapTask;
import com.netflix.spinnaker.orca.chap.tasks.MonitorChapTask;
import com.netflix.spinnaker.orca.pipeline.model.Execution;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class ChapStage implements StageDefinitionBuilder, CancellableStage {

  @Autowired
  public ChapService chapService;

  @Override
  public <T extends Execution<T>> void taskGraph(Stage<T> stage, TaskNode.Builder builder {
    builder
      .withTask("beginChap", BeginChapTask.class)
      .withTask("monitorChap", MonitorChapTask.class);
  }

  @Override
  public CancellableStage.Result cancel(Stage stage) {
    Run run = (Run) stage.getContext().get("run");
    if (run != null) {
      Run latestDetails = chapService.cancelChap(run.id.toString(), "");
      return new CancellableStage.Result(stage, ImmutableMap.of("run", latestDetails));
    }

    return null;
  }
}

Task classes

A task class must implement a com.netflix.spinnaker.orca.api.pipeline.Task , or an interface that extends it, such as:

A RetryableTask can be retried if it fails.
A PreconditionTask defines preconditions that the task will enforce.

To communicate that a task failed, throw a RuntimeException.

In our example, the ChapStage consists of two tasks:

BeginChapTask
MonitorChapTask

BeginChapTask

The BeginChapTask:

retrieves the testCaseId specified by the user during the task configuration stage
calls the ChAP service via REST API call (see ChapService.startChap)
returns a DefaultTaskResult, passing it the response from the ChAP REST API call

package com.netflix.spinnaker.orca.chap.tasks;

import com.netflix.spinnaker.orca.*;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class BeginChapTask implements RetryableTask {

  @Override
  public TaskResult execute(Stage stage) {
    Map<String, Object> ctx = stage.getContext();
    Object testCaseId = ctx.get("testCaseId");

    if(testCaseId == null || !(testCaseId instanceof String)) {
      throw new RuntimeException("Cannot begin ChAP experiment without a testCaseId.");
    }

    Map<String, Object> params = new HashMap<>();
    params.put("testCaseId", testCaseId);
    Run chapRun = chapService.startChap(params);

    Map<String, Object> map = new HashMap<>();
    map.put("run", chapRun);
    return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  }

  public ChapService getChapService() {
    return chapService;
  }

  public void setChapService(ChapService chapService) {
    this.chapService = chapService;
  }

  @Autowired
  private ChapService chapService;

  @Override
  public long getBackoffPeriod() {
    return TimeUnit.SECONDS.toMillis(5);
  }

  @Override
  public long getTimeout() {
    return TimeUnit.MINUTES.toMillis(1);
  }
}

MonitorChapTask

The MonitorChapTask polls the ChAP service for the status of the ChAP run.

package com.netflix.spinnaker.orca.chap.tasks;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.netflix.spinnaker.orca.DefaultTaskResult;
import com.netflix.spinnaker.orca.ExecutionStatus;
import com.netflix.spinnaker.orca.RetryableTask;
import com.netflix.spinnaker.orca.TaskResult;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class MonitorChapTask implements RetryableTask {

  @Autowired
  private ObjectMapper objectMapper;

  @Autowired
  public ChapService chapService;

  @Override
  public TaskResult execute(Stage stage) {
    Map<String, Object> ctx = stage.getContext();

    Run run = objectMapper.convertValue(ctx.get("run"), Run.class);

    if (run == null) {
      throw new RuntimeException("Cannot monitor Chap task without a valid Run in the context.");
    }

    Run latestDetails = chapService.getChap(run.id.toString());

    Map<String, Object> map = new HashMap<>();
    map.put("run", latestDetails);

    if(latestDetails.outcome == Run.Outcome.PASSED){
      return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
    }

    ExecutionStatus status;

    switch (latestDetails.state) {
      case COMPLETED:
        //workflow is complete, but the outcome didnt pass, consider this a failure.
      case FAILED:
        throw new RuntimeException("ChAP experiment failed.");
      case CANCELLED:
        status = ExecutionStatus.CANCELED;
        break;
      default:
        status = ExecutionStatus.RUNNING;
        break;
    }


    return new DefaultTaskResult(status, map);
  }

  public ChapService getChapService() {
    return chapService;
  }

  public void setChapService(ChapService chapService) {
    this.chapService = chapService;
  }

  @Override
  public long getBackoffPeriod() {
    return TimeUnit.MINUTES.toMillis(1);
  }

  @Override
  public long getTimeout() {
    return TimeUnit.DAYS.toMillis(1);
  }

  public ObjectMapper getObjectMapper() {
    return objectMapper;
  }

  public void setObjectMapper(ObjectMapper objectMapper) {
    this.objectMapper = objectMapper;
  }
}

Other classes used

The details of the com.netflix.spinnaker.orca.chap.Run class and com.netflix.spinnaker.orca.chap.ChapService interface aren’t directly relevant to learning how to write a Spinnaker stage, but for completeness, here’s what those look like for this case:

Run

The Run class is a Java object that is deserialized from JSON.

package com.netflix.spinnaker.orca.chap;

import com.fasterxml.jackson.annotation.JsonAnyGetter;
import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

import java.util.HashMap;
import java.util.Map;
import java.util.UUID;

@JsonIgnoreProperties(ignoreUnknown = true)
public class Run {

  public UUID id;
  // Other properties not shown
  // ...

  // Support arbitrary properties without needing to define them explicitly
  public Map<String, Object> properties = new HashMap<>();

  @JsonAnySetter
  public void set(String fieldName, Object value) {
    this.properties.put(fieldName, value);
  }

  @JsonAnyGetter
  public Object get(String fieldName) {
    return this.properties.get(fieldName);
  }
}

ChapService

The ChapService defines a REST client API for talking to the ChAP service. It uses the Retrofit library.

package com.netflix.spinnaker.orca.chap;

import retrofit.http.Body;
import retrofit.http.GET;
import retrofit.http.POST;
import retrofit.http.Path;

import java.util.Map;

public interface ChapService {
  @POST("/v1/runs")
  Run startChap(@Body Map params);

  @GET("/v1/runs/{id}")
  Run getChap(@Path("id") String id);

  @POST("/v1/runs/{id}/stop")
  Run cancelChap(@Path("id") String id, @Body String body);
}

ChapConfig

To implement the ChapService interface, we do not define a class that extends the interface. Instead, we define a class named ChapConfig with a Spring @Configuration annotation. Note that this implementation uses the chap.baseUrl configuration value that is defined in a separate Spinnaker configuration file.

package com.netflix.spinnaker.config;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.retrofit.Slf4jRetrofitLogger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import retrofit.Endpoint;
import retrofit.RestAdapter;
import retrofit.client.Client;
import retrofit.converter.JacksonConverter;

import static retrofit.Endpoints.newFixedEndpoint;

@Configuration
@ComponentScan({
  "com.netflix.spinnaker.orca.chap.pipeline",
  "com.netflix.spinnaker.orca.chap.tasks"
})
@ConditionalOnProperty(value = "chap.baseUrl")
public class ChapConfig {

  @Bean
  Endpoint chapEndpoint(@Value("${chap.baseUrl}") String chapBaseUrl) {
    return newFixedEndpoint(chapBaseUrl);
  }

  @Bean
  ChapService chapService(Endpoint chapEndpoint,
                          Client retrofitClient,
                          RestAdapter.LogLevel retrofitLogLevel,
                          ObjectMapper objectMapper) {
    return new RestAdapter.Builder()
      .setEndpoint(chapEndpoint)
      .setClient(retrofitClient)
      .setLogLevel(retrofitLogLevel)
      .setLog(new Slf4jRetrofitLogger(ChapService.class))
      .setConverter(new JacksonConverter(objectMapper))
      .build()
      .create(ChapService.class);
  }
}