Crossplane – The Control-Plane of the future

VSHN has been a fan of Crossplane since its very early days in 2019. Since then the project has matured a lot and is now used in production by VSHN and many others. In mid 2020 Crossplane became a CNCF Sandbox project and lately applied to be promoted to be a CNCF Incubation project. It’s time for an introduction to Crossplane, why it matters to VSHN and talk about our production usage.

This blog article is also available as a video talk (with corresponding slides).

Use Case: Self-Service Marketplace with Crossplane Service Broker

The very first use case we were able to fulfill by using Crossplane is a project for a VSHN customer who provides a self-service marketplace to their internal customers (developer and service teams). This marketplace is available in their internal Cloudfoundry environment, presented as a one-click service provisioning web-interface. Via this interface, the end-user can order a MariaDB Galera Cluster or a Redis Cluster with different characteristics (e.g. available storage or memory), called Service Plans, with one click. This infrastructure runs in an on-premise datacenter which doesn’t provide any of the well-known hyperscaler services and APIs. We were able to use the Crossplane Helm Provider to deploy services which are specified by Crossplane Composite Resources and Compositions.

In the world of Cloudfoundry the Open Service Broker API is used to provision and manage services. To have a native integration in to the marketplace we developed a Crossplane Service Broker which maps the concepts of the Open Service Broker API specification to the concepts of Crossplane. As they match very well, the integration and translation between these two APIs is very easy.

The Crossplane Service Broker is Open Source and available under https://github.com/vshn/crossplane-service-broker.

This concept lays the foundation for many upcoming new services of VSHN, under the name “Application Catalog”. Watch out this space for more articles about this topic!

What is Crossplane?

In short:

Crossplane is an open source Kubernetes add-on that enables platform teams to assemble infrastructure from multiple vendors, and expose higher level self-service APIs for application teams to consume, without having to write any code. 

(Source: crossplane.io)

To achieve this promise, Crossplane brings three main features with it:

• Providers: These are the pluggable building blocks to provision and manage infrastructure. Each provider enables the use of an upstream or third-party API, for example of a cloud provider, and manages the abstraction to it by bringing Kubernetes custom resources (CRDs) with it. These custom resources are called “managed resources” in the Crossplane world and resemble the upstream API as closely as possible. As each upstream provider has its own opinionated API, Crossplane aligns these interface by providing its own opinionated structure, the Crossplane Resource Model, abbreviated XRM. This allows for example to have a unified API for things like status conditions and references.
  Crossplane itself brings already a lot of providers out-of-the-box and under the crossplane-contrib GitHub organization a lot of third-party providers are being developed.
• Compositions: This is a Crossplane specific construct which enables the possibility to define new custom APIs – called “composite resources” (XR) – which provide a pre-configured set of managed resources. By predefining a set of managed resources – called “composition” –  the end-user of the platform (e.g. the developer) is being enabled to get infrastructure in an actually usable self-service way. The user doesn’t have to care about the inner details of for example an AWS RDS instance which most of the time needs a lot of configuration and other objects (VPC, networking, firewalling, access control). This work is done by the platform team.
• Packages: Sharing opinionated infrastructures is done by packaging up all the resources (XRD, Provider, Compositions) in to a package and re-distributing it as a standard OCI image.

The Crossplane developer Nic Cope has written a very good overview in Crossplane vs Cloud Provider Infrastructure Addons and the Crossplane documentation gives another good introduction.

The project overview slides Crossplane – CNCF Project Overview walks the reader through the project (As of March 2021).

Why do we care?

The three core features described above in itself are already very cool, but we feel that there is much more behind it.

As Crossplane leverages the Kubernetes API and concepts, it enables a lot of possibilities:

• Usage of a well-known API. There is no need to learn a completely new API.
• This allows to re-use tooling like GitOps to declaratively manage infrastructure.
• The infrastructure is defined in the same language (Kubernetes API style) as the application deployment is described. No need to learn a new domain-specific language.
• With that there can be one place which describes everything needed to run an application, including all the infrastructure needs (databaes, caches, indexes, queues, …).
• As Crossplane is a Kubernetes operator, it has reconciliation built into its heart and therefore all the time actively makes sure that the infrastructure adheres to the defined state. Manual changes to the infrastructure will be rolled-back immediately. No configuration drift is possible this way.
• Battle-tested Kubernetes RBAC rules help to control access to infrastructure provisioning and management.

Kubernetes is much more than “just” container orchestration. It’s the platform aspect that counts, the well-defined API and the concepts of the control-loop. Crossplane brings this to the next level, making Kubernetes more independent of containers than ever before.

VSHN lately updated its company beliefs in which we set out to bet on Kubernetes as the technical foundation for everything we do in the future. With Crossplane we can now even provision all the infrastructure we need straight out of Kubernetes, no need to use another, disconnected tool anymore.

One API to rule them all:

Core Kubernetes APIs to orchestrate application containers
Crossplane APIs to orchestrate infrastructure

Comparison to other Infrastructure as Code tools

The above description might already shed some light where the differences to other Infrastructure as code tools like Terraform, Pulumi or Ansible are.

Why not Terraform?

When directly comparing Crossplane with Terraform the most important aspect is that Terraform is a command-line (CLI) tool acting on control-planes, where Crossplane is a control-plane itself which is active all the time.

To configure infrastructure with Terraform, the user has to declare the intended architecture in the domain specific HashiCorp Configuration Language (HCL). After that the CLI has to be invoked manually which starts Terraform to plan and actually apply the configuration. After that the current state is stored in a state file to represent the current infrastructure. When something changes in the infrastructure without telling Terraform about it, the stored state differs from the actual state and on the next CLI invocation no-one knows what happens. Also, when only wanting to change something on one of the probably many provisioned services, Terraform always configures all services which could take a long time and affect other services as well, unintended. Automating Terraform is very hard because of these and many other issues. Many more aspects are discussed in Crossplane vs Terraform.

That doesn’t mean Terraform is bad, it’s just a completely different approach to manage infrastructure.

Why not cloud specific Kubernetes operators?

Many, if not all, of the big cloud providers (Hyperscaler) are providing a native Kubernetes operator to manage their cloud of out Kubernetes: Google Cloud has their Config Connector, Azure the Service Operator and AWS the Controllers for Kubernetes. All these operators are specific to the cloud they are engineered for and are providing low-level access to their services. While it’s perfectly fine to use them, Crossplane provides an abstraction layer where the same APIs can be used cross-cloud and presents the platform user the same API, independent on which cloud the cluster is running on and how all the services are named. By leveraging the Composition feature of Crossplane the user doesn’t have to care what all is needed to properly provision a service: For example a production-grade AWS RDS instance has hundreds of configuration values and needs networking, security group, API connection secret, user, schema and grants. This all can be easily abstracted by Crossplane Compositions. An in-depth discussion can be found in Crossplane vs Cloud Provider Infrastructure Addons.

Keeping up with infrastructure providers

Cloud providers are constantly adding and changing services. The Crossplane providers somehow have to keep up with that. As it’s nearly impossible and also impractical to manually keep up with the changes, the Crossplane developers have engineered tools to generate Crossplane providers out of already existing and well maintained SDKs.

Examples:

• Because Terraform has a huge user base, a lot of upstream providers are already supported and are implemented as Terraform providers. Crossplane is working on a way to generate Crossplane providers out of Terraform providers. See Adding VMware support to Crossplane using Terraform Provider Generation for an example how VMware support is provided by leveraging the already very complete Terraform provider.
• Hyperscaler clouds like AWS, Azure or Google already provide SDKs for automating their clouds. The respective Crossplane providers are generated out of these SDKs. The blog post Accelerating Crossplane provider coverage with ACK and Azure code generation describes that in detail.

Crossplane Terminology

To better understand Crossplane, a few Crossplane specific terms will need an explanation:

• Providers: Extends Crossplane to enable infrastructure resource provisioning. In order to provision a resource, a Custom Resource Definition (CRD) needs to be registered in the Kubernetes cluster and its controller should be watching the Custom Resources those CRDs define.
• Managed Resources: The Crossplane representation of the cloud provider resources and they are considered primitive low level custom resources that can be used directly to provision external cloud resources for an application or as part of an infrastructure composition.
• Crossplane Resource Model (XRM): Standardization between all providers, defining for example the status properties and resource references.
• Composite Resource (XR): A special kind of custom resource that is composed of other resources. Its schema is user-defined.
• Composite Resource Definition (XRD): Defines a new kind of composite resource, and optionally the claim it offers.
• Composite Resource Claim (XRC): Declares that an application requires a particular kind of infrastructure, as well as specifying how to configure it.
• Composition: Specifies how Crossplane should reconcile a composite infrastructure resource – i.e. what infrastructure resources it should compose. It can be used to build a catalog of custom resources and classes of configuration that fit the needs and opinions of an organization.

Video introduction