First Pre-Release of Project Syn Tools

We have been working hard since the initial announcement of Project Syn back in November 2019, and are proud to announce version 0.1.0, the first pre-release of a set of Project Syn tools.
Quick reminder about what Project Syn is about:

Project Syn is a pre-integrated set of tools to provision, update, backup, observe and react/alert production applications on Kubernetes and in the cloud. It supports DevOps through full self-service and automation using containers, Kubernetes and GitOps. And best of all: it is Open Source.

TL;DR: The code is on GitHub, under its own organization: https://github.com/projectsyn. The official documentation is in https://docs.syn.tools/ (The documentation is open source too!)

What does Project Syn do?

Short answer: it enables the management of many Kubernetes clusters, and provides a set of services to the users of those clusters. Project Syn is composed by many tools; some specially developed for the project, some already existing, all Open Source. It’s not only about tooling, it’s also about processes and best practices.
The actual story is a bit longer.

Features of version 0.1.0

To manage a big fleet of Kubernetes clusters, we need an inventory with the following information:

• The cloud providers they are running on;
• Locations;
• Tenants each cluster belongs to;
• Kubernetes versions deployed;
• Kubernetes flavor / distribution used;
• …and a lot more!

This is what the Project Syn tool Lieutenant (written in Go) gives us: an inventory application to register clusters, to assign them to a tenant and to store inventory data. It consists of a REST API (based on the OpenAPI 3 specification) and a Kubernetes Operator, to store data directly in the underlying Kubernetes cluster (in CRDs) and to act on events.
Knowing about clusters is just one part. Another important element is to continuously deploy and monitor system applications (like K8up, Prometheus, …) on Project Syn enabled Kubernetes clusters. This is all done with the GitOps pattern, managed by Argo CD, which is deployed to every cluster. Thanks to Argo CD we can make sure that the applications deployed to the cluster are exactly configured as specified in the corresponding Git repository, and that they are running just fine.
Each Project Syn enabled Kubernetes Cluster has its own so-called Catalog Git Repository. This contains a set of YAML files specifically crafted for each cluster, containing the system tools to operate the cluster, and to give access to well configured self-service tooling to the user of the cluster.
The generation of these YAML files is the responsibility of the Project Syn tool Commodore (written in Python). Commodore is based upon the Open Source tool Kapitan by leveraging inventory data from Lieutenant. After gathering all needed data about a cluster from the inventory, Commodore can fetch all defined components, parameterize them with configuration data from a hierarchical GIT data structure and generate the final YAML files, ready to be applied by Argo CD to the Kubernetes Cluster. The Lieutenant API also knows where the catalog Git repository is located, and Commodore is therefore able to automatically push the catalog to the matching Git repository.
Secrets are never stored in GitOps repositories. They are instead stored securely in Hashicorp Vault, and only retrieved during the „apply“ phase, directly on the destination Kubernetes Cluster. This process is supported by the Kapitan secret management feature and by Commodore, who prepares the secret references during the catalog generation. Argo CD calls kapitan secrets --reveal  during the manifest apply phase, which then actually connects to Vault to retrieve the secrets and stores them in the Kubernetes Cluster, ready to be consumed by the application.
The management of all these Git repositories is the responsibility of the Lieutenant Operator (written in Go, based on Red Hat’s Operator SDK). It is able to manage remote Git repositories (GitLab, GitHub, Bitbucket, etc) and prepare them for Commodore and Argo CD, for example by configuring an SSH deploy key.
The Project Syn tool Steward (written in Go) has the responsibility of enabling Project Syn in a Kubernetes Cluster, communicating with the Lieutenant API, to perform the initial bootstrapping of Argo CD. This bootstrapping includes basic maintenance tasks: should Argo CD be removed from the cluster inadvertently, Steward will automatically reinstall it. An SSH deploy key is generated during bootstrapping and transmitted back to the API. With this procedure it is possible to bootstrap the whole GitOps workflow without any manual interaction.

Analogies with Puppet

For those familiar with Puppet, there are some similarities with the design of Project Syn:

• Puppet Server: Commodore and Kapitan to generate the catalog, matching the facts from the cluster.
• Puppet DB: Lieutenant acting as inventory / facts registry.
• Hiera: Kapitan with its hierarchical configuration model.
• Puppet Agent: Steward and Argo CD on the cluster. Steward to communicate with the API and Argo CD to apply the catalog.
• Puppet Modules: Commodore Components, bringing modularity into Kubernetes application deployment.

Many of these concepts are documented in the Project Syn documentation pages, specifically the Syn Design Documents, documenting all the design decisions (even though they are still in „work-in-progress“ stages).

What are the next steps for Project Syn?

This is really just the beginning! There are a lot of plans and ideas for the future evolution of Project Syn. We have crafted an initial roadmap, and we published it as part of the official Project Syn documentation.
This initial pre-release is just the tip of the iceberg. Under the surface there is a lot more brewing, to be released as soon as possible. To reiterate: It’s not only about tools, but also about concepts and processes, which also means a lot of documentation will emerge over the next months.
One of the focus of this initial pre-release was to lay the foundation for future development. It has a strong focus on the operations side. Future milestones will broaden the focus to include more and more self-service possibilities for the user, including tight integration of Crossplane for easy and fully automated cloud service provisioning.
We at VSHN are now starting to use Project Syn for an initial set of managed Kubernetes clusters, and will continue to develop the concept, tools and processes while we learn about more use cases and with the real-life experience we gather.

How can I contribute?

Project Syn is a young project and is making the first initial steps in the open world. Many things are just getting started, just like the documentation and the contribution guidelines. Testing and giving feedback through GitHub issues is certainly a great way to start contributing. And of course, if you are looking for a Managed Kubernetes or Managed OpenShift cluster, get in touch with us with the form at the bottom of this page!

Learn more

Second Beta Release of Project Syn Tools