Kurator – Blog

Blog: Kurator v0.5.0: Crafting a Unified Multi-Cluster Backup and Storage Experience

Mon, 30 Oct 2023 00:00:00 +0000

Kurator is an open-source distributed cloud-native suite, designed to provide a one-stop solution for users looking to build their distributed cloud-native platform. In the latest version v0.5.0, Kurator has enhanced its capabilities in application backup and recovery, as well as storage management in multi-cluster environments, to meet the needs of complex deployments. This update mainly introduces two new features:

Unified Cluster Backup, Recovery, and Migration: Kurator now supports customized one-click backup and recovery of applications and resources across multiple clusters, with real-time monitoring of progress through a unified view. It also offers a one-click migration feature for cross-cluster resources.
Unified Distributed Storage: Kurator implements a consistent distributed storage solution, enabling users to easily implement block, file, and object storage in a multi-cluster environment with one-stop deployment.

Unified Cluster Backup, Recovery, and Migration

In the evolving landscape of multi-cloud and distributed environments, data security and recoverability have become paramount concerns for users. For enterprises, data loss can be a devastating blow, potentially leading to severe business interruptions and reputational damage. With Kubernetes becoming the industry standard, the complexity of data management increases alongside the growth in service numbers and cluster size, making efficient and flexible backup strategies crucial.

Facing the expanding demands and challenges, traditional backup tools often show limitations in multi-environment setups, failing to provide a seamless unified solution. Hence, Kurator’s unified backup solution was developed to offer a backup solution in this field. Based on Velero, Kurator provides a one-click operation experience, allowing users to customize backups and recover applications and resources across multiple clusters. With Kurator’s unified view feature, users can monitor the status and progress of backups in each cluster in real-time. Its coverage includes Kubernetes native resources like Pods, Deployments, Services, and PersistentVolumes, meeting the diverse data protection needs of modern enterprises.

Unified Backup

Kurator’s backup solution offers a variety of choices to suit different data protection needs in various scenarios. Its flexibility ensures that suitable backup strategies can be found for different business scenarios.

Instant Backup: In scenarios with frequent data changes, “instant backup” can quickly provide protection, ensuring the integrity of critical data at key moments.
Scheduled Backup: For data that changes less frequently but still requires persistent protection, “scheduled backup” can automatically perform backups at preset intervals, meeting compliance requirements and ensuring data security.

Additionally, Kurator provides a range of highly customizable backup options. For example, “specific cluster backup” allows operational teams to selectively back up specific clusters based on policies or specific needs. The “resource filtering” feature provides fine-grained control, enabling administrators to precisely define the scope of backups based on attributes such as resource names, namespaces, or labels.

These backup strategies’ diversity and automation capabilities provide stable and reliable data protection in the ever-changing business landscape.

Here is a practical example of unified backup:

apiVersion: backup.kurator.dev/v1alpha1
kind: Backup
metadata:
  ...
  name: select-labels
  namespace: default
spec:
  destination:
    fleet: quickstart
  policy:
    resourceFilter:
      labelSelector:
        matchLabels:
          app: busybox
    ttl: 720h
status:
  backupDetails:
  - backupNameInCluster: kurator-member1-backup-default-select-labels
    backupStatusInCluster:
      completionTimestamp: "2023-10-28T03:37:13Z"
      expiration: "2023-11-27T03:37:07Z"
      formatVersion: 1.1.0
      phase: Completed
      progress:
        itemsBackedUp: 1
        totalItems: 1
      startTimestamp: "2023-10-28T03:37:07Z"
      version: 1
    clusterKind: AttachedCluster
    clusterName: kurator-member1
  - backupNameInCluster: kurator-member2-backup-default-select-labels
    backupStatusInCluster:
      completionTimestamp: "2023-10-28T03:37:13Z"
      expiration: "2023-11-27T03:37:07Z"
      formatVersion: 1.1.0
      phase: Completed
      progress: {}
      startTimestamp: "2023-10-28T03:37:07Z"
      version: 1
    clusterKind: AttachedCluster
    clusterName: kurator-member2
  ...

Observing the spec configuration, the backup target is resources labeled with app:busybox across all clusters in the Fleet. By configuring policies in the spec, relevant resources are ensured to be backed up. The status allows real-time tracking of the backup task’s execution in each cluster, such as kurator-member1 and kurator-member2, maintaining transparency in operations.

🔗 For more examples and details, please refer to: Kurator Backup Documentation

Unified Recovery

Based on the backup data generated from unified backups, Kurator supports cross-cluster application and resource recovery through its unified recovery feature.

Instant Backup Recovery: Based on the backup data created from “instant backup,” you can quickly restore to a specified critical moment’s state.
Scheduled Backup Recovery: For “scheduled backup,” Kurator supports restoring backup data to the point of the last successful backup execution.

Similar to the backup function, Kurator also provides diverse and customizable options in recovery. For example, “specific cluster recovery” allows users to restore data to designated clusters without covering all clusters included in the backup. The “resource filtering” feature enables further selection of backup data, allowing selective restoration of needed data items. Users can define the scope of recovery based on backup names, namespaces, or labels, enhancing the recovery process’s flexibility and ensuring high precision.

Refer to the following operational example to understand how to use Kurator for unified recovery:

apiVersion: backup.kurator.dev/v1alpha1
kind: Restore
metadata:
  ...
  name: minimal
  namespace: default
spec:
  backupName: select-labels
status:
  restoreDetails:
  - clusterKind: AttachedCluster
    clusterName: kurator-member1
    restoreNameInCluster: kurator-member1-restore-default-minimal
    restoreStatusInCluster:
      completionTimestamp: "2023-10-28T09:24:07Z"
      phase: Completed
      progress:
        itemsRestored: 2
        totalItems: 2
      startTimestamp: "2023-10-28T09:24:05Z"
  - clusterKind: AttachedCluster
    clusterName: kurator-member2
    restoreNameInCluster: kurator-member2-restore-default-minimal
    restoreStatusInCluster:
      completionTimestamp: "2023-10-28T09:24:07Z"
      phase: Completed
      progress:
        itemsRestored: 2
        totalItems: 2
      startTimestamp: "2023-10-28T09:24:05Z"
  ...

By examining the recovery task’s spec configuration, we can confirm that this recovery operation targets the previously mentioned backup data labeled as select-labels. Here, the minimum configuration is used, not filtering during recovery but directly restoring according to the backup’s settings. In the status, the execution condition of the recovery task in each cluster can be tracked in real-time.

🔗 For more examples and details, please refer to: Kurator Recovery Documentation

Unified Migration

Unified migration aims to simplify the process of migrating applications and their resources from one cluster to several others. Users need to define a migrate-type resource configuration, specifying source clusters, target clusters, and related policies. Similar to Kurator’s unified backup and recovery functions, users can also perform a wealth of customized configurations.

Once the configuration is complete, Kurator’s corresponding controllers will automatically start the migration tasks. This series of tasks includes uploading resources from the source cluster to object storage and eventually migrating to the designated target clusters. The specific migration process can be referred to in the following diagram:

Compared to using Velero, Kurator provides a more integrated and clear migration process description. All necessary configuration details are centralized in a single migrate object, thus reducing the configuration burden as the number of target clusters increases. Additionally, Kurator automatically manages the entire process from creating backups to completing migration, simplifying operational procedures and reducing the risk of manual errors. Moreover, users can also monitor the migration progress in multiple clusters in real-time through this single object, staying informed of the latest migration status and ensuring the entire process runs as expected.

Here is a practical example of unified migration:

apiVersion: backup.kurator.dev/v1alpha1
kind: Migrate
metadata:
  ...
  name: select-labels
  namespace: default
spec:
  policy:
    resourceFilter:
      labelSelector:
        matchLabels:
          app: busybox
  sourceCluster:
    clusters:
    - kind: AttachedCluster
      name: kurator-member1
    fleet: quickstart
  targetCluster:
    clusters:
    - kind: AttachedCluster
      name: kurator-member2
    fleet: quickstart
status:
  conditions:
  - lastTransitionTime: "2023-10-28T15:55:23Z"
    status: "True"
    type: sourceReady
  phase: Completed
  sourceClusterStatus:
    backupNameInCluster: kurator-member1-migrate-default-select-labels
    backupStatusInCluster:
      completionTimestamp: "2023-10-28T15:55:18Z"
      expiration: "2023-11-27T15:55:13Z"
      formatVersion: 1.1.0
      phase: Completed
      progress: {}
      startTimestamp: "2023-10-28T15:55:13Z"
      version: 1
    clusterKind: AttachedCluster
    clusterName: kurator-member1
  targetClusterStatus:
  - clusterKind: AttachedCluster
    clusterName: kurator-member2
    restoreNameInCluster: kurator-member2-migrate-default-select-labels
    restoreStatusInCluster:
      completionTimestamp: "2023-10-28T15:56:00Z"
      phase: Completed
      startTimestamp: "2023-10-28T15:55:58Z"
  ...

In the spec configuration, the source cluster is set as kurator-member1, with the target cluster being kurator-member2, focusing the migration process only on resources containing the label app:busybox. In the status, the migration phase displays as Completed, indicating that the migration operation has been accomplished. sourceClusterStatus and targetClusterStatus provide details of the backup in the source cluster and the recovery in the target cluster, respectively.

🔗 For more details, please refer to: Kurator Migration Documentation

Unified Distributed Storage

Distributed storage, an indispensable part of modern cloud-native architectures, provides scalability and reliability of data. However, implementing a consistent distributed storage solution across different clusters often involves complex configurations and management tasks.

Kurator is committed to simplifying the deployment and management of distributed storage. Based on the leading open-source project Rook, Kurator supports easy and automated management of distributed storage across multiple clusters. This includes various storage types such as block, filesystem, and object storage, to meet the demands of various application scenarios.

Utilizing the Fleet plugin, Kurator offers a one-click cross-cluster deployment solution for distributed storage, simplifying configuration steps and significantly reducing the likelihood of configuration errors. The architecture is illustrated in the following diagram:

Here is an example of deploying multi-cluster distributed storage through the Fleet plugin:

apiVersion: fleet.kurator.dev/v1alpha1
kind: Fleet
metadata:
  name: quickstart
  namespace: default
spec:
  clusters:
    - name: kurator-member1
      kind: AttachedCluster
    - name: kurator-member2
      kind: AttachedCluster
  plugin:
    distributedStorage:
      storage:
        dataDirHostPath: /var/lib/rook
        monitor:
          count: 3
          labels:
            role: MonitorNodeLabel
        manager:
          count: 2
          labels:
            role: ManagerNodeLabel

In the spec, clusters indicate where the storage will be deployed. In the status, under the plugin configuration, distributedStorage signifies the installation of a distributed storage plugin. Additionally, dataDirHostPath defines the storage path, while monitor and manager configurations specify parameters for the Ceph components.

🔗 For more examples and details, please refer to: Kurator Distributed Storage Documentation

Reference Links

Unified Backup Recovery Migration Feature Introduction: Kurator Backup Recovery Migration
Fleet Backup Plugin Installation: Fleet Backup Plugin
Unified Backup Operation Guide: Kurator Backup Operation
Unified Recovery Operation Guide: Kurator Recovery Operation
Unified Migration Operation Guide: Kurator Migration Operation
Unified Distributed Storage Operation Guide: Kurator Distributed Storage Operation

Community Contact Information

GitHub: Kurator on GitHub
Kurator Homepage: Kurator Website
Slack: Join Kurator on Slack

Blog: Kurator v0.4.0: Leading a New Chapter in Distributed Cloud-Native Management

Wed, 28 Jun 2023 00:00:00 +0000

Kurator is an open-source distributed cloud-native platform that integrates numerous mainstream cloud-native software stacks, such as Kubernetes, Istio, Prometheus, etc. It is designed to help users build and manage their distributed cloud-native infrastructure, thus driving the digital transformation of enterprises. Kurator embodies the concept of “Infrastructure as Code,” allowing users to manage infrastructure in cloud, edge, or local environments declaratively. Its “out-of-the-box” feature enables users to install cloud-native software stacks with a single click. Leveraging Fleet, Kurator further offers unified management of multi-cloud and multi-cluster environments, greatly enhancing management efficiency.

In the newly released version v0.4.0, Kurator further enriches the unified management capabilities for applications in distributed cloud-native scenarios to better meet the complex requirements of multi-cloud environments. This update primarily includes the following four aspects:

Adopting GitOps and utilizing Fleet for unified application distribution in multi-cloud environments. This new approach reduces the complexity of configuration in multi-cloud heterogeneous environments and simplifies the management process of distributed deployment.
Providing a unified cluster metric monitoring solution based on Fleet, Prometheus, and Thanos. This solution aims to improve the comprehensiveness, accuracy, and real-time nature of metric monitoring in complex multi-cloud and multi-cluster environments, thereby increasing operational efficiency and reducing complexity.
Utilizing Kyverno and Fleet for unified policy management in multi-cloud and multi-cluster environments. This feature will enhance the efficiency of policy management, ensuring consistency and security of policies across all clusters.
Introducing a new cluster type called “Attached Cluster.” This type allows Kurator to manage Kubernetes clusters built by any tool, located anywhere, further strengthening Kurator’s management of distributed cloud environments.

Unified Application Distribution

As multi-cloud and multi-cluster become more common, effectively deploying and distributing applications in distributed cloud-native environments is becoming increasingly important. Kurator’s unified application distribution function aims to solve the following problems:

Cumbersome configuration for multi-cloud, multi-cluster environments: In traditional multi-cloud environments, deploying the same application in each environment requires complex configuration, increasing the difficulty of deployment and consuming unnecessary time and human resources.
Maintaining version consistency challenges: In distributed multi-cloud environments, keeping the application version consistent across all clusters and updating them in a timely manner is challenging.
Difficulties in managing distributed deployments: After deploying applications in various clusters, one must enter each cluster separately to check whether the deployment was successful and view the deployment status.

Kurator’s unified application distribution function uses GitOps, making it possible to deploy applications to multiple cloud environments with one click and simplifying the configuration process. This method ensures that the application versions in each cluster remain consistent and can be updated promptly. On the Kurator host cluster, users can view and manage the application deployment status of all clusters in a unified manner, thereby improving operational efficiency.

Kurator, based on FluxCD, optimizes deployment efficiency and accuracy with automated application synchronization and deployment processes. Leveraging the advantages of Fleet, it can flexibly adapt to various business and cluster needs, meeting the diverse requirements of users for application distribution.

Kurator’s unified application distribution function provides rich and flexible configuration options. Users can define key parameters such as the application’s source and synchronization policy through YAML configuration files. Additionally, Kurator supports a combination of various source types (including gitRepository, helmRelease, etc.) and synchronization strategies.

Here is an example of unified application distribution:

apiVersion: apps.kurator.dev/v1alpha1
kind: Application
metadata:
  name: gitrepo-kustomization-demo
  namespace: default
spec:
  source:
    gitRepository:
      interval: 3m0s
      ref:
        branch: master
      timeout: 1m0s
      url: https://github.com/stefanprodan/podinfo
  syncPolicies:
    - destination:
        fleet: quickstart
      kustomization:
        interval: 5m0s
        path: ./deploy/webapp
        prune: true
        timeout: 2m0s
    - destination:
        fleet: quickstart
      kustomization:
        targetNamespace: default
        interval: 5m0s
        path: ./kustomize
        prune: true
        timeout: 2m0s

This example configuration demonstrates how to achieve unified application distribution across multiple clusters with Kurator: retrieving application configuration from a Git source, followed by synchronization and deployment through Fleet. Users only need simple configuration to quickly deploy applications to multiple clusters.

For more examples and related information, please refer to: Kurator Unified Application Distribution Documentation

Unified Cluster Metric Monitoring

In complex multi-cloud and multi-cluster environments, unified cluster metric monitoring can enhance work efficiency and reduce operational complexity. Many enterprises face the challenge of effectively monitoring and managing across various clusters to ensure service stability and optimize resource utilization.

Single monitoring tools often fail to meet the needs for comprehensive, timely, and accurate monitoring. This requires operational personnel to enter each cluster separately for checks, increasing the workload and potentially leading to missed or delayed critical metric information. Moreover, different clusters may have different requirements, making management even more complex.

To address these issues, Kurator offers a multi-cluster metric monitoring solution based on Prometheus, Thanos, Grafana, and Fleet, enabling users to easily achieve unified metric monitoring across multiple clusters.

Typically, the process of achieving unified metric monitoring in multi-cluster environments can be summarized as follows:

Each cluster runs a Prometheus instance responsible for collecting local monitoring data.
Each Prometheus instance is accompanied by a Thanos Sidecar, which pushes the data collected by Prometheus to remote storage.
Thanos Query aggregates data from all Thanos Sidecars and remote storage, providing a unified query interface.
Grafana connects to Thanos Query, thus displaying a unified monitoring view of all clusters.

With the capabilities of Kurator’s Fleet, users do not have to handle the above complex process themselves. Users simply define related configurations in Fleet, and the Fleet Manager automatically completes the process.

Here is an example Fleet configuration that can complete the above process:

apiVersion: fleet.kurator.dev/v1alpha1
kind: Fleet
metadata:
  name: quickstart
  namespace: default
spec:
  clusters:
    - name: kurator-member1
      kind: AttachedCluster
    - name: kurator-member2
      kind: AttachedCluster
  plugin:
    metric:
      thanos:
        objectStoreConfig:
          secretName: thanos-objstore
grafana: {}

After executing the above configuration, the Fleet Manager will install Prometheus and Thanos Sidecar on both the kurator-member1 and kurator-member2 clusters. Then, users can view the unified monitoring view of all clusters on the Kurator host through the Grafana dashboard.

For more details on using unified cluster metric monitoring, please refer to: Kurator Unified Cluster Metric Monitoring Documentation

Unified Policy Management

In the distributed cloud environment, to meet the unified security protection needs of multi-cloud and multi-cluster, Kurator introduces unified policy management functionality to solve the following problems:

Inability to manage policies across multiple clusters and apply the same policy across clusters.
Scattered management of policies in multiple sub-clusters, leading to redundancy and high complexity, preventing unified, efficient configuration and management.
Inability to uniformly limit resource usage across multiple clusters to ensure all clusters follow the same operational rules and business requirements.

Kurator’s policy management capability is based on Kyverno and utilizes Fleet to implement the cross-cluster distribution and application of policies. This mechanism allows for unified and efficient management of policies across the entire cluster group, avoiding the complexity of managing policies in each sub-cluster separately.

Kurator’s policy management capability is nearly identical to policy management in a single Kubernetes cluster, allowing users to quickly familiarize themselves and get started. Here is an example of implementing unified policy management in Fleet with Kurator:

apiVersion: fleet.kurator.dev/v1alpha1
kind: Fleet
metadata:
  name: quickstart
  namespace: default
spec:
  clusters:
    - name: kurator-member1
      kind: AttachedCluster
    - name: kurator-member2
      kind: Cluster
    - name: kurator-member3
      kind: CustomCluster
  plugin:
    policy:
      kyverno:
        podSecurity:
          standard: baseline
          severity: high
          validationFailureAction: Audit

In the above configuration file, we apply a Pod security policy with podSecurityStandard as baseline and podSecuritySeverity as high to the clusters in Fleet. When Pod configurations violate the security policy, the corresponding events are recorded in the PolicyReport during their creation process; when validationFailureAction is set to Enforce, the creation or update of illegal resources will be intercepted. All clusters in Fleet will apply this policy, and application operations and development personnel will adjust and configure applications following these Pod security regulations. With Kurator’s unified policy management capability, policy management efficiency can be effectively improved while ensuring the consistency and security of policies across all clusters.

For more information on Kurator’s unified policy management, please refer to: Kurator Unified Policy Management Documentation

AttachedCluster

In the world of cloud-native, the complexity and diversity of infrastructure are unavoidable challenges. For large organizations or companies, they may have deployed multiple Kubernetes clusters in different environments, created by various tools and distributed around the world. To better address this issue, Kurator introduces a new type of cluster in its latest version, the AttachedCluster.

The main purpose of the AttachedCluster is to manage Kubernetes clusters that were not created by Kurator but need to be included in the Kurator fleet management scope. These clusters can be created by any tool and located anywhere. The introduction of this new cluster type extends Kurator’s management capabilities to effectively manage truly distributed cloud environments. In practice, users need to create AttachedCluster resources for Kubernetes clusters they plan to manage with Kurator. These resources contain the cluster’s connection and authentication information, securely stored and managed through Secret. With this information, Kurator can effectively interact with and manage these clusters.

Here is an example:

apiVersion: cluster.kurator.dev/v1alpha1
kind: AttachedCluster
metadata:
  name: kurator-member1
  namespace: default
spec:
  kubeconfig:
    name: kurator-member1
    key: kurator-member1.config

Once the AttachedCluster resources are created, users also need to add these resources to the Kurator Fleet to include these clusters in Kurator’s management scope. In this way, regardless of where these clusters are or what tools created them, they can be uniformly managed and monitored in Kurator.

Here is an example of adding the above AttachedCluster to the Fleet:

apiVersion: fleet.kurator.dev/v1alpha1
kind: Fleet 
metadata:
  name: quickstart
  namespace: default
spec:
  clusters:
    - name: kurator-member1 
      kind: AttachedCluster

With the introduction of AttachedCluster, Kurator provides convenient management of all Kubernetes clusters on a unified platform, avoiding frequent switching between various tools and effectively monitoring and managing clusters in the distributed cloud environment. This improvement not only strengthens Kurator’s management capabilities in the cloud-native field but also expands its management scope, significantly enhancing Kurator’s adaptability and management efficiency in handling complex and diverse cloud computing environments.

For more information on Kurator’s unified policy management, please refer to: Kurator AttachedCluster Documentation

Reference Links

Release Notes

Unified Application Distribution Documentation

Unified Cluster Metrics Monitoring Documentation

Unified Policy Management Documentation

AttachedCluster Documentation

Fleet Manager Documentation

GitHub Address

Kurator Homepage

Slack Address

Blog: Kubernetes Cluster Management: Kurator or Kubespray

Sat, 27 May 2023 00:00:00 +0000

As cloud computing technology rapidly evolves, Kubernetes has become the de facto standard in the container orchestration domain. Users can automate the deployment, scaling, and management of containerized applications through Kubernetes. However, creating highly reliable Kubernetes clusters in different cloud environments can be complex and time-consuming. In response, many users have started looking for tools that can automate the deployment and management of Kubernetes clusters, with Kubespray and Kurator being typical representatives of such open-source tools. This article will compare these two tools.

Kubespray

Kubespray is an open-source project aimed at helping users deploy and manage Kubernetes clusters in multi-cloud environments. To achieve this, Kubespray utilizes Ansible, a trusted open-source automation tool used for automated application deployment, configuration management, and task execution. Based on this, Kubespray can deploy on various cloud platforms such as AWS, GCE, Azure, OpenStack, as well as on bare metal hardware. Additionally, Kubespray has the following advantages:

Support for high-availability clusters
Composable properties like network plugins
Support for various popular Linux distributions
Continuous integration testing

Using Kubespray, users can choose to execute an Ansible script, which then communicates with each target host via SSH protocol and performs tasks such as cluster deployment, cleanup, and upgrades based on the script.

Kurator

Kurator, developed by a cloud-native team, draws from years of excellent practice in the field of distributed cloud-native technology. While Kurator manages the lifecycle of local data center clusters based on Kubespray, its main difference lies in the more understandable and configurable cloud-native declarative approach to cluster management. Specifically, Kurator has designed declarative APIs to express the desired state of a Kubernetes cluster (such as cluster version, node scale, network configuration, etc.) and manages the cluster lifecycle through the Cluster Operator. This approach greatly simplifies user operations: users only need to declare the desired state in the API object, and all remaining tasks can be automatically completed by Kurator’s Cluster Operator. In distributed cloud scenarios, this declarative method provides higher automation and better scalability, making management and operation more convenient and efficient.

The following example shows how to use Kurator to deploy a local data center Kubernetes cluster:

Install Kurator’s Cluster Operator on machines with an already installed Kubernetes cluster.
Create a secret containing SSH keys (an object in Kubernetes used to store sensitive data).
Declare CRDs (Custom Resource Definitions) containing machine and cluster information.
Apply these CRDs to the cluster, and the Cluster Operator will start the automatic installation of the target cluster.

Here are some CRD instances used in the above steps:

CustomMachine for declaring target cluster host information
CustomCluster for declaring cluster properties like network plugins and high availability
KubeadmControlPlane for declaring cluster control plane configurations

The following code example shows how to define ‘CustomMachine’ and ‘CustomCluster’.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: CustomMachine
metadata:
  name: cc-custommachine
  namespace: default
spec:
  master:
    - hostName: master1
      publicIP: 200.x.x.1 
      privateIP: 192.x.x.1 
      sshKey:
        apiVersion: v1
        kind: Secret
        name: cluster-secret
  node:
    - hostName: node1
      publicIP: 200.x.x.2
      privateIP: 192.x.x.2
      sshKey:
        apiVersion: v1
        kind: Secret
        name: cluster-secret
  ...
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: CustomCluster
metadata:
  name: cc-customcluster
  namespace: default
spec:
  cni:
    type: cilium
  controlPlaneConfig:
    address: 192.x.x.0
    certSANs: [200.x.x.1,200.x.x.2]
  machineRef:
    apiVersion: cluster.kurator.dev/v1alpha1
    kind: CustomMachine
    name: cc-custommachine
    namespace: default
  ...

Implementation details of the Cluster Operator

In a Kubernetes environment, once these custom resources (CRs) are created or updated, related events are sent to the API server. The Cluster Operator listens to these CR-related events and creates corresponding manager workers based on the desired state (defined by the spec fields) to adjust the state, i.e., from the current state to the desired state.

These manager workers, created and managed by the Cluster Operator, are special Pods capable of executing the corresponding Ansible cluster management commands to adjust the state. Depending on the difference between the current state and the desired state, the Cluster Operator creates different workers for different adjustments. The Cluster Operator also monitors the completion of these workers to confirm task completion and state updates.

Apart from cluster creation and deletion, processes like cluster node scaling and upgrades follow the same procedure. For example, to upgrade, one simply modifies the version field in the KubeadmControlPlane. Similarly, for scaling, one just adds or deletes node information fields in CustomMachine.

The entire process is illustrated in the following diagram:

Comparison of Kubespray and Kurator

Both Kurator and Kubespray can deploy production-ready Kubernetes clusters in various cloud environments. Kubespray has the previously mentioned advantages, including high-availability options. Kurator inherits Kubespray’s capabilities and also supports high availability. However, these two differ significantly in technical implementation, user experience, and project vision and community.

Technical Implementation

In Kubespray, cluster configuration is mainly done through inventory files and variable files. The inventory file defines the hosts that Ansible needs to manage, while the variable file customizes the Kubernetes cluster. In terms of cluster management, Kubespray relies on executing a series of Ansible playbook commands. Each type of cluster operation has a corresponding Ansible script, covering cluster deployment, scaling, upgrades, and lifecycle management. Executing these scripts requires using the Ansible-playbook command, including the script, access permissions, and other parameter information.

In contrast, Kurator’s implementation method differs. In Kurator, all configuration information is unified in API objects. This means users do not need to manage these configurations from Ansible’s perspective but through declaring API object states. For example, once a user declares the desired state of an API object, the Cluster Operator automatically triggers and executes the corresponding operation. Users do not need to know the specific Ansible scripts used, making the operation more concise and intuitive.

Overall, while Kubespray provides more flexible customization, Kurator offers a simpler and more intuitive cluster configuration and management method with better scalability in a cloud-native environment.

User Experience

As mentioned above, in Kubespray, users need to configure inventory files and variable files, adjust, and execute corresponding Ansible commands based on their needs.

Kurator, on the other hand, uses declarative configuration to manage local Kubernetes clusters, aligning highly with Kubernetes’ core design philosophy. Therefore, for Kubernetes users, Kurator is easier to understand and has a lower learning curve. Compared to Kurator’s approach of merely describing the desired state, Kubespray’s use of Ansible commands may pose a learning challenge for users without Ansible experience.

Additionally, by using API objects, Kurator can save cluster information and management operation records, facilitating user review and tracking. Since the current cluster information is preserved, Kurator can also perform pre-checks before executing operations. For instance, when a user wants to upgrade the Kubernetes version of a cluster, Kurator can judge the appropriateness of the upgrade span before starting the operation.

With the Operator pattern, Kurator can automate the creation and management of clusters. If a cluster operation fails, users can delete the erroneous worker, and Kurator will immediately automatically create a new, functionally identical cluster management worker, ensuring operation idempotency, i.e., repeated operations do not change the system state.

This highly automated approach helps reduce the time and cost of manual intervention, thereby lowering the incidence of human error and improving overall efficiency.

Project Vision and Community

In terms of community vision, Kubespray is positioned to deploy production-ready Kubernetes clusters in various cloud environments and does not focus on managing clusters in distributed cloud environments. In contrast, Kurator aims to be a one-click distributed cloud-native suite. Beyond supporting cluster deployment in various cloud environments, Kurator’s broader goal is to help users build a personalized distributed cloud-native infrastructure to support business distributed upgrades across different cloud and edge environments. Therefore, Kurator’s latest version introduces the concept of “fleet” to provide the ability to consistently manage clusters distributed in any cloud environment.

Additionally, there is a clear difference in the stages of community development. Kubespray is currently very active and mature, with a large number of contributors and users. In contrast, Kurator is still in its early stages but is full of potential and innovation. The Kurator community gathers experienced open-source project contributors and highly values and respects the discussions and contributions of each participant. For newcomers to Kubernetes, Kurator can help easily create clusters and integrate common open-source tools, facilitating a better experience with cloud-native technology. For developers seeking breakthroughs and innovation, Kurator also offers ample space for exploration.

Conclusion and Outlook

The following comparison table summarizes the differences between Kubespray and Kurator in various aspects:

Comparison Aspect	Kubespray	Kurator
Underlying Implementation	Ansible	Ansible
Multi-Cloud Environment Cluster Deployment	✔️	✔️
Reliability	High	High
Cluster Configuration and Management	Inventory/Variable Files + Ansible Commands	API Objects + Automated Cluster Operator
User Friendliness	High customization capability, possibly requiring more learning	Simplified configuration management, reducing learning costs
Community	Active and mature community	Early stage, full of innovation and development potential
Suitable Scenarios	Deploying and managing clusters	Building distributed cloud platforms

From the comparison above, Kubespray and Kurator each have their strengths and characteristics. Kubespray is a mature project with an active community and a high degree of cluster customization. However, Kurator offers more simplified and user-friendly configuration management, making it easier for users to get started and use. Although Kurator’s community is currently not as large as Kubespray’s, it is full of innovation and development potential. Therefore, whether you are a newcomer to Kubernetes or a developer seeking innovation and potential, Kurator can meet your needs.

In future development plans, Kurator will further strengthen its management of the Kubernetes cluster lifecycle, enhancing cluster management capabilities, providing cross-cloud, cross-region, cross-cluster unified and consistent policy management to ensure security and compliance

Blog: Kurator v0.3.0 Version Released! Cluster Fleet Powering Unified Management of Distributed Clouds

Sat, 08 Apr 2023 00:00:00 +0000

April 8, 2023 - Kurator officially releases version v0.3.0.

Kurator is an open-source distributed cloud-native suite that helps users construct a personalized distributed cloud-native infrastructure by integrating mainstream open-source technology stacks, thus facilitating the digital transformation and distributed upgrade of enterprise businesses. The v0.2 version of Kurator already had the core capability to manage multi-cloud and heterogeneous infrastructures, introducing the Cluster Operator component to support lifecycle management features such as cluster creation and cleanup for “AWS self-built clusters” and “local data center clusters.”

In the latest v0.3.0 release, Cluster Operator not only enhances the lifecycle management capabilities for both types of clusters but also abstracts multiple API objects from the v0.2.0 version into a single API object, cluster, for ease of use. Additionally, based on the cluster object, Kurator introduces the concept of a fleet. Each fleet represents a physical cluster group, facilitating unified orchestration, scheduling, traffic governance, and monitoring operations for Kurator in the future. Currently, Kurator’s fleet management supports several features, including the lifecycle management of the fleet control plane and cluster registration and deregistration to the fleet.

With this, Kurator unifies the cluster view through cluster fleets. This means that Kurator now possesses the ability to manage clusters distributed across any cloud with a consistent user experience, further assisting users in upgrading their distributed cloud-native architecture.

Kurator v0.3.0 Key Features Introduction

Enhanced Cluster Lifecycle Management

Kurator manages the lifecycle of clusters through the Cluster Operator component. Based on Cluster API, the Cluster Operator not only manages the cluster lifecycle but also unifies and simplifies the configurations needed to create clusters, providing a simple and user-friendly API for managing clusters on different cloud platforms. Currently, Cluster Operator supports “local data center clusters” and “AWS self-built clusters.”

On-Premise Clusters

Kurator manages the lifecycle of local data center clusters based on kubespray. Different from kubespray, Kurator uses an easier-to-understand and configure cloud-native declarative approach to manage clusters.

Compared to the v0.2.0 version, the v0.3.0 version of Kurator brings the following enhanced features:

Batch scaling of worker nodes. Kurator now supports adding, deleting, or replacing multiple worker nodes in an existing cluster in a declarative manner. Users only need to declare the final desired state of the worker nodes, and Kurator can complete the batch scaling of the nodes without any external intervention.
Cluster version upgrade. Users can declare the Kubernetes version they want to upgrade to on the API object, and Kurator will automatically upgrade the nodes of the target cluster.
Enhanced high availability of the cluster control plane. Kurator provides a VIP-based enhanced cluster control plane high availability solution. In this scheme, Kurator utilizes the capabilities of kube-vip to achieve inbound traffic load balancing across multiple control plane replicas using a VIP.

User Manual: Kurator on-premise Cluster Lifecycle

AWS Self-Built Clusters

Kurator manages the lifecycle of AWS self-built clusters through the Cluster Operator. Compared to the Cluster API support for AWS self-built clusters, Kurator simplifies the deployment model provided by the Cluster API, obtaining full management capabilities by deploying the Kurator cluster operator component.

v0.3.0 brings the following feature enhancements:

Improved usability. Kurator has added a series of user experience improvements, including validating credentials before creating a cluster, automatically managing IAM roles and policies required by cloud platform operators, verifying the existence of dependent resources, and centrally displaying error information.
One-click association of IAM with K8s identity. By associating AWS IAM roles with Kubernetes Pod identities, IAM can verify and accept tokens issued by Kubernetes, eliminating the need to create and distribute AWS credentials. This association has advantages like minimum privilege, credential isolation, and auditability, but it requires multiple steps to set up. Kurator now enables this feature with one click through Cluster.Spec.PodIdentity, simplifying the configuration.

apiVersion: cluster.kurator.dev/v1alpha1
kind: Cluster
metadata:
 name: pod-identity
 namespace: default
spec:
 ...
 podIdentity:
   enabled: true

User Manuals:

Kurator Cluster API

AWS IRSA

Cloud-Native Fleet Management

Kurator introduces the logical unit “fleet,” representing a physical cluster group, aiming to manage a set of clusters uniformly. Fleets allow you to manage clusters distributed across any cloud easily and consistently.

Kurator implements fleet control plane lifecycle management through the Fleet Manager and can easily add or remove clusters from a fleet. In the future, Kurator will support Fleet-level application orchestration and provide unified namespaces, ServiceAccounts, and Services across all clusters in the Fleet to enable service discovery and communication between multiple clusters. Additionally, Kurator will aggregate monitoring metrics from all clusters.

The Kurator Fleet Manager operates as a Kubernetes Operator, responsible for the lifecycle management of the Fleet control plane and the registration and deregistration of clusters.

User Manual: Kurator Fleet Manager

Kurator: One-Click Construction of Distributed Cloud-Native Platform

Visit Kurator Release v0.3.0 to experience and upgrade to the latest version of Kurator v0.3.0, and build your personalized distributed cloud-native platform. If you are interested in or have insights into the new features of Kurator, you are also welcome to join the Kurator community to participate in community discussions and development.

GitHub Address: Kurator on GitHub
Slack Address: Join Kurator on Slack

Blog: Distributed Cloud-Native Platform Kurator v0.2.0 Officially Released! One-Click Construction of Distributed Cloud-Native Platform

Thu, 09 Feb 2023 00:00:00 +0000

Beijing Time, February 9, 2023 - Kurator has officially released version v0.2.0.

Kurator is an open-source distributed cloud-native platform that helps users build their own distributed cloud-native infrastructure, aiding in the digital transformation of enterprises. Version v0.1 of Kurator integrated mainstream open-source projects like Karmada, Volcano, Istio, and Prometheus, providing unified multi-cluster management, scheduling, traffic governance, and unified application monitoring capabilities for distributed cloud-native environments.

In the latest release, v0.2.0, Kurator has added two major categories of key features, enhancing observability and introducing cluster lifecycle management, including the following significant updates:

Multi-cluster monitoring and metric persistence storage based on Thanos
Real-time K8s application monitoring based on Pixie
Support for lifecycle management of local data center clusters
Support for lifecycle management of self-built clusters on AWS Cloud

With these updates, Kurator now provides management capabilities for distributed cloud-native infrastructure, meaning that Kurator can rely on infrastructure and Kubernetes clusters to better manage various cloud-native middleware, providing out-of-the-box distributed cloud-native capabilities for users.

Key Features of Kurator

Observability

Multi-Cluster Monitoring and Metric Persistence Storage Based on Thanos: Thanos is an open-source, highly available Prometheus solution with persistent storage capabilities. It is a CNCF incubation project and one of the most popular multi-cluster monitoring projects. Core features of Thanos include global query views, high availability, data backup, history, and low-cost data access.

Kurator offers a simple command to install Thanos, facilitating the quick construction of multi-cloud and multi-cluster monitoring systems.

User Manual: Thanos Documentation
Real-Time K8s Application Monitoring with Pixie: Pixie, an open-source Kubernetes observability tool, enables users to view advanced cluster states (service topology, cluster resources, application traffic) and delve into more detailed views (Pod status, flame graphs, individual full-body application requests). Pixie uses eBPF to automatically collect telemetry data such as requests, resource and network metrics, application profiles, offering a completely non-intrusive approach to application monitoring with very low CPU usage, typically under 2%. Users can now enable Pixie’s real-time monitoring with one click.

User Manual: Pixie Vizier Documentation

Cluster Lifecycle Management

New Component, Cluster Operator: Provides cloud-native ways to manage Kubernetes clusters. Cluster Operator can easily manage the lifecycle of Kubernetes clusters across various infrastructures, including public clouds, hybrid clouds, and local data centers.

The overall architecture of the Kurator Cluster Operator is as follows:

The Kurator Cluster Operator is equipped with various controllers that monitor cluster API objects and then create Kubernetes clusters on designated infrastructures.

Design Document: Cluster Operator Documentation

Support for Local Data Center Self-Built Clusters: Lifecycle management capabilities for local data center clusters are primarily based on KubeSpray. It allows for the deployment of production-grade Kubernetes clusters on existing virtual machines and bare-metal servers using a declarative API and manages their lifecycle. Local data center self-built clusters fully utilize the open-source K8s software stack for rapid deployment, setting up container networking, DNS servers, and more.

User Manual: On-Premise Deploy Documentation
Support for AWS Cloud Self-Built Clusters: Kurator supports building clusters in the AWS public cloud using the integrated Cluster API Provider AWS. It offers an experience consistent with the AWS managed Kubernetes service EKS, leveraging AWS infrastructure (Elastic Cloud Servers, VPC networks, load balancers, etc.). Resources required for AWS self-built clusters include:
- Cluster: Represents a complete cluster, including references to the KubeadmControlPlane and the infrastructure AWSCluster.
- KubeadmControlPlane: Represents all the configurations needed for kubeadm to install the Kubernetes control plane.
- AWSCluster: Represents AWS as the infrastructure for the Cluster.
- AWSMachineTemplate: Represents the template for creating AWS EC2 instances.
The cluster resource topology structure is as follows:

Through the Cluster, KubeadmControlPlane, and other Cluster API controllers, as well as the provider offered by AWS, Kurator manages the lifecycle of self-built clusters in AWS according to given configurations.

User Manual: AWS Deploy Documentation

Kurator: One-Click Construction of Distributed Cloud-Native Platform

Kurator, first launched in June 2022, is the industry’s first open-source distributed cloud-native platform. By integrating mainstream open-source technology stacks and excellent cloud-native fleet management performance, Kurator provides a one-stop, ready-to-use distributed cloud-native capability, building a solid foundation for distributed cloud-native technology and helping businesses upgrade to distributed, cloud, and edge computing.

GitHub Address: Kurator on GitHub