The Power of GitOps

• Understanding GitOps: We'll start by dissecting the concept of GitOps, exploring its origins, and why it has become a game-changer in the Kubernetes ecosystem.

• ArgoCD Unveiled: A deep dive into ArgoCD, including its architecture, components, and how it manages Kubernetes applications declaratively.

• Git Repository as the Source of Truth: Learn why your Git repository becomes the ultimate source of truth for your Kubernetes configurations, bringing transparency, accountability, and robust version control to your deployments.

Setting Up ArgoCD

Installation and Configuration

• To get started with ArgoCD, you'll first need to install it in your Kubernetes cluster. This can be achieved using a simple Kubernetes manifest or a Helm chart. We will walk you through both methods, providing guidance on choosing the one that best fits your requirements.

  Example:

    # ArgoCD Installation via Helm
    helm repo add argo https://argoproj.github.io/argo-helm
    helm install my-argocd argo/argo-cd
  

  Once installed, you'll need to configure ArgoCD to connect to your Git repository where your Kubernetes manifests and ArgoCD Application configurations are stored. We'll show you how to set up these configurations, including connecting to private Git repositories securely.

  Multi-Cluster Support

  ArgoCD excels in managing applications across multiple Kubernetes clusters. We'll guide you through the process of configuring ArgoCD to support multi-cluster environments, allowing you to deploy and manage applications seamlessly, no matter how complex your infrastructure.

  Role-Based Access Control (RBAC)

  Securing your ArgoCD instance is paramount. We'll provide best practices for implementing Role-Based Access Control (RBAC) within ArgoCD. You'll learn how to create custom roles, define permissions, and manage user access effectively.

  Example:

    # Define a custom Role in ArgoCD
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: my-custom-role
    rules:
    - apiGroups: [""]
      resources: ["applications"]
      verbs: ["get", "list", "create", "update", "delete"]
  

Deploying Applications with ArgoCD

Now that you have ArgoCD set up, it's time to dive into deploying applications. In this chapter, we'll guide you through defining Application Custom Resource Definitions (CRDs) and deploying your first application.

Defining Application CRDs

Applications in ArgoCD are defined using Application Custom Resource Definitions (CRDs). These YAML manifests declare how ArgoCD should deploy and manage your application.

Example:

# Application CRD for a sample application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-app
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/your-org/your-repo
    targetRevision: HEAD
    path: apps/sample
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: sample-namespace
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Here, we're defining an application named sample-app that resides in a Git repository. ArgoCD will automatically synchronize this application according to the defined sync policy.

Syncing Applications

ArgoCD continuously monitors your Git repositories for changes. When it detects updates, it syncs applications to match the desired state defined in your Git repository.

Example:

# Triggering a manual sync for an application
argocd app sync sample-app

ArgoCD also provides a web-based dashboard for easy application monitoring and manual syncing.

Application Rollbacks

Sometimes, things don't go as planned. ArgoCD supports application rollbacks, allowing you to easily revert to a previous version of your application when issues arise.

Example:

# Rollback an application to a previous version
argocd app rollback sample-app

Manifests and Application Resources

In this section, we'll explore how ArgoCD deals with Kubernetes manifests and different application resources. By the end, you'll have a solid understanding of how to manage and deploy complex applications using ArgoCD.

Managing Manifests

ArgoCD shines when it comes to managing Kubernetes manifests. You can keep your application manifests in a Git repository, and ArgoCD will automatically synchronize them with your Kubernetes clusters.

Example: Here's a simplified directory structure for your application's manifests:

my-app/
├── deployment.yaml
├── service.yaml
├── configmap.yaml
├── route.yaml

ArgoCD will track changes in this repository and ensure that the application state matches the definitions in these manifests.

Resource Dependency

Kubernetes applications often consist of multiple resources that depend on each other. For example, a Deployment may depend on a ConfigMap and a Service.

Example: In the deployment.yaml, you might reference a ConfigMap:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
spec:
  template:
    spec:
      containers:
        - name: my-app
          image: my-app-image
          envFrom:
            - configMapRef:
                name: my-app-config

ArgoCD understands these dependencies and ensures that resources are synchronized in the correct order.

Application Resources

Applications often involve various Kubernetes resources. ArgoCD makes managing these resources straightforward.

Example: Consider a YAML manifest for a Service:

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080

ArgoCD handles resources like Services, Deployments, ConfigMaps, and Routes seamlessly, deploying and managing them according to your Git repository's state.

Deployment and Synchronization

Whenever you push changes to your Git repository, ArgoCD automatically detects the updates and synchronizes your application to match the desired state.

Example: After you push an update to your manifests, ArgoCD will sync the application:

# Triggering a sync
argocd app sync my-app

Sealed Secrets with ArgoCD

In this section, we'll dive deeper into the world of Sealed Secrets and see how ArgoCD can securely simplify the management of secrets.

What are Sealed Secrets?

Sealed Secrets are an innovative way of managing Kubernetes secrets in a GitOps workflow. They work by encrypting your sensitive data and storing it as a SealedSecret custom resource in your Git repository.

How to Use Sealed Secrets with ArgoCD

Step 1: Install Sealed Secrets Controller

The first step is to install the Sealed Secrets controller in your Kubernetes cluster. This controller is responsible for decrypting SealedSecrets objects during runtime.

shellCopy code# Install Sealed Secrets Controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml

Step 2: Create a Sealed Secret

Now, let's create a Sealed Secret from a regular Kubernetes Secret. You'll need to use the kubeseal command-line tool.

shellCopy code# Create a Sealed Secret from a regular Secret
kubectl create secret generic my-secret --dry-run=client --from-literal=my-key=my-value -o json | kubeseal > my-sealed-secret.yaml

This command generates a SealedSecret YAML file that you can store in your Git repository.

Step 3: Add Sealed Secret to Git Repository

Commit the my-sealed-secret.yaml file to your Git repository, just like you would with other Kubernetes manifests.

Step 4: Deploy Sealed Secret with ArgoCD

ArgoCD can automatically deploy Sealed Secrets. Create an ArgoCD Application resource for your Sealed Secret, just as you would for any other resource.

yamlCopy codeapiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  project: default
  source:
    repoURL: <URL_TO_YOUR_GIT_REPO>
    targetRevision: HEAD
    path: path/to/secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: my-namespace

ArgoCD will handle the deployment of the Sealed Secret to your cluster securely.

Benefits of Using Sealed Secrets with ArgoCD

1. Security: Sensitive data is encrypted and stored in version control, reducing the risk of exposure.

2. GitOps: Sealed Secrets fit perfectly into the GitOps workflow. You define, version, and track your secrets in your Git repository.

3. Auditing: All changes to secrets are traceable through Git commits, providing a clear audit trail.

4. Easy Management: ArgoCD simplifies the deployment and synchronization of Sealed Secrets, making secret management a breeze.

Enhancing Security with Sealed Secrets

• The Challenge of Secrets: Delve into the critical issue of securing sensitive data within Kubernetes and why traditional Secrets fall short.

• Sealed Secrets Revealed: Learn about the architecture of Sealed Secrets and how it encrypts your secrets into a safe, Git-friendly format.

• Public Key Encryption: Understand how Sealed Secrets leverages public key encryption to ensure that only your target cluster can decrypt and utilize the secrets

The Future of Kubernetes Deployments

• Emerging Trends: Get a glimpse into the future of Kubernetes deployments, including the evolution of GitOps, new security paradigms, and the latest tools on the horizon.

This blog aims to provide you with a comprehensive understanding of these powerful Kubernetes tools, offering the knowledge and expertise you need to optimize your Kubernetes deployments, enhance security, and stay ahead in the ever-changing world of DevOps. Get ready to embark on a deep dive into the world of ArgoCD and Sealed Secrets.

In the rapidly evolving world of Artificial Intelligence (AI), the ability to analyze and understand text data is becoming increasingly important. Amazon Web Services (AWS) offers a powerful service called Amazon Comprehend that allows developers to build Natural Language Processing (NLP) applications without the need for extensive machine learning expertise. In this blog post, we will explore the process of prototyping AI APIs using AWS Comprehend to extract valuable insights from text data.

Understanding AWS Comprehend

Amazon Comprehend is a fully managed NLP service that enables businesses to extract useful information and relationships from unstructured text data. It can identify entities, key phrases, sentiments, language, and more from a variety of text sources, including documents, social media, and customer feedback.

Getting Started: Setting Up AWS Comprehend:

1. AWS Account Setup: If you don't already have an AWS account, sign up and create one.

2. Access Amazon Comprehend: Navigate to the AWS Management Console and locate the Amazon Comprehend service.

3. Create a Custom IAM Role: Create an IAM role that grants necessary permissions to Amazon Comprehend.

Prototyping AI APIs using AWS Comprehend:

1. Text Sentiment Analysis:

   • Use the AWS SDK to interact with the Comprehend service programmatically.

   •   $ aws comprehend help
       NAME
              comprehend -
     
       DESCRIPTION
              Amazon Comprehend is an Amazon Web Services service for gaining insight
              into the content of documents. Use these actions to determine the  top-
              ics  contained in your documents, the topics they discuss, the predomi-
              nant sentiment expressed in them, the predominant  language  used,  and
              more.
     
       AVAILABLE COMMANDS
              o batch-detect-dominant-language
     
              o batch-detect-entities
     
              o batch-detect-key-phrases
     
              o batch-detect-sentiment
     
              o batch-detect-syntax
     
              o batch-detect-targeted-sentiment
     
              o classify-document
     
              o contains-pii-entities
     
              o create-dataset
     
              o create-document-classifier
     
              o create-endpoint
     
              o create-entity-recognizer
     
              o create-flywheel
     
              o delete-document-classifier
     
              o delete-endpoint
     
              o delete-entity-recognizer
     
              o delete-flywheel
     
              o delete-resource-policy
     
              o describe-dataset
     

   • Let's say we want to do the sentiment analysis of text - "I love Python Programming language"

       $ aws comprehend detect-sentiment --language-code "en" --text "I love Python programming language"
       {
           "Sentiment": "POSITIVE",
           "SentimentScore": {
               "Positive": 0.9986007809638977,
               "Negative": 9.051552478922531e-05,
               "Neutral": 0.0011894343188032508,
               "Mixed": 0.00011932779307244346
           }
       }
     

     We can see the sentiment analysis is POSITIVE which is very true.

   • Now let's say we want to analyse a document or a data from website which is very simple. We will use lynx command to get the data from website into a variable.

       $ TEXT=`lynx -dump https://en.wikipedia.org/wiki/Python_\(programming_language\) | head -c 5000`
       $
     

   • Pass the $TEXT variable to a detect-sentiment command

       $ aws comprehend detect-sentiment --language-code "en" --text "$TEXT"
       {
           "Sentiment": "NEUTRAL",
           "SentimentScore": {
               "Positive": 0.00020325076184235513,
               "Negative": 0.0005437986110337079,
               "Neutral": 0.999243974685669,
               "Mixed": 9.053033863892779e-06
           }
       }
     

   • This is how Comprehend classifies text as positive, negative, neutral, or mixed sentiment.

1. Entity Recognition:

   • Detects named entities in input text when you use the pre-trained model. Detects custom entities if you have a custom entity recognition model.

     When detecting named entities using the pre-trained model, use plain text as the input.

   • Let's say we want to analysis our website data and want to know about word count for each word.

       $ aws comprehend detect-entities --language-code "en" --text "$TEXT" --output text | cut -f 5 |sort|uniq -c |sort -nr -k 1
             4 Wikipedia
             2 Python
             2 Norsk
             2 English
             2 Bahasa
             2 한국어
             1 粵語
             1 日本語
             1 吴语
             1 中文
             1 සිංහල
             1 မြန်မာဘာသာ
             1 বাংলা
             1 অসমীয়া
             1 മലയാളം
             1 தமிழ்
     

   • Explore how Comprehend can enhance data extraction and organization.

2. Key Phrase Extraction:

   • Detects the key noun phrases found in the text.

       $ aws comprehend detect-key-phrases --language-code "en" --text "$TEXT" --output text
       KEYPHRASES      3       6       0.8861472606658936      #[1
       KEYPHRASES      7       29      0.7588788866996765      alternate [2]Wikipedia
       KEYPHRASES      35      57      0.8782252073287964      [3]Wikipedia Atom feed
       KEYPHRASES      62      69      0.7381764054298401      [4]Jump
       KEYPHRASES      73      80      0.5808557271957397      content
       KEYPHRASES      85      98      0.5795749425888062      [ ] Main menu
       KEYPHRASES      107     111     0.5609327554702759      menu
       KEYPHRASES      116     122     0.9981253743171692      BUTTON
       KEYPHRASES      141     147     0.9974844455718994      BUTTON
       KEYPHRASES      157     167     0.9399071335792542      Navigation
       KEYPHRASES      175     187     0.8160272836685181      [5]Main page
       KEYPHRASES      195     206     0.7837936878204346      [6]Contents
       KEYPHRASES      215     231     0.7777291536331177      7]Current events
       KEYPHRASES      239     256     0.858264684677124       [8]Random article
       KEYPHRASES      264     267     0.7898728847503662      [9]
       KEYPHRASES      273     282     0.8571096062660217      Wikipedia
       KEYPHRASES      290     294     0.7877490520477295      [10]
       KEYPHRASES      312     337     0.8975982069969177      [11]Donate
     

Use Case: Analyzing Customer Feedback

This kind of tool is very helpful, where a company can use AWS Comprehend to analyze customer feedback from various sources. They can improve customer satisfaction by doing sentiment analysis, entity recognition, and key phrase extraction.

Understanding Kubernetes Operators

Before we dive into the technical details, let's understand what a Kubernetes Operator is. An Operator is an extension of Kubernetes that automates the deployment, management, and scaling of applications. It encapsulates domain-specific knowledge to ensure your applications run seamlessly within a Kubernetes environment.

Why GoLang for Developing Operators?

Go, also known as Golang, has gained immense popularity for its simplicity, efficiency, and performance. It's an ideal choice for building Kubernetes Operators due to its strong concurrency support and suitability for systems programming.

Operator SDK

The Operator SDK provides building blocks that simplify Operator development. The Operator SDK makes it easier to build Kubernetes native applications, a process that can require deep, application-specific operational knowledge. Use the SDK to create controllers, handlers, and operators that follow best practices and ensure consistency across your project.

Start building the Operator

Prerequisite

You should have a working Kubernetes cluster or Openshift cluster.

• git

• go

• docker

• kubectl and access to a Kubernetes cluster of a compatible version.

Gihub Repository used for this tutorial

Code for this tutorial is here -

https://github.com/techwithhuz/kubernetes-operator/tree/main/techwithhuz-operator

Step 1: Set Up Your Development Environment

Start by installing the necessary tools: Go, the Kubernetes command-line tool (kubectl), and the Operator SDK.

Install the Operator SDK CLI. We will use the GitHub release method to install Operator SDK. For Mac users brew install is available.

Install from GitHub release

Prerequisites

• curl

• gpg version 2.0+

1.Download the release binary

Set platform information:

$export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
$export OS=$(uname | awk '{print tolower($0)}')

Download the binary for your platform:

$export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.31.0
$curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}

2. Verify the downloaded binary

Import the operator-sdk release GPG key from keyserver.ubuntu.com:

$gpg --keyserver keyserver.ubuntu.com --recv-keys 052996E2A20B5C7E

Download the checksums file and its signature, then verify the signature:

$curl -LO ${OPERATOR_SDK_DL_URL}/checksums.txt
$curl -LO ${OPERATOR_SDK_DL_URL}/checksums.txt.asc
$gpg -u "Operator SDK (release) <cncf-operator-sdk@cncf.io>" --verify checksums.txt.asc

You should see something similar to the following:

gpg: assuming signed data in 'checksums.txt'
gpg: Signature made Fri 30 Oct 2020 12:15:15 PM PDT
gpg:                using RSA key ADE83605E945FA5A1BD8639C59E5B47624962185
gpg: Good signature from "Operator SDK (release) <cncf-operator-sdk@cncf.io>" [ultimate]

Make sure the checksums match:

$grep operator-sdk_${OS}_${ARCH} checksums.txt | sha256sum -c -

You should see something similar to the following:

operator-sdk_linux_amd64: OK

1. Install Go

   Download the latest version of Go - https://go.dev/dl

   Then unzip it.

$ wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
Saving to: 'go1.21.0.linux-amd64.tar.gz'

go1.21.0.linux-amd64.tar.gz          100%[===================================================================>]  63.40M  85.2MB/s    in 0.7s    

2023-08-13 08:30:42 (85.2 MB/s) - 'go1.21.0.linux-amd64.tar.gz' saved [66479500/66479500]

$tar -xvf go1.21.0.linux-amd64.tar.gz

$pwd
/root/operator

1. Set PATH variable for GO

   You can set the custom path where you downloaded the GO binary or you can copy the binaries to the default /usr/local path

   Here, we will be set to the custom path and check the version

$ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/operator/go/bin:/snap/bin:/root/operator/go/bin:/snap/bin

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/operator/go/bin:/snap/bin:/root/operator/go/bin:/snap/bin

 $ go version
go version go1.21.0 linux/amd64

Step 2: Start building the Operator

We will create and deploy one demo application with the name techwithhuz through the GoLang operator and we will expose it with a service. The application will be a simple Deployment yaml which will have a nginx image.

1. Create a new project

   Use the CLI to create a new techwithhuz-operator project

$ mkdir techwithuz-operator
$ cd techwithuz-operator/

$ operator-sdk init --domain techwithhuz.com --repo github.com/techwithhuz/techwithhuz-operator
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.14.1
go: downloading sigs.k8s.io/controller-runtime v0.14.1
go: downloading k8s.io/apimachinery v0.26.0
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/google/gofuzz v1.1.0
go: downloading github.com/go-logr/logr v1.2.3
go: downloading k8s.io/client-go v0.26.0
go: downloading k8s.io/klog/v2 v2.80.1
go: downloading k8s.io/utils v0.0.0-20221128185143-99ec85e7a448
go: downloading github.com/prometheus/client_golang v1.14.0
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3
go: downloading golang.org/x/net v0.3.1-0.20221206200815-1e63c2f08a10
go: downloading github.com/evanphx/json-patch/v5 v5.6.0
go: downloading github.com/evanphx/json-patch v4.12.0+incompatible
go: downloading golang.org/x/time v0.3.0
go: downloading gomodules.xyz/jsonpatch/v2 v2.2.0
go: downloading k8s.io/api v0.26.0
go: downloading k8s.io/apiextensions-apiserver v0.26.0
go: downloading github.com/imdario/mergo v0.3.6
go: downloading github.com/spf13/pflag v1.0.5
go: downloading golang.org/x/term v0.3.0
go: downloading k8s.io/component-base v0.26.0
go: downloading github.com/prometheus/client_model v0.3.0
go: downloading github.com/prometheus/common v0.37.0
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading sigs.k8s.io/json v0.0.0-20220713155537-f223a00ba0e2
go: downloading github.com/json-iterator/go v1.1.12
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b
go: downloading github.com/pkg/errors v0.9.1
go: downloading k8s.io/kube-openapi v0.0.0-20221012153701-172d655c2280
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/google/gnostic v0.5.7-v3refs
go: downloading golang.org/x/sys v0.3.0
go: downloading sigs.k8s.io/yaml v1.3.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/prometheus/procfs v0.8.0
go: downloading google.golang.org/protobuf v1.28.1
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.2
go: downloading github.com/google/uuid v1.1.2
go: downloading github.com/fsnotify/fsnotify v1.6.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading golang.org/x/text v0.5.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/google/go-cmp v0.5.9
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
go: downloading github.com/emicklei/go-restful/v3 v3.9.0
go: downloading github.com/go-openapi/swag v0.19.14
go: downloading github.com/go-openapi/jsonreference v0.20.0
go: downloading github.com/mailru/easyjson v0.7.6
go: downloading github.com/go-openapi/jsonpointer v0.19.5
go: downloading github.com/josharian/intern v1.0.0
Update dependencies:
$ go mod tidy
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/onsi/ginkgo/v2 v2.6.0
go: downloading github.com/onsi/gomega v1.24.1
go: downloading github.com/go-logr/zapr v1.2.3
go: downloading go.uber.org/zap v1.24.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading go.uber.org/goleak v1.2.0
go: downloading go.uber.org/atomic v1.7.0
go: downloading go.uber.org/multierr v1.6.0
go: downloading gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: downloading github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/benbjohnson/clock v1.1.0
Next: define a resource with:
$ operator-sdk create api

$ pwd
/root/techwithuz-operator
$ ls
Dockerfile  Makefile  PROJECT  README.md  config  go.mod  go.sum  hack  main.go

The above operator-sdk command creates all the necessary folder structures for us inside the techwithhuz-operator folder.

1. Create the api for techwithhuz

$ operator-sdk create api --group cache --version v1alpha1 --kind Techwithhuz --resource --controller
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1alpha1/techwithhuz_types.go
controllers/techwithhuz_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
test -s /root/techwithuz-operator/bin/controller-gen && /root/techwithuz-operator/bin/controller-gen --version | grep -q v0.11.1 || \
GOBIN=/root/techwithuz-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.1
go: downloading github.com/spf13/cobra v1.6.1
go: downloading github.com/gobuffalo/flect v0.3.0
go: downloading golang.org/x/tools v0.4.0
go: downloading github.com/fatih/color v1.13.0
go: downloading k8s.io/utils v0.0.0-20221107191617-1a15be271d1d
go: downloading github.com/mattn/go-colorable v0.1.9
go: downloading github.com/mattn/go-isatty v0.0.14
go: downloading golang.org/x/mod v0.7.0
go: downloading golang.org/x/net v0.4.0
/root/techwithuz-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
$ 
$ ls
Dockerfile  Makefile  PROJECT  README.md  api  bin  config  controllers  go.mod  go.sum  hack  main.go

$ ls api/v1alpha1/techwithhuz_types.go 
api/v1alpha1/techwithhuz_types.go

Now we have the API folder created with the above command. Also, we can see we have the types.go file available.

Step 3: Define the API

To begin, we will represent our API by defining the Techwithhuz type, which will have a Techwithhuz.Size and Techwithhuz.ContainerPort field to set the number of our instances (CRs) to be deployed and container port to be used, and a TechwithhuzStatus.Conditions field to store a CR’s Conditions.

Define the API for the Techwithhuz Custom Resource(CR) by modifying the Go type definitions at api/v1alpha1/techwithhuz_types.go to have the following spec and status:

// TechwithhuzSpec defines the desired state of Techwithhuz
type TechwithhuzSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // Foo is an example field of Techwithhuz. Edit techwithhuz_types.go to remove/update
    //Foo string `json:"foo,omitempty"`
    //Add size and containerport property for which values can be passed through Custom Resource(CR) file.
    Size          int32 `json:"size,omitempty"`
    ContainerPort int32 `json:"containerPort,omitempty"`
}

type TechwithhuzStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    // Conditions store the status conditions of the TechWithHuz instances
    // +operator-sdk:csv:customresourcedefinitions:type=status
    Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,1,rep,name=conditions"`
}

Now add the +kubebuilder:subresource:status marker to add a status subresource to the CRD manifest so that the controller can update the CR status without changing the rest of the CR object:

// Techwithhuz is the Schema for the techwithhuzs API
type Techwithhuz struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   TechwithhuzSpec   `json:"spec,omitempty"`
    Status TechwithhuzStatus `json:"status,omitempty"`
}

After modifying the *_types.go file always runs the following command to update the generated code for that resource type

$ make generate
test -s /root/kubernetes-operator/techwithhuz-operator/bin/controller-gen && /root/kubernetes-operator/techwithhuz-operator/bin/controller-gen --version | grep -q v0.11.1 || \
GOBIN=/root/kubernetes-operator/techwithhuz-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.1
/root/kubernetes-operator/techwithhuz-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."

The above makefile target will invoke the controller-gen utility to update the api/v1alpha1/zz_generated.deepcopy.go file to ensure our API’s Go type definitions implement the runtime.Object interface that all Kind types must implement.

Step 4: Generating CRD manifests

Once the API is defined with spec/status fields and CRD validation markers, the CRD manifests can be generated and updated with the following command.

$ make manifests
test -s /root/kubernetes-operator/techwithhuz-operator/bin/controller-gen && /root/kubernetes-operator/techwithhuz-operator/bin/controller-gen --version | grep -q v0.11.1 || \
GOBIN=/root/kubernetes-operator/techwithhuz-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.1
/root/kubernetes-operator/techwithhuz-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

This makefile target will invoke controller-gen to generate the CRD manifests at config/crd/bases/cache.example.com_memcacheds.yaml.

Step 5: Implement the Controller

The controller file will contain the reconciliation logic. We have the controller file already generated controller/techwithuz_controller.go. Now we need to write the logic like in our case we want to create a kind Deployment for our application Techwithhuz.

So we need to write the yaml file of kind deployment but in Go lang format.

We will see now how to do that.

// deploymentForTechwithhuz returns a Techwithhuz Deployment object
func (r *TechwithhuzReconciler) deploymentForTechwithhuz(techwithhuz *cachev1alpha1.Techwithhuz) (*appsv1.Deployment, error) {
    replicas := techwithhuz.Spec.Size
    ls := labelsForTechwithhuz(techwithhuz.Name)

    // Get the Operand image
    image := "nginx:latest"
    dep := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      techwithhuz.Name,
            Namespace: techwithhuz.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: ls,
                },
                Spec: corev1.PodSpec{
                    SecurityContext: &corev1.PodSecurityContext{
                        RunAsNonRoot: &[]bool{true}[0],
                        SeccompProfile: &corev1.SeccompProfile{
                            Type: corev1.SeccompProfileTypeRuntimeDefault,
                        },
                    },
                    Containers: []corev1.Container{{
                        Image:           image,
                        Name:            "techwithhuz",
                        ImagePullPolicy: corev1.PullIfNotPresent,
                        SecurityContext: &corev1.SecurityContext{
                            RunAsNonRoot:             &[]bool{true}[0],
                            RunAsUser:                &[]int64{1001}[0],
                            AllowPrivilegeEscalation: &[]bool{false}[0],
                            Capabilities: &corev1.Capabilities{
                                Drop: []corev1.Capability{
                                    "ALL",
                                },
                            },
                        },
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: techwithhuz.Spec.ContainerPort,
                            Name:          "techwithhuz",
                        }},
                        Command: []string{"sleep", "1000s"},
                    }},
                },
            },
        },
    }
    if err := ctrl.SetControllerReference(techwithhuz, dep, r.Scheme); err != nil {
        return nil, err
    }
    return dep, nil
}

As we can see in the above code we have written the logic for Kind Deployment in the Go language.

To know more about how to write the above code and what all functions are available refer to this link - https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#deploymentspec-v1-apps also you can refer to the API document link from GO - https://pkg.go.dev/k8s.io/api/core/v1

Now we have to write the logic for reconciliation like when the user set Size as 3 in the CR file, then our reconcile logic should increase the pod to 3.

func (r *TechwithhuzReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    //_ = log.FromContext(ctx)
    log := log.FromContext(ctx)

    // TODO(user): your logic here
    // Fetch the Techwithhuz instance
    // The purpose is check if the Custom Resource for the Kind Techwithhuz
    // is applied on the cluster if not we return nil to stop the reconciliation
    techwithhuz := &cachev1alpha1.Techwithhuz{}
    err := r.Get(ctx, req.NamespacedName, techwithhuz)
    if err != nil {
        if apierrors.IsNotFound(err) {
            // If the custom resource is not found then, it usually means that it was deleted or not created
            // In this way, we will stop the reconciliation
            log.Info("techwithhuz resource not found. Ignoring since object must be deleted")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request.
        log.Error(err, "Failed to get techwithhuz")
        return ctrl.Result{}, err
    }
    // Let's just set the status as Unknown when no status are available
    if techwithhuz.Status.Conditions == nil || len(techwithhuz.Status.Conditions) == 0 {
        meta.SetStatusCondition(&techwithhuz.Status.Conditions, metav1.Condition{Type: typeAvailableTechwithhuz, Status: metav1.ConditionUnknown, Reason: "Reconciling", Message: "Starting reconciliation"})
        if err = r.Status().Update(ctx, techwithhuz); err != nil {
            log.Error(err, "Failed to update techwithhuz status")
            return ctrl.Result{}, err
        }

        // Let's re-fetch the techwithhuz Custom Resource after update the status
        // so that we have the latest state of the resource on the cluster and we will avoid
        // raise the issue "the object has been modified, please apply
        // your changes to the latest version and try again" which would re-trigger the reconciliation
        // if we try to update it again in the following operations
        if err := r.Get(ctx, req.NamespacedName, techwithhuz); err != nil {
            log.Error(err, "Failed to re-fetch techwithhuz")
            return ctrl.Result{}, err
        }
    }
    // Let's add a finalizer. Then, we can define some operations which should
    // occurs before the custom resource to be deleted.
    // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers
    if !controllerutil.ContainsFinalizer(techwithhuz, techwithhuzFinalizer) {
        log.Info("Adding Finalizer for techwithhuz")
        if ok := controllerutil.AddFinalizer(techwithhuz, techwithhuzFinalizer); !ok {
            log.Error(err, "Failed to add finalizer into the custom resource")
            return ctrl.Result{Requeue: true}, nil
        }

        if err = r.Update(ctx, techwithhuz); err != nil {
            log.Error(err, "Failed to update custom resource to add finalizer")
            return ctrl.Result{}, err
        }
    }
    // Check if the Techwithhuz instance is marked to be deleted, which is
    // indicated by the deletion timestamp being set.
    isTechwithhuzMarkedToBeDeleted := techwithhuz.GetDeletionTimestamp() != nil
    if isTechwithhuzMarkedToBeDeleted {
        if controllerutil.ContainsFinalizer(techwithhuz, techwithhuzFinalizer) {
            log.Info("Performing Finalizer Operations for techwithhuz before delete CR")

            // Let's add here an status "Downgrade" to define that this resource begin its process to be terminated.
            meta.SetStatusCondition(&techwithhuz.Status.Conditions, metav1.Condition{Type: typeDegradedTechwithhuz,
                Status: metav1.ConditionUnknown, Reason: "Finalizing",
                Message: fmt.Sprintf("Performing finalizer operations for the custom resource: %s ", techwithhuz.Name)})

            if err := r.Status().Update(ctx, techwithhuz); err != nil {
                log.Error(err, "Failed to update techwithhuz status")
                return ctrl.Result{}, err
            }

            // Perform all operations required before remove the finalizer and allow
            // the Kubernetes API to remove the custom resource.
            r.doFinalizerOperationsForTechwithhuz(techwithhuz)

            // Re-fetch the techwithhuz Custom Resource before update the status
            // so that we have the latest state of the resource on the cluster and we will avoid
            // raise the issue "the object has been modified, please apply
            // your changes to the latest version and try again" which would re-trigger the reconciliation
            if err := r.Get(ctx, req.NamespacedName, techwithhuz); err != nil {
                log.Error(err, "Failed to re-fetch techwithhuz")
                return ctrl.Result{}, err
            }

            meta.SetStatusCondition(&techwithhuz.Status.Conditions, metav1.Condition{Type: typeDegradedTechwithhuz,
                Status: metav1.ConditionTrue, Reason: "Finalizing",
                Message: fmt.Sprintf("Finalizer operations for custom resource %s name were successfully accomplished", techwithhuz.Name)})

            if err := r.Status().Update(ctx, techwithhuz); err != nil {
                log.Error(err, "Failed to update Techwithhuz status")
                return ctrl.Result{}, err
            }

            log.Info("Removing Finalizer for Techwithhuz after successfully perform the operations")
            if ok := controllerutil.RemoveFinalizer(techwithhuz, techwithhuzFinalizer); !ok {
                log.Error(err, "Failed to remove finalizer for techwithhuz")
                return ctrl.Result{Requeue: true}, nil
            }

            if err := r.Update(ctx, techwithhuz); err != nil {
                log.Error(err, "Failed to remove finalizer for techwithhuz")
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil
    }
    // Check if the deployment already exists, if not create a new one
    found := &appsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: techwithhuz.Name, Namespace: techwithhuz.Namespace}, found)
    if err != nil && apierrors.IsNotFound(err) {
        // Define a new deployment
        dep, err := r.deploymentForTechwithhuz(techwithhuz)
        if err != nil {
            log.Error(err, "Failed to define new Deployment resource for techwithhuz")

            // The following implementation will update the status
            meta.SetStatusCondition(&techwithhuz.Status.Conditions, metav1.Condition{Type: typeAvailableTechwithhuz,
                Status: metav1.ConditionFalse, Reason: "Reconciling",
                Message: fmt.Sprintf("Failed to create Deployment for the custom resource (%s): (%s)", techwithhuz.Name, err)})

            if err := r.Status().Update(ctx, techwithhuz); err != nil {
                log.Error(err, "Failed to update techwithhuz status")
                return ctrl.Result{}, err
            }

            return ctrl.Result{}, err
        }

        log.Info("Creating a new Deployment",
            "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
        if err = r.Create(ctx, dep); err != nil {
            log.Error(err, "Failed to create new Deployment",
                "Deployment.Namespace", dep.Namespace, "Deployment.Name", dep.Name)
            return ctrl.Result{}, err
        }

        // Deployment created successfully
        // We will requeue the reconciliation so that we can ensure the state
        // and move forward for the next operations
        return ctrl.Result{RequeueAfter: time.Minute}, nil
    } else if err != nil {
        log.Error(err, "Failed to get Deployment")
        // Let's return the error for the reconciliation be re-trigged again
        return ctrl.Result{}, err
    }

    // The CRD API is defining that the techwithhuz type, have a TechwithhuzSpec.Size field
    // to set the quantity of Deployment instances is the desired state on the cluster.
    // Therefore, the following code will ensure the Deployment size is the same as defined
    // via the Size spec of the Custom Resource which we are reconciling.
    size := techwithhuz.Spec.Size
    if *found.Spec.Replicas != size {
        found.Spec.Replicas = &size
        if err = r.Update(ctx, found); err != nil {
            log.Error(err, "Failed to update Deployment",
                "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)

            // Re-fetch the techwithhuz Custom Resource before update the status
            // so that we have the latest state of the resource on the cluster and we will avoid
            // raise the issue "the object has been modified, please apply
            // your changes to the latest version and try again" which would re-trigger the reconciliation
            if err := r.Get(ctx, req.NamespacedName, techwithhuz); err != nil {
                log.Error(err, "Failed to re-fetch techwithhuz")
                return ctrl.Result{}, err
            }

            // The following implementation will update the status
            meta.SetStatusCondition(&techwithhuz.Status.Conditions, metav1.Condition{Type: typeAvailableTechwithhuz,
                Status: metav1.ConditionFalse, Reason: "Resizing",
                Message: fmt.Sprintf("Failed to update the size for the custom resource (%s): (%s)", techwithhuz.Name, err)})

            if err := r.Status().Update(ctx, techwithhuz); err != nil {
                log.Error(err, "Failed to update techwithhuz status")
                return ctrl.Result{}, err
            }

            return ctrl.Result{}, err
        }

        // Now, that we update the size we want to requeue the reconciliation
        // so that we can ensure that we have the latest state of the resource before
        // update. Also, it will help ensure the desired state on the cluster
        return ctrl.Result{Requeue: true}, nil
    }

    // The following implementation will update the status
    meta.SetStatusCondition(&techwithhuz.Status.Conditions, metav1.Condition{Type: typeAvailableTechwithhuz,
        Status: metav1.ConditionTrue, Reason: "Reconciling",
        Message: fmt.Sprintf("Deployment for custom resource (%s) with %d replicas created successfully", techwithhuz.Name, size)})

    if err := r.Status().Update(ctx, techwithhuz); err != nil {
        log.Error(err, "Failed to update techwithhuz status")
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

Lastly, we have to update the SetupWithManager function and tell him to monitor our kind deployment.

// SetupWithManager sets up the controller with the Manager.
func (r *TechwithhuzReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1alpha1.Techwithhuz{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

Above controller/techwithhuz_controller.go file can be referred from here - https://github.com/techwithhuz/kubernetes-operator/blob/feature/techwithhuz-operator/techwithhuz-operator/controllers/techwithhuz_controller.go

Step 6: To run the operator

To run and test the operator code we need to first test it locally before creating the Docker image and deploying the operator in an environment.

To test the operator locally run this command

$ make install run
test -s /root/kubernetes-operator/techwithhuz-operator/bin/controller-gen && /root/kubernetes-operator/techwithhuz-operator/bin/controller-gen --version | grep -q v0.11.1 || \
GOBIN=/root/kubernetes-operator/techwithhuz-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.1
/root/kubernetes-operator/techwithhuz-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/kubernetes-operator/techwithhuz-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/techwithhuzs.cache.techwithhuz.com created
/root/kubernetes-operator/techwithhuz-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go: downloading github.com/onsi/ginkgo/v2 v2.6.0
go: downloading github.com/onsi/gomega v1.24.1

go run ./main.go
2023-08-16T12:41:36Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-08-16T12:41:36Z    INFO    setup   starting manager
2023-08-16T12:41:36Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-08-16T12:41:36Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-08-16T12:41:36Z    INFO    Starting EventSource    {"controller": "techwithhuz", "controllerGroup": "cache.techwithhuz.com", "controllerKind": "Techwithhuz", "source": "kind source: *v1alpha1.Techwithhuz"}
2023-08-16T12:41:36Z    INFO    Starting EventSource    {"controller": "techwithhuz", "controllerGroup": "cache.techwithhuz.com", "controllerKind": "Techwithhuz", "source": "kind source: *v1.Deployment"}
2023-08-16T12:41:36Z    INFO    Starting Controller     {"controller": "techwithhuz", "controllerGroup": "cache.techwithhuz.com", "controllerKind": "Techwithhuz"}
2023-08-16T12:41:36Z    INFO    Starting workers        {"controller": "techwithhuz", "controllerGroup": "cache.techwithhuz.com", "controllerKind": "Techwithhuz", "worker count": 1}

If it throws any error related to variables or functions then first fix those errors.

Successful logs will look like the above.

Step 7: Deploy the Custom Resource Definition(CRD) file

Now open another tab or take another ssh of your server where our operator is running and deploy the Custom Resource Definition file.

$ pwd   
/root/kubernetes-operator/techwithhuz-operator/config/crd/bases

$ kubectl apply -f cache.techwithhuz.com_techwithhuzs.yaml 
customresourcedefinition.apiextensions.k8s.io/techwithhuzs.cache.techwithhuz.com configured

$ kubectl get crds techwithhuzs.cache.techwithhuz.com 
NAME                                 CREATED AT
techwithhuzs.cache.techwithhuz.com   2023-08-16T12:39:16Z

Step 8: Deploy the Custom Resource (CR) file

Now the last step is to deploy the CR file of Kind Techwithhuz and see the magic.

$ pwd
/root/kubernetes-operator/techwithhuz-operator/config/samples

$ cat techwithhuz-cr.yaml 
apiVersion: cache.techwithhuz.com/v1alpha1 
kind: Techwithhuz
metadata:
  name: techwithhuz-sample
spec:
  size: 1
  containerPort: 8080

$ kubectl get techwithhuz
NAME                 AGE
techwithhuz-sample   3m22s

$ kubectl get pods
No resources found in default namespace.

$ kubectl apply -f techwithhuz-cr.yaml 
techwithhuz.cache.techwithhuz.com/techwithhuz-sample created

Go to the previous console where our operator code is already running

2023-08-16T12:47:39Z    INFO    Adding Finalizer for techwithhuz        {"controller": "techwithhuz", "controllerGroup": "cache.techwithhuz.com", "controllerKind": "Techwithhuz", "Techwithhuz": {"name":"techwithhuz-sample","namespace":"default"}, "namespace": "default", "name": "techwithhuz-sample", "reconcileID": "674549cf-7a73-437e-a6cd-fd4805db6651"}
2023-08-16T12:47:40Z    INFO    Creating a new Deployment       {"controller": "techwithhuz", "controllerGroup": "cache.techwithhuz.com", "controllerKind": "Techwithhuz", "Techwithhuz": {"name":"techwithhuz-sample","namespace":"default"}, "namespace": "default", "name": "techwithhuz-sample", "reconcileID": "674549cf-7a73-437e-a6cd-fd4805db6651", "Deployment.Namespace": "default", "Deployment.Name": "techwithhuz-sample"}

We can see in the logs that the operator has detected that a CR is applied with kind Techwithhuz with spec.Size as 1 pod and spec.ContainerPort as 8080. It will then run the reconcile loop and deploy our Deployment Go code and create a new deployment.

$ kubectl get deployments
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
techwithhuz-sample   0/1     1            0           70s

$ kubectl get pods
NAME                                  READY   STATUS              RESTARTS   AGE
techwithhuz-sample-6f6b66478b-jnbqg   0/1     ContainerCreating   0          85s

So now we have the deployment created and our application - techwithhuz is managed by our Go Lang operator.

Similar to how deployment kind is created, other Kubernetes objects like services, secrets, routes, configmaps, pvc etc. can also be created following the same logic.

This approach of running the operator is local but for production, a Docker Image of the Operator needs to be created. Dockerfile is already present in the repository which get generated by default when we run the operator sdk command.

That's it for this blog post. We can make the code more modular and generic so that we can deploy and manage multiple applications and make our work and life easier.

Lastly, I would like to say building the Golang operator is a little challenging and complicated but once you have it ready then it is very very easy to manage and also it is very stable for use in a production environment.

Feel free to put in the comment box if any queries.