Build, Test, and Automate a Kubernetes Interfacing Application in Go | by Fernando Diaz | Apr, 2022

Photograph by Clovis Wood Photography on Unsplash

Kubernetes’ Client-Go supplies all kinds of packages for interacting along with your cluster. These packages embody the next:

  • The kubernetes bundle incorporates the clientset to entry Kubernetes API.
  • The discovery bundle is used to find APIs supported by a Kubernetes API server.
  • The dynamic bundle incorporates a dynamic shopper that may carry out generic operations on arbitrary Kubernetes API objects.
  • The plugin/pkg/shopper/auth packages comprise optionally available authentication plugins for acquiring credentials from exterior sources.
  • The transport bundle is used to arrange auth and begin a connection.
  • The instruments/cache bundle is beneficial for writing controllers.

Together with all of the above packages, Consumer-Go additionally incorporates a pretend shopper, which lets you mockup the creation, studying, modifying, and removing of a selected Kubernetes useful resource with the intention to simply enhance unit-test protection.

Again in KubeCon Europe 2019, I introduced on performing unit checks utilizing the Go-Consumer pretend shopper:

At this time, I’d wish to go over making a easy out-of-cluster software used to carry out actions in your Kubernetes cluster. Then I’ll present you the way to mock out the Kubernetes API calls in your unit checks, and automate operating these checks and extra with GitLab!

So as to get began, there are a couple of packages you need to have put in:

  • Go— an open supply programming language.
  • MiniKube— a software that shortly units up an area Kubernetes cluster on macOS, Linux, and Home windows. Additionally, you will want a virtualization driver for MiniKube to run akin to Docker, HyperV, Podman, and many others. Extra info could be discovered here.

Be aware: You should utilize one other cluster apart from minikube, and it’ll work simply effective based mostly on the kubectl context which is ready. This information is written utilizing minikube, as a result of it’s accessible to everybody.

You will need to even have information of the next:

  • Go Basics — The code I’ve written and can go over is in Go. Ensure you have a fundamental understanding.
  • Kubernetes Basics — Perceive the way to work with Kubernetes clusters, and utilizing kubectl.
  • Kubernetes Secrets — Perceive what secrets and techniques are in Kubernetes.

It’s good to know the next:

  • Git Basics — Good to know Git to create forks and make modifications to the code your self.
  • GitLab CI— My code is housed in GitLab, and I’m utilizing the GitLab CI with the intention to run my unit checks and safety scans.

I will likely be utilizing my secreto-server undertaking which performs the next features:

  • Generates generic Kubernetes secrets and techniques
  • Lists the secrets and techniques per a given namespace
  • Obtains secret knowledge which incorporates the payload
  • Deletes secrets and techniques

To begin off, we’ll create a cluster utilizing MiniKube and ensure we are able to work together with Kubernetes secrets and techniques. Then we’ll launch the applying domestically and confirm all the applying’s features.

Creating Kubernetes Cluster

  1. Set up MiniKube.You’ll have the ability to obtain a model of Minikube based mostly in your OS and Structure on the Getting Started Page.

2. Run Minikube. It might take a couple of minutes to obtain the required packages, however operating Minikube is easy so long as you have got the required virtualization drivers. Docker Desktop works for many, however I’m utilizing podman to attempt one thing completely different. See the Minikube documentation for extra info on drivers.

$ minikube beginminikube v1.25.2 on Darwin 12.3 (arm64)
Utilizing the podman (experimental) driver based mostly on present profile
Beginning management airplane node minikube in cluster minikube
Pulling base picture ...
E0321 11:05:07.616563 66007 cache.go:203] Error downloading kic artifacts: not but carried out, see problem #8426
Restarting present podman container for "minikube" ...
Making ready Kubernetes v1.23.3 on Docker 20.10.12 ...E0321 11:05:13.251398 66007 begin.go:126] Unable to get host IP: RoutableHostIPFromInside is at the moment solely carried out for linux
▪ kubelet.housekeeping-interval=5m
Verifying Kubernetes parts...
▪ Utilizing picture gcr.io/k8s-minikube/storage-provisioner:v5
Enabled addons: storage-provisioner, default-storageclass
kubectl not discovered. In case you want it, attempt: 'minikube kubectl -- get pods -A'
Accomplished! kubectl is now configured to make use of "minikube" cluster and "default" namespace by default

3. Confirm that Minikube is operating appropriately. We will do that by checking if the Minikube node has a STATUS of prepared.

$ minikube kubectl get nodesNAME       STATUS   ROLES                  AGE     VERSION
minikube Prepared control-plane,grasp 3m50s v1.23.3

4. Create a secret. We are going to do that by means of kubectl simply to confirm that we’ve entry to the secrets and techniques API. I’m making a generic secret with the literal[shhh]=supersecret .

$ minikube kubectl create secret generic my-secret -- --from-literal=shhh=supersecretsecret/my-secret created

5. Now, let’s confirm that the key was created efficiently

We will seize it in YAML mode and see the base64 encoded supersecret (c3VwZXJzZWNyZXQ=).

$ minikube kubectl get secrets and techniques my-secret -- -o yamlminikube kubectl get secrets and techniques my-secret -- -o yaml
apiVersion: v1
knowledge:
shhh: c3VwZXJzZWNyZXQ=
form: Secret
metadata:
creationTimestamp: "2022-03-20T21:16:48Z"
identify: my-secret
namespace: default
resourceVersion: "728"
uid: 9fcb7814-77f1-44dc-b476-066db11598bd
sort: Opaque
  1. Clone the applying to your GOPATH
$ git clone git@gitlab.com:k2511/secreto-server.gitgit clone git@gitlab.com:k2511/secreto-server.git
Cloning into 'secreto-server'...
distant: Enumerating objects: 235, finished.
distant: Counting objects: 100% (232/232), finished.
distant: Compressing objects: 100% (121/121), finished.
distant: Complete 235 (delta 97), reused 177 (delta 69), pack-reused 3
Receiving objects: 100% (235/235), 282.99 KiB | 3.11 MiB/s, finished.
Resolving deltas: 100% (97/97), finished.
$ cd secreto-server

2. Construct the applying executable. I created a Makefile which makes it simple. As soon as this command is run, there ought to be a brand new executable created named secreto-server.

$ make constructgo mod obtain
GOOS=darwin GOARCH=arm64 go construct -o secreto-server .
chmod +x secreto-server

Be aware: You could want to alter $GOOS and $GOARCH variables within the Makefile in the event you aren’t operating on an M1 Mac. Extra particulars here.

3. Run the applying domestically. That is finished by simply passing the -local flag when operating the executable. Operating it with out the -local flag, would require the applying to be operating within the Kubernetes cluster as a result of it makes use of a distinct auth method.

$ ./secreto-server -local2022/03/20 16:18:30 KubeClient operating with native configuration
2022/03/20 16:18:30 Beginning server on the port 8080

It’s also possible to change the port by setting the SECRETO_PORT setting variable earlier than executing this system.

Now the applying is operating. Let’s go forward and ensure it’s working. We will do that by opening up one other terminal and sending a request to the server to acquire its model.

$ curl http://localhost:8080/api/secreto/model"model":1

We now have a working software! Let’s confirm a number of of the applying’s features.

Including a secret

We will add a secret by passing a identify and payload to the secreto API path. Ensure you additionally add the namespace to the top of the trail as seen beneath.

$ curl -X POST http://localhost:8080/api/secreto/default -d '"identify": "my-secret2", "payload": "my-secret-yoo"'"Secret Created Efficiently":"identify":"my-secret2","namespace":"default","date":"2022-03-20 17:39:41 -0500 CDT","payload":"my-secret-yoo"

Viewing secrets and techniques

We will checklist all of our secrets and techniques, kind by namespace, and even lookup the payload. That is finished by performing a GET on completely different paths:

  • /api/secreto: lists all secrets and techniques
  • /api/secreto/namespace: lists all secrets and techniques in namespace
  • /api/secreto/namespace/identify: lists knowledge for secret identify in namespace
$ curl -X GET http://localhost:8080/api/secreto["name":"default-token-m9wsq","namespace":"default","date":"2022-03-20 16:10:27 -0500 CDT","payload":"","name":"my-secret","namespace":"default","date":"2022-03-20 16:16:48 -0500 CDT","payload":"supersecret","name":"yeet","namespace":"default","date":"2022-03-20 16:56:24 -0500 CDT","payload":"my-secret-yoo","name":"default-token-dq5wr","namespace":"kube-node-lease","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"default-token-nwbxx","namespace":"kube-public","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"attachdetach-controller-token-cdfl4","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"bootstrap-signer-token-ljx9n","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"bootstrap-token-81dbvo","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"certificate-controller-token-9nqdf","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":"","name":"clusterrole-aggregation-controller-token-wb95r","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"coredns-token-7sldt","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"cronjob-controller-token-l5msx","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":"","name":"daemon-set-controller-token-ppr8p","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":"","name":"default-token-mxzhs","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"deployment-controller-token-ctsrt","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"disruption-controller-token-kf9qs","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"endpoint-controller-token-kkp5b","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"endpointslice-controller-token-b4bwk","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"endpointslicemirroring-controller-token-g7bqq","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":"","name":"ephemeral-volume-controller-token-t7s6h","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"expand-controller-token-wvhn8","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"generic-garbage-collector-token-q62cw","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"horizontal-pod-autoscaler-token-wmkcc","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"job-controller-token-9492p","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"kube-proxy-token-6z9ht","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"namespace-controller-token-lrwx8","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":"","name":"node-controller-token-x9vwn","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":"","name":"persistent-volume-binder-token-vdw68","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"pod-garbage-collector-token-jl9z2","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":"","name":"pv-protection-controller-token-jv9d8","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"pvc-protection-controller-token-d4ccm","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"replicaset-controller-token-hbdj6","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":"","name":"replication-controller-token-74kl8","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"resourcequota-controller-token-767r2","namespace":"kube-system","date":"2022-03-20 16:10:13 -0500 CDT","payload":"","name":"root-ca-cert-publisher-token-7zbhn","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":"","name":"service-account-controller-token-vdxgt","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"service-controller-token-nvt8n","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":"","name":"statefulset-controller-token-97d8r","namespace":"kube-system","date":"2022-03-20 16:10:26 -0500 CDT","payload":"","name":"storage-provisioner-token-nsblb","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":"","name":"token-cleaner-token-wdbdn","namespace":"kube-system","date":"2022-03-20 16:10:15 -0500 CDT","payload":"","name":"ttl-after-finished-controller-token-rgjt4","namespace":"kube-system","date":"2022-03-20 16:10:16 -0500 CDT","payload":"","name":"ttl-controller-token-tzjfc","namespace":"kube-system","date":"2022-03-20 16:10:14 -0500 CDT","payload":""]$ curl -X GET http://localhost:8080/api/secreto/default["name":"default-token-m9wsq","namespace":"default","date":"2022-03-20 16:10:27 -0500 CDT","payload":"","name":"my-secret","namespace":"default","date":"2022-03-20 16:16:48 -0500 CDT","payload":"supersecret","name":"my-secret2","namespace":"default","date":"2022-03-20 16:56:24 -0500 CDT","payload":"my-secret-yoo"]$ curl -X GET http://localhost:8080/api/secreto/default/my-secret"identify":"my-secret","namespace":"default","date":"2022-03-20 16:16:48 -0500 CDT","payload":"supersecret"

Deleting secrets and techniques

Now, let’s go forward and delete a secret we created earlier. That is finished by performing a DELETE on the complete path of a secret.

$ curl -X DELETE http://localhost:8080/api/secreto/default/my-secret"Secret Deleted Efficiently":"my-secret"

Now let’s dive into the application code. I’m primarily going to go over the components which work together with the Kubernetes API. The appliance is break up up within the following approach:

  • middleware: Incorporates all the applying logic for processing a request and producing a response. This consists of speaking with the Kubernetes API with the intention to carry out completely different features on secrets and techniques.
  • router: Routes calls from specific URIs to the applying logic within the middleware.
  • mannequin: Incorporates structs which are used within the software to show and create secrets and techniques.
  • essential.go: Begins the applying, loading the web-server.

Now let’s take a dive into how the supply code works.

Authentication and setup

Authenticating with a Kubernetes shopper could be seen in middleware.go. On this article, we will likely be going over out-of-cluster authentication.

This code was taken from the out-of-cluster config example of Consumer-Go. The principle components to note are:

  • The Kubernetes config is ready by what’s energetic within the ~/.kube/config folder
  • If no house listing exists, then we should cross -kubeconfig together with the trail of our Kubernetes configuration to ensure that it to load correctly
  • There’s a international variable ClientSet=shopper set in order that we don’t must hold loading the kubeconfig and might simply run instructions with the ClientSet

Including a secret

So as to add a secret, the code beneath accepts a request, calls the personal createSecret operate, after which returns the response to the consumer. The principle components to note are:

  • Units headers for CORS, for instance, Entry-Management-Enable-Strategies which can enable browsers to make use of several types of strategies
  • Grabs parameters for the namespace from the URI route namespace outlined in router.go
  • Generates a Kubernetes API name based mostly on the gadgets within the request physique
  • Encodes both the precise secret and returns it, or returns an error

The code beneath is a non-public operate that makes use of Consumer-Go with the intention to create a secret. The principle components to note are:

  • metav1.ObjectMeta is setup together with the key payload which is encapsulated within the maps secretData and secretDataBytes
  • A Kubernetes v1.Secret based mostly on the info despatched within the operate is handed to the Kubernetes API for the creation of the key
  • If the technology of the key is profitable then we generate a Secreto object and fill it with knowledge from the key earlier than returning it

Viewing a secret

Now we have a number of features used to view secrets and techniques. They seize Kubernetes’ secrets and techniques and their information utilizing Consumer-Go. I’m simply going to go over acquiring the key particulars since most features are comparable. The principle components to note are:

  • Units headers for CORS, for instance, Entry-Management-Enable-Strategies which can enable browsers to make use of several types of strategies
  • Grabs parameters for the namespace and identify from the URI route namespace/identify outlined in router.go, and generates a name to the Kubernetes API
  • Encodes both the precise secret and returns it, or returns an error

The code beneath is a non-public operate that makes use of Consumer-Go with the intention to describe a secret acquiring its particulars with Consumer-Go.

func getSecretDetails(namespace string, identify string) (*v1.Secret, error) 
secret, err := ClientSet.CoreV1().Secrets and techniques(namespace).Get(context.TODO(), identify, metav1.GetOptions)

if err != nil
return nil, err

return secret, nil

Deleting a secret

Deleting a secret is fairly simple. The principle components to note are:

  • Units headers for CORS, for instance, Entry-Management-Enable-Strategies which can enable browsers to make use of several types of strategies
  • Grabs parameters for the namespace and identify from the URI route namespace/identify outlined in router.go, and generates a name to the Kubernetes API
  • Returns a message saying that the key was efficiently deleted, or returns an error

The code beneath is a non-public operate that makes use of Consumer-Go with the intention to delete a secret:

func deleteSecret(identify string, namespace string) error 
err := ClientSet.CoreV1().Secrets and techniques(namespace).Delete(context.TODO(), identify, metav1.DeleteOptions)
if err != nil
return err

return nil

Now that we’ve seen all of the completely different features throughout the software, we’re gonna go forward and write some unit checks. Unit checks are particular person checks on completely different components of our software, that are necessary for verifying our logic and ensuring our software is doing what it ought to.

All of the unit checks I’ve written are situated in middleware_test.go.

Normal setup

I created a operate that simply units up take a look at values for the completely different unit checks. You possibly can see the setupSecrets() operate which can generate completely different secrets and techniques in addition to anticipated values to search for.

Mocking Consumer-Go

ClientSet is a world variable outlined in middleware.go. Inside the checks, we overwrite it, permitting all of the requests to return pretend values with out speaking to our Kubernetes cluster. The pretend shopper makes requests returning mock objects and values.

ClientSet = pretend.NewSimpleClientset()

Mocking requests

Requests could be mocked utilizing httptest, which supplies utilities for HTTP testing and permits us to “file” requests. A number of issues to note are:

  • A request is created and take a look at variables are handed within the requestBody
  • The CreateSecret operate is known as with the pretend w http.ResponseWriter, r *http.Request that we generated

Operating checks

Now that we’ve gone over the checks, we are able to go forward and run them. This may be finished by operating the next command:

$ make take a look atgo take a look at -v ./...
? gitlab.com/k2511/kube-secreto [no test files]
=== RUN TestProcessSecrets
--- PASS: TestProcessSecrets (0.00s)
=== RUN TestGetSecretsClient
--- PASS: TestGetSecretsClient (0.00s)
=== RUN TestGetSecretDetailsClient
--- PASS: TestGetSecretDetailsClient (0.00s)
=== RUN TestCreateSecretClient
--- PASS: TestCreateSecretClient (0.00s)
=== RUN TestDeleteSecretClient
--- PASS: TestDeleteSecretClient (0.00s)
=== RUN TestGetSecretsByNamespace
--- PASS: TestGetSecretsByNamespace (0.00s)
=== RUN TestGetSecretDetails
--- PASS: TestGetSecretDetails (0.00s)
=== RUN TestCreateSecret
--- PASS: TestCreateSecret (0.00s)
=== RUN TestDeleteSecret
--- PASS: TestDeleteSecret (0.00s)
=== RUN TestGetVersion
--- PASS: TestGetVersion (0.00s)
PASS
okay gitlab.com/k2511/kube-secreto/inside/server/middleware 1.406s
? gitlab.com/k2511/kube-secreto/inside/server/fashions [no test files]
? gitlab.com/k2511/kube-secreto/inside/server/router [no test files]

I’m utilizing GitLab for CI to automate constructing, testing, and pushing the applying container to my registry. This makes it in order that I don’t need to manually carry out these features every time I push code.

I can configure GitLab in order that my software is robotically examined so I can confirm my software logic. My software can be constructed, containerized, and pushed to my container registry. I can even examine if my supply code is safe utilizing GitLab SAST.

My GitLab Yaml seems as follows:

Now, if we take a look at the most recent operating GitLab Pipeline, we are able to see the next:

  • Construct stage: runs construct which builds the applying and docker-build which builds the applying container and pushes it to my container registry
  • Check stage: unit runs unit checks and generates a protection report, gosec-sast and semgrep-sast makes use of gosec and semgrep respectively to scan the applying supply code for vulnerabilities

When clicking on the Safety tab, we are able to see a couple of vulnerabilities we should always resolve, sorted by severity.

Clicking on one supplies an outline, location, CVE, and resolution. It’s also possible to dismiss the vulnerability or create a confidential problem to work on remediation with others with out alerting these with out permission.

There’s additionally extra security scanners you possibly can look into in addition to different cool CICD instruments at GitLab.

More Posts