Kubernetes operator for declaratively deploying wasmCloud applications (via wadm) and hosts on Kubernetes.

Overview

wasmcloud-operator

An operator for managing a set of wasmCloud hosts running on Kubernetes and manage wasmCloud applications using wadm. The goal is to easily be able to run WasmCloud hosts on a Kubernetes cluster.

WasmCloudHostConfig Custom Resource Definition (CRD)

The WasmCloudHostConfig CRD describes the desired state of a set of wasmCloud hosts connected to the same lattice.

apiVersion: k8s.wasmcloud.dev/v1alpha1
kind: WasmCloudHostConfig
metadata:
  name: my-wasmcloud-cluster
spec:
  # The number of wasmCloud host pods to run
  hostReplicas: 2
  # The cluster issuers to use for each host
  issuers:
    # This needs to be the public key of a cluster seed generated by
    # `wash keys gen cluster`
    - CDK...
  # The lattice to connect the hosts to
  lattice: 83a5b52e-17cf-4080-bac8-f844099f142e
  # Additional labels to apply to the host other than the defaults set in the operator
  hostLabels:
    some-label: value
  # Which wasmCloud version to use
  version: 0.82.0
  # The name of a secret in the same namespace that provides the required secrets.
  secretName: cluster-secrets
  # Enable the following to run the wasmCloud hosts as a DaemonSet
  #daemonset: true
  # The name of the image pull secret to use with wasmCloud hosts so that they
  # can authenticate to a private registry to pull components.
  # registryCredentialsSecret: my-registry-secret

The CRD requires a Kubernetes Secret with the following keys:

apiVersion: v1
kind: Secret
metadata:
  name: my-wasmcloud-cluster
data:
  # You can generate this with wash:
  # `wash keys gen cluster`
  WASMCLOUD_CLUSTER_SEED: <seed>
  # Only required if using a NATS creds file
  # nats.creds: <base64 encoded creds file>

The operator will fail to provision the wasmCloud Deployment if any of these secrets are missing!

Image Pull Secrets

You can also specify an image pull secret to use use with the wasmCloud hosts so that they can pull components from a private registry. This secret needs to be in the same namespace as the WasmCloudHostConfig CRD and must be a kubernetes.io/dockerconfigjson type secret. See the Kubernetes documentation for more information on how to provision that secret.

Once it is created, you can reference it in the WasmCloudHostConfig CRD by setting the registryCredentialsSecret field to the name of the secret.

Deploying the operator

A wasmCloud cluster requires a few things to run:

  • A NATS cluster with Jetstream enabled
  • WADM connected to the NATS cluster in order to support applications

If you are running locally, you can use the following commands to start a NATS cluster and WADM in your Kubernetes cluster.

Running NATS

Use the upstream NATS Helm chart to start a cluster with the following values.yaml file:

config:
  cluster:
    enabled: true
    replicas: 3
  leafnodes:
    enabled: true
  jetstream:
    enabled: true
    fileStore:
      pvc:
        size: 10Gi
    merge:
      domain: default
helm upgrade --install -f values.yaml nats-cluster nats/nats

Running Wadm

You can run Wadm in your Kubernetes cluster using our Helm chart. For a minimal deployment using the NATS server deployed above, all you need in your values.yaml file is:

wadm:
  config:
    nats:
      server: "nats-cluster.default.svc.cluster.local:4222"

You can deploy Wadm using your values file and Helm:

helm install wadm -f values.yaml --version 0.2.0 oci://ghcr.io/wasmcloud/charts/wadm

Start the operator

kubectl kustomize build deploy/local | kubectl apply -f -

Automatically Syncing Kubernetes Services

The operator automatically creates Kubernetes Services for wasmCloud applications. Right now this is limited only to applications that deploy the wasmCloud httpserver component using a daemonscaler, but additional support for spreadscalers will be added in the future.

If you specify host label selectors on the daemonscaler then the operator will honor those labels and will only create a service for the pods that match those label selectors.

Argo CD Health Check

Argo CD provides a way to define a custom health check that it then runs against a given resource to determine whether or not the resource is in healthy state.

For this purpose, we specifically expose a status.phase field, which exposes the underlying status information from wadm.

With the following ConfigMap, a custom health check can be added to an existing Argo CD installation for tracking the health of wadm applications.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations: |
    core.oam.dev/Application:
      health.lua: |
        hs = {}
        hs.status = "Progressing"
        hs.message = "Reconciling application state"
        if obj.status ~= nil and obj.status.phase ~= nil then
          if obj.status.phase == "Deployed" then
            hs.status = "Healthy"
            hs.message = "Application is ready"
          end
          if obj.status.phase == "Reconciling" then
            hs.status = "Progressing"
            hs.message = "Application has been deployed"
          end
          if obj.status.phase == "Failed" then
            hs.status = "Degraded"
            hs.message = "Application failed to deploy"
          end
          if obj.status.phase == "Undeployed" then
            hs.status = "Suspended"
            hs.message = "Application is undeployed"
          end
        end
        return hs

Testing

  • Make sure you have a Kubernetes cluster running locally. Some good options include Kind or Docker Desktop.
  • RUST_LOG=info cargo run

Types crate

This repo stores the types for any CRDs used by the operator in a separate crate (wasmcloud-operator-types) so that they can be reused in other projects.

Comments
  • feat: enable additional scheduling options for wasmCloud host pods

    feat: enable additional scheduling options for wasmCloud host pods

    Feature or Problem

    This refactors the WasmCloudHostConfig CRD so that it has a single field (schedulingOptions) for configuring how the underlying Pods are scheduled in Kubernetes. This includes:

    • Relocating the daemonset option to this new field
    • Relocating the resources option to this new field
    • Adding a new pod_template_additions field that allows you to set any valid option in a PodSpec

    Doing so allows cluster operators to do things like set node affinity and node selector rules, along with any other valid PodSpec option. The only thing that cannot be done is adding additional containers to the pod, since that is all handled by the controller. We could look at exposing that option if users want to be able to add additional sidecars.

    Related Issues

    Release Information

    next

    Consumer Impact

    Testing

    Tested locally in kind by setting daemonset: true and adding a node selector, both of which correctly changed the configuration of the underlying pods.

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by protochron 2
  • chore: update wadm and wash-lib to the latest released versions

    chore: update wadm and wash-lib to the latest released versions

    Feature or Problem

    Update to use the latest wasmCloud 1.0 compatible versions of wadm and wash-lib.

    Related Issues

    Release Information

    0.2.0

    Consumer Impact

    Testing

    Tested locally in kind and everything worked!

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by protochron 1
  • feat(*): Bumps to 0.2 for wasmcloud 1.0 support

    feat(*): Bumps to 0.2 for wasmcloud 1.0 support

    This bumps both the version of the operator and the default host version in the example to 1.0. Also updates the README to use the released wadm chart

    opened by thomastaylor312 1
  • feat: rework the operator for wasmCloud 1.0 compatibility

    feat: rework the operator for wasmCloud 1.0 compatibility

    Feature or Problem

    wasmCloud 1.0 fully converts over to using components among other large changes, so we need to rework the operator to account for it. 1.0 is still in the works, so this updates the operator to depend on wasmCloud 1.0 libraries along with refactoring the service creation code to use the updated application manifest format.

    Once this is in we will likely cut a 0.2.0 pre-release and wait for an actual release until wasmCloud 1.0.0 and wadm 0.11 are out.

    This also adds a new field to the CRD so that you can specify the full image reference instead of just the version. The image parameter takes precedence.

    Related Issues

    Release Information

    0.2.0

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by protochron 1
  • feat: automatically create Kubernetes Services for apps using a httpserver component

    feat: automatically create Kubernetes Services for apps using a httpserver component

    Feature or Problem

    One advantage of wasmCloud applications is that we always know what components make up an application and therefore can intelligently make decisions based on what interfaces we're using. This makes it much easier for software like the operator to take particular actions based on the contents of an application manifest.

    This change adds an additional set of reconciliations we can perform on a per-lattice basis to automatically create Kubernetes Services for applications that deploy a httpserver component. The operator uses a NATS consumer that watches for manifest deploy and undeploy events and triggers a reconciliation on a managed Service. Right now this is restricted only to daemonscalers, since we do not have to do any bookeeping on where components are deployed if they are running on all of the hosts in a lattice or in a subset identified by label. We will add support for spreadscalers in a future PR.

    This allows for some interesting deployment scenarios, such as wasmCloud applications that span multiple Kubernetes deployments in the same namespace, or potentially in multiple namespaces if we decide to implement support for them. Initially the operator is only creating endpoints that route traffic from a service to pods in the same namespace, but we may add an option to relax that assumption.

    Related Issues

    Release Information

    0.2

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by protochron 1
  • chore(deps): Bump h2 from 0.3.25 to 0.3.26

    chore(deps): Bump h2 from 0.3.25 to 0.3.26

    Bumps h2 from 0.3.25 to 0.3.26.

    Release notes

    Sourced from h2's releases.

    v0.3.26

    What's Changed

    • Limit number of CONTINUATION frames for misbehaving connections.

    See https://seanmonstar.com/blog/hyper-http2-continuation-flood/ for more info.

    Changelog

    Sourced from h2's changelog.

    0.3.26 (April 3, 2024)

    • Limit number of CONTINUATION frames for misbehaving connections.
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies 
    opened by dependabot[bot] 1
  • Use the full chart path for helm package

    Use the full chart path for helm package

    Feature or Problem

    Currently we're not using the full path, which makes helm package fail

    Related Issues

    Release Information

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by joonas 0
  • Add Pods and EndpointSlices permissions to wasmcloud-operator ClusterRole

    Add Pods and EndpointSlices permissions to wasmcloud-operator ClusterRole

    Feature or Problem

    Brings permissions changes from #14 to the chart

    Related Issues

    Release Information

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by joonas 0
  • Add conditions to Endpoints part of the EndpointSlices reflecting wadm application Service entries

    Add conditions to Endpoints part of the EndpointSlices reflecting wadm application Service entries

    Feature or Problem

    While working on something somewhat related, I noticed that third-party software that was trying to connect to the Services we create for wasmCloud applications could not do so.

    After some spelunking, it turned out that the software in question was looking for the Ready condition of the Endpoints section of a given EndpointSlice, which was previously missing.

    This fixes that.

    Related Issues

    Release Information

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by joonas 0
  • Fix up some README typos and add relevant links

    Fix up some README typos and add relevant links

    Feature or Problem

    Let's help the potential reader with some links in case the Operator README is their first contact with wasmCloud.

    Related Issues

    Release Information

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by joonas 0
  • chore: Update charts to be published under separate namespace

    chore: Update charts to be published under separate namespace

    Feature or Problem

    To avoid conflicting names with the containers and potential confusion, this moves charts to be published under a separate namespace (charts/<chart-name>).

    Related Issues

    Release Information

    Consumer Impact

    Testing

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by joonas 0
  • feat: add additional host configuration options for images, policy

    feat: add additional host configuration options for images, policy

    Feature or Problem

    Add additional host options for the policy service, allowing insecure registries and pulling the latest tag of an OCI ref.

    Related Issues

    Release Information

    next

    Consumer Impact

    Testing

    Tested locally in a kind cluster

    Unit Test(s)

    Acceptance or Integration

    Manual Verification

    opened by protochron 1
  • Expose information from the downward API to wasmCloud hosts

    Expose information from the downward API to wasmCloud hosts

    Right now it's impossible to expose information from the downward API either as a volume or as environment variables since we don't expose either option in the CRD. Technically we do expose the ability to define volumes, but you have no way to provide a volume mapping. We can fix this in a few ways:

    1. add a new option to the CRD (eg. downwardAPI: true) to the CRD to tell the controller to load downward API information automatically. The controller would load all available values as either env vars or through the volume mount depending on how the data is exposed
    2. add a env config option so that users can add their own environment variables to hosts. The controller would need to add those first and merge in the set of environment variables it generates for the pod template
    3. add a volumeMounts configuration option for allowing users to pass in volume mount mappings corresponding to a downward API volume. This would likely end up simply being the ability to define arbitrary volume mappings, one of which could just be a downward API volume.

    Options 2 and 3 seem like things we should do anyway, but option 1 would provide us the most control over what we expose without needing to implement environment variable and volume support. However it would require the maintainers to add all available information from the downward API and keep that up to date.

    opened by protochron 0
  • Pod template additions shouldn't be required

    Pod template additions shouldn't be required

    In the latest 0.2.0-alpha.1, if you set schedulingOptions, you get this error:

    The WasmCloudHostConfig "my-wasmcloud-cluster" is invalid: spec.schedulingOptions.pod_template_additions: Required value
    

    Based on our code, this field should be optional so we should update the spec

    bug 
    opened by thomastaylor312 0
  • `WasmCloudHostConfig` should have a status that indicates if it can't scale all the way

    `WasmCloudHostConfig` should have a status that indicates if it can't scale all the way

    I was running in GKE in autopilot mode and only 1 node had spun up. Even though I had requested 2 replicas in daemonset mode, it only gave me one (which makes sense). However, we should probably have some sort of indicator for when nodes are full or otherwise unavailable that the number of hosts desired is not possible.

    This is really low priority, was just pointing it out since I discovered it

    enhancement 
    opened by thomastaylor312 0
  • The operator should support using a different set of NATS credentials for communicating with wadm

    The operator should support using a different set of NATS credentials for communicating with wadm

    Add a configuration option so that we can load a NATS credentials file stored as a secret and use it to configure the client we use to communicate with the NATS cluster. These credentials should only be used to configure the operator's consumer and the stream that we create when bootstrapping. If a WasmCloudHostConfig references a secret that has NATS credentials then the operator should use those credentials when creating a client to manage wasmCloud applications. Otherwise it should fall back to either its own credentials or using unauthenticated calls.

    opened by protochron 0
Releases(v0.2.0)
  • v0.2.0(Apr 17, 2024)

    Breaking changes

    This version is now only compatible with wasmCloud 1.0+. If you need to run workloads targeting 0.82 or below then use v0.1.1 of the operator for now until you are able to migrate.

    The WasmCloudHostConfig CRD in this release moves the daemonset option down a level nested under the schedulingOptions field. If you have have been using this feature then you will want to either re-deploy your cluster or temporarily revert to using a Deployment by removing the value and then setting it after you upgrade the operator.

    What's Changed

    • fix: Address GHSA-g98v-hv3f-hcfr by @joonas in https://github.com/wasmCloud/wasmcloud-operator/pull/4
    • chore(deps): Bump unsafe-libyaml from 0.2.9 to 0.2.11 by @dependabot in https://github.com/wasmCloud/wasmcloud-operator/pull/5
    • chore(deps): Bump h2 from 0.3.21 to 0.3.25 by @dependabot in https://github.com/wasmCloud/wasmcloud-operator/pull/6
    • chore: Add dependency-review workflow for pull requests by @joonas in https://github.com/wasmCloud/wasmcloud-operator/pull/7
    • feat: Add helm chart by @joonas in https://github.com/wasmCloud/wasmcloud-operator/pull/8
    • feat: enable additional scheduling options for wasmCloud host pods by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/9
    • chore: Update charts to be published under separate namespace by @joonas in https://github.com/wasmCloud/wasmcloud-operator/pull/10
    • chore(deps): Bump h2 from 0.3.25 to 0.3.26 by @dependabot in https://github.com/wasmCloud/wasmcloud-operator/pull/13
    • feat: automatically create Kubernetes Services for apps using a httpserver component by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/14
    • feat: rework the operator for wasmCloud 1.0 compatibility by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/18
    • ref(*): Refactors some copied code and bumps version by @thomastaylor312 in https://github.com/wasmCloud/wasmcloud-operator/pull/21
    • Add wadm instructions to readme by @ericgregory in https://github.com/wasmCloud/wasmcloud-operator/pull/20
    • Fix up some README typos and add relevant links by @joonas in https://github.com/wasmCloud/wasmcloud-operator/pull/19
    • feat(*): Bumps to 0.2 for wasmcloud 1.0 support by @thomastaylor312 in https://github.com/wasmCloud/wasmcloud-operator/pull/25
    • chore: update wadm and wash-lib to the latest released versions by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/26

    New Contributors

    • @joonas made their first contribution in https://github.com/wasmCloud/wasmcloud-operator/pull/4
    • @thomastaylor312 made their first contribution in https://github.com/wasmCloud/wasmcloud-operator/pull/21
    • @ericgregory made their first contribution in https://github.com/wasmCloud/wasmcloud-operator/pull/20

    Full Changelog: https://github.com/wasmCloud/wasmcloud-operator/compare/v0.1.1...v0.2.0

    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Mar 28, 2024)

    What's Changed

    • feat: add option to start hosts as a DaemonSet by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/1
    • chore: add docs on how to use the registryCredentialsSecret option by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/2
    • fix: use the correct OCI registry environment variables for newer hosts by @protochron in https://github.com/wasmCloud/wasmcloud-operator/pull/3

    Full Changelog: https://github.com/wasmCloud/wasmcloud-operator/compare/v0.1.0...v0.1.1

    Source code(tar.gz)
    Source code(zip)
Owner
wasmcloud
Bringing joy to the act of building distributed applications with low boilerplate, secure by default WebAssembly components. CNCF sandbox application runtime.
wasmcloud
⏱ Kubernetes operator that allows to set maximum lifetime for pods

Pod Lifetime Limiter Hi! ?? So you deal with a crappy application which stops working after some period of time and you want to restart it every N hou

Viktor 27 Sep 8, 2022
Rust based Kubernetes Operator to deploy K8s objects minimally.

kube-nimble nimble /ˈnɪmbl/ - quick and light in movement or action; agile. This project began from a place of curiosity about Kubernetes CRDs and the

Meet Vasani 3 Feb 26, 2024
Warp is a blazingly fast, Rust-based terminal that makes you and your team more productive at running, debugging, and deploying code and infrastructure.

Warp is a blazingly fast, Rust-based terminal that makes you and your team more productive at running, debugging, and deploying code and infrastructure.

Warp 10.4k Jan 4, 2023
1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel

massh 1 library and 2 binary crates to run SSH/SCP commands on a "mass" of hosts in parallel. The binary crates are CLI and GUI "frontends" for the li

null 2 Oct 16, 2022
Tool for mass import of hosts into Zabbix (and other API functions)

zabbix-tools A CLI tool for interacting with Zabbix API built in Rust. Designed for Zabbix 6.0. Functions added to test API and add hosts manually or

null 1 Apr 21, 2022
🐙 Loads config and hosts for gh CLI in Rust.

gh-config-rs Loads config and hosts for gh CLI in Rust. Getting started [dependencies] gh-config = "0.2" Usage use std::error::Error; use gh_config::*

Naoki Ikeguchi 2 Jul 23, 2022
locdev is a handy CLI tool that simplifies the process of adding, removing, and listing entries in the hosts file.

locdev ??️ locdev is a handy CLI tool that simplifies the process of adding, removing, and listing entries in the hosts file. You no longer need to de

Nick Rempel 20 Jun 5, 2023
rewrite of hosts-creator in rust

hc-rs -> hosts-creator-rust fetch and merge multiple hosts files this is a WIP whats done fetching hosts files merging hosts files removing duplicate

null 2 Nov 20, 2022
IntMaxRollup operator node & cli tools.

Intmax Rollup Operator Int max operator node Prepara Install rustup. rustup override set nightly

null 5 Jul 26, 2022
Expand your possibilities with the Try ? Operator

Expand your possibilities with the Try ? Operator Have you ever found yourself writing a function which may return early based on some condition? fn m

EC 1 Feb 1, 2022
Set of tools that make it easier for the operator to manage a TAPLE network.

⚠️ TAPLE is in early development and should not be used in production ⚠️ TAPLE Tools TAPLE (pronounced T+ ?? ['tapəl]) stands for Tracking (Autonomous

Open Canarias 5 Jan 25, 2023
A simple, C-like, ternary operator for cleaner syntax.

A simple ternary operator macro in rust. the iff! macro is the only item exported by this crate, it simply takes three expressions, seperated by ? and

KaitlynEthylia 42 May 23, 2023
CLI utility that screencaptures your Linux desktop and streams it to Kodi via UPNP/DLNA and RTSP

desktopcast Desktopcast is a little CLI application that allows you to cast your Linux desktop to any UPNP/DLNA device capable of the AVTransfer servi

Markus Ebner 25 Apr 16, 2023
Cli tool for git productivity written in Rust and packaged for consumption via NPM

crust ?? cli tool for git productivity written in Rust and packaged for consumption via NPM This repo is identical with @skyneticist/golee in terms of

null 2 Jul 30, 2022
A library and binary for testing unhooking ntdll by identifying hooks via in-memory disassembly

(First Public?) Sample of unhooking ntdll (All Exports & IAT imports) hooks in Rust using in-memory disassembly, avoiding direct syscalls and all hooked functions (incl. hooked NtProtectVirtualMemory)

Signal Labs 52 Apr 9, 2023
Sets of libraries and tools to write applications and libraries mixing OCaml and Rust

Sets of libraries and tools to write applications and libraries mixing OCaml and Rust. These libraries will help keeping your types and data structures synchronized, and enable seamless exchange between OCaml and Rust

Meta 36 Jan 28, 2023
Rust command line utility to quickly display useful secrets in a Kubernetes namespace

kube-secrets This is a command line utility for quickly looking at secrets in a Kubernetes namespace that are typically looked at by humans. It specif

Frank Wiles 8 Feb 10, 2022
Terminal text styling via ANSI escape sequences.

Iridescent Features iridescent is a library for styling terminal text easily. It supports basic ANSI sequences, Xterm-256 colors, and RGB. You can ope

Rob 2 Oct 20, 2022
🔔 CLI utility to send notifications to Slack via integration webhooks

Slack notifier Just a simple CLI tool to send notifications to Slack. Please note that this project is just a playground to start learning Rust, it is

Green.Mod 2 May 21, 2022