Feat: Pluggable Event Watcher System

[antweiss]

Pluggable Event Watcher System

The Pluggable Event Watcher System extends the KHook Controller to support multiple event sources beyond Kubernetes events. The idea is to use Go's plugin system to enable dynamic loading of event source implementations at startup, allowing for extensibility and customization without modifying the core controller code.

Architecture Overview

graph TB
    A[Plugin Manager] --> B[Event Source Registry]
    A --> C[Event Mapping Loader]
    B --> D[Kubernetes Event Source]
    B --> E[Custom Event Source Plugin]
    B --> F[External API Event Source]
    C --> G[Event Mappings Config]
    D --> H[Event Router]
    E --> H
    F --> H
    H --> I[Event Processing Pipeline]

Loading

Core Components

1. Plugin Manager: Loads and manages event source plugins at startup
2. Event Source Registry: Maintains registry of available event sources
3. Event Mapping Loader: Loads event type mappings from configuration files
4. Event Router: Routes events from multiple sources to processing pipeline
5. Event Source Plugins: Individual implementations for different event sources

The existing Kubernetes event watcher will be rewritten as a plugin and enabled by default. Additional event source plugins will be optional.

[antweiss]

@EItanya this is the initial feature definition for pluggable event sources. I've been working on this since last week, hope to have a working prototype on Sept 25.

[EItanya]

Before commenting on this exact design I do potentially want to take a tiny step back and ask a question. What is khook really? The part of khook that is unique is the k8s API, and the integration with kagent. Event management is a complex beast and I'm a little worried we may be rewriting the wheel here. Before we get too deep on designing our own system, is there prior art we can use here?

The idea is to use Go's plugin system

We should avoid this if possible. I've worked with it before and it's a NIGHTMARE. I understand the impetus but we should try and figure out another way if possible.

[antweiss]

Thanks for this comment. Yeah, it's possible I'm a little carried away with the vision of sending any possible event to any appropriate agent. Let's discuss the options. I know go plugins have issues but still thought I'd see for myself. 😉 But I trust your experience.

[EItanya]

@antweiss and I had a meeting offline this morning to discuss this question, and it got us digging into the nature of khook and how we may want the architecture to look for the long-term. I made this very very simplistic excalidraw if that helps anyone 😆.

We discussed other projects/products that have similar extensions, and Keda was brought up as an interesting corollary, it's an event driven framework with extensions, we are going to research it further in an effort to learn more. The architecture diagram can be found here.

Slightly less relevant, but I also mentioned Cert Manager, which also has an extension mechanism build into the API, but that mechanism uses an API contract via the kubernetes API.

[antweiss]

Thanks @EItanya for this great summary. So I looked more into KEDA scalers. Basically the majority of them are built-in objects imlementing this interface with their code found here: https://github.com/kedacore/keda/tree/main/pkg/scalers - maintained with the Keda core code. But there's also a possibilty of building external GRPC scalers based on https://github.com/kedacore/keda/blob/main/pkg/scalers/externalscaler/externalscaler.proto. It's all described here: https://keda.sh/docs/2.17/concepts/external-scalers

We could definitely go this path. It looks like the community isn't very eager to build external scalers. There are only 5 of them listed vs. 69 built-in ones. So probably built-in event-watchers will be ok for Khook too. And for those who like independence we can provide protobufs and support for external sources too.

@EItanya also mentioned using WASM as an option, which I gave some thought to but I feel like the tech hasn't proven itself enough and building and loading WASM modules is still pretty cumbersome - don't want to add unnecessary complexity here.