(Go) Spelunk

Spelunk is a Golang library for extracting secrets from various sources (Kubernetes, Vault, env vars, files) using a unified URI-based string we are calling Secret Coordinates. Here are some example of coordinates:

# Secret from the namespace `ns`,
# stored inside the secret `my-team-secret`
# at data key `the-key`
k8s://ns/my-team-secret/the-key

# Secret provided in the form
# of base64-encoded string
base64://bXktYmlnLXNlY3JldAo=

# Secret stored in a JSON
# file at a specific field
file://kafka-credentials.json?jp=$.kafka.password

Spelunk simplifies the access to secrets by just providing the coordinates for "digging up" configuration values in cloud-native CLI tools and applications.

Its primary application is command line tools, but... you do you! Users point at a secret from any source, providing the right coordinates: your tool/service/software can use Spelunk to adapt dynamically and fetch the secret.

With a single library, the source of secrets is flexible and adapts to your environment, situation and/or needs.

Spelunk can be configured to support more Sources, and users can apply Modifiers to "prepare" the secret in the exact way they need it.

Get started

Add the library to your project:

# Pull the main library
go get github.com/detro/spelunk
# Pull optional plugins
go get github.com/detro/spelunk/plugin/source/kubernetes

Setup a new Spelunker and start digging up secrets:

package main

import (
	"github.com/detro/spelunk"
	"github.com/detro/spelunk/plugin/source/kubernetes"
	v1 "k8s.io/client-go/kubernetes/typed/core/v1"
)

// Initialize the Kubernetes client...
k8sClient, err := v1.NewForConfig(restConfig)

// Create a Spelunker
spelunker := spelunk.NewSpelunker(
	kubernetes.WithKubernetes(k8sClient),
)

// Get coordinates to a secret from one of many supported sources:
// from Kubernetes... 
coord, err := types.NewSecretCoord("k8s://my-namespace/my-secret/my-data-key")
// ... or from plain text (please don't!)
coord, err := types.NewSecretCoord("plain://MY_PLAINTEXT_SECRET")
// ... or from a local file
coord, err := types.NewSecretCoord("file://secrets.json?jp=$.kafka.password")
// ... or from environment variable
coord, err := types.NewSecretCoord("env://GITHUB_PRIVATE_TOKEN")

// Dig-up secrets!
secret, _ := spelunker.DigUp(ctx, coord)

Examples

Find some useful /examples directory for how to use spelunk with various libraries for configuration or command line arguments parsing.

built-in vs plug-in

Spelunk comes with a bunch of features: Sources and Modifiers, the role of which is explained below. Some are built-in, and a new spelunk.Spelunker instance comes with those enabled; others are plug-in, and you will have to enable them as SpelunkerOption provided at construction time:

package main

import (
	"github.com/detro/spelunk"
	"github.com/detro/spelunk/plugin/source/kubernetes"
	"github.com/detro/spelunk/plugin/source/vault"
)

_ = spelunk.NewSpelunker(
	kubernetes.WithKubernetes(k8sClient),
	vault.WithVault(vaultClient),
)

Key Types

spelunk.Spelunker is the entry point type, and it does its job using the following types.

Coordinates (SecretCoord)

This is the starting point: take a string containing Secret Coordinates as documented above, and use types.NewSecretCoord to turn it into a SecretCoord.

This is a generic, secret-type-agnostic representation of how to find a secret. And it's all that Spelunker needs to dig-up the secret.

From user input to SecretCoord

SecretCoord implements encoding.TextUnmarshaler, so it can be created through the unmarshalling of command-line user input, through json.Unmarshal and any other type-aware process.

For example, when using the awesome Kong library:

package main

import "github.com/detro/spelunk"

type CLI struct {
	Password spelunk.SecretCoord `name:"password" short:"p" help:"your password"`
	// ...
}

Sources (SecretSource)

Sources are places out of which a secret can be "dug-up". Some are built-in to spelunk.Spelunker, others are plug-in and need to be enabled.

Source (of Secrets)	Type (scheme)	Available as	Status	Doc
Environment Variables	env://	built-in	✅	link
File	file://	built-in	✅	link
Plaintext	plain://	built-in	✅	link
Base64 encoded	base64://	built-in	✅	link
Kubernetes Secrets	k8s://	plug-in	✅	link
Vault	vault://	plug-in	✅	link
Azure Key Vault	az://	plug-in	⏳
AWS Secrets Manager	aws://	plug-in	✅	link
GCP Secrets Manager	gcp://	plug-in	✅	link
1Password	opass://	plug-in	⏳
LastPass	lpass://	plug-in	⏳
Bitwarden	bw://	plug-in	⏳
Keeper	keeper://	plug-in	⏳
Dashlane	dashlane://	plug-in	⏳

Modifiers (SecretModifier)

Modifiers are optional behaviour applied to a secret after it has been dug-up by Spelunk. It can be seen as a function in the mathematical sense:

$$ Modifier(SecretVal, Input) = ModifiedSecVal $$

Each modifier is applied in the same order provided in the secret coordinates:

<type>://<location>?mod1=A&mod2=B&mod1=C

will result in this sequence:

• Spelunker digs-up the secret <value> of type <type> from the <location>
• mod1 takes the <value> and applies mod1(<value>, A) = <value_A>
• mod2 takes the <value_A> and applies mod2(<value_A>, B) = <value_A_B>
• mod1 takes the <value_A_B> and applies mod1(<value_A_B>, C) = <value_A_B_C>
• client code is returned the final <value_A_B_C>

Modifier (of Secrets)	Type (query)	Available as	Status	Doc
JSONPath extractor	?jp=<JSONPath>	built-in	✅	link
Base64 encoder	?b64	built-in	✅	link
XPath extractor	?xp=<XPath>	plug-in	⏳
YAML JSONPath extractor	?yp=<JSONPath>	plug-in	⏳
TOML JSONPath extractor	?tp=<JSONPath>	plug-in	⏳
SHA-2/3 / BLAKE-2/3 / ... hasher	TBD	plug-in	⏳

Contributing

If you are interested in contributing (for example, you have a brilliant idea for a plug-in), we have some contribution guidelines.

License

This project is shared under the MIT license.

Links

• Architecture documentation: understand how Spelunk works internally
• Contribution guidelines: setting some ground rules
• Agents documentation: helps LLM-agent augmented developers in their contribution journey
• Changelog: track the evolution of the project