Replies: 8 comments 17 replies
-
|
Hello @ilp-sys , please take a look of the above plan as discussed earlier and let me know if I missed anything or for any mistakes. |
Beta Was this translation helpful? Give feedback.
-
|
Sure, thank you for this outline. Let's get started!! |
Beta Was this translation helpful? Give feedback.
-
Phase 1 Report
Problem Statement
Theoretical BackgroundLocation of Annotations in OCI ImagesTo integrate unikernels into container environments, OCI images are used to package both the application and the necessary metadata. The OCI Image Specification Annotations are optional metadata fields that can appear in three components of the specification, namely in the image index, image manifest and the labels in the image configuration. The OCI specification also defines that the image configuration object is the primary input used to generate the OCI runtime configuration. As a result, metadata stored in the OCI image does not automatically become visible to the container runtime. Only metadata that is propagated into the OCI runtime specification becomes accessible during container execution. Annotation Propagation from Image to RuntimeContainer engines and runtimes (e.g., Docker, Podman, containerd) are responsible for converting an OCI image into an OCI runtime bundle. During this conversion process, metadata stored in the image may be omitted depending on the implementation. The general metadata flow can be summarized as follows: Only metadata that is explicitly propagated into the runtime specification (config.json) becomes visible to the low-level container runtime. Metadata stored solely in the image manifest or other image components may not be preserved during this conversion step. Initial Investigation ResultsInitial investigation reports how image labels and OCI annotations are preserved or lost across different container tools and runtime stages. It investigates whether metadata stored in an OCI image is preserved when containers are created and executed using container engines and runtimes such as Previously Reported Observations
Key findings
Further InvestigationIn non-Kubernetes deployments, Cases Where Annotations Are LostThe loss of annotations can be observed through the following experiment. First, we verify that the OCI image manifest actually contains annotations. ╰─$ sudo ctr content get sha256:60503bb1db41f94083b99f047448e4b7bcc3ace5c9a4999ec0ef4014ac3c52c9 | jq '.annotations'
{
"com.test.foo": "bar",
"org.opencontainers.image.base.digest": "sha256:5c8b2c0a6c745bc177669abfaa716b4bc57d58e2ea3882fb5da67f4d59e3dda5",
"org.opencontainers.image.base.name": "docker.io/library/ubuntu:22.04"
}Next, we run a container using the image. sudo dlv exec ./bin/ctr -- \
--address /run/containerd/containerd.sock \
-n default run --rm localhost:5000/test-annotation:latest test1We can observe that the code responsible for retrieving the image configuration returns only the config descriptor from the manifest and ignores the manifest annotations.
261: // configuration of the image.
262: func Config(ctx context.Context, provider content.Provider, image ocispec.Descriptor, platform platforms.MatchComparer) (ocispec.Descriptor, error) {
263: manifest, err := Manifest(ctx, provider, image, platform)
=> 264: if err != nil {
265: return ocispec.Descriptor{}, err
266: }
267: return manifest.Config, nil
268: }
269:
(dlv) p manifest
github.com/opencontainers/image-spec/specs-go/v1.Manifest {
Versioned: github.com/opencontainers/image-spec/specs-go.Versioned {SchemaVersion: 2},
MediaType: "application/vnd.oci.image.manifest.v1+json",
ArtifactType: "",
Config: github.com/opencontainers/image-spec/specs-go/v1.Descriptor {
MediaType: "application/vnd.oci.image.config.v1+json",
Digest: "sha256:d47671acdbf35974d8548f4f659ee3a453dfd180b49313d0a51a16410...+7 more",
Size: 1579,
URLs: []string len: 0, cap: 0, nil,
Annotations: map[string]string nil,
Data: []uint8 len: 0, cap: 0, nil,
Platform: *github.com/opencontainers/image-spec/specs-go/v1.Platform nil,
ArtifactType: "",},
Layers: []github.com/opencontainers/image-spec/specs-go/v1.Descriptor len: 2, cap: 2, [
(*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc00046e0f0),
(*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc00046e168),
],
Subject: *github.com/opencontainers/image-spec/specs-go/v1.Descriptor nil,
Annotations: map[string]string [
"com.test.foo": "bar",
"org.opencontainers.image.base.digest": "sha256:5c8b2c0a6c745bc177669abfaa716b4bc57d58e2ea3882fb5da67f4d5...+7 more",
"org.opencontainers.image.base.name": "docker.io/library/ubuntu:22.04",
],}
(dlv) bt
0 0x0000000000dc64b3 in github.com/containerd/containerd/v2/core/images.Config
at ./core/images/image.go:264
1 0x0000000000dc35be in github.com/containerd/containerd/v2/core/images.(*Image).Config
at ./core/images/image.go:108
2 0x0000000000fabdef in github.com/containerd/containerd/v2/client.(*image).Config
at ./client/image.go:205
3 0x0000000000fa1b04 in github.com/containerd/containerd/v2/client.WithImageConfigLabels.func1
at ./client/container_opts.go:118
4 0x0000000000f8d293 in github.com/containerd/containerd/v2/client.(*Client).NewContainer
at ./client/client.go:361
5 0x00000000011a80a6 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer
at ./cmd/ctr/commands/run/run_unix.go:457
6 0x000000000119f6a5 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.init.func1
at ./cmd/ctr/commands/run/run.go:184
7 0x00000000006dd942 in github.com/urfave/cli/v2.(*Command).Run
at ./vendor/github.com/urfave/cli/v2/command.go:276
8 0x00000000006dd885 in github.com/urfave/cli/v2.(*Command).Run
at ./vendor/github.com/urfave/cli/v2/command.go:269
9 0x00000000006d8a25 in github.com/urfave/cli/v2.(*App).RunContext
at ./vendor/github.com/urfave/cli/v2/app.go:333
10 0x00000000006d870b in github.com/urfave/cli/v2.(*App).Run
at ./vendor/github.com/urfave/cli/v2/app.go:307
11 0x000000000134f7e5 in main.main
at ./cmd/ctr/main.go:32
12 0x00000000004499e7 in runtime.main
at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/proc.go:283
13 0x0000000000485601 in runtime.goexit
at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/asm_amd64.s:1700
(dlv) n
> github.com/containerd/containerd/v2/core/images.Config() ./core/images/image.go:264 (PC: 0xdc64b3)
259: //
260: // The caller can then use the descriptor to resolve and process the
261: // configuration of the image.
262: func Config(ctx context.Context, provider content.Provider, image ocispec.Descriptor, platform platforms.MatchComparer) (ocispec.Descriptor, error) {
263: manifest, err := Manifest(ctx, provider, image, platform)
=> 264: if err != nil {
265: return ocispec.Descriptor{}, err
266: }
267: return manifest.Config, nil
268: }
269:
(dlv) p manifest
github.com/opencontainers/image-spec/specs-go/v1.Manifest {
Versioned: github.com/opencontainers/image-spec/specs-go.Versioned {SchemaVersion: 2},
MediaType: "application/vnd.oci.image.manifest.v1+json",
ArtifactType: "",
Config: github.com/opencontainers/image-spec/specs-go/v1.Descriptor {
MediaType: "application/vnd.oci.image.config.v1+json",
Digest: "sha256:d47671acdbf35974d8548f4f659ee3a453dfd180b49313d0a51a16410...+7 more",
Size: 1579,
URLs: []string len: 0, cap: 0, nil,
Annotations: map[string]string nil,
Data: []uint8 len: 0, cap: 0, nil,
Platform: *github.com/opencontainers/image-spec/specs-go/v1.Platform nil,
ArtifactType: "",},
Layers: []github.com/opencontainers/image-spec/specs-go/v1.Descriptor len: 2, cap: 2, [
(*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc000018000),
(*"github.com/opencontainers/image-spec/specs-go/v1.Descriptor")(0xc000018078),
],
Subject: *github.com/opencontainers/image-spec/specs-go/v1.Descriptor nil,
Annotations: map[string]string [
"com.test.foo": "bar",
"org.opencontainers.image.base.digest": "sha256:5c8b2c0a6c745bc177669abfaa716b4bc57d58e2ea3882fb5da67f4d5...+7 more",
"org.opencontainers.image.base.name": "docker.io/library/ubuntu:22.04",
],}
(dlv) bt
0 0x0000000000dc64b3 in github.com/containerd/containerd/v2/core/images.Config
at ./core/images/image.go:264
1 0x0000000000dc35be in github.com/containerd/containerd/v2/core/images.(*Image).Config
at ./core/images/image.go:108
2 0x0000000000fabdef in github.com/containerd/containerd/v2/client.(*image).Config
at ./client/image.go:205
3 0x0000000000f0c14b in github.com/containerd/containerd/v2/pkg/oci.WithImageConfigArgs.func1
at ./pkg/oci/spec_opts.go:372
4 0x0000000000f06936 in github.com/containerd/containerd/v2/pkg/oci.ApplyOpts
at ./pkg/oci/spec.go:110
5 0x0000000000fa3ee5 in github.com/containerd/containerd/v2/client.WithSpec.func1
at ./client/container_opts.go:318
6 0x0000000000f8d293 in github.com/containerd/containerd/v2/client.(*Client).NewContainer
at ./client/client.go:361
7 0x00000000011a80a6 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer
at ./cmd/ctr/commands/run/run_unix.go:457
8 0x000000000119f6a5 in github.com/containerd/containerd/v2/cmd/ctr/commands/run.init.func1
at ./cmd/ctr/commands/run/run.go:184
9 0x00000000006dd942 in github.com/urfave/cli/v2.(*Command).Run
at ./vendor/github.com/urfave/cli/v2/command.go:276
10 0x00000000006dd885 in github.com/urfave/cli/v2.(*Command).Run
at ./vendor/github.com/urfave/cli/v2/command.go:269
11 0x00000000006d8a25 in github.com/urfave/cli/v2.(*App).RunContext
at ./vendor/github.com/urfave/cli/v2/app.go:333
12 0x00000000006d870b in github.com/urfave/cli/v2.(*App).Run
at ./vendor/github.com/urfave/cli/v2/app.go:307
13 0x000000000134f7e5 in main.main
at ./cmd/ctr/main.go:32
14 0x00000000004499e7 in runtime.main
at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/proc.go:283
15 0x0000000000485601 in runtime.goexit
at /home/jiwoo/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/runtime/asm_amd64.s:1700
(dlv) |
Beta Was this translation helpful? Give feedback.
-
|
Hey @cmainas @panosmaurikos , I plan to use this discussion to keep track of my progress and findings. I’ll keep this comment updated as I go through phase 1, so feel free to check in and leave any feedback or thoughts you might have. I want to keep everything transparent and leave a solid record of how we’re approaching this—from the background concepts to the actual implementation. I’ll reach out again if I run into any questions, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , thank you, it sounds like a good idea. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys, a small recap of our sync today (10/03/2026):
Also, regarding the tracing of Go programs, it might be helpful to check the trace Go package and the respective go tool trace. Using these two you can trace the function calls inside containerd. Moreover, containerd has some documentation about tracing, but I have never tried it out. At last, here is the |
Beta Was this translation helpful? Give feedback.
-
|
Hello @cmainas, I have added the function trace for the non-cri container execution path. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , thank you for the trace. If I udnerstand correctly the trace you shared refers to the creation of the rootfs for the container. However, we are mostly interested in the trace for the creation of the OCI spec for the runtime. Looking a bit in the code, it should be this variable https://github.com/containerd/containerd/blob/main/cmd/ctr/commands/run/run_unix.go#L124 which is later passed as argument in Please let me know if I misunderstood something in the trace. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @cmainas I’ve updated the trace. Previously I set the breakpoint directly on |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , thank you for the update. It seems that the The questions that we need to answer:
|
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , a small recap for the next steps from our sync today (17/03/2026), since it was a bit chaotic:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @cmainas for the summary. It really was chaotic but still helpful. |
Beta Was this translation helpful? Give feedback.
-
Phase 2 ReportSummary of Phase 1According to the OCI Image Specification, the runtime bundle is constructed solely from the image’s Configuration, without incorporating information from the Manifest or Index. where containerd sets
|
Beta Was this translation helpful? Give feedback.
-
|
According to the conversion specification, |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , a small recap for the next steps from our last sync (31/03/2026),
|
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , yes NRI can be used for various purposes and especially for cases where users want to modify the "usual" operation of the low level runtime (adding hooks, changing the spec, etc.), without modifying the low level runtime. However, since we control the low level runtime (and the shim) we can modify their operation as we wish. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @cmainas I wanted to clarify something from yesterday’s meeting. My understanding is that we should check whether there is an API or plugin that enables metadata such as annotations in config.json (or equivalent labels) to be passed from the shim to the low-level runtime. I recall that this approach was mentioned as potentially sufficient, but I may have misunderstood. If that is the case, I am wondering why this mechanism would be necessary, since config.json could be accessed by urunc. I am not entirely sure which direction I should focus on at this point. Should I be looking into what information containerd passes to the shim, or should I instead examine how the shim reads config.json through specific code paths? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , our goal is to get the annotations form the image manifest or Config.Labels from contianerd to urunc. There are two ways we can do that:
So, if contianerd somehow was passing this information to the shim everything would be great. Unfortunately, according to our yesterday's sync, this does not seem to be the case. Therefore, we need to explore contianerd's API and retrieve this information through a specific call. |
Beta Was this translation helpful? Give feedback.
-
|
The shim writes into It looks like the shim may be able to query containerd directly: retrieve the image reference through the Containers API, resolve the target descriptor through the Images API, and then read the corresponding blob through the Content API. If the target is the image manifest, we could extract annotations from there. One question: is it acceptable for the shim to establish a new connection to containerd for this? func fetchManifestAnnotations(ctx context.Context, ttrpcAddr, containerID, ns string) (map[string]string, error) {
// connect to containerd ttrpc
conn, err := net.DialTimeout("unix", ttrpcAddr, 10*time.Second)
if err != nil {
return nil, fmt.Errorf("dial unix %s: %w", ttrpcAddr, err)
}
client := ttrpc.NewClient(conn)
defer client.Close()
ctx = namespaces.WithNamespace(ctx, ns)
// containers API → image ref
cClient := containersapi.NewTTRPCContainersClient(client)
cResp, err := cClient.Get(ctx, &containersapi.GetContainerRequest{ID: containerID})
if err != nil {
return nil, fmt.Errorf("get container: %w", err)
}
imageRef := cResp.Container.Image
if imageRef == "" {
return nil, fmt.Errorf("no image ref on container %s", containerID)
}
// images API → manifest descriptor
iClient := imagesapi.NewTTRPCImagesClient(client)
iResp, err := iClient.Get(ctx, &imagesapi.GetImageRequest{Name: imageRef})
if err != nil {
return nil, fmt.Errorf("get image: %w", err)
}
target := iResp.Image.Target
// content API → manifest blob
csClient := contentapi.NewTTRPCContentClient(client)
stream, err := csClient.Read(ctx, &contentapi.ReadContentRequest{
Digest: target.Digest,
Size: target.Size,
})
if err != nil {
return nil, fmt.Errorf("content read: %w", err)
}
var raw []byte
for {
resp, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
return nil, fmt.Errorf("stream recv: %w", err)
}
raw = append(raw, resp.Data...)
}
var manifest ocispec.Manifest
if err := json.Unmarshal(raw, &manifest); err != nil {
return nil, fmt.Errorf("unmarshal manifest: %w", err)
}
result := make(map[string]string)
for k, v := range manifest.Annotations {
if strings.HasPrefix(k, uruncPrefix) {
result[k] = v
}
}
return result, nil
} |
Beta Was this translation helpful? Give feedback.
-
That is some good news.
The shim already has a connection with contianerd. However, there is a high chance that this connection might be limited to specific operations (e.g. specific to containers lifecycle). So ideally, we should use the existing connection, but If that is not possibe, then we will need to establish a new connection. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , a small recap for the next steps from our last sync (07/04/2026),
|
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , a small recap from our last sync (14/04/2026):
|
Beta Was this translation helpful? Give feedback.
-
|
Hey @cmainas, I’ve created a draft PR for #12. It is still in progress, I wanted to check in on the current status. Beyond functional validation, there are still several tasks remaining, including improving logging, error handling, and adding comments. And I also wanted to ask: do images built with bunny automatically include those annotations? And what about the images listed in the https://github.com/urunc-dev/urunc/blob/main/docs/Sample-images.md —do they include the urunc-specific annotations as well? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @ilp-sys , thank you for sharing the code. The images that bunny generates include the annotations, but be careful with docker (See https://github.com/nubificus/bunny/blob/main/docs/annotations.md). Some of the images in https://github.com/urunc-dev/urunc/blob/main/docs/Sample-images.md have lost the annotations due to docker. They should still have the annotations as "docker labels" though. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Mentorship plan: Investigate missing custom OCI annotations in urunc containers
This discussion outlines the proposed plan as discussed with @ilp-sys for the CNCF mentorship project "Investigate missing custom OCI annotations in urunc containers",
The plan is structured into three phases. Each phase has clear goals and specific outcomes, along with suggested tasks and sub-tasks to help guide the work. The listed sub-tasks are meant as guidance and reference points, they are not strict requirements. The exact order of tasks within each phase can be adjusted as long as the main outcomes are achieved.
Phase 1
The goal of Phase 1 is to build the necessary background and get familiar with OCI annotations, containerd and the annotating container images at build time. This phase is expected to last up to 3 weeks (02/03/2026 – 22/03/2026). During this period, the focus will be on understanding the theoretical aspects of OCI images, establishing the knowledge required to effectively apply these concepts in the later implementation phases.
Tentative tasks and sub-tasks
Outcome
Deadline: Completed no later than 22/03/2026 (AoE).
Description: A report with the:
Phase 2
The goal of Phase 2 is to investigate where annotations are getting lost in the containerd / urunc workflow and create a Proof of Concept to resolve the metadata passing from images to the container runtime. This phase is expected to last up to 5 weeks (23/03/2026 – 03/05/2026). During this time, the focus will shift to the practical investigation of the missing OCI annotations and the implementation of a PoC that resolves the overall issue (without or without annotations).
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 02/05/2026 (AoE).
Description: The following items:
Phase 3
The goal of Phase 3 is to finalize and polish the solution created in the previous phase. This phase is expected to last up to 4 weeks (04/05/2026 – 28/05/2026), with an emphasis on refinement, validation, and broader ecosystem exploration. During this period, the focus will be on refining the solution created in the previous phase, enhance it with the necessary tests explore ways to overwrite the metadata passed to the runtime at runtime and explore the compatibility of the current approach with other high level container runtimes (e.g. CRI-O),
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 29/05/2026 (AoE).
Description: The following items:
Beta Was this translation helpful? Give feedback.
All reactions