Replies: 16 comments 11 replies
-
|
Hello @sidneychang , please take a look of the above plan as discussed earlier and let me know if I missed anything or for any mistakes. |
Beta Was this translation helpful? Give feedback.
-
|
The following content captures my current understanding and findings from this week's exploration of containerd, rootfs setup, and snapshotters. It may still contain incomplete sections or inaccuracies, and I will continue improving and refining it as I explore the topic further. 1. Overall Architecture and Container rootfs Construction Workflow
A typical example appears in // 1. Pull the image and unpack it into the snapshotter
image, err := client.Pull(context, "docker.io/library/redis:latest", containerd.WithPullUnpack)
// 2. Allocate a rootfs and OCI spec for the container based on the image
redis, err := client.NewContainer(context, "redis-master",
containerd.WithNewSnapshot("redis-rootfs", image),
containerd.WithNewSpec(oci.WithImageConfig(image)),
)
// 3. Turn the container object into a runnable task
task, err := redis.NewTask(context, cio.NewCreator(cio.WithStdio))
defer task.Delete(context)
pid := task.Pid()
err = task.Start(context)
status, err := task.Wait(context)This workflow can be abstracted as:
2. rootfs Preparation and the Role of Snapshotter2.1
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you @sidneychang for the detailed explanation. Everything looks good. |
Beta Was this translation helpful? Give feedback.
-
|
Last week I also ran some preliminary startup latency tests in my prototype impletation. In this prototype, the snapshot view is created in the shim before launching urunc. The idea is to prepare a read-only view of the container snapshot and expose the required artifacts (e.g., unikernel, initrd, and configuration files) to the runtime without directly copying them from the snapshot device. To understand the impact of this approach on startup latency, I added instrumentation logs around the snapshot view creation path in the shim. The experiments were conducted using the nginx-qemu-linux-raw image. The results suggest that the dominant cost comes from the devmapper snapshot view creation itself. A typical run shows:
This results in roughly ~170 ms of additional control-plane overhead in the startup path when the snapshot view is created. For comparison, in the previous implementation we directly copied the required files (unikernel, initrd, and This difference is also reflected in the end-to-end startup measurements:
From these preliminary results, the additional latency appears to come mainly from the control-plane operations required to create the snapshot view (interaction with containerd, snapshot service calls, and metadata updates). For relatively small images such as That said, snapshot views may still be beneficial in other scenarios (for example larger images, repeated snapshot reuse, or cases where reducing filesystem duplication is more important than minimizing startup latency). I would be interested to hear whether this observation aligns with the intended usage model of snapshot views, or if there are scenarios where this approach would be expected to perform better. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @sidneychang for this early evaluation. To be honest, I was not expecting the snapshot creation to require that much time. On the other hand, the small overhead of copying files in urunc can be explained from the fact that these operations take place in tmpfs. However, as you already mentioned snapshot views will be beneficial in some scenarios. I do not believe that these scenarios are larger images, since we are only interested in some specific files. However, in case we deploy multiple containers with the same image (and hence reducing duplication), snapshot views will perform better than copying files. For example with the numbers form the early evaluation spawning ~12 containers with the same image could have the same overhead as creating a single snapshot view and sharing it among all of the containers. Also, this overhead will affect entirely the first container and not the rest. Another point that would be helpful to search is if the size of the image affects the time required for the creation of the view snapshot. If that is the case, then we can explore ways to decrease the size of the snapshot we are interested (e.g. only first layer of the image, I am not sure if this is possible). |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the suggestion. I also think that sharing the same snapshot view across multiple containers is a feasible direction. In fact, I had considered this possibility earlier as well. My main concern is how to properly manage the lifecycle of such a view. Since this view would not be created directly by containerd, we would need a mechanism to ensure that containerd can reclaim it at the appropriate time. I will investigate how this could be handled safely. Regarding the snapshot size, I noticed that when containerd creates a view snapshot, most of the work seems to involve database updates and metadata writes. Because of that, I am not entirely sure whether the size of the selected layer actually has a significant impact on the creation time. Even if it does, interacting with containerd through APIs (for example when using lightweight operations like leases) already takes a few milliseconds. Therefore, it might be difficult for snapshot views alone to significantly improve container startup latency. |
Beta Was this translation helpful? Give feedback.
-
|
During some startup latency experiments with urunc + containerd + devmapper snapshotter, I observed several behaviors that might be interesting to share. The goal of the experiment was to explore container startup latency under different snapshot usage patterns. One important note about the environment: the experiments were conducted on a VM with relatively limited memory, but according to system monitoring during the tests, no swap space was used, so the observed latency does not appear to be related to swapping. Testing MethodOriginally, the testing pattern was:
Under this pattern, the startup latency remained relatively stable. Later, to explore snapshot reuse behavior, the testing method was changed to:
Observation 1: Higher Startup Latency When Another Container Is RunningWhen using the devmapper snapshotter, it appears that if there is already a container running based on the same image, starting additional containers tends to have higher startup latency. From instrumentation logs added around the runtime startup path, the increased latency is not due to urunc itself, but mainly comes from Example results (baseline implementation without snapshot views): Compared with the first container startup (~0.8–1.0s), additional containers tend to start around 1.2–1.5s when another container based on the same image is already running. Observation 2: Using a Shared Snapshot ViewI experimented with an approach intended to reuse the snapshot view more explicitly: Create a snapshot view Mount the view block device on the host filesystem Share the mounted filesystem with multiple containers via bind mounts In other words, instead of letting each container mount its own snapshot, multiple containers reuse the same mounted filesystem derived from the snapshot view. In this setup, I observed that container startup latency fluctuates significantly more, again mainly due to TaskService.Create, rather than urunc itself. At this stage, the focus is not on whether the view-based implementation is faster or slower, but rather on the latency variability observed during repeated container startups. Overall, compared with the baseline pattern, the view-based approach showed noticeably larger startup latency variance in the tests. I am continuing to investigate the underlying causes in the containerd snapshotter and runtime startup path, and I will update this thread if I find additional details. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , thank you for this information and the evaluation. I have a few questions and some comments that it would be nice if we could discuss.
|
Beta Was this translation helpful? Give feedback.
-
|
I reran the snapshot-view experiments on a much stronger machine than the one used in earlier rounds, so the absolute startup times are better across the board.
SummaryThe current snapshot-view implementation is working correctly in terms of reuse, but on devmapper it is still not improving startup latency. In serial startup it is slightly slower than the original path, and under concurrency it is clearly slower. How the current implementation worksThe current implementation creates one shared read-only view per root snapshot and reuses it across containers. Conceptually, it does this: At create time, the shim:
At delete time, it removes the container’s usage marker, and only the last user removes the shared view. So the design already does shared reuse. The question is not whether reuse exists, but why latency is still worse. Why the view path is slowerThe main reason is that this path is additive, not substitutive.
Test methodFor serial startup, I ran 10 starts and kept the first container. Every later container was deleted immediately after startup. The command shape was: The serial script logic was: For parallel startup, we launched 10 containers concurrently: ResultsWarm serial startup:
So in steady-state serial startup, the snapshot-view path was slightly slower. 10-way parallel startup:
So under concurrency, the gap becomes much larger. Profiling evidenceTo understand where the time went, I added startup profiling. The most important finding is that the current implementation does reuse the shared view:
This rules out “one view per container” as the explanation. But profiling also showed why latency is still poor:
Then I reduced the lock scope around shared-view lifecycle management. That helped serial startup and removed lock contention as the main suspect:
However, parallel startup still did not improve meaningfully:
So after that change, the bottleneck was no longer lock contention. The bottleneck was the cost of the first shared view creation itself. ConclusionAt this point, the evidence suggests:
So in its current form, this snapshot-view approach behaves more like a storage/layout optimization than a startup-latency optimization. |
Beta Was this translation helpful? Give feedback.
-
|
I have recently been looking into the startup behavior of From the implementation model, In my own tests, the extra overhead introduced by concurrent startup is quite significant, and I found that similar behavior has already been observed by others in the community. For example, in containerd issue #12335, someone analyzed the case of starting 100 Pods concurrently and pointed out that device creation and In addition, there have already been reports that In earlier discussions, we had planned to introduce a view to avoid one extra copy. But now I increasingly suspect that this view may not actually provide real acceleration. Although it may save a small amount of copy time (which is actually very small in practice), it also means adding one more object creation and handling step on top of the existing So my current tentative conclusion is that, for |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , unfortunately, this is true, devmapper can be much slower compared to other snapshotters. However, the root cause might be the implementation of the snapshotter. Therefore, as mentioned in the Friday's sync, it would be helpful to also evaluate the blockfile snapshotter. Furthermore, despite the latency in concurrent execution, there still can be use cases where storing the boot files for the sandbox in urunc might cause harm in other ways (e.g. storage). We also need to check if the devmapper snapshotter could potentially help towards this direction (using less storage / memory). In general, a block-based snapshotter is important for urunc sandboxes, because a lot of VMMs and software-based sandboxes might offer only block-based support for storage. Therefore, devmapper and similar snapshots help a lot on preparing the rootfs for such sandboxes. |
Beta Was this translation helpful? Give feedback.
-
Some observations on |
| Configuration | Concurrency | Avg (s) | Min (s) | Max (s) | Stddev (s) |
|---|---|---|---|---|---|
| devmapper original | 1 | 0.6628 | 0.6628 | 0.6628 | 0.0000 |
| devmapper original | 10 | 1.7104 | 1.0709 | 2.0166 | 0.3157 |
| devmapper original | 32 | 4.3700 | 2.3930 | 5.9973 | 0.9732 |
| devmapper original | 64 | 7.4371 | 2.7701 | 10.4343 | 1.7315 |
| devmapper + view | 1 | 0.7694 | 0.7694 | 0.7694 | 0.0000 |
| devmapper + view | 10 | 2.4569 | 2.1186 | 2.8414 | 0.2085 |
| devmapper + view | 32 | 6.6032 | 5.7286 | 7.6330 | 0.5345 |
| devmapper + view | 64 | 12.9197 | 10.6997 | 15.1199 | 0.8107 |
| blockfile no-view | 1 | 2.8293 | 2.8293 | 2.8293 | 0.0000 |
| blockfile no-view | 10 | 14.2522 | 3.1052 | 24.2788 | 7.4659 |
| blockfile no-view | 32 | 40.9451 | 3.3203 | 76.3530 | 22.1804 |
| blockfile no-view | 64 | 78.5843 | 3.3048 | 152.3767 | 44.7836 |
| blockfile + view | 1 | 2.8091 | 2.8091 | 2.8091 | 0.0000 |
| blockfile + view | 10 | 20.5956 | 10.5745 | 24.9275 | 5.0656 |
| blockfile + view | 32 | 45.7393 | 13.8223 | 76.0734 | 18.9432 |
| blockfile + view | 64 | 86.7734 | 12.5340 | 153.2913 | 43.6929 |
This makes one thing very clear: the disadvantage of blockfile grows as concurrency increases.
Raw timing data
I am also including the raw timing samples here for completeness.
Devmapper and no-View
Concurrency 10.6628
Concurrency 101.0709, 1.3500, 1.5362, 1.6140, 1.7670, 1.8301, 1.9639, 1.9738, 1.9814, 2.0166
Concurrency 322.3930, 2.6542, 2.6574, 2.7670, 2.9605, 3.3636, 3.5575, 3.7255, 3.8011, 3.8804, 4.0815, 4.1321, 4.2901, 4.4287, 4.4254, 4.4210, 4.6252, 4.6250, 4.6504, 4.6867, 4.9246, 4.9363, 5.0816, 5.0885, 5.1085, 5.1294, 5.2772, 5.2824, 5.4455, 5.6246, 5.8174, 5.9973
Concurrency 642.7701, 4.5399, 4.7248, 4.8749, 5.0216, 5.0842, 5.0728, 5.2251, 5.4171, 5.5718, 5.6370, 5.6393, 5.7806, 5.7849, 5.9254, 6.1339, 6.3559, 6.4928, 6.5081, 6.5614, 6.5567, 6.6351, 6.7030, 6.7752, 6.9231, 7.1336, 7.1446, 7.1357, 7.1534, 7.2977, 7.3375, 7.3851, 7.3970, 7.5273, 7.5483, 7.5850, 7.6290, 7.7287, 7.9482, 8.0302, 8.0462, 8.0679, 8.1471, 8.1746, 8.3648, 8.4431, 8.4655, 8.5913, 8.6074, 8.7862, 8.9875, 9.1598, 9.4432, 9.5231, 9.6505, 9.7177, 9.7644, 9.8202, 9.9882, 10.1671, 10.1993, 10.3592, 10.3698, 10.4343
Devmapper and use a shared View
Concurrency 10.7694
Concurrency 102.1186, 2.3058, 2.3246, 2.3466, 2.3768, 2.4333, 2.5755, 2.6090, 2.6372, 2.8414
Concurrency 325.7289, 5.7286, 5.8149, 6.0301, 6.0415, 6.0520, 6.0568, 6.1095, 6.1845, 6.1841, 6.1799, 6.2860, 6.3462, 6.4996, 6.5602, 6.5710, 6.6444, 6.7601, 6.7831, 6.8113, 6.8260, 6.8560, 6.9147, 6.9711, 7.0443, 7.1982, 7.2028, 7.2312, 7.2446, 7.3874, 7.4214, 7.6330
Concurrency 6410.6997, 10.9039, 11.8133, 12.0998, 12.2745, 12.3020, 12.3212, 12.4462, 12.4751, 12.4979, 12.4890, 12.4923, 12.5055, 12.5458, 12.5875, 12.6142, 12.6149, 12.5991, 12.6052, 12.6180, 12.6263, 12.6172, 12.6258, 12.6284, 12.6282, 12.6370, 12.6202, 12.6262, 12.6396, 12.6284, 12.6219, 12.6247, 12.6383, 12.6318, 12.6324, 12.6388, 12.6613, 12.6698, 12.6727, 12.7091, 12.7566, 12.8544, 12.8862, 12.8994, 12.9759, 13.0184, 13.2214, 13.2467, 13.2813, 13.3032, 13.3918, 13.3900, 13.4350, 13.7252, 13.8153, 13.9175, 14.0878, 14.3330, 14.3349, 14.4974, 14.5386, 14.6449, 14.9029, 15.1199
blockfile and no View
Concurrency 12.8293
Concurrency 103.1052, 5.5674, 7.9705, 10.3697, 12.7532, 15.1741, 20.1096, 20.6956, 22.4978, 24.2788
Concurrency 323.3203, 7.4115, 8.1950, 10.4836, 15.4757, 15.8389, 18.0301, 20.3336, 22.7309, 25.2223, 27.5892, 29.8835, 34.5640, 35.2944, 37.1073, 39.3863, 42.0385, 46.4552, 47.5370, 50.1888, 51.3144, 55.7064, 56.0546, 58.6294, 60.8470, 62.6859, 65.8204, 68.2748, 70.1479, 72.3413, 74.9816, 76.3530
Concurrency 643.3048, 5.6381, 7.9732, 10.4862, 12.9163, 15.1246, 17.5128, 19.9351, 22.2346, 24.5225, 26.7343, 30.1488, 31.4135, 33.4584, 35.8712, 38.1004, 40.7972, 43.0258, 45.3449, 50.4551, 51.1847, 54.0341, 54.9797, 57.5968, 60.1320, 62.2911, 64.7874, 66.9418, 69.1502, 72.0469, 74.2890, 77.0926, 81.6011, 82.4126, 83.9984, 86.1275, 88.7347, 91.2607, 93.9695, 96.4191, 98.3960, 100.2426, 105.7345, 106.5355, 110.1670, 110.7324, 113.7679, 117.8232, 118.6740, 123.3320, 123.5920, 124.9864, 127.8304, 130.5702, 132.6665, 134.4144, 137.9454, 139.9549, 142.5985, 144.9472, 146.7368, 149.4373, 151.8834, 152.3767
blockfile and use a shared View
Concurrency 12.8091
Concurrency 1010.5745, 15.4463, 16.4722, 18.5294, 21.8350, 24.2985, 24.4854, 24.6818, 24.7056, 24.9275
Concurrency 3213.8223, 14.5868, 17.0446, 19.1483, 22.0457, 24.5364, 25.7591, 27.0690, 30.2287, 32.9242, 34.5675, 38.0236, 41.2834, 42.6369, 44.4846, 47.1391, 50.9122, 51.8467, 53.3780, 54.3230, 54.3776, 54.4034, 57.8684, 58.8422, 60.2175, 65.0495, 65.7195, 67.6173, 70.0243, 71.8504, 75.8557, 76.0734
Concurrency 6412.5340, 15.3160, 16.5870, 20.0854, 21.5345, 24.2904, 25.6475, 27.6185, 31.1265, 33.8792, 36.0453, 39.4449, 41.8233, 42.7420, 45.8095, 49.1573, 50.1146, 52.8277, 54.6968, 57.3054, 59.1127, 62.2059, 64.1450, 66.1186, 68.2766, 71.5521, 73.9081, 76.5338, 78.0364, 81.2729, 83.6384, 86.2146, 88.5108, 92.2084, 93.0919, 95.7777, 98.0437, 99.7103, 102.1815, 105.1966, 107.5742, 109.4366, 112.2623, 114.5843, 116.8198, 120.6527, 121.4012, 123.7027, 126.1087, 128.4059, 131.1994, 132.5935, 134.2823, 138.1150, 139.2946, 142.8230, 145.4999, 147.7409, 152.1521, 152.2853, 152.7977, 153.0632, 153.0917, 153.2913
All of the above values come directly from my recorded runs.
What I found
1. devmapper is clearly better than blockfile
From the averages alone, devmapper is much faster than blockfile, and the gap gets larger as concurrency increases. That suggests the difference is not just a fixed cold-start cost. It looks more like a difference in scalability under concurrent startup.
2. I did not see evidence that view reduces startup latency
On devmapper, view is slower than the original implementation. On blockfile, view is also slower than no-view for most concurrency levels. So based on this dataset, I do not see evidence that view helps reduce startup latency. If anything, it introduces additional overhead.
3. On devmapper, view looks slower but more stable
With devmapper, the average gets worse with view, but the standard deviation becomes smaller. That suggests view may be introducing a more uniform path: not randomly hurting a few runs, but consistently adding work to most or all runs.
4. On blockfile, the backend already looks heavy, and view just makes it worse
Both blockfile variants show very large variance and very long tails at higher concurrency. At concurrency 64, both max values are around 150 seconds, and the standard deviations are above 40 seconds. So the main issue on blockfile appears to be the backend itself under concurrency, while view just adds more overhead on top.
5. In my implementation, the minimum time for the view case is close to the first container passing the shared-view critical path
Because my shared view is protected by a lock, the minimum completion time in the view case is likely close to the container that first acquires the lock, finishes shared view initialization, and then completes startup. From that perspective, on blockfile, view looks like it adds a front-loaded barrier, while on devmapper, the rise in the minimum time is more pronounced, which suggests that the shared view path is more directly exposed on the critical path there.
My current hypotheses
My current interpretation is:
The main differences here are more likely caused by the
snapshotter/ runtime path than by image format.The large fluctuations I observed on
devmappermay indeed be related to the design or runtime state ofdevmapper, such as metadata state, cache warmth, prepared device state, or whether some initialization path was effectively reused.In its current form,
viewseems to be adding complexity to the startup path rather than reducing the amount of work on the critical path.The main bottleneck for
blockfileseems to be poor scalability under concurrency, notviewalone. Even withoutview, it already shows severe queueing and long-tail behavior.
Conclusion
My current conclusion is fairly simple:
devmapperperforms much better thanblockfile, especially as concurrency increases.I do not currently see
viewproviding meaningful startup-latency reduction.The startup-time fluctuation I observed on
devmappermay be related todevmapperdesign or runtime state, but I would need more stage-level timing to verify that.On
blockfile, the major problem seems to be backend scalability and queueing under concurrency, andviewmostly makes that worse rather than fixing it.
Beta Was this translation helpful? Give feedback.
-
|
This is my script for the evalutaion: #!/usr/bin/env bash
# Concurrently start containers and measure nerdctl run latency;
# ensure the image exists before starting; remove containers created by this script afterward.
# Each execution runs only one concurrency level:
# 1 (single container, for comparison with manual `time nerdctl run`),
# 10, 32, 64 (passed as the first argument).
# Default namespace is nerdctl's "default".
#
# Usage:
# ./run_concurrent_start_bench.sh 1
# ./run_concurrent_start_bench.sh 10
# ./run_concurrent_start_bench.sh --blockfile 64
# ./run_concurrent_start_bench.sh --devmapper 32
# ./run_concurrent_start_bench.sh -s overlayfs 10
#
# Environment variables: NS, IMAGE, SNAPSHOTTER, RUNTIME, SKIP_RESET, RESET_SCRIPT, NAME_PREFIX
# (Command-line options override SNAPSHOTTER)
#
# Image & snapshotter notes:
# containerd shares image blobs, but unpacked writable rootfs snapshots
# (devmapper / blockfile / overlayfs) are separated per snapshotter.
# If only pulled under devmapper, switching to blockfile may require re-pull or first-run unpack.
set -euo pipefail
NS="${NS:-default}"
IMAGE="${IMAGE:-harbor.nbfc.io/nubificus/urunc/nginx-qemu-linux-raw:latest}"
SNAPSHOTTER="${SNAPSHOTTER:-devmapper}"
# If set to 1, always run nerdctl pull (ensure unpacked under current snapshotter)
FORCE_PULL="${FORCE_PULL:-0}"
RUNTIME="${RUNTIME:-io.containerd.urunc.v2}"
RESET_SCRIPT="${RESET_SCRIPT:-$(dirname "$0")/reset_urunc_bench_state.sh}"
# Container name prefix for identification and cleanup
NAME_PREFIX="${NAME_PREFIX:-urunc-concbench}"
SCRIPT_TAG="concbench-$$"
log() { printf '%s\n' "$*"; }
ensure_image() {
local need_pull=0
if [[ "${FORCE_PULL}" == "1" ]]; then
need_pull=1
log "[pull] FORCE_PULL=1, pulling image under snapshotter=$SNAPSHOTTER"
elif ! sudo nerdctl -n "$NS" --snapshotter "$SNAPSHOTTER" image inspect "$IMAGE" >/dev/null 2>&1; then
need_pull=1
log "[pull] Image not found under snapshotter=$SNAPSHOTTER, pulling: $IMAGE"
fi
if [[ "${need_pull}" == "1" ]]; then
sudo nerdctl -n "$NS" --snapshotter "$SNAPSHOTTER" pull "$IMAGE"
else
log "[pull] Inspect passed under snapshotter=$SNAPSHOTTER, skipping pull"
fi
}
run_reset() {
if [[ "${SKIP_RESET:-0}" == "1" ]]; then
log "[reset] SKIP_RESET=1, skipping reset_urunc_bench_state"
return 0
fi
log "[reset] Cleaning namespace runtime state..."
NS="$NS" SNAPSHOTTER="$SNAPSHOTTER" "$RESET_SCRIPT"
}
stats_from_times_file() {
local f=$1
if [[ ! -s "$f" ]]; then
log " (no valid samples)"
return 0
fi
awk '
NF {
v[++n] = $1 + 0
sum += v[n]
if (n == 1 || v[n] < min) min = v[n]
if (n == 1 || v[n] > max) max = v[n]
}
END {
if (n < 1) exit 0
mean = sum / n
for (i = 1; i <= n; i++) { ss += (v[i] - mean) ^ 2 }
sd = (n > 1) ? sqrt(ss / (n - 1)) : 0
printf " Sample count: %d\n", n
printf " Latencies (s):"
for (i = 1; i <= n; i++) printf " %.4f", v[i]
printf "\n"
printf " Mean (s): %.4f\n", mean
printf " Min (s): %.4f Max (s): %.4f\n", min, max
printf " Stddev (s): %.4f\n", sd
}
' "$f"
}
run_batch() {
local n=$1
local work
work="$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_TAG}-batch${n}-XXXXXX")"
log ""
log "======== Concurrency: $n ========"
local i pids=()
for ((i = 1; i <= n; i++)); do
(
set +e
local t0 t1 elapsed cid name rc
name="${NAME_PREFIX}-${SCRIPT_TAG}-${n}-${i}"
t0=$(date +%s.%N)
cid=$(sudo nerdctl -n "$NS" --snapshotter "$SNAPSHOTTER" run -d \
--name "$name" \
--runtime "$RUNTIME" \
"$IMAGE" 2>/dev/null | tail -1)
rc=$?
t1=$(date +%s.%N)
elapsed=$(awk -v a="$t0" -v b="$t1" 'BEGIN { printf "%.6f", b - a }')
if [[ $rc -eq 0 && -n "$cid" ]]; then
printf '%s\n' "$elapsed" >>"$work/times.ok"
printf '%s\n' "$cid" >>"$work/cids"
else
printf '%s nerdctl_rc=%s\n' "$elapsed" "$rc" >>"$work/failures.log"
fi
) &
pids+=($!)
done
for pid in "${pids[@]}"; do
wait "$pid" || true
done
if [[ -f "$work/times.ok" ]]; then
stats_from_times_file "$work/times.ok"
fi
if [[ -f "$work/failures.log" ]]; then
log " Failed or empty container ID records (elapsed, rc):"
sed 's/^/ /' "$work/failures.log" || true
fi
if [[ -f "$work/cids" ]]; then
cat "$work/cids" >>"$ALL_CIDS_FILE"
fi
rm -rf "$work"
}
cleanup_all() {
local f=$1
if [[ ! -f "$f" ]] || [[ ! -s "$f" ]]; then
log "[cleanup] No container IDs found, skipping"
return 0
fi
log "[cleanup] Stopping and removing containers..."
sort -u "$f" -o "$f.uniq"
while read -r id; do
[[ -z "$id" ]] && continue
sudo nerdctl -n "$NS" rm -f "$id" >/dev/null 2>&1 || true
done <"$f.uniq"
rm -f "$f.uniq"
log "[cleanup] Done"
}
ALL_CIDS_FILE=""
print_help() {
log "Usage: $0 [options] <1|10|32|64>"
log ""
log "Options:"
log " --blockfile, -B Use blockfile snapshotter"
log " --devmapper, -D Use devmapper snapshotter (default)"
log " --snapshotter, -s <name> Specify snapshotter"
log " -h, --help Show help"
log ""
log "Examples:"
log " $0 1"
log " $0 --blockfile 64"
log " SNAPSHOTTER=blockfile $0 10"
log ""
log "Env: FORCE_PULL=1 forces image pull"
}
usage() {
print_help
exit 1
}
usage_help() {
print_help
exit 0
}
main() {
while [[ $# -gt 0 ]]; do
case "$1" in
--blockfile|-B)
SNAPSHOTTER=blockfile
shift
;;
--devmapper|-D)
SNAPSHOTTER=devmapper
shift
;;
--snapshotter|-s)
SNAPSHOTTER=$2
shift 2
;;
-h|--help)
usage_help
;;
-*)
log "Error: unknown option: $1"
usage
;;
*)
break
;;
esac
done
local concurrency="${1:-}"
case "$concurrency" in
1|10|32|64) ;;
*)
log "Error: concurrency must be 1, 10, 32, or 64"
usage
;;
esac
log "== Concurrent startup benchmark =="
log "NS=$NS IMAGE=$IMAGE SNAPSHOTTER=$SNAPSHOTTER RUNTIME=$RUNTIME"
log "Concurrency: $concurrency NAME_PREFIX=$NAME_PREFIX"
run_reset
ensure_image
ALL_CIDS_FILE="$(mktemp "${TMPDIR:-/tmp}/${SCRIPT_TAG}-all-cids.XXXXXX")"
trap 'cleanup_all "$ALL_CIDS_FILE"; rm -f "$ALL_CIDS_FILE" 2>/dev/null || true' EXIT
run_batch "$concurrency"
log ""
log "======== Finished, cleaning containers ========"
cleanup_all "$ALL_CIDS_FILE"
rm -f "$ALL_CIDS_FILE"
ALL_CIDS_FILE=""
trap - EXIT
}
main "$@" |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @sidneychang it would also be nice to have the code for the view snapshot. Preferably in a repository. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , here is a brief overview of the today's meeting (02/04/2026):
|
Beta Was this translation helpful? Give feedback.
-
urunc
|
| snapshotter | N | no_view drop | with_view drop | diff (with - no) |
|---|---|---|---|---|
| devmapper | 10 | 1.663 | 1.651 | -0.012 |
| devmapper | 20 | 2.365 | 2.347 | -0.018 |
| devmapper | 30 | 3.218 | 3.351 | +0.133 |
| devmapper | 40 | 4.617 | 4.535 | -0.082 |
| devmapper | 50 | 5.823 | 5.503 | -0.320 |
| blockfile | 10 | 0.954 | 0.950 | -0.004 |
| blockfile | 20 | 2.089 | 1.998 | -0.091 |
| blockfile | 30 | 3.202 | 2.984 | -0.218 |
| blockfile | 40 | 4.209 | 3.974 | -0.235 |
| blockfile | 50 | 5.375 | 5.101 | -0.274 |
Using this view, with_view has a smaller drop in most points (except devmapper N=30), but the magnitude is modest and should be interpreted together with RSS and cleanup behavior.
Key raw data excerpts
1) Points after all N containers are started devmapper
| mode | N | mem_avail_kb | containerd_main_rss_kb | shim_urunc_rss_sum_kb | qemu_rss_sum_kb | snapshotter_root_bytes | run_containerd_bytes | snapshot_count | running_containers |
|---|---|---|---|---|---|---|---|---|---|
| no_view | 10 | 248314904 | 32508 | 53228 | 1024724 | 1179648 | 125957698 | 29 | 10 |
| with_view | 10 | 248478440 | 37868 | 143864 | 1022620 | 1179648 | 126100588 | 30 | 10 |
| no_view | 20 | 246843956 | 32508 | 118824 | 2069828 | 1179648 | 251906448 | 40 | 20 |
| with_view | 20 | 247027776 | 40620 | 288884 | 2065772 | 1179648 | 252192253 | 41 | 20 |
| no_view | 30 | 245839760 | 32508 | 164484 | 3097156 | 1179648 | 377861598 | 48 | 30 |
| with_view | 30 | 245956880 | 35684 | 410324 | 3081752 | 1179648 | 378290452 | 49 | 30 |
| no_view | 40 | 244673168 | 32508 | 233116 | 4114892 | 1179648 | 503823148 | 59 | 40 |
| with_view | 40 | 244732788 | 35832 | 547356 | 4115332 | 1179648 | 504394954 | 60 | 40 |
| no_view | 50 | 243444748 | 35268 | 283300 | 5160976 | 1179648 | 629791098 | 69 | 50 |
| with_view | 50 | 243586560 | 51908 | 691148 | 5173860 | 1179648 | 630505850 | 71 | 50 |
2) Points after all N containers are started blockfile
| mode | N | mem_avail_kb | containerd_main_rss_kb | shim_urunc_rss_sum_kb | qemu_rss_sum_kb | snapshotter_root_bytes | run_containerd_bytes | snapshot_count | running_containers |
|---|---|---|---|---|---|---|---|---|---|
| no_view | 10 | 248500376 | 32508 | 57876 | 1024656 | 15204614144 | 125957698 | 28 | 10 |
| with_view | 10 | 248340752 | 46996 | 146856 | 1026228 | 15204614144 | 126099043 | 29 | 10 |
| no_view | 20 | 247306408 | 32508 | 115700 | 2063624 | 20447494144 | 251906448 | 38 | 20 |
| with_view | 20 | 247258696 | 51788 | 276760 | 2059460 | 20447494144 | 252189173 | 39 | 20 |
| no_view | 30 | 246017552 | 32508 | 168880 | 3104948 | 25690374144 | 377861598 | 48 | 30 |
| with_view | 30 | 246077888 | 52752 | 408188 | 3102048 | 25690374144 | 378285703 | 49 | 30 |
| no_view | 40 | 245042532 | 35016 | 225292 | 4140136 | 30933254144 | 503823148 | 58 | 40 |
| with_view | 40 | 245035992 | 44232 | 553748 | 4134308 | 30933254144 | 504388633 | 59 | 40 |
| no_view | 50 | 243810548 | 39748 | 286236 | 5162880 | 36176134144 | 629791098 | 68 | 50 |
| with_view | 50 | 243843284 | 57308 | 683500 | 5158616 | 36176134144 | 630497963 | 69 | 50 |
3) Points after cleanup is completed devmapper
| mode | N | batch_elapsed_sec | mem_avail_kb | snapshotter_root_bytes | run_containerd_bytes | snapshot_count | running_containers |
|---|---|---|---|---|---|---|---|
| no_view | 10 | 11.930104 | 249229028 | 1179648 | 15348 | 18 | 0 |
| with_view | 10 | 11.910459 | 249370956 | 1179648 | 15348 | 18 | 0 |
| no_view | 20 | 24.590242 | 249013080 | 1179648 | 15348 | 18 | 0 |
| with_view | 20 | 25.210551 | 249217436 | 1179648 | 15348 | 18 | 0 |
| no_view | 30 | 38.386461 | 249256648 | 1179648 | 15348 | 18 | 0 |
| with_view | 30 | 39.363232 | 249156184 | 1179648 | 15348 | 18 | 0 |
| no_view | 40 | 53.939509 | 249309072 | 1179648 | 15348 | 18 | 0 |
| with_view | 40 | 54.497614 | 249130544 | 1179648 | 15348 | 18 | 0 |
| no_view | 50 | 69.225268 | 249261788 | 1179648 | 15348 | 18 | 0 |
| with_view | 50 | 70.900510 | 249115928 | 1179648 | 15348 | 18 | 0 |
4) Points after cleanup is completed blockfile
| mode | N | batch_elapsed_sec | mem_avail_kb | snapshotter_root_bytes | run_containerd_bytes | snapshot_count | running_containers |
|---|---|---|---|---|---|---|---|
| no_view | 10 | 32.722383 | 249421000 | 12058886144 | 15348 | 18 | 0 |
| with_view | 10 | 32.865807 | 249296476 | 12058886144 | 15348 | 18 | 0 |
| no_view | 20 | 65.979176 | 249346708 | 12058886144 | 15348 | 18 | 0 |
| with_view | 20 | 66.430903 | 249118792 | 12058886144 | 15348 | 18 | 0 |
| no_view | 30 | 101.300709 | 249364368 | 12583174144 | 15348 | 18 | 0 |
| with_view | 30 | 102.049901 | 249099836 | 12058886144 | 15348 | 18 | 0 |
| no_view | 40 | 136.507869 | 249376008 | 12058886144 | 15348 | 18 | 0 |
| with_view | 40 | 137.544523 | 249056540 | 11534598144 | 15348 | 18 | 0 |
| no_view | 50 | 172.589100 | 249345964 | 12058886144 | 15348 | 18 | 0 |
| with_view | 50 | 173.901752 | 249008724 | 12583174144 | 15348 | 18 | 0 |
Quick summary
- In this run,
viewdoes not show a stable net system-level gain. - Peak RSS is higher with
with_viewin both snapshotters. - devmapper shows small positive signals in
mem_avail, but the magnitude is limited. - Under blockfile, fixed allocation effects are very strong and can hide local view-path benefits.
- Cleanup is generally a bit slower with
with_view. - Based on code-path understanding, this currently looks like: one local copy-saving optimization offset by other overheads.
Beta Was this translation helpful? Give feedback.
-
|
There is an issue with the previous interpretation of the storage results. The From the thin-pool perspective, The detailed thin-pool comparison is as follows:
These results indicate that However, In summary, the benefit of |
Beta Was this translation helpful? Give feedback.
-
|
I investigated the memory delta between view and no-view in the
These numbers suggest the extra RSS growth is mainly tied to the shim path after introducing and executing the |
Beta Was this translation helpful? Give feedback.
-
Memory Growth Analysis: View vs No-View (100 Containers)Based on the experimental data, I think PSS provides a more accurate estimate of the actual memory overhead, as it avoids double counting of shared pages across shim processes. Figures
The second figure supports the idea that RSS is not a reliable metric for total memory usage in this setting, while PSS is more appropriate. At small |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , the RSS and PPS values are process specific and not exactly what we are trying to evaluate. We need to evaluate the storage overhead of using view snapshots vs copying files (non-view snapshot). This overhead is not specific to a process, but to the system (storage). |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , here is a brief overview of the yesterday's meeting (08/04/2026):
Also, please try to have a repository where you place all your code, so we can check it easier. |
Beta Was this translation helpful? Give feedback.
-
|
I’m currently in the process of cleaning up the related code implementation. The code still contains some noise, leftover comments, and extra log prints, which I will address later. My rough implementation is here: shared-view d68ab01#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6 Some scripts I used for testing (not yet fully organized) are here: d8a5215#diff-de5a0901e44e85d6676bd788da033bd68b0f0c09b6bcbd9346bbe908534fb39d |
Beta Was this translation helpful? Give feedback.
-
|
@cmainas Hello, I followed the second approach you suggested: instead of measuring from the host mount namespace, I first spawned the container, then entered the corresponding mount namespace and inspected the bundle from urunc’s view. What I mainly want to confirm is whether the evidence below is sufficient to support the following claim:
My current reading is the following. What I checkedI used This seems important because, as you mentioned, measuring from the host namespace is not enough for this case, and the relevant state is the one visible inside the container’s mount namespace. Current observations1. Non-view caseInside the namespace, I see:
At the same time, QEMU is launched with:
So my current interpretation is that, in the non-view path, I also noticed that 2. View caseIn the view case, on the host side I see an extra shared-view mount:
Then, inside the urunc/QEMU mount namespace, I see:
QEMU is still launched with:
So my current interpretation is that, in the view path, we are no longer copying I also noticed that Tentative conclusionBased on these observations, my current understanding is:
QuestionWould you consider this evidence strong enough to support that claim? More specifically, does the combination of:
justify saying that we have moved from “copy into tmpfs” to “reuse the existing shared view” for these artifacts? If this interpretation is correct, the next things I plan to analyze are:
Full raw commands and outputsNon-view: raw commands and outputs(base) root@sev:~/zxd/urunc# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 26G 2.7M 26G 1% /run
efivarfs 128K 23K 101K 19% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv 492G 260G 211G 56% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
tmpfs 126G 220K 126G 1% /run/qemu
/dev/sda2 2.0G 289M 1.6G 16% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 26G 44K 26G 1% /run/user/0
(base) root@sev:~/zxd/urunc# nerdctl ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
(base) root@sev:~/zxd/urunc# nerdctl -n default --snapshotter devmapper run -d --name urunc-noview-check \
--runtime io.containerd.urunc.v2 \
harbor.nbfc.io/nubificus/urunc/nginx-qemu-linux-raw:latest
103f7c493efc574cc11985a51845cda405aa8448af5c5387150b715da32ad2d9
(base) root@sev:~/zxd/urunc# nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
103f7c493efc harbor.nbfc.io/nubificus/urunc/nginx-qemu-linux-raw:latest "/urunit /usr/sbin/n…" 41 seconds ago Up urunc-noview-check
(base) root@sev:~/zxd/urunc# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 26G 5.4M 26G 1% /run
efivarfs 128K 23K 101K 19% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv 492G 260G 211G 56% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
tmpfs 126G 220K 126G 1% /run/qemu
/dev/sda2 2.0G 289M 1.6G 16% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 26G 44K 26G 1% /run/user/0
/dev/mapper/containerd-pool-snap-7625 1007M 50M 906M 6% /run/containerd/io.containerd.runtime.v2.task/default/103f7c493efc574cc11985a51845cda405aa8448af5c5387150b715da32ad2d9/rootfs
(base) root@sev:~/zxd/urunc# ps aux | grep urunc
root 1473339 0.0 0.0 1233708 4704 ? Sl 21:39 0:00 /usr/local/bin/containerd-shim-urunc-v2 -namespace default -id 103f7c493efc574cc11985a51845cda405aa8448af5c5387150b715da32ad2d9 -address /run/containerd/containerd.sock -debug
root 1473384 99.3 0.0 907672 68608 ? Ssl 21:39 0:58 /opt/urunc/bin/qemu-system-x86_64 -m 268M -L /usr/share/qemu -cpu host -enable-kvm -display none -vga none -serial stdio -monitor null -smp 1 --sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -kernel /.boot/kernel -net nic,model=virtio,macaddr=1e:63:8a:f6:17:60 -net tap,script=no,downscript=no,ifname=tap0_urunc -device virtio-blk-pci,serial=rootfs,drive=rootfs -drive format=raw,if=none,id=rootfs,file=/dev/mapper/containerd-pool-snap-7625 -initrd /urunit.conf -no-reboot -nodefaults -append panic=-1 console=ttyS0 root=/dev/vda rw ip=10.4.0.198::10.4.0.1:255.255.255.0:urunc:eth0:off retain_initrd URUNIT_CONFIG=/sys/firmware/initrd init=/urunit -- /usr/sbin/nginx -g 'daemon off;error_log stderr debug;
root 1481730 0.0 0.0 6548 1544 pts/10 S+ 21:40 0:00 grep --color=auto urunc
(base) root@sev:~/zxd/urunc# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=131322468k,nr_inodes=32830617,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26272656k,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime,stripe=64)
...
/dev/mapper/containerd-pool-snap-7625 on /run/containerd/io.containerd.runtime.v2.task/default/103f7c493efc574cc11985a51845cda405aa8448af5c5387150b715da32ad2d9/rootfs type ext2 (rw,relatime,stripe=16)
(base) root@sev:~/zxd/urunc# cntr attach 1473384
(base) root@sev:/var/lib/cntr# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 492G 260G 211G 56% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 26G 5.4M 26G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 126G 220K 126G 1% /run/qemu
tmpfs 26G 44K 26G 1% /run/user/0
/dev/mapper/containerd-pool-snap-7625 1007M 50M 906M 6% /run/containerd/io.containerd.runtime.v2.task/default/103f7c493efc574cc11985a51845cda405aa8448af5c5387150b715da32ad2d9/rootfs
efivarfs 128K 23K 101K 19% /sys/firmware/efi/efivars
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
/dev/sda2 2.0G 289M 1.6G 16% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 126G 0 126G 0% /var/lib/cntr
tmpfs 64M 0 64M 0% /var/lib/cntr/dev
tmpfs 64M 0 64M 0% /var/lib/cntr/tmp
(base) root@sev:/var/lib/cntr# mount
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime,stripe=64)
...
tmpfs on /var/lib/cntr/.boot type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26272656k,mode=755,inode64)
tmpfs on /var/lib/cntr/urunc.json type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26272656k,mode=755,inode64)
tmpfs on /var/lib/cntr/tmp type tmpfs (rw,nosuid,noexec,size=65536k,inode64)
tmpfs on /var/lib/cntr/urunit.conf type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26272656k,mode=755,inode64)View: raw commands and outputs(base) root@sev:~/zxd/urunc# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 26G 2.6M 26G 1% /run
efivarfs 128K 23K 101K 19% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv 492G 260G 211G 56% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
tmpfs 126G 220K 126G 1% /run/qemu
/dev/sda2 2.0G 289M 1.6G 16% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 26G 48K 26G 1% /run/user/0
(base) root@sev:~/zxd/urunc# nerdctl ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
(base) root@sev:~/zxd/urunc# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
...
(base) root@sev:~/zxd/urunc# nerdctl -n default --snapshotter devmapper run -d --name urunc-view-check \
--runtime io.containerd.urunc.v2 \
harbor.nbfc.io/nubificus/urunc/nginx-qemu-linux-raw:latest
4b67ced7d507e0e7debcb787bc101e48b1f4749edb8ea3af7461158bd8172254
(base) root@sev:~/zxd/urunc# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 26G 2.7M 26G 1% /run
efivarfs 128K 23K 101K 19% /sys/firmware/efi/efivars
/dev/mapper/ubuntu--vg-ubuntu--lv 492G 260G 211G 56% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
tmpfs 126G 220K 126G 1% /run/qemu
/dev/sda2 2.0G 289M 1.6G 16% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 26G 48K 26G 1% /run/user/0
/dev/mapper/containerd-pool-snap-7628 1007M 53M 903M 6% /run/urunc/shared-views/devmapper_default_sha256:69bd1c06dccfa3cf9ab836e1e11911873c36d453403dc97d34572b50365b6dc1/data
/dev/mapper/containerd-pool-snap-7627 1007M 53M 903M 6% /run/containerd/io.containerd.runtime.v2.task/default/4b67ced7d507e0e7debcb787bc101e48b1f4749edb8ea3af7461158bd8172254/rootfs
(base) root@sev:~/zxd/urunc# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
...
/dev/mapper/containerd-pool-snap-7628 on /run/urunc/shared-views/devmapper_default_sha256:69bd1c06dccfa3cf9ab836e1e11911873c36d453403dc97d34572b50365b6dc1/data type ext2 (ro,noatime,stripe=16)
/dev/mapper/containerd-pool-snap-7627 on /run/containerd/io.containerd.runtime.v2.task/default/4b67ced7d507e0e7debcb787bc101e48b1f4749edb8ea3af7461158bd8172254/rootfs type ext2 (rw,relatime,stripe=16)
(base) root@sev:~/zxd/urunc# ps aux | grep urunc
root 1493736 0.0 0.0 1240692 12576 ? Sl 21:47 0:00 /usr/local/bin/containerd-shim-urunc-v2 -namespace default -id 4b67ced7d507e0e7debcb787bc101e48b1f4749edb8ea3af7461158bd8172254 -address /run/containerd/containerd.sock -debug
root 1493806 98.5 0.0 915868 67696 ? Ssl 21:47 0:24 /opt/urunc/bin/qemu-system-x86_64 -m 268M -L /usr/share/qemu -cpu host -enable-kvm -display none -vga none -serial stdio -monitor null -smp 1 --sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -kernel /.boot/kernel -net nic,model=virtio,macaddr=8e:1f:91:1a:ae:2e -net tap,script=no,downscript=no,ifname=tap0_urunc -device virtio-blk-pci,serial=rootfs,drive=rootfs -drive format=raw,if=none,id=rootfs,file=/dev/mapper/containerd-pool-snap-7627 -initrd /urunit.conf -no-reboot -nodefaults -append panic=-1 console=ttyS0 root=/dev/vda rw ip=10.4.0.199::10.4.0.1:255.255.255.0:urunc:eth0:off retain_initrd URUNIT_CONFIG=/sys/firmware/initrd init=/urunit -- /usr/sbin/nginx -g 'daemon off;error_log stderr debug;
root 1494961 0.0 0.0 6548 1556 pts/10 S+ 21:47 0:00 grep --color=auto urunc
(base) root@sev:~/zxd/urunc# cntr attach 1493806
(base) root@sev:/var/lib/cntr# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 492G 260G 211G 56% /
tmpfs 126G 0 126G 0% /dev/shm
tmpfs 26G 2.7M 26G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 126G 220K 126G 1% /run/qemu
tmpfs 26G 48K 26G 1% /run/user/0
/dev/mapper/containerd-pool-snap-7628 1007M 53M 903M 6% /run/urunc/shared-views/devmapper_default_sha256:69bd1c06dccfa3cf9ab836e1e11911873c36d453403dc97d34572b50365b6dc1/data
/dev/mapper/containerd-pool-snap-7627 1007M 53M 903M 6% /run/containerd/io.containerd.runtime.v2.task/default/4b67ced7d507e0e7debcb787bc101e48b1f4749edb8ea3af7461158bd8172254/rootfs
efivarfs 128K 23K 101K 19% /sys/firmware/efi/efivars
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
/dev/sda2 2.0G 289M 1.6G 16% /boot
/dev/sda1 1.1G 6.2M 1.1G 1% /boot/efi
tmpfs 126G 0 126G 0% /var/lib/cntr
tmpfs 64M 0 64M 0% /var/lib/cntr/dev
tmpfs 64M 0 64M 0% /var/lib/cntr/tmp
(base) root@sev:/var/lib/cntr# mount
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime,stripe=64)
...
tmpfs on /var/lib/cntr/.boot type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26272656k,mode=755,inode64)
/dev/mapper/containerd-pool-snap-7628 on /var/lib/cntr/.boot/kernel type ext2 (ro,noatime,stripe=16)
/dev/mapper/containerd-pool-snap-7628 on /var/lib/cntr/urunc.json type ext2 (ro,noatime,stripe=16)
tmpfs on /var/lib/cntr/tmp type tmpfs (rw,nosuid,noexec,size=65536k,inode64)
tmpfs on /var/lib/cntr/urunit.conf type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26272656k,mode=755,inode64) |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , I am surprised that cntr worked without any issues. However, I noticed something that we need to be aware of. Cntr makes use of FUSE to create the view that we see in
Does this refer to the measurement form inside cntr, or from the host?
We will need to be more careful with cntr and not focus on the results of df and mount. Your observation about the growing size of
I would say that this is the case. If I understand correctly, cntr bind mounts in |
Beta Was this translation helpful? Give feedback.
-
|
Hi @cmainas
I observed the same results both inside the container and on the host.
Here is the full output I captured before and after container startup. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sidneychang , here is a brief overview of the last week's meeting (15/04/2026):
So, as next steps:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the feedback from the 15/04/2026 meeting.I ran another round of
Environment and run model
How each metric is collected, and why use it
Scope note for the dm-thin metrics The Other storage areas may include the containerd content store, containerd metadata, snapshotter metadata, runtime state under Reference: containerd config documentation: https://containerd.io/docs/1.7/man/containerd-config.toml.5/ Therefore, the conclusion supported by
Meaningful raw excerpts ( Column meanings:
Additional raw excerpt during start progression (
Interpretation
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Mentorship plan: Optimize rootfs handling with block-based snapshotters
This discussion outlines the proposed plan as discussed with @sidneychang for the CNCF mentorship project "Optimize rootfs handling with block-based snapshotters" for urunc.
The plan is structured into three phases. Each phase has clear goals and specific outcomes, along with suggested tasks and sub-tasks to help guide the work. The listed sub-tasks are meant as guidance and reference points, they are not strict requirements. The exact order of tasks within each phase can be adjusted as long as the main outcomes are achieved.
Phase 1
The goal of Phase 1 is to build the necessary background and get familiar with containerd and its snapshotters. This phase is expected to last up to 3 weeks (02/03/2026 – 22/03/2026). During this time, the focus will be on understanding contianerd, the various snapshotters, and getting familiar with snapshots.
Tentative tasks and sub-tasks
Outcomes
Deadline: Completed no later than 22/03/2026 (AoE).
Description: A report with:
Phase 2
The goal of Phase 2 is to apply the knowledge gained in Phase 1 to urunc and begin implementing the use of view snapshots for obtaining the required boot files for the monitors. This phase is expected to last up to 5 weeks (23/03/2026 – 03/05/2026). During this time, the focus will shift to the practical implementation within urunc.
Tentative tasks and sub-tasks
Outcomes:
Deadline: Completed no later than 02/05/2026 (AoE).
Description: The following items:
Phase 3
The goal of Phase 3 is to finalize and polish the integration of view snapshots for obtaining the necessary boot files for the monitors.
This phase will also include an evaluation to demonstrate the benefits of the new approach (view snapshots) compared to the previous method (mounting and copying files). This phase is expected to last up to 4 weeks (04/05/2026 – 28/05/2026), with an emphasis on refinement, validation, and broader ecosystem exploration. During this period, the focus will be on refining the implementation, validating the design decisions through evaluation, and exploring related approaches in other high level runtimes.
Tentative tasks and sub-tasks
Outcomes:
Deadline:: Completed no later than 29/05/2026 (AoE).
Description: The following items:
Beta Was this translation helpful? Give feedback.
All reactions