Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .just/security.just
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Security scanning with Trivy (https://trivy.dev)

trivy_image := "aquasec/trivy:latest"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Trivy container image latest tag best practices CI/CD reproducibility

💡 Result:

Recommendation: don’t use :latest for Trivy in CI/CD if you care about reproducibility

latest (and other floating tags) can be retagged to different image contents over time, which breaks determinism. Docker explicitly recommends pinning by digest when you need fully repeatable builds/scans. [1]

Best practices for reproducible Trivy container-image scanning

  1. Pin the Trivy tool version (and ideally the image digest)

    • Prefer an explicit Trivy release version over latest.
    • If you use GitHub Actions, pin the action and the Trivy version it installs (the action supports version: latest or version: vX.Y.Z; don’t use latest for reproducibility). [2]
    • Avoid “canary” images/builds for production pipelines. [3]
  2. Control the vulnerability database lifecycle (this is a major source of “same image, different results”)

    • Trivy’s vulnerability DB is updated frequently (published every ~6 hours), so scan results can change even when the target image is unchanged. [4]
    • For reproducible runs, cache the DB between CI jobs and/or runs (Trivy explicitly calls out CI caching benefits). [4]
    • For “frozen” results (e.g., audits), run with --skip-db-update to prevent pulling a newer DB during the scan. [4]
  3. If you need strict repeatability across networks/regions, self-host or mirror the Trivy DB

    • Trivy distributes DBs as OCI artifacts (notably via GHCR), and its “air-gapped/offline” guidance is designed around controlling that dependency. [5]
    • In locked-down enterprise CI, mirror/self-host the DB and point Trivy at your controlled source (so the exact DB used is explicit and reviewable). [5]
  4. Separate “reproducible security gate” from “continuously updated security signal”

    • Common pattern:
      • PR gate: pinned Trivy version + cached/frozen DB (stable pass/fail).
      • Scheduled job (daily): same Trivy version (or intentionally updated), DB updates enabled to detect newly disclosed CVEs.

Sources: Docker pinning/digest guidance for determinism [1]; Trivy Action supports explicit Trivy version pinning [2]; Trivy warns about canary builds [3]; Trivy DB update frequency + skip-update + caching [4]; Trivy DB distribution via GHCR / offline control concepts [5].


🏁 Script executed:

# Examine the .just/security.just file to understand context
cat -n .just/security.just

Repository: evstack/ev-node

Length of output: 1566


🏁 Script executed:

# Check if there are other references to trivy_image or container images in the project
rg "trivy_image|docker|container|image" --type-list | head -20
fd -t f "*.just" -exec rg "trivy_image" {} +

Repository: evstack/ev-node

Length of output: 693


Pin Trivy version and consider database update strategy.

Line 3 uses :latest, which allows the Trivy vulnerability database to update automatically (~every 6 hours), causing identical image scans to produce different results. Pin an explicit Trivy version (and optionally the image digest) for reproducible security scans.

The project already caches the Trivy database via trivy_cache_volume, which helps. For further reproducibility, consider:

  • Pinning to a specific Trivy release (e.g., aquasec/trivy:0.48.0) or digest
  • Using --skip-db-update in PR/gate scans to freeze results; enable DB updates only in scheduled/continuous jobs
🔧 Suggested fix
-trivy_image := "aquasec/trivy:latest"
+trivy_image := env("TRIVY_IMAGE", "aquasec/trivy:0.48.0")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
trivy_image := "aquasec/trivy:latest"
trivy_image := env("TRIVY_IMAGE", "aquasec/trivy:0.48.0")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.just/security.just at line 3, The Trivy image is pinned to "latest" via the
trivy_image variable which allows non-reproducible scans; change trivy_image to
a specific release tag or digest (e.g., aquasec/trivy:0.48.0 or the
image@sha256:...) to lock the tool version, and update any scan invocations
(where Trivy is run) to optionally include --skip-db-update for PR/gate scans
while reserving DB updates for scheduled jobs; update the trivy_image variable
and adjust the scan invocation logic that references trivy_image to implement
these changes.

trivy_severity := env("TRIVY_SEVERITY", "CRITICAL,HIGH")
trivy_cache_volume := "trivy-cache"
scan_images := env("SCAN_IMAGES", "evstack:local-dev")

trivy_run := "docker run --rm -v " + trivy_cache_volume + ":/root/.cache/ -e TRIVY_SEVERITY=" + trivy_severity

# Run all Trivy security scans (filesystem + Docker images)
trivy-scan: trivy-scan-fs trivy-scan-image

# Scan repo for dependency vulnerabilities, misconfigs, and secrets
trivy-scan-fs:
@echo "--> Scanning repository filesystem with Trivy"
@{{trivy_run}} \
-v {{justfile_directory()}}:/workspace \
{{trivy_image}} \
fs --scanners vuln,misconfig,secret \
--severity {{trivy_severity}} \
/workspace
@echo "--> Filesystem scan complete"

# Scan built Docker images for vulnerabilities
trivy-scan-image:
@echo "--> Scanning Docker images with Trivy"
@for img in {{scan_images}}; do \
echo "--> Scanning image: $img"; \
{{trivy_run}} \
-v /var/run/docker.sock:/var/run/docker.sock \
{{trivy_image}} \
image --severity {{trivy_severity}} \
$img; \
done
Comment on lines +11 to +34
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Trivy exit code vulnerabilities found documentation

💡 Result:

Trivy’s default behavior is to exit with code 0 even if vulnerabilities (or other security issues) are found. To make a CI job fail when findings exist, set a non‑zero exit code via --exit-code (or the config/action equivalent). (trivy.dev)

CLI (Trivy)

  • --exit-code <n>: “Exit code when vulnerabilities were found (default: 0).” (trivy.dev)
  • Common pattern (fail build if matching findings are present):
    • trivy image --severity HIGH,CRITICAL --exit-code 1 <image>
    • trivy fs --severity HIGH,CRITICAL --exit-code 1 <path> (trivy.dev)

GitHub Action (aquasecurity/trivy-action)

  • exit-code input: “Exit code when specified vulnerabilities are found” (default 0). (github.com)

Citations:


Make scans fail on findings.

The current implementation lacks explicit failure behavior. By default, Trivy exits with code 0 even when vulnerabilities are found. Add --exit-code 1 (or a configurable equivalent) to both trivy-scan-fs and trivy-scan-image so that CI jobs fail when security issues are detected.

🔧 Suggested fix
 trivy_severity := env("TRIVY_SEVERITY", "CRITICAL,HIGH")
+trivy_exit_code := env("TRIVY_EXIT_CODE", "1")
@@
         fs --scanners vuln,misconfig,secret \
+        --exit-code {{trivy_exit_code}} \
         --severity {{trivy_severity}} \
         /workspace
@@
             {{trivy_image}} \
             image --severity {{trivy_severity}} \
+            --exit-code {{trivy_exit_code}} \
             $img; \
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.just/security.just around lines 11 - 34, The Trivy targets trivy-scan-fs
and trivy-scan-image currently do not fail CI when findings exist; update both
command invocations (the fs command inside target trivy-scan-fs and the image
command inside target trivy-scan-image) to include an exit code flag (e.g., add
--exit-code 1) so Trivy returns non-zero on findings; optionally expose a
variable (e.g., trivy_exit_code with default 1) and use {{trivy_exit_code}} in
both invocations so the behavior is configurable while ensuring CI fails on
detected issues.

@echo "--> Image scan complete"
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changes

- Store pending blocks separately from executed blocks key. [#3073](https://github.com/evstack/ev-node/pull/3073)
- **BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as non-root user `ev-node` (uid 1000) instead of `root`. Existing volumes or bind mounts with root-owned files may require a `chown` to uid 1000. See the [migration guide](https://ev.xyz/guides/migrate-docker-nonroot).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add PR reference for changelog traceability.

Line 27 should include the PR link like neighboring entries.

🔧 Suggested fix
-- **BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as non-root user `ev-node` (uid 1000) instead of `root`. Existing volumes or bind mounts with root-owned files may require a `chown` to uid 1000. See the [migration guide](https://ev.xyz/guides/migrate-docker-nonroot).
+- **BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as non-root user `ev-node` (uid 1000) instead of `root`. Existing volumes or bind mounts with root-owned files may require a `chown` to uid 1000. See the [migration guide](https://ev.xyz/guides/migrate-docker-nonroot). [`#3082`](https://github.com/evstack/ev-node/pull/3082)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- **BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as non-root user `ev-node` (uid 1000) instead of `root`. Existing volumes or bind mounts with root-owned files may require a `chown` to uid 1000. See the [migration guide](https://ev.xyz/guides/migrate-docker-nonroot).
- **BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as non-root user `ev-node` (uid 1000) instead of `root`. Existing volumes or bind mounts with root-owned files may require a `chown` to uid 1000. See the [migration guide](https://ev.xyz/guides/migrate-docker-nonroot). [`#3082`](https://github.com/evstack/ev-node/pull/3082)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@CHANGELOG.md` at line 27, Update the changelog entry that starts with
"**BREAKING:** Docker images for `evm`, `testapp`, and `local-da` now run as
non-root user `ev-node` (uid 1000)..." to append the PR reference (e.g.,
"(`#1234`)") consistent with neighboring entries; edit the line to include the PR
link or number so the entry matches the format used elsewhere for traceability.

- Fixes issues with force inclusion verification on sync nodes. [#3057](https://github.com/evstack/ev-node/pull/3057)
- Add flag to `local-da` to produce empty DA blocks (closer to the real system). [#3057](https://github.com/evstack/ev-node/pull/3057)
- Validate P2P DA height hints against the latest known DA height to prevent malicious peers from triggering runaway catchup . [#3128](https://github.com/evstack/ev-node/pull/3128)
Expand Down
10 changes: 8 additions & 2 deletions apps/evm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,16 @@ FROM alpine:3.22.2
#hadolint ignore=DL3018
RUN apk --no-cache add ca-certificates curl

WORKDIR /root
RUN addgroup -g 1000 ev-node && \
adduser -u 1000 -G ev-node -s /bin/sh -D ev-node
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The adduser command uses the -D flag, which prevents the creation of a home directory. However, the subsequent WORKDIR /home/ev-node implies that /home/ev-node is intended to be the user's home directory. For consistency with apps/testapp/Dockerfile and clearer intent, it's better to allow adduser to create the home directory by removing the -D flag, or explicitly create it if -D is strictly necessary for other reasons. Removing -D is the most straightforward approach to align with the WORKDIR and chown commands.

    adduser -u 1000 -G ev-node -s /bin/sh ev-node


WORKDIR /home/ev-node

COPY --from=build-env /src/apps/evm/evm /usr/bin/evm
COPY apps/evm/entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh && \
chown -R ev-node:ev-node /home/ev-node

USER ev-node
Comment on lines +23 to +30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify entrypoint depends on HOME and Dockerfile state
echo "=== Checking entrypoint.sh for HOME usage ==="
rg -n '\$HOME|--home|\.evm' apps/evm/entrypoint.sh
echo ""
echo "=== Checking Dockerfile for WORKDIR, USER, ENV HOME ==="
rg -n 'WORKDIR|USER|ENV HOME' apps/evm/Dockerfile

Repository: evstack/ev-node

Length of output: 571


Set HOME explicitly before USER ev-node.

The entrypoint.sh defaults to $HOME/.evm (line 8) when the --home flag is not provided. The Dockerfile switches to non-root user but does not set the HOME environment variable. Without ENV HOME=/home/ev-node, the HOME variable may still resolve to /root at runtime, causing the process to attempt writes to /root/.evm and fail with permission errors.

🔧 Suggested fix
 WORKDIR /home/ev-node
+ENV HOME=/home/ev-node
 
 COPY --from=build-env /src/apps/evm/evm /usr/bin/evm
 COPY apps/evm/entrypoint.sh /usr/bin/entrypoint.sh
 RUN chmod +x /usr/bin/entrypoint.sh && \
     chown -R ev-node:ev-node /home/ev-node
 
 USER ev-node
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
WORKDIR /home/ev-node
COPY --from=build-env /src/apps/evm/evm /usr/bin/evm
COPY apps/evm/entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh && \
chown -R ev-node:ev-node /home/ev-node
USER ev-node
WORKDIR /home/ev-node
ENV HOME=/home/ev-node
COPY --from=build-env /src/apps/evm/evm /usr/bin/evm
COPY apps/evm/entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod +x /usr/bin/entrypoint.sh && \
chown -R ev-node:ev-node /home/ev-node
USER ev-node
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@apps/evm/Dockerfile` around lines 23 - 30, The Dockerfile switches to the
non-root user (USER ev-node) but never sets HOME, so entrypoint.sh (which
defaults to $HOME/.evm) may still resolve to /root; add an ENV
HOME=/home/ev-node before the USER ev-node instruction so $HOME points to
/home/ev-node at runtime, and keep the existing WORKDIR /home/ev-node and
chown/chmod steps to ensure ownership/permissions for entrypoint.sh and the home
directory.


ENTRYPOINT ["/usr/bin/entrypoint.sh"]
8 changes: 7 additions & 1 deletion apps/testapp/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FROM golang:1.25 AS base

#hadolint ignore=DL3018
RUN apt-get update && \

Check failure on line 4 in apps/testapp/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint

DL3008 warning: Pin versions in apt get install. Instead of `apt-get install <package>` use `apt-get install <package>=<version>`
apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
Expand All @@ -21,14 +21,20 @@
COPY . .

# Now download dependencies and build
RUN go mod download && cd apps/testapp && go install .

Check failure on line 24 in apps/testapp/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint

DL3062 warning: Pin versions in go. Instead of `go install <package>` use `go install <package>@<version>`

Check failure on line 24 in apps/testapp/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint

DL3003 warning: Use WORKDIR to switch to a directory

## prep the final image.
#
FROM base

RUN groupadd -g 1000 ev-node && \
useradd -u 1000 -g ev-node -s /bin/sh -m ev-node

COPY --from=builder /go/bin/testapp /usr/bin

WORKDIR /apps
WORKDIR /home/ev-node
RUN chown -R ev-node:ev-node /home/ev-node

USER ev-node

ENTRYPOINT ["testapp"]
4 changes: 4 additions & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,10 @@ function sidebarHome() {
text: "Migrating to ev-abci",
link: "/guides/migrating-to-ev-abci",
},
{
text: "Migrate Docker to non-root",
link: "/guides/migrate-docker-nonroot",
},
{
text: "Create genesis for your chain",
link: "/guides/create-genesis",
Expand Down
112 changes: 112 additions & 0 deletions docs/guides/migrate-docker-nonroot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Migrating Docker Containers to Non-Root User

Starting with this release, the `evm`, `testapp`, and `local-da` Docker images run as a non-root user (`ev-node`, uid/gid 1000) instead of `root`. This aligns with the `grpc` image, which already ran as non-root.

If you are running any of these containers with **persistent volumes or bind mounts**, you need to fix file ownership before upgrading. Containers running without persistent storage (ephemeral) require no action.

## Who is affected

You are affected if **all** of the following are true:

- You run `evm`, `testapp`, or `local-da` via Docker (or docker-compose / Kubernetes)
- You use a volume or bind mount for the container's data directory
- The files in that volume were created by a previous (root-based) image

## Migration steps

### 1. Stop the running container

```bash
docker stop <container-name>
```

### 2. Fix file ownership on the volume

For **bind mounts** (host directory), run `chown` directly on the host:

```bash
# Replace /path/to/data with your actual data directory
sudo chown -R 1000:1000 /path/to/data
```

For **named Docker volumes**, use a temporary container:

```bash
# Replace <volume-name> with your Docker volume name
docker run --rm -v <volume-name>:/data alpine chown -R 1000:1000 /data
```

### 3. Pull the new image and restart

```bash
docker pull <image>
docker start <container-name>
```
Comment on lines +39 to +44
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

docker pull image docker start container does it use new image automatically

💡 Result:

No.

  • docker pull repo:tag only updates your local image (it may move the repo:tag label to a new image ID). Existing containers keep pointing to the image ID they were created from, and that never changes. (forums.docker.com)
  • docker start <container> starts an already-created container, so it cannot switch to a newer image. (stackoverflow.com)

To use the newly pulled image, you must recreate the container (e.g., docker rm <container> then docker run ... repo:tag), or force pulling when creating a new container with docker run --pull=always .... (docs.docker.com)

Citations:


Step 3 must recreate the container to use the newly pulled image.

The current step pulls a new image but then runs docker start on the existing container, which continues to use the old image. Containers retain references to the specific image ID they were created with; pulling only updates the local tag label. To apply the new image, the container must be recreated with docker rm <container-name> followed by docker run (or equivalent docker-compose recreate), otherwise the migration appears successful but the old image remains active.

🔧 Suggested fix
 ### 3. Pull the new image and restart
 
 ```bash
 docker pull <image>
-docker start <container-name>
+# Recreate container to use the newly pulled image
+docker rm <container-name>
+docker run ... <image> ...
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @docs/guides/migrate-docker-nonroot.md around lines 39 - 44, The step
currently pulls the new image but incorrectly uses docker start on the existing
container (so the old image stays in use); replace the docker start
instruction with commands that remove and recreate the
container so it runs the new image (e.g., docker rm followed by
docker run ... ...), or instruct users to use docker-compose recreate /
docker-compose up -d --force-recreate when using compose; ensure the docs
explicitly state that the container must be recreated to pick up the new image.


</details>

<!-- fingerprinting:phantom:poseidon:hawk -->

<!-- This is an auto-generated comment by CodeRabbit -->


### Kubernetes / docker-compose

If you manage containers through orchestration, you have two options:

**Option A: Init container (recommended for Kubernetes)**

Add an init container that fixes ownership before the main container starts:

```yaml
initContainers:
- name: fix-permissions
image: alpine:3.22
command: ["chown", "-R", "1000:1000", "/home/ev-node"]
volumeMounts:
- name: data
mountPath: /home/ev-node
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
```

**Option B: Set `fsGroup` in the pod security context**

If your volume driver supports it, setting `fsGroup: 1000` will automatically fix ownership on mount:

```yaml
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
```

**docker-compose**: update your `docker-compose.yml` to set the user:

```yaml
services:
evm:
image: evm:latest
user: "1000:1000"
volumes:
- evm-data:/home/ev-node
```

## Verifying the migration

After restarting, confirm the container runs as the correct user:

```bash
docker exec <container-name> id
# Expected: uid=1000(ev-node) gid=1000(ev-node)
```

Check that the process can read and write its data directory:

```bash
docker exec <container-name> ls -la /home/ev-node
# All files should be owned by ev-node:ev-node
```

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `Permission denied` on startup | Volume files still owned by root | Re-run the `chown` step above |
| Container exits immediately | Data directory not writable | Check ownership and directory permissions (`drwxr-xr-x` or more permissive for uid 1000) |
| Application writes to wrong path | Old `WORKDIR` was `/root` or `/apps` | Update any custom volume mounts to target `/home/ev-node` instead |
1 change: 1 addition & 0 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import '.just/lint.just'
import '.just/codegen.just'
import '.just/run.just'
import '.just/tools.just'
import '.just/security.just'

# List available recipes when running `just` with no args
default:
Expand Down
8 changes: 7 additions & 1 deletion tools/local-da/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,22 @@

COPY . .

RUN mkdir -p build && cd tools/local-da && go build -o /src/build/local-da .

Check failure on line 13 in tools/local-da/Dockerfile

View workflow job for this annotation

GitHub Actions / lint / hadolint

DL3003 warning: Use WORKDIR to switch to a directory

FROM alpine:3.22.2

#hadolint ignore=DL3018
RUN apk --no-cache add ca-certificates curl

WORKDIR /root
RUN addgroup -g 1000 ev-node && \
adduser -u 1000 -G ev-node -s /bin/sh -D ev-node

WORKDIR /home/ev-node
RUN chown -R ev-node:ev-node /home/ev-node

COPY --from=build-env /src/build/local-da /usr/bin/local-da

USER ev-node

ENTRYPOINT ["/usr/bin/local-da"]
CMD ["-listen-all"]
Loading