Tests, Scripts, GET impl

docker/README.md

@@ -10,7 +10,7 @@ docker/
 │   ├── docker-compose.dsv.yml                    # App only
 │   ├── docker-compose.dsv-redis.yml              # App + Redis
 │   ├── docker-compose.dsv-redis-kafka.yml        # App + Redis + Kafka
-│   └── docker-compose.dsv-redis-kafka-3nodes.yml # Three DSV app instances
+│   └── docker-compose.dsv-redis-kafka-3nodes.yml # Three DSV app instances + per-node Redis
 ├── redis/
 │   ├── docker-compose.redis.yml                  # Redis only
 │   └── redis.conf                                # Redis persistence and security config
@@ -54,15 +54,15 @@ The API listens on `http://localhost:8080`.
 
 ## Three App Nodes
 
-For local cluster-like testing, run three DSV app instances against the same Redis and Kafka services:
+For local cluster-like testing, run three DSV app instances against shared Kafka and one Redis service per app node:
 
 ```bash
 ./mvnw clean package -DskipTests
 mkdir -p target/dependency && (cd target/dependency && jar -xf ../*.jar)
 docker compose -f docker/dsv/docker-compose.dsv-redis-kafka-3nodes.yml up -d --build
 ```
 
-Apps listen on `8081`, `8082`, and `8083`. Each instance sets a different `NODE_NAME` so Kafka consumer groups differ and every node receives `secrets-commit` messages.
+Apps listen on `8081`, `8082`, and `8083`. Redis instances for those nodes are published on `6381`, `6382`, and `6383`. Each app points at its own Redis service and sets a different `NODE_NAME` so Kafka consumer groups differ and every node receives `secrets-commit` messages.
 
 Automated check:
 
@@ -81,7 +81,7 @@ docker logs dsv-app-3 2>&1 | grep -i "Received commit" | tail -3
 
 ## Services
 
-`redis` stores secret shards durably with AOF persistence and password auth.
+`redis` stores secret shards durably with AOF persistence and password auth. In the three-node stack this is split into `redis1`, `redis2`, and `redis3`, one per app node.
 
 `kafka` provides commit fanout and ordering infrastructure in KRaft mode.
 
@@ -108,4 +108,4 @@ docker compose -f docker/dsv/docker-compose.dsv-redis-kafka.yml down
 docker compose -f docker/dsv/docker-compose.dsv-redis-kafka.yml down -v
 ```
 
-All services communicate on the `dsv-network` bridge network. The app connects to Redis as `redis` and Kafka as `kafka:29092`.
+All services communicate on the `dsv-network` bridge network. Single-app stacks connect to Redis as `redis`; the three-node stack connects `app1`, `app2`, and `app3` to `redis1`, `redis2`, and `redis3`. Kafka is reachable as `kafka:29092`.

docker/dsv/docker-compose.dsv-redis-kafka-3nodes.yml

@@ -1,16 +1,56 @@
-# Same stack as docker-compose.dsv-redis-kafka.yml but with three DSV app instances.
+# Three DSV app instances with one Redis instance per app node.
 # HTTP: localhost:8081 (app1), :8082 (app2), :8083 (app3)
-name: "Distributed Secrets Vault + Redis + Kafka"
+name: "Distributed Secrets Vault + Per-Node Redis + Kafka"
 
 services:
-  redis:
+  redis1:
     image: redis:8.6-alpine
-    container_name: dsv-redis
+    container_name: dsv-redis-1
     ports:
-      - "6379:6379"
+      - "6381:6379"
+    environment:
+      - REDIS_PASSWORD=${REDIS_PASSWORD:-REDIS_PASSWORD}
+    command: redis-server /usr/local/etc/redis/redis.conf --requirepass ${REDIS_PASSWORD:-REDIS_PASSWORD}
+    volumes:
+      - redis-data-1:/data
+      - ../redis/redis.conf:/usr/local/etc/redis/redis.conf:ro
+    healthcheck:
+      test: ["CMD-SHELL", "redis-cli -a \"$${REDIS_PASSWORD:-REDIS_PASSWORD}\" ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+    networks:
+      - dsv-network
+
+  redis2:
+    image: redis:8.6-alpine
+    container_name: dsv-redis-2
+    ports:
+      - "6382:6379"
+    environment:
+      - REDIS_PASSWORD=${REDIS_PASSWORD:-REDIS_PASSWORD}
+    command: redis-server /usr/local/etc/redis/redis.conf --requirepass ${REDIS_PASSWORD:-REDIS_PASSWORD}
+    volumes:
+      - redis-data-2:/data
+      - ../redis/redis.conf:/usr/local/etc/redis/redis.conf:ro
+    healthcheck:
+      test: ["CMD-SHELL", "redis-cli -a \"$${REDIS_PASSWORD:-REDIS_PASSWORD}\" ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+    networks:
+      - dsv-network
+
+  redis3:
+    image: redis:8.6-alpine
+    container_name: dsv-redis-3
+    ports:
+      - "6383:6379"
+    environment:
+      - REDIS_PASSWORD=${REDIS_PASSWORD:-REDIS_PASSWORD}
     command: redis-server /usr/local/etc/redis/redis.conf --requirepass ${REDIS_PASSWORD:-REDIS_PASSWORD}
     volumes:
-      - redis-data:/var/lib/redis
+      - redis-data-3:/data
       - ../redis/redis.conf:/usr/local/etc/redis/redis.conf:ro
     healthcheck:
       test: ["CMD-SHELL", "redis-cli -a \"$${REDIS_PASSWORD:-REDIS_PASSWORD}\" ping"]
@@ -63,14 +103,19 @@ services:
       - "8081:8080"
     environment:
       - NODE_NAME=node-1
-      - SPRING_DATA_REDIS_HOST=redis
+      - POD_IP=app1
+      - SERVER_PORT=8080
+      - CLUSTER_PORT=4801
+      - SEED_DNS_HOST=app1
+      - SEED_DNS_PORT=4801
+      - SPRING_DATA_REDIS_HOST=redis1
       - SPRING_DATA_REDIS_PORT=6379
       - SPRING_DATA_REDIS_PASSWORD=${REDIS_PASSWORD:-REDIS_PASSWORD}
       - SPRING_PROFILES_ACTIVE=${SPRING_PROFILES_ACTIVE:-dev}
       - KAFKA_BOOTSTRAP_SERVERS=kafka:29092
-      - SEED_DNS_HOST=localhost
+      - JAVA_TOOL_OPTIONS=-Dcluster.totalNodes=3 -Dcluster.thresholdK=2 -Dcluster.quorumM=2
     depends_on:
-      redis:
+      redis1:
         condition: service_healthy
       kafka:
         condition: service_healthy
@@ -86,14 +131,19 @@ services:
       - "8082:8080"
     environment:
       - NODE_NAME=node-2
-      - SPRING_DATA_REDIS_HOST=redis
+      - POD_IP=app2
+      - SERVER_PORT=8080
+      - CLUSTER_PORT=4801
+      - SEED_DNS_HOST=app1
+      - SEED_DNS_PORT=4801
+      - SPRING_DATA_REDIS_HOST=redis2
       - SPRING_DATA_REDIS_PORT=6379
       - SPRING_DATA_REDIS_PASSWORD=${REDIS_PASSWORD:-REDIS_PASSWORD}
       - SPRING_PROFILES_ACTIVE=${SPRING_PROFILES_ACTIVE:-dev}
       - KAFKA_BOOTSTRAP_SERVERS=kafka:29092
-      - SEED_DNS_HOST=localhost
+      - JAVA_TOOL_OPTIONS=-Dcluster.totalNodes=3 -Dcluster.thresholdK=2 -Dcluster.quorumM=2
     depends_on:
-      redis:
+      redis2:
         condition: service_healthy
       kafka:
         condition: service_healthy
@@ -109,22 +159,29 @@ services:
       - "8083:8080"
     environment:
       - NODE_NAME=node-3
-      - SPRING_DATA_REDIS_HOST=redis
+      - POD_IP=app3
+      - SERVER_PORT=8080
+      - CLUSTER_PORT=4801
+      - SEED_DNS_HOST=app1
+      - SEED_DNS_PORT=4801
+      - SPRING_DATA_REDIS_HOST=redis3
       - SPRING_DATA_REDIS_PORT=6379
       - SPRING_DATA_REDIS_PASSWORD=${REDIS_PASSWORD:-REDIS_PASSWORD}
       - SPRING_PROFILES_ACTIVE=${SPRING_PROFILES_ACTIVE:-dev}
       - KAFKA_BOOTSTRAP_SERVERS=kafka:29092
-      - SEED_DNS_HOST=localhost
+      - JAVA_TOOL_OPTIONS=-Dcluster.totalNodes=3 -Dcluster.thresholdK=2 -Dcluster.quorumM=2
     depends_on:
-      redis:
+      redis3:
         condition: service_healthy
       kafka:
         condition: service_healthy
     networks:
       - dsv-network
 
 volumes:
-  redis-data:
+  redis-data-1:
+  redis-data-2:
+  redis-data-3:
   kafka-data:
 
 networks:

docs/demo-cluster-lifecycle.md

@@ -0,0 +1,87 @@
+# Local Cluster Lifecycle Demo
+
+This demo runs a local Docker-based Distributed Secrets Vault cluster and exercises the full secret lifecycle across multiple clients and nodes. The script builds the Spring Boot application, generates a temporary Docker Compose file at `target/dsv-demo/docker-compose.demo.yml`, starts Kafka plus one Redis service per app node, runs the demo checks, and then tears the stack down.
+
+## What It Demonstrates
+
+The default run starts a 3-node cluster:
+
+- `app1` on `http://127.0.0.1:8081` with `redis1`
+- `app2` on `http://127.0.0.1:8082` with `redis2`
+- `app3` on `http://127.0.0.1:8083` with `redis3`
+- Kafka for commit fanout and cluster coordination
+
+The demo validates:
+
+- secret creation, latest reads, updates, version reads, all-version reads, and deletion
+- multiple users storing the same secret name without leaking values across users
+- parallel clients creating secrets against different app nodes
+- continued quorum operation when one node and its Redis service are stopped
+- node reboot and recovery after the stopped app and Redis service restart
+- read rejection when fewer than `K` shards are online
+- write rejection when fewer than `M` nodes are available for quorum
+
+With the defaults, the script uses `NODE_COUNT=3`, `THRESHOLD_K=2`, and `QUORUM_M=2`.
+
+## Prerequisites
+
+Install and start:
+
+- Docker with Docker Compose
+- Java and the Maven wrapper requirements used by the project
+- `curl`
+- `jar`, unless running with `SKIP_BUILD=1`
+
+From the project root, create a local environment file if you do not already have one:
+
+```bash
+cp .env.example .env
+```
+
+The demo defaults to `REDIS_PASSWORD=REDIS_PASSWORD`, matching the local development configuration.
+
+## Run The Demo
+
+From the repository root:
+
+```bash
+./scripts/demo-cluster-lifecycle.sh
+```
+
+The first run may take longer because Docker needs to pull Redis, Kafka, and Java base images.
+
+## Useful Run Modes
+
+Customize the quorum and reconstruction threshold:
+
+```bash
+NODE_COUNT=5 THRESHOLD_K=3 QUORUM_M=3 ./scripts/demo-cluster-lifecycle.sh
+```
+
+Move the published ports if the defaults are already in use:
+
+```bash
+BASE_PORT=9081 REDIS_BASE_PORT=7381 KAFKA_HOST_PORT=29092 ./scripts/demo-cluster-lifecycle.sh
+```
+
+## Configuration Reference
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `NODE_COUNT` | `3` | Number of app nodes and Redis instances to run. Must be between `3` and `10`. |
+| `BASE_PORT` | `8081` | Host port for app node 1. Node N uses `BASE_PORT + N - 1`. |
+| `REDIS_BASE_PORT` | `6381` | Host port for Redis node 1. Redis N uses `REDIS_BASE_PORT + N - 1`. |
+| `KAFKA_HOST_PORT` | `19092` | Host port published for Kafka. |
+| `THRESHOLD_K` | majority | Number of Shamir shards required to reconstruct a secret. |
+| `QUORUM_M` | majority | Number of write acknowledgements required to commit. Must be greater than or equal to `THRESHOLD_K`. |
+| `REDIS_PASSWORD` | `REDIS_PASSWORD` | Password configured for every local Redis instance. |
+| `SPRING_PROFILES_ACTIVE` | `dev` | Spring profile used by the app containers. |
+| `KEEP_STACK` | `0` | Set to `1` to leave containers running after the demo. |
+| `SKIP_BUILD` | `0` | Set to `1` to skip Maven packaging and reuse `target/dependency`. |
+| `PROJECT_NAME` | `dsv-demo` | Docker Compose project and container name prefix. |
+
+## Troubleshooting
+
+If Docker reports that a port is already allocated, rerun with different `BASE_PORT`, `REDIS_BASE_PORT`, or `KAFKA_HOST_PORT` values.
+
+If the script cannot connect to the Docker daemon, make sure Docker Desktop or the Docker service is running and that your shell has permission to access the Docker socket.

docs/docker.md

@@ -24,7 +24,7 @@ The application will be available at:
 | `docker/dsv/docker-compose.dsv.yml` | App only |
 | `docker/dsv/docker-compose.dsv-redis.yml` | App + Redis |
 | `docker/dsv/docker-compose.dsv-redis-kafka.yml` | App + Redis + Kafka |
-| `docker/dsv/docker-compose.dsv-redis-kafka-3nodes.yml` | Three app nodes + Redis + Kafka |
+| `docker/dsv/docker-compose.dsv-redis-kafka-3nodes.yml` | Three app nodes + per-node Redis + Kafka |
 | `docker/redis/docker-compose.redis.yml` | Redis only |
 | `docker/kafka/docker-compose.kafka.yml` | Kafka only |
 
@@ -50,6 +50,12 @@ The apps listen on:
 - `http://127.0.0.1:8082`
 - `http://127.0.0.1:8083`
 
+Each app in the three-node stack has its own Redis service, matching the Kubernetes sidecar model:
+
+- app1 -> redis1, published at `localhost:6381`
+- app2 -> redis2, published at `localhost:6382`
+- app3 -> redis3, published at `localhost:6383`
+
 ## Configuration
 
 Configuration is loaded from `.env` in the project root when you run Docker Compose from the project root: