Conversation
bwalsh
force-pushed
the
feature/dynamic-buckets
branch
from
4db6cdc to
14365b0
Compare
github now rendering ```mermaid !
| **Flow (validate):** | ||
| AuthN → Admin RBAC → resolve secret/assume role → optional HEAD to storage → return `{ok, reason}` (no secrets). | ||
|
|
||
| ## 3) Data Model (Shared DB) |
There was a problem hiding this comment.
We should not have a shared db. We need to abstract microservices from each other with APIs as the only coupling.
|
|
||
| **Separation of concerns** | ||
| - `presigned-urls`: *read-only* DB, read secrets or assume-role, generate presigned URLs. No DB writes. | ||
| - `secret-admin`: CRUD + validate; never returns raw secrets; enforces admin RBAC. |
There was a problem hiding this comment.
The raw secrets are required for signing URLs
| This package contains a focused **architecture + rollout plan** to split signing from admin into two services: | ||
|
|
||
| - **`presigned-urls`** — high-throughput signer (download/upload single + multipart), read-only DB + secrets access. | ||
| - **`secret-admin`** — low-QPS (queries per second) admin CRUD + validation for the bucket registry, write access to DB; validate-only access to secrets/STS. |
There was a problem hiding this comment.
We will need this service to be able to be configured as included or excluded, b/c we will likely not utilize it. We'd like to still have the option of the current state: configure creds explicitly for the presigned-urls service
|
|
||
| ### A) `presigned-urls` | ||
| **Endpoints (Fence-compatible):** | ||
| - `GET /data/download/{guid}` → signed GET |
There was a problem hiding this comment.
should also include /drs/{{guid}}/access endpoints
| * **`FENCE_DYNAMIC_BUCKETS=true`** | ||
|
|
||
| * Split service deployment. Fence continues to handle auth flows and GA4GH endpoints. | ||
| * Presign and bucket-admin endpoints are served by new services. |
There was a problem hiding this comment.
GA4GH should be included in presigned. presigned needs a config flag on whether or not to rely on bucket-admin or just look for creds in configuration
motivation |
|
|
||
| * **`FENCE_DYNAMIC_BUCKETS=false` (default)** | ||
|
|
||
| * Fence behaves as today: all endpoints are served by the Fence service itself. |
There was a problem hiding this comment.
we should consider a clean split, at least for AWS. Have PRs to remove AWS signed URL logic from Fence and add new service(s) that handle AWS, GCP. The new presigned url service should handle GCP without all the crazy group management we have currently in Fence. We don't need that anymore. We should model the AWS approach in GCP in the new service. e.g. one admin service account signing URLs.
|
As a next step can you please fill out this community feature doc? https://docs.google.com/document/d/1P2dfqnSH-e7OX1Hw62sDL8zcR7gZp4d152TlDBlomDc/edit?tab=t.0#heading=h.5e0lej9k5tiv Please make a copy, and provide with your PR either as a shareable link to a completed version. Sharing as a Google doc will be helpful to allow for comments and discussion between you and the Gen3 team. |
3 participants
Overview: https://github.com/calypr/fence/blob/feature/dynamic-buckets/docs/fence-buckets/docs/index.md
New Features
S3_BUCKETSYAML lookups at runtime).FENCE_ENABLE_DYNAMIC_BUCKETSwith YAML fallback for backward compatibility.Breaking Changes
Bug Fixes
Improvements
Dependency updates
Optional runtime deps when corresponding backend is used:
boto3(for AWS STS/S3 clients in examples)kubernetes(for Kubernetes Secret API mode)No mandatory dependency changes for existing (YAML‑only) deployments.
Deployment changes
New DB objects:
bucket,bucket_prefix(optional),bucket_policy_binding, and audit table (optional).App config/env:
FENCE_ENABLE_DYNAMIC_BUCKETS=trueto activateFENCE_SECRETS_BACKEND=aws_sm|k8s(etc.)FENCE_BUCKET_CACHE_TTL_SECONDS,FENCE_SECRET_CACHE_TTL_SECONDSAuthZ/IAM:
sts:AssumeRolefor configured ARNs.Helm/K8s:
Migration:
fence-create sync-buckets-from-yaml) before enabling the flag.