Skip to content

Commit 44471d1

Browse files
keIIy-kimjk-kim0claude
authored
feat(app): add shared object storage foundation (#752)
## 배경 `app` / `common` 레이어에서 공통 object storage foundation이 필요해서, storage 계약과 provider adapter를 먼저 올리는 PR입니다. 이번 PR은 foundation만 다룹니다. - 공통 storage contract - provider별 adapter wiring - blob backing store - compression / smoke / 회귀 테스트 - backend reference 문서 상위 asset metadata 모델과 서비스별 attachment 통합은 의도적으로 제외했습니다. ## 이번 PR에서 포함한 범위 - `common` - `ObjectStoragePort` - `StorageUri` - `StoreObjectRequest`, `OpenDownloadRequest`, `DownloadHandle` - `app` - `S3CompatibleObjectStorageAdapter` - `MinioObjectStorageAdapter` - `TencentCosObjectStorageAdapter` - `BlobObjectStorageAdapter` - `ObjectStorageRegistry` - blob backing store migration / entity / repository - 문서 - `docs/reference/backend/object-storage.md` - 관련 backend index 갱신 - MinIO smoke / compression 작업 메모 추가 ## 주요 변경 - `storage_uri`는 public URL이 아니라 internal locator로만 사용합니다. - backend alias는 논리 이름 `minio` / `tencent-cos` / `blob`으로 정리했습니다. - S3-compatible adapter는 metadata round-trip을 보장합니다. - MinIO는 path-style access를 강제합니다. - blob backend는 bucket-qualified target을 reject합니다. - blob download도 `filename` / `disposition` 계약을 맞췄습니다. - `Content-Disposition`은 filename이 없어도 `attachment` / `inline`을 유지하고, filename이 있으면 ASCII fallback + `filename*` UTF-8 encoding을 사용합니다. - oversized blob 업로드는 configurable limit로 방어합니다. - `blob`는 텍스트 계열 payload에 대해 선택적 `GZIP` compression을 지원합니다. - `S3CompatibleObjectStorageAdapter.head()`는 provider 편차를 고려해 header / metadata / range / list fallback을 사용합니다. - env-gated `MinIO` smoke test를 추가했습니다. ## MinIO smoke 메모 - 이 개발기에서는 `39000` 포트를 `ZscalerTunnel`이 이미 점유하고 있었습니다. - 그래서 `http://127.0.0.1:39000`은 MinIO가 아니라 `Proxy-Agent: Ztunnel/1.0` 응답이 나왔습니다. - 실제 smoke는 `39100:9000`으로 MinIO를 띄워서 검증했습니다. - smoke test는 endpoint env를 그대로 사용하므로 특정 포트 고정에 의존하지 않습니다. ## 설정/기본값 - local profile 기본 provider: `minio` - dev / ci 기본 provider: `blob` - prod profile 기본 provider: `blob` - `Tencent COS` adapter는 wiring과 테스트까지 포함하지만, 운영 기본값 전환은 별도 판단으로 남겨뒀습니다. ## 리뷰 반영 이번 PR에서 리뷰로 반영한 핵심은 아래입니다. - blob adapter의 `readBytes()` 제거 및 max size guard 추가 - `BlobStoredObjectEntity.id` non-nullable 정리 - S3 metadata round-trip 반영 - MinIO path-style wiring 강제 및 회귀 테스트 추가 - blob bucket-qualified target reject - `Content-Disposition` filename 없는 케이스까지 보강 - backend alias를 `minio` / `tencent-cos` / `blob`으로 정리 - 실제 MinIO smoke에서 드러난 port 하드코딩 assertion 제거 ## 검증 아래 focused suite로 확인했습니다. ```bash RUN_MINIO_SMOKE_TEST=true \ MINIO_SMOKE_TEST_ENDPOINT=http://127.0.0.1:39100 \ MINIO_SMOKE_TEST_REGION=us-east-1 \ MINIO_SMOKE_TEST_ACCESS_KEY=minio \ MINIO_SMOKE_TEST_SECRET_KEY=minio123 \ MINIO_SMOKE_TEST_BUCKET=deck-smoke \ ./gradlew --no-build-cache :common:test --tests "io.deck.common.api.storage.StorageUriTest" \ :app:test --tests "io.deck.app.storage.blob.BlobObjectStorageAdapterTest" \ --tests "io.deck.app.storage.s3.S3CompatibleObjectStorageAdapterTest" \ --tests "io.deck.app.storage.minio.MinioObjectStorageSmokeTest" ``` 또한 fresh DB에서 `app` migration만 지정해 blob backing store migration을 검증했습니다. ## 주의 사항 - `:app` Gradle Flyway task는 기본적으로 `db/migration/ci-seed/V9999__ci-seed.sql`까지 함께 스캔합니다. - 이번 fresh migration 검증은 `-Dflyway.locations=filesystem:src/main/resources/db/migration/app`로 `app` migration만 지정해서 확인했습니다. ## 이번 PR에서 제외한 범위 - `StoredAsset` 공통 자산 메타데이터 모델 - `PUBLIC` / `PRIVATE` asset serving 정책 - `meetpie og_image`의 object storage 이전 - `meetpie` namecard 내부 이미지의 persisted `imageUrl/data:image` 제거 - `deskpie` note/file attachment 통합 ## 후속 설계 상위 자산 모델과 서비스 통합 방향은 별도 이슈로 분리했습니다. - #756 - #756 --------- Co-authored-by: JK <jk@chequer.io> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 9b21b02 commit 44471d1

36 files changed

Lines changed: 2724 additions & 1 deletion

‎backend/app/build.gradle.kts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,10 @@ dependencies {
5858
// Cache
5959
implementation(libs.caffeine)
6060

61+
// AWS SDK
62+
implementation(platform(libs.aws.bom))
63+
implementation(libs.aws.s3)
64+
6165
// Database
6266
runtimeOnly(libs.postgresql)
6367
implementation("org.springframework.boot:spring-boot-starter-flyway")

‎backend/app/src/main/kotlin/io/deck/app/config/ObjectStorageConfig.kt‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
package io.deck.app.config
2+
3+
import io.deck.app.storage.ObjectStorageRegistry
4+
import io.deck.app.storage.blob.BlobObjectStorageAdapter
5+
import io.deck.app.storage.blob.BlobStoredObjectPayloadRepository
6+
import io.deck.app.storage.blob.BlobStoredObjectRepository
7+
import io.deck.app.storage.cos.TencentCosObjectStorageAdapter
8+
import io.deck.app.storage.minio.MinioObjectStorageAdapter
9+
import io.deck.common.api.config.ObjectStorageProperties
10+
import io.deck.common.api.storage.ObjectStoragePort
11+
import org.springframework.beans.factory.ObjectProvider
12+
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty
13+
import org.springframework.boot.context.properties.EnableConfigurationProperties
14+
import org.springframework.context.annotation.Bean
15+
import org.springframework.context.annotation.Configuration
16+
import org.springframework.context.annotation.Primary
17+
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
18+
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
19+
import software.amazon.awssdk.regions.Region
20+
import software.amazon.awssdk.services.s3.S3Client
21+
import software.amazon.awssdk.services.s3.S3Configuration
22+
import software.amazon.awssdk.services.s3.presigner.S3Presigner
23+
import java.net.URI
24+
25+
@Configuration(proxyBeanMethods = false)
26+
@EnableConfigurationProperties(ObjectStorageProperties::class)
27+
class ObjectStorageConfig {
28+
@Bean(name = ["minioS3Client"], destroyMethod = "close")
29+
@ConditionalOnProperty(
30+
prefix = "app.object-storage.minio",
31+
name = ["endpoint", "region", "access-key", "secret-key"],
32+
)
33+
fun minioS3Client(properties: ObjectStorageProperties): S3Client =
34+
buildS3Client(
35+
endpoint = properties.minio.endpoint,
36+
region = properties.minio.region,
37+
accessKey = properties.minio.accessKey,
38+
secretKey = properties.minio.secretKey,
39+
pathStyleAccess = true,
40+
)
41+
42+
@Bean(name = ["minioS3Presigner"], destroyMethod = "close")
43+
@ConditionalOnProperty(
44+
prefix = "app.object-storage.minio",
45+
name = ["endpoint", "region", "access-key", "secret-key"],
46+
)
47+
fun minioS3Presigner(properties: ObjectStorageProperties): S3Presigner =
48+
buildS3Presigner(
49+
endpoint = properties.minio.endpoint,
50+
region = properties.minio.region,
51+
accessKey = properties.minio.accessKey,
52+
secretKey = properties.minio.secretKey,
53+
pathStyleAccess = true,
54+
)
55+
56+
@Bean(name = ["tencentCosS3Client"], destroyMethod = "close")
57+
@ConditionalOnProperty(
58+
prefix = "app.object-storage.cos",
59+
name = ["endpoint", "region", "secret-id", "secret-key"],
60+
)
61+
fun tencentCosS3Client(properties: ObjectStorageProperties): S3Client =
62+
buildS3Client(
63+
endpoint = properties.cos.endpoint,
64+
region = properties.cos.region,
65+
accessKey = properties.cos.secretId,
66+
secretKey = properties.cos.secretKey,
67+
pathStyleAccess = properties.cos.pathStyleAccess,
68+
)
69+
70+
@Bean(name = ["tencentCosS3Presigner"], destroyMethod = "close")
71+
@ConditionalOnProperty(
72+
prefix = "app.object-storage.cos",
73+
name = ["endpoint", "region", "secret-id", "secret-key"],
74+
)
75+
fun tencentCosS3Presigner(properties: ObjectStorageProperties): S3Presigner =
76+
buildS3Presigner(
77+
endpoint = properties.cos.endpoint,
78+
region = properties.cos.region,
79+
accessKey = properties.cos.secretId,
80+
secretKey = properties.cos.secretKey,
81+
pathStyleAccess = properties.cos.pathStyleAccess,
82+
)
83+
84+
@Bean(name = ["minioObjectStoragePort"])
85+
@ConditionalOnProperty(
86+
prefix = "app.object-storage.minio",
87+
name = ["endpoint", "region", "access-key", "secret-key"],
88+
)
89+
fun minioObjectStoragePort(
90+
properties: ObjectStorageProperties,
91+
minioS3Client: S3Client,
92+
minioS3Presigner: S3Presigner,
93+
): MinioObjectStorageAdapter =
94+
MinioObjectStorageAdapter(
95+
backend = MINIO_BACKEND,
96+
bucket = properties.bucket,
97+
properties = properties.minio,
98+
s3Client = minioS3Client,
99+
presigner = minioS3Presigner,
100+
)
101+
102+
@Bean(name = ["tencentCosObjectStoragePort"])
103+
@ConditionalOnProperty(
104+
prefix = "app.object-storage.cos",
105+
name = ["endpoint", "region", "secret-id", "secret-key"],
106+
)
107+
fun tencentCosObjectStoragePort(
108+
properties: ObjectStorageProperties,
109+
tencentCosS3Client: S3Client,
110+
tencentCosS3Presigner: S3Presigner,
111+
): TencentCosObjectStorageAdapter =
112+
TencentCosObjectStorageAdapter(
113+
backend = TENCENT_COS_BACKEND,
114+
bucket = properties.bucket,
115+
properties = properties.cos,
116+
s3Client = tencentCosS3Client,
117+
presigner = tencentCosS3Presigner,
118+
)
119+
120+
@Bean(name = ["blobObjectStoragePort"])
121+
fun blobObjectStoragePort(
122+
properties: ObjectStorageProperties,
123+
objectRepository: BlobStoredObjectRepository,
124+
payloadRepository: BlobStoredObjectPayloadRepository,
125+
): BlobObjectStorageAdapter =
126+
BlobObjectStorageAdapter(
127+
backend = BLOB_BACKEND,
128+
objectRepository = objectRepository,
129+
payloadRepository = payloadRepository,
130+
maxObjectSizeBytes = properties.blob.maxObjectSizeBytes,
131+
)
132+
133+
@Bean
134+
fun objectStorageRegistry(
135+
minioObjectStoragePort: ObjectProvider<MinioObjectStorageAdapter>,
136+
tencentCosObjectStoragePort: ObjectProvider<TencentCosObjectStorageAdapter>,
137+
blobObjectStoragePort: BlobObjectStorageAdapter,
138+
): ObjectStorageRegistry =
139+
ObjectStorageRegistry(
140+
adaptersByBackend = buildMap {
141+
minioObjectStoragePort.getIfAvailable()?.let { put(MINIO_BACKEND, it) }
142+
tencentCosObjectStoragePort.getIfAvailable()?.let { put(TENCENT_COS_BACKEND, it) }
143+
put(BLOB_BACKEND, blobObjectStoragePort)
144+
},
145+
)
146+
147+
@Bean
148+
@Primary
149+
fun objectStoragePort(
150+
properties: ObjectStorageProperties,
151+
minioObjectStoragePort: ObjectProvider<MinioObjectStorageAdapter>,
152+
tencentCosObjectStoragePort: ObjectProvider<TencentCosObjectStorageAdapter>,
153+
blobObjectStoragePort: BlobObjectStorageAdapter,
154+
): ObjectStoragePort =
155+
when (properties.provider) {
156+
ObjectStorageProperties.Provider.MINIO -> {
157+
minioObjectStoragePort.getIfAvailable()
158+
?: throw IllegalStateException("MinIO object storage is not configured")
159+
}
160+
161+
ObjectStorageProperties.Provider.TENCENT_COS -> {
162+
tencentCosObjectStoragePort.getIfAvailable()
163+
?: throw IllegalStateException("Tencent COS object storage is not configured")
164+
}
165+
166+
ObjectStorageProperties.Provider.BLOB -> {
167+
blobObjectStoragePort
168+
}
169+
}
170+
171+
private fun buildS3Client(
172+
endpoint: String,
173+
region: String,
174+
accessKey: String,
175+
secretKey: String,
176+
pathStyleAccess: Boolean,
177+
): S3Client =
178+
S3Client
179+
.builder()
180+
.endpointOverride(URI.create(endpoint))
181+
.region(Region.of(region))
182+
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)))
183+
.serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(pathStyleAccess).build())
184+
.build()
185+
186+
private fun buildS3Presigner(
187+
endpoint: String,
188+
region: String,
189+
accessKey: String,
190+
secretKey: String,
191+
pathStyleAccess: Boolean,
192+
): S3Presigner =
193+
S3Presigner
194+
.builder()
195+
.endpointOverride(URI.create(endpoint))
196+
.region(Region.of(region))
197+
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)))
198+
.serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(pathStyleAccess).build())
199+
.build()
200+
201+
private companion object {
202+
const val MINIO_BACKEND = "minio"
203+
const val TENCENT_COS_BACKEND = "tencent-cos"
204+
const val BLOB_BACKEND = "blob"
205+
}
206+
}

‎backend/app/src/main/kotlin/io/deck/app/storage/ObjectStorageRegistry.kt‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
package io.deck.app.storage
2+
3+
import io.deck.common.api.storage.ObjectStoragePort
4+
import io.deck.common.api.storage.StorageUri
5+
6+
class ObjectStorageRegistry(
7+
private val adaptersByBackend: Map<String, ObjectStoragePort>,
8+
) {
9+
fun find(storageUri: StorageUri): ObjectStoragePort =
10+
adaptersByBackend[storageUri.backend]
11+
?: throw IllegalArgumentException("Unsupported storage backend: ${storageUri.backend}")
12+
}

0 commit comments

Comments
 (0)