Skip to content

Commit 4d29fb0

Browse files
a-klosa-klos
authored
refactor: gha (#33)
This pull request introduces a major restructuring of the repository to improve organization and streamline workflows. Key changes include migrating core libraries and services into separate `libs` and `services` directories, updating references across configuration files, and enhancing CI/CD workflows to support the new structure. ### Repository Restructuring: * Removed submodules for `rag-infrastructure` and `rag-core-library` from `.gitmodules`, consolidating their contents into `libs` and `services` directories. * Updated paths in `.vscode/launch.json` to reflect the new directory structure for `libs` and `services`. [[1]](diffhunk://#diff-bd5430ee7c51dc892a67b3f2829d1f5b6d223f0fd48b82322cfd45baf9f5e945L37-R47) [[2]](diffhunk://#diff-bd5430ee7c51dc892a67b3f2829d1f5b6d223f0fd48b82322cfd45baf9f5e945L68-R78) [[3]](diffhunk://#diff-bd5430ee7c51dc892a67b3f2829d1f5b6d223f0fd48b82322cfd45baf9f5e945L99-R109) * Adjusted Python analysis paths in `.vscode/settings.json` to point to the new `libs` directory. ### Workflow Enhancements: * Replaced the `update-submodules` job with `build-and-lint-services` and `build-and-lint-libs`, introducing matrix strategies for linting and testing individual services and libraries. ### Documentation Updates: * Updated `README.md` to reflect the new directory structure, replacing references to submodules with local paths and updating setup instructions. [[1]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L3-R3) [[2]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L47-R70) [[3]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L81-R102) [[4]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L108-R131) [[5]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L179-R208) [[6]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L234-R269) ### Build and Makefile Updates: * Updated `Makefile` to use the new `libs` and `services` directories for linting, testing, and Docker builds.
1 parent 7e9bb63 commit 4d29fb0

63 files changed

Lines changed: 15141 additions & 27946 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.template‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# RAG Template Environment Variables
2+
# Copy this file to .env and fill in your actual values
3+
4+
# =============================================================================
5+
# S3 Object Storage Configuration (Required)
6+
# =============================================================================
7+
S3_ACCESS_KEY_ID=your_s3_access_key_here
8+
S3_SECRET_ACCESS_KEY=your_s3_secret_key_here
9+
10+
# =============================================================================
11+
# Basic Authentication (Required)
12+
# =============================================================================
13+
BASIC_AUTH="foo:$apr1$ryE0iE7H$F2SlPDNoFdGoaHrcla2HL/"
14+
15+
# =============================================================================
16+
# Langfuse Configuration (Required for observability)
17+
# =============================================================================
18+
LANGFUSE_PUBLIC_KEY=pk-lf-your-public-key-here
19+
LANGFUSE_SECRET_KEY=sk-lf-your-secret-key-here
20+
21+
# Langfuse Initialization (Optional - for pre-creating organization/project)
22+
LANGFUSE_INIT_ORG_ID="my-org"
23+
LANGFUSE_INIT_PROJECT_ID="my-project"
24+
LANGFUSE_INIT_PROJECT_PUBLIC_KEY=pk-lf-your-public-key-here
25+
LANGFUSE_INIT_PROJECT_SECRET_KEY=sk-lf-your-secret-key-here
26+
LANGFUSE_INIT_USER_EMAIL="user@stackit.cloud"
27+
LANGFUSE_INIT_USER_NAME="stackiteer"
28+
LANGFUSE_INIT_USER_PASSWORD="stackit123"
29+
30+
# =============================================================================
31+
# Frontend Authentication (Required)
32+
# =============================================================================
33+
VITE_AUTH_USERNAME=foo
34+
VITE_AUTH_PASSWORD=bar
35+
36+
# =============================================================================
37+
# LLM Provider API Keys (Choose one or more)
38+
# =============================================================================
39+
40+
# STACKIT AI Model Serving as provider (Recommended)
41+
# A guide how to get your API key by using the STACKIT portal UI: https://docs.stackit.cloud/stackit/en/getting-started-with-the-stackit-portal-ui-319914591.html
42+
# A guide how to get your API key by using the Product API: https://docs.stackit.cloud/stackit/en/getting-started-with-the-product-api-319914605.html
43+
STACKIT_VLLM_API_KEY=your-stackit-vllm-api-key
44+
STACKIT_EMBEDDER_API_KEY=your-stackit-embedder-api-key
45+
46+
# OpenAI (For RAGAS evaluation - Optional)
47+
RAGAS_OPENAI_API_KEY=sk-your-openai-api-key-here
48+
49+
# =============================================================================
50+
# Confluence Integration (Optional)
51+
# =============================================================================
52+
CONFLUENCE_TOKEN="your-confluence-api-token,your-confluence-api-token"
53+
CONFLUENCE_URL="https://your-company.atlassian.net,https://your-company.atlassian.net"
54+
CONFLUENCE_SPACE_KEY="YOUR_SPACE_KEY_1,YOUR_SPACE_KEY_2"
55+
CONFLUENCE_VERIFY_SSL="True,True"
56+
CONFLUENCE_DOCUMENT_NAME="document1,document2"
57+
58+
# =============================================================================
59+
# Additional Notes:
60+
# =============================================================================
61+
# 1. Required variables must be set for the system to work properly
62+
# 2. S3-compatible storage is required for document storage
63+
# 3. Choose either STACKIT or configure Ollama for LLM providers
64+
# 4. Confluence integration is optional but enables document ingestion from Confluence
65+
# 5. For production deployment, see infrastructure/README.md for additional configuration

‎.github/workflows/lint-and-test.yml‎

Lines changed: 182 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,48 @@
11
name: lint-and-test
22

33
on:
4-
pull_request:
5-
branches:
4+
pull_request:
5+
branches:
66
- main
7-
workflow_dispatch:
7+
workflow_dispatch:
8+
9+
env:
10+
NODE_VERSION: '18'
11+
PYTHON_VERSION: '3.11'
812

913
jobs:
14+
changes:
15+
name: Detect Changes
16+
runs-on: ubuntu-latest
17+
outputs:
18+
services: ${{ steps.changes.outputs.services }}
19+
libs: ${{ steps.changes.outputs.libs }}
20+
frontend: ${{ steps.changes.outputs.frontend }}
21+
infrastructure: ${{ steps.changes.outputs.infrastructure }}
22+
steps:
23+
- name: Checkout
24+
uses: actions/checkout@v4
25+
with:
26+
fetch-depth: 0
27+
28+
- name: Detect changes
29+
id: changes
30+
uses: dorny/paths-filter@v2
31+
with:
32+
filters: |
33+
services:
34+
- 'services/admin-backend/**'
35+
- 'services/document-extractor/**'
36+
- 'services/rag-backend/**'
37+
- 'services/mcp-server/**'
38+
libs:
39+
- 'libs/**'
40+
frontend:
41+
- 'services/frontend/**'
42+
infrastructure:
43+
- 'infrastructure/**'
44+
- 'Tiltfile'
45+
1046
sanitize-branch-name:
1147
runs-on: ubuntu-latest
1248
outputs:
@@ -15,76 +51,176 @@ jobs:
1551
- name: sanitize-branch-name
1652
id: sanitize
1753
run: |
18-
SANITIZED_REF=$(echo "${GITHUB_HEAD_REF}" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9' '-')
54+
SANITIZED_REF=$(echo "${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9' '-')
1955
SANITIZED_REF=${SANITIZED_REF#-}
2056
SANITIZED_REF=${SANITIZED_REF%-}
2157
SANITIZED_REF=${SANITIZED_REF:0:63}
2258
if [[ -z "$SANITIZED_REF" || "$SANITIZED_REF" =~ ^-+$ ]]; then
2359
SANITIZED_REF="tmp-branch"
2460
fi
25-
echo "::set-output name=sanitized_ref::${SANITIZED_REF}"
61+
echo "sanitized_ref=${SANITIZED_REF}" >> $GITHUB_OUTPUT
2662
shell: bash
27-
env:
28-
GITHUB_HEAD_REF: ${{ github.head_ref }}
2963

30-
update-submodules:
31-
name: update-submodules
64+
build-and-test-services:
65+
name: Build and Test Services
3266
runs-on: ubuntu-latest
33-
permissions:
34-
contents: write
67+
needs: [changes, sanitize-branch-name]
68+
if: needs.changes.outputs.services == 'true' || github.event_name == 'workflow_dispatch'
69+
strategy:
70+
fail-fast: false
71+
matrix:
72+
service: ["admin-backend", "document-extractor", "rag-backend", "mcp-server"]
3573
steps:
36-
- name: checkout
74+
- name: Checkout code
3775
uses: actions/checkout@v4
38-
with:
39-
fetch-depth: 0
40-
ref: ${{ github.event.pull_request.head.ref }}
41-
submodules: false
4276

43-
- name: update-submodules
44-
id: update-submodules
77+
- name: Set Docker Image Name
4578
run: |
46-
git config --global url."https://github.com/".insteadOf "git@github.com:"
47-
git submodule sync --recursive
48-
git submodule update --init --remote --recursive
49-
50-
if [ -n "$(git status --porcelain)" ]; then
51-
echo "Submodules have new commits. Committing and pushing..."
52-
git config user.name "github-actions[bot]"
53-
git config user.email "github-actions[bot]@users.noreply.github.com"
54-
git commit -am "chore: update submodules to latest main"
55-
git push
56-
else
57-
echo "No submodule changes."
58-
fi
79+
echo "IMAGE_NAME=${{ matrix.service }}:${{ needs.sanitize-branch-name.outputs.sanitized_ref }}-${{ github.run_number }}" >> $GITHUB_ENV
80+
shell: bash
81+
82+
- name: Build Docker image
83+
run: |
84+
docker build -t $IMAGE_NAME --build-arg dev=1 -f services/${{ matrix.service }}/Dockerfile .
85+
86+
- name: Run linting
87+
run: |
88+
docker run --rm $IMAGE_NAME make lint
89+
90+
- name: Run tests
91+
run: |
92+
docker run --rm $IMAGE_NAME make test
5993
60-
build-and-lint:
61-
name: Build and Lint
94+
build-and-test-libs:
95+
name: Build and Test Libraries
6296
runs-on: ubuntu-latest
63-
needs: [sanitize-branch-name, update-submodules]
97+
needs: [changes, sanitize-branch-name]
98+
if: needs.changes.outputs.libs == 'true' || github.event_name == 'workflow_dispatch'
6499
strategy:
65-
fail-fast: true
66-
matrix:
67-
service: [ "admin-backend", "document-extractor", "rag-backend"]
100+
fail-fast: false
101+
matrix:
102+
library: ["rag-core-lib", "rag-core-api", "admin-api-lib", "extractor-api-lib"]
68103
steps:
69104
- name: Checkout code
70105
uses: actions/checkout@v4
71-
with:
72-
ref: ${{ github.event.pull_request.head.ref }}
73-
submodules: true
74106

75-
- name: Set Docker Image Name
107+
- name: Set Docker Image Names
76108
run: |
77-
echo "IMAGE_NAME=${{ matrix.service }}:${{ needs.sanitize-branch-name.outputs.sanitized_ref }}-${{ github.run_number }}" >> $GITHUB_ENV
109+
echo "LINT_IMAGE_NAME=${{ matrix.library }}-lint:${{ needs.sanitize-branch-name.outputs.sanitized_ref }}-${{ github.run_number }}" >> $GITHUB_ENV
110+
echo "TEST_IMAGE_NAME=${{ matrix.library }}-test:${{ needs.sanitize-branch-name.outputs.sanitized_ref }}-${{ github.run_number }}" >> $GITHUB_ENV
78111
shell: bash
79112

80113
- name: Build lint image
81114
run: |
82-
docker build -t $IMAGE_NAME --build-arg dev=1 --build-arg TAG=debug -f ${{ matrix.service }}/Dockerfile .
115+
docker build -t $LINT_IMAGE_NAME --build-arg TEST=0 -f libs/Dockerfile libs
83116
84-
- name: Generate lint report
117+
- name: Run linting
85118
run: |
86-
docker run --rm --entrypoint make "$IMAGE_NAME" lint
119+
docker run --rm $LINT_IMAGE_NAME make lint
120+
121+
- name: Build test image
122+
run: |
123+
docker build -t $TEST_IMAGE_NAME --build-arg DIRECTORY=${{ matrix.library }} --build-arg TEST=1 -f libs/Dockerfile libs
87124
88125
- name: Run tests
89126
run: |
90-
docker run --rm --entrypoint make "$IMAGE_NAME" test
127+
docker run --rm $TEST_IMAGE_NAME make test
128+
129+
build-and-test-frontend:
130+
name: Build and Test Frontend
131+
runs-on: ubuntu-latest
132+
needs: [changes]
133+
if: needs.changes.outputs.frontend == 'true' || github.event_name == 'workflow_dispatch'
134+
steps:
135+
- name: Checkout code
136+
uses: actions/checkout@v4
137+
138+
- name: Setup Node.js
139+
uses: actions/setup-node@v4
140+
with:
141+
node-version: ${{ env.NODE_VERSION }}
142+
cache: 'npm'
143+
cache-dependency-path: services/frontend/package-lock.json
144+
145+
- name: Install dependencies
146+
working-directory: services/frontend
147+
run: npm ci
148+
149+
- name: Run linting
150+
working-directory: services/frontend
151+
run: npm run eslint
152+
153+
- name: Run tests
154+
working-directory: services/frontend
155+
run: npm run test
156+
157+
- name: Build chat app
158+
working-directory: services/frontend
159+
run: npm run chat:build
160+
161+
- name: Build admin app
162+
working-directory: services/frontend
163+
run: npm run admin:build
164+
165+
- name: Upload build artifacts
166+
uses: actions/upload-artifact@v4
167+
with:
168+
name: frontend-build-artifacts
169+
path: |
170+
services/frontend/apps/*/dist/
171+
retention-days: 7
172+
173+
validate-infrastructure:
174+
name: Validate Infrastructure
175+
runs-on: ubuntu-latest
176+
needs: [changes]
177+
if: needs.changes.outputs.infrastructure == 'true' || github.event_name == 'workflow_dispatch'
178+
steps:
179+
- name: Checkout code
180+
uses: actions/checkout@v4
181+
182+
- name: Setup Terraform
183+
uses: hashicorp/setup-terraform@v3
184+
with:
185+
terraform_version: "~1.0"
186+
187+
- name: Terraform Format Check
188+
working-directory: infrastructure/terraform
189+
run: terraform fmt -check -recursive
190+
191+
- name: Terraform Init
192+
working-directory: infrastructure/terraform
193+
run: terraform init -backend=false
194+
195+
- name: Terraform Validate
196+
working-directory: infrastructure/terraform
197+
run: terraform validate
198+
199+
- name: Validate Helm Charts
200+
run: |
201+
curl https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz | tar xz
202+
sudo mv linux-amd64/helm /usr/local/bin/helm
203+
helm lint infrastructure/rag/
204+
205+
summary:
206+
name: CI Summary
207+
runs-on: ubuntu-latest
208+
needs: [build-and-test-services, build-and-test-libs, build-and-test-frontend, validate-infrastructure]
209+
if: always()
210+
steps:
211+
- name: Check results
212+
run: |
213+
echo "Services: ${{ needs.build-and-test-services.result }}"
214+
echo "Libraries: ${{ needs.build-and-test-libs.result }}"
215+
echo "Frontend: ${{ needs.build-and-test-frontend.result }}"
216+
echo "Infrastructure: ${{ needs.validate-infrastructure.result }}"
217+
218+
if [[ "${{ needs.build-and-test-services.result }}" == "failure" ]] || \
219+
[[ "${{ needs.build-and-test-libs.result }}" == "failure" ]] || \
220+
[[ "${{ needs.build-and-test-frontend.result }}" == "failure" ]] || \
221+
[[ "${{ needs.validate-infrastructure.result }}" == "failure" ]]; then
222+
echo "❌ One or more jobs failed"
223+
exit 1
224+
else
225+
echo "✅ All jobs passed or were skipped"
226+
fi

0 commit comments

Comments
 (0)