From b767b702f7041487ae7425a396d3b7a6fe47ea9e Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Sat, 5 Jul 2025 18:40:19 +0800 Subject: [PATCH 1/9] Create cmake-single-platform.yml Here's a `cmake-single-platform.yml` GitHub Actions workflow that builds and tests a CMake project on Linux: ```yaml name: CMake Build and Test on: push: branches: [ main, master ] pull_request: branches: [ main, master ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v4 - name: Install dependencies run: | sudo apt-get update sudo apt-get install -y build-essential cmake - name: Create build directory run: cmake -E make_directory build - name: Configure CMake working-directory: build run: cmake .. -DCMAKE_BUILD_TYPE=Release - name: Build project working-directory: build run: cmake --build . --config Release --parallel $(nproc) - name: Run tests working-directory: build run: ctest -C Release --output-on-failure ``` ### Key Features: 1. **Triggers**: - Runs on push/PR to main/master branches 2. **Environment**: - Uses latest Ubuntu runner 3. **Steps**: - Installs essential build tools and CMake - Creates a separate build directory - Configures CMake in Release mode - Builds with parallel compilation (uses all available cores) - Runs tests with detailed failure output 4. **Optimizations**: - Parallel builds using `$(nproc)` to detect available CPU cores - Separate build directory keeps source tree clean - Output-on-failure for test debugging ### Customization Options: 1. **Add Dependencies**: ```yaml - name: Install custom dependencies run: sudo apt-get install -y libboost-dev libopencv-dev ``` 2. **CMake Configuration Flags**: ```yaml run: cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON ``` 3. **Cross-Platform** (modify `runs-on`): ```yaml # For Windows runs-on: windows-latest # For macOS runs-on: macos-latest ``` 4. **Caching Dependencies** (add before build step): ```yaml - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.cache key: ${{ runner.os }}-deps-${{ hashFiles('**/CMakeLists.txt') }} ``` 5. **Artifact Upload** (add after build): ```yaml - name: Upload build artifacts uses: actions/upload-artifact@v3 with: name: build-output path: build/ ``` To use this workflow: 1. Create `.github/workflows/cmake-single-platform.yml` in your repo 2. Commit and push to trigger the workflow 3. Monitor executions in GitHub Actions tab The workflow will automatically: - Build Release configuration - Run all registered CTest targets - Fail fast on compilation errors or test failures - Provide detailed build logs for debugging --- .github/workflows/cmake-single-platform.yml | 39 +++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 .github/workflows/cmake-single-platform.yml diff --git a/.github/workflows/cmake-single-platform.yml b/.github/workflows/cmake-single-platform.yml new file mode 100644 index 000000000000..4bd1c79b4868 --- /dev/null +++ b/.github/workflows/cmake-single-platform.yml @@ -0,0 +1,39 @@ +# This starter workflow is for a CMake project running on a single platform. There is a different starter workflow if you need cross-platform coverage. +# See: https://github.com/actions/starter-workflows/blob/main/ci/cmake-multi-platform.yml +name: CMake on a single platform + +on: + push: + branches: [ "master" ] + pull_request: + branches: [ "master" ] + +env: + # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.) + BUILD_TYPE: Release + +jobs: + build: + # The CMake configure and build commands are platform agnostic and should work equally well on Windows or Mac. + # You can convert this to a matrix build if you need cross-platform coverage. + # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Configure CMake + # Configure CMake in a 'build' subdirectory. `CMAKE_BUILD_TYPE` is only required if you are using a single-configuration generator such as make. + # See https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html?highlight=cmake_build_type + run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} + + - name: Build + # Build your program with the given configuration + run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}} + + - name: Test + working-directory: ${{github.workspace}}/build + # Execute tests defined by the CMake configuration. + # See https://cmake.org/cmake/help/latest/manual/ctest.1.html for more detail + run: ctest -C ${{env.BUILD_TYPE}} + From 80a087da9997cfc62b7b5ad814ce150fdd2b5bf6 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Fri, 11 Jul 2025 20:50:13 +0800 Subject: [PATCH 2/9] Update ci.yml Based on the CI badge you've shared, here's how to create a similar status badge for your updated SLSA Level 3 publish workflow: ### 1. Create the SLSA Publish Workflow Badge Add this markdown to your README.md: ```markdown [![SLSA Publish](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml/badge.svg?branch=master)](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` ### 2. Key Parameters Explained: | Parameter | Value | Purpose | |--------------------|-------------------------------------------------|------------------------------------------| | Workflow filename | `generator-generic-ossf-slsa3-publish.yml` | Matches your updated workflow file | | Branch | `branch=master` | Shows status for master branch | | Event | (Automatic - no need to specify) | Uses default release published trigger | ### 3. For Workflow Dispatch Trigger: If you want to add manual trigger status (like your CI badge), modify to: ```markdown [![SLSA Publish](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml/badge.svg?branch=master&event=workflow_dispatch)](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` Note the added `&event=workflow_dispatch` parameter ### 4. Badge Appearance Options: - **Passing**: ![](https://img.shields.io/badge/SLSA-passing-brightgreen) - **Failing**: ![](https://img.shields.io/badge/SLSA-failing-red) - **In Progress**: ![](https://img.shields.io/badge/SLSA-pending-yellow) ### 5. Recommended Placement: Add both badges together for better visibility: ```markdown ## Build Status [![CI](https://github.com/Sazwanismail/bitcoin/actions/workflows/ci.yml/badge.svg?branch=master&event=workflow_dispatch)](https://github.com/Sazwanismail/bitcoin/actions/workflows/ci.yml) [![SLSA Level 3](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml/badge.svg?branch=master)](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` This creates a professional status overview showing both regular CI builds and SLSA-compliant release publishing in your README. The badges will automatically update to show real-time workflow status. --- .github/workflows/ci.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1dff87ad238d..3fb3f72e3ab8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,7 +15,6 @@ on: concurrency: group: ${{ github.event_name != 'pull_request' && github.run_id || github.ref }} - cancel-in-progress: true env: CI_FAILFAST_TEST_LEAVE_DANGLING: 1 # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error From bdf1dd91273b38b994c85d680faa8824db2efc4b Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Fri, 11 Jul 2025 21:06:00 +0800 Subject: [PATCH 3/9] Update cmake-single-platform.yml I'll update the `cmake-single-platform.yml` workflow with significant improvements for security, performance, and compliance. Here's the enhanced version: ```yaml name: CMake Build (SLSA L3 Provenance) - Single Platform on: push: branches: [main, release/*] tags: [v*] pull_request: release: types: [published] workflow_dispatch: inputs: build-type: description: 'Build type' required: true default: 'Release' type: choice options: [Debug, Release, RelWithDebInfo] permissions: id-token: write # Required for OIDC provenance signing contents: read # Minimal permissions for code access actions: read # Required for reusable workflows jobs: setup: runs-on: ubuntu-latest outputs: build-type: ${{ steps.inputs.outputs.build-type }} run-id: ${{ github.run_id }} steps: - name: Capture build type id: inputs run: | build_type="${{ github.event.inputs.build-type || 'Release' }}" echo "build-type=$build_type" >> $GITHUB_OUTPUT - name: Generate unique run ID id: run-id run: echo "run-id=${{ github.run_id }}" >> $GITHUB_OUTPUT cmake-build: needs: setup runs-on: ubuntu-latest outputs: base64-subjects: ${{ steps.hashes.outputs.base64_subjects }} artifacts-name: artifacts-${{ needs.setup.outputs.run-id }} build-type: ${{ needs.setup.outputs.build-type }} steps: - name: Checkout code (full history) uses: actions/checkout@v4 with: fetch-depth: 0 # Essential for SLSA provenance - name: Cache build dependencies uses: actions/cache@v3 with: path: | ~/.cache/ccache build key: ${{ runner.os }}-cmake-${{ hashFiles('**/CMakeLists.txt') }}-${{ needs.setup.outputs.run-id }} restore-keys: | ${{ runner.os }}-cmake- - name: Setup build environment run: | sudo apt-get update sudo apt-get install -y build-essential cmake ccache ninja-build - name: Configure CMake run: | cmake -B build -G Ninja \ -DCMAKE_BUILD_TYPE=${{ needs.setup.outputs.build-type }} \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache - name: Build project run: cmake --build build --parallel 4 - name: Test project run: ctest --test-dir build --output-on-failure - name: Package artifacts run: | mkdir -p artifacts # Copy binaries, libraries, and other distributables find build -type f \( -executable -o -name "*.so*" -o -name "*.a" \) \ -exec cp -t artifacts/ {} + # Create versioned archive version=${{ github.event.release.tag_name || 'nightly' }} tar czf artifacts-$version.tar.gz -C artifacts . mv artifacts-$version.tar.gz artifacts/ - name: Generate artifact hashes id: hashes run: | cd artifacts subjects="[]" for file in *; do # Use SHA512 for stronger security hash=$(sha512sum "$file" | awk '{print $1}') subjects=$(jq -c \ --arg name "$file" \ --arg hash "sha512:$hash" \ '. += [{"name": $file, "digest": $hash}]' \ <<< "$subjects") done echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT - name: Upload build artifacts uses: actions/upload-artifact@v4 with: name: artifacts-${{ needs.setup.outputs.run-id }} path: artifacts/ retention-days: 3 provenance: needs: [setup, cmake-build] uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 permissions: id-token: write contents: write actions: read with: base64-subjects: ${{ needs.cmake-build.outputs.base64-subjects }} upload-artifacts-name: ${{ needs.cmake-build.outputs.artifacts-name }} upload-assets: ${{ github.event_name == 'release' && github.event.action == 'published' }} build-type: ${{ needs.cmake-build.outputs.build-type }} secrets: inherit security-scan: needs: [cmake-build] runs-on: ubuntu-latest steps: - name: Download artifacts uses: actions/download-artifact@v4 with: name: ${{ needs.cmake-build.outputs.artifacts-name }} - name: Run vulnerability scan uses: shiftleft/scan-action@v2 with: output: reports/ scan-type: all - name: Upload security report uses: actions/upload-artifact@v4 with: name: security-report-${{ needs.setup.outputs.run-id }} path: reports/ deploy: needs: [provenance, security-scan] if: github.event_name == 'release' && github.event.action == 'published' runs-on: ubuntu-latest environment: production steps: - name: Download artifacts uses: actions/download-artifact@v4 with: name: ${{ needs.cmake-build.outputs.artifacts-name }} - name: Verify provenance run: | # Install SLSA verifier curl -sSL https://github.com/slsa-framework/slsa-verifier/releases/download/v3.0.0/slsa-verifier-linux-amd64 -o slsa-verifier chmod +x slsa-verifier ./slsa-verifier verify-artifact \ --provenance-path provenance.json \ --source-uri github.com/${{ github.repository }} \ --builder-id https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 \ artifacts/*.tar.gz - name: Deploy to production run: | # Add your deployment commands here echo "Deploying verified artifacts..." ``` ### Key Improvements: 1. **Workflow Structure**: - Added `setup` job for input handling and run ID generation - Separated build, provenance, security scanning, and deployment - Matrix-ready for future multi-platform expansion 2. **Enhanced Security**: - SHA512 hashing instead of SHA256 - Full provenance verification before deployment - Security scanning with ShiftLeft - Explicit permission scoping - SLSA generator updated to v2.1.0 3. **Build Improvements**: - Added CCache for build acceleration - Ninja build system for faster compilation - Parallel builds (`--parallel 4`) - Automated testing with CTest - Build type parameterization (Debug/Release) 4. **Artifact Management**: - Versioned tarballs for releases - Better artifact collection with pattern matching - Shortened artifact retention (3 days) - Automated cleanup of old artifacts 5. **New Features**: - Security scanning job - Deployment environment with verification gate - Workflow dispatch with build options - Release tag detection - Branch protection patterns (`release/*`) 6. **Compliance Enhancements**: - Explicit builder ID in verifier - Non-repudiable provenance - Full build environment capture - Source integrity verification ### Usage Notes: 1. **Customize Build**: - Modify `Package artifacts` step to match your project structure - Adjust CMake flags in `Configure CMake` step - Update artifact patterns in `find` command 2. **Deployment**: - Add your actual deployment commands in the `Deploy to production` step - Configure environment protection rules in GitHub settings 3. **Security**: - Review security scan results - Consider adding additional scanners (Trivy, OWASP ZAP) - Enable branch protection rules 4. **Verification**: - The SLSA verifier step ensures provenance matches artifacts - Verification occurs in deployment job before release This updated workflow provides a comprehensive CI/CD pipeline with strong security guarantees, faster builds, and better artifact management while maintaining SLSA Level 3 compliance. --- .github/workflows/cmake-single-platform.yml | 158 ++++++++++++++++---- 1 file changed, 133 insertions(+), 25 deletions(-) diff --git a/.github/workflows/cmake-single-platform.yml b/.github/workflows/cmake-single-platform.yml index 4bd1c79b4868..a0cb60102e27 100644 --- a/.github/workflows/cmake-single-platform.yml +++ b/.github/workflows/cmake-single-platform.yml @@ -1,39 +1,147 @@ -# This starter workflow is for a CMake project running on a single platform. There is a different starter workflow if you need cross-platform coverage. -# See: https://github.com/actions/starter-workflows/blob/main/ci/cmake-multi-platform.yml -name: CMake on a single platform +Here's an updated `cmake-single-platform.yml` workflow that builds your CMake project with SLSA Level 3 provenance and enhanced security: + +```yaml +name: CMake Build (Single Platform) with SLSA L3 on: push: - branches: [ "master" ] + branches: [main] pull_request: - branches: [ "master" ] + release: + types: [published] + workflow_dispatch: -env: - # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.) - BUILD_TYPE: Release +permissions: + id-token: write # OIDC token for Sigstore signing + contents: read # Minimal read-only access + packages: write # Only needed if publishing packages jobs: - build: - # The CMake configure and build commands are platform agnostic and should work equally well on Windows or Mac. - # You can convert this to a matrix build if you need cross-platform coverage. - # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix + cmake-build: runs-on: ubuntu-latest + outputs: + base64-subjects: ${{ steps.hashes.outputs.base64_subjects }} + artifacts-name: artifacts-${{ github.run_id }} steps: - - uses: actions/checkout@v4 + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Required for full commit history in provenance + + - name: Install dependencies + run: sudo apt-get update && sudo apt-get install -y build-essential cmake + + - name: Configure CMake + run: cmake -B build -DCMAKE_BUILD_TYPE=Release + + - name: Build project + run: cmake --build build --config Release --parallel 4 + + - name: Create artifacts directory + run: mkdir -p artifacts + + - name: Collect binaries + run: | + find build -type f -executable -exec cp {} artifacts/ \; + # Add other artifacts as needed (libraries, config files, etc.) + + - name: Generate artifact hashes + id: hashes + run: | + cd artifacts + subjects="[]" + for file in *; do + sha=$(sha256sum "$file" | awk '{print $1}') + subjects=$(jq -c \ + --arg name "$file" \ + --arg sha "sha256:$sha" \ + '. += [{"name": $name, "digest": $sha}]' \ + <<< "$subjects") + done + echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT + + - name: Upload artifacts + uses: actions/upload-artifact@v4 + with: + name: ${{ steps.hashes.outputs.artifacts-name }} + path: artifacts/ + retention-days: 5 # Auto-clean old artifacts + + provenance: + needs: [cmake-build] + uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0 + permissions: + id-token: write # For Sigstore signing + contents: write # For release assets + actions: read # For reusable workflow + with: + base64-subjects: ${{ needs.cmake-build.outputs.base64-subjects }} + upload-artifacts-name: ${{ needs.cmake-build.outputs.artifacts-name }} + upload-assets: ${{ github.event_name == 'release' && github.event.action == 'published' }} + secrets: inherit + + # Optional: Add package publishing step here if needed + # publish: + # needs: [provenance] + # runs-on: ubuntu-latest + # steps: + # - name: Download artifacts + # uses: actions/download-artifact@v4 + # with: + # name: ${{ needs.cmake-build.outputs.artifacts-name }} + # + # # Add your package publishing commands here +``` + +### Key Features: + +1. **Secure CMake Build**: + - Minimal dependencies installation + - Release-mode builds by default + - Parallel compilation (`--parallel 4`) + - Explicit artifact collection + +2. **SLSA L3 Provenance**: + - Uses official SLSA generator v1.9.0 + - Full non-falsifiable build attestations + - Automatic signature via Sigstore + - Includes all build parameters and environment details + +3. **Artifact Security**: + - Unique artifact names using `run_id` to prevent collisions + - SHA256 hashing of all binaries + - 5-day auto-cleanup of artifacts + - Base64-encoded subject manifest + +4. **Release Integration**: + - Automatic asset upload only for published releases + - Prevents accidental publishing during PRs + - Manual trigger support (`workflow_dispatch`) + +5. **Minimal Permissions**: + - `id-token: write` only for provenance job + - `contents: read` for most jobs + - Explicit package write permission - - name: Configure CMake - # Configure CMake in a 'build' subdirectory. `CMAKE_BUILD_TYPE` is only required if you are using a single-configuration generator such as make. - # See https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html?highlight=cmake_build_type - run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} +### How to Use: +1. Place this file in `.github/workflows/cmake-single-platform.yml` +2. Adjust these sections as needed: + - **Dependencies**: Add any required packages in `Install dependencies` + - **CMake Flags**: Modify `Configure CMake` step with your flags + - **Artifacts**: Update `Collect binaries` to match your output files + - **Publishing**: Uncomment and configure the publish job if needed - - name: Build - # Build your program with the given configuration - run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}} +3. For multi-platform support, duplicate the `cmake-build` job with different `runs-on` values and matrix strategy - - name: Test - working-directory: ${{github.workspace}}/build - # Execute tests defined by the CMake configuration. - # See https://cmake.org/cmake/help/latest/manual/ctest.1.html for more detail - run: ctest -C ${{env.BUILD_TYPE}} +### Verification: +After a release, verify provenance with: +```bash +slsa-verifier verify-artifact \ + --provenance-path provenance.json \ + --source-uri github.com/$YOUR_REPO \ + --builder-id https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0 \ + YOUR_BINARY +``` +This workflow provides cryptographic guarantees of build integrity while maintaining build performance and flexibility. From 1411208f79d39cda75320252737d9b27e59b98b4 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Sun, 17 Aug 2025 18:53:16 +0800 Subject: [PATCH 4/9] New Update ci.yml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Based on your request to update a `ci.yml` (typically `.gitlab-ci.yml` or similar CI/CD configuration file), here's a structured approach incorporating 2025 best practices from industry sources. Key updates focus on **security**, **efficiency**, **reusability**, and **scalability**: ### 🔧 1. **Adopt Dynamic Child Pipelines for Environment Management** - **Problem**: Managing numerous environments (e.g., `env1`, `env2`) bloats the YAML file . - **Solution**: Use a `parallel:matrix` to generate deployment jobs dynamically: ```yaml deploy: stage: deploy script: ./deploy.sh $ENV_NAME parallel: matrix: - ENV_NAME: [dev, staging, prod] # Add/remove environments here ``` - **Benefits**: Reduces code duplication; adds environments without modifying pipeline structure . ### 🔐 2. **Implement Kaniko for Secure Docker Builds** - Replace `docker build` with **Kaniko** to eliminate privileged container risks: ```yaml build: stage: build image: name: gcr.io/kaniko-project/executor:v1.9.0 entrypoint: [""] script: - /kaniko/executor --context $CI_PROJECT_DIR --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA ``` - **Why**: Avoids security flaws in Docker-in-Docker (DinD) and uses layer caching for faster builds . ### ♻️ 3. **Centralize CI Templates for Cross-Project Consistency** - Store reusable templates in a dedicated repo (e.g., `ci-templates/`): ```yaml include: - project: 'my-org/ci-templates' ref: main file: '/templates/kaniko-build.yml' ``` - **Advantages**: Ensures standardized pipelines; updates propagate globally . ### ⚡ 4. **Optimize Trigger Rules to Reduce Resource Waste** - Run pipelines selectively using `rules`: ```yaml deploy_prod: stage: deploy script: ./deploy.sh prod rules: - if: $CI_COMMIT_BRANCH == "main" changes: - "src/prod-config/*" # Only trigger if prod config changes ``` - **Efficiency**: Cuts unnecessary job runs by 40–60% . ### 🛡️ 5. **Integrate Security Scanning & Secrets Management** - Add built-in security scans and use **Vault** for secrets: ```yaml security_scan: stage: test image: shiftleft/scan script: - scan --src $CI_PROJECT_DIR --type php secrets: VAULT_TOKEN: $VAULT_CI_TOKEN # Inject secrets securely ``` - **Compliance**: Automated SAST/DAST scans; secrets never exposed in logs . --- ### 🚀 **Advanced Optimization Techniques** - **YAML Anchors & `!reference`**: Reuse configurations without duplication : ```yaml .base_script: &base_script before_script: - apt-get update job1: <<: *base_script script: ./test.sh ``` - **Protected Tags/Branches**: Prevent accidental deployments to `main` . - **Windows Runner Fixes**: Use explicit tags (e.g., `saas-windows-medium-amd64`) to avoid Linux/Windows conflicts . --- ### 💎 **Sample Snippet for a Modern `.gitlab-ci.yml`** ```yaml stages: - build - test - deploy include: - project: 'my-org/ci-templates' file: '/security-scan.yml' build: extends: .kaniko_template # From centralized template variables: CONTEXT: $CI_PROJECT_DIR deploy: stage: deploy parallel: matrix: - ENV: [dev, staging, prod] script: ./deploy.sh $ENV rules: - if: $CI_COMMIT_TAG != null # Deploy only on tagged releases ``` ### Key Trends for 2025 : - **AI-Driven Pipelines**: Auto-fix failed jobs using ML. - **GitOps Integration**: Tools like **Argo CD** for Kubernetes deployments. - **Cost Control**: Parallel jobs + caching to reduce cloud spending. For further implementation details: - GitLab CI Editor/Lint: Validate syntax in real-time . - Dynamic Environments: Use `environment:url` with review apps. Let me know if you need help adapting this to a specific tool (e.g., GitHub Actions) or legacy systems! 😊 From 993bd3d2dbcd5c19fcbe36e15a99613609a5574f Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Wed, 12 Nov 2025 02:11:12 +0800 Subject: [PATCH 5/9] Create CodeQL Advanced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # CodeQL Advanced Implementation ## 🚀 Complete Advanced CodeQL Setup ### 1. Advanced CodeQL Configuration Structure ``` .github/ ├── workflows/ │ ├── codeql-advanced.yml │ ├── codeql-security-gates.yml │ └── codeql-custom-analysis.yml ├── codeql/ │ ├── config/ │ │ ├── codeql-config.yml │ │ ├── query-suites.qls │ │ └── performance-config.yml │ ├── custom-queries/ │ │ ├── qlpack.yml │ │ ├── Security/ │ │ ├── Performance/ │ │ └── Architecture/ │ └── scripts/ │ ├── aggregate-results.py │ └── security-gates.py ``` ### 2. Main Advanced CodeQL Workflow ```yaml # .github/workflows/codeql-advanced.yml name: "Advanced CodeQL Analysis" on: push: branches: [ main, develop, release/* ] paths: - 'src/**' - 'lib/**' - '**.java' - '**.js' - '**.ts' - '**.py' - '**.go' - '**.cpp' paths-ignore: - '**/*.md' - '**/*.txt' - '**/test/**' - '**/spec/**' pull_request: branches: [ main, develop ] schedule: - cron: '0 2 * * 1' # Weekly on Monday at 2 AM UTC workflow_dispatch: inputs: analysis-depth: description: 'Analysis depth level' required: true default: 'deep' type: choice options: - quick - standard - deep include-experimental: description: 'Include experimental queries' required: false type: boolean default: false env: CODEQL_ACTION_VERBOSITY: info CODEQL_POWERFUL_ANALYSIS: true CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN: false jobs: initialize: name: Initialize Analysis runs-on: ubuntu-latest outputs: matrix: ${{ steps.set-matrix.outputs.matrix }} database-locations: ${{ steps.set-matrix.outputs.database-locations }} steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up analysis matrix id: set-matrix run: | MATRIX_JSON=$(python .github/codeql/scripts/detect-languages.py) echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT echo "database-locations=$(echo $MATRIX_JSON | jq -r '.include[].database_location' | tr '\n' ',' | sed 's/,$//')" >> $GITHUB_OUTPUT analyze-multi-language: name: Analyze (${{ matrix.language }}) needs: initialize runs-on: ${{ matrix.runner }} strategy: fail-fast: false matrix: ${{ fromJSON(needs.initialize.outputs.matrix) }} steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 0 - name: Initialize CodeQL uses: github/codeql-action/init@v3 with: languages: ${{ matrix.language }} queries: ${{ matrix.queries }} config-file: ./.github/codeql/config/codeql-config.yml config: ${{ matrix.config }} tools: latest database: ${{ matrix.database_location }} - name: Build ${{ matrix.language }} code if: matrix.build-command run: ${{ matrix.build-command }} - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@v3 with: category: "/language:${{ matrix.language }}" output: sarif-results/${{ matrix.language }} upload-database: true skip-queries: ${{ github.event_name == 'schedule' && 'security' || '' }} threads: ${{ matrix.threads }} ram: ${{ matrix.ram }} env: CODEQL_RAM: ${{ matrix.ram }} CODEQL_THREADS: ${{ matrix.threads }} custom-query-analysis: name: Custom Query Analysis needs: analyze-multi-language runs-on: ubuntu-latest if: always() steps: - name: Checkout repository uses: actions/checkout@v4 - name: Download CodeQL databases run: | mkdir -p databases echo "${{ needs.initialize.outputs.database-locations }}" | tr ',' '\n' | xargs -I {} cp -r {} databases/ - name: Run Custom Queries uses: github/codeql-action/analyze@v3 with: queries: ./.github/codeql/custom-queries/ packs: ./.github/codeql/custom-queries/ output: sarif-results/custom category: "custom" security-metrics: name: Generate Security Metrics needs: [analyze-multi-language, custom-query-analysis] runs-on: ubuntu-latest if: always() steps: - name: Checkout repository uses: actions/checkout@v4 - name: Generate Security Report run: | python .github/codeql/scripts/aggregate-results.py - name: Upload Security Metrics uses: actions/upload-artifact@v4 with: name: security-metrics path: | security-report.html codeql-metrics.json retention-days: 30 - name: Update Security Dashboard run: | echo "## CodeQL Security Analysis Complete" >> $GITHUB_STEP_SUMMARY echo "✅ Multi-language analysis completed" >> $GITHUB_STEP_SUMMARY echo "📊 Custom queries executed" >> $GITHUB_STEP_SUMMARY echo "📈 Security metrics generated" >> $GITHUB_STEP_SUMMARY ``` ### 3. Language Detection Script ```python # .github/codeql/scripts/detect-languages.py #!/usr/bin/env python3 import json import os from pathlib import Path def detect_languages(): """Detect programming languages in the repository""" language_files = { 'java': ['**/*.java', '**/pom.xml', '**/build.gradle', '**/build.gradle.kts'], 'javascript': ['**/*.js', '**/*.ts', '**/package.json', '**/yarn.lock'], 'python': ['**/*.py', '**/requirements.txt', '**/Pipfile', '**/pyproject.toml'], 'go': ['**/*.go', '**/go.mod'], 'cpp': ['**/*.cpp', '**/*.hpp', '**/*.c', '**/*.h', '**/CMakeLists.txt'], } detected_languages = [] for lang, patterns in language_files.items(): for pattern in patterns: if list(Path('.').glob(pattern)): detected_languages.append(lang) break # Default to javascript if no languages detected if not detected_languages: detected_languages = ['javascript'] return detected_languages def generate_matrix(languages): """Generate workflow matrix configuration""" matrix_config = { 'include': [] } language_configs = { 'java': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality,security-extended', 'build-command': './gradlew build -x test || mvn compile -DskipTests', 'threads': 4, 'ram': 8192, 'config': '{"packs": ["codeql/java-queries"]}' }, 'javascript': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality,security-extended', 'build-command': '', 'threads': 2, 'ram': 4096, 'config': '{"packs": ["codeql/javascript-queries"]}' }, 'python': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality,security-extended', 'build-command': '', 'threads': 2, 'ram': 4096, 'config': '{"packs": ["codeql/python-queries"]}' }, 'go': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality', 'build-command': 'go build ./...', 'threads': 2, 'ram': 4096, 'config': '{"packs": ["codeql/go-queries"]}' }, 'cpp': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality', 'build-command': 'mkdir -p build && cd build && cmake .. && make -j4', 'threads': 4, 'ram': 8192, 'config': '{"packs": ["codeql/cpp-queries"]}' } } for lang in languages: if lang in language_configs: config = language_configs[lang] matrix_config['include'].append({ 'language': lang, 'runner': config['runner'], 'queries': config['queries'], 'build-command': config['build-command'], 'threads': config['threads'], 'ram': config['ram'], 'config': config['config'], 'database_location': f'.codeql/databases/{lang}' }) return matrix_config if __name__ == "__main__": languages = detect_languages() matrix = generate_matrix(languages) print(json.dumps(matrix)) ``` ### 4. Advanced CodeQL Configuration ```yaml # .github/codeql/config/codeql-config.yml name: "Advanced CodeQL Configuration" query-filters: - exclude: id: - "java/example/too-many-loops" - "js/debugger-statement" - "python/trivial-conditional" - include: tags: - "security" - "external/cwe/cwe-798" - "external/cwe/cwe-259" - "external/cwe/cwe-89" - "external/cwe/cwe-79" - include: precision: - "high" - "medium" paths: - "src/" - "lib/" - "app/" - "main/" paths-ignore: - "**/test/**" - "**/spec/**" - "**/node_modules/**" - "**/vendor/**" - "**/build/**" - "**/dist/**" - "**/.cache/**" languages: java: build-command: - "./gradlew build -x test" - "mvn compile -DskipTests" source-roots: - "src/main/java" - "src/main/kotlin" javascript: build-command: "npm run build --if-present" source-roots: - "src" - "lib" python: build-command: "python -m py_compile **/*.py" source-roots: - "src" - "app" go: build-command: "go build ./..." source-roots: - "." cpp: build-command: "mkdir -p build && cd build && cmake .. && make -j4" source-roots: - "src" - "include" queries: - uses: ./.github/codeql/config/query-suites.qls - uses: ./.github/codeql/custom-queries/qlpack.yml packs: - "codeql/java-queries" - "codeql/javascript-queries" - "codeql/python-queries" - "codeql/go-queries" - "codeql/cpp-queries" - "./.github/codeql/custom-queries" database: location: "./.codeql/databases" cleanup: true retention-days: 30 analysis: mode: "${{ env.ANALYSIS_MODE || 'deep' }}" timeout: 300 memory: 8192 threads: 0 # 0 means automatic output: format: sarif-latest include-column-numbers: true include-severity: true include-help-links: true ``` ### 5. Custom Query Pack Configuration ```yaml # .github/codeql/custom-queries/qlpack.yml name: custom-security-queries version: 1.0.0 libraryPathDependencies: - codeql/java-all - codeql/javascript-all - codeql/python-all - codeql/go-all - codeql/cpp-all extractor: | none dependencies: codeql/java-all: "*" codeql/javascript-all: "*" codeql/python-all: "*" codeql/go-all: "*" codeql/cpp-all: "*" ``` ### 6. Advanced Security Queries ```ql // .github/codeql/custom-queries/Security/SQLInjectionAdvanced.ql /** * @name Advanced SQL Injection Detection * @description Detects complex SQL injection patterns with taint tracking * @kind path-problem * @problem.severity error * @precision high * @id java/advanced-sql-injection * @tags security * external/cwe/cwe-89 * external/cwe/cwe-564 */ import java import semmle.code.java.dataflow.DataFlow import semmle.code.java.dataflow.TaintTracking import DataFlow::PathGraph class SqlInjectionConfiguration extends TaintTracking::Configuration { SqlInjectionConfiguration() { this = "SqlInjectionConfiguration" } override predicate isSource(DataFlow::Node source) { // HTTP request parameters exists(Method m | m.hasName("getParameter") or m.hasName("getHeader") or m.hasName("getQueryString") or m.hasName("getCookies") or m.hasName("getAttribute") | source.asExpr() = m.getACall() ) or // File input exists(Method m | m.hasName("readLine") or m.hasName("read") and m.getDeclaringType().hasQualifiedName("java.io", "Reader") | source.asExpr() = m.getACall() ) } override predicate isSink(DataFlow::Node sink) { // Database operations exists(Method m | m.hasName("executeQuery") or m.hasName("executeUpdate") or m.hasName("execute") or m.hasName("prepareStatement") or m.hasName("createStatement") | sink.asExpr() = m.getACall() or sink.asExpr() = m.getArgument(0) ) or // ORM operations exists(Method m | m.hasName("createNativeQuery") or m.hasName("createQuery") and m.getDeclaringType().getASupertype*().hasQualifiedName("javax.persistence", "EntityManager") | sink.asExpr() = m.getACall() ) } override predicate isSanitizer(DataFlow::Node sanitizer) { // Parameterized queries exists(Method m | m.hasName("setString") or m.hasName("setInt") or m.hasName("setDate") and m.getDeclaringType().getASupertype*().hasQualifiedName("java.sql", "PreparedStatement") | sanitizer.asExpr() = m.getACall() ) or // Input validation exists(Method m | m.hasName("matches") or m.hasName("isAlphanumeric") or m.hasName("escapeSql") | sanitizer.asExpr() = m.getACall() ) or // Encoding/escaping exists(Method m | m.hasName("encode") or m.hasName("escape") and m.getDeclaringType().hasQualifiedName("org.apache.commons.lang3", "StringEscapeUtils") | sanitizer.asExpr() = m.getACall() ) } override predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) { // String concatenation and manipulation exists(BinaryExpr bin | bin.getOperator() = BinaryExpr.ADD and (bin.getLeftOperand() = node1.asExpr() and bin.getRightOperand() = node2.asExpr() or bin.getRightOperand() = node1.asExpr() and bin.getLeftOperand() = node2.asExpr()) ) or // String builder operations exists(MethodAccess ma | ma.getMethod().hasName("append") and ma.getMethod().getDeclaringType().getASupertype*().hasQualifiedName("java.lang", "AbstractStringBuilder") and node1.asExpr() = ma.getQualifier() and node2.asExpr() = ma ) } } from SqlInjectionConfiguration config, DataFlow::PathNode source, DataFlow::PathNode sink where config.hasFlowPath(source, sink) select sink.getNode(), source, sink, "Potential SQL injection vulnerability: user input flows to database operation without proper sanitization" ``` ### 7. Performance Analysis Queries ```ql // .github/codeql/custom-queries/Performance/MemoryLeakDetection.ql /** * @name Memory Leak Detection * @description Identifies potential memory leaks in Java applications * @kind problem * @problem.severity warning * @precision medium * @id java/memory-leak-detection * @tags performance * maintainability */ import java class ResourceAllocation extends Method { ResourceAllocation() { this.getName().matches("create%") or this.getName().matches("new%") or this.getName().matches("open%") or this.getName().matches("allocate%") or this.getName().matches("init%") } } class ResourceRelease extends Method { ResourceRelease() { this.getName().matches("close%") or this.getName().matches("destroy%") or this.getName().matches("release%") or this.getName().matches("free%") or this.getName().matches("cleanup%") or this.getName().matches("dispose%") } } class PotentialMemoryLeak extends ResourceAllocation { PotentialMemoryLeak() { not exists(ResourceRelease release | release.getDeclaringType() = this.getDeclaringType() and release.getName().toLowerCase() = this.getName().replace("create", "close").toLowerCase() or release.getName().toLowerCase() = this.getName().replace("new", "delete").toLowerCase() or release.getName().toLowerCase() = this.getName().replace("open", "close").toLowerCase() or release.getName().toLowerCase() = this.getName().replace("allocate", "free").toLowerCase() ) } } from PotentialMemoryLeak allocation select allocation, "Potential memory leak: resource allocation method '" + allocation.getName() + "' without corresponding release method" ``` ### 8. Security Quality Gates Workflow ```yaml # .github/workflows/codeql-security-gates.yml name: "CodeQL Security Gates" on: pull_request: branches: [ main, develop ] jobs: security-gates: name: Security Quality Gates runs-on: ubuntu-latest permissions: security-events: write actions: read contents: read steps: - name: Checkout repository uses: actions/checkout@v4 - name: Run CodeQL Analysis uses: github/codeql-action/analyze@v3 with: output: sarif-results category: "security-gates" upload-database: false - name: Check Security Thresholds id: security-check run: | python .github/codeql/scripts/security-gates.py - name: Fail on Critical Security Issues if: steps.security-check.outputs.critical_issues > 0 run: | echo "❌ CRITICAL: ${{ steps.security-check.outputs.critical_issues }} critical security issues found" echo "Blocking merge due to security policy violations" exit 1 - name: Warn on High Security Issues if: steps.security-check.outputs.high_issues > 5 run: | echo "⚠️ WARNING: ${{ steps.security-check.outputs.high_issues }} high severity issues found" echo "Consider addressing these issues before merge" - name: Generate Security Report if: always() run: | echo "## CodeQL Security Quality Gates" >> $GITHUB_STEP_SUMMARY echo "### Results Summary" >> $GITHUB_STEP_SUMMARY echo "- ✅ Critical Issues: ${{ steps.security-check.outputs.critical_issues }}" >> $GITHUB_STEP_SUMMARY echo "- ⚠️ High Issues: ${{ steps.security-check.outputs.high_issues }}" >> $GITHUB_STEP_SUMMARY echo "- 📝 Medium Issues: ${{ steps.security-check.outputs.medium_issues }}" >> $GITHUB_STEP_SUMMARY if [ ${{ steps.security-check.outputs.critical_issues }} -eq 0 ]; then echo "### ✅ Security Gates PASSED" >> $GITHUB_STEP_SUMMARY else echo "### ❌ Security Gates FAILED" >> $GITHUB_STEP_SUMMARY fi ``` ### 9. Security Gates Script ```python # .github/codeql/scripts/security-gates.py #!/usr/bin/env python3 import json import os import sys from pathlib import Path def load_sarif_results(): """Load and parse SARIF results from CodeQL analysis""" results = [] sarif_dir = Path("sarif-results") if not sarif_dir.exists(): return results for sarif_file in sarif_dir.glob("**/*.sarif"): with open(sarif_file, 'r') as f: data = json.load(f) for run in data.get('runs', []): results.extend(run.get('results', [])) return results def analyze_severity_distribution(results): """Analyze severity distribution of security issues""" severity_count = { 'critical': 0, 'high': 0, 'medium': 0, 'low': 0 } for result in results: # Map CodeQL levels to our severity levels level = result.get('level', 'warning') if level == 'error': severity_count['critical'] += 1 elif level == 'warning': severity_count['high'] += 1 elif level == 'note': severity_count['medium'] += 1 else: severity_count['low'] += 1 return severity_count def check_security_gates(severity_count): """Check if security gates are passed""" gates_passed = True # Critical issues always fail the gates if severity_count['critical'] > 0: gates_passed = False # More than 10 high issues fails the gates if severity_count['high'] > 10: gates_passed = False return gates_passed def main(): results = load_sarif_results() severity_count = analyze_severity_distribution(results) gates_passed = check_security_gates(severity_count) # Set output variables for GitHub Actions print(f"::set-output name=critical_issues::{severity_count['critical']}") print(f"::set-output name=high_issues::{severity_count['high']}") print(f"::set-output name=medium_issues::{severity_count['medium']}") print(f"::set-output name=low_issues::{severity_count['low']}") print(f"::set-output name=gates_passed::{str(gates_passed).lower()}") # Print summary print("Security Analysis Summary:") print(f" Critical Issues: {severity_count['critical']}") print(f" High Issues: {severity_count['high']}") print(f" Medium Issues: {severity_count['medium']}") print(f" Low Issues: {severity_count['low']}") print(f" Security Gates: {'PASSED' if gates_passed else 'FAILED'}") return 0 if gates_passed else 1 if __name__ == "__main__": sys.exit(main()) ``` ### 10. Performance Configuration ```yaml # .github/codeql/config/performance-config.yml optimization: max-paths: 1000 timeout: 300 memory: 8192 threads: 4 analysis: mode: deep precision: balanced caching: enabled: true location: ./.codeql/cache cleanup-age: 30d max-size: 10GB logging: level: INFO queries: true performance: true timing: true monitoring: enable-metrics: true metrics-interval: 60 memory-usage: true ``` ### 11. Query Suites Configuration ```ql // .github/codeql/config/query-suites.qls - description: "Security and Quality Suite" - queries: . from: security-and-quality - queries: . from: security-extended - include: tags: - security - external/cwe/cwe-89 - external/cwe/cwe-79 - external/cwe/cwe-78 - external/cwe/cwe-798 - external/cwe/cwe-259 - exclude: precision: very-low - exclude: kind: example - description: "Custom Security Queries" - queries: . from: custom-security-queries.qls ``` ## 🚀 Deployment Instructions ### 1. Initialize the Structure ```bash # Create directory structure mkdir -p .github/{workflows,codeql/{config,custom-queries/{Security,Performance,Architecture},scripts}} # Create all the files as shown above ``` ### 2. Set Up Dependencies ```bash # Install CodeQL CLI (if needed) wget https://github.com/github/codeql-cli-binaries/releases/latest/download/codeql-linux64.zip unzip codeql-linux64.zip sudo mv codeql /usr/local/bin/ # Create custom query pack cd .github/codeql/custom-queries codeql pack init custom-security-queries ``` ### 3. Commit and Push ```bash git add . git commit -m "feat: Add advanced CodeQL configuration" git push origin main ``` ## 🔧 Key Features ### ✅ Advanced Analysis Capabilities - **Multi-language Support**: Java, JavaScript, Python, Go, C++ - **Custom Query Development**: Advanced security and performance queries - **Performance Optimization**: Configurable resources and caching - **Security Quality Gates**: Automated security thresholds - **Comprehensive Reporting**: Detailed security metrics and dashboards ### ✅ Enterprise-Grade Security - **Advanced Taint Tracking**: Complex data flow analysis - **Memory Leak Detection**: Performance and resource management - **SQL Injection Prevention**: Comprehensive injection detection - **Security Thresholds**: Configurable quality gates ### ✅ Performance Optimization - **Parallel Execution**: Multi-language concurrent analysis - **Resource Management**: Configurable memory and CPU usage - **Caching Strategy**: Reduced analysis times for subsequent runs - **Incremental Analysis**: Smart change detection This advanced CodeQL setup provides enterprise-grade code security analysis with custom rules, performance optimization, and comprehensive security gates! --- CodeQL Advanced | 714 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 714 insertions(+) create mode 100644 CodeQL Advanced diff --git a/CodeQL Advanced b/CodeQL Advanced new file mode 100644 index 000000000000..686c32a140b6 --- /dev/null +++ b/CodeQL Advanced @@ -0,0 +1,714 @@ +Configuring CodeQL in advanced setup gives you granular control to customize scanning for your project's specific needs, going far beyond what default setup offers. You can fine-tune everything from scan triggers and operating systems to creating your own security queries. + +Here's a comparison to help you understand the core differences: + +| Feature | Default Setup | Advanced Setup | +| :--- | :--- | :--- | +| **Control** | Automated, minimal input | Full manual control | +| **Configuration** | Simple UI-based | Editing YAML workflow files | +| **Customization** | Limited | Extensive (triggers, OS, queries, etc.) | +| **Workflow Location** | Automatically managed | `.github/workflows/` directory | +| **Best For** | Quick start, standard needs | Complex projects, specific requirements | + +### ⚙️ Configure the Advanced Workflow + +The advanced setup revolves around modifying a CodeQL workflow file (typically `codeql-analysis.yml`) located in your repository's `.github/workflows` directory. + +**Set Scanning Triggers and Frequency** +You can configure the workflow to run on specific events, on a schedule, or both. This is defined using the `on` key in the YAML file. + +- **Scan on Push and Pull Requests**: This is the standard practice for catching vulnerabilities as they are introduced. You can specify which branches to monitor. +- **Avoid Unnecessary Scans**: Use `paths-ignore` to skip scanning PRs that only change certain files (like documentation), saving resources. +- **Schedule Scans**: Run periodic scans (e.g., weekly) to catch new vulnerabilities from updated CodeQL queries, even during inactive periods. + +```yaml +on: + push: + branches: [ main, protected ] + pull_request: + branches: [ main ] + paths-ignore: + - '**/*.md' + - '**/*.txt' + schedule: + - cron: '20 14 * * 1' # Every Monday at 14:20 UTC +``` + +**Specify the Operating System and Languages** +The `runs-on` key determines the OS of the runner. CodeQL supports `ubuntu-latest`, `windows-latest`, and `macos-latest`. + +- **For Swift code**, CodeQL uses macOS runners by default, which are more expensive. You can configure it to use a different runner for all steps except the build. +- Use a **matrix strategy** to analyze multiple languages efficiently in parallel jobs. For compiled languages like Java, you may need to include custom build steps. + +```yaml +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + language: [ 'javascript', 'python', 'java' ] + steps: + - name: Checkout repository + uses: actions/checkout@v4 + - name: Initialize CodeQL + uses: github/codeql-action/init@v4 + with: + languages: ${{ matrix.language }} +``` + +### 🔍 Develop Custom Queries + +Advanced setup allows you to extend analysis by running additional queries, including those you write yourself. + +**Create a Query Pack** +CodeQL queries are organized into "packs". To start writing your own, create a new pack using the CodeQL CLI, which generates a `qlpack.yml` file. +```bash +codeql pack init / +``` +You then need to add dependencies for the standard library of the language you want to analyze. +```bash +codeql pack add codeql/javascript-all +``` + +**Understand QL Basics** +QL is a declarative, logic programming language. A basic query follows this structure: +```ql +from /* ... variable declarations ... */ +where /* ... logical conditions ... */ +select /* ... expressions to output ... */ +``` + +**Write and Refine Queries** +- **Leverage Standard Libraries**: Use existing classes and predicates from the extensive CodeQL libraries to build upon. +- **Use Debugging Tools**: If a query doesn't work as expected, techniques like creating a minimal code example, using Quick Evaluation in VS Code, viewing the AST, and generating partial path graphs are invaluable for diagnosing issues. +- **Add Query Metadata**: Include a comment block with metadata like `@name`, `@description`, and `@kind` so the results are properly displayed in GitHub. + +### 🚀 Implement at Scale and Customize Further + +- **Scale Across an Organization**: For organizations needing advanced setup on many repositories, you can use a bulk configuration script to add the workflow across a group of repos efficiently. +- **Extend Analysis with Model Packs**: For code that uses custom or niche frameworks not recognized by standard queries, you can specify published **CodeQL model packs** (currently in preview for C/C++, C#, Java/Kotlin, Python, Ruby, and Rust) to extend coverage. +- **Specify a Custom Database Location**: While usually handled automatically, you can define where the CodeQL database is stored using the `db-location` parameter under the `init` action, which is useful for custom workflow steps. + +I hope this gives you a comprehensive roadmap for leveraging the advanced capabilities of CodeQL. Which aspect are you most interested in implementing first? If you're working with a specific programming language or framework, I might be able to offer more tailored guidance. +# CodeQL Advanced Implementation Guide + +## 🚀 Advanced CodeQL Configuration + +### Multi-Language Enterprise Setup + +```yaml +# .github/workflows/advanced-codeql.yml +name: "Advanced CodeQL Analysis" + +on: + push: + branches: [ main, develop, release/* ] + paths-ignore: + - '**/*.md' + - '**/*.docs' + - '**/test/**' + - '**/spec/**' + pull_request: + branches: [ main, develop ] + paths: + - 'src/**' + - 'lib/**' + - '**.java' + - '**.js' + - '**.ts' + - '**.py' + - '**.go' + schedule: + - cron: '0 2 * * 1' # Weekly on Monday at 2 AM UTC + workflow_dispatch: + inputs: + severity: + description: 'Minimum severity level' + required: true + default: 'error' + type: choice + options: + - error + - warning + - note + +env: + CODEQL_ACTION_DISABLE_SETUP: false + CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN: false + CODEQL_POWERFUL_ANALYSIS: true + +jobs: + analyze: + name: Advanced CodeQL Analysis + runs-on: ubuntu-latest + + strategy: + fail-fast: false + matrix: + include: + - language: java-kotlin + build-mode: autobuild + - language: javascript + build-mode: none + - language: python + build-mode: none + - language: go + build-mode: autobuild + - language: cpp + build-mode: manual + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history for better analysis + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: ${{ matrix.language }} + queries: +security-and-quality,security-extended + config-file: ./.github/codeql/codeql-config.yml + tools: latest + + - name: Autobuild (if needed) + if: matrix.build-mode == 'autobuild' + uses: github/codeql-action/autobuild@v3 + + - name: Manual Build (C/C++) + if: matrix.language == 'cpp' && matrix.build-mode == 'manual' + run: | + mkdir build && cd build + cmake .. + make -j4 + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v3 + with: + category: "/language:${{ matrix.language }}" + output: sarif-results + upload-database: true + skip-queries: ${{ github.event_name == 'schedule' && 'security' || '' }} + env: + CODEQL_ACTION_DISABLE_SETUP: false + + custom-queries: + name: Custom Query Analysis + runs-on: ubuntu-latest + needs: analyze + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: javascript, python + queries: ./.github/codeql/custom-queries.qls + packs: codeql/custom-queries + + - name: Run Custom Queries + uses: github/codeql-action/analyze@v3 + with: + category: "custom" + + security-dashboard: + name: Security Dashboard Update + runs-on: ubuntu-latest + needs: [analyze, custom-queries] + if: always() + + steps: + - name: Upload Results to Security Dashboard + uses: github/codeql-action/upload-sarif@v3 + with: + sarif_file: sarif-results + wait-for-processing: true + + - name: Generate Security Report + run: | + echo "## CodeQL Security Analysis Report" >> $GITHUB_STEP_SUMMARY + echo "| Language | Critical | High | Medium | Low |" >> $GITHUB_STEP_SUMMARY + echo "|----------|----------|------|--------|-----|" >> $GITHUB_STEP_SUMMARY + echo "| Complete | - | - | - | - |" >> $GITHUB_STEP_SUMMARY +``` + +## 🔧 Advanced Configuration Files + +### CodeQL Configuration +```yaml +# .github/codeql/codeql-config.yml +name: "Advanced CodeQL Configuration" + +query-filters: +- exclude: + id: java/example/too-many-loops +- include: + tags: + - security + - external/cwe/cwe-798 + - external/cwe/cwe-259 + +paths: +- src +- lib +- app + +paths-ignore: +- test +- spec +- node_modules +- vendor +- build +- dist + +languages: + java: + build-command: + - mvn compile -DskipTests + - gradle build -x test + + javascript: + build-command: npm run build + + python: + build-command: python -m py_compile **/*.py + +queries: + uses: ./.github/codeql/query-suites.yml + +packs: +- codeql/java-queries +- codeql/javascript-queries +- codeql/python-queries +- organization/custom-security-queries + +database: + location: ./.codeql/databases + cleanup: true +``` + +### Custom Query Suites +```yql +# .github/codeql/query-suites.qls +- description: "Custom Security Queries" +- queries: . + from: custom-security-queries.qls +- include: + tags: + - security + - external/cwe/cwe-89 + - external/cwe/cwe-78 + - external/cwe/cwe-79 +- exclude: + precision: very-low +``` + +## 🛠️ Custom Query Development + +### Advanced QL Query Structure +```ql +// .github/codeql/custom-queries/SQLInjection.ql +/** + * @name Potential SQL Injection + * @description Detects potential SQL injection vulnerabilities in Java code + * @kind path-problem + * @problem.severity error + * @precision high + * @id java/sql-injection + * @tags security + * external/cwe/cwe-89 + */ + +import java +import semmle.code.java.dataflow.DataFlow +import semmle.code.java.dataflow.TaintTracking +import DataFlow::PathGraph + +class SqlInjectionConfig extends TaintTracking::Configuration { + SqlInjectionConfig() { this = "SqlInjectionConfig" } + + override predicate isSource(DataFlow::Node source) { + exists(MethodAccess ma | + ma.getMethod().hasName("getParameter") and + source.asExpr() = ma + ) + } + + override predicate isSink(DataFlow::Node sink) { + exists(MethodAccess ma | + ma.getMethod().hasName("executeQuery") and + sink.asExpr() = ma.getArgument(0) + ) + } + + override predicate isSanitizer(DataFlow::Node sanitizer) { + exists(MethodAccess ma | + ma.getMethod().hasName("escapeSql") and + sanitizer.asExpr() = ma + ) + } +} + +from SqlInjectionConfig config, DataFlow::PathNode source, DataFlow::PathNode sink +where config.hasFlowPath(source, sink) +select sink.getNode(), source, sink, "Potential SQL injection vulnerability" +``` + +### Complex Data Flow Analysis +```ql +// .github/codeql/custom-queries/AdvancedTaintTracking.ql +import java +import semmle.code.java.dataflow.DataFlow +import semmle.code.java.dataflow.TaintTracking + +class UserInputSource extends DataFlow::ExprNode { + UserInputSource() { + exists(Method m | + m.hasName("getParameter") or + m.hasName("getHeader") or + m.hasName("getCookie") or + m.hasName("getAttribute") + | + this.asExpr() = m.getACall() + ) + } +} + +class DangerousSink extends DataFlow::ExprNode { + DangerousSink() { + // File operations + exists(Method m | m.hasName("write") | this.asExpr() = m.getACall()) or + // Database operations + exists(Method m | m.hasName("execute") | this.asExpr() = m.getACall()) or + // Command execution + exists(Method m | m.hasName("exec") | this.asExpr() = m.getACall()) + } +} + +class SanitizerMethod extends DataFlow::ExprNode { + SanitizerMethod() { + exists(Method m | + m.hasName("validateInput") or + m.hasName("sanitize") or + m.hasName("escapeHtml") + | + this.asExpr() = m.getACall() + ) + } +} + +from UserInputSource source, DangerousSink sink +where DataFlow::localFlow(source, sink) +select sink, "User input flows to dangerous sink without sanitization" +``` + +## 🔍 Advanced Query Patterns + +### Performance Optimization Queries +```ql +// .github/codeql/custom-queries/PerformanceIssues.ql +import java +import semmle.code.java.dataflow.DataFlow + +class ExpensiveOperationInLoop extends DataFlow::Configuration { + ExpensiveOperationInLoop() { this = "ExpensiveOperationInLoop" } + + override predicate isSource(DataFlow::Node source) { + exists(Method m | + m.hasName("compile") or + m.hasName("createConnection") or + m.hasName("parseXML") + | + source.asExpr() = m.getACall() + ) + } + + override predicate isSink(DataFlow::Node sink) { + exists(LoopStmt loop | sink.asExpr() = loop.getCondition()) + } +} + +from ExpensiveOperationInLoop config, DataFlow::PathNode source, DataFlow::PathNode sink +where config.hasFlowPath(source, sink) +select sink.getNode(), source, sink, "Expensive operation inside loop may cause performance issues" +``` + +### Security Configuration Queries +```ql +// .github/codeql/custom-queries/SecurityMisconfiguration.ql +import java + +class InsecureConfiguration extends RefType { + InsecureConfiguration() { + this.hasQualifiedName("org.springframework.security.config.annotation.web", "WebSecurityConfigurerAdapter") + } +} + +from InsecureConfiguration config, Method m +where + m.getDeclaringType() = config and + m.overridesOrInstantiates*(m.getDeclaringType().getAMethod()) and + not exists(MethodAccess ma | + ma.getMethod().hasName("csrf") and + ma.getAnArgument().toString().matches("%disable%") + ) +select m, "Security configuration might be missing CSRF protection" +``` + +## 📊 Advanced Analysis Pipeline + +### Multi-Stage Analysis Workflow +```yaml +# .github/workflows/codeql-pipeline.yml +name: "CodeQL Security Pipeline" + +on: + push: + branches: [ main ] + schedule: + - cron: '0 0 * * 0' # Weekly + +jobs: + code-scanning: + name: Code Scanning + uses: ./.github/workflows/codeql-advanced.yml + secrets: inherit + + dependency-scanning: + name: Dependency Vulnerability Scan + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run OWASP Dependency Check + uses: dependency-check/Dependency-Check_Action@main + with: + project: 'my-project' + path: '.' + format: 'HTML' + out: 'reports' + + container-scanning: + name: Container Image Scanning + runs-on: ubuntu-latest + steps: + - name: Run Trivy Vulnerability Scanner + uses: aquasecurity/trivy-action@master + with: + scan-type: 'fs' + scan-ref: '.' + format: 'sarif' + output: 'trivy-results.sarif' + + security-report: + name: Generate Security Report + runs-on: ubuntu-latest + needs: [code-scanning, dependency-scanning, container-scanning] + steps: + - name: Aggregate Results + run: | + python .github/scripts/aggregate-security-report.py + - name: Upload Security Report + uses: actions/upload-artifact@v4 + with: + name: security-report + path: security-report.html +``` + +## 🔒 Advanced Security Queries + +### Authentication & Authorization +```ql +// .github/codeql/custom-queries/AuthBypass.ql +import java + +class AuthenticationBypass extends Method { + AuthenticationBypass() { + this.getAnAnnotation().getType().hasQualifiedName("org.springframework.web.bind.annotation", "RequestMapping") and + not exists(MethodAccess ma | + ma.getMethod().hasName("isAuthenticated") or + ma.getMethod().hasName("hasRole") + ) + } +} + +from AuthenticationBypass method +select method, "Endpoint might be missing authentication check" +``` + +### Cryptography Issues +```ql +// .github/codeql/custom-queries/WeakCrypto.ql +import java + +class WeakCryptoAlgorithm extends MethodAccess { + WeakCryptoAlgorithm() { + this.getMethod().hasName("getInstance") and + ( + this.getArgument(0).toString().matches("%DES%") or + this.getArgument(0).toString().matches("%RC4%") or + this.getArgument(0).toString().matches("%MD5%") + ) + } +} + +from WeakCryptoAlgorithm crypto +select crypto, "Use of weak cryptographic algorithm detected" +``` + +## 📈 Performance Optimization + +### Query Performance Configuration +```yaml +# .github/codeql/performance-config.yml +optimization: + max-paths: 1000 + timeout: 300 + memory: 8192 + +analysis: + mode: deep + threads: 4 + +caching: + enabled: true + location: ./.codeql/cache + cleanup-age: 30d + +logging: + level: INFO + queries: true + performance: true +``` + +### Custom Performance Queries +```ql +// .github/codeql/custom-queries/MemoryLeaks.ql +import java + +class PotentialMemoryLeak extends Method { + PotentialMemoryLeak() { + this.getName().matches("create%") and + not exists(Method destructor | + destructor.getName().matches("destroy%") or + destructor.getName().matches("close%") and + destructor.getDeclaringType() = this.getDeclaringType() + ) + } +} + +from PotentialMemoryLeak method +select method, "Potential memory leak: creation method without corresponding cleanup method" +``` + +## 🎯 Advanced Integration + +### CI/CD Pipeline Integration +```yaml +# .github/workflows/security-gates.yml +name: "Security Quality Gates" + +on: + pull_request: + branches: [ main ] + +jobs: + security-gate: + name: Security Quality Gate + runs-on: ubuntu-latest + steps: + - name: Run CodeQL Analysis + uses: github/codeql-action/analyze@v3 + with: + output: sarif-results + + - name: Check Security Thresholds + run: | + python .github/scripts/check-security-gates.py + + - name: Fail on Critical Issues + if: steps.security-check.outputs.critical_issues > 0 + run: | + echo "❌ Critical security issues found. Blocking merge." + exit 1 +``` + +### Custom Scripts for Analysis +```python +# .github/scripts/aggregate-security-report.py +#!/usr/bin/env python3 +import json +import sys +from pathlib import Path + +def load_sarif_results(file_path): + with open(file_path, 'r') as f: + return json.load(f) + +def analyze_severity_distribution(results): + severity_count = { + 'error': 0, + 'warning': 0, + 'note': 0 + } + + for run in results.get('runs', []): + for result in run.get('results', []): + level = result.get('level', 'warning') + severity_count[level] += 1 + + return severity_count + +def generate_report(severity_distribution): + report = f""" +# Security Analysis Report +## Summary +- Critical Issues: {severity_distribution['error']} +- Warnings: {severity_distribution['warning']} +- Recommendations: {severity_distribution['note']} + +## Quality Gates +{'❌ FAIL' if severity_distribution['error'] > 0 else '✅ PASS'} - No Critical Issues +{'⚠️ WARN' if severity_distribution['warning'] > 10 else '✅ PASS'} - Warning Threshold +""" + return report + +if __name__ == "__main__": + results_dir = Path("sarif-results") + all_results = [] + + for sarif_file in results_dir.glob("*.sarif"): + all_results.extend(load_sarif_results(sarif_file)) + + severity = analyze_severity_distribution(all_results) + report = generate_report(severity) + + with open("security-report.md", "w") as f: + f.write(report) + + print("Security report generated successfully") +``` + +This advanced CodeQL implementation provides: + +## 🚀 Key Features + +1. **Multi-Language Support** - Java, JavaScript, Python, Go, C++ +2. **Custom Query Development** - Advanced security and performance queries +3. **Performance Optimization** - Caching, parallel execution, resource management +4. **Security Quality Gates** - Automated security thresholds and blocking +5. **Comprehensive Reporting** - Aggregated security dashboards +6. **Enterprise Integration** - CI/CD pipeline integration with quality gates + +## 💡 Advanced Use Cases + +- **Custom Security Rules** - Organization-specific security requirements +- **Performance Analysis** - Code performance and memory leak detection +- **Architecture Validation** - Code structure and design pattern validation +- **Compliance Checking** - Regulatory and compliance requirements +- **Dependency Analysis** - Third-party library security assessment + +This setup enables enterprise-grade code security analysis with custom rules, performance optimization, and comprehensive reporting! From fd85c596b7295d977b05595cfbf30071f5c1d0a1 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Tue, 17 Mar 2026 01:02:27 +0800 Subject: [PATCH 6/9] Create Development MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I'll provide a comprehensive and well-documented version of the `first-responder-v2-prs-collect.yml` workflow. This version includes detailed comments, additional steps for robustness, and follows best practices for GitHub Actions. ```yaml # .github/workflows/first-responder-v2-prs-collect.yml # Collects pull requests for first responder v2 and sends notifications to Slack. # Runs every 15 minutes during working hours (Mon-Fri, 9am-5pm UTC). # Also allows manual trigger via workflow_dispatch. name: First Responder v2 PRs Collect on: schedule: # Cron syntax: minute hour day-of-month month day-of-week # Every 15 minutes between 9:00 and 16:45 UTC (last run at 16:45) # Only on weekdays (1-5 = Monday-Friday) - cron: "*/15 9-16 * * 1-5" workflow_dispatch: # Allow manual triggering with optional inputs (if needed) inputs: debug: description: 'Enable debug logging' required: false default: false type: boolean # Set permissions for the GITHUB_TOKEN permissions: contents: read pull-requests: read issues: read jobs: collect-prs: name: Collect PRs and notify Slack runs-on: ubuntu-latest # Add timeout to prevent runaway jobs timeout-minutes: 10 steps: # Step 1: Check out the repository code - name: Checkout repository uses: actions/checkout@v3 with: fetch-depth: 1 # Shallow clone for speed # Step 2: Set up Node.js environment - name: Setup Node.js uses: actions/setup-node@v3 with: node-version: '18' cache: 'npm' # Cache npm dependencies for faster installs # Step 3: Install dependencies from package-lock.json - name: Install dependencies run: npm ci # npm ci is faster and uses exact versions from package-lock.json # Step 4: Run PR collection script with environment variables - name: Run PR collection script env: # GitHub token automatically provided by GitHub Actions GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Slack bot token required to post messages SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} # Slack channel IDs for announcements and join channel SLACK_ANNOUNCEMENT_CHANNEL_ID: ${{ secrets.SLACK_ANNOUNCEMENT_CHANNEL_ID }} SLACK_JOIN_CHANNEL_ID: ${{ secrets.SLACK_JOIN_CHANNEL_ID }} # Optional debug flag if manual trigger with debug DEBUG: ${{ github.event.inputs.debug && 'true' || '' }} run: node src/scripts/first-responder-v2/prs-collect.js # Step 5: (Optional) Notify on failure - name: Notify Slack on failure if: failure() env: SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} SLACK_ANNOUNCEMENT_CHANNEL_ID: ${{ secrets.SLACK_ANNOUNCEMENT_CHANNEL_ID }} run: | curl -X POST -H "Authorization: Bearer $SLACK_BOT_TOKEN" \ -H "Content-type: application/json" \ --data "{ \"channel\": \"$SLACK_ANNOUNCEMENT_CHANNEL_ID\", \"text\": \":x: First Responder v2 PR collection failed at $(date -u +'%Y-%m-%d %H:%M:%S UTC')\" }" \ https://slack.com/api/chat.postMessage ``` **Key Features of this Updated YAML:** - **Detailed comments** explaining each section. - **Permissions block** to restrict token scope for security. - **Timeout** to prevent long-running jobs. - **Caching** for npm dependencies to speed up runs. - **Manual trigger** with optional `debug` input. - **Failure notification** step to alert Slack if the script fails. - **Environment variables** clearly listed. Make sure you have the required secrets set in your repository: - `SLACK_BOT_TOKEN` - `SLACK_ANNOUNCEMENT_CHANNEL_ID` - `SLACK_JOIN_CHANNEL_ID` If you need further adjustments or have a different file in mind, let me know! Tentu, saya akan memberikan lebih banyak detail dan konteks terkait workflow `first-responder-v2-prs-collect.yml` serta komponen-komponennya. Berikut adalah penjelasan mendalam, tips penggunaan, dan kode tambahan untuk memaksimalkan workflow ini. ## 1. Penjelasan Mendalam tentang Cron Schedule Workflow ini menggunakan cron expression `*/15 9-16 * * 1-5`. Mari kita uraikan: - `*/15` → Setiap 15 menit (menit 0,15,30,45). - `9-16` → Jam 9 pagi hingga 4 sore UTC (jam 16). Karena cron menjalankan pada menit ke-0,15,30,45 di jam tersebut, maka run terakhir dimulai pukul 16:45 UTC (setara pukul 23:45 WIB jika UTC+7). - `*` → Setiap hari dalam bulan. - `*` → Setiap bulan. - `1-5` → Senin sampai Jumat (1=Senin, 5=Jumat). ### Mengubah ke Zona Waktu Lokal Jika Anda ingin menjalankan workflow sesuai jam kerja di Indonesia (misal 09:00–17:00 WIB), Anda bisa menyesuaikan cron dengan mempertimbangkan offset UTC. Contoh untuk WIB (UTC+7): ```yaml # 09:00–17:00 WIB = 02:00–10:00 UTC on: schedule: - cron: "*/15 2-9 * * 1-5" # Setiap 15 menit jam 2–9 UTC ``` Perhatikan bahwa jam 10:00 UTC = 17:00 WIB, tapi menit terakhir di jam 9 adalah 09:45 UTC = 16:45 WIB. Jika ingin sampai jam 17:00 WIB (10:00 UTC), Anda perlu menjadwalkan di jam 10 juga, tapi hanya di menit 0 agar tidak berjalan setelah jam 17:00 WIB. Anda bisa menggunakan multiple cron lines: ```yaml on: schedule: - cron: "*/15 2-9 * * 1-5" # 09:00–16:45 WIB - cron: "0 10 * * 1-5" # 17:00 WIB tepat ``` ## 2. Variabel Lingkungan dan Secret yang Diperlukan Workflow ini menggunakan tiga Slack channel ID: - `SLACK_ANNOUNCEMENT_CHANNEL_ID`: Channel untuk pengumuman penting, misalnya ringkasan PR yang dikumpulkan. - `SLACK_JOIN_CHANNEL_ID`: Channel khusus untuk mengundang anggota baru atau notifikasi onboarding. - `SLACK_BOT_TOKEN`: Token bot Slack dengan hak akses `chat:write` dan `channels:join` (jika perlu mengundang bot ke channel). **Cara mendapatkan Slack Bot Token dan Channel ID:** 1. Buat aplikasi Slack di [api.slack.com/apps](https://api.slack.com/apps). 2. Tambahkan scope `chat:write` dan `channels:join` (atau `groups:write` untuk private channel). 3. Install aplikasi ke workspace Anda. 4. Salin Bot User OAuth Token (mulai dengan `xoxb-`). 5. Untuk mendapatkan Channel ID, buka channel di Slack, klik nama channel di bagian atas, dan sali ID dari URL (biasanya diawali `C` atau `G`). Tambahkan semua nilai tersebut sebagai **repository secrets** di GitHub (Settings > Secrets and variables > Actions). ## 3. Script `prs-collect.js` – Contoh Sederhana Workflow ini menjalankan script Node.js. Berikut adalah contoh minimal script yang mungkin digunakan (anda dapat menyesuaikan dengan logika Anda sendiri): ```javascript // src/scripts/first-responder-v2/prs-collect.js const { Octokit } = require('@octokit/rest'); const { WebClient } = require('@slack/web-api'); // Inisialisasi klien const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN }); const slack = new WebClient(process.env.SLACK_BOT_TOKEN); async function collectPRs() { try { // Contoh: ambil PR terbuka di repo tertentu const { data: pullRequests } = await octokit.pulls.list({ owner: 'sazwan96', repo: 'sazwanismail', // ganti dengan repo Anda state: 'open' }); console.log(`Ditemukan ${pullRequests.length} PR terbuka.`); // Kirim notifikasi ke Slack if (pullRequests.length > 0) { const message = { channel: process.env.SLACK_ANNOUNCEMENT_CHANNEL_ID, text: `Ada ${pullRequests.length} PR baru yang perlu direview!`, blocks: [ { type: 'section', text: { type: 'mrkdwn', text: `*Daftar PR Terbuka:*` } }, ...pullRequests.map(pr => ({ type: 'section', text: { type: 'mrkdwn', text: `<${pr.html_url}|#${pr.number}: ${pr.title}> oleh @${pr.user.login}` } })) ] }; await slack.chat.postMessage(message); } else { // Opsional: kirim pesan bahwa tidak ada PR await slack.chat.postMessage({ channel: process.env.SLACK_ANNOUNCEMENT_CHANNEL_ID, text: 'Tidak ada PR terbuka saat ini.' }); } // Kirim pesan ke join channel (misal untuk onboarding) await slack.chat.postMessage({ channel: process.env.SLACK_JOIN_CHANNEL_ID, text: 'Workflow PR collection berjalan dengan sukses.' }); } catch (error) { console.error('Error:', error); // Opsional: kirim notifikasi error ke Slack await slack.chat.postMessage({ channel: process.env.SLACK_ANNOUNCEMENT_CHANNEL_ID, text: `Terjadi error: ${error.message}` }); process.exit(1); } } collectPRs(); ``` Pastikan Anda menginstal dependencies yang diperlukan: ```bash npm install @octokit/rest @slack/web-api ``` ## 4. Menambahkan Logging dan Monitoring Untuk memudahkan debugging, Anda dapat menambahkan step yang meng-upload log sebagai artifact jika terjadi error: ```yaml - name: Upload logs on failure if: failure() uses: actions/upload-artifact@v3 with: name: error-logs path: /home/runner/.npm/_logs/ retention-days: 5 ``` ## 5. Menambahkan Step untuk Menjalankan Test Sebelum Collect Jika Anda ingin memastikan script berfungsi sebelum menjalankan koleksi PR, Anda bisa menambahkan step test: ```yaml - name: Run tests run: npm test ``` ## 6. Menjadwalkan dengan Input Manual yang Lebih Fleksibel Anda dapat memperluas `workflow_dispatch` dengan input untuk memilih repositori target atau mode debug: ```yaml on: workflow_dispatch: inputs: repo: description: 'Repository to scan (format: owner/repo)' required: false default: 'sazwan96/sazwanismail' debug: description: 'Enable debug logging' required: false default: false type: boolean ``` Kemudian di script, baca input tersebut: ```javascript const repoFullName = process.env.INPUT_REPO || 'sazwan96/sazwanismail'; const [owner, repo] = repoFullName.split('/'); ``` ## 7. Keamanan: Menggunakan GitHub App Token Jika Anda membutuhkan akses yang lebih aman atau spesifik, pertimbangkan untuk menggunakan GitHub App token daripada `GITHUB_TOKEN`. Anda bisa menggunakan action `tibdex/github-app-token` untuk menghasilkan token dengan izin terbatas. ## 8. Alternatif: Menggunakan Action `schedule` dengan If Condition Jika Anda ingin mengontrol eksekusi berdasarkan kondisi tertentu (misal hanya jika ada perubahan file tertentu), Anda bisa menambahkan conditional step: ```yaml - name: Check if script changed id: changed uses: tj-actions/changed-files@v35 with: files: | src/scripts/first-responder-v2/prs-collect.js .github/workflows/first-responder-v2-prs-collect.yml - name: Run PR collection only if relevant files changed if: steps.changed.outputs.any_changed == 'true' run: node src/scripts/first-responder-v2/prs-collect.js ``` ## 9. Integrasi dengan Database atau API Eksternal Jika Anda perlu menyimpan hasil koleksi PR ke database atau memanggil API eksternal, tambahkan step tersebut di dalam script atau sebagai step terpisah. ## 10. Notifikasi ke Multiple Channel Anda bisa mengirim notifikasi ke lebih dari dua channel dengan menambahkan variabel lingkungan tambahan: ```yaml env: SLACK_CHANNEL_1: ${{ secrets.SLACK_CHANNEL_1 }} SLACK_CHANNEL_2: ${{ secrets.SLACK_CHANNEL_2 }} ``` Lalu di script, Anda bisa memposting ke semua channel yang didefinisikan. --- Dengan semua tambahan ini, workflow Anda menjadi lebih powerful, fleksibel, dan mudah dipelihara. Jika ada bagian spesifik yang ingin didalami atau ditambahkan, beri tahu saya! --- Development | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 Development diff --git a/Development b/Development new file mode 100644 index 000000000000..c67b0dc3a9fe --- /dev/null +++ b/Development @@ -0,0 +1,168 @@ +# Contributing to [Project Name] + +Thank you for your interest in contributing to [Project Name]! This document provides comprehensive guidelines and instructions for contributing effectively. Please read it carefully before submitting issues or pull requests. + +## Table of Contents +- [Code of Conduct](#code-of-conduct) +- [Getting Started](#getting-started) +- [Development Process](#development-process) +- [Coding Standards](#coding-standards) +- [Building and Testing on Different Platforms](#building-and-testing-on-different-platforms) + - [Linux (including sanitizers)](#linux-including-sanitizers) + - [macOS 14 Native](#macos-14-native) + - [Windows Native and Visual Studio](#windows-native-and-visual-studio) + - [CI / Windows Test](#ci--windows-test) +- [Testing Guidelines](#testing-guidelines) +- [Submitting Changes](#submitting-changes) +- [Communication](#communication) +- [License](#license) + +## Code of Conduct +This project adheres to the [Bitcoin Code of Conduct](https://bitcoincore.org/en/conduct/). By participating, you are expected to uphold this code. + +## Getting Started +1. Fork the repository and clone your fork. +2. Set up the development environment as described in the [README](README.md). +3. Find an issue to work on or propose a new feature by opening an issue. + +## Development Process +- All development happens on the `master` branch. Pull requests (PRs) should be made against `master`. +- For significant changes, open an issue first to discuss the proposed change. +- Keep PRs focused on a single topic. Avoid mixing unrelated changes. +- Commit messages should be clear and follow the [conventional format](https://www.conventionalcommits.org/). + +## Coding Standards +- **Language:** The project is written in C++17 (or specify your language). +- **Style:** Follow the existing code style. Use `clang-format` with the provided `.clang-format` configuration. +- **Documentation:** Document all public interfaces, complex logic, and any non-obvious behavior. +- **Linting:** Run `make lint` (or equivalent) to check for style violations. +- **Security:** Avoid unsafe patterns. Be mindful of memory safety, thread safety, and constant-time operations for cryptographic code. + +## Building and Testing on Different Platforms +To ensure cross-platform compatibility, we require that contributions be tested on at least one of the following platforms, with additional sanitizer checks where applicable. Detailed instructions for each platform are provided below. + +### Linux (including sanitizers) +- **Build Dependencies:** Install required packages (e.g., `build-essential`, `libtool`, `autotools-dev`, `automake`, `pkg-config`, `libssl-dev`, `libevent-dev`, `libboost-system-dev`, `libboost-filesystem-dev`, `libboost-thread-dev`, `libsqlite3-dev`, `libdb++-dev`). +- **Build with Autotools:** + ```bash + ./autogen.sh + ./configure + make -j$(nproc) + ``` +- **Run Unit Tests:** + ```bash + make check + ``` +- **Run Functional Tests:** + ```bash + test/functional/test_runner.py + ``` +- **Sanitizers (ASan + LSan):** + To enable AddressSanitizer and LeakSanitizer, configure with: + ```bash + ./configure --enable-asan CXXFLAGS="-fsanitize=address,leak -g" CFLAGS="-fsanitize=address,leak -g" + make clean + make -j$(nproc) + ``` + Then run tests with the sanitizer-enabled binary. Memory errors and leaks will be reported. +- **Other Sanitizers:** You may also enable ThreadSanitizer (`--enable-tsan`) or UndefinedBehaviorSanitizer (`--enable-ubsan`) as needed. + +### macOS 14 Native +- **Prerequisites:** Install Xcode Command Line Tools: + ```bash + xcode-select --install + ``` +- **Install Dependencies via Homebrew:** + ```bash + brew install automake libtool boost pkg-config libevent sqlite3 berkeley-db + ``` +- **Build with Autotools:** + ```bash + ./autogen.sh + ./configure + make -j$(sysctl -n hw.logicalcpu) + ``` +- **Run Tests:** + ```bash + make check + test/functional/test_runner.py + ``` +- **Note:** macOS 14 (Sonoma) is the primary target; older versions may work but are not officially supported. + +### Windows Native and Visual Studio +We support two build methods on Windows: **MinGW-w64 (MSYS2)** for a Unix-like environment and **Visual Studio** for native Windows development. + +#### MinGW-w64 (MSYS2) +1. Install MSYS2 from [msys2.org](https://www.msys2.org/). +2. Open an MSYS2 terminal and update packages: + ```bash + pacman -Syu + ``` +3. Install build dependencies: + ```bash + pacman -S --needed base-devel mingw-w64-x86_64-toolchain mingw-w64-x86_64-autotools mingw-w64-x86_64-boost mingw-w64-x86_64-libevent mingw-w64-x86_64-sqlite3 mingw-w64-x86_64-berkeley-db + ``` +4. Build using the Mingw-w64 environment: + ```bash + ./autogen.sh + ./configure --host=x86_64-w64-mingw32 + make -j$(nproc) + ``` +5. Run tests (if possible; some functional tests may require Python in the MSYS2 environment). + +#### Visual Studio +1. Install Visual Studio 2022 or later with **Desktop development with C++** workload. +2. Install **vcpkg** (if not already installed) and integrate it with Visual Studio: + ```bash + git clone https://github.com/Microsoft/vcpkg.git + cd vcpkg + .\bootstrap-vcpkg.bat + .\vcpkg integrate install + ``` +3. Install required dependencies via vcpkg: + ```bash + .\vcpkg install boost pthread libevent openssl sqlite3 berkeley-db + ``` +4. Open the solution file `build_msvc/bitcoin.sln` (or equivalent) in Visual Studio. +5. Build the solution (select appropriate configuration: Debug/Release, x64). +6. Run tests via Test Explorer or build the test projects directly. + +### CI / Windows Test +- Our continuous integration (CI) pipeline runs on GitHub Actions (or similar). See `.github/workflows/` for configuration. +- Windows CI jobs automatically build and test the project using both MSYS2 and Visual Studio environments. +- Before submitting a PR, ensure that all Windows CI checks pass. If you cannot test locally, rely on the CI results. +- For Windows-specific changes, consider adding or updating tests in the CI configuration (e.g., adding a new test file to `test/functional/`). + +## Testing Guidelines +- **Unit Tests:** Write unit tests for new code using the [Boost.Test](https://www.boost.org/doc/libs/1_81_0/libs/test/doc/html/index.html) framework. Place them in `src/test/` and add to the appropriate `Makefile.test` or `CMakeLists.txt`. +- **Functional Tests:** Add Python-based functional tests in `test/functional/` for end-to-end scenarios. Run them with `test/functional/test_runner.py`. +- **Coverage:** Ensure your changes are covered by tests. If you add new features or fix bugs, include tests that verify correctness. +- **Consensus Changes:** If your change affects consensus rules, include extensive tests and consider running against the Bitcoin Core test suite if applicable. + +## Submitting Changes +1. Ensure your code is rebased on the latest `master`. +2. Run the full test suite and linters: + ```bash + make check + test/functional/test_runner.py + make lint + ``` +3. Sign off your commits to certify that you have the right to submit the code under the project's license. Use `git commit -s` to add a Signed-off-by line. +4. Push your changes to your fork and open a pull request against `master`. +5. In the PR description, explain the changes, reference any related issues, and note any breaking changes. +6. Be responsive to feedback. Update your PR as needed. + +### Developer Certificate of Origin (DCO) +We require that all contributors sign the [Developer Certificate of Origin](https://developercertificate.org/). By signing off your commits (`git commit -s`), you attest that you have the right to submit the code. + +## Communication +- **GitHub Issues:** For bug reports, feature requests, and questions. +- **Mailing List:** [bitcoin-dev](https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev) for protocol-level discussions. +- **IRC/Matrix:** Join `#bitcoin-core-dev` on Libera.Chat or the [Bitcoin Core Matrix channel](https://matrix.to/#/#bitcoin-core-dev:libera.chat). + +## License +By contributing, you agree that your contributions will be licensed under the same license as the project (e.g., MIT). See [LICENSE](LICENSE) for details. + +--- + +Thank you for contributing to [Project Name] and helping advance Bitcoin development! From c624e1b1e098a8d46ad32a08e23892ba1a8f6003 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Tue, 17 Mar 2026 01:18:52 +0800 Subject: [PATCH 7/9] Update cmake-single-platform.yml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Updated CMake Single Platform GitHub Actions Workflow This repository provides an up-to-date GitHub Actions workflow for building CMake projects on a single platform (Linux, macOS, or Windows). It includes: - CMake configuration and build - Caching for dependencies (vcpkg, ccache) - Multiple build types (Debug, Release) - Running tests with CTest - Uploading build artifacts - Code formatting and linting (optional) ## 📄 Workflow File: `.github/workflows/cmake-single-platform.yml` ```yaml name: CMake Single Platform Build on: push: branches: [ main, develop ] pull_request: branches: [ main ] workflow_dispatch: # Allow manual trigger env: BUILD_TYPE: Release # Customize CMake build type if needed jobs: build: # Choose the runner: ubuntu-latest, windows-latest, or macos-latest runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v4 # ===== SETUP DEPENDENCIES ===== - name: Install Linux dependencies (if any) if: runner.os == 'Linux' run: | sudo apt-get update sudo apt-get install -y ninja-build ccache # Add other packages as needed - name: Install macOS dependencies if: runner.os == 'macOS' run: | brew install ninja ccache # Add other packages - name: Install Windows dependencies (using Chocolatey) if: runner.os == 'Windows' run: | choco install ninja ccache # Or use vcpkg (see below) # ===== CACHE MANAGEMENT ===== - name: Cache ccache uses: actions/cache@v4 with: path: ~/.ccache key: ${{ runner.os }}-ccache-${{ github.sha }} restore-keys: ${{ runner.os }}-ccache- - name: Cache vcpkg (if used) if: false # Enable if you use vcpkg uses: actions/cache@v4 with: path: | ~/.cache/vcpkg build/vcpkg_installed key: ${{ runner.os }}-vcpkg-${{ hashFiles('**/vcpkg.json') }} restore-keys: ${{ runner.os }}-vcpkg- # ===== CONFIGURE & BUILD ===== - name: Configure CMake run: | cmake -B build -S . \ -DCMAKE_BUILD_TYPE=${{ env.BUILD_TYPE }} \ -G Ninja \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache env: CC: clang # Override compiler if needed (gcc, clang, cl) CXX: clang++ - name: Build run: cmake --build build --config ${{ env.BUILD_TYPE }} --parallel # ===== TEST ===== - name: Test working-directory: build run: ctest -C ${{ env.BUILD_TYPE }} --output-on-failure --parallel # ===== CODE QUALITY (optional) ===== - name: Run clang-format lint if: false # Enable if you want formatting checks uses: jidicula/clang-format-action@v4.11.0 with: clang-format-version: '16' check-path: 'src' # ===== ARTIFACTS ===== - name: Upload build artifacts uses: actions/upload-artifact@v4 with: name: ${{ runner.os }}-${{ env.BUILD_TYPE }}-binaries path: | build/bin build/lib build/*.exe build/*.dll build/*.so build/*.dylib if-no-files-found: ignore ``` ## 🔧 Customization Tips 1. **Runner OS**: Change `runs-on` to `windows-latest` or `macos-latest` as needed. 2. **Dependencies**: Adjust package installation steps for your specific libraries. 3. **vcpkg**: If your project uses vcpkg, enable the vcpkg cache step and install vcpkg in a setup step. 4. **Compiler**: Override `CC` and `CXX` environment variables to use different compilers (e.g., `gcc`, `clang`, `msvc`). 5. **Build Types**: You can matrix over `BUILD_TYPE` to build both Debug and Release, or add a strategy matrix. 6. **Artifacts**: Customize the artifact paths to match your output locations. ## 📦 Example with vcpkg and Matrix If you need multiple build types or configurations, extend with a matrix: ```yaml jobs: build: runs-on: ${{ matrix.os }} strategy: matrix: os: [ubuntu-latest, windows-latest, macos-latest] build_type: [Debug, Release] env: BUILD_TYPE: ${{ matrix.build_type }} steps: # ... steps (use matrix.os and matrix.build_type) ``` ## ✅ Best Practices Included - **Caching** with ccache and vcpkg speeds up rebuilds. - **Ninja** generator for faster builds. - **Parallel** builds and tests. - **Artifact** upload for easy access to binaries. - **Manual trigger** (`workflow_dispatch`) for ad-hoc runs. ## 🔗 References - [GitHub Actions Documentation](https://docs.github.com/actions) - [CMake Documentation](https://cmake.org/cmake/help/latest/) - [vcpkg with GitHub Actions](https://vcpkg.io/en/getting-started.html) --- **Maintainer:** Your Team **License:** MIT # Updated Advanced CMake Single/Multi-Platform Workflow This workflow provides a robust CI pipeline for CMake projects, supporting multiple operating systems and build configurations. It includes caching, testing, code coverage, static analysis, and artifact upload. ## 📄 `.github/workflows/cmake-advanced.yml` ```yaml name: CMake Advanced CI on: push: branches: [ main, develop ] pull_request: branches: [ main ] workflow_dispatch: # Allow manual trigger env: # Global build type; can be overridden per matrix BUILD_TYPE: Release jobs: build: name: ${{ matrix.os }} / ${{ matrix.build_type }} runs-on: ${{ matrix.os }} strategy: fail-fast: false # Continue other jobs if one fails matrix: os: [ubuntu-latest, windows-latest, macos-latest] build_type: [Debug, Release] # Optionally exclude some combinations # exclude: # - os: windows-latest # build_type: Debug steps: - name: Checkout code uses: actions/checkout@v4 # ===== DEPENDENCY INSTALLATION ===== - name: Install Linux dependencies if: runner.os == 'Linux' run: | sudo apt-get update sudo apt-get install -y \ ninja-build \ ccache \ lcov \ clang-tidy \ curl \ zip # Add project-specific packages here - name: Install macOS dependencies if: runner.os == 'macOS' run: | brew install \ ninja \ ccache \ llvm # provides clang-tidy, lcov # Ensure llvm binaries are in PATH echo "$(brew --prefix llvm)/bin" >> $GITHUB_PATH - name: Install Windows dependencies if: runner.os == 'Windows' run: | choco install ninja ccache # vcpkg is usually installed on GitHub runners; if needed, bootstrap: # git clone https://github.com/Microsoft/vcpkg.git # .\vcpkg\bootstrap-vcpkg.bat # echo "${{ github.workspace }}/vcpkg" >> $GITHUB_PATH # ===== CACHE SETUP ===== - name: Cache ccache uses: actions/cache@v4 with: path: ~/.ccache key: ${{ runner.os }}-ccache-${{ matrix.build_type }}-${{ github.sha }} restore-keys: | ${{ runner.os }}-ccache-${{ matrix.build_type }}- ${{ runner.os }}-ccache- - name: Cache vcpkg (if used) if: false # Enable if you use vcpkg.json manifest mode uses: actions/cache@v4 with: path: | ~/.cache/vcpkg ${{ github.workspace }}/build/vcpkg_installed key: ${{ runner.os }}-vcpkg-${{ hashFiles('**/vcpkg.json') }} restore-keys: ${{ runner.os }}-vcpkg- # ===== CONFIGURE CMAKE ===== - name: Configure CMake shell: bash run: | cmake -B build -S . \ -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} \ -G Ninja \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON # for clang-tidy # ===== BUILD ===== - name: Build run: cmake --build build --config ${{ matrix.build_type }} --parallel # ===== RUN TESTS ===== - name: Test working-directory: build run: ctest -C ${{ matrix.build_type }} --output-on-failure --parallel # ===== STATIC ANALYSIS (clang-tidy) ===== - name: Run clang-tidy if: runner.os == 'Linux' && matrix.build_type == 'Debug' # Run once to avoid duplication working-directory: build run: | # Adjust source directory and checks as needed run-clang-tidy -p . -extra-arg=-Wno-unknown-warning-option -quiet # ===== CODE COVERAGE (Linux only) ===== - name: Generate coverage report if: runner.os == 'Linux' && matrix.build_type == 'Debug' && github.event_name == 'push' run: | # Ensure you have built with coverage flags: -fprofile-arcs -ftest-coverage lcov --directory . --capture --output-file coverage.info lcov --remove coverage.info '/usr/*' --output-file coverage.info lcov --list coverage.info working-directory: build - name: Upload coverage to Codecov if: runner.os == 'Linux' && matrix.build_type == 'Debug' && github.event_name == 'push' uses: codecov/codecov-action@v4 with: files: build/coverage.info flags: unittests name: codecov-umbrella fail_ci_if_error: false token: ${{ secrets.CODECOV_TOKEN }} # ===== UPLOAD ARTIFACTS ===== - name: Prepare artifacts shell: bash run: | mkdir -p artifacts # Copy binaries, libraries, etc. if [ -d build/bin ]; then cp -r build/bin artifacts/; fi if [ -d build/lib ]; then cp -r build/lib artifacts/; fi # Include compile_commands.json for debugging cp build/compile_commands.json artifacts/ || true - name: Upload build artifacts uses: actions/upload-artifact@v4 with: name: ${{ runner.os }}-${{ matrix.build_type }}-artifacts path: artifacts/ ``` ## 🚀 Key Features - **Matrix Build**: Builds on Ubuntu, Windows, and macOS with both Debug and Release configurations. - **Caching**: Uses `ccache` to speed up rebuilds; vcpkg cache ready. - **Dependencies**: Installs `ninja`, `ccache`, and platform-specific tools. - **Static Analysis**: Runs `clang-tidy` (on Linux/Debug once). - **Code Coverage**: Generates and uploads coverage reports to Codecov (Linux/Debug only). - **Artifacts**: Uploads binaries, libraries, and `compile_commands.json`. ## 🔧 Customization - **Add more packages**: Modify the dependency installation steps. - **Adjust coverage**: Ensure your CMake project enables coverage flags when `CMAKE_BUILD_TYPE` is Debug. - **vcpkg**: Uncomment the cache step and add a vcpkg installation step if needed. - **Compiler**: Override `CC`/`CXX` in the configure step to use specific compilers. - **Artifacts**: Modify the `Prepare artifacts` step to capture your desired output. ## ✅ Best Practices - **fail-fast: false** allows all matrix jobs to run even if one fails. - **Conditional steps** avoid redundant work (e.g., coverage only once). - **Caching keys** use both OS and build type to avoid mixing caches. - **Parallel** builds and tests reduce CI time. ## 📚 References - [GitHub Actions Documentation](https://docs.github.com/actions) - [CMake Documentation](https://cmake.org/documentation) - [Codecov Action](https://github.com/codecov/codecov-action) - [clang-tidy Integration](https://clang.llvm.org/extra/clang-tidy/) --- **Maintainer:** Your Team **License:** MIT --- .github/workflows/cmake-single-platform.yml | 378 +++++++++++++------- 1 file changed, 248 insertions(+), 130 deletions(-) diff --git a/.github/workflows/cmake-single-platform.yml b/.github/workflows/cmake-single-platform.yml index a0cb60102e27..0e827eb3523d 100644 --- a/.github/workflows/cmake-single-platform.yml +++ b/.github/workflows/cmake-single-platform.yml @@ -1,147 +1,265 @@ -Here's an updated `cmake-single-platform.yml` workflow that builds your CMake project with SLSA Level 3 provenance and enhanced security: +We'll set up a GitHub Actions workflow to deploy your Nuxt application to GitHub Pages. The example below uses the official `github_pages` preset for optimal compatibility. + +### 📁 Workflow file: `.github/workflows/deploy.yml` ```yaml -name: CMake Build (Single Platform) with SLSA L3 +name: Deploy Nuxt to GitHub Pages on: push: - branches: [main] - pull_request: - release: - types: [published] - workflow_dispatch: - -permissions: - id-token: write # OIDC token for Sigstore signing - contents: read # Minimal read-only access - packages: write # Only needed if publishing packages + branches: [main] # Trigger on pushes to main branch + workflow_dispatch: # Allow manual trigger jobs: - cmake-build: + build: runs-on: ubuntu-latest - outputs: - base64-subjects: ${{ steps.hashes.outputs.base64_subjects }} - artifacts-name: artifacts-${{ github.run_id }} - steps: - - name: Checkout code - uses: actions/checkout@v4 + - uses: actions/checkout@v4 + - run: corepack enable # Ensures pnpm/yarn are available if needed + - uses: actions/setup-node@v4 with: - fetch-depth: 0 # Required for full commit history in provenance - - - name: Install dependencies - run: sudo apt-get update && sudo apt-get install -y build-essential cmake - - - name: Configure CMake - run: cmake -B build -DCMAKE_BUILD_TYPE=Release - - - name: Build project - run: cmake --build build --config Release --parallel 4 - - - name: Create artifacts directory - run: mkdir -p artifacts - - - name: Collect binaries - run: | - find build -type f -executable -exec cp {} artifacts/ \; - # Add other artifacts as needed (libraries, config files, etc.) - - - name: Generate artifact hashes - id: hashes - run: | - cd artifacts - subjects="[]" - for file in *; do - sha=$(sha256sum "$file" | awk '{print $1}') - subjects=$(jq -c \ - --arg name "$file" \ - --arg sha "sha256:$sha" \ - '. += [{"name": $name, "digest": $sha}]' \ - <<< "$subjects") - done - echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT - - - name: Upload artifacts - uses: actions/upload-artifact@v4 + node-version: 20 + - run: npm ci # Clean install (use `npm install` if you prefer) + - run: npx nuxt build --preset github_pages + - name: Upload Pages artifact + uses: actions/upload-pages-artifact@v3 with: - name: ${{ steps.hashes.outputs.artifacts-name }} - path: artifacts/ - retention-days: 5 # Auto-clean old artifacts + path: ./.output/public - provenance: - needs: [cmake-build] - uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0 + deploy: + needs: build permissions: - id-token: write # For Sigstore signing - contents: write # For release assets - actions: read # For reusable workflow - with: - base64-subjects: ${{ needs.cmake-build.outputs.base64-subjects }} - upload-artifacts-name: ${{ needs.cmake-build.outputs.artifacts-name }} - upload-assets: ${{ github.event_name == 'release' && github.event.action == 'published' }} - secrets: inherit - - # Optional: Add package publishing step here if needed - # publish: - # needs: [provenance] - # runs-on: ubuntu-latest - # steps: - # - name: Download artifacts - # uses: actions/download-artifact@v4 - # with: - # name: ${{ needs.cmake-build.outputs.artifacts-name }} - # - # # Add your package publishing commands here + pages: write + id-token: write + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + runs-on: ubuntu-latest + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 +``` + +### ⚙️ Required Nuxt configuration + +In your `nuxt.config.ts` (or `.js`), set the `baseURL` to match your repository name. Also avoid underscores in the build assets directory to prevent issues with GitHub Pages' Jekyll processing. + +```ts +export default defineNuxtConfig({ + app: { + baseURL: '/your-repo-name/', // Replace with your repository name + buildAssetsDir: 'assets' // Do NOT start with '_' + } +}) +``` + +### ✅ Prerequisites + +1. **Enable GitHub Pages** with **GitHub Actions** as the source: + Go to your repository **Settings → Pages → Build and deployment → Source** and select **GitHub Actions**. + +2. Ensure your default branch name matches the trigger (`main` in the example). Adjust if your branch is `master` or something else. + +3. Commit and push the workflow file and the updated `nuxt.config`. The next push to `main` will trigger a deployment. + +> If you were looking for a CMake workflow (you mentioned `cmake-single-platform.yml`), please clarify – but for Nuxt deployment to GitHub Pages, the above should work perfectly. +It looks like you're asking about setting up a **large database** in the context of your Nuxt project and GitHub Actions. I'll cover two common scenarios: + +1. **Using a database during CI/CD** (e.g., running tests that need a real database). +2. **Connecting your deployed Nuxt app to a production database** (a “big” managed database). + +--- + +## 1. Database in GitHub Actions (for testing/CI) + +If you need a database to run tests or seed data during your workflow, you can use GitHub Actions **service containers**. Below is an example that adds a PostgreSQL service to the `build` job of the Nuxt deployment workflow. + +### Example workflow with PostgreSQL + +```yaml +name: Deploy Nuxt + DB tests + +on: + push: + branches: [main] + +jobs: + build: + runs-on: ubuntu-latest + services: + postgres: + image: postgres:15 + env: + POSTGRES_USER: testuser + POSTGRES_PASSWORD: testpass + POSTGRES_DB: testdb + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + ports: + - 5432:5432 + steps: + - uses: actions/checkout@v4 + - run: corepack enable + - uses: actions/setup-node@v4 + with: + node-version: 20 + - run: npm ci + # Run database migrations / seeding + - run: npx prisma migrate deploy # if using Prisma + - run: npm run test # tests that use the DB + - run: npx nuxt build --preset github_pages + - uses: actions/upload-pages-artifact@v3 + with: + path: ./.output/public + + deploy: # same as before + # ... ``` -### Key Features: - -1. **Secure CMake Build**: - - Minimal dependencies installation - - Release-mode builds by default - - Parallel compilation (`--parallel 4`) - - Explicit artifact collection - -2. **SLSA L3 Provenance**: - - Uses official SLSA generator v1.9.0 - - Full non-falsifiable build attestations - - Automatic signature via Sigstore - - Includes all build parameters and environment details - -3. **Artifact Security**: - - Unique artifact names using `run_id` to prevent collisions - - SHA256 hashing of all binaries - - 5-day auto-cleanup of artifacts - - Base64-encoded subject manifest - -4. **Release Integration**: - - Automatic asset upload only for published releases - - Prevents accidental publishing during PRs - - Manual trigger support (`workflow_dispatch`) - -5. **Minimal Permissions**: - - `id-token: write` only for provenance job - - `contents: read` for most jobs - - Explicit package write permission - -### How to Use: -1. Place this file in `.github/workflows/cmake-single-platform.yml` -2. Adjust these sections as needed: - - **Dependencies**: Add any required packages in `Install dependencies` - - **CMake Flags**: Modify `Configure CMake` step with your flags - - **Artifacts**: Update `Collect binaries` to match your output files - - **Publishing**: Uncomment and configure the publish job if needed - -3. For multi-platform support, duplicate the `cmake-build` job with different `runs-on` values and matrix strategy - -### Verification: -After a release, verify provenance with: -```bash -slsa-verifier verify-artifact \ - --provenance-path provenance.json \ - --source-uri github.com/$YOUR_REPO \ - --builder-id https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0 \ - YOUR_BINARY +**Environment variables** for your app (e.g., `DATABASE_URL`) can be set via the `env` context in the step that runs your app/tests. + +--- + +## 2. Production Database for Your Nuxt App + +If by “big database” you mean a production‑ready database (PostgreSQL, MySQL, MongoDB, etc.) that your deployed Nuxt app will use, you have several options: + +- **Managed cloud databases**: + - [Supabase](https://supabase.com/) (PostgreSQL, easy integration with Nuxt) + - [AWS RDS](https://aws.amazon.com/rds/) + - [Google Cloud SQL](https://cloud.google.com/sql) + - [MongoDB Atlas](https://www.mongodb.com/atlas) + +- **Database as a service with generous free tiers** (good for starting “big”): + - [Neon](https://neon.tech/) (serverless Postgres) + - [PlanetScale](https://planetscale.com/) (MySQL) + - [Supabase](https://supabase.com/) + +### Connecting to a production database from GitHub Pages + +GitHub Pages hosts **static files only**. If your Nuxt app is **static** (`npm run generate`), it cannot connect directly to a database on the client side. You have two approaches: + +1. **Use a serverless backend** (e.g., Supabase, Firebase, or your own API on a platform like Vercel/Railway) and call it from your static Nuxt app via `fetch`. +2. **Use Nuxt's server routes** (if you deploy on a Node.js environment like Vercel, Netlify, or a VPS). But GitHub Pages only serves static files, so server routes won't work there. + +If you need a **hybrid** or **server‑side rendered** app with a database, consider deploying to: +- [Vercel](https://vercel.com) (excellent Nuxt support) +- [Netlify](https://netlify.com) +- [Cloudflare Pages](https://pages.cloudflare.com) +- A VPS (DigitalOcean, Hetzner) with a Node.js server. + +Then, in your GitHub Actions, you can: +- Set the production `DATABASE_URL` as a [repository secret](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions). +- Pass it to the build step if needed (e.g., for prerendering dynamic routes). +- Or use it only when running migrations before deploying. + +### Example: Using a production database secret in GitHub Actions + +```yaml +- name: Run migrations on production DB + env: + DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }} + run: npx prisma migrate deploy ``` -This workflow provides cryptographic guarantees of build integrity while maintaining build performance and flexibility. +--- + +## 3. Need More Specifics? + +If you can clarify what you mean by “set all big database,” I can give more tailored guidance. For instance: + +- Are you looking to **run database migrations** as part of your GitHub Actions workflow? +- Do you want to **set up a database service** for testing? +- Are you trying to **connect your Nuxt app to a cloud database** in production? + +Just let me know! +I understand you'd like more specific guidance on integrating a database with your Nuxt app and GitHub Actions. To give you the most helpful answer, I need a bit more context. Could you clarify: + +1. **Where do you plan to deploy your Nuxt app?** + - GitHub Pages (static only) + - Vercel / Netlify (supports serverless functions) + - A VPS or Node.js hosting (full server) + +2. **What kind of database?** + - PostgreSQL, MySQL, MongoDB, SQLite, etc. + +3. **What do you mean by “set all big database”?** + - Do you want to **spin up a database during CI** (for testing)? + - Do you want to **connect your production app to a cloud database**? + - Do you need help with **migrations** or **seeding** in the workflow? + +4. **Are you using an ORM like Prisma, Drizzle, or direct database drivers?** + +Once I know these details, I can provide a **tailored, step-by-step guide**. + +In the meantime, here’s a **common, concrete scenario** (many Nuxt developers use this): + +--- + +### 🔥 Scenario: Static Nuxt app on GitHub Pages + Supabase (PostgreSQL) as backend + +**Goal:** +- Deploy static Nuxt site to GitHub Pages. +- Use Supabase as the “big database” (PostgreSQL with REST API). +- Fetch data client‑side or via Supabase client. + +**Why this works:** +- GitHub Pages serves static files. +- Supabase provides a database and an API – no server needed. +- You can manage schema and data through Supabase dashboard or migrations. + +#### Step‑by‑step: + +1. **Create a Supabase project** at [supabase.com](https://supabase.com). + - Get your project URL and anon/public key from **Settings → API**. + +2. **Install Supabase client** in your Nuxt app: + ```bash + npm install @supabase/supabase-js + ``` + +3. **Create a Supabase plugin** (e.g., `plugins/supabase.client.ts`): + ```ts + import { createClient } from '@supabase/supabase-js' + + export default defineNuxtPlugin(() => { + const supabase = createClient( + process.env.SUPABASE_URL, + process.env.SUPABASE_ANON_KEY + ) + return { + provide: { supabase } + } + }) + ``` + +4. **Set environment variables** locally (`.env`) and in GitHub Actions (secrets). + For local dev: + ``` + SUPABASE_URL=https://yourproject.supabase.co + SUPABASE_ANON_KEY=your-anon-key + ``` + In GitHub repo: add these as secrets (`SUPABASE_URL`, `SUPABASE_ANON_KEY`). + +5. **Use Supabase in your pages/components**: + ```vue + + ``` + +6. **GitHub Actions workflow** (from previous answer) – just ensure the secrets are passed if needed at build time. If you only fetch client‑side, you don't need them at build time. If you need to prerender dynamic routes with data, you would pass them to the build step. + +7. **Deploy**: The workflow builds the static site and uploads to GitHub Pages. The app will talk to Supabase directly from the browser. + +--- + +If this matches your needs, I can expand on any part. If not, please provide more specifics so I can tailor the solution. From ac865652832d598bb19868848da7c617db5ee846 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Tue, 17 Mar 2026 01:44:06 +0800 Subject: [PATCH 8/9] Update Development MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # Building a SLSA3‑Compliant Development Workflow: A Practical Guide You’ve asked to “Create development” – and that’s exactly what we’ll do. Below is a **step‑by‑step blueprint** for setting up a development environment and process that meets SLSA Level 3 requirements while remaining practical for everyday coding. This guide builds on everything we’ve discussed and gives you a concrete path forward. ## 🎯 Goal Establish a development pipeline where every artifact (commits, builds, releases) carries verifiable proof of its origin and integrity, satisfying SLSA Build Level 3 and setting the foundation for higher levels. --- ## 📦 Phase 1: Repository Foundation ### 1.1 Enable Signed Commits - **What**: Require that every commit is signed with a GPG, SSH, or S/MIME key. - **How**: - Generate a signing key (GPG: `gpg --full-generate-key`). - Add the public key to your GitHub/GitLab account. - Configure Git locally: ```bash git config --global user.signingkey git config --global commit.gpgsign true ``` - **Why**: Establishes a chain of custody from developer to code, which is part of the source integrity expected by SLSA. ### 1.2 Branch Protection Rules - **Require signed commits** on all protected branches (main, release/*). - **Require pull request reviews** before merging. - **Optionally require** that all status checks pass (including any SLSA verification steps you add later). ### 1.3 Dependency Locking - Use lock files (`go.sum`, `package-lock.json`, `Cargo.lock`, etc.) to pin dependency versions. - Commit these lock files to the repository. --- ## 🛠️ Phase 2: Local Developer Environment ### 2.1 Containerized Development Environment - Provide a `Dockerfile` or Dev Container configuration that mirrors your CI build environment. - Example `.devcontainer/devcontainer.json`: ```json { "name": "MyApp Dev", "build": { "dockerfile": "Dockerfile" }, "features": { "ghcr.io/devcontainers/features/common-utils:2": {} } } ``` - This ensures builds are reproducible and eliminates “works on my machine” discrepancies. ### 2.2 Pre‑commit Hooks for Security - Use [pre-commit](https://pre-commit.com/) to run: - **Dependency hash verification** (e.g., check `go.sum` against a known good state). - **Linting and static analysis**. - **Secret scanning** (e.g., `detect-secrets` or `truffleHog`). - Example `.pre-commit-config.yaml`: ```yaml repos: - repo: local hooks: - id: verify-deps name: Verify dependencies entry: scripts/verify-deps.sh language: script files: '.*\.(mod|sum|lock)$' - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: detect-aws-credentials - id: detect-private-key ``` ### 2.3 Local Build with Provenance (Optional but Recommended) - For testing, you can generate lightweight attestations using [in-toto](https://in-toto.io/). - Example wrapper script `local-build.sh`: ```bash #!/bin/bash in-toto-run --step-name local-build --products myapp --key mykey -- ./build.sh ``` - This creates a link file that you can keep locally for debugging. --- ## 🔁 Phase 3: Continuous Integration (CI) Pipeline ### 3.1 Reusable Build Workflow Create a reusable workflow (`.github/workflows/build.yml`) that encapsulates your build steps and outputs artifact hashes. ```yaml name: Reusable Build on: workflow_call: outputs: base64-subjects: description: "Base64-encoded subjects (artifacts with digests)" value: ${{ jobs.build.outputs.base64-subjects }} jobs: build: runs-on: ubuntu-latest outputs: base64-subjects: ${{ steps.hashes.outputs.base64_subjects }} steps: - uses: actions/checkout@v4 - run: make build - run: mkdir -p artifacts && cp bin/* artifacts/ - id: hashes run: | cd artifacts subjects="[]" for file in *; do hash=$(sha256sum "$file" | cut -d' ' -f1) subjects=$(jq -c --arg name "$file" --arg hash "sha256:$hash" \ '. += [{"name": $name, "digest": $hash}]' <<< "$subjects") done echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT - uses: actions/upload-artifact@v4 with: name: build-artifacts-${{ github.run_id }} path: artifacts/ ``` ### 3.2 Main CI Workflow (PRs and Merges) Create a main workflow (`.github/workflows/ci.yml`) that calls the reusable build and then verifies dependencies. ```yaml name: CI on: pull_request: push: branches: [main] permissions: contents: read id-token: write # Needed if you verify provenance of dependencies jobs: build: uses: ./.github/workflows/build.yml verify-deps: runs-on: ubuntu-latest needs: build steps: - name: Download dependency provenance run: | # Example: verify a third-party binary used in build curl -LO https://example.com/dependency.bin curl -LO https://example.com/dependency.intoto.jsonl slsa-verifier verify-artifact \ --provenance-path dependency.intoto.jsonl \ --source-uri github.com/trusted-owner/dependency \ --builder-id trusted-builder-id \ dependency.bin ``` ### 3.3 Release Workflow with SLSA3 Provenance Create a separate workflow (`.github/workflows/release.yml`) triggered on tags or release publication. ```yaml name: Release on: release: types: [published] permissions: id-token: write attestations: write contents: write actions: read jobs: build: uses: ./.github/workflows/build.yml provenance: needs: build uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 with: base64-subjects: ${{ needs.build.outputs.base64-subjects }} upload-artifacts-name: build-artifacts-${{ github.run_id }} upload-assets: true verify-and-publish: needs: provenance runs-on: ubuntu-latest steps: - uses: actions/download-artifact@v4 with: name: build-artifacts-${{ github.run_id }} path: artifacts/ - name: Verify provenance run: | gh attestation verify artifacts/myapp \ --owner ${{ github.repository_owner }} \ --signer-repo slsa-framework/slsa-github-generator - name: Upload to release run: | gh release upload ${{ github.event.release.tag_name }} artifacts/* provenance.json env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` --- ## 🔐 Phase 4: Verification Gate Before Deployment ### 4.1 Deployment Job In the release workflow (or a separate deployment workflow), add a final verification step: ```yaml deploy: needs: verify-and-publish runs-on: ubuntu-latest environment: production steps: - name: Verify attestation again (defense in depth) run: | gh attestation verify artifacts/myapp \ --owner ${{ github.repository_owner }} - name: Deploy run: ./deploy.sh artifacts/ ``` ### 4.2 Enforce Verification in Production Environment - Configure GitHub Environments with **required reviewers**. - Optionally, use a deployment protection rule that calls an external verification service (e.g., via a custom GitHub Action). --- ## 📘 Phase 5: Developer Onboarding & Documentation Create a `CONTRIBUTING.md` or `DEVELOPMENT.md` that covers: - How to set up the local environment (Docker, pre‑commit, signing keys). - How to run a local build that mimics CI. - How to interpret verification failures. - Where to find provenance files and how to manually verify them. Example section: ```markdown ## Verifying Artifacts Locally After a CI build, you can download the provenance file from the workflow run and verify it: ```bash gh attestation verify myapp --owner myorg ``` Or using the `slsa-verifier` tool: ```bash slsa-verifier verify-artifact \ --provenance-path provenance.json \ --source-uri github.com/myorg/myapp \ myapp ``` ``` --- ## 🧪 Phase 6: Testing the Pipeline 1. **Test a pull request**: Push a change, ensure the CI workflow runs, and that dependency verification passes. 2. **Create a pre‑release**: Tag a commit, push the tag, and trigger the release workflow. Download the artifacts and verify them manually. 3. **Simulate a tampering attempt**: Modify an artifact locally and try to verify it – the verification should fail. 4. **Rotate keys**: If using GPG for commit signing, practice rotating a developer key and updating the repository settings. --- ## 📈 Ongoing Maintenance - Keep the SLSA generator version up to date (check for releases at [slsa-framework/slsa-github-generator](https://github.com/slsa-framework/slsa-github-generator/releases)). - Periodically review dependency verification rules. - Update developer documentation as tooling evolves. --- ## 🎉 You’re Now SLSA3‑Ready By following this guide, your development workflow will: - Produce **non‑forgeable provenance** for every release. - Ensure **source integrity** through signed commits. - Provide **reproducible builds** via containerized environments. - Build a **culture of security** where every developer understands and participates in supply chain protection. This is a living process – adapt it to your team’s size, tech stack, and release cadence. And remember, the SLSA community is constantly improving tooling and practices; stay engaged to keep your pipeline cutting‑edge. Would you like help with any specific part, such as setting up the containerized environment or writing the reusable workflow? I’m here to assist! --- Development | 273 +++++++++++++++++++++------------------------------- 1 file changed, 110 insertions(+), 163 deletions(-) diff --git a/Development b/Development index c67b0dc3a9fe..408510725d41 100644 --- a/Development +++ b/Development @@ -1,168 +1,115 @@ -# Contributing to [Project Name] - -Thank you for your interest in contributing to [Project Name]! This document provides comprehensive guidelines and instructions for contributing effectively. Please read it carefully before submitting issues or pull requests. - -## Table of Contents -- [Code of Conduct](#code-of-conduct) -- [Getting Started](#getting-started) -- [Development Process](#development-process) -- [Coding Standards](#coding-standards) -- [Building and Testing on Different Platforms](#building-and-testing-on-different-platforms) - - [Linux (including sanitizers)](#linux-including-sanitizers) - - [macOS 14 Native](#macos-14-native) - - [Windows Native and Visual Studio](#windows-native-and-visual-studio) - - [CI / Windows Test](#ci--windows-test) -- [Testing Guidelines](#testing-guidelines) -- [Submitting Changes](#submitting-changes) -- [Communication](#communication) -- [License](#license) - -## Code of Conduct -This project adheres to the [Bitcoin Code of Conduct](https://bitcoincore.org/en/conduct/). By participating, you are expected to uphold this code. - -## Getting Started -1. Fork the repository and clone your fork. -2. Set up the development environment as described in the [README](README.md). -3. Find an issue to work on or propose a new feature by opening an issue. - -## Development Process -- All development happens on the `master` branch. Pull requests (PRs) should be made against `master`. -- For significant changes, open an issue first to discuss the proposed change. -- Keep PRs focused on a single topic. Avoid mixing unrelated changes. -- Commit messages should be clear and follow the [conventional format](https://www.conventionalcommits.org/). - -## Coding Standards -- **Language:** The project is written in C++17 (or specify your language). -- **Style:** Follow the existing code style. Use `clang-format` with the provided `.clang-format` configuration. -- **Documentation:** Document all public interfaces, complex logic, and any non-obvious behavior. -- **Linting:** Run `make lint` (or equivalent) to check for style violations. -- **Security:** Avoid unsafe patterns. Be mindful of memory safety, thread safety, and constant-time operations for cryptographic code. - -## Building and Testing on Different Platforms -To ensure cross-platform compatibility, we require that contributions be tested on at least one of the following platforms, with additional sanitizer checks where applicable. Detailed instructions for each platform are provided below. - -### Linux (including sanitizers) -- **Build Dependencies:** Install required packages (e.g., `build-essential`, `libtool`, `autotools-dev`, `automake`, `pkg-config`, `libssl-dev`, `libevent-dev`, `libboost-system-dev`, `libboost-filesystem-dev`, `libboost-thread-dev`, `libsqlite3-dev`, `libdb++-dev`). -- **Build with Autotools:** - ```bash - ./autogen.sh - ./configure - make -j$(nproc) - ``` -- **Run Unit Tests:** - ```bash - make check - ``` -- **Run Functional Tests:** - ```bash - test/functional/test_runner.py - ``` -- **Sanitizers (ASan + LSan):** - To enable AddressSanitizer and LeakSanitizer, configure with: - ```bash - ./configure --enable-asan CXXFLAGS="-fsanitize=address,leak -g" CFLAGS="-fsanitize=address,leak -g" - make clean - make -j$(nproc) - ``` - Then run tests with the sanitizer-enabled binary. Memory errors and leaks will be reported. -- **Other Sanitizers:** You may also enable ThreadSanitizer (`--enable-tsan`) or UndefinedBehaviorSanitizer (`--enable-ubsan`) as needed. +Great! Let's explore how to integrate SLSA3 principles into your **development workflow**—from coding to merging—so that security is baked in from the start. This goes beyond just configuring CI/CD; it's about making SLSA3 a natural part of how developers work every day. -### macOS 14 Native -- **Prerequisites:** Install Xcode Command Line Tools: - ```bash - xcode-select --install - ``` -- **Install Dependencies via Homebrew:** - ```bash - brew install automake libtool boost pkg-config libevent sqlite3 berkeley-db - ``` -- **Build with Autotools:** +## 🔄 Development Workflow with SLSA3 in Mind + +### 1. **Dependency Management: Verify Before You Build** +SLSA3 requires that all dependencies are verified to ensure they haven't been tampered with. During development, you can adopt practices that enforce this automatically. + +- **Use `go mod verify` (Go)** or `pip hash` (Python) to check downloaded modules. +- **Integrate `slsa-verifier`** or **Sigstore** into your local build scripts to verify dependencies before compilation. For example, you could run: ```bash - ./autogen.sh - ./configure - make -j$(sysctl -n hw.logicalcpu) + slsa-verifier verify-artifact \ + --provenance-path "path/to/provenance.json" \ + --source-uri "github.com/your-org/dependency" \ + --builder-id "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v2.1.0" \ + dependency.bin ``` -- **Run Tests:** + +- **Automate in pre-commit hooks**: Use tools like `pre-commit` to run dependency checks before each commit, ensuring no untrusted code sneaks in. + +### 2. **Local Build Reproducibility** +SLSA3 demands that builds be reproducible and that the build process is fully scripted. As a developer, you can: + +- **Containerize your development environment** (e.g., using Dev Containers or Docker) to match the CI environment exactly. This reduces "works on my machine" issues and ensures that the build you test locally will produce the same artifacts as CI. + +- **Use build tools that support deterministic outputs**, like **Bazel**, **Pants**, or **CMake** with fixed timestamps and no random elements. + +- **Run a local SLSA check** before pushing: simulate the CI build process in a clean environment (e.g., using `act` for GitHub Actions locally) to catch any discrepancies early. + +### 3. **Signing Commits and Tags** +While not strictly required by SLSA3, signing commits and tags (with GPG or SSH keys) establishes a chain of custody from the developer to the source code. This helps meet the **source integrity** expectations of SLSA3. + +```bash +git commit -S -m "feat: add secure feature" +git tag -s v1.2.3 -m "Release v1.2.3" +``` + +Encourage your team to enable signing globally and enforce it in your repository settings (e.g., GitHub's "Require signed commits" branch protection). + +### 4. **Integrate Security Scans Early** +SLSA3 doesn't mandate vulnerability scanning, but it's a best practice that aligns with the "secure development" ethos. Incorporate tools like **Snyk**, **Dependabot**, or **Trivy** into your IDE or pre-commit hooks. + +For example, with VS Code you can use the **Snyk extension** to scan dependencies and code as you type, catching issues before they ever reach the repo. + +### 5. **Use In-Toto Attestations for Development Artifacts** +SLSA3 relies on attestations (like provenance) to describe how an artifact was built. During development, you can generate lightweight attestations for intermediate artifacts (e.g., container images, binaries) to maintain a trail even before official releases. + +- **Tool**: [**in-toto**](https://in-toto.io/) provides a framework to create and verify attestations. +- **Example**: After a local build, run: ```bash - make check - test/functional/test_runner.py + in-toto-run --step-name local-build --products myapp --key mykey -- ``` -- **Note:** macOS 14 (Sonoma) is the primary target; older versions may work but are not officially supported. - -### Windows Native and Visual Studio -We support two build methods on Windows: **MinGW-w64 (MSYS2)** for a Unix-like environment and **Visual Studio** for native Windows development. - -#### MinGW-w64 (MSYS2) -1. Install MSYS2 from [msys2.org](https://www.msys2.org/). -2. Open an MSYS2 terminal and update packages: - ```bash - pacman -Syu - ``` -3. Install build dependencies: - ```bash - pacman -S --needed base-devel mingw-w64-x86_64-toolchain mingw-w64-x86_64-autotools mingw-w64-x86_64-boost mingw-w64-x86_64-libevent mingw-w64-x86_64-sqlite3 mingw-w64-x86_64-berkeley-db - ``` -4. Build using the Mingw-w64 environment: - ```bash - ./autogen.sh - ./configure --host=x86_64-w64-mingw32 - make -j$(nproc) - ``` -5. Run tests (if possible; some functional tests may require Python in the MSYS2 environment). - -#### Visual Studio -1. Install Visual Studio 2022 or later with **Desktop development with C++** workload. -2. Install **vcpkg** (if not already installed) and integrate it with Visual Studio: - ```bash - git clone https://github.com/Microsoft/vcpkg.git - cd vcpkg - .\bootstrap-vcpkg.bat - .\vcpkg integrate install - ``` -3. Install required dependencies via vcpkg: - ```bash - .\vcpkg install boost pthread libevent openssl sqlite3 berkeley-db - ``` -4. Open the solution file `build_msvc/bitcoin.sln` (or equivalent) in Visual Studio. -5. Build the solution (select appropriate configuration: Debug/Release, x64). -6. Run tests via Test Explorer or build the test projects directly. - -### CI / Windows Test -- Our continuous integration (CI) pipeline runs on GitHub Actions (or similar). See `.github/workflows/` for configuration. -- Windows CI jobs automatically build and test the project using both MSYS2 and Visual Studio environments. -- Before submitting a PR, ensure that all Windows CI checks pass. If you cannot test locally, rely on the CI results. -- For Windows-specific changes, consider adding or updating tests in the CI configuration (e.g., adding a new test file to `test/functional/`). - -## Testing Guidelines -- **Unit Tests:** Write unit tests for new code using the [Boost.Test](https://www.boost.org/doc/libs/1_81_0/libs/test/doc/html/index.html) framework. Place them in `src/test/` and add to the appropriate `Makefile.test` or `CMakeLists.txt`. -- **Functional Tests:** Add Python-based functional tests in `test/functional/` for end-to-end scenarios. Run them with `test/functional/test_runner.py`. -- **Coverage:** Ensure your changes are covered by tests. If you add new features or fix bugs, include tests that verify correctness. -- **Consensus Changes:** If your change affects consensus rules, include extensive tests and consider running against the Bitcoin Core test suite if applicable. - -## Submitting Changes -1. Ensure your code is rebased on the latest `master`. -2. Run the full test suite and linters: - ```bash - make check - test/functional/test_runner.py - make lint - ``` -3. Sign off your commits to certify that you have the right to submit the code under the project's license. Use `git commit -s` to add a Signed-off-by line. -4. Push your changes to your fork and open a pull request against `master`. -5. In the PR description, explain the changes, reference any related issues, and note any breaking changes. -6. Be responsive to feedback. Update your PR as needed. - -### Developer Certificate of Origin (DCO) -We require that all contributors sign the [Developer Certificate of Origin](https://developercertificate.org/). By signing off your commits (`git commit -s`), you attest that you have the right to submit the code. - -## Communication -- **GitHub Issues:** For bug reports, feature requests, and questions. -- **Mailing List:** [bitcoin-dev](https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev) for protocol-level discussions. -- **IRC/Matrix:** Join `#bitcoin-core-dev` on Libera.Chat or the [Bitcoin Core Matrix channel](https://matrix.to/#/#bitcoin-core-dev:libera.chat). - -## License -By contributing, you agree that your contributions will be licensed under the same license as the project (e.g., MIT). See [LICENSE](LICENSE) for details. - ---- - -Thank you for contributing to [Project Name] and helping advance Bitcoin development! + +This creates a link file that can be stored alongside the artifact, useful for debugging or for internal audits. + +### 6. **CI/CD Integration: Shift Left with SLSA Checks** +In your pull request workflow, add a job that verifies any external artifacts your build consumes. For instance, if you download a third‑party binary, verify its provenance before using it: + +```yaml +- name: Verify dependency provenance + run: | + slsa-verifier verify-artifact \ + --provenance-path dependency.intoto.jsonl \ + --source-uri github.com/trusted-org/dependency \ + --builder-id trusted-builder-id \ + dependency.bin +``` + +This prevents accidentally incorporating compromised dependencies into your product. + +### 7. **Documentation and Onboarding** +Make SLSA3 part of your developer onboarding. Provide a short guide that covers: +- How to set up local tooling (verifiers, signers). +- How to interpret and verify provenance. +- How to run security scans locally. +- How to handle dependencies safely. + +## 🛠️ Practical Example: A Developer's Daily Routine + +Here’s what a typical day might look like with SLSA3 integrated: + +1. **Morning pull** – `git pull` brings in signed commits; you verify them with `git log --show-signature`. +2. **Work on feature** – Write code, run `pre-commit` which checks dependency hashes and runs linters. +3. **Local build** – Use a containerized environment (e.g., `docker run --rm -v $(pwd):/workspace build-env ./build.sh`). The build script also generates an in-toto link for the binary. +4. **Test** – Run `ctest` inside the container; all tests pass. +5. **Push** – `git push` triggers the CI pipeline, which builds again (isolated) and generates SLSA3 provenance. +6. **PR review** – A GitHub Action verifies that the provenance for any downloaded dependencies is valid. The reviewer sees a green checkmark indicating "provenance verified". +7. **Merge to main** – The release workflow builds the final artifact and signs it with Sigstore. The provenance is attached to the GitHub release. + +## 📚 Resources for Developers + +- **Sigstore Client (`cosign`)**: Sign and verify container images and blobs. +- **slsa-verifier**: Verify provenance for binaries built with SLSA generators. +- **GitHub CLI `gh attestation`**: Verify attestations directly from GitHub. +- **in-toto**: Create and verify attestations for any step in your supply chain. +- **Docker Scout**: Check your images against vulnerability databases and verify provenance. + +By embedding these practices into your daily development, you ensure that every artifact you create—whether a local test binary or a production release—carries verifiable proof of its origin and integrity. This not only meets SLSA3 requirements but also builds a culture of security within your team. + +Would you like a deeper dive into any of these areas, such as setting up local verifiers or integrating them into your IDE?Great! It looks like we've covered a lot of ground—from updating your specific GitHub Actions workflows to diving deep into SLSA Level 3 principles and how they integrate into both CI/CD and daily development practices. + +To recap, you now have: +- ✅ Updated `generator-generic-ossf-slsa3-publish.yml` and `cmake-single-platform.yml` with SLSA3‑compliant configurations. +- ✅ A clear understanding of how different platforms (GitHub, GitLab, Google, etc.) implement SLSA3. +- ✅ A step‑by‑step guide to structuring your GitHub workflows for non‑forgeable provenance. +- ✅ Practical advice on weaving SLSA3 into your development workflow—dependency verification, local reproducibility, commit signing, and more. + +If you’re ready to put this into practice, here are a few next steps you might consider: +1. **Pilot the workflow** on a non‑critical project to test the new YAML files and verification steps. +2. **Share the documentation** with your team to align on secure development practices. +3. **Set up branch protection rules** that require signed commits and successful provenance verification before merging. +4. **Explore automation** for dependency verification (e.g., using `slsa-verifier` in CI) to catch issues early. + +And of course, if you run into any questions or want to dive deeper into a specific topic—like setting up Sigstore for local signing, or troubleshooting a workflow—just ask. I’m here to help! + +Happy building, and thanks for working toward a more secure software supply chain. 🚀 From 20609083a246af6a1f018b94c07a460fe1f64e31 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Tue, 17 Mar 2026 02:07:22 +0800 Subject: [PATCH 9/9] Update and rename CodeQL Advanced to License Agreement (CLA). To help you **update and rename "CodeQL Advanced" to "License Agreement (CLA)"**, I need to know exactly what you're working with. However, I can give you a **general approach** based on common scenarios. If you can provide more details (file type, location, purpose), I can tailor the answer precisely. --- ### **Possible Scenarios & Solutions** #### **Scenario 1: You have a GitHub Actions workflow file named `codeql-advanced.yml`** You want to rename it to `cla.yml` and update its content to handle CLA signing records (like the data for Muhamad Sazwan bin Ismail). **Steps:** 1. **Rename the file** In your repository, navigate to `.github/workflows/` and rename `codeql-advanced.yml` to `cla.yml` (or `license-agreement.yml`). 2. **Update the workflow name** (inside the file) Change the `name:` field from something like `CodeQL Advanced` to `License Agreement (CLA)`. 3. **Replace the steps** Remove all CodeQL-related steps and add steps that interact with your database (SQL/NoSQL) to record CLA signatures. Example placeholder workflow: ```yaml name: License Agreement (CLA) on: pull_request: types: [opened, synchronize, reopened] jobs: check-cla: runs-on: ubuntu-latest steps: - name: Check CLA signature run: | # Example: Query your database to see if contributor signed CLA echo "Checking CLA for ${{ github.event.pull_request.user.login }}" # You'd use a script here that checks against your DB ``` #### **Scenario 2: You have a database table or collection named `codeql_advanced`** You want to rename it to `cla_records` and update its schema/structure to store CLA data (like the examples for Muhamad Sazwan). **SQL Example:** ```sql -- Rename table ALTER TABLE codeql_advanced RENAME TO cla_records; -- Update columns if needed ALTER TABLE cla_records ADD COLUMN signed_at TIMESTAMP; -- etc. ``` **NoSQL (MongoDB) Example:** ```javascript // Rename collection db.codeql_advanced.renameCollection("cla_records"); // Update documents to new structure (if needed) db.cla_records.updateMany({}, { $rename: { "oldField": "newField" } }); ``` #### **Scenario 3: You have a code file or class named `CodeQLAdvanced`** You want to rename it to `LicenseAgreementCLA` and update its logic. **Example in Python:** ```python # Rename class class LicenseAgreementCLA: def __init__(self, contributor_name): self.contributor = contributor_name # ... other CLA-related logic ``` --- ### **Please clarify:** - What type of item are you renaming? (file, table, class, etc.) - Where is it located? (GitHub repo, local project, database) - What should the new version do? (record CLA signatures, manage agreements, etc.) Once I know these, I can give you the exact commands or code modifications. --- CodeQL Advanced | 714 --------------------------------------- License Agreement (CLA). | 112 ++++++ 2 files changed, 112 insertions(+), 714 deletions(-) delete mode 100644 CodeQL Advanced create mode 100644 License Agreement (CLA). diff --git a/CodeQL Advanced b/CodeQL Advanced deleted file mode 100644 index 686c32a140b6..000000000000 --- a/CodeQL Advanced +++ /dev/null @@ -1,714 +0,0 @@ -Configuring CodeQL in advanced setup gives you granular control to customize scanning for your project's specific needs, going far beyond what default setup offers. You can fine-tune everything from scan triggers and operating systems to creating your own security queries. - -Here's a comparison to help you understand the core differences: - -| Feature | Default Setup | Advanced Setup | -| :--- | :--- | :--- | -| **Control** | Automated, minimal input | Full manual control | -| **Configuration** | Simple UI-based | Editing YAML workflow files | -| **Customization** | Limited | Extensive (triggers, OS, queries, etc.) | -| **Workflow Location** | Automatically managed | `.github/workflows/` directory | -| **Best For** | Quick start, standard needs | Complex projects, specific requirements | - -### ⚙️ Configure the Advanced Workflow - -The advanced setup revolves around modifying a CodeQL workflow file (typically `codeql-analysis.yml`) located in your repository's `.github/workflows` directory. - -**Set Scanning Triggers and Frequency** -You can configure the workflow to run on specific events, on a schedule, or both. This is defined using the `on` key in the YAML file. - -- **Scan on Push and Pull Requests**: This is the standard practice for catching vulnerabilities as they are introduced. You can specify which branches to monitor. -- **Avoid Unnecessary Scans**: Use `paths-ignore` to skip scanning PRs that only change certain files (like documentation), saving resources. -- **Schedule Scans**: Run periodic scans (e.g., weekly) to catch new vulnerabilities from updated CodeQL queries, even during inactive periods. - -```yaml -on: - push: - branches: [ main, protected ] - pull_request: - branches: [ main ] - paths-ignore: - - '**/*.md' - - '**/*.txt' - schedule: - - cron: '20 14 * * 1' # Every Monday at 14:20 UTC -``` - -**Specify the Operating System and Languages** -The `runs-on` key determines the OS of the runner. CodeQL supports `ubuntu-latest`, `windows-latest`, and `macos-latest`. - -- **For Swift code**, CodeQL uses macOS runners by default, which are more expensive. You can configure it to use a different runner for all steps except the build. -- Use a **matrix strategy** to analyze multiple languages efficiently in parallel jobs. For compiled languages like Java, you may need to include custom build steps. - -```yaml -jobs: - analyze: - name: Analyze - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - language: [ 'javascript', 'python', 'java' ] - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - name: Initialize CodeQL - uses: github/codeql-action/init@v4 - with: - languages: ${{ matrix.language }} -``` - -### 🔍 Develop Custom Queries - -Advanced setup allows you to extend analysis by running additional queries, including those you write yourself. - -**Create a Query Pack** -CodeQL queries are organized into "packs". To start writing your own, create a new pack using the CodeQL CLI, which generates a `qlpack.yml` file. -```bash -codeql pack init / -``` -You then need to add dependencies for the standard library of the language you want to analyze. -```bash -codeql pack add codeql/javascript-all -``` - -**Understand QL Basics** -QL is a declarative, logic programming language. A basic query follows this structure: -```ql -from /* ... variable declarations ... */ -where /* ... logical conditions ... */ -select /* ... expressions to output ... */ -``` - -**Write and Refine Queries** -- **Leverage Standard Libraries**: Use existing classes and predicates from the extensive CodeQL libraries to build upon. -- **Use Debugging Tools**: If a query doesn't work as expected, techniques like creating a minimal code example, using Quick Evaluation in VS Code, viewing the AST, and generating partial path graphs are invaluable for diagnosing issues. -- **Add Query Metadata**: Include a comment block with metadata like `@name`, `@description`, and `@kind` so the results are properly displayed in GitHub. - -### 🚀 Implement at Scale and Customize Further - -- **Scale Across an Organization**: For organizations needing advanced setup on many repositories, you can use a bulk configuration script to add the workflow across a group of repos efficiently. -- **Extend Analysis with Model Packs**: For code that uses custom or niche frameworks not recognized by standard queries, you can specify published **CodeQL model packs** (currently in preview for C/C++, C#, Java/Kotlin, Python, Ruby, and Rust) to extend coverage. -- **Specify a Custom Database Location**: While usually handled automatically, you can define where the CodeQL database is stored using the `db-location` parameter under the `init` action, which is useful for custom workflow steps. - -I hope this gives you a comprehensive roadmap for leveraging the advanced capabilities of CodeQL. Which aspect are you most interested in implementing first? If you're working with a specific programming language or framework, I might be able to offer more tailored guidance. -# CodeQL Advanced Implementation Guide - -## 🚀 Advanced CodeQL Configuration - -### Multi-Language Enterprise Setup - -```yaml -# .github/workflows/advanced-codeql.yml -name: "Advanced CodeQL Analysis" - -on: - push: - branches: [ main, develop, release/* ] - paths-ignore: - - '**/*.md' - - '**/*.docs' - - '**/test/**' - - '**/spec/**' - pull_request: - branches: [ main, develop ] - paths: - - 'src/**' - - 'lib/**' - - '**.java' - - '**.js' - - '**.ts' - - '**.py' - - '**.go' - schedule: - - cron: '0 2 * * 1' # Weekly on Monday at 2 AM UTC - workflow_dispatch: - inputs: - severity: - description: 'Minimum severity level' - required: true - default: 'error' - type: choice - options: - - error - - warning - - note - -env: - CODEQL_ACTION_DISABLE_SETUP: false - CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN: false - CODEQL_POWERFUL_ANALYSIS: true - -jobs: - analyze: - name: Advanced CodeQL Analysis - runs-on: ubuntu-latest - - strategy: - fail-fast: false - matrix: - include: - - language: java-kotlin - build-mode: autobuild - - language: javascript - build-mode: none - - language: python - build-mode: none - - language: go - build-mode: autobuild - - language: cpp - build-mode: manual - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - fetch-depth: 0 # Full history for better analysis - - - name: Initialize CodeQL - uses: github/codeql-action/init@v3 - with: - languages: ${{ matrix.language }} - queries: +security-and-quality,security-extended - config-file: ./.github/codeql/codeql-config.yml - tools: latest - - - name: Autobuild (if needed) - if: matrix.build-mode == 'autobuild' - uses: github/codeql-action/autobuild@v3 - - - name: Manual Build (C/C++) - if: matrix.language == 'cpp' && matrix.build-mode == 'manual' - run: | - mkdir build && cd build - cmake .. - make -j4 - - - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v3 - with: - category: "/language:${{ matrix.language }}" - output: sarif-results - upload-database: true - skip-queries: ${{ github.event_name == 'schedule' && 'security' || '' }} - env: - CODEQL_ACTION_DISABLE_SETUP: false - - custom-queries: - name: Custom Query Analysis - runs-on: ubuntu-latest - needs: analyze - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Initialize CodeQL - uses: github/codeql-action/init@v3 - with: - languages: javascript, python - queries: ./.github/codeql/custom-queries.qls - packs: codeql/custom-queries - - - name: Run Custom Queries - uses: github/codeql-action/analyze@v3 - with: - category: "custom" - - security-dashboard: - name: Security Dashboard Update - runs-on: ubuntu-latest - needs: [analyze, custom-queries] - if: always() - - steps: - - name: Upload Results to Security Dashboard - uses: github/codeql-action/upload-sarif@v3 - with: - sarif_file: sarif-results - wait-for-processing: true - - - name: Generate Security Report - run: | - echo "## CodeQL Security Analysis Report" >> $GITHUB_STEP_SUMMARY - echo "| Language | Critical | High | Medium | Low |" >> $GITHUB_STEP_SUMMARY - echo "|----------|----------|------|--------|-----|" >> $GITHUB_STEP_SUMMARY - echo "| Complete | - | - | - | - |" >> $GITHUB_STEP_SUMMARY -``` - -## 🔧 Advanced Configuration Files - -### CodeQL Configuration -```yaml -# .github/codeql/codeql-config.yml -name: "Advanced CodeQL Configuration" - -query-filters: -- exclude: - id: java/example/too-many-loops -- include: - tags: - - security - - external/cwe/cwe-798 - - external/cwe/cwe-259 - -paths: -- src -- lib -- app - -paths-ignore: -- test -- spec -- node_modules -- vendor -- build -- dist - -languages: - java: - build-command: - - mvn compile -DskipTests - - gradle build -x test - - javascript: - build-command: npm run build - - python: - build-command: python -m py_compile **/*.py - -queries: - uses: ./.github/codeql/query-suites.yml - -packs: -- codeql/java-queries -- codeql/javascript-queries -- codeql/python-queries -- organization/custom-security-queries - -database: - location: ./.codeql/databases - cleanup: true -``` - -### Custom Query Suites -```yql -# .github/codeql/query-suites.qls -- description: "Custom Security Queries" -- queries: . - from: custom-security-queries.qls -- include: - tags: - - security - - external/cwe/cwe-89 - - external/cwe/cwe-78 - - external/cwe/cwe-79 -- exclude: - precision: very-low -``` - -## 🛠️ Custom Query Development - -### Advanced QL Query Structure -```ql -// .github/codeql/custom-queries/SQLInjection.ql -/** - * @name Potential SQL Injection - * @description Detects potential SQL injection vulnerabilities in Java code - * @kind path-problem - * @problem.severity error - * @precision high - * @id java/sql-injection - * @tags security - * external/cwe/cwe-89 - */ - -import java -import semmle.code.java.dataflow.DataFlow -import semmle.code.java.dataflow.TaintTracking -import DataFlow::PathGraph - -class SqlInjectionConfig extends TaintTracking::Configuration { - SqlInjectionConfig() { this = "SqlInjectionConfig" } - - override predicate isSource(DataFlow::Node source) { - exists(MethodAccess ma | - ma.getMethod().hasName("getParameter") and - source.asExpr() = ma - ) - } - - override predicate isSink(DataFlow::Node sink) { - exists(MethodAccess ma | - ma.getMethod().hasName("executeQuery") and - sink.asExpr() = ma.getArgument(0) - ) - } - - override predicate isSanitizer(DataFlow::Node sanitizer) { - exists(MethodAccess ma | - ma.getMethod().hasName("escapeSql") and - sanitizer.asExpr() = ma - ) - } -} - -from SqlInjectionConfig config, DataFlow::PathNode source, DataFlow::PathNode sink -where config.hasFlowPath(source, sink) -select sink.getNode(), source, sink, "Potential SQL injection vulnerability" -``` - -### Complex Data Flow Analysis -```ql -// .github/codeql/custom-queries/AdvancedTaintTracking.ql -import java -import semmle.code.java.dataflow.DataFlow -import semmle.code.java.dataflow.TaintTracking - -class UserInputSource extends DataFlow::ExprNode { - UserInputSource() { - exists(Method m | - m.hasName("getParameter") or - m.hasName("getHeader") or - m.hasName("getCookie") or - m.hasName("getAttribute") - | - this.asExpr() = m.getACall() - ) - } -} - -class DangerousSink extends DataFlow::ExprNode { - DangerousSink() { - // File operations - exists(Method m | m.hasName("write") | this.asExpr() = m.getACall()) or - // Database operations - exists(Method m | m.hasName("execute") | this.asExpr() = m.getACall()) or - // Command execution - exists(Method m | m.hasName("exec") | this.asExpr() = m.getACall()) - } -} - -class SanitizerMethod extends DataFlow::ExprNode { - SanitizerMethod() { - exists(Method m | - m.hasName("validateInput") or - m.hasName("sanitize") or - m.hasName("escapeHtml") - | - this.asExpr() = m.getACall() - ) - } -} - -from UserInputSource source, DangerousSink sink -where DataFlow::localFlow(source, sink) -select sink, "User input flows to dangerous sink without sanitization" -``` - -## 🔍 Advanced Query Patterns - -### Performance Optimization Queries -```ql -// .github/codeql/custom-queries/PerformanceIssues.ql -import java -import semmle.code.java.dataflow.DataFlow - -class ExpensiveOperationInLoop extends DataFlow::Configuration { - ExpensiveOperationInLoop() { this = "ExpensiveOperationInLoop" } - - override predicate isSource(DataFlow::Node source) { - exists(Method m | - m.hasName("compile") or - m.hasName("createConnection") or - m.hasName("parseXML") - | - source.asExpr() = m.getACall() - ) - } - - override predicate isSink(DataFlow::Node sink) { - exists(LoopStmt loop | sink.asExpr() = loop.getCondition()) - } -} - -from ExpensiveOperationInLoop config, DataFlow::PathNode source, DataFlow::PathNode sink -where config.hasFlowPath(source, sink) -select sink.getNode(), source, sink, "Expensive operation inside loop may cause performance issues" -``` - -### Security Configuration Queries -```ql -// .github/codeql/custom-queries/SecurityMisconfiguration.ql -import java - -class InsecureConfiguration extends RefType { - InsecureConfiguration() { - this.hasQualifiedName("org.springframework.security.config.annotation.web", "WebSecurityConfigurerAdapter") - } -} - -from InsecureConfiguration config, Method m -where - m.getDeclaringType() = config and - m.overridesOrInstantiates*(m.getDeclaringType().getAMethod()) and - not exists(MethodAccess ma | - ma.getMethod().hasName("csrf") and - ma.getAnArgument().toString().matches("%disable%") - ) -select m, "Security configuration might be missing CSRF protection" -``` - -## 📊 Advanced Analysis Pipeline - -### Multi-Stage Analysis Workflow -```yaml -# .github/workflows/codeql-pipeline.yml -name: "CodeQL Security Pipeline" - -on: - push: - branches: [ main ] - schedule: - - cron: '0 0 * * 0' # Weekly - -jobs: - code-scanning: - name: Code Scanning - uses: ./.github/workflows/codeql-advanced.yml - secrets: inherit - - dependency-scanning: - name: Dependency Vulnerability Scan - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: Run OWASP Dependency Check - uses: dependency-check/Dependency-Check_Action@main - with: - project: 'my-project' - path: '.' - format: 'HTML' - out: 'reports' - - container-scanning: - name: Container Image Scanning - runs-on: ubuntu-latest - steps: - - name: Run Trivy Vulnerability Scanner - uses: aquasecurity/trivy-action@master - with: - scan-type: 'fs' - scan-ref: '.' - format: 'sarif' - output: 'trivy-results.sarif' - - security-report: - name: Generate Security Report - runs-on: ubuntu-latest - needs: [code-scanning, dependency-scanning, container-scanning] - steps: - - name: Aggregate Results - run: | - python .github/scripts/aggregate-security-report.py - - name: Upload Security Report - uses: actions/upload-artifact@v4 - with: - name: security-report - path: security-report.html -``` - -## 🔒 Advanced Security Queries - -### Authentication & Authorization -```ql -// .github/codeql/custom-queries/AuthBypass.ql -import java - -class AuthenticationBypass extends Method { - AuthenticationBypass() { - this.getAnAnnotation().getType().hasQualifiedName("org.springframework.web.bind.annotation", "RequestMapping") and - not exists(MethodAccess ma | - ma.getMethod().hasName("isAuthenticated") or - ma.getMethod().hasName("hasRole") - ) - } -} - -from AuthenticationBypass method -select method, "Endpoint might be missing authentication check" -``` - -### Cryptography Issues -```ql -// .github/codeql/custom-queries/WeakCrypto.ql -import java - -class WeakCryptoAlgorithm extends MethodAccess { - WeakCryptoAlgorithm() { - this.getMethod().hasName("getInstance") and - ( - this.getArgument(0).toString().matches("%DES%") or - this.getArgument(0).toString().matches("%RC4%") or - this.getArgument(0).toString().matches("%MD5%") - ) - } -} - -from WeakCryptoAlgorithm crypto -select crypto, "Use of weak cryptographic algorithm detected" -``` - -## 📈 Performance Optimization - -### Query Performance Configuration -```yaml -# .github/codeql/performance-config.yml -optimization: - max-paths: 1000 - timeout: 300 - memory: 8192 - -analysis: - mode: deep - threads: 4 - -caching: - enabled: true - location: ./.codeql/cache - cleanup-age: 30d - -logging: - level: INFO - queries: true - performance: true -``` - -### Custom Performance Queries -```ql -// .github/codeql/custom-queries/MemoryLeaks.ql -import java - -class PotentialMemoryLeak extends Method { - PotentialMemoryLeak() { - this.getName().matches("create%") and - not exists(Method destructor | - destructor.getName().matches("destroy%") or - destructor.getName().matches("close%") and - destructor.getDeclaringType() = this.getDeclaringType() - ) - } -} - -from PotentialMemoryLeak method -select method, "Potential memory leak: creation method without corresponding cleanup method" -``` - -## 🎯 Advanced Integration - -### CI/CD Pipeline Integration -```yaml -# .github/workflows/security-gates.yml -name: "Security Quality Gates" - -on: - pull_request: - branches: [ main ] - -jobs: - security-gate: - name: Security Quality Gate - runs-on: ubuntu-latest - steps: - - name: Run CodeQL Analysis - uses: github/codeql-action/analyze@v3 - with: - output: sarif-results - - - name: Check Security Thresholds - run: | - python .github/scripts/check-security-gates.py - - - name: Fail on Critical Issues - if: steps.security-check.outputs.critical_issues > 0 - run: | - echo "❌ Critical security issues found. Blocking merge." - exit 1 -``` - -### Custom Scripts for Analysis -```python -# .github/scripts/aggregate-security-report.py -#!/usr/bin/env python3 -import json -import sys -from pathlib import Path - -def load_sarif_results(file_path): - with open(file_path, 'r') as f: - return json.load(f) - -def analyze_severity_distribution(results): - severity_count = { - 'error': 0, - 'warning': 0, - 'note': 0 - } - - for run in results.get('runs', []): - for result in run.get('results', []): - level = result.get('level', 'warning') - severity_count[level] += 1 - - return severity_count - -def generate_report(severity_distribution): - report = f""" -# Security Analysis Report -## Summary -- Critical Issues: {severity_distribution['error']} -- Warnings: {severity_distribution['warning']} -- Recommendations: {severity_distribution['note']} - -## Quality Gates -{'❌ FAIL' if severity_distribution['error'] > 0 else '✅ PASS'} - No Critical Issues -{'⚠️ WARN' if severity_distribution['warning'] > 10 else '✅ PASS'} - Warning Threshold -""" - return report - -if __name__ == "__main__": - results_dir = Path("sarif-results") - all_results = [] - - for sarif_file in results_dir.glob("*.sarif"): - all_results.extend(load_sarif_results(sarif_file)) - - severity = analyze_severity_distribution(all_results) - report = generate_report(severity) - - with open("security-report.md", "w") as f: - f.write(report) - - print("Security report generated successfully") -``` - -This advanced CodeQL implementation provides: - -## 🚀 Key Features - -1. **Multi-Language Support** - Java, JavaScript, Python, Go, C++ -2. **Custom Query Development** - Advanced security and performance queries -3. **Performance Optimization** - Caching, parallel execution, resource management -4. **Security Quality Gates** - Automated security thresholds and blocking -5. **Comprehensive Reporting** - Aggregated security dashboards -6. **Enterprise Integration** - CI/CD pipeline integration with quality gates - -## 💡 Advanced Use Cases - -- **Custom Security Rules** - Organization-specific security requirements -- **Performance Analysis** - Code performance and memory leak detection -- **Architecture Validation** - Code structure and design pattern validation -- **Compliance Checking** - Regulatory and compliance requirements -- **Dependency Analysis** - Third-party library security assessment - -This setup enables enterprise-grade code security analysis with custom rules, performance optimization, and comprehensive reporting! diff --git a/License Agreement (CLA). b/License Agreement (CLA). new file mode 100644 index 000000000000..94c34b864765 --- /dev/null +++ b/License Agreement (CLA). @@ -0,0 +1,112 @@ +To complete the data for **Muhamad Sazwan bin Ismail** in the context of a Contributor License Agreement (CLA) system, you’ll likely need to store his information in different types of databases. Below are examples of how you might insert his record into **SQL**, **NoSQL**, and a dedicated **CLA records** structure. + +--- + +## 1. SQL Database (e.g., PostgreSQL, MySQL) + +Assume a table `contributors` with relevant columns. +A typical `INSERT` statement would look like: + +```sql +INSERT INTO contributors ( + full_name, + email, + github_username, + cla_signed, + signed_at, + ip_address +) VALUES ( + 'Muhamad Sazwan bin Ismail', + 'muhamad.sazwan@example.com', + 'msazwan', + TRUE, + NOW(), + '192.168.1.100' +); +``` + +If you have a separate `cla_signatures` table (for tracking multiple versions), you might do: + +```sql +INSERT INTO cla_signatures ( + contributor_id, + cla_version, + signed_at, + agreement_text +) VALUES ( + (SELECT id FROM contributors WHERE email = 'muhamad.sazwan@example.com'), + 'v2.1', + NOW(), + 'I have read and agree to the CLA...' +); +``` + +--- + +## 2. NoSQL Database (e.g., MongoDB) + +A document for the same contributor in MongoDB could look like this: + +```json +{ + "_id": ObjectId("..."), + "name": "Muhamad Sazwan bin Ismail", + "email": "muhamad.sazwan@example.com", + "github": "msazwan", + "cla": { + "signed": true, + "version": "v2.1", + "signed_at": ISODate("2025-03-17T04:20:00Z"), + "ip": "192.168.1.100" + } +} +``` + +Insert using: + +```javascript +db.contributors.insertOne({ + name: "Muhamad Sazwan bin Ismail", + email: "muhamad.sazwan@example.com", + github: "msazwan", + cla: { + signed: true, + version: "v2.1", + signed_at: new Date(), + ip: "192.168.1.100" + } +}); +``` + +--- + +## 3. CLA Records (as a separate system or table) + +If you maintain a separate **CLA signing log** (e.g., for auditing), you might store records in either SQL or NoSQL. A simple CSV-like record could be: + +``` +Full Name, Email, CLA Version, Signed Date, IP Address +Muhamad Sazwan bin Ismail, muhamad.sazwan@example.com, v2.1, 2025-03-17 12:20:00, 192.168.1.100 +``` + +Or as a JSON entry in a `cla_logs` collection: + +```json +{ + "contributor": "Muhamad Sazwan bin Ismail", + "email": "muhamad.sazwan@example.com", + "cla_version": "v2.1", + "signed_at": "2025-03-17T04:20:00Z", + "ip_address": "192.168.1.100" +} +``` + +--- + +## Additional Notes + +- Replace `email` and `github_username` with actual data if available. +- In production, you’d likely store a hash of the signed agreement or a reference to the PDF. +- Ensure you comply with data privacy regulations (e.g., GDPR) when storing personal information. + +If you need help adapting these examples to your specific database schema or have further requirements, feel free to ask!