Skip to content

INSTRUCTIONS.md

Latest commit

 

History

History
153 lines (115 loc) · 7.57 KB

INSTRUCTIONS.md

File metadata and controls

153 lines (115 loc) · 7.57 KB

PyMigrate Compliance Audit Migration Framework

Overview

A specialized data migration framework designed for compliance auditors overseeing data migrations in regulated industries. This implementation provides comprehensive evidence collection, cryptographic proof of data lineage, and detailed transformation tracking to meet stringent regulatory requirements and support audit processes.

Persona Description

A compliance auditor overseeing data migrations in regulated industries who needs comprehensive evidence of data lineage and transformation. He requires detailed documentation of every data movement and modification.

Key Requirements

  1. Immutable audit trail with cryptographic proof of records

    • Creates tamper-proof audit logs using cryptographic hash chains and digital signatures. Every data operation is recorded with timestamps, actors, and actions, providing irrefutable evidence for regulatory examinations and ensuring compliance with standards like SOX, GDPR, and Basel III.

  2. Data lineage visualization with transformation tracking

    • Provides complete visibility into data flow from source to destination, including all transformations applied. Critical for demonstrating data integrity and compliance with regulations requiring clear documentation of data handling processes.

  3. Compliance rule engine for regulatory validation

    • Enforces regulatory requirements through configurable rules that validate data handling, retention, and privacy requirements. Automatically flags violations and prevents non-compliant operations from proceeding, ensuring continuous compliance.

  4. Evidence package generation for audit submissions

    • Automatically compiles comprehensive audit packages including logs, lineage diagrams, validation reports, and cryptographic proofs. Formats evidence according to regulatory requirements, significantly reducing audit preparation time.

  5. Role-based access control with privileged operation logging

    • Implements granular access controls with complete logging of all privileged operations. Tracks who accessed what data and when, supporting principles of least privilege and providing accountability required by regulations.

Technical Requirements

Testability Requirements

  • All components must be testable via pytest without manual intervention
  • Cryptographic operations validation without compromising security
  • Mock regulatory rule scenarios
  • Audit package generation testing

Performance Expectations

  • Audit log writing with <10ms latency
  • Support for 1M+ audit events per day
  • Lineage graph generation for complex migrations in <30 seconds
  • Evidence package compilation in <5 minutes

Integration Points

  • Blockchain or distributed ledger for immutable storage
  • Regulatory reporting systems (XBRL, SFTP)
  • Identity management systems for access control
  • Visualization tools for lineage diagrams

Key Constraints

  • Cryptographic non-repudiation of all records
  • Long-term audit log retention (7+ years)
  • Support for multiple regulatory frameworks
  • Tamper-evident storage mechanisms

IMPORTANT: The implementation should have NO UI/UX components. All functionality must be implemented as testable Python modules and classes that can be thoroughly tested using pytest. Focus on creating well-defined APIs and interfaces rather than user interfaces.

Core Functionality

The framework must provide:

  1. Cryptographic Audit Logger: Implements hash chain integrity with SHA-256 or stronger, applies digital signatures to audit entries, provides timestamp authority integration, and supports distributed consensus mechanisms

  2. Lineage Tracking Engine: Captures complete data flow paths, records all transformation operations, maintains parent-child relationships, and generates visual lineage graphs

  3. Compliance Rule Processor: Evaluates data against regulatory rules, implements rule versioning and history, provides real-time compliance checking, and generates violation reports

  4. Evidence Package Compiler: Aggregates all relevant audit data, formats according to regulatory templates, includes cryptographic proofs, and provides package integrity verification

  5. Access Control Manager: Implements fine-grained permissions, logs all access attempts and changes, provides privileged access management, and maintains access review trails

Testing Requirements

Key Functionalities to Verify

  • Audit trail immutability and tamper detection
  • Complete data lineage capture without gaps
  • Compliance rule accuracy for various regulations
  • Evidence package completeness and format compliance
  • Access control enforcement and logging accuracy

Critical User Scenarios

  • Demonstrating GDPR compliance for customer data migration
  • Providing SOX evidence for financial data transformations
  • Tracking HIPAA-compliant healthcare data movements
  • Generating Basel III reports for risk data migrations
  • Supporting forensic investigation of data breaches

Performance Benchmarks

  • Generate 100K audit events with cryptographic proofs per hour
  • Track lineage for migrations involving 1M+ records
  • Evaluate 1000 compliance rules per second
  • Compile 1GB evidence packages in <5 minutes
  • Handle 10K concurrent access control checks

Edge Cases and Error Conditions

  • Clock synchronization issues affecting timestamps
  • Audit log storage approaching capacity
  • Conflicting compliance rules
  • Corrupted cryptographic signatures
  • Access control system unavailability

Required Test Coverage

  • Minimum 95% code coverage for compliance-critical components
  • Cryptographic algorithm correctness tests
  • Regulatory rule validation across frameworks
  • Evidence package format compliance tests
  • Security penetration testing for access controls

IMPORTANT:

  • ALL functionality must be testable via pytest without any manual intervention
  • Tests should verify behavior against requirements, not implementation details
  • Tests should be designed to validate the WHAT (requirements) not the HOW (implementation)
  • Tests should be comprehensive enough to verify all aspects of the requirements
  • Tests should not assume or dictate specific implementation approaches
  • REQUIRED: Tests must be run with pytest-json-report to generate a pytest_results.json file: 
    pip install pytest-json-report
pytest --json-report --json-report-file=pytest_results.json
  • The pytest_results.json file must be included as proof that all tests pass

Success Criteria

The implementation is successful when:

  1. All tests pass when run with pytest, with 95%+ code coverage for compliance components
  2. A valid pytest_results.json file is generated showing all tests passing
  3. Audit trails are cryptographically secure and tamper-evident
  4. Data lineage provides complete visibility without gaps
  5. Compliance validation correctly enforces all configured rules
  6. Evidence packages meet regulatory submission requirements
  7. Access control provides complete accountability

REQUIRED FOR SUCCESS:

  • All tests must pass when run with pytest
  • A valid pytest_results.json file must be generated showing all tests passing
  • The implementation must satisfy all key requirements specified for this persona

Environment Setup

To set up the development environment:

cd /path/to/data_migration_framework_compliance_auditor
uv venv
source .venv/bin/activate
uv pip install -e .

Install the project and run tests:

pip install pytest-json-report
pytest --json-report --json-report-file=pytest_results.json

REMINDER: The pytest_results.json file is MANDATORY and must be included to demonstrate that all tests pass successfully.