Skip to content

INSTRUCTIONS.md

Latest commit

 

History

History
153 lines (115 loc) · 7.55 KB

INSTRUCTIONS.md

File metadata and controls

153 lines (115 loc) · 7.55 KB

PyMigrate Microservices Migration Framework

Overview

A specialized data migration framework designed for CTOs transitioning from monolithic databases to microservices architectures. This implementation enables zero-downtime, incremental migration with bi-directional synchronization, ensuring data consistency during the critical transition period while both systems remain operational.

Persona Description

A CTO at a fast-growing startup migrating from a monolithic database to microservices architecture who needs to ensure zero data loss during the transition. He wants to migrate incrementally while keeping both systems in sync during the transition period.

Key Requirements

  1. Bi-directional sync mode for gradual migration with conflict resolution

    • Essential for maintaining data consistency when both monolith and microservices are actively used. Provides configurable conflict resolution strategies (last-write-wins, priority-based, custom rules) to handle concurrent modifications.

  2. Service boundary detection for automatic data partitioning

    • Analyzes data access patterns and relationships to suggest optimal service boundaries. Critical for avoiding distributed transactions and ensuring microservices independence while maintaining data cohesion.

  3. API endpoint generation for migrated data services

    • Automatically generates RESTful API endpoints for newly created microservices. Ensures consistent API design patterns and includes versioning, pagination, and filtering capabilities out of the box.

  4. Real-time data consistency validation between old and new systems

    • Continuously monitors data integrity across both systems with configurable consistency levels. Provides alerts for divergence and automatic reconciliation options to maintain trust during migration.

  5. Automated traffic routing with percentage-based migration control

    • Enables gradual traffic shifting from monolith to microservices with fine-grained control. Supports canary deployments, A/B testing, and instant rollback capabilities for risk mitigation.

Technical Requirements

Testability Requirements

  • All components must be testable via pytest without manual intervention
  • Mock monolithic and microservice databases for testing
  • Simulated network conditions for sync testing
  • Traffic routing simulation capabilities

Performance Expectations

  • Sub-100ms sync latency for critical data paths
  • Support for 10,000+ transactions per second
  • Conflict detection within 1 second of occurrence
  • API generation completing within 30 seconds per service

Integration Points

  • Database change capture mechanisms (triggers, CDC, binary logs)
  • Service mesh integration for traffic management
  • API gateway compatibility for routing rules
  • Message queue interfaces for async synchronization

Key Constraints

  • Zero data loss guarantee during migration
  • Eventual consistency with configurable lag tolerance
  • Backward compatibility for existing monolith clients
  • Stateless migration components for horizontal scaling

IMPORTANT: The implementation should have NO UI/UX components. All functionality must be implemented as testable Python modules and classes that can be thoroughly tested using pytest. Focus on creating well-defined APIs and interfaces rather than user interfaces.

Core Functionality

The framework must provide:

  1. Bi-directional Sync Engine: Captures changes from both systems, applies transformation rules, manages conflict resolution, and ensures eventual consistency with configurable sync intervals

  2. Service Boundary Analyzer: Examines data models, access patterns, and transaction boundaries to recommend optimal microservice decomposition with minimal cross-service dependencies

  3. API Generator: Creates standardized REST endpoints with OpenAPI specifications, request/response validation, error handling, and built-in monitoring hooks

  4. Consistency Validator: Performs continuous data comparison with efficient checksumming, identifies discrepancies, triggers reconciliation, and maintains audit logs

  5. Traffic Router: Implements percentage-based routing with sticky sessions, health-check aware routing, gradual rollout capabilities, and instant rollback mechanisms

Testing Requirements

Key Functionalities to Verify

  • Bi-directional sync accuracy under concurrent modifications
  • Service boundary recommendations for various data models
  • API endpoint generation completeness and correctness
  • Consistency validation detecting all types of discrepancies
  • Traffic routing accuracy and failover behavior

Critical User Scenarios

  • Migrating user authentication service while users are logging in
  • Handling order processing during payment service extraction
  • Maintaining inventory consistency during warehouse service migration
  • Preserving session state during gradual traffic migration
  • Rolling back a problematic service migration without data loss

Performance Benchmarks

  • Sync latency <100ms for 95th percentile of changes
  • Process 10,000 concurrent modifications without data loss
  • Generate API for 50-table service in <30 seconds
  • Validate 1M records for consistency in <60 seconds
  • Route 100K requests/second with <1ms overhead

Edge Cases and Error Conditions

  • Network partition during bi-directional sync
  • Conflicting schema changes in both systems
  • Circular dependencies in service boundaries
  • Database connection failures during migration
  • Memory pressure with large data volumes

Required Test Coverage

  • Minimum 90% code coverage with pytest
  • Integration tests for sync scenarios
  • Load tests for performance validation
  • Chaos engineering tests for failure handling
  • End-to-end migration scenario tests

IMPORTANT:

  • ALL functionality must be testable via pytest without any manual intervention
  • Tests should verify behavior against requirements, not implementation details
  • Tests should be designed to validate the WHAT (requirements) not the HOW (implementation)
  • Tests should be comprehensive enough to verify all aspects of the requirements
  • Tests should not assume or dictate specific implementation approaches
  • REQUIRED: Tests must be run with pytest-json-report to generate a pytest_results.json file: 
    pip install pytest-json-report
pytest --json-report --json-report-file=pytest_results.json
  • The pytest_results.json file must be included as proof that all tests pass

Success Criteria

The implementation is successful when:

  1. All tests pass when run with pytest, with 90%+ code coverage
  2. A valid pytest_results.json file is generated showing all tests passing
  3. Bi-directional sync maintains consistency with <100ms latency
  4. Service boundaries are correctly identified for a 100-table monolith
  5. API generation produces working endpoints for all migrated services
  6. Consistency validation detects 100% of data discrepancies
  7. Traffic routing successfully migrates load without dropping requests

REQUIRED FOR SUCCESS:

  • All tests must pass when run with pytest
  • A valid pytest_results.json file must be generated showing all tests passing
  • The implementation must satisfy all key requirements specified for this persona

Environment Setup

To set up the development environment:

cd /path/to/data_migration_framework_startup_cto
uv venv
source .venv/bin/activate
uv pip install -e .

Install the project and run tests:

pip install pytest-json-report
pytest --json-report --json-report-file=pytest_results.json

REMINDER: The pytest_results.json file is MANDATORY and must be included to demonstrate that all tests pass successfully.