diff --git a/EnterpriseIntegrationPlatform/tutorials/01-introduction.md b/EnterpriseIntegrationPlatform/tutorials/01-introduction.md index 3709872..e2c8042 100644 --- a/EnterpriseIntegrationPlatform/tutorials/01-introduction.md +++ b/EnterpriseIntegrationPlatform/tutorials/01-introduction.md @@ -1,188 +1,120 @@ # Tutorial 01 — Introduction to Enterprise Integration -## What You'll Learn - -- What enterprise integration is and why it matters -- The messaging approach to integration -- The Enterprise Integration Patterns (EIP) book and its relevance -- How this platform implements EIP patterns in a modern .NET 10 architecture - ---- - -## What Is Enterprise Integration? - -Enterprise integration is the challenge of connecting multiple applications, services, and systems so they can work together and share data. In any organization, different systems need to communicate: - -- An **order system** creates an order → the **warehouse system** needs to know about it -- A **CRM** updates a customer record → the **billing system** needs the change -- An **external partner** sends an invoice → your **ERP** needs to process it - -### The Four Integration Styles - -The EIP book identifies four fundamental integration styles: - -| Style | Description | Example | -|-------|-------------|---------| -| **File Transfer** | Systems share data by writing and reading files | Nightly CSV export/import | -| **Shared Database** | Systems read/write to the same database | Two apps sharing a SQL table | -| **Remote Procedure Invocation** | Systems call each other's APIs directly | REST API calls, gRPC | -| **Messaging** | Systems communicate through an intermediary message broker | Kafka topics, NATS subjects | - -### Why Messaging Wins - -While all four styles work, **messaging** has unique advantages for enterprise integration: - -- **Loose Coupling** — Sender and receiver don't need to know about each other -- **Reliability** — Messages persist in the broker even if the receiver is offline -- **Scalability** — Add more consumers to handle increased load -- **Flexibility** — Route, transform, filter, and enrich messages in transit -- **Resilience** — If a system crashes, messages wait in the queue until it recovers - ---- - -## The Enterprise Integration Patterns Book - -Published in 2003 by Gregor Hohpe and Bobby Woolf, *Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions* catalogs 65 patterns for messaging-based integration. These patterns are timeless — they apply whether you use Kafka, RabbitMQ, NATS, or any other message broker. - -The patterns are organized into categories: - -``` -Enterprise Integration Patterns -├── Integration Styles (File Transfer, Shared DB, RPC, Messaging) -├── Messaging Systems (Channel, Message, Pipes & Filters, Router, Translator) -├── Messaging Channels (Point-to-Point, Pub-Sub, Dead Letter, ...) -├── Message Construction (Command, Document, Event, Request-Reply, ...) -├── Message Routing (Content-Based Router, Filter, Splitter, Aggregator, ...) -├── Message Transformation (Envelope Wrapper, Enricher, Normalizer, ...) -├── Messaging Endpoints (Gateway, Consumer types, Dispatcher, ...) -└── System Management (Control Bus, Wire Tap, Message History, ...) +Enterprise integration connects applications through messaging. This platform implements 65+ EIP patterns in .NET 10. + +## Key Types + +```csharp +// The universal message wrapper — every message in the platform is an IntegrationEnvelope +// src/Contracts/IntegrationEnvelope.cs +public record IntegrationEnvelope +{ + public Guid MessageId { get; init; } + public Guid CorrelationId { get; init; } + public Guid? CausationId { get; init; } + public string Source { get; init; } + public string MessageType { get; init; } + public T Payload { get; init; } + public MessagePriority Priority { get; init; } + public string SchemaVersion { get; init; } + public MessageIntent? Intent { get; init; } + public IReadOnlyDictionary Metadata { get; init; } +} + +// Publish and consume via broker abstractions +// src/Ingestion/IMessageBrokerProducer.cs +public interface IMessageBrokerProducer +{ + Task PublishAsync(IntegrationEnvelope envelope, string topic, CancellationToken ct = default); +} + +// src/Ingestion/IMessageBrokerConsumer.cs +public interface IMessageBrokerConsumer : IAsyncDisposable +{ + Task SubscribeAsync(string topic, Func, Task> handler, CancellationToken ct = default); +} ``` -### Why These Patterns Still Matter - -Despite being 20+ years old, these patterns are the foundation of every modern integration platform: - -- **Apache Camel** implements them in Java -- **MuleSoft** and **Azure Integration Services** implement them as cloud services -- **Microsoft BizTalk Server** implemented them (now legacy) -- **This platform** implements them in modern .NET 10 with cloud-native architecture +## Exercises ---- - -## This Platform: A Modern EIP Implementation +### 1. Create an envelope and verify auto-generated fields -The Enterprise Integration Platform replaces legacy middleware (like BizTalk Server) with a modern, cloud-native architecture: +```csharp +var envelope = IntegrationEnvelope.Create( + payload: "Hello, EIP!", + source: "Tutorial01", + messageType: "greeting.created"); -``` - Enterprise Integration Platform -┌──────────────────────────────────────────────────────────────────────────┐ -│ │ -│ ┌──────────┐ ┌───────────┐ ┌───────────┐ ┌──────────────────┐ │ -│ │ Ingress │───▶│ Broker │───▶│ Temporal │───▶│ Activities │ │ -│ │ Adapters │ │ Layer │ │ Workflows │ │ (Transform/Route)│ │ -│ └──────────┘ └───────────┘ └───────────┘ └────────┬─────────┘ │ -│ ▲ │ │ │ -│ │ ▼ ▼ │ -│ ┌────┴─────┐ ┌───────────┐ ┌──────────────────┐ │ -│ │ External │ │ DLQ │ │ Connectors │ │ -│ │ Systems │ │ Topics │ │ (HTTP/SFTP/Email)│ │ -│ └──────────┘ └───────────┘ └──────────────────┘ │ -│ │ -└──────────────────────────────────────────────────────────────────────────┘ +Assert.That(envelope.MessageId, Is.Not.EqualTo(Guid.Empty)); +Assert.That(envelope.CorrelationId, Is.Not.EqualTo(Guid.Empty)); +Assert.That(envelope.Timestamp, Is.Not.EqualTo(default(DateTimeOffset))); +Assert.That(envelope.Source, Is.EqualTo("Tutorial01")); +Assert.That(envelope.Payload, Is.EqualTo("Hello, EIP!")); ``` -### Key Technology Choices +### 2. Check default values on a new envelope -| Component | Technology | Why | -|-----------|-----------|-----| -| **Runtime** | .NET 10 (C# 14) | Latest LTS, high performance, cross-platform | -| **Orchestration** | .NET Aspire | Single-command local dev with all dependencies | -| **Event Streaming** | Apache Kafka | Best for ordered event streams and audit logs | -| **Task Delivery** | NATS JetStream / Pulsar | No head-of-line blocking for task queues | -| **Workflows** | Temporal.io | Durable execution with saga compensation | -| **Storage** | Apache Cassandra | Write-optimized distributed storage | -| **Observability** | OpenTelemetry + Grafana | End-to-end distributed tracing and metrics | -| **AI** | Ollama + RagFlow | Self-hosted RAG for developer assistance | +```csharp +var envelope = IntegrationEnvelope.Create("payload", "source", "type"); -### Design Principles - -1. **Zero Message Loss** — Every accepted message is either delivered or routed to a Dead Letter Queue. No silent drops. -2. **Ack/Nack Loopback** — Every integration publishes an acknowledgment (Ack) on success or negative acknowledgment (Nack) on failure. -3. **Atomic Processing** — Temporal workflows ensure all-or-nothing execution with compensation. -4. **Multi-Tenant Isolation** — Tenant A's messages never mix with Tenant B's. -5. **AI-Driven Generation** — Write a minimal spec, let AI generate production-ready integrations. - ---- +Assert.That(envelope.SchemaVersion, Is.EqualTo("1.0")); +Assert.That(envelope.Priority, Is.EqualTo(MessagePriority.Normal)); +Assert.That(envelope.CausationId, Is.Null); +Assert.That(envelope.ReplyTo, Is.Null); +Assert.That(envelope.Metadata, Is.Empty); +``` -## How the Platform Maps to EIP Patterns +### 3. Set message intent using `with` expression -Every EIP pattern has a corresponding platform component: +```csharp +var command = IntegrationEnvelope.Create( + "PlaceOrder", "OrderService", "order.place") with +{ + Intent = MessageIntent.Command, +}; -| EIP Pattern | Platform Component | Project | -|-------------|-------------------|---------| -| Message | `IntegrationEnvelope` | `src/Contracts/` | -| Message Channel | Broker abstraction | `src/Ingestion/` | -| Content-Based Router | `IContentBasedRouter` | `src/Processing.Routing/` | -| Message Translator | `IMessageTranslator` | `src/Processing.Translator/` | -| Splitter | `IMessageSplitter` | `src/Processing.Splitter/` | -| Aggregator | `IMessageAggregator` | `src/Processing.Aggregator/` | -| Dead Letter Channel | `IDeadLetterPublisher` | `src/Processing.DeadLetter/` | -| Process Manager | Temporal Workflows | `src/Workflow.Temporal/` | -| Channel Adapter | `IConnector` | `src/Connector.*` | -| Wire Tap | OpenTelemetry | `src/Observability/` | +Assert.That(command.Intent, Is.EqualTo(MessageIntent.Command)); +``` -You'll explore each of these in detail throughout this course. +### 4. Verify platform types exist (EIP pattern mapping) ---- +```csharp +// EIP: Message Channel → IMessageBrokerProducer +var producerType = typeof(IMessageBrokerProducer); +Assert.That(producerType.IsInterface, Is.True); +Assert.That(producerType.GetMethod("PublishAsync"), Is.Not.Null); -## What You'll Build +// EIP: Message Endpoint → IMessageBrokerConsumer +var consumerType = typeof(IMessageBrokerConsumer); +Assert.That(consumerType.IsInterface, Is.True); +Assert.That(consumerType.GetMethod("SubscribeAsync"), Is.Not.Null); +``` -By the end of this course, you'll understand how to: +### 5. Verify IntegrationEnvelope is a C# record with value equality -1. **Design** integration solutions using EIP patterns -2. **Build** message-driven pipelines with routing, transformation, and delivery -3. **Configure** message brokers (Kafka, NATS, Pulsar) for different workloads -4. **Orchestrate** complex workflows with Temporal and saga compensation -5. **Monitor** message flow with OpenTelemetry and the OpenClaw UI -6. **Deploy** to Kubernetes with Helm and Kustomize -7. **Secure** integrations with input sanitization, encryption, and multi-tenancy -8. **Scale** with competing consumers, throttling, and backpressure -9. **Test** integrations with unit, contract, integration, and load tests -10. **Operate** in production with disaster recovery and performance profiling +```csharp +var envelopeType = typeof(IntegrationEnvelope); +Assert.That(envelopeType.IsClass, Is.True); ---- +var equatable = typeof(IEquatable>); +Assert.That(equatable.IsAssignableFrom(envelopeType), Is.True); +``` ## Lab -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial01/Lab.cs`](../tests/TutorialLabs/Tutorial01/Lab.cs) - -**Objective:** Map EIP pattern categories to concrete platform components and trace how the Pipes and Filters architecture enables scalable message processing. - -### Step 1: Map Patterns to Projects - -Open [`docs/eip-mapping.md`](../docs/eip-mapping.md). For each of the following EIP categories, identify the `src/` project that implements it and the primary interface it exposes: - -| Category | Project | Interface | -|----------|---------|-----------| -| Message Construction | `src/Contracts/` | ? | -| Content-Based Router | `src/Processing.Routing/` | ? | -| Message Translator | `src/Processing.Translator/` | ? | -| Splitter | `src/Processing.Splitter/` | ? | -| Dead Letter Channel | `src/Processing.DeadLetter/` | ? | +Run the full lab: [`tests/TutorialLabs/Tutorial01/Lab.cs`](../tests/TutorialLabs/Tutorial01/Lab.cs) -### Step 2: Trace the Pipes and Filters Chain - -Open [`docs/architecture-overview.md`](../docs/architecture-overview.md) and trace how a single message flows through the platform: Ingress → Broker → Workflow → Activities → Connectors. For each stage, write down which EIP pattern it implements and how the platform guarantees **atomicity** (hint: look at Temporal workflows and Ack/Nack). - -### Step 3: Evaluate Scalability Points - -Identify three places in the architecture where **horizontal scaling** is possible without code changes. Consider: broker partitions, Competing Consumers (`src/Processing.CompetingConsumers/`), and workflow workers. For each, explain what happens to in-flight messages when a new instance is added. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial01.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial01/Exam.cs`](../tests/TutorialLabs/Tutorial01/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial01/Exam.cs`](../tests/TutorialLabs/Tutorial01/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial01.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/02-environment-setup.md b/EnterpriseIntegrationPlatform/tutorials/02-environment-setup.md index 6b1b9ba..b0644b8 100644 --- a/EnterpriseIntegrationPlatform/tutorials/02-environment-setup.md +++ b/EnterpriseIntegrationPlatform/tutorials/02-environment-setup.md @@ -1,297 +1,131 @@ # Tutorial 02 — Setting Up Your Environment -## What You'll Learn +Verify your .NET 10 environment by confirming that all core platform types, enums, and namespaces are present and correctly structured. -- Install .NET 10 SDK and Docker Desktop -- Clone and build the platform -- Run the test suite -- Launch the platform with .NET Aspire -- Navigate the project structure +## Key Types ---- +```csharp +// src/Contracts/IntegrationEnvelope.cs +public record IntegrationEnvelope { /* ... */ } -## Prerequisites +// src/Contracts/MessagePriority.cs +public enum MessagePriority { Low = 0, Normal = 1, High = 2, Critical = 3 } -### .NET 10 SDK +// src/Contracts/MessageIntent.cs +public enum MessageIntent { Command = 0, Document = 1, Event = 2 } -The platform targets **.NET 10** with C# 14. Install the SDK: +// src/Contracts/MessageHeaders.cs +public static class MessageHeaders +{ + public const string TraceId = "trace-id"; + public const string ContentType = "content-type"; + public const string SourceTopic = "source-topic"; + // ... 13 well-known header keys +} -**Windows (winget):** -```powershell -winget install Microsoft.DotNet.SDK.10 -``` +// src/Ingestion/IMessageBrokerProducer.cs +public interface IMessageBrokerProducer { /* ... */ } -**macOS (Homebrew):** -```bash -brew install dotnet-sdk@10 -``` +// src/Ingestion/IMessageBrokerConsumer.cs +public interface IMessageBrokerConsumer : IAsyncDisposable { /* ... */ } -**Ubuntu/Debian:** -```bash -sudo apt-get update -sudo apt-get install -y dotnet-sdk-10.0 -``` +// src/Ingestion/BrokerOptions.cs +public sealed class BrokerOptions +{ + public BrokerType BrokerType { get; set; } = BrokerType.NatsJetStream; + public string ConnectionString { get; set; } = string.Empty; + public int TransactionTimeoutSeconds { get; set; } = 30; +} -**Verify installation:** -```bash -dotnet --version -# Should output 10.0.x +// src/Ingestion/BrokerType.cs +public enum BrokerType { NatsJetStream = 0, Kafka = 1, Pulsar = 2 } ``` -### Docker Desktop +## Exercises -Docker is required to run infrastructure containers (Kafka, NATS, Temporal, Cassandra, Ollama) via .NET Aspire: +### 1. Verify core types exist -- Download from [docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop/) -- Ensure Docker is running before starting the platform +```csharp +var envelopeType = typeof(IntegrationEnvelope); +Assert.That(envelopeType, Is.Not.Null); +Assert.That(envelopeType.IsGenericType || envelopeType.IsClass, Is.True); -### .NET Aspire Templates +var producerType = typeof(IMessageBrokerProducer); +Assert.That(producerType.IsInterface, Is.True); -```bash -dotnet new install Aspire.ProjectTemplates +var consumerType = typeof(IMessageBrokerConsumer); +Assert.That(consumerType.IsInterface, Is.True); +Assert.That(typeof(IAsyncDisposable).IsAssignableFrom(consumerType), Is.True); ``` ---- - -## Clone and Build - -### Step 1: Clone the Repository +### 2. Verify BrokerType enum has exactly three values -```bash -git clone https://github.com/devstress/My3DLearning.git -cd My3DLearning/EnterpriseIntegrationPlatform -``` - -### Step 2: Restore Dependencies +```csharp +Assert.That(Enum.IsDefined(typeof(BrokerType), BrokerType.NatsJetStream), Is.True); +Assert.That(Enum.IsDefined(typeof(BrokerType), BrokerType.Kafka), Is.True); +Assert.That(Enum.IsDefined(typeof(BrokerType), BrokerType.Pulsar), Is.True); -```bash -dotnet restore EnterpriseIntegrationPlatform.sln +var values = Enum.GetValues(); +Assert.That(values, Has.Length.EqualTo(3)); ``` -This downloads all NuGet packages defined in `Directory.Packages.props` (central package management). +### 3. Verify MessagePriority ordinal values -### Step 3: Build the Solution +```csharp +Assert.That((int)MessagePriority.Low, Is.EqualTo(0)); +Assert.That((int)MessagePriority.Normal, Is.EqualTo(1)); +Assert.That((int)MessagePriority.High, Is.EqualTo(2)); +Assert.That((int)MessagePriority.Critical, Is.EqualTo(3)); -```bash -dotnet build EnterpriseIntegrationPlatform.sln +var values = Enum.GetValues(); +Assert.That(values, Has.Length.EqualTo(4)); ``` -A clean build should complete with **0 errors**. The solution contains many projects — this takes 30–60 seconds on first build. +### 4. Verify Contracts namespace contains expected types -### Step 4: Run the Tests +```csharp +var assembly = typeof(IntegrationEnvelope<>).Assembly; +var typeNames = assembly.GetTypes() + .Where(t => t.Namespace == "EnterpriseIntegrationPlatform.Contracts") + .Select(t => t.Name) + .ToList(); -```bash -dotnet test EnterpriseIntegrationPlatform.sln +Assert.That(typeNames, Does.Contain("MessagePriority")); +Assert.That(typeNames, Does.Contain("MessageIntent")); +Assert.That(typeNames, Does.Contain("MessageHeaders")); ``` -The test suite includes: +### 5. Verify Ingestion namespace contains expected types -| Test Project | Description | -|-------------|-------------| -| UnitTests | Fast, isolated tests for every component (most numerous) | -| ContractTests | Contract verification between services | -| WorkflowTests | Temporal workflow behavior tests | -| IntegrationTests | Testcontainers-based tests with real infrastructure | -| PlaywrightTests | End-to-end browser tests for Admin dashboard & OpenClaw UI | -| LoadTests | Performance and throughput benchmarks | +```csharp +var assembly = typeof(IMessageBrokerProducer).Assembly; +var typeNames = assembly.GetTypes() + .Where(t => t.Namespace == "EnterpriseIntegrationPlatform.Ingestion") + .Select(t => t.Name) + .ToList(); -> **Note:** IntegrationTests and PlaywrightTests require Docker to be running. - ---- - -## Launch with .NET Aspire - -```bash -cd src/AppHost -dotnet run +Assert.That(typeNames, Does.Contain("IMessageBrokerProducer")); +Assert.That(typeNames, Does.Contain("IMessageBrokerConsumer")); +Assert.That(typeNames, Does.Contain("BrokerOptions")); +Assert.That(typeNames, Does.Contain("BrokerType")); ``` -The Aspire dashboard opens automatically in your browser. You'll see: - -- **All platform services** — Gateway.Api, Admin.Api, OpenClaw.Web, Demo.Pipeline -- **Infrastructure containers** — Kafka, NATS, Temporal, Cassandra, Ollama -- **Logs** — Structured logs from every service in one view -- **Traces** — Distributed traces showing message flow -- **Metrics** — Prometheus-compatible metrics - -### The Aspire Dashboard - -The dashboard at `https://localhost:15888` (or the URL shown in console output) shows: - -``` -┌─────────────────────────────────────────────────────────┐ -│ Aspire Dashboard │ -│ │ -│ Resources: │ -│ ✅ gateway-api Running :8080 │ -│ ✅ admin-api Running :8081 │ -│ ✅ openclaw-web Running :8082 │ -│ ✅ kafka Running :15092 │ -│ ✅ nats Running :15222 │ -│ ✅ temporal Running :15233 │ -│ ✅ cassandra Running :15942 │ -│ ✅ ollama Running :15434 │ -│ │ -│ [Logs] [Traces] [Metrics] [Structured] │ -└─────────────────────────────────────────────────────────┘ -``` - -> **Port Range:** The platform uses ports in the 15xxx range to avoid conflicts with existing services on your machine. - ---- - -## Project Structure Overview - -``` -EnterpriseIntegrationPlatform/ -├── src/ # Source code -│ ├── AppHost/ # .NET Aspire orchestrator -│ ├── ServiceDefaults/ # Shared OpenTelemetry & health checks -│ ├── Contracts/ # IntegrationEnvelope & shared interfaces -│ ├── Ingestion/ # Broker abstraction layer -│ ├── Ingestion.Kafka/ # Apache Kafka provider -│ ├── Ingestion.Nats/ # NATS JetStream provider -│ ├── Ingestion.Pulsar/ # Apache Pulsar provider -│ ├── Workflow.Temporal/ # Temporal workflow worker -│ ├── Activities/ # Workflow activity implementations -│ ├── Processing.*/ # Message processing patterns -│ ├── Connector.*/ # Protocol-specific connectors -│ ├── Gateway.Api/ # API gateway (Messaging Gateway) -│ ├── Admin.Api/ # Administration REST API (Control Bus) -│ └── ... # And more (see full list in README) -│ -├── tests/ # Test projects -│ ├── UnitTests/ # Fast, isolated unit tests -│ ├── ContractTests/ # Contract verification tests -│ ├── WorkflowTests/ # Temporal workflow tests -│ ├── IntegrationTests/ # Testcontainers integration tests -│ ├── PlaywrightTests/ # E2E browser tests -│ └── LoadTests/ # Performance benchmarks -│ -├── docs/ # Architecture & design documentation -├── deploy/ # Helm charts, Kustomize, K8s manifests -├── rules/ # Development standards & milestones -└── tutorials/ # This tutorial course -``` - -### Key Files - -| File | Purpose | -|------|---------| -| `Directory.Build.props` | Shared MSBuild properties (target framework, nullable, implicit usings) | -| `Directory.Packages.props` | Central NuGet package version management | -| `global.json` | SDK version constraint with `rollForward: latestMinor` | -| `EnterpriseIntegrationPlatform.sln` | Solution file linking all projects | - ---- - -## IDE Setup - -### Visual Studio 2022 (v17.12+) - -1. Open `EnterpriseIntegrationPlatform.sln` -2. Set `AppHost` as the startup project -3. Press **F5** to launch with the Aspire dashboard - -### Visual Studio Code - -Install these extensions: -- [C# Dev Kit](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csdevkit) -- [.NET Aspire](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.dotnet-aspire) - -### JetBrains Rider (2024.3+) - -- Open the `.sln` file — Aspire support is built-in - ---- - -## Verify Your Setup - -Run this quick verification: - -```bash -# 1. Build should succeed -dotnet build EnterpriseIntegrationPlatform.sln - -# 2. Unit tests should pass -dotnet test tests/UnitTests/UnitTests.csproj - -# 3. Aspire should start (Ctrl+C to stop) -cd src/AppHost && dotnet run -``` - -If all three succeed, your environment is ready. - ---- - -## Troubleshooting - -### "The SDK 'Microsoft.NET.Sdk' specified could not be found" - -Install .NET 10 SDK. The `global.json` requires SDK 10.x. - -### Aspire AppHost fails to start - -1. Ensure Docker Desktop is running -2. Check that ports in the 15xxx range are free -3. Run `dotnet restore` to ensure NuGet packages are resolved - -### Tests fail with framework errors - -```bash -dotnet --list-runtimes -``` - -You need `Microsoft.NETCore.App 10.x.x` and `Microsoft.AspNetCore.App 10.x.x`. - ---- - ## Lab -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial02/Lab.cs`](../tests/TutorialLabs/Tutorial02/Lab.cs) - -**Objective:** Build the solution, launch the Aspire orchestrator, and explore how the platform's service topology implements the EIP Messaging Gateway and Control Bus patterns. - -### Step 1: Build and Launch - -Open a terminal in the repository root and execute: +Run the full lab: [`tests/TutorialLabs/Tutorial02/Lab.cs`](../tests/TutorialLabs/Tutorial02/Lab.cs) ```bash -dotnet restore EnterpriseIntegrationPlatform.sln -dotnet build EnterpriseIntegrationPlatform.sln +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial02.Lab" ``` -Confirm the build succeeds with zero errors and zero warnings. - -### Step 2: Explore the Aspire Service Topology +## Exam -Start the orchestrator: +Coding challenges: [`tests/TutorialLabs/Tutorial02/Exam.cs`](../tests/TutorialLabs/Tutorial02/Exam.cs) ```bash -cd src/AppHost -dotnet run +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial02.Exam" ``` -Open the Aspire dashboard URL printed in the console. Identify each service and classify it by EIP role: - -| Service | EIP Role | -|---------|----------| -| Gateway.Api | Messaging Gateway — single entry point for external systems | -| Admin.Api | Control Bus — runtime administration and monitoring | -| OpenClaw.Web | ? (identify its role) | - -Click each resource's health endpoint. Explain why health checks are essential for **scalability** — what happens when a load balancer cannot determine service health? - -### Step 3: Trace a Message Path Through Services - -Using the Aspire dashboard's **Traces** tab, identify the OpenTelemetry spans created when a message enters the Gateway. Draw the message flow: Gateway → Broker → Workflow → Activities → Connector. For each hop, note which EIP pattern is being applied (e.g., Gateway = Messaging Gateway, Broker = Message Channel, Workflow = Process Manager). - -## Exam - -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial02/Exam.cs`](../tests/TutorialLabs/Tutorial02/Exam.cs) - -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. - --- **Previous: [← Tutorial 01 — Introduction](01-introduction.md)** | **Next: [Tutorial 03 — Your First Message →](03-first-message.md)** diff --git a/EnterpriseIntegrationPlatform/tutorials/03-first-message.md b/EnterpriseIntegrationPlatform/tutorials/03-first-message.md index a9e4079..dc50998 100644 --- a/EnterpriseIntegrationPlatform/tutorials/03-first-message.md +++ b/EnterpriseIntegrationPlatform/tutorials/03-first-message.md @@ -1,315 +1,141 @@ # Tutorial 03 — Your First Message -## What You'll Learn +Create an `IntegrationEnvelope`, publish it through a mocked broker, and consume it on the other side using NSubstitute. -- Create an `IntegrationEnvelope` message -- Publish it to a message broker -- Consume it from the broker -- Understand the message lifecycle - ---- - -## The IntegrationEnvelope - -Every message in the platform is wrapped in an `IntegrationEnvelope`. This is the canonical message format — no matter where a message comes from or where it's going, it always travels inside an envelope. +## Key Types ```csharp -// Location: src/Contracts/IntegrationEnvelope.cs - +// src/Contracts/IntegrationEnvelope.cs — static factory creates envelopes with auto-generated IDs public record IntegrationEnvelope { - public required Guid MessageId { get; init; } - public required Guid CorrelationId { get; init; } - public Guid? CausationId { get; init; } - public required DateTimeOffset Timestamp { get; init; } - public required string Source { get; init; } - public required string MessageType { get; init; } - public string SchemaVersion { get; init; } = "1.0"; - public MessagePriority Priority { get; init; } = MessagePriority.Normal; - public required T Payload { get; init; } - public Dictionary Metadata { get; init; } = new(); - public string? ReplyTo { get; init; } - public DateTimeOffset? ExpiresAt { get; init; } - public int? SequenceNumber { get; init; } - public int? TotalCount { get; init; } - public MessageIntent? Intent { get; init; } + public static IntegrationEnvelope Create(T payload, string source, string messageType, + Guid? correlationId = null); + // MessageId, CorrelationId, Timestamp auto-generated } -``` - -### Key Fields Explained - -| Field | Purpose | -|-------|---------| -| `MessageId` | Unique identifier for this specific message | -| `CorrelationId` | Links related messages together (e.g., a split batch) | -| `CausationId` | The MessageId of the message that caused this one | -| `Source` | Where the message originated (e.g., "order-system") | -| `MessageType` | Describes the payload type (e.g., "OrderCreated") | -| `Payload` | The actual message content (generic type `T`) | -| `Priority` | Low, Normal, High, or Critical | -| `SchemaVersion` | Schema version of the message contract (default: `"1.0"`) | -| `Intent` | Command, Document, or Event — nullable (EIP message construction patterns) | -| `Metadata` | Key-value pairs for headers (TraceId, ContentType, etc.) | - ---- - -## Creating Your First Message - -Here's how to create an envelope carrying an order payload: - -```csharp -// Define your payload -public record OrderPayload( - string OrderId, - string CustomerId, - decimal Amount, - string Currency); - -// Create the envelope -var order = new OrderPayload("ORD-001", "CUST-42", 149.99m, "USD"); - -var envelope = new IntegrationEnvelope -{ - MessageId = Guid.NewGuid(), - CorrelationId = Guid.NewGuid(), - Timestamp = DateTimeOffset.UtcNow, - Source = "order-system", - MessageType = "OrderCreated", - Payload = order, - Priority = MessagePriority.Normal, - Intent = MessageIntent.Event, - Metadata = new Dictionary - { - [MessageHeaders.ContentType] = "application/json", - [MessageHeaders.SourceTopic] = "orders.created" - } -}; -``` - -### What Each Field Means in This Context - -- **MessageId** — A unique ID for this specific order event -- **CorrelationId** — Links all messages related to this order (the initial event, any transforms, delivery confirmations) -- **Source** — The "order-system" produced this message -- **MessageType** — "OrderCreated" tells consumers what happened -- **Intent** — `Event` means "something happened" (vs. `Command` = "do something" or `Document` = "here's data") - ---- - -## Publishing to a Broker - -The platform abstracts the broker behind `IMessageBrokerProducer`: - -```csharp -// Location: src/Ingestion/IMessageBrokerProducer.cs +// src/Ingestion/IMessageBrokerProducer.cs public interface IMessageBrokerProducer { - Task PublishAsync( - IntegrationEnvelope envelope, - string topic, - CancellationToken cancellationToken = default); + Task PublishAsync(IntegrationEnvelope envelope, string topic, CancellationToken ct = default); } -``` - -Publishing our order: -```csharp -// Inject the producer via DI -public class OrderService(IMessageBrokerProducer producer) +// src/Ingestion/IMessageBrokerConsumer.cs +public interface IMessageBrokerConsumer : IAsyncDisposable { - public async Task CreateOrderAsync(OrderPayload order) - { - var envelope = new IntegrationEnvelope - { - MessageId = Guid.NewGuid(), - CorrelationId = Guid.NewGuid(), - Timestamp = DateTimeOffset.UtcNow, - Source = "order-system", - MessageType = "OrderCreated", - Payload = order, - Priority = MessagePriority.Normal, - Intent = MessageIntent.Event, - Metadata = new Dictionary - { - [MessageHeaders.ContentType] = "application/json" - } - }; - - await producer.PublishAsync(envelope, "orders.created"); - } + Task SubscribeAsync(string topic, string consumerGroup, + Func, Task> handler, CancellationToken ct = default); } ``` -The producer publishes to the **configured broker** — NATS JetStream by default, Kafka for streaming, or Pulsar for production. The code doesn't change when you switch brokers. +## Exercises ---- - -## Consuming from a Broker - -The consumer side uses `IMessageBrokerConsumer`: +### 1. Create an envelope with a string payload ```csharp -// Location: src/Ingestion/IMessageBrokerConsumer.cs - -public interface IMessageBrokerConsumer : IAsyncDisposable -{ - Task SubscribeAsync( - string topic, - string consumerGroup, - Func, Task> handler, - CancellationToken cancellationToken = default); -} +var envelope = IntegrationEnvelope.Create( + payload: "Hello, Messaging!", + source: "Tutorial03", + messageType: "greeting"); + +Assert.That(envelope.Payload, Is.EqualTo("Hello, Messaging!")); +Assert.That(envelope.Source, Is.EqualTo("Tutorial03")); +Assert.That(envelope.MessageType, Is.EqualTo("greeting")); +Assert.That(envelope.MessageId, Is.Not.EqualTo(Guid.Empty)); +Assert.That(envelope.CorrelationId, Is.Not.EqualTo(Guid.Empty)); ``` -Consuming our order: +### 2. Create an envelope with a domain object payload ```csharp -public class OrderProcessor(IMessageBrokerConsumer consumer) -{ - public async Task StartAsync(CancellationToken ct) - { - await consumer.SubscribeAsync( - topic: "orders.created", - consumerGroup: "order-processors", - handler: async envelope => - { - Console.WriteLine($"Received order: {envelope.Payload.OrderId}"); - Console.WriteLine($" Amount: {envelope.Payload.Amount} {envelope.Payload.Currency}"); - Console.WriteLine($" CorrelationId: {envelope.CorrelationId}"); - Console.WriteLine($" Timestamp: {envelope.Timestamp}"); - }, - cancellationToken: ct); - } -} -``` - -### Consumer Groups - -The `consumerGroup` parameter is important: - -- **Same consumer group** = messages are distributed across consumers (load balancing) -- **Different consumer groups** = each group gets every message (fan-out) +public sealed record OrderPayload(string OrderId, string Product, int Quantity); -This maps directly to the EIP patterns: -- Same group → **Competing Consumers** pattern -- Different groups → **Publish-Subscribe Channel** pattern +var order = new OrderPayload("ORD-100", "Gadget", 3); ---- - -## The Message Lifecycle - -When you publish a message, here's what happens: +var envelope = IntegrationEnvelope.Create( + payload: order, + source: "OrderService", + messageType: "order.created"); +Assert.That(envelope.Payload, Is.EqualTo(order)); +Assert.That(envelope.Payload.OrderId, Is.EqualTo("ORD-100")); +Assert.That(envelope.Payload.Product, Is.EqualTo("Gadget")); +Assert.That(envelope.Payload.Quantity, Is.EqualTo(3)); ``` -1. CREATE → IntegrationEnvelope created with unique MessageId -2. PUBLISH → IMessageBrokerProducer publishes to broker topic -3. PERSIST → Broker durably stores the message (Kafka log / NATS stream) -4. CONSUME → IMessageBrokerConsumer picks up the message -5. WORKFLOW → Temporal workflow orchestrates processing -6. ACTIVITIES → Validate → Transform → Route → Deliver -7. ACK/NACK → Success = Ack published; Failure = Nack published -8. OBSERVE → OpenTelemetry traces, logs, and metrics recorded at every step -``` - -This lifecycle is the foundation of everything in the platform. Every tutorial builds on this flow. - ---- - -## How It Connects to EIP Patterns -What we just did touches several EIP patterns: +### 3. Publish through a mocked producer and verify the call -| Pattern | Where We Used It | -|---------|-----------------| -| **Message** | `IntegrationEnvelope` wraps our payload | -| **Message Channel** | The `"orders.created"` topic is a channel | -| **Document Message** / **Event Message** | The `Intent` field distinguishes message types | -| **Correlation Identifier** | `CorrelationId` links related messages | -| **Envelope Wrapper** | The envelope wraps the raw `OrderPayload` with metadata | -| **Format Indicator** | `MessageHeaders.ContentType` tells consumers the format | - ---- +```csharp +var producer = Substitute.For(); -## Writing a Test +var envelope = IntegrationEnvelope.Create( + "first-message", "Tutorial03", "demo.publish"); -The platform uses NUnit 4.4.0 for testing. Here's how you'd test envelope creation: +await producer.PublishAsync(envelope, "demo-topic"); -```csharp -[TestFixture] -public class IntegrationEnvelopeTests -{ - [Test] - public void Create_SetsMessageId() - { - var envelope = new IntegrationEnvelope - { - MessageId = Guid.NewGuid(), - CorrelationId = Guid.NewGuid(), - Timestamp = DateTimeOffset.UtcNow, - Source = "test", - MessageType = "TestMessage", - Payload = "hello", - Priority = MessagePriority.Normal, - Intent = MessageIntent.Document, - Metadata = new Dictionary() - }; - - Assert.That(envelope.MessageId, Is.Not.EqualTo(Guid.Empty)); - Assert.That(envelope.Payload, Is.EqualTo("hello")); - Assert.That(envelope.Intent, Is.EqualTo(MessageIntent.Document)); - } -} +await producer.Received(1).PublishAsync( + Arg.Is>(e => e.Payload == "first-message"), + Arg.Is("demo-topic"), + Arg.Any()); ``` -> **Testing convention:** The platform uses NUnit with `[SetUp]` for per-test initialization and NSubstitute for mocking. See `rules/coding-standards.md` for full conventions. +### 4. Subscribe with a mocked consumer and simulate message delivery ---- +```csharp +var consumer = Substitute.For(); +Func, Task>? capturedHandler = null; -## Lab +consumer.SubscribeAsync( + Arg.Any(), Arg.Any(), + Arg.Do, Task>>(h => capturedHandler = h), + Arg.Any()) + .Returns(Task.CompletedTask); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial03/Lab.cs`](../tests/TutorialLabs/Tutorial03/Lab.cs) +await consumer.SubscribeAsync( + "demo-topic", "demo-group", msg => Task.CompletedTask); -**Objective:** Create an `IntegrationEnvelope`, publish it to a Message Channel, and trace the Correlation Identifier through a publish-subscribe round-trip. +var envelope = IntegrationEnvelope.Create( + "consumed-payload", "Producer", "demo.event"); -### Step 1: Create and Inspect an Integration Envelope +Assert.That(capturedHandler, Is.Not.Null); -Using the static factory method, create an envelope and inspect the EIP Message pattern fields it populates automatically: +IntegrationEnvelope? received = null; +capturedHandler = msg => { received = msg; return Task.CompletedTask; }; +await capturedHandler(envelope); -```csharp -var envelope = IntegrationEnvelope.Create( - payload: "{\"orderId\": 42, \"amount\": 99.95}", - source: "OrderService", - messageType: "order.created"); +Assert.That(received, Is.Not.Null); +Assert.That(received!.Payload, Is.EqualTo("consumed-payload")); ``` -Verify: `MessageId` is a non-empty `Guid` (Message Identity), `CorrelationId` is generated (Correlation Identifier pattern), `Timestamp` is UTC (for ordering and expiration), and `Priority` defaults to `Normal`. +### 5. Verify subscribe was called with correct topic and consumer group -### Step 2: Trace the Message Lifecycle +```csharp +var consumer = Substitute.For(); -Draw the 8-step message lifecycle from the tutorial on paper or whiteboard: +await consumer.SubscribeAsync( + "events-topic", "my-consumer-group", _ => Task.CompletedTask); +await consumer.Received(1).SubscribeAsync( + Arg.Is("events-topic"), + Arg.Is("my-consumer-group"), + Arg.Any, Task>>(), + Arg.Any()); ``` -CREATE → PUBLISH → PERSIST → CONSUME → WORKFLOW → ACTIVITIES → ACK/NACK → OBSERVE -``` - -For each step, identify: (a) which EIP pattern applies, (b) where **atomicity** is enforced (hint: PERSIST ensures durability, WORKFLOW ensures all-or-nothing), and (c) which step enables **scalability** through parallel processing (hint: CONSUME with consumer groups). -### Step 3: Design a Multi-Consumer Topology - -Imagine you need both an **analytics service** and a **billing service** to receive `order.created` messages. Design the consumer group configuration: +## Lab -- Analytics: consumer group = `"analytics-processors"` (receives every message) -- Billing: consumer group = `"billing-processors"` (receives every message) -- Within billing, 3 instances share the load +Run the full lab: [`tests/TutorialLabs/Tutorial03/Lab.cs`](../tests/TutorialLabs/Tutorial03/Lab.cs) -Explain which EIP patterns are at play: **Publish-Subscribe Channel** (different groups) vs. **Competing Consumers** (same group, multiple instances). Why does this design scale without code changes? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial03.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial03/Exam.cs`](../tests/TutorialLabs/Tutorial03/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial03/Exam.cs`](../tests/TutorialLabs/Tutorial03/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial03.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/04-integration-envelope.md b/EnterpriseIntegrationPlatform/tutorials/04-integration-envelope.md index 0beea4c..fe41334 100644 --- a/EnterpriseIntegrationPlatform/tutorials/04-integration-envelope.md +++ b/EnterpriseIntegrationPlatform/tutorials/04-integration-envelope.md @@ -1,137 +1,35 @@ # Tutorial 04 — The Integration Envelope -## What You'll Learn +Deep dive into every `IntegrationEnvelope` property: identity, expiration, metadata headers, sequence numbers, and immutable record semantics. -- Every field of `IntegrationEnvelope` and when to use it -- Message headers and metadata conventions -- How the envelope implements multiple EIP patterns -- Envelope immutability and the causation chain - ---- - -## The Canonical Message Format - -The `IntegrationEnvelope` is the single most important type in the platform. Every message — whether it's an HTTP request body, an SFTP file, an email, or an internal event — is wrapped in this envelope before entering the processing pipeline. - -``` -┌─────────────────────────────────────────────────────┐ -│ IntegrationEnvelope │ -│ │ -│ ┌─── Identity ──────────────────────────────────┐ │ -│ │ MessageId : Guid (unique per message) │ │ -│ │ CorrelationId : Guid (links related msgs) │ │ -│ │ CausationId : Guid? (parent message) │ │ -│ └───────────────────────────────────────────────┘ │ -│ │ -│ ┌─── Source & Type ─────────────────────────────┐ │ -│ │ Source : string ("order-system") │ │ -│ │ MessageType : string ("OrderCreated") │ │ -│ │ SchemaVersion : string (default "1.0") │ │ -│ │ Timestamp : DateTimeOffset │ │ -│ └───────────────────────────────────────────────┘ │ -│ │ -│ ┌─── Payload ───────────────────────────────────┐ │ -│ │ Payload : T (the actual data) │ │ -│ └───────────────────────────────────────────────┘ │ -│ │ -│ ┌─── Quality of Service ────────────────────────┐ │ -│ │ Priority : Low | Normal | High | Crit │ │ -│ │ ReplyTo : string? (return address) │ │ -│ │ ExpiresAt : DateTimeOffset? (TTL) │ │ -│ └───────────────────────────────────────────────┘ │ -│ │ -│ ┌─── Sequencing ────────────────────────────────┐ │ -│ │ SequenceNumber : int? (position in batch) │ │ -│ │ TotalCount : int? (total in batch) │ │ -│ └───────────────────────────────────────────────┘ │ -│ │ -│ ┌─── Classification ───────────────────────────┐ │ -│ │ Intent : Command | Document | Event? │ │ -│ └───────────────────────────────────────────────┘ │ -│ │ -│ ┌─── Metadata ──────────────────────────────────┐ │ -│ │ Dictionary (extensible headers)│ │ -│ └───────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────┘ -``` - ---- - -## Field-by-Field Deep Dive - -### Identity Fields - -**MessageId** — Every message gets a unique `Guid`. This is used for: -- Idempotent processing (deduplication in Cassandra) -- Audit trail -- DLQ tracking - -**CorrelationId** — Links all messages that belong to the same logical operation. When a message is split into parts, all parts share the same `CorrelationId`. When a request gets a reply, both carry the same `CorrelationId`. - -**CausationId** — Points to the `MessageId` of the message that caused this one. This builds a causal chain: - -``` -Original Order (MessageId: A, CausationId: null) - └─ Validated Order (MessageId: B, CausationId: A) - └─ Transformed Order (MessageId: C, CausationId: B) - └─ Delivery Confirmation (MessageId: D, CausationId: C) -``` - -### Source and Type - -**Source** — Identifies the originating system. Examples: `"order-system"`, `"crm"`, `"partner-api"`. - -**MessageType** — A string discriminator for the payload type. Routing rules often match on this field. - -**SchemaVersion** — Version of the message contract schema (default: `"1.0"`). Allows consumers to handle schema evolution. - -**Timestamp** — When the message was created (UTC). Used for ordering, expiration checks, and audit. - -### Quality of Service - -**Priority** — Determines processing order when queues have backlog: - -| Priority | Use Case | -|----------|----------| -| `Critical` | System alerts, circuit breaker notifications | -| `High` | Financial transactions, SLA-bound deliveries | -| `Normal` | Standard business messages (default) | -| `Low` | Analytics, batch reports, non-urgent | - -**ReplyTo** — Implements the **Return Address** EIP pattern. Tells the receiver where to send the response. - -**ExpiresAt** — Implements the **Message Expiration** EIP pattern. The `MessageExpirationChecker` in `Processing.DeadLetter` routes expired messages to the DLQ. - -### Sequencing - -**SequenceNumber** and **TotalCount** — Used by the **Splitter** and **Resequencer** patterns: - -``` -Original batch (5 items) → Splitter produces: - Item 1: SequenceNumber=1, TotalCount=5, CorrelationId=X - Item 2: SequenceNumber=2, TotalCount=5, CorrelationId=X - Item 3: SequenceNumber=3, TotalCount=5, CorrelationId=X - Item 4: SequenceNumber=4, TotalCount=5, CorrelationId=X - Item 5: SequenceNumber=5, TotalCount=5, CorrelationId=X -``` - -The **Aggregator** uses `TotalCount` to know when all parts have arrived. - -### Message Intent - -**Intent** (`MessageIntent?`, nullable) — Implements three EIP **Message Construction** patterns: - -| Intent | EIP Pattern | Meaning | Example | -|--------|------------|---------|---------| -| `Command` | Command Message | "Do this" — an instruction to perform an action | "ProcessPayment" | -| `Document` | Document Message | "Here's data" — information transfer with no expected action | "QuarterlyReport" | -| `Event` | Event Message | "This happened" — notification of something that occurred | "OrderCreated" | - -### Metadata - -The `Metadata` dictionary carries extensible key-value headers. The `MessageHeaders` static class defines well-known keys: +## Key Types ```csharp +// src/Contracts/IntegrationEnvelope.cs +public record IntegrationEnvelope +{ + public Guid MessageId { get; init; } + public Guid CorrelationId { get; init; } + public Guid? CausationId { get; init; } + public DateTimeOffset Timestamp { get; init; } + public string Source { get; init; } + public string MessageType { get; init; } + public string SchemaVersion { get; init; } = "1.0"; + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public T Payload { get; init; } + public Dictionary Metadata { get; init; } = new(); + public string? ReplyTo { get; init; } + public DateTimeOffset? ExpiresAt { get; init; } + public int? SequenceNumber { get; init; } + public int? TotalCount { get; init; } + public MessageIntent? Intent { get; init; } + public bool IsExpired { get; } + + public static IntegrationEnvelope Create(T payload, string source, string messageType, + Guid? correlationId = null); +} + +// src/Contracts/MessageHeaders.cs public static class MessageHeaders { public const string TraceId = "trace-id"; @@ -141,107 +39,145 @@ public static class MessageHeaders public const string ConsumerGroup = "consumer-group"; public const string RetryCount = "retry-count"; public const string SequenceNumber = "sequence-number"; - // ... and more + public const string TotalCount = "total-count"; } ``` ---- - -## EIP Patterns Implemented by the Envelope - -The envelope alone implements **9 EIP patterns**: - -| EIP Pattern | Envelope Feature | -|-------------|-----------------| -| **Message** | The envelope IS the message | -| **Envelope Wrapper** | Wraps raw payload with standardized metadata | -| **Correlation Identifier** | `CorrelationId` field | -| **Return Address** | `ReplyTo` field | -| **Message Expiration** | `ExpiresAt` field | -| **Message Sequence** | `SequenceNumber` + `TotalCount` fields | -| **Format Indicator** | `MessageHeaders.ContentType` in metadata | -| **Command Message** | `Intent = Command` | -| **Document Message** | `Intent = Document` | -| **Event Message** | `Intent = Event` | - ---- - -## Envelope Immutability +## Exercises -`IntegrationEnvelope` is a C# `record`, which means it's **immutable by default**. When a processing step needs to modify the envelope (e.g., add metadata, change the payload), it creates a **new** envelope using `with`: +### 1. Set all properties on a complex payload envelope ```csharp -// Original envelope -var original = new IntegrationEnvelope { /* ... */ }; +public sealed record ShipmentPayload(string ShipmentId, string Carrier, decimal WeightKg, string[] Items); -// Create a new envelope with updated metadata -var enriched = original with +var items = new[] { "SKU-001", "SKU-002" }; +var shipment = new ShipmentPayload("SHIP-1", "FedEx", 12.5m, items); +var correlationId = Guid.NewGuid(); + +var envelope = IntegrationEnvelope.Create( + payload: shipment, + source: "WarehouseService", + messageType: "shipment.dispatched", + correlationId: correlationId) with { - Metadata = new Dictionary(original.Metadata) - { - ["enriched-at"] = DateTimeOffset.UtcNow.ToString("O") - } + SchemaVersion = "2.0", + Priority = MessagePriority.High, + Intent = MessageIntent.Event, + ReplyTo = "shipment-replies", + ExpiresAt = DateTimeOffset.UtcNow.AddHours(1), + SequenceNumber = 0, + TotalCount = 3, }; -// The original is unchanged -// enriched has the new metadata entry +Assert.That(envelope.Payload.ShipmentId, Is.EqualTo("SHIP-1")); +Assert.That(envelope.Payload.Carrier, Is.EqualTo("FedEx")); +Assert.That(envelope.Payload.WeightKg, Is.EqualTo(12.5m)); +Assert.That(envelope.Payload.Items, Has.Length.EqualTo(2)); +Assert.That(envelope.CorrelationId, Is.EqualTo(correlationId)); +Assert.That(envelope.SchemaVersion, Is.EqualTo("2.0")); +Assert.That(envelope.Priority, Is.EqualTo(MessagePriority.High)); +Assert.That(envelope.Intent, Is.EqualTo(MessageIntent.Event)); +Assert.That(envelope.ReplyTo, Is.EqualTo("shipment-replies")); +Assert.That(envelope.ExpiresAt, Is.Not.Null); +Assert.That(envelope.SequenceNumber, Is.EqualTo(0)); +Assert.That(envelope.TotalCount, Is.EqualTo(3)); ``` -This immutability is critical for: -- **Thread safety** — Multiple activities can read the same envelope concurrently -- **Audit trail** — Each step produces a new envelope; the original is preserved -- **Replay** — You can always replay the original, unmodified envelope +### 2. Verify unique MessageId generation and independent CorrelationIds ---- - -## The Causation Chain +```csharp +var ids = Enumerable.Range(0, 100) + .Select(_ => IntegrationEnvelope.Create("payload", "source", "type").MessageId) + .ToList(); -As a message flows through the pipeline, each step creates a new envelope with `CausationId` set to the previous step's `MessageId`. This creates a traceable chain: +Assert.That(ids.Distinct().Count(), Is.EqualTo(100), + "Each envelope must have a globally unique MessageId"); +var env1 = IntegrationEnvelope.Create("a", "src", "type"); +var env2 = IntegrationEnvelope.Create("b", "src", "type"); +Assert.That(env1.CorrelationId, Is.Not.EqualTo(env2.CorrelationId)); ``` -Ingress → Validate → Transform → Route → Deliver -Envelope A (Ingress): MessageId=aaa, CausationId=null -Envelope B (Validate): MessageId=bbb, CausationId=aaa -Envelope C (Transform): MessageId=ccc, CausationId=bbb -Envelope D (Route): MessageId=ddd, CausationId=ccc -Envelope E (Deliver): MessageId=eee, CausationId=ddd -``` +### 3. Test IsExpired with past, future, and null ExpiresAt -All five envelopes share the same `CorrelationId`. This lets you: -- Query "show me all messages for this order" (by CorrelationId) -- Query "what caused this message?" (by CausationId) -- Reconstruct the full processing history +```csharp +var expired = IntegrationEnvelope.Create("stale", "source", "type") with +{ + ExpiresAt = DateTimeOffset.UtcNow.AddMinutes(-5), +}; +Assert.That(expired.IsExpired, Is.True); ---- +var fresh = IntegrationEnvelope.Create("fresh", "source", "type") with +{ + ExpiresAt = DateTimeOffset.UtcNow.AddHours(1), +}; +Assert.That(fresh.IsExpired, Is.False); -## Lab +var immortal = IntegrationEnvelope.Create("immortal", "source", "type"); +Assert.That(immortal.ExpiresAt, Is.Null); +Assert.That(immortal.IsExpired, Is.False); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial04/Lab.cs`](../tests/TutorialLabs/Tutorial04/Lab.cs) +### 4. Add and read metadata headers -**Objective:** Build causation chains and sequenced message sets that demonstrate how the Envelope Wrapper pattern preserves **atomicity** and **traceability** across a multi-step integration pipeline. +```csharp +var envelope = IntegrationEnvelope.Create("payload", "source", "type") with +{ + Metadata = new Dictionary + { + [MessageHeaders.ContentType] = "application/json", + [MessageHeaders.TraceId] = "abc-123-trace", + [MessageHeaders.SourceTopic] = "orders-topic", + }, +}; -### Step 1: Build a Causation Chain (Message Lineage) +Assert.That(envelope.Metadata[MessageHeaders.ContentType], Is.EqualTo("application/json")); +Assert.That(envelope.Metadata[MessageHeaders.TraceId], Is.EqualTo("abc-123-trace")); +Assert.That(envelope.Metadata[MessageHeaders.SourceTopic], Is.EqualTo("orders-topic")); +Assert.That(envelope.Metadata, Has.Count.EqualTo(3)); +``` -Write code that simulates a three-step processing pipeline. Create an original envelope with `IntegrationEnvelope.Create()`. Then create a second envelope (transformation result) using a `with` expression — set its `CausationId` to the first envelope's `MessageId` and keep the same `CorrelationId`. Create a third envelope whose `CausationId` is the second. Verify all three share the same `CorrelationId` but have distinct `MessageId` values. +### 5. Model a Splitter output with sequence numbers -This lineage is essential for **atomicity**: if step 3 fails, the saga compensation engine uses the `CausationId` chain to identify and roll back exactly the right upstream steps. +```csharp +var correlationId = Guid.NewGuid(); +var parts = Enumerable.Range(0, 3) + .Select(i => IntegrationEnvelope.Create( + payload: $"Part-{i}", + source: "Splitter", + messageType: "order.part", + correlationId: correlationId) with + { + SequenceNumber = i, + TotalCount = 3, + }) + .ToList(); -### Step 2: Model a Splitter Output with Sequencing +Assert.That(parts, Has.Count.EqualTo(3)); -Create three envelopes representing a Splitter's output. Use `with` expressions to set `SequenceNumber` (0, 1, 2) and `TotalCount` (3). Also set `ExpiresAt` on one envelope to 5 minutes from now, and on another to a time in the past. Verify `IsExpired` returns the correct value. +for (var i = 0; i < 3; i++) +{ + Assert.That(parts[i].SequenceNumber, Is.EqualTo(i)); + Assert.That(parts[i].TotalCount, Is.EqualTo(3)); + Assert.That(parts[i].CorrelationId, Is.EqualTo(correlationId)); +} +``` -Explain why the **Message Expiration** pattern is critical for scalability: in a high-throughput system, stale messages must be routed to the Dead Letter Queue rather than consuming resources processing outdated data. +## Lab -### Step 3: Design an Atomicity Scenario +Run the full lab: [`tests/TutorialLabs/Tutorial04/Lab.cs`](../tests/TutorialLabs/Tutorial04/Lab.cs) -Imagine an order message is split into 3 line-item messages. Line-item 2 fails delivery. Using the envelope fields (`CorrelationId`, `CausationId`, `SequenceNumber`, `TotalCount`), describe how the platform can: (a) identify all 3 messages as belonging to the same operation, (b) determine which specific message failed, and (c) trigger compensation for line-items 1 and 3 that already succeeded. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial04.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial04/Exam.cs`](../tests/TutorialLabs/Tutorial04/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial04/Exam.cs`](../tests/TutorialLabs/Tutorial04/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial04.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/05-message-brokers.md b/EnterpriseIntegrationPlatform/tutorials/05-message-brokers.md index 287b45f..79f4159 100644 --- a/EnterpriseIntegrationPlatform/tutorials/05-message-brokers.md +++ b/EnterpriseIntegrationPlatform/tutorials/05-message-brokers.md @@ -1,278 +1,149 @@ # Tutorial 05 — Message Brokers -## What You'll Learn +Configure the three broker implementations (NATS JetStream, Kafka, Pulsar) via `BrokerOptions` and publish messages through the broker abstraction. -- The three message brokers in the platform and when to use each -- Kafka for event streaming vs. NATS/Pulsar for task delivery -- How the broker abstraction lets you switch without code changes -- Head-of-line blocking and why it matters - ---- - -## The Three-Broker Architecture - -The platform doesn't use just one message broker — it uses the **right broker for each job**: - -``` -┌──────────────────────────────────────────────────────────┐ -│ Broker Layer │ -│ │ -│ ┌──────────────┐ ┌───────────────┐ ┌───────────────┐ │ -│ │ Kafka │ │ NATS │ │ Pulsar │ │ -│ │ │ │ JetStream │ │ Key_Shared │ │ -│ │ Event │ │ │ │ │ │ -│ │ Streaming │ │ Task │ │ Task │ │ -│ │ + Audit │ │ Delivery │ │ Delivery │ │ -│ │ │ │ (default) │ │ (prod scale) │ │ -│ └──────────────┘ └───────────────┘ └───────────────┘ │ -│ │ -│ IMessageBrokerProducer / IMessageBrokerConsumer │ -└──────────────────────────────────────────────────────────┘ -``` - -### When to Use Each Broker - -| Broker | Best For | Why | -|--------|----------|-----| -| **Kafka** | Event streams, audit logs, fan-out analytics | Partitioned log with ordering guarantees; excellent for high-throughput streaming | -| **NATS JetStream** | Task delivery (default), dev/test | Lightweight single binary; per-subject filtering; no head-of-line blocking | -| **Apache Pulsar** | Large-scale production task delivery | Key_Shared subscriptions; messages for recipient A never block recipient B | - ---- - -## The Head-of-Line Blocking Problem - -This is the most important concept for understanding the broker choice. - -### Kafka's Model - -Kafka organizes messages into **partitions**. Within a consumer group, each partition is consumed by **exactly one consumer**: - -``` -Topic: orders (3 partitions) - -Partition 0: [msg1] [msg3] [msg6] ──→ Consumer A -Partition 1: [msg2] [msg4] [msg7] ──→ Consumer B -Partition 2: [msg5] [msg8] [msg9] ──→ Consumer C -``` - -**The problem**: If `msg3` takes 30 seconds to process, `msg6` waits behind it. This is **head-of-line (HOL) blocking**. Message 6 might be for a completely different customer, but it's stuck. - -### NATS JetStream's Model - -NATS uses **subjects** with **queue groups**. Any consumer in the group can pick up any message: - -``` -Subject: orders.created - -Queue Group "processors": - Consumer A picks msg1 → processing... - Consumer B picks msg2 → done - Consumer B picks msg3 → done (B is fast, takes more) - Consumer C picks msg4 → done -``` - -No single slow message blocks others — each consumer processes independently. - -### Pulsar Key_Shared - -Pulsar with **Key_Shared** subscriptions routes messages by key: - -``` -Key_Shared on recipientId: - -recipientId=alice → Consumer A (all Alice's messages, in order) -recipientId=bob → Consumer B (all Bob's messages, in order) -recipientId=carol → Consumer C (all Carol's messages, in order) -``` - -**Recipient A never blocks Recipient B**, even with millions of recipients. Each recipient's messages stay ordered. - ---- - -## The Broker Abstraction - -The platform abstracts all three brokers behind two interfaces: +## Key Types ```csharp -// src/Ingestion/IMessageBrokerProducer.cs -public interface IMessageBrokerProducer +// src/Ingestion/BrokerType.cs +public enum BrokerType { - Task PublishAsync( - IntegrationEnvelope envelope, - string topic, - CancellationToken cancellationToken = default); + NatsJetStream = 0, // Default — lightweight, no HOL blocking + Kafka = 1, // Event streaming, audit logs, fan-out + Pulsar = 2, // Key_Shared — per-recipient ordering at scale } -// src/Ingestion/IMessageBrokerConsumer.cs -public interface IMessageBrokerConsumer : IAsyncDisposable +// src/Ingestion/BrokerOptions.cs +public sealed class BrokerOptions { - Task SubscribeAsync( - string topic, - string consumerGroup, - Func, Task> handler, - CancellationToken cancellationToken = default); + public BrokerType BrokerType { get; set; } = BrokerType.NatsJetStream; + public string ConnectionString { get; set; } = string.Empty; + public int TransactionTimeoutSeconds { get; set; } = 30; +} + +// src/Ingestion/IMessageBrokerProducer.cs +public interface IMessageBrokerProducer +{ + Task PublishAsync(IntegrationEnvelope envelope, string topic, CancellationToken ct = default); } ``` -### Switching Brokers +## Exercises -The broker is a **deployment-time configuration** — no code changes required: +### 1. Configure BrokerOptions for each broker ```csharp -// In DI registration (simplified) -services.AddIngestion(options => +var nats = new BrokerOptions +{ + BrokerType = BrokerType.NatsJetStream, + ConnectionString = "nats://localhost:15222", + TransactionTimeoutSeconds = 30, +}; +Assert.That(nats.BrokerType, Is.EqualTo(BrokerType.NatsJetStream)); +Assert.That(nats.ConnectionString, Is.EqualTo("nats://localhost:15222")); + +var kafka = new BrokerOptions { - options.BrokerType = BrokerType.NatsJetStream; // Default - // options.BrokerType = BrokerType.Kafka; // For streaming - // options.BrokerType = BrokerType.Pulsar; // For production scale -}); + BrokerType = BrokerType.Kafka, + ConnectionString = "localhost:9092", + TransactionTimeoutSeconds = 60, +}; +Assert.That(kafka.BrokerType, Is.EqualTo(BrokerType.Kafka)); +Assert.That(kafka.TransactionTimeoutSeconds, Is.EqualTo(60)); + +var pulsar = new BrokerOptions +{ + BrokerType = BrokerType.Pulsar, + ConnectionString = "pulsar://localhost:6650", + TransactionTimeoutSeconds = 45, +}; +Assert.That(pulsar.BrokerType, Is.EqualTo(BrokerType.Pulsar)); ``` -Your publishing and consuming code stays identical: +### 2. Publish through a mocked NATS producer ```csharp -// This code works with ANY broker -await producer.PublishAsync(envelope, "orders.created"); - -await consumer.SubscribeAsync( - "orders.created", "processors", HandleOrder); -``` - ---- +var producer = Substitute.For(); -## Kafka: The Event Streaming Backbone +var envelope = IntegrationEnvelope.Create( + "nats-message", "NatsService", "nats.event"); -Kafka excels at: +await producer.PublishAsync(envelope, "nats-events"); -- **Ordered event streams** — Events within a partition maintain strict order -- **Audit logging** — Immutable append-only log with configurable retention -- **Fan-out analytics** — Multiple consumer groups each get every message -- **Replay** — Read from any offset to reprocess historical events - -The platform uses Kafka specifically for: - -| Use Case | Topic Pattern | -|----------|---------------| -| Audit events | `audit.*` | -| Event streaming | `events.*` | -| Analytics fan-out | `analytics.*` | -| Ack/Nack notifications | `notifications.ack.*`, `notifications.nack.*` | - -### Kafka Configuration - -```csharp -// src/Ingestion.Kafka/ provides the Kafka implementation -// Key settings: -// - Bootstrap servers (connection) -// - Consumer group IDs -// - Partition count per topic -// - Replication factor -// - Retention period +await producer.Received(1).PublishAsync( + Arg.Is>(e => e.Payload == "nats-message"), + Arg.Is("nats-events"), + Arg.Any()); ``` ---- +### 3. Publish through a mocked Kafka producer -## NATS JetStream: The Default Task Broker +```csharp +var producer = Substitute.For(); -NATS JetStream is the **default broker** for task-oriented message delivery: +var envelope = IntegrationEnvelope.Create( + "kafka-message", "KafkaService", "kafka.event"); -- **Lightweight** — Single binary, minimal configuration -- **Per-subject filtering** — Subscribe to specific subjects with wildcards -- **Queue groups** — Built-in competing consumers without HOL blocking -- **Cloud-native** — Perfect for local development and cloud deployments +await producer.PublishAsync(envelope, "kafka-events"); +await producer.Received(1).PublishAsync( + Arg.Is>(e => e.Payload == "kafka-message"), + Arg.Is("kafka-events"), + Arg.Any()); ``` -NATS Subject Hierarchy: - -eip.> (all platform messages) -eip.orders.> (all order messages) -eip.orders.created (specific subject) -eip.orders.validated -eip.orders.processed -eip.dlq.> (all dead letter messages) -``` - -### NATS Wildcards - -| Pattern | Matches | -|---------|---------| -| `eip.orders.*` | `eip.orders.created`, `eip.orders.validated` | -| `eip.orders.>` | `eip.orders.created`, `eip.orders.us.created` (multi-level) | -| `eip.*.created` | `eip.orders.created`, `eip.invoices.created` | ---- +### 4. Publish to multiple topics and verify each -## Apache Pulsar: Production Scale +```csharp +var producer = Substitute.For(); -For large-scale on-premises production, Pulsar with **Key_Shared** subscriptions provides: +var orderEnvelope = IntegrationEnvelope.Create( + "new-order", "OrderService", "order.created"); +var paymentEnvelope = IntegrationEnvelope.Create( + "payment-received", "PaymentService", "payment.received"); +var shippingEnvelope = IntegrationEnvelope.Create( + "shipment-dispatched", "ShippingService", "shipment.dispatched"); -- **Key-based routing** — All messages for the same key go to the same consumer -- **No cross-key blocking** — Different keys are processed independently -- **Built-in multi-tenancy** — Lightweight topic creation scales to millions of tenants -- **Tiered storage** — Automatic offloading to S3/GCS for cost efficiency +await producer.PublishAsync(orderEnvelope, "orders-topic"); +await producer.PublishAsync(paymentEnvelope, "payments-topic"); +await producer.PublishAsync(shippingEnvelope, "shipping-topic"); ---- +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("orders-topic"), + Arg.Any()); -## Choosing the Right Broker - -``` -Decision Tree: +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("payments-topic"), + Arg.Any()); -Is this an event stream or audit log? - → YES → Use Kafka +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("shipping-topic"), + Arg.Any()); -Is this task delivery (process and acknowledge)? - → YES → - Running locally or in the cloud? - → Local/cloud → Use NATS JetStream (default) - Large-scale on-premises production? - → On-prem → Use Apache Pulsar with Key_Shared +await producer.Received(3).PublishAsync( + Arg.Any>(), + Arg.Any(), + Arg.Any()); ``` ---- - ## Lab -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial05/Lab.cs`](../tests/TutorialLabs/Tutorial05/Lab.cs) - -**Objective:** Design a broker topic hierarchy for a multi-tenant system and analyze how different broker architectures affect **scalability** and **message ordering guarantees**. - -### Step 1: Design a Multi-Region Topic Hierarchy - -Design a NATS subject hierarchy for a multi-region e-commerce system with: orders, payments, and refunds across three regions (US, EU, APAC). Use NATS conventions (`.` for levels, `*` for single-level wildcard, `>` for multi-level wildcard): +Run the full lab: [`tests/TutorialLabs/Tutorial05/Lab.cs`](../tests/TutorialLabs/Tutorial05/Lab.cs) +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial05.Lab" ``` -eip.{region}.{domain}.{event} -Example: eip.us.orders.created -``` - -Write subscriber patterns for: (a) all events in EU: `eip.eu.>`, (b) all order events globally: `eip.*.orders.*`, (c) only payment completions in APAC: `eip.apac.payments.completed`. - -Explain how this hierarchy enables **horizontal scalability** — new regions can be added without changing existing subscribers. - -### Step 2: Compare Broker Scalability Characteristics - -Create a comparison table for Kafka, NATS JetStream, and Pulsar: - -| Characteristic | Kafka | NATS JetStream | Pulsar | -|---------------|-------|----------------|--------| -| Ordering guarantee | Per-partition | Per-subject | Per-key (Key_Shared) | -| HOL blocking risk | ? | ? | ? | -| Multi-tenant isolation | ? | ? | ? | -| Scale-out mechanism | ? | ? | ? | - -For each cell, explain the implication for a platform processing 10,000 messages/second from 50 tenants. - -### Step 3: Design for Atomicity Across Broker Switches - -The platform uses `IMessageBrokerProducer` / `IMessageBrokerConsumer` to abstract the broker. Describe a scenario where switching from NATS to Kafka for a specific message type would change the **atomicity** guarantees (hint: Kafka's transactional producer vs. NATS at-least-once). What compensating design would the platform need? ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial05/Exam.cs`](../tests/TutorialLabs/Tutorial05/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial05/Exam.cs`](../tests/TutorialLabs/Tutorial05/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial05.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/06-messaging-channels.md b/EnterpriseIntegrationPlatform/tutorials/06-messaging-channels.md index 90f7f86..49e4b2a 100644 --- a/EnterpriseIntegrationPlatform/tutorials/06-messaging-channels.md +++ b/EnterpriseIntegrationPlatform/tutorials/06-messaging-channels.md @@ -1,47 +1,10 @@ # Tutorial 06 — Messaging Channels -## What You'll Learn - -- The difference between Point-to-Point and Publish-Subscribe channels -- Datatype Channel for format-specific routing -- Invalid Message Channel for validation failures -- Messaging Bridge for cross-broker communication - ---- - -## What Is a Messaging Channel? - -A **Messaging Channel** is a named conduit through which messages flow. In the EIP book, a channel is the logical pipe connecting a sender to a receiver. In this platform, channels are implemented on top of the broker abstraction, adding semantic behavior. - -``` - Messaging Channels -┌─────────────────────────────────────────────────────┐ -│ │ -│ Point-to-Point Publish-Subscribe │ -│ ┌───────────┐ ┌───────────┐ │ -│ │ 1 msg → │ │ 1 msg → │ │ -│ │ 1 consumer│ │ N consumers│ │ -│ └───────────┘ └───────────┘ │ -│ │ -│ Datatype Invalid Message │ -│ ┌───────────┐ ┌───────────┐ │ -│ │ Route by │ │ Failed │ │ -│ │ content │ │ validation│ │ -│ │ type │ │ → quarantine│ │ -│ └───────────┘ └───────────┘ │ -│ │ -│ Messaging Bridge │ -│ ┌───────────────────────────────────┐ │ -│ │ Source broker ──→ Target broker │ │ -│ └───────────────────────────────────┘ │ -└─────────────────────────────────────────────────────┘ -``` +Point-to-Point, Publish-Subscribe, Datatype, Invalid Message, and Messaging Bridge channel patterns. --- -## Point-to-Point Channel - -The **Point-to-Point Channel** ensures each message is consumed by **exactly one** consumer. This is the classic work queue pattern. +## Key Types ```csharp // src/Ingestion/Channels/IPointToPointChannel.cs @@ -60,24 +23,6 @@ public interface IPointToPointChannel } ``` -### How It Works - -``` -Producer → [ Queue ] → Consumer A (gets msg 1) - → Consumer B (gets msg 2) - → Consumer C (gets msg 3) - -Each message goes to ONE consumer (load balanced) -``` - -**Use when:** You have work items that should be processed exactly once — orders to fulfill, payments to process, files to transform. - ---- - -## Publish-Subscribe Channel - -The **Publish-Subscribe Channel** broadcasts each message to **all** subscribers. Every subscriber gets every message. - ```csharp // src/Ingestion/Channels/IPublishSubscribeChannel.cs public interface IPublishSubscribeChannel @@ -95,33 +40,6 @@ public interface IPublishSubscribeChannel } ``` -### How It Works - -``` -Producer → [ Topic ] → Subscriber A (gets ALL messages) - → Subscriber B (gets ALL messages) - → Subscriber C (gets ALL messages) - -Each message goes to EVERY subscriber (fan-out) -``` - -**Use when:** Multiple independent systems need the same events — audit logging, analytics, notification services. - -### Point-to-Point vs. Publish-Subscribe - -| Feature | Point-to-Point | Publish-Subscribe | -|---------|---------------|-------------------| -| Delivery | One consumer per message | All subscribers per message | -| Use case | Work distribution | Event notification | -| Scaling | Add consumers to share load | Each subscriber independent | -| EIP Name | Point-to-Point Channel | Publish-Subscribe Channel | - ---- - -## Datatype Channel - -The **Datatype Channel** routes messages to a dedicated channel derived from their `MessageType`. Each distinct message type flows on its own channel, ensuring type-safe consumption. Use `ResolveChannel` to look up the channel name for a given message type. - ```csharp // src/Ingestion/Channels/IDatatypeChannel.cs public interface IDatatypeChannel @@ -134,24 +52,6 @@ public interface IDatatypeChannel } ``` -### How It Works - -``` - ┌─→ [OrderCreated channel] (messageType: "OrderCreated") -Message ─→ Publish┤ - ├─→ [InvoiceReceived channel](messageType: "InvoiceReceived") - │ - └─→ [ShipmentDispatched channel](messageType: "ShipmentDispatched") -``` - -**Use when:** You have distinct message types and each type should flow on its own dedicated channel. - ---- - -## Invalid Message Channel - -The **Invalid Message Channel** quarantines messages that fail validation. Instead of silently dropping bad messages, they're routed to a special channel for inspection. - ```csharp // src/Ingestion/Channels/IInvalidMessageChannel.cs public interface IInvalidMessageChannel @@ -169,30 +69,6 @@ public interface IInvalidMessageChannel } ``` -> `RouteInvalidAsync` handles messages that were parsed into an envelope but failed validation. `RouteRawInvalidAsync` handles raw data that could not be deserialized into an envelope at all. - -### How It Works - -``` -Message arrives → Validate - ├─ Valid → Continue processing - └─ Invalid → Invalid Message Channel → Inspect / Fix / Discard -``` - -The `InvalidMessageEnvelope` wraps the original message with error context: -- Original envelope (preserved exactly as received) -- Validation error description -- Timestamp of rejection -- Retry count (if this was a retry attempt) - -**Use when:** You need to capture and diagnose malformed messages without losing them. - ---- - -## Messaging Bridge - -The **Messaging Bridge** connects two different messaging systems. It consumes from one broker/channel and republishes to another. - ```csharp // src/Ingestion/Channels/IMessagingBridge.cs public interface IMessagingBridge : IAsyncDisposable @@ -207,88 +83,142 @@ public interface IMessagingBridge : IAsyncDisposable } ``` -### How It Works +--- -``` -┌──────────┐ ┌──────────────────┐ ┌──────────┐ -│ NATS │ ──→ │ Messaging Bridge │ ──→ │ Kafka │ -│ (tasks) │ │ (consume+pub) │ │ (events) │ -└──────────┘ └──────────────────┘ └──────────┘ -``` +## Exercises -**Use when:** You need to move messages between different broker types — for example, bridging task delivery (NATS) to event streaming (Kafka) for audit. +### 1. Point-to-Point: single consumer receives the message ---- +```csharp +var producer = Substitute.For(); -## Channel Patterns in Practice +var envelope = IntegrationEnvelope.Create( + payload: "order-123", + source: "OrderService", + messageType: "order.created") with +{ + Intent = MessageIntent.Command, +}; -Here's how a typical message flow uses multiple channel types: +await producer.PublishAsync(envelope, "orders.point-to-point"); +await producer.Received(1).PublishAsync( + Arg.Is>(e => e.Payload == "order-123"), + Arg.Is("orders.point-to-point"), + Arg.Any()); ``` -1. External system sends HTTP request to Gateway.Api -2. Gateway publishes to Point-to-Point channel (task delivery) - → One worker picks it up +### 2. Publish-Subscribe: every consumer group gets a copy -3. Worker validates the message - → Invalid? → Invalid Message Channel (quarantine) - → Valid? → Continue +```csharp +var consumer = Substitute.For(); +var producer = Substitute.For(); -4. Worker checks content type via Datatype Channel - → JSON? → JSON processor - → XML? → XML-to-JSON normalizer first +var envelope = IntegrationEnvelope.Create( + "event-data", "EventService", "event.published") with +{ + Intent = MessageIntent.Event, +}; -5. Processed result published to Publish-Subscribe channel - → Subscriber A: Audit system (records the event) - → Subscriber B: Analytics (updates dashboards) - → Subscriber C: Notification service (sends confirmation) -``` +var groups = new[] { "billing-group", "analytics-group", "notifications-group" }; ---- +foreach (var group in groups) +{ + await consumer.SubscribeAsync( + "events.pubsub", group, _ => Task.CompletedTask); +} -## Lab +await producer.PublishAsync(envelope, "events.pubsub"); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial06/Lab.cs`](../tests/TutorialLabs/Tutorial06/Lab.cs) +await consumer.Received(3).SubscribeAsync( + Arg.Is("events.pubsub"), + Arg.Any(), + Arg.Any, Task>>(), + Arg.Any()); -**Objective:** Classify messaging scenarios by channel type and design a channel topology that ensures **atomic delivery** and **scalable fan-out**. +foreach (var group in groups) +{ + await consumer.Received(1).SubscribeAsync( + Arg.Is("events.pubsub"), + Arg.Is(group), + Arg.Any, Task>>(), + Arg.Any()); +} +``` -### Step 1: Map Scenarios to Channel Types +### 3. Datatype Channel: each message type routes to its own topic -For each scenario, identify the correct EIP channel pattern and the platform class that implements it: +```csharp +var producer = Substitute.For(); + +var orderEnvelope = IntegrationEnvelope.Create( + "new-order", "OrderService", "order.created"); +var paymentEnvelope = IntegrationEnvelope.Create( + "payment-received", "PaymentService", "payment.completed"); +var inventoryEnvelope = IntegrationEnvelope.Create( + "stock-updated", "InventoryService", "inventory.adjusted"); + +await producer.PublishAsync(orderEnvelope, "datatype.order.created"); +await producer.PublishAsync(paymentEnvelope, "datatype.payment.completed"); +await producer.PublishAsync(inventoryEnvelope, "datatype.inventory.adjusted"); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("datatype.order.created"), + Arg.Any()); +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("datatype.payment.completed"), + Arg.Any()); +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("datatype.inventory.adjusted"), + Arg.Any()); +``` + +### 4. Invalid Message Channel: expired envelopes -| Scenario | Channel Pattern | Platform Class | -|----------|----------------|----------------| -| Processing purchase orders (one processor per order) | ? | `PointToPointChannel` or `PublishSubscribeChannel`? | -| Notifying 5 systems when a shipment is dispatched | ? | ? | -| Handling messages in XML or JSON from different partners | ? | ? | -| Quarantining messages with missing required fields | ? | ? | +```csharp +var expired = IntegrationEnvelope.Create( + "stale-data", "LegacySystem", "legacy.update") with +{ + ExpiresAt = DateTimeOffset.UtcNow.AddMinutes(-5), +}; -Open `src/Ingestion/Channels/` and verify your answers against the actual implementations. +Assert.That(expired.IsExpired, Is.True); -### Step 2: Design a Messaging Bridge for Broker Migration +var valid = IntegrationEnvelope.Create( + "fresh-data", "ModernSystem", "modern.update") with +{ + ExpiresAt = DateTimeOffset.UtcNow.AddHours(1), +}; -Your company uses Kafka for all integrations but wants to add NATS for new microservices. Using the `MessagingBridge` class in `src/Ingestion/Channels/`, design a bridge configuration that: +Assert.That(valid.IsExpired, Is.False); -- Reads from Kafka topic `legacy.orders.created` -- Publishes to NATS subject `eip.orders.created` -- Preserves the `CorrelationId` and all `Metadata` across the bridge +var noExpiry = IntegrationEnvelope.Create( + "persistent-data", "CoreService", "core.event"); -Draw the message flow and identify where **atomicity** could be lost (hint: what if the bridge crashes after reading from Kafka but before publishing to NATS?). How does the platform's Ack/Nack pattern mitigate this? +Assert.That(noExpiry.ExpiresAt, Is.Null); +Assert.That(noExpiry.IsExpired, Is.False); +``` -### Step 3: Evaluate Scalability of Channel Patterns +--- -Compare Point-to-Point and Publish-Subscribe channels under high load: +## Lab -- Point-to-Point with 3 competing consumers processing 10,000 messages/second -- Pub-Sub with 5 subscriber groups, each with 2 consumers +> 💻 [`tests/TutorialLabs/Tutorial06/Lab.cs`](../tests/TutorialLabs/Tutorial06/Lab.cs) -For each, explain: How does adding more consumers affect throughput? What happens to in-flight messages? Where is the bottleneck? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial06.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial06/Exam.cs`](../tests/TutorialLabs/Tutorial06/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial06/Exam.cs`](../tests/TutorialLabs/Tutorial06/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial06.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/07-temporal-workflows.md b/EnterpriseIntegrationPlatform/tutorials/07-temporal-workflows.md index d46c007..5b55d99 100644 --- a/EnterpriseIntegrationPlatform/tutorials/07-temporal-workflows.md +++ b/EnterpriseIntegrationPlatform/tutorials/07-temporal-workflows.md @@ -1,390 +1,178 @@ # Tutorial 07 — Temporal Workflows -## What You'll Learn - -- What Temporal.io is and why the platform uses it -- How workflows orchestrate message processing -- Durable execution and fault tolerance -- The IntegrationPipelineWorkflow and AtomicPipelineWorkflow -- Saga compensation for distributed transactions +Durable workflow orchestration with `IntegrationPipelineWorkflow`, `AtomicPipelineWorkflow`, and saga compensation. --- -## Why Temporal? - -Traditional message processing is fragile: - -``` -Receive → Validate → Transform → Route → Deliver - ↑ - Server crashes here. - Message is lost. -``` - -**Temporal.io** solves this with **durable execution**. Every workflow step is persisted. If the server crashes, Temporal automatically resumes from the last completed step: +## Key Types +```csharp +// src/Workflow.Temporal/TemporalOptions.cs +public sealed class TemporalOptions +{ + public const string SectionName = "Temporal"; + public string ServerAddress { get; set; } = "localhost:15233"; + public string Namespace { get; set; } = "default"; + public string TaskQueue { get; set; } = "integration-workflows"; +} ``` -Receive → Validate → Transform → [CRASH] → [RESTART] → Route → Deliver - ↑ - Resumes from here -``` - -### Key Temporal Concepts - -| Concept | Description | -|---------|-------------| -| **Workflow** | A durable, long-running function that orchestrates activities | -| **Activity** | A single unit of work (validate, transform, route, deliver) | -| **Worker** | A process that polls for and executes workflows/activities | -| **Task Queue** | Named queue where workflows and activities are dispatched | -| **Signal** | External input to a running workflow (e.g., manual approval) | -| **Query** | Read the current state of a running workflow | - ---- - -## The Integration Pipeline Workflow - -The core workflow orchestrates every message through the processing pipeline: ```csharp -// src/Workflow.Temporal/Workflows/IntegrationPipelineWorkflow.cs (simplified) - +// src/Workflow.Temporal/Workflows/IntegrationPipelineWorkflow.cs [Workflow] public class IntegrationPipelineWorkflow { [WorkflowRun] public async Task RunAsync( - IntegrationPipelineInput input) - { - // Step 1: Persist the message (status: Pending) - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => act.PersistMessageAsync(input), - PipelineActivityOptions); - - // Step 2: Log Received lifecycle event - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => - act.LogStageAsync(input.MessageId, input.MessageType, "Received"), - PipelineActivityOptions); - - // Step 3: Validate the message - var validation = await Workflow.ExecuteActivityAsync( - (IntegrationActivities act) => - act.ValidateMessageAsync(input.MessageType, input.PayloadJson), - ValidationActivityOptions); - - if (!validation.IsValid) - { - // Publish Nack and update status to Failed - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => - act.UpdateDeliveryStatusAsync( - input.MessageId, input.CorrelationId, - input.Timestamp, "Failed"), - PipelineActivityOptions); - - if (input.NotificationsEnabled) - { - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => - act.PublishNackAsync( - input.MessageId, input.CorrelationId, - validation.Reason ?? "Validation failed", input.NackSubject), - PipelineActivityOptions); - } - - return new IntegrationPipelineResult(input.MessageId, false, validation.Reason); - } - - // Step 4: Update status to Delivered - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => - act.UpdateDeliveryStatusAsync( - input.MessageId, input.CorrelationId, - input.Timestamp, "Delivered"), - PipelineActivityOptions); - - // Step 5: Publish Ack - if (input.NotificationsEnabled) - { - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => - act.PublishAckAsync(input.MessageId, input.CorrelationId, input.AckSubject), - PipelineActivityOptions); - } - - return new IntegrationPipelineResult(input.MessageId, true); - } + IntegrationPipelineInput input) { /* Persist → Log → Validate → Ack/Nack */ } } ``` -### What Makes This Durable - -Each `ExecuteActivityAsync` call is recorded by Temporal. If the worker crashes after Step 2 but before Step 3, Temporal will: - -1. Detect the worker is gone -2. Assign the workflow to another worker -3. Replay Steps 1 and 2 (already completed — just fast-forward) -4. Execute Step 3 from where it left off - -**Zero message loss, guaranteed.** - ---- - -## The Atomic Pipeline Workflow - -The `AtomicPipelineWorkflow` adds **saga compensation** — if a step fails after earlier steps have committed side effects, compensation activities undo those effects: - -``` -Step 1: Reserve inventory ✅ (committed) -Step 2: Charge payment ✅ (committed) -Step 3: Send to warehouse ❌ (FAILED) - -Compensation (reverse order): -Step 2 comp: Refund payment ✅ -Step 1 comp: Release inventory ✅ -Publish Nack with compensation details -``` - ```csharp -// src/Workflow.Temporal/Workflows/AtomicPipelineWorkflow.cs (simplified) +// src/Workflow.Temporal/Workflows/AtomicPipelineWorkflow.cs [Workflow] public class AtomicPipelineWorkflow { [WorkflowRun] public async Task RunAsync( - IntegrationPipelineInput input) - { - var completedSteps = new List(); - - // Step 1: Persist message as Pending - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => act.PersistMessageAsync(input), - PipelineActivityOptions); - completedSteps.Add("PersistMessage"); - - // Step 2: Validate message - var validation = await Workflow.ExecuteActivityAsync( - (IntegrationActivities act) => - act.ValidateMessageAsync(input.MessageType, input.PayloadJson), - ValidationActivityOptions); - - if (!validation.IsValid) - { - // Compensate all previously completed steps in reverse order - foreach (var step in Enumerable.Reverse(completedSteps)) - { - await Workflow.ExecuteActivityAsync( - (SagaCompensationActivities act) => - act.CompensateStepAsync(input.CorrelationId, step), - CompensationActivityOptions); - } - - // Save fault and publish Nack - return new AtomicPipelineResult( - input.MessageId, false, validation.Reason); - } - - // Step 3: Update status to Delivered and Publish Ack - await Workflow.ExecuteActivityAsync( - (PipelineActivities act) => - act.UpdateDeliveryStatusAsync( - input.MessageId, input.CorrelationId, - input.Timestamp, "Delivered"), - PipelineActivityOptions); - - return new AtomicPipelineResult(input.MessageId, true); - } + IntegrationPipelineInput input) { /* Persist → Validate → Compensate on failure */ } } ``` ---- - -## Workflow Activities - -Activities are the building blocks that workflows orchestrate. Each activity is a stateless function that performs one task: - ```csharp -// src/Workflow.Temporal/Activities/IntegrationActivities.cs (simplified) -// Handles validation and processing-stage logging - -public sealed class IntegrationActivities +// src/Activities/IMessageValidationService.cs +public interface IMessageValidationService { - [Activity] - public async Task ValidateMessageAsync( - string messageType, string payloadJson) - { - // Validate the message against schema and business rules - return await _validation.ValidateAsync(messageType, payloadJson); - } - - [Activity] - public async Task LogProcessingStageAsync( - Guid messageId, string messageType, string stage) - { - // Record a lifecycle stage for observability - } + Task ValidateAsync(string messageType, string payloadJson); } -// src/Workflow.Temporal/Activities/PipelineActivities.cs (simplified) -// Handles persistence, delivery status, acknowledgments, and faults - -public sealed class PipelineActivities +public record MessageValidationResult(bool IsValid, string? Reason = null) { - [Activity] - public async Task PersistMessageAsync(IntegrationPipelineInput input) - { - // Save message to Cassandra with status: Pending - await _persistence.SaveMessageAsync(input); - } - - [Activity] - public async Task PublishAckAsync( - Guid messageId, Guid correlationId, string topic) - { - // Publish acknowledgment to Ack topic - await _notification.PublishAckAsync(messageId, correlationId, topic); - } - - [Activity] - public async Task PublishNackAsync( - Guid messageId, Guid correlationId, string reason, string topic) - { - // Publish negative acknowledgment to Nack topic - await _notification.PublishNackAsync(messageId, correlationId, reason, topic); - } + public static MessageValidationResult Success { get; } = new(true); + public static MessageValidationResult Failure(string reason) => new(false, reason); } ``` -> **Note:** Activities are split across two classes: `IntegrationActivities` (validation and logging) and `PipelineActivities` (persistence and notifications). A third class, `SagaCompensationActivities`, handles rollback (see Tutorial 47). - -### Activity Design Principles - -1. **Stateless** — Activities don't hold state between executions -2. **Idempotent** — Running twice produces the same result (critical for retries) -3. **Single responsibility** — Each activity does one thing -4. **Testable** — Activities can be unit tested in isolation with mocked dependencies +```csharp +// src/Activities/IMessageLoggingService.cs +public interface IMessageLoggingService +{ + Task LogAsync(Guid messageId, string messageType, string stage); +} +``` --- -## Ack/Nack Notification Loopback +## Exercises -Every workflow ends by publishing either an **Ack** (success) or **Nack** (failure): +### 1. Verify workflow types exist in the Temporal assembly -``` -Workflow completes successfully: - → Publish Ack to "notifications.ack.{messageType}" - → External systems subscribe to confirm delivery - -Workflow fails (after retries): - → Compensate prior steps (if saga) - → Publish Nack to "notifications.nack.{messageType}" - → External systems subscribe to handle failure - → Message routed to DLQ for inspection -``` - -This ensures **closed-loop integration** — the sender always knows the outcome. - ---- +```csharp +var assembly = typeof(TemporalOptions).Assembly; -## How Messages Trigger Workflows +var pipeline = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "IntegrationPipelineWorkflow"); +Assert.That(pipeline, Is.Not.Null); +Assert.That(pipeline!.IsClass, Is.True); -The `Demo.Pipeline` project shows how messages flow from the broker to Temporal: +var atomic = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "AtomicPipelineWorkflow"); +Assert.That(atomic, Is.Not.Null); -``` -1. IntegrationPipelineWorker (BackgroundService) subscribes to broker topic -2. Message arrives → Worker creates IntegrationPipelineInput -3. Worker calls TemporalWorkflowDispatcher to start a workflow -4. Temporal assigns the workflow to a worker on the task queue -5. Workflow executes activities in sequence -6. Activities call domain services (persistence, validation, routing, delivery) -7. Workflow publishes Ack or Nack and completes +var saga = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "SagaCompensationWorkflow"); +Assert.That(saga, Is.Not.Null); ``` ---- +### 2. TemporalOptions defaults and overrides -## Testing Workflows +```csharp +var options = new TemporalOptions(); -Workflow tests use Temporal's local dev server: +Assert.That(options.ServerAddress, Is.EqualTo("localhost:15233")); +Assert.That(options.Namespace, Is.EqualTo("default")); +Assert.That(options.TaskQueue, Is.EqualTo("integration-workflows")); +Assert.That(TemporalOptions.SectionName, Is.EqualTo("Temporal")); -```csharp -[TestFixture] -public class IntegrationPipelineWorkflowTests +options = new TemporalOptions { - [Test] - public async Task RunAsync_ValidMessage_PublishesAck() - { - // Arrange: create a valid envelope and mock services - var input = CreateValidInput(); - - // Act: run the workflow - var result = await RunWorkflow(input); - - // Assert: workflow succeeded, Ack was published - Assert.That(result.IsSuccess, Is.True); - Assert.That(result.AckPublished, Is.True); - } - - [Test] - public async Task RunAsync_InvalidMessage_PublishesNack() - { - // Arrange: create an invalid envelope - var input = CreateInvalidInput(); - - // Act: run the workflow - var result = await RunWorkflow(input); - - // Assert: workflow failed, Nack was published - Assert.That(result.IsSuccess, Is.False); - Assert.That(result.NackPublished, Is.True); - } -} + ServerAddress = "temporal.prod.internal:7233", + Namespace = "production", + TaskQueue = "prod-integration", +}; + +Assert.That(options.ServerAddress, Is.EqualTo("temporal.prod.internal:7233")); +Assert.That(options.Namespace, Is.EqualTo("production")); +Assert.That(options.TaskQueue, Is.EqualTo("prod-integration")); ``` ---- - -## Lab - -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial07/Lab.cs`](../tests/TutorialLabs/Tutorial07/Lab.cs) +### 3. Verify activity classes expose expected methods -**Objective:** Trace how Temporal workflows enforce **atomic processing** with saga compensation, and design a failure recovery strategy for a multi-step integration pipeline. +```csharp +var assembly = typeof(TemporalOptions).Assembly; + +var integrationActivities = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "IntegrationActivities"); +Assert.That(integrationActivities, Is.Not.Null); +Assert.That(integrationActivities!.GetMethod("ValidateMessageAsync"), Is.Not.Null); +Assert.That(integrationActivities.GetMethod("LogProcessingStageAsync"), Is.Not.Null); + +var pipelineActivities = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "PipelineActivities"); +Assert.That(pipelineActivities, Is.Not.Null); + +var methodNames = pipelineActivities! + .GetMethods(BindingFlags.Public | BindingFlags.Instance) + .Select(m => m.Name).ToList(); + +Assert.That(methodNames, Does.Contain("PersistMessageAsync")); +Assert.That(methodNames, Does.Contain("UpdateDeliveryStatusAsync")); +Assert.That(methodNames, Does.Contain("PublishAckAsync")); +Assert.That(methodNames, Does.Contain("PublishNackAsync")); +Assert.That(methodNames, Does.Contain("LogStageAsync")); +``` -### Step 1: Trace a Failure Recovery Path +### 4. Mock workflow activity chain: Validate → Log -A workflow has 4 steps: Validate → Transform → Route → Deliver. Step 3 (Route) fails after Step 2 has already committed its result. Open `src/Workflow.Temporal/` and trace the code path: +```csharp +var validationService = Substitute.For(); +var loggingService = Substitute.For(); -1. What does Temporal do when Step 3 throws an exception? (hint: retry policy) -2. If all retries are exhausted, how does the `AtomicPipelineWorkflow` trigger saga compensation? -3. What does `SagaCompensationActivities.CompensateStepAsync` do for Steps 1 and 2? +var messageId = Guid.NewGuid(); +const string messageType = "order.created"; +const string payloadJson = "{\"orderId\": \"ORD-001\"}"; -Draw the timeline showing: original steps executed, failure point, compensation steps in reverse order. +validationService.ValidateAsync(messageType, payloadJson) + .Returns(MessageValidationResult.Success); +loggingService.LogAsync(messageId, messageType, Arg.Any()) + .Returns(Task.CompletedTask); -### Step 2: Design Compensation for a Business Scenario +var validationResult = await validationService.ValidateAsync(messageType, payloadJson); +Assert.That(validationResult.IsValid, Is.True); -Design saga compensation for an order fulfilment workflow: +await loggingService.LogAsync(messageId, messageType, "Validated"); -| Step | Action | Compensation | -|------|--------|-------------| -| 1 | Create customer record in CRM | ? | -| 2 | Reserve inventory in warehouse | ? | -| 3 | Charge payment via gateway | ? | -| 4 | Send confirmation email | ? | +await validationService.Received(1).ValidateAsync(messageType, payloadJson); +await loggingService.Received(1).LogAsync(messageId, messageType, "Validated"); +``` -For each compensation, identify: Is it idempotent? What happens if the compensation itself fails? How does the `CorrelationId` link the original action to its compensation? +--- -### Step 3: Evaluate Scalability of Workflow Workers +## Lab -Temporal workers poll task queues for workflow and activity tasks. Consider a scenario with 100 concurrent integrations: +> 💻 [`tests/TutorialLabs/Tutorial07/Lab.cs`](../tests/TutorialLabs/Tutorial07/Lab.cs) -- How many workflow workers should you run? What happens when you add more? -- What is the relationship between worker count and **throughput**? -- Why does Temporal's durable execution model prevent duplicate processing even when workers scale horizontally? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial07.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial07/Exam.cs`](../tests/TutorialLabs/Tutorial07/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial07/Exam.cs`](../tests/TutorialLabs/Tutorial07/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial07.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/08-activities-pipeline.md b/EnterpriseIntegrationPlatform/tutorials/08-activities-pipeline.md index 2c77e9d..536f60b 100644 --- a/EnterpriseIntegrationPlatform/tutorials/08-activities-pipeline.md +++ b/EnterpriseIntegrationPlatform/tutorials/08-activities-pipeline.md @@ -1,39 +1,10 @@ # Tutorial 08 — Activities and the Pipeline -## What You'll Learn - -- How activities are the building blocks of integration pipelines -- The activity service interfaces (persistence, validation, notification) -- How the Demo.Pipeline orchestrates end-to-end message flow -- The relationship between activities, workflows, and brokers - ---- - -## What Are Activities? - -In the EIP world, **Pipes and Filters** is a fundamental pattern — messages flow through a series of processing steps, each step doing one thing. In this platform, each "filter" is a Temporal **activity**. - -``` - Pipes and Filters -┌────────┐ ┌──────────┐ ┌───────────┐ ┌─────────┐ ┌─────────┐ -│ Ingest │───▶│ Validate │───▶│ Transform │───▶│ Route │───▶│ Deliver │ -└────────┘ └──────────┘ └───────────┘ └─────────┘ └─────────┘ - Activity Activity Activity Activity Activity -``` - -Each activity: -- Receives an `IntegrationEnvelope` (or input derived from it) -- Performs one operation -- Returns a result -- Is orchestrated by a Temporal workflow +Activity service interfaces, the Pipes-and-Filters pipeline, and end-to-end message orchestration via Temporal. --- -## Activity Service Interfaces - -Activities delegate to **service interfaces** defined in `src/Activities/`. This separation keeps activities thin and services independently testable. - -### IPersistenceActivityService +## Key Types ```csharp // src/Activities/IPersistenceActivityService.cs @@ -44,34 +15,23 @@ public interface IPersistenceActivityService CancellationToken cancellationToken = default); Task UpdateDeliveryStatusAsync( - Guid messageId, - Guid correlationId, - DateTimeOffset recordedAt, - string status, + Guid messageId, Guid correlationId, + DateTimeOffset recordedAt, string status, CancellationToken cancellationToken = default); Task SaveFaultAsync( - Guid messageId, - Guid correlationId, - string messageType, - string faultedBy, - string reason, - int retryCount, + Guid messageId, Guid correlationId, + string messageType, string faultedBy, string reason, int retryCount, CancellationToken cancellationToken = default); } ``` -**Purpose:** Save messages to Cassandra with delivery status tracking. Every message is persisted on entry (status: `Pending`), updated as it progresses (`InFlight`, `Delivered`, `Failed`). - -### IMessageValidationService - ```csharp // src/Activities/IMessageValidationService.cs public interface IMessageValidationService { Task ValidateAsync( - string messageType, - string payloadJson); + string messageType, string payloadJson); } public record MessageValidationResult(bool IsValid, string? Reason = null) @@ -81,267 +41,172 @@ public record MessageValidationResult(bool IsValid, string? Reason = null) } ``` -**Purpose:** Validate message content — schema validation, required fields, business rules. Returns a `MessageValidationResult` indicating success or the reason for failure. - -### INotificationActivityService - ```csharp // src/Activities/INotificationActivityService.cs public interface INotificationActivityService { Task PublishAckAsync( - Guid messageId, - Guid correlationId, - string topic, + Guid messageId, Guid correlationId, string topic, CancellationToken cancellationToken = default); Task PublishNackAsync( - Guid messageId, - Guid correlationId, - string reason, - string topic, + Guid messageId, Guid correlationId, string reason, string topic, CancellationToken cancellationToken = default); } ``` -**Purpose:** Publish Ack/Nack notifications. On success, publish Ack so downstream systems know the message was processed. On failure, publish Nack with a reason so they can react. - -### ICompensationActivityService - ```csharp // src/Activities/ICompensationActivityService.cs public interface ICompensationActivityService { - Task CompensateAsync( - Guid correlationId, - string stepName); + Task CompensateAsync(Guid correlationId, string stepName); } ``` -**Purpose:** Undo the effects of a completed step during saga compensation. Returns `true` on success, `false` on failure. - ---- - -## The Demo Pipeline - -The `src/Demo.Pipeline/` project shows a complete end-to-end integration pipeline: - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Demo.Pipeline │ -│ │ -│ ┌─────────────────────┐ │ -│ │ IntegrationPipeline │ BackgroundService that │ -│ │ Worker │ subscribes to broker topic │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ PipelineOrchestrator│ Coordinates message flow │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ TemporalWorkflow │ Dispatches to Temporal │ -│ │ Dispatcher │ workflow execution │ -│ └──────────┬──────────┘ │ -│ │ │ -│ ▼ │ -│ ┌─────────────────────┐ │ -│ │ Temporal Workflow │ Orchestrates activities: │ -│ │ │ Persist → Validate → Route → │ -│ │ │ Deliver → Ack/Nack │ -│ └─────────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ -``` - -### IntegrationPipelineWorker - -The worker is a .NET `BackgroundService` that continuously listens for messages: - ```csharp -// Simplified from src/Demo.Pipeline/IntegrationPipelineWorker.cs -public class IntegrationPipelineWorker : BackgroundService -{ - protected override async Task ExecuteAsync(CancellationToken ct) - { - await _consumer.SubscribeAsync( - topic: _options.InboundSubject, - consumerGroup: _options.ConsumerGroup, - handler: async envelope => - { - // Dispatch to pipeline orchestrator - await _orchestrator.ProcessAsync(envelope, ct); - }, - cancellationToken: ct); - } -} -``` - -### PipelineOrchestrator - -The orchestrator wraps the message in pipeline input and dispatches to Temporal: - -```csharp -// Simplified from src/Demo.Pipeline/PipelineOrchestrator.cs -public sealed class PipelineOrchestrator : IPipelineOrchestrator -{ - public async Task ProcessAsync( - IntegrationEnvelope envelope, - CancellationToken cancellationToken = default) - { - var input = new IntegrationPipelineInput - { - MessageId = envelope.MessageId, - CorrelationId = envelope.CorrelationId, - // ... map from envelope to workflow input - }; - - await _dispatcher.DispatchAsync(input, cancellationToken); - } -} +// src/Contracts/IntegrationPipelineInput.cs +public sealed record IntegrationPipelineInput( + Guid MessageId, Guid CorrelationId, Guid? CausationId, + DateTimeOffset Timestamp, string Source, string MessageType, + string SchemaVersion, int Priority, string PayloadJson, + string? MetadataJson, string AckSubject, string NackSubject); ``` --- -## End-to-End Message Flow +## Exercises -Here's the complete journey of a message through the platform: +### 1. Verify activity classes exist with expected methods -``` -1. EXTERNAL SYSTEM - └─ Sends HTTP POST to Gateway.Api - -2. GATEWAY.API (Messaging Gateway pattern) - ├─ Validates the request - ├─ Wraps payload in IntegrationEnvelope - └─ Publishes to broker topic "eip.inbound.orders" - -3. BROKER (NATS JetStream / Kafka / Pulsar) - └─ Durably stores the message - -4. INTEGRATION PIPELINE WORKER (Demo.Pipeline) - ├─ Subscribes to "eip.inbound.orders" - ├─ Picks up the message - └─ Dispatches to Temporal - -5. TEMPORAL WORKFLOW - ├─ Activity 1: Persist message (Cassandra, status: Pending) - ├─ Activity 2: Validate message (schema + business rules) - ├─ Activity 3: Update status (InFlight) - ├─ Activity 4: Transform payload (if needed) - ├─ Activity 5: Route to destination - ├─ Activity 6: Deliver via connector (HTTP/SFTP/Email/File) - ├─ Activity 7: Update status (Delivered) - └─ Activity 8: Publish Ack - -6. ACK/NACK NOTIFICATION - └─ Published to "notifications.ack.orders" - -7. OBSERVABILITY - └─ OpenTelemetry traces, logs, and metrics recorded at every step +```csharp +var assembly = typeof(TemporalOptions).Assembly; + +var integrationActivities = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "IntegrationActivities"); +Assert.That(integrationActivities, Is.Not.Null); +Assert.That(integrationActivities!.GetMethod("ValidateMessageAsync"), Is.Not.Null); +Assert.That(integrationActivities.GetMethod("LogProcessingStageAsync"), Is.Not.Null); + +var pipelineActivities = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "PipelineActivities"); +Assert.That(pipelineActivities, Is.Not.Null); + +var methodNames = pipelineActivities! + .GetMethods(BindingFlags.Public | BindingFlags.Instance) + .Select(m => m.Name).ToList(); + +Assert.That(methodNames, Does.Contain("PersistMessageAsync")); +Assert.That(methodNames, Does.Contain("UpdateDeliveryStatusAsync")); +Assert.That(methodNames, Does.Contain("SaveFaultAsync")); +Assert.That(methodNames, Does.Contain("PublishAckAsync")); +Assert.That(methodNames, Does.Contain("PublishNackAsync")); +Assert.That(methodNames, Does.Contain("LogStageAsync")); + +var sagaActivities = assembly.GetTypes() + .FirstOrDefault(t => t.Name == "SagaCompensationActivities"); +Assert.That(sagaActivities, Is.Not.Null); +Assert.That(sagaActivities!.GetMethod("CompensateStepAsync"), Is.Not.Null); ``` ---- +### 2. Pipeline: Create → Validate → Transform → Route -## Pipeline Configuration +```csharp +var validationService = Substitute.For(); +var loggingService = Substitute.For(); +var producer = Substitute.For(); -The pipeline is configured via `PipelineOptions`: +const string messageType = "order.created"; +const string payloadJson = "{\"orderId\": \"ORD-500\"}"; -```csharp -public class PipelineOptions +// Step 1: Create envelope +var envelope = IntegrationEnvelope.Create( + payloadJson, "OrderService", messageType) with { - public string NatsUrl { get; set; } // NATS server URL - public string InboundSubject { get; set; } // Where to listen - public string AckSubject { get; set; } // Ack notification subject - public string NackSubject { get; set; } // Nack notification subject - public string ConsumerGroup { get; set; } // Consumer group name - public string TemporalServerAddress { get; set; } // Temporal gRPC address - public string TemporalNamespace { get; set; } // Temporal namespace - public string TemporalTaskQueue { get; set; } // Temporal task queue - public TimeSpan WorkflowTimeout { get; set; } // Workflow timeout -} + Intent = MessageIntent.Command, +}; +Assert.That(envelope.MessageId, Is.Not.EqualTo(Guid.Empty)); + +// Step 2: Validate +validationService.ValidateAsync(messageType, payloadJson) + .Returns(MessageValidationResult.Success); +var validationResult = await validationService.ValidateAsync(messageType, payloadJson); +Assert.That(validationResult.IsValid, Is.True); + +// Step 3: Transform — enrich metadata +envelope = envelope with +{ + Metadata = new Dictionary(envelope.Metadata) + { + ["region"] = "us-east", + ["validated"] = "true", + }, +}; +Assert.That(envelope.Metadata["region"], Is.EqualTo("us-east")); + +// Step 4: Route — publish to destination +await producer.PublishAsync(envelope, "orders.us-east"); + +await producer.Received(1).PublishAsync( + Arg.Is>( + e => e.Metadata.ContainsKey("region") && e.Metadata["region"] == "us-east"), + Arg.Is("orders.us-east"), + Arg.Any()); ``` ---- - -## Testing Activities - -Activities are tested in isolation with mocked dependencies: +### 3. Chained activities: Persist → Log → Validate → Log ```csharp -[TestFixture] -public class PersistenceActivityTests +var persistenceService = Substitute.For(); +var loggingService = Substitute.For(); +var validationService = Substitute.For(); + +var input = new IntegrationPipelineInput( + MessageId: Guid.NewGuid(), CorrelationId: Guid.NewGuid(), + CausationId: null, Timestamp: DateTimeOffset.UtcNow, + Source: "Lab08", MessageType: "lab.pipeline", + SchemaVersion: "1.0", Priority: 1, + PayloadJson: "{\"item\": \"widget\"}", MetadataJson: null, + AckSubject: "ack.lab08", NackSubject: "nack.lab08"); + +persistenceService.SaveMessageAsync(input, Arg.Any()) + .Returns(Task.CompletedTask); +loggingService.LogAsync(input.MessageId, input.MessageType, Arg.Any()) + .Returns(Task.CompletedTask); +validationService.ValidateAsync(input.MessageType, input.PayloadJson) + .Returns(MessageValidationResult.Success); + +await persistenceService.SaveMessageAsync(input); +await loggingService.LogAsync(input.MessageId, input.MessageType, "Received"); +var result = await validationService.ValidateAsync(input.MessageType, input.PayloadJson); +await loggingService.LogAsync(input.MessageId, input.MessageType, + result.IsValid ? "Validated" : "ValidationFailed"); + +Received.InOrder(() => { - private IPersistenceActivityService _persistence; - - [SetUp] - public void SetUp() - { - _persistence = Substitute.For(); - } - - [Test] - public async Task SaveMessage_StoresWithPendingStatus() - { - var input = CreateTestPipelineInput(); - - await _persistence.SaveMessageAsync(input); - - await _persistence.Received(1).SaveMessageAsync( - input, Arg.Any()); - } -} + persistenceService.SaveMessageAsync(input, Arg.Any()); + loggingService.LogAsync(input.MessageId, input.MessageType, "Received"); + validationService.ValidateAsync(input.MessageType, input.PayloadJson); + loggingService.LogAsync(input.MessageId, input.MessageType, "Validated"); +}); ``` --- ## Lab -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial08/Lab.cs`](../tests/TutorialLabs/Tutorial08/Lab.cs) - -**Objective:** Design an activity pipeline for a real integration scenario, analyze failure modes, and identify where the Pipes and Filters pattern enables **independent scaling** of each stage. - -### Step 1: Design a Pipeline for XML Invoice Processing - -You receive XML invoices via SFTP. Design the complete activity sequence using the platform's activity classes: - -| Step | Activity | Class | Purpose | -|------|----------|-------|---------| -| 1 | Validate | `IntegrationActivities.ValidateMessageAsync` | Schema + payload checks | -| 2 | ? | ? | Sanitize input (XSS, SQL injection) | -| 3 | ? | ? | Transform XML → canonical JSON | -| 4 | ? | ? | Enrich with customer data from CRM | -| 5 | ? | ? | Route to correct downstream system | -| 6 | ? | ? | Deliver via HTTP connector | -| 7 | ? | ? | Persist to Cassandra | -| 8 | ? | ? | Send Ack/Nack notification | - -Open `src/Activities/` and `src/Workflow.Temporal/Activities/` to find the actual activity classes. +> 💻 [`tests/TutorialLabs/Tutorial08/Lab.cs`](../tests/TutorialLabs/Tutorial08/Lab.cs) -### Step 2: Analyze Failure Modes and Atomicity - -For your pipeline above, analyze what happens at each failure point: - -- Step 3 fails with a **transient** error (network timeout) — what retry policy applies? -- Step 3 fails with a **permanent** error (invalid XML schema) — where does the message go? -- Step 6 fails after Step 7 already persisted — what compensation is needed? - -Explain how the Ack/Nack pattern at Step 8 ensures the originating system knows the final outcome, preserving **end-to-end atomicity**. - -### Step 3: Evaluate Per-Stage Scalability - -The Pipes and Filters pattern allows each activity to scale independently. For your pipeline: - -- Which step is likely the bottleneck under high load? (hint: external API calls) -- How would you scale Step 4 (CRM enrichment) without affecting Steps 1-3? -- What is the advantage of Temporal's activity-level retry over retrying the entire pipeline? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial08.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial08/Exam.cs`](../tests/TutorialLabs/Tutorial08/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial08/Exam.cs`](../tests/TutorialLabs/Tutorial08/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial08.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/09-content-based-router.md b/EnterpriseIntegrationPlatform/tutorials/09-content-based-router.md index 8bf31ca..2089436 100644 --- a/EnterpriseIntegrationPlatform/tutorials/09-content-based-router.md +++ b/EnterpriseIntegrationPlatform/tutorials/09-content-based-router.md @@ -1,36 +1,10 @@ # Tutorial 09 — Content-Based Router -## What You'll Learn - -- The EIP Content-Based Router pattern and when to apply it -- How `IContentBasedRouter` evaluates routing rules by priority -- The `RoutingRule` model with Field / Operator / Value / TargetTopic / Priority -- The `RoutingOperator` enum and pre-compiled regex support -- Why a stateless router scales horizontally without coordination +Priority-ordered routing rules with `IContentBasedRouter`, `RoutingRule`, `RoutingOperator`, and `RoutingDecision`. --- -## EIP Pattern: Content-Based Router - -> *"Use a Content-Based Router to route each message to the correct recipient based on message content."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────────┐ - │ Content-Based│ - ──Message──▶ │ Router │──▶ Topic A (rule 1 matched) - │ │──▶ Topic B (rule 2 matched) - │ │──▶ Default (no rule matched) - └──────────────┘ -``` - -The router inspects a field inside the message (header, metadata key, or JSON payload path) and selects the output topic by evaluating rules in priority order. The first rule that matches wins. - ---- - -## Platform Implementation - -### IContentBasedRouter +## Key Types ```csharp // src/Processing.Routing/IContentBasedRouter.cs @@ -42,14 +16,12 @@ public interface IContentBasedRouter } ``` -### RoutingRule - ```csharp // src/Processing.Routing/RoutingRule.cs public sealed record RoutingRule { public required int Priority { get; init; } - public required string FieldName { get; init; } // e.g. "MessageType", "Payload.order.region" + public required string FieldName { get; init; } public required RoutingOperator Operator { get; init; } public required string Value { get; init; } public required string TargetTopic { get; init; } @@ -57,8 +29,6 @@ public sealed record RoutingRule } ``` -### RoutingOperator - ```csharp // src/Processing.Routing/RoutingOperator.cs public enum RoutingOperator @@ -70,73 +40,208 @@ public enum RoutingOperator } ``` -Rules are sorted by ascending `Priority`. Regex patterns are compiled once at startup (`RegexOptions.Compiled`) and cached in a dictionary keyed by rule, avoiding per-message compilation overhead. - -### RoutingDecision - ```csharp +// src/Processing.Routing/RoutingDecision.cs public sealed record RoutingDecision( string TargetTopic, RoutingRule? MatchedRule, bool IsDefault); ``` -When no rule matches and a default topic is configured, `IsDefault = true`. When no rule matches and no default exists, the router throws `InvalidOperationException`. - --- -## Scalability Dimension +## Exercises -The `ContentBasedRouter` is **stateless** — it holds no per-message state between invocations. Routing rules are loaded once from configuration and shared read-only across all requests. This means you can run N replicas behind a competing-consumer group and every replica makes identical routing decisions. Horizontal scaling is limited only by broker throughput, not by router coordination. +### 1. Route by MessageType with Equals operator ---- +```csharp +var producer = Substitute.For(); -## Atomicity Dimension +var options = Options.Create(new RouterOptions +{ + Rules = + [ + new RoutingRule + { + Priority = 1, FieldName = "MessageType", + Operator = RoutingOperator.Equals, + Value = "order.created", TargetTopic = "orders-topic", + Name = "OrderCreated", + }, + new RoutingRule + { + Priority = 2, FieldName = "MessageType", + Operator = RoutingOperator.Equals, + Value = "payment.received", TargetTopic = "payments-topic", + Name = "PaymentReceived", + }, + ], + DefaultTopic = "unmatched-topic", +}); + +var router = new ContentBasedRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "order-data", "OrderService", "order.created"); + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.TargetTopic, Is.EqualTo("orders-topic")); +Assert.That(decision.IsDefault, Is.False); +Assert.That(decision.MatchedRule, Is.Not.Null); +Assert.That(decision.MatchedRule!.Name, Is.EqualTo("OrderCreated")); +``` -The router publishes to the selected topic via the broker producer **before** acknowledging the source message. If the publish fails, the source message is Nacked and the broker redelivers it. If the process crashes after publish but before Ack, the message may be routed twice — downstream consumers must be idempotent (the `IntegrationEnvelope.MessageId` enables deduplication). Combined with Temporal workflow orchestration, the platform guarantees **zero message loss**. +### 2. Route by Metadata field with Contains operator ---- +```csharp +var producer = Substitute.For(); -## Lab +var options = Options.Create(new RouterOptions +{ + Rules = + [ + new RoutingRule + { + Priority = 1, FieldName = "Metadata.region", + Operator = RoutingOperator.Contains, + Value = "europe", TargetTopic = "eu-topic", + Name = "EuropeRegion", + }, + ], + DefaultTopic = "global-topic", +}); + +var router = new ContentBasedRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "eu-data", "RegionalService", "data.regional") with +{ + Metadata = new Dictionary { ["region"] = "western-europe-1" }, +}; + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.TargetTopic, Is.EqualTo("eu-topic")); +Assert.That(decision.IsDefault, Is.False); +Assert.That(decision.MatchedRule!.Name, Is.EqualTo("EuropeRegion")); +``` + +### 3. Route by MessageType with Regex operator -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial09/Lab.cs`](../tests/TutorialLabs/Tutorial09/Lab.cs) +```csharp +var producer = Substitute.For(); -**Objective:** Configure routing rules with priorities, trace how the Content-Based Router dispatches messages, and analyze routing **scalability** under high-throughput conditions. +var options = Options.Create(new RouterOptions +{ + Rules = + [ + new RoutingRule + { + Priority = 1, FieldName = "MessageType", + Operator = RoutingOperator.Regex, + Value = @"^order\..+", TargetTopic = "order-events", + Name = "AllOrderEvents", + }, + ], + DefaultTopic = "other-events", +}); + +var router = new ContentBasedRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "shipped-data", "OrderService", "order.shipped"); + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.TargetTopic, Is.EqualTo("order-events")); +Assert.That(decision.MatchedRule!.Name, Is.EqualTo("AllOrderEvents")); +``` -### Step 1: Configure a Multi-Rule Routing Table +### 4. No rule matches — falls back to default topic -Open `src/Processing.Routing/ContentBasedRouter.cs`. Create a routing configuration for an e-commerce platform: +```csharp +var producer = Substitute.For(); -| Priority | Field | Operator | Value | Output Topic | -|----------|-------|----------|-------|-------------| -| 1 | `Payload.customer.tier` | Equals | `"platinum"` | `priority-processing` | -| 5 | `MessageType` | Equals | `"OrderCreated"` | `orders.standard` | -| 10 | `MessageType` | Regex | `"Return.*"` | `returns.processing` | -| 100 | (default) | — | — | `general.inbox` | +var options = Options.Create(new RouterOptions +{ + Rules = + [ + new RoutingRule + { + Priority = 1, FieldName = "MessageType", + Operator = RoutingOperator.Equals, + Value = "order.created", TargetTopic = "orders-topic", + }, + ], + DefaultTopic = "catch-all-topic", +}); + +var router = new ContentBasedRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "unknown-data", "UnknownService", "unknown.event"); + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.TargetTopic, Is.EqualTo("catch-all-topic")); +Assert.That(decision.IsDefault, Is.True); +Assert.That(decision.MatchedRule, Is.Null); +``` -A message arrives with `MessageType = "OrderCreated"` and `Payload.customer.tier = "platinum"`. Which topic does it route to? Explain how priority ordering ensures deterministic routing. +### 5. RoutingDecision exposes full matched rule details -### Step 2: Trace the Routing Decision Path +```csharp +var producer = Substitute.For(); -Using the `RoutingDecision` record, trace the router's decision path for a message that matches rules at priorities 1 and 5. Open the router implementation and identify: +var options = Options.Create(new RouterOptions +{ + Rules = + [ + new RoutingRule + { + Priority = 10, FieldName = "Source", + Operator = RoutingOperator.Equals, + Value = "CriticalService", TargetTopic = "critical-topic", + Name = "CriticalSource", + }, + ], + DefaultTopic = "default-topic", +}); + +var router = new ContentBasedRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "critical-payload", "CriticalService", "alert.triggered"); + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.MatchedRule, Is.Not.Null); +Assert.That(decision.MatchedRule!.Priority, Is.EqualTo(10)); +Assert.That(decision.MatchedRule.FieldName, Is.EqualTo("Source")); +Assert.That(decision.MatchedRule.Operator, Is.EqualTo(RoutingOperator.Equals)); +Assert.That(decision.MatchedRule.Value, Is.EqualTo("CriticalService")); +Assert.That(decision.MatchedRule.TargetTopic, Is.EqualTo("critical-topic")); +Assert.That(decision.MatchedRule.Name, Is.EqualTo("CriticalSource")); +``` -- How does the router evaluate rules? (sequential scan vs. sorted by priority?) -- Does evaluation stop at the first match, or are all rules evaluated? -- What `RoutingDecision` is returned — does it include the matched rule for auditing? +--- -### Step 3: Design for Routing Scalability +## Lab -Consider a Content-Based Router processing 50,000 messages/second with 200 routing rules: +> 💻 [`tests/TutorialLabs/Tutorial09/Lab.cs`](../tests/TutorialLabs/Tutorial09/Lab.cs) -- What is the computational cost per message? (hint: O(n) for n rules) -- How does pre-compiling regex patterns (`RoutingOperator.Regex`) improve throughput? -- If you need to route to different brokers (Kafka for audit, NATS for real-time), how would the router's output topic abstraction enable this without code changes? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial09.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial09/Exam.cs`](../tests/TutorialLabs/Tutorial09/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial09/Exam.cs`](../tests/TutorialLabs/Tutorial09/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial09.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/10-message-filter.md b/EnterpriseIntegrationPlatform/tutorials/10-message-filter.md index 90e5b50..d4e75df 100644 --- a/EnterpriseIntegrationPlatform/tutorials/10-message-filter.md +++ b/EnterpriseIntegrationPlatform/tutorials/10-message-filter.md @@ -1,35 +1,10 @@ # Tutorial 10 — Message Filter -## What You'll Learn - -- The EIP Message Filter pattern and how it differs from routing -- How `IMessageFilter` evaluates predicates with AND/OR logic -- The `MessageFilterResult` with Passed / DiscardReason / DiscardTopic -- How `RuleCondition` and `RuleLogicOperator` compose complex predicates -- Why discarded messages are never silently dropped — they go to a DLQ +Predicate-based filtering with `IMessageFilter`, `MessageFilterResult`, `MessageFilterOptions`, and `RuleCondition`. --- -## EIP Pattern: Message Filter - -> *"Use a Message Filter to eliminate undesired messages from a channel based on a set of criteria."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌────────────────┐ - │ Message Filter │ - ──Message──▶ │ (predicate) │──▶ Output Topic (passed) - │ │──▶ Discard / DLQ (failed predicate) - └────────────────┘ -``` - -Unlike the Content-Based Router (which selects *which* topic), the Message Filter decides *whether* the message continues at all. Messages that fail the predicate are discarded — but in this platform, "discarded" does not mean lost. - ---- - -## Platform Implementation - -### IMessageFilter +## Key Types ```csharp // src/Processing.Routing/IMessageFilter.cs @@ -41,8 +16,6 @@ public interface IMessageFilter } ``` -### MessageFilterResult - ```csharp // src/Processing.Routing/MessageFilterResult.cs public sealed record MessageFilterResult( @@ -51,11 +24,6 @@ public sealed record MessageFilterResult( string Reason); ``` -`Passed = true` → message published to `OutputTopic`. -`Passed = false` → message routed to the configured `DiscardTopic` (DLQ) or, only if no discard topic is set, silently dropped. - -### MessageFilterOptions - ```csharp // src/Processing.Routing/MessageFilterOptions.cs public sealed class MessageFilterOptions @@ -68,8 +36,6 @@ public sealed class MessageFilterOptions } ``` -### RuleCondition (from RuleEngine) - ```csharp // src/RuleEngine/RuleCondition.cs public sealed record RuleCondition @@ -80,75 +46,199 @@ public sealed record RuleCondition } ``` -Multiple conditions are combined with `RuleLogicOperator.And` (all must match) or `RuleLogicOperator.Or` (at least one must match). - --- -## Scalability Dimension +## Exercises -The filter is **stateless** — each evaluation depends only on the envelope content and the immutable predicate configuration. Any number of replicas can evaluate concurrently with no shared state. Scaling out is as simple as adding consumer instances to the competing-consumer group. +### 1. Accept filter: message passes when predicate matches ---- - -## Atomicity Dimension +```csharp +var producer = Substitute.For(); -The platform enforces **no silent drops** in production deployments. When a `DiscardTopic` is configured, every filtered-out message is published to that topic with the discard reason before the source message is Acked. If the DLQ publish fails, the source message is Nacked and redelivered. This guarantees every message is accounted for — either it reaches the output topic or it reaches the discard topic with a reason. +var options = Options.Create(new MessageFilterOptions +{ + Conditions = + [ + new RuleCondition + { + FieldName = "MessageType", + Operator = RuleConditionOperator.Equals, + Value = "order.created", + }, + ], + Logic = RuleLogicOperator.And, + OutputTopic = "orders-accepted", + DiscardTopic = "orders-rejected", +}); ---- +var filter = new MessageFilter(producer, options, NullLogger.Instance); -## Lab +var envelope = IntegrationEnvelope.Create( + "valid-order", "OrderService", "order.created"); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial10/Lab.cs`](../tests/TutorialLabs/Tutorial10/Lab.cs) +var result = await filter.FilterAsync(envelope); -**Objective:** Configure message filter rules, analyze the no-silent-drop guarantee with `RequireDiscardTopic`, and design a filter topology for **scalable** multi-stage message processing. +Assert.That(result.Passed, Is.True); +Assert.That(result.OutputTopic, Is.EqualTo("orders-accepted")); +Assert.That(result.Reason, Is.EqualTo("Predicate matched")); -### Step 1: Configure a Filter with Discard Routing +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("orders-accepted"), + Arg.Any()); +``` -Write a `MessageFilterOptions` configuration that passes only messages where `MessageType = "OrderCreated"` AND `Payload.total > 100`: +### 2. Reject filter: message discarded to DLQ ```csharp -var options = new MessageFilterOptions +var producer = Substitute.For(); + +var options = Options.Create(new MessageFilterOptions { - Conditions = [ - new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Equals, Value = "OrderCreated" }, - new RuleCondition { FieldName = "Payload.total", Operator = RuleConditionOperator.GreaterThan, Value = "100" } + Conditions = + [ + new RuleCondition + { + FieldName = "MessageType", + Operator = RuleConditionOperator.Equals, + Value = "order.created", + }, ], Logic = RuleLogicOperator.And, - OutputTopic = "high-value-orders", - DiscardTopic = "filtered-out.orders", - RequireDiscardTopic = true -}; + OutputTopic = "orders-accepted", + DiscardTopic = "orders-rejected", +}); + +var filter = new MessageFilter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "unknown-data", "UnknownService", "unknown.event"); + +var result = await filter.FilterAsync(envelope); + +Assert.That(result.Passed, Is.False); +Assert.That(result.OutputTopic, Is.EqualTo("orders-rejected")); +Assert.That(result.Reason, Does.Contain("discard")); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("orders-rejected"), + Arg.Any()); ``` -Explain what happens when `RequireDiscardTopic = true` and no `DiscardTopic` is configured — how does this enforce **zero message loss**? +### 3. No conditions configured — everything passes through -### Step 2: Trace the Filter's Atomicity Guarantee +```csharp +var producer = Substitute.For(); + +var options = Options.Create(new MessageFilterOptions +{ + Conditions = [], + OutputTopic = "pass-through-topic", +}); + +var filter = new MessageFilter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "any-data", "AnyService", "any.event"); + +var result = await filter.FilterAsync(envelope); + +Assert.That(result.Passed, Is.True); +Assert.That(result.OutputTopic, Is.EqualTo("pass-through-topic")); +``` + +### 4. No discard topic — silent drop, no publish + +```csharp +var producer = Substitute.For(); + +var options = Options.Create(new MessageFilterOptions +{ + Conditions = + [ + new RuleCondition + { + FieldName = "MessageType", + Operator = RuleConditionOperator.Equals, + Value = "expected.type", + }, + ], + Logic = RuleLogicOperator.And, + OutputTopic = "output-topic", +}); -Open `src/Processing.Routing/MessageFilter.cs`. Trace the code path for a message that fails all conditions: +var filter = new MessageFilter(producer, options, NullLogger.Instance); -1. The filter evaluates conditions → all fail → `MessageFilterResult.Passed = false` -2. With `DiscardTopic` set → message is published to the discard topic -3. With `DiscardTopic` null and `RequireDiscardTopic = true` → what exception is thrown? +var envelope = IntegrationEnvelope.Create( + "wrong-data", "Service", "wrong.type"); -Draw the decision tree and explain how this guarantees every message is either delivered to `OutputTopic` or explicitly routed to `DiscardTopic` — never silently dropped. +var result = await filter.FilterAsync(envelope); -### Step 3: Design a Multi-Stage Filter Pipeline +Assert.That(result.Passed, Is.False); +Assert.That(result.OutputTopic, Is.Null); +Assert.That(result.Reason, Does.Contain("silently discarded")); -Design a pipeline with three cascading filters for an insurance claims system: +await producer.DidNotReceive().PublishAsync( + Arg.Any>(), + Arg.Any(), + Arg.Any()); +``` + +### 5. Filter result contains correct reason and topic for both outcomes -| Stage | Filter Criteria | Output | Discard | -|-------|----------------|--------|---------| -| 1 | Claim amount > $0 and valid policy number | `claims.validated` | `claims.invalid` | -| 2 | Claim type is "auto" or "home" | `claims.supported` | `claims.unsupported` | -| 3 | Claim amount < $50,000 (auto-approve threshold) | `claims.auto-approve` | `claims.manual-review` | +```csharp +var producer = Substitute.For(); -How does each filter's **discard topic** become a different team's input? How does this design scale — can each filter stage run independently with its own consumer group? +var options = Options.Create(new MessageFilterOptions +{ + Conditions = + [ + new RuleCondition + { + FieldName = "Source", + Operator = RuleConditionOperator.Equals, + Value = "TrustedService", + }, + ], + Logic = RuleLogicOperator.And, + OutputTopic = "trusted-output", + DiscardTopic = "untrusted-dlq", +}); + +var filter = new MessageFilter(producer, options, NullLogger.Instance); + +var trusted = IntegrationEnvelope.Create( + "trusted-data", "TrustedService", "data.event"); +var passResult = await filter.FilterAsync(trusted); +Assert.That(passResult.Passed, Is.True); +Assert.That(passResult.Reason, Is.EqualTo("Predicate matched")); + +var untrusted = IntegrationEnvelope.Create( + "untrusted-data", "UntrustedService", "data.event"); +var failResult = await filter.FilterAsync(untrusted); +Assert.That(failResult.Passed, Is.False); +Assert.That(failResult.OutputTopic, Is.EqualTo("untrusted-dlq")); +Assert.That(failResult.Reason, Does.Contain("discard")); +``` + +--- + +## Lab + +> 💻 [`tests/TutorialLabs/Tutorial10/Lab.cs`](../tests/TutorialLabs/Tutorial10/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial10.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial10/Exam.cs`](../tests/TutorialLabs/Tutorial10/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial10/Exam.cs`](../tests/TutorialLabs/Tutorial10/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial10.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/11-dynamic-router.md b/EnterpriseIntegrationPlatform/tutorials/11-dynamic-router.md index 6793d77..1bf6207 100644 --- a/EnterpriseIntegrationPlatform/tutorials/11-dynamic-router.md +++ b/EnterpriseIntegrationPlatform/tutorials/11-dynamic-router.md @@ -1,39 +1,10 @@ # Tutorial 11 — Dynamic Router -## What You'll Learn - -- The EIP Dynamic Router pattern and how it differs from static routing -- How `IDynamicRouter` resolves destinations from a runtime routing table -- How `IRouterControlChannel` lets downstream participants register/unregister -- The `DynamicRoutingDecision` with fallback handling -- How the routing table is built at runtime, not at deployment time - ---- - -## EIP Pattern: Dynamic Router - -> *"Use a Dynamic Router, a Router that can self-configure based on special configuration messages from participating destinations."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - Participant A ──register("typeA","topic-a")──▶ ┌──────────────────┐ - Participant B ──register("typeB","topic-b")──▶ │ Control Channel │ - Participant C ──unregister("typeC")──────────▶ │ │ - └────────┬─────────┘ - │ updates - ▼ - ┌──────────────────┐ - ──Message──▶ │ Dynamic Router │──▶ topic-a / topic-b / fallback - └──────────────────┘ -``` - -Unlike the Content-Based Router whose rules are fixed at startup, the Dynamic Router's routing table is **learned at runtime** as downstream participants register and unregister through the control channel. +A router whose routing table is built at runtime as downstream participants register and unregister through a control channel. --- -## Platform Implementation - -### IDynamicRouter +## Key Types ```csharp // src/Processing.Routing/IDynamicRouter.cs @@ -43,11 +14,7 @@ public interface IDynamicRouter IntegrationEnvelope envelope, CancellationToken cancellationToken = default); } -``` - -### IRouterControlChannel -```csharp // src/Processing.Routing/IRouterControlChannel.cs public interface IRouterControlChannel { @@ -63,24 +30,15 @@ public interface IRouterControlChannel IReadOnlyDictionary GetRoutingTable(); } -``` - -Downstream services call `RegisterAsync` on startup with their condition key and destination topic. The router looks up the condition field from the envelope, finds the matching entry, and publishes the message. - -### DynamicRoutingDecision -```csharp // src/Processing.Routing/DynamicRoutingDecision.cs public sealed record DynamicRoutingDecision( string Destination, DynamicRouteEntry? MatchedEntry, bool IsFallback, string? ConditionValue); -``` -### DynamicRouteEntry - -```csharp +// src/Processing.Routing/DynamicRouteEntry.cs public sealed record DynamicRouteEntry( string ConditionKey, string Destination, @@ -88,63 +46,143 @@ public sealed record DynamicRouteEntry( DateTimeOffset RegisteredAtUtc); ``` -When no routing table entry matches, the router uses the configured fallback topic (`IsFallback = true`). If no fallback is configured, it throws `InvalidOperationException`. - --- -## Scalability Dimension +## Exercises -The routing table is stored in a **thread-safe concurrent dictionary** shared across requests within one process. Multiple router replicas each maintain their own copy. For cross-replica consistency, control channel registrations should be broadcast (e.g. via a shared broker topic) so every replica converges on the same table. The routing evaluation itself is stateless and lock-free on read. +### 1. Register a route and route a matching message ---- +```csharp +var options = Options.Create(new DynamicRouterOptions +{ + ConditionField = "MessageType", + FallbackTopic = "unmatched-topic", +}); -## Atomicity Dimension +var router = new DynamicRouter(producer, options, NullLogger.Instance); -Routing decisions are deterministic for a given routing-table snapshot. If the process crashes after publish but before Ack, the message is redelivered and the same decision is made again (assuming the table has not changed). Registration events should also be durable — publishing registrations through the broker guarantees that table state can be rebuilt from the event log after a crash. +await router.RegisterAsync("order.created", "orders-topic", "OrderService"); ---- +var envelope = IntegrationEnvelope.Create( + "order-data", "OrderService", "order.created"); -## Lab +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.Destination, Is.EqualTo("orders-topic")); +Assert.That(decision.IsFallback, Is.False); +Assert.That(decision.MatchedEntry, Is.Not.Null); +Assert.That(decision.MatchedEntry!.ParticipantId, Is.EqualTo("OrderService")); +Assert.That(decision.ConditionValue, Is.EqualTo("order.created")); +``` + +### 2. Unmatched message falls back to FallbackTopic + +```csharp +var options = Options.Create(new DynamicRouterOptions +{ + ConditionField = "MessageType", + FallbackTopic = "catch-all-topic", +}); + +var router = new DynamicRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "unknown-data", "UnknownService", "unknown.event"); + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.Destination, Is.EqualTo("catch-all-topic")); +Assert.That(decision.IsFallback, Is.True); +Assert.That(decision.MatchedEntry, Is.Null); +``` + +### 3. Unregister removes route — subsequent message uses fallback + +```csharp +var router = new DynamicRouter(producer, options, NullLogger.Instance); + +await router.RegisterAsync("order.created", "orders-topic"); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial11/Lab.cs`](../tests/TutorialLabs/Tutorial11/Lab.cs) +var removed = await router.UnregisterAsync("order.created"); +Assert.That(removed, Is.True); -**Objective:** Trace how the Dynamic Router updates its routing table at runtime, analyze the EIP pattern's role in **scalable** integration topologies, and design a consistent routing strategy for distributed deployments. +var envelope = IntegrationEnvelope.Create( + "order-data", "OrderService", "order.created"); -### Step 1: Trace a Dynamic Registration Flow +var decision = await router.RouteAsync(envelope); +Assert.That(decision.IsFallback, Is.True); +Assert.That(decision.Destination, Is.EqualTo("fallback-topic")); +``` -Open `src/Processing.Routing/DynamicRouter.cs`. A new participant registers with `conditionKey = "invoices"` and destination `"invoice-processing"`. Then a message arrives with `MessageType = "invoices"`. Trace the code path: +### 4. Case-insensitive routing matches regardless of case -1. How does `RegisterAsync` store the mapping? -2. How does `RouteAsync` look up the destination? -3. What `RoutingDecision` is returned — does it include the matched condition for auditing? +```csharp +var options = Options.Create(new DynamicRouterOptions +{ + ConditionField = "MessageType", + FallbackTopic = "fallback", + CaseInsensitive = true, +}); -Now: Participant unregisters. A new message with the same key arrives. What happens? Where does the message go? +var router = new DynamicRouter(producer, options, NullLogger.Instance); -### Step 2: Design for Multi-Replica Consistency +await router.RegisterAsync("order.created", "orders-topic"); -You have 5 Dynamic Router replicas behind a load balancer. Participant D registers on Replica 1, but Replica 3 doesn't know about it. Design a solution using the platform's broker infrastructure: +var envelope = IntegrationEnvelope.Create( + "data", "Service", "Order.Created"); -- Publish registration events to a `routing.registrations` topic -- Each replica subscribes and updates its local table -- How does this use the **Publish-Subscribe Channel** pattern to keep all replicas consistent? -- What happens to messages during the propagation delay? Is this an **atomicity** concern? +var decision = await router.RouteAsync(envelope); -### Step 3: Compare Dynamic Router Scalability vs. Content-Based Router +Assert.That(decision.Destination, Is.EqualTo("orders-topic")); +Assert.That(decision.IsFallback, Is.False); +``` -| Aspect | Content-Based Router | Dynamic Router | -|--------|---------------------|---------------| -| Rule source | Static configuration | Runtime registrations | -| Adding new routes | ? | ? | -| Scalability model | ? | ? | -| Consistency across replicas | ? | ? | +### 5. Route by metadata field -When would you choose a Dynamic Router over a Content-Based Router in a multi-tenant SaaS platform? +```csharp +var options = Options.Create(new DynamicRouterOptions +{ + ConditionField = "Metadata.region", + FallbackTopic = "global-topic", +}); + +var router = new DynamicRouter(producer, options, NullLogger.Instance); + +await router.RegisterAsync("eu-west", "eu-west-topic", "EUService"); + +var envelope = IntegrationEnvelope.Create( + "eu-data", "RegionalService", "data.sync") with +{ + Metadata = new Dictionary + { + ["region"] = "eu-west", + }, +}; + +var decision = await router.RouteAsync(envelope); + +Assert.That(decision.Destination, Is.EqualTo("eu-west-topic")); +Assert.That(decision.IsFallback, Is.False); +Assert.That(decision.MatchedEntry!.ParticipantId, Is.EqualTo("EUService")); +``` + +--- + +## Lab + +> 💻 [`tests/TutorialLabs/Tutorial11/Lab.cs`](../tests/TutorialLabs/Tutorial11/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial11.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial11/Exam.cs`](../tests/TutorialLabs/Tutorial11/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial11/Exam.cs`](../tests/TutorialLabs/Tutorial11/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial11.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/12-recipient-list.md b/EnterpriseIntegrationPlatform/tutorials/12-recipient-list.md index 3fed366..2def6c8 100644 --- a/EnterpriseIntegrationPlatform/tutorials/12-recipient-list.md +++ b/EnterpriseIntegrationPlatform/tutorials/12-recipient-list.md @@ -1,36 +1,10 @@ # Tutorial 12 — Recipient List -## What You'll Learn - -- The EIP Recipient List pattern and how it enables fan-out messaging -- How `IRecipientList` / `RecipientListRouter` resolves and publishes to ALL destinations -- Rule-based and metadata-based recipient resolution -- How parallel publishing avoids blocking on slow consumers -- The `RecipientListResult` with deduplication reporting - ---- - -## EIP Pattern: Recipient List - -> *"Route a message to a list of dynamically specified recipients."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌─────────────────┐ - │ Recipient List │──▶ Topic A - ──Message──▶ │ (fan-out) │──▶ Topic B - │ │──▶ Topic C - └─────────────────┘ - Same unmodified message sent to every resolved destination. -``` - -Unlike the Content-Based Router (one winner), the Recipient List sends the **same message to every resolved destination**. This is pure fan-out. +Fan-out routing that sends the same message to every resolved destination, with rule-based and metadata-based recipient resolution and deduplication. --- -## Platform Implementation - -### IRecipientList +## Key Types ```csharp // src/Processing.Routing/IRecipientList.cs @@ -40,102 +14,205 @@ public interface IRecipientList IntegrationEnvelope envelope, CancellationToken cancellationToken = default); } -``` -### RecipientListRouter (concrete) - -```csharp -// src/Processing.Routing/RecipientListRouter.cs -public sealed class RecipientListRouter : IRecipientList -{ - // Resolves destinations from: - // 1. Rule-based: ALL matching RecipientListRule rules contribute destinations - // 2. Metadata-based: comma-separated value from envelope metadata key - - public async Task RouteAsync( - IntegrationEnvelope envelope, - CancellationToken cancellationToken = default) { ... } -} -``` - -The router pre-compiles regex patterns at construction time (`RegexOptions.Compiled`) and caches them per rule. Duplicate destinations are removed (case-insensitive) before publishing. - -### RecipientListResult - -```csharp // src/Processing.Routing/RecipientListResult.cs public sealed record RecipientListResult( IReadOnlyList Destinations, int ResolvedCount, int DuplicatesRemoved); -``` - -Publishing is done **concurrently** using `Task.WhenAll` — the router does not wait for Topic A to complete before publishing to Topic B. This prevents one slow consumer from blocking the entire fan-out. - ---- - -## Scalability Dimension -The `RecipientListRouter` is **stateless** — destination resolution depends only on the immutable rule set and the envelope content. Multiple replicas can fan-out independently. The fan-out multiplies downstream load by the number of recipients, so capacity planning must account for the amplification factor. If a message resolves to 5 destinations, each replica generates 5 publishes per inbound message. +// src/Processing.Routing/RecipientListRule.cs (used in RecipientListOptions) +public sealed class RecipientListRule +{ + public string FieldName { get; set; } + public RoutingOperator Operator { get; set; } + public string Value { get; set; } + public List Destinations { get; set; } + public string? Name { get; set; } +} +``` --- -## Atomicity Dimension +## Exercises -Fan-out introduces an **atomicity challenge**: what if 3 of 5 publishes succeed but 2 fail? The platform strategy is: +### 1. Single rule matches — fan-out to multiple destinations -1. Attempt all publishes concurrently. -2. If any publish fails, the entire operation is considered failed and the source message is Nacked. -3. The broker redelivers the message, and the router retries all publishes (downstream consumers must be idempotent on `MessageId`). +```csharp +var options = Options.Create(new RecipientListOptions +{ + Rules = + [ + new RecipientListRule + { + FieldName = "MessageType", + Operator = RoutingOperator.Equals, + Value = "order.created", + Destinations = ["audit-topic", "analytics-topic", "fulfilment-topic"], + Name = "OrderFanOut", + }, + ], +}); + +var router = new RecipientListRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "order-data", "OrderService", "order.created"); + +var result = await router.RouteAsync(envelope); + +Assert.That(result.ResolvedCount, Is.EqualTo(3)); +Assert.That(result.Destinations, Contains.Item("audit-topic")); +Assert.That(result.Destinations, Contains.Item("analytics-topic")); +Assert.That(result.Destinations, Contains.Item("fulfilment-topic")); +``` -This ensures either all recipients get the message or the source is redelivered. +### 2. Duplicate destinations are deduplicated ---- - -## Lab +```csharp +var options = Options.Create(new RecipientListOptions +{ + Rules = + [ + new RecipientListRule + { + FieldName = "MessageType", + Operator = RoutingOperator.Contains, + Value = "order", + Destinations = ["audit-topic", "analytics-topic"], + }, + new RecipientListRule + { + FieldName = "Source", + Operator = RoutingOperator.Equals, + Value = "OrderService", + Destinations = ["audit-topic", "fulfilment-topic"], + }, + ], +}); + +var router = new RecipientListRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "order-data", "OrderService", "order.created"); + +var result = await router.RouteAsync(envelope); + +Assert.That(result.ResolvedCount, Is.EqualTo(3)); +Assert.That(result.DuplicatesRemoved, Is.EqualTo(1)); +Assert.That(result.Destinations, Contains.Item("audit-topic")); +Assert.That(result.Destinations, Contains.Item("analytics-topic")); +Assert.That(result.Destinations, Contains.Item("fulfilment-topic")); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial12/Lab.cs`](../tests/TutorialLabs/Tutorial12/Lab.cs) +### 3. No rule matches — empty result -**Objective:** Analyze how the Recipient List pattern enables **scalable fan-out** to multiple destinations, design duplicate-safe publishing, and measure the performance impact of parallel vs. sequential delivery. +```csharp +var options = Options.Create(new RecipientListOptions +{ + Rules = + [ + new RecipientListRule + { + FieldName = "MessageType", + Operator = RoutingOperator.Equals, + Value = "order.created", + Destinations = ["orders-topic"], + }, + ], +}); + +var router = new RecipientListRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "payment-data", "PaymentService", "payment.received"); + +var result = await router.RouteAsync(envelope); + +Assert.That(result.ResolvedCount, Is.EqualTo(0)); +Assert.That(result.Destinations, Is.Empty); +``` -### Step 1: Trace a Recipient List Resolution +### 4. Metadata-based recipient resolution -A message matches two routing rules that produce destinations `["audit", "billing", "audit"]`. Open `src/Processing.Routing/RecipientListRouter.cs` and trace: +```csharp +var options = Options.Create(new RecipientListOptions +{ + Rules = [], + MetadataRecipientsKey = "recipients", +}); -1. How are duplicate destinations handled? What does `RecipientListResult.DuplicatesRemoved` report? -2. What is the final `ResolvedCount`? -3. How does the router publish to each destination — sequentially or in parallel? +var router = new RecipientListRouter(producer, options, NullLogger.Instance); -### Step 2: Design a Metadata-Driven Recipient List +var envelope = IntegrationEnvelope.Create( + "data", "Service", "event.occurred") with +{ + Metadata = new Dictionary + { + ["recipients"] = "topic-a,topic-b,topic-c", + }, +}; + +var result = await router.RouteAsync(envelope); + +Assert.That(result.ResolvedCount, Is.EqualTo(3)); +Assert.That(result.Destinations, Contains.Item("topic-a")); +Assert.That(result.Destinations, Contains.Item("topic-b")); +Assert.That(result.Destinations, Contains.Item("topic-c")); +``` -Some integration scenarios require the **sender** to specify recipients dynamically via envelope metadata: +### 5. Verify producer receives all publish calls ```csharp -envelope.Metadata["recipients"] = "audit,billing,compliance"; +var options = Options.Create(new RecipientListOptions +{ + Rules = + [ + new RecipientListRule + { + FieldName = "MessageType", + Operator = RoutingOperator.Equals, + Value = "order.created", + Destinations = ["topic-a", "topic-b"], + }, + ], +}); + +var router = new RecipientListRouter(producer, options, NullLogger.Instance); + +var envelope = IntegrationEnvelope.Create( + "data", "OrderService", "order.created"); + +await router.RouteAsync(envelope); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("topic-a"), + Arg.Any()); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + Arg.Is("topic-b"), + Arg.Any()); ``` -Design this approach and compare trade-offs: - -| Approach | Pros | Cons | -|----------|------|------| -| Rule-based (server-side) | Centralized control, auditable | ? | -| Metadata-based (sender-specified) | ? | Sender must know all destinations | - -Which approach provides better **atomicity** guarantees? (hint: what if the sender specifies a non-existent topic?) +--- -### Step 3: Analyze Fan-Out Scalability +## Lab -With 10 recipients and one slow destination (3-second latency): +> 💻 [`tests/TutorialLabs/Tutorial12/Lab.cs`](../tests/TutorialLabs/Tutorial12/Lab.cs) -- How does parallel publishing (platform's default) compare to sequential publishing? -- What is the total latency for parallel vs. sequential? (hint: parallel ≈ max latency, sequential ≈ sum) -- If the slow destination fails, should the message be Ack'd or Nack'd for the other 9 successful deliveries? Design your atomicity strategy. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial12.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial12/Exam.cs`](../tests/TutorialLabs/Tutorial12/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial12/Exam.cs`](../tests/TutorialLabs/Tutorial12/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial12.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/13-routing-slip.md b/EnterpriseIntegrationPlatform/tutorials/13-routing-slip.md index c0b06a4..bfa546e 100644 --- a/EnterpriseIntegrationPlatform/tutorials/13-routing-slip.md +++ b/EnterpriseIntegrationPlatform/tutorials/13-routing-slip.md @@ -1,46 +1,17 @@ # Tutorial 13 — Routing Slip -## What You'll Learn - -- The EIP Routing Slip pattern for per-message dynamic pipelines -- How `RoutingSlip` and `RoutingSlipStep` attach a processing plan to a message -- How `IRoutingSlipRouter` executes the current step and calls `Advance()` -- How `IRoutingSlipStepHandler` dispatches to named step implementations -- The difference between a routing slip and a fixed pipeline - ---- - -## EIP Pattern: Routing Slip - -> *"Attach a Routing Slip to each message, specifying the sequence of processing steps."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - Envelope + RoutingSlip [Validate, Transform, Enrich] - │ - ▼ - ┌──────────┐ ┌───────────┐ ┌─────────┐ - │ Validate │────▶│ Transform │────▶│ Enrich │────▶ Done - └──────────┘ └───────────┘ └─────────┘ - slip.Advance() slip.Advance() slip.IsComplete -``` - -Unlike a fixed pipeline where every message follows the same path, a routing slip lets **each message carry its own processing plan**. Different messages can visit different steps. +Each message carries its own processing itinerary — steps are executed sequentially and the slip advances after each one. --- -## Platform Implementation - -### RoutingSlip & RoutingSlipStep (Contracts) +## Key Types ```csharp // src/Contracts/RoutingSlip.cs public sealed record RoutingSlip(IReadOnlyList Steps) { public const string MetadataKey = "RoutingSlip"; - public bool IsComplete => Steps.Count == 0; - public RoutingSlipStep? CurrentStep => Steps.Count > 0 ? Steps[0] : null; public RoutingSlip Advance() @@ -56,11 +27,7 @@ public sealed record RoutingSlipStep( string StepName, string? DestinationTopic = null, IReadOnlyDictionary? Parameters = null); -``` - -### IRoutingSlipRouter -```csharp // src/Processing.Routing/IRoutingSlipRouter.cs public interface IRoutingSlipRouter { @@ -68,11 +35,7 @@ public interface IRoutingSlipRouter IntegrationEnvelope envelope, CancellationToken cancellationToken = default); } -``` -### IRoutingSlipStepHandler - -```csharp // src/Processing.Routing/IRoutingSlipStepHandler.cs public interface IRoutingSlipStepHandler { @@ -82,11 +45,8 @@ public interface IRoutingSlipStepHandler IReadOnlyDictionary? parameters, CancellationToken cancellationToken = default); } -``` - -### RoutingSlipStepResult -```csharp +// src/Processing.Routing/RoutingSlipStepResult.cs public sealed record RoutingSlipStepResult( string StepName, bool Succeeded, @@ -95,76 +55,182 @@ public sealed record RoutingSlipStepResult( string? ForwardedToTopic); ``` -After execution, the router calls `Advance()` to consume the completed step. If `DestinationTopic` is set on the current step, the message is forwarded to that topic; otherwise the next step runs in-process. - --- -## Scalability Dimension +## Exercises -Each step can be a separate service consuming from its own topic. Step handlers are resolved from DI by `StepName`, so new steps can be deployed independently. The slip travels with the message — no central orchestrator is required. This enables **per-step horizontal scaling**: a "Transform" step with high load can have 10 replicas while a "Validate" step needs only 2. +### 1. Execute a single step successfully ---- +```csharp +var handler = Substitute.For(); +handler.StepName.Returns("Validate"); +handler.HandleAsync( + Arg.Any>(), + Arg.Any?>(), + Arg.Any()) + .Returns(true); + +var router = new RoutingSlipRouter( + [handler], producer, NullLogger.Instance); + +var slip = new RoutingSlip([new RoutingSlipStep("Validate", "output-topic")]); +var envelope = IntegrationEnvelope.Create( + "payload", "Service", "event.type") with +{ + Metadata = new Dictionary + { + [RoutingSlip.MetadataKey] = JsonSerializer.Serialize(slip.Steps), + }, +}; -## Atomicity Dimension +var result = await router.ExecuteCurrentStepAsync(envelope); -The routing slip is stored in the envelope's `Metadata` dictionary as serialised JSON. After each step the advanced slip is written back to metadata before forwarding. If a step fails, the message is Nacked and redelivered with the **original** slip (the step is retried, not skipped). If a step handler returns `false`, the `RoutingSlipStepResult.Succeeded` is `false` and the failure reason is recorded. +Assert.That(result.StepName, Is.EqualTo("Validate")); +Assert.That(result.Succeeded, Is.True); +Assert.That(result.FailureReason, Is.Null); +Assert.That(result.RemainingSlip.IsComplete, Is.True); +Assert.That(result.ForwardedToTopic, Is.EqualTo("output-topic")); +``` ---- +### 2. Multi-step slip — advance through steps -## Lab +```csharp +var slip = new RoutingSlip([ + new RoutingSlipStep("Validate"), + new RoutingSlipStep("Transform", "transform-topic"), +]); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial13/Lab.cs`](../tests/TutorialLabs/Tutorial13/Lab.cs) +var envelope = IntegrationEnvelope.Create( + "payload", "Service", "event.type") with +{ + Metadata = new Dictionary + { + [RoutingSlip.MetadataKey] = JsonSerializer.Serialize(slip.Steps), + }, +}; -**Objective:** Build a Routing Slip, trace failure recovery with partial completion, and compare the Routing Slip pattern's **scalability** against Process Manager workflows. +var result = await router.ExecuteCurrentStepAsync(envelope); -### Step 1: Build a Routing Slip with Parameters +Assert.That(result.StepName, Is.EqualTo("Validate")); +Assert.That(result.Succeeded, Is.True); +Assert.That(result.RemainingSlip.Steps, Has.Count.EqualTo(1)); +Assert.That(result.RemainingSlip.CurrentStep!.StepName, Is.EqualTo("Transform")); +Assert.That(result.ForwardedToTopic, Is.Null); +``` -Write C# code to construct a `RoutingSlip` with three steps: +### 3. Step with parameters passed to handler ```csharp -var slip = new RoutingSlip([ - new RoutingSlipStep("Validate"), - new RoutingSlipStep("Transform", Parameters: new Dictionary +IReadOnlyDictionary? receivedParams = null; + +var handler = Substitute.For(); +handler.StepName.Returns("Enrich"); +handler.HandleAsync( + Arg.Any>(), + Arg.Any?>(), + Arg.Any()) + .Returns(ci => { - ["targetFormat"] = "XML", - ["schemaVersion"] = "2.0" - }), - new RoutingSlipStep("Deliver", Parameters: new Dictionary + receivedParams = ci.ArgAt?>(1); + return true; + }); + +var parameters = new Dictionary +{ + ["lookupUrl"] = "https://api.example.com/enrich", + ["timeout"] = "30", +}; + +var slip = new RoutingSlip([new RoutingSlipStep("Enrich", null, parameters)]); +var envelope = IntegrationEnvelope.Create( + "payload", "Service", "event.type") with +{ + Metadata = new Dictionary { - ["endpoint"] = "https://partner.example.com/api/orders" - }) + [RoutingSlip.MetadataKey] = JsonSerializer.Serialize(slip.Steps), + }, +}; + +var result = await router.ExecuteCurrentStepAsync(envelope); + +Assert.That(result.Succeeded, Is.True); +Assert.That(receivedParams, Is.Not.Null); +Assert.That(receivedParams!["lookupUrl"], Is.EqualTo("https://api.example.com/enrich")); +Assert.That(receivedParams["timeout"], Is.EqualTo("30")); +``` + +### 4. RoutingSlip.Advance() consumes current step + +```csharp +var slip = new RoutingSlip([ + new RoutingSlipStep("Step1"), + new RoutingSlipStep("Step2"), + new RoutingSlipStep("Step3"), ]); + +Assert.That(slip.IsComplete, Is.False); +Assert.That(slip.CurrentStep!.StepName, Is.EqualTo("Step1")); + +var advanced = slip.Advance(); +Assert.That(advanced.CurrentStep!.StepName, Is.EqualTo("Step2")); +Assert.That(advanced.Steps, Has.Count.EqualTo(2)); + +var advanced2 = advanced.Advance(); +Assert.That(advanced2.CurrentStep!.StepName, Is.EqualTo("Step3")); + +var completed = advanced2.Advance(); +Assert.That(completed.IsComplete, Is.True); +Assert.That(completed.CurrentStep, Is.Null); ``` -Open `src/Contracts/RoutingSlip.cs` and verify the record structure. How does each step carry its own parameters? Why is this important for **atomicity** — each step is self-contained with all the data it needs. +### 5. Handler throws exception — step fails gracefully -### Step 2: Trace a Partial-Completion Recovery +```csharp +var handler = Substitute.For(); +handler.StepName.Returns("RiskyStep"); +handler.HandleAsync( + Arg.Any>(), + Arg.Any?>(), + Arg.Any()) + .Returns(_ => throw new InvalidOperationException("Connection timed out")); + +var router = new RoutingSlipRouter( + [handler], producer, NullLogger.Instance); + +var slip = new RoutingSlip([new RoutingSlipStep("RiskyStep", "output-topic")]); +var envelope = IntegrationEnvelope.Create( + "payload", "Service", "event.type") with +{ + Metadata = new Dictionary + { + [RoutingSlip.MetadataKey] = JsonSerializer.Serialize(slip.Steps), + }, +}; -A message has completed Validate and Transform but the worker crashes during Deliver. The message is redelivered with the slip attached: +var result = await router.ExecuteCurrentStepAsync(envelope); -1. What does `RemainingSlip` contain? (hint: only Deliver remains) -2. How does the platform know which steps already completed? -3. Are Validate and Transform re-executed? Why or why not? +Assert.That(result.Succeeded, Is.False); +Assert.That(result.FailureReason, Does.Contain("Connection timed out")); +Assert.That(result.ForwardedToTopic, Is.Null); +``` -Draw the recovery timeline and explain how the Routing Slip pattern achieves **idempotent resume** — crashed messages resume from exactly where they left off. +--- -### Step 3: Compare Routing Slip vs. Temporal Workflow +## Lab -| Aspect | Routing Slip | Temporal Workflow (Process Manager) | -|--------|-------------|-------------------------------------| -| State persistence | In the message itself | In Temporal's event history | -| Dynamic step addition | ? | ? | -| Compensation support | ? | ? | -| Scalability | ? | ? | -| Best for | ? | ? | +> 💻 [`tests/TutorialLabs/Tutorial13/Lab.cs`](../tests/TutorialLabs/Tutorial13/Lab.cs) -When would you choose a Routing Slip over a full Temporal workflow? Consider: simple linear pipelines vs. complex branching logic. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial13.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial13/Exam.cs`](../tests/TutorialLabs/Tutorial13/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial13/Exam.cs`](../tests/TutorialLabs/Tutorial13/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial13.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/14-process-manager.md b/EnterpriseIntegrationPlatform/tutorials/14-process-manager.md index cc92cdd..8bdd41c 100644 --- a/EnterpriseIntegrationPlatform/tutorials/14-process-manager.md +++ b/EnterpriseIntegrationPlatform/tutorials/14-process-manager.md @@ -1,179 +1,204 @@ # Tutorial 14 — Process Manager -## What You'll Learn - -- The EIP Process Manager pattern for long-running stateful orchestration -- How Temporal workflows implement the Process Manager role -- `IntegrationPipelineWorkflow` and `AtomicPipelineWorkflow` as concrete managers -- Saga compensation via `SagaCompensationActivities` -- The difference between a process manager and a routing slip - ---- - -## EIP Pattern: Process Manager - -> *"Use a central processing unit, a Process Manager, to maintain the state of the sequence and determine the next processing step based on intermediate results."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌────────────────────────────────────────────┐ - │ Process Manager │ - │ (maintains state, decides next step) │ - │ │ - │ Step 1 ──▶ Step 2 ──▶ Step 3 │ - │ ✅ ✅ ❌ │ - │ │ - │ Compensation: Step 2⁻¹ ──▶ Step 1⁻¹ │ - └────────────────────────────────────────────┘ -``` - -Unlike a routing slip (decentralised, message-carried state), the Process Manager is a **centralised stateful orchestrator** that decides the next step based on intermediate results and can branch, loop, or compensate. +Centralised stateful orchestration via Temporal workflows — decides the next step based on intermediate results and compensates on failure. --- -## Platform Implementation - -### IntegrationPipelineWorkflow +## Key Types ```csharp -// src/Workflow.Temporal/Workflows/IntegrationPipelineWorkflow.cs (simplified) -[Workflow] -public class IntegrationPipelineWorkflow +// src/Contracts/IntegrationPipelineInput.cs +public sealed record IntegrationPipelineInput( + Guid MessageId, + Guid CorrelationId, + Guid? CausationId, + DateTimeOffset Timestamp, + string Source, + string MessageType, + string SchemaVersion, + int Priority, + string PayloadJson, + string? MetadataJson, + string AckSubject, + string NackSubject) { - [WorkflowRun] - public async Task RunAsync( - IntegrationPipelineInput input) - { - await Workflow.ExecuteActivityAsync( - (PipelineActivities a) => a.PersistMessageAsync(input), opts); + public bool NotificationsEnabled { get; init; } +} - var validation = await Workflow.ExecuteActivityAsync( - (IntegrationActivities a) => - a.ValidateMessageAsync(input.MessageType, input.PayloadJson), opts); +// src/Contracts/IntegrationPipelineResult.cs +public sealed record IntegrationPipelineResult( + Guid MessageId, + bool IsSuccess, + string? FailureReason = null); - if (!validation.IsValid) - return await HandleFailureAsync(input, validation.Reason); +// src/Demo.Pipeline/PipelineOrchestrator.cs +public sealed class PipelineOrchestrator +{ + public Task ProcessAsync(IntegrationEnvelope envelope) { ... } +} - return await HandleSuccessAsync(input); - } +// src/Demo.Pipeline/ITemporalWorkflowDispatcher.cs +public interface ITemporalWorkflowDispatcher +{ + Task DispatchAsync( + IntegrationPipelineInput input, + string workflowId, + CancellationToken cancellationToken = default); } ``` -The workflow **decides** the next step based on intermediate results (`validation.IsValid`). This is the core Process Manager behaviour — it is not a fixed pipeline. +--- + +## Exercises -### AtomicPipelineWorkflow (Saga variant) +### 1. Successful dispatch — workflow completes without error ```csharp -// src/Workflow.Temporal/Workflows/AtomicPipelineWorkflow.cs (simplified) -[Workflow] -public class AtomicPipelineWorkflow +var dispatcher = Substitute.For(); +dispatcher.DispatchAsync( + Arg.Any(), + Arg.Any(), + Arg.Any()) + .Returns(ci => new IntegrationPipelineResult( + ci.ArgAt(0).MessageId, + IsSuccess: true)); + +var options = Options.Create(new PipelineOptions { - [WorkflowRun] - public async Task RunAsync( - IntegrationPipelineInput input) - { - var completedSteps = new List(); - // Execute steps, tracking each ... - // On failure → compensate in reverse order - } + AckSubject = "integration.ack", + NackSubject = "integration.nack", +}); - private async Task HandleNackWithRollbackAsync( - IntegrationPipelineInput input, - List completedSteps, - string failureReason) - { - foreach (var step in Enumerable.Reverse(completedSteps)) - { - await Workflow.ExecuteActivityAsync( - (SagaCompensationActivities a) => - a.CompensateStepAsync(input.CorrelationId, step), opts); - } - // Publish Nack ... - } -} -``` +var orchestrator = new PipelineOrchestrator( + dispatcher, options, NullLogger.Instance); -### SagaCompensationActivities +var json = JsonSerializer.Deserialize( + """{"orderId": "ORD-1", "amount": 100}"""); -```csharp -// src/Workflow.Temporal/Activities/SagaCompensationActivities.cs -public sealed class SagaCompensationActivities -{ - [Activity] - public async Task CompensateStepAsync(Guid correlationId, string stepName) - { - await _logging.LogAsync(correlationId, stepName, $"CompensationStarted:{stepName}"); - var success = await _compensationService.CompensateAsync(correlationId, stepName); - var stage = success ? $"CompensationSucceeded:{stepName}" : $"CompensationFailed:{stepName}"; - await _logging.LogAsync(correlationId, stepName, stage); - return success; - } -} -``` +var envelope = IntegrationEnvelope.Create( + json, "OrderService", "order.created"); ---- +await orchestrator.ProcessAsync(envelope); -## Scalability Dimension +await dispatcher.Received(1).DispatchAsync( + Arg.Any(), + Arg.Any(), + Arg.Any()); +``` -Temporal workers scale horizontally — multiple workers poll the same task queue and Temporal distributes workflow executions across them. Each workflow execution is single-threaded (deterministic replay) but thousands of concurrent workflow instances can run in parallel across the worker fleet. The process manager state lives in Temporal's persistence layer, not in the worker memory. +### 2. Input mapping — envelope fields map to pipeline input ---- +```csharp +IntegrationPipelineInput? capturedInput = null; -## Atomicity Dimension +dispatcher.DispatchAsync( + Arg.Any(), + Arg.Any(), + Arg.Any()) + .Returns(ci => + { + capturedInput = ci.ArgAt(0); + return new IntegrationPipelineResult(capturedInput.MessageId, IsSuccess: true); + }); -The `AtomicPipelineWorkflow` implements full **saga compensation**. Completed steps are tracked in a list. On failure, compensation activities run in reverse order to undo side effects. Even if a worker crashes mid-compensation, Temporal replays the workflow from the last durable checkpoint and continues compensating. The workflow always terminates with either an Ack (success) or a Nack (failure with compensation details), guaranteeing closed-loop notification. +var envelope = IntegrationEnvelope.Create( + json, "TestService", "test.event") with +{ + Priority = MessagePriority.High, + SchemaVersion = "2.0", + Metadata = new Dictionary { ["tenant"] = "acme" }, +}; + +await orchestrator.ProcessAsync(envelope); + +Assert.That(capturedInput!.MessageId, Is.EqualTo(envelope.MessageId)); +Assert.That(capturedInput.CorrelationId, Is.EqualTo(envelope.CorrelationId)); +Assert.That(capturedInput.Source, Is.EqualTo("TestService")); +Assert.That(capturedInput.MessageType, Is.EqualTo("test.event")); +Assert.That(capturedInput.SchemaVersion, Is.EqualTo("2.0")); +Assert.That(capturedInput.Priority, Is.EqualTo((int)MessagePriority.High)); +Assert.That(capturedInput.AckSubject, Is.EqualTo("test.ack")); +Assert.That(capturedInput.NackSubject, Is.EqualTo("test.nack")); +Assert.That(capturedInput.PayloadJson, Does.Contain("value")); +Assert.That(capturedInput.MetadataJson, Does.Contain("acme")); +``` ---- +### 3. Workflow ID derived from MessageId -## Lab +```csharp +string? capturedWorkflowId = null; -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial14/Lab.cs`](../tests/TutorialLabs/Tutorial14/Lab.cs) +dispatcher.DispatchAsync( + Arg.Any(), + Arg.Any(), + Arg.Any()) + .Returns(ci => + { + capturedWorkflowId = ci.ArgAt(1); + var input = ci.ArgAt(0); + return new IntegrationPipelineResult(input.MessageId, IsSuccess: true); + }); -**Objective:** Trace the Process Manager's orchestration of multi-step workflows with saga compensation, and analyze how centralized coordination enables **atomic** all-or-nothing processing. +var envelope = IntegrationEnvelope.Create( + json, "Service", "event.type"); -### Step 1: Trace a Compensation Sequence +await orchestrator.ProcessAsync(envelope); -A workflow has steps: Persist → Validate → Transform → Deliver. Transform succeeds but Deliver fails after all retries. Open `src/Workflow.Temporal/Workflows/AtomicPipelineWorkflow.cs` and trace: +Assert.That(capturedWorkflowId, Is.Not.Null); +Assert.That(capturedWorkflowId, Is.EqualTo($"integration-{envelope.MessageId}")); +``` -1. Which steps need compensation? (only steps that committed work) -2. In what order do compensation steps execute? (hint: reverse) -3. What does `SagaCompensationActivities.CompensateStepAsync` do for each step? +### 4. Failed workflow completes without throwing -List the compensation sequence: +```csharp +dispatcher.DispatchAsync( + Arg.Any(), + Arg.Any(), + Arg.Any()) + .Returns(ci => new IntegrationPipelineResult( + ci.ArgAt(0).MessageId, + IsSuccess: false, + FailureReason: "Validation failed")); + +var envelope = IntegrationEnvelope.Create( + json, "Service", "event.type"); + +Assert.DoesNotThrowAsync(() => orchestrator.ProcessAsync(envelope)); +``` -| Order | Compensating | Original Step | -|-------|-------------|---------------| -| 1 | Undo Transform | Transform | -| 2 | ? | ? | -| 3 | ? | ? | +### 5. IntegrationPipelineResult record shape -### Step 2: Handle Compensation Failures +```csharp +var messageId = Guid.NewGuid(); -The compensation for "Persist" itself fails. Open `src/Workflow.Temporal/Activities/SagaCompensationActivities.cs` and answer: +var success = new IntegrationPipelineResult(messageId, IsSuccess: true); +Assert.That(success.MessageId, Is.EqualTo(messageId)); +Assert.That(success.IsSuccess, Is.True); +Assert.That(success.FailureReason, Is.Null); -- What does the `CompensateStepAsync` method return when compensation fails? -- Does Temporal retry the compensation? With what policy? -- What is logged? How does the operations team know that manual intervention is required? +var failure = new IntegrationPipelineResult( + messageId, IsSuccess: false, FailureReason: "Timeout exceeded"); +Assert.That(failure.IsSuccess, Is.False); +Assert.That(failure.FailureReason, Is.EqualTo("Timeout exceeded")); +``` -Design an alerting strategy for compensation failures — this is the **atomicity boundary** of the system. +--- -### Step 3: Compare Process Manager vs. Routing Slip +## Lab -| Aspect | Process Manager | Routing Slip | -|--------|----------------|-------------| -| Coordination | Centralized (Temporal) | Decentralized (in-message) | -| Compensation | Full saga support | Limited / manual | -| Visibility | Full execution history | ? | -| Scalability bottleneck | Temporal server | ? | -| Best for | Complex branching, compensation | ? | +> 💻 [`tests/TutorialLabs/Tutorial14/Lab.cs`](../tests/TutorialLabs/Tutorial14/Lab.cs) -When would a Process Manager's centralized coordination be worth the **scalability** trade-off vs. a Routing Slip? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial14.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial14/Exam.cs`](../tests/TutorialLabs/Tutorial14/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial14/Exam.cs`](../tests/TutorialLabs/Tutorial14/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial14.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/15-message-translator.md b/EnterpriseIntegrationPlatform/tutorials/15-message-translator.md index 3d3c625..c4e0593 100644 --- a/EnterpriseIntegrationPlatform/tutorials/15-message-translator.md +++ b/EnterpriseIntegrationPlatform/tutorials/15-message-translator.md @@ -1,35 +1,10 @@ # Tutorial 15 — Message Translator -## What You'll Learn - -- The EIP Message Translator pattern for payload format conversion -- How `IMessageTranslator` transforms an entire envelope -- How `IPayloadTransform` encapsulates the actual mapping logic -- Built-in transforms: `FuncPayloadTransform`, `JsonFieldMappingTransform` -- The `FieldMapping` model for declarative JSON field mapping -- `TranslationResult` that carries the translated envelope and target topic +Converts a message payload from one format to another while preserving envelope identity (CorrelationId, CausationId chain). --- -## EIP Pattern: Message Translator - -> *"Use a Message Translator, a special filter, between other filters or applications to translate one data format into another."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ - │ Source App │───▶│ Message Translator │───▶│ Target App │ - │ (Format A) │ │ A ──▶ B │ │ (Format B) │ - └──────────────┘ └────────────────────┘ └──────────────┘ -``` - -When two systems speak different data formats (JSON vs XML, different JSON schemas), the translator converts the payload without changing the messaging infrastructure. - ---- - -## Platform Implementation - -### IMessageTranslator +## Key Types ```csharp // src/Processing.Translator/IMessageTranslator.cs @@ -39,113 +14,149 @@ public interface IMessageTranslator IntegrationEnvelope source, CancellationToken cancellationToken = default); } -``` - -### IPayloadTransform -```csharp // src/Processing.Translator/IPayloadTransform.cs public interface IPayloadTransform { TOut Transform(TIn source); } -``` - -The translator delegates to an `IPayloadTransform` for the actual mapping, then wraps the result in a new `IntegrationEnvelope` preserving `CorrelationId`, `MessageId`, and metadata. -### FuncPayloadTransform - -A convenience implementation that accepts a `Func` delegate — useful for simple inline transformations without creating a full class. - -### JsonFieldMappingTransform +// src/Processing.Translator/FuncPayloadTransform.cs +public sealed class FuncPayloadTransform : IPayloadTransform +{ + public FuncPayloadTransform(Func transform) { ... } + public TOut Transform(TIn source) => _transform(source); +} -Declarative field mapping driven by a list of `FieldMapping` records: +// src/Processing.Translator/TranslationResult.cs +public sealed record TranslationResult( + IntegrationEnvelope TranslatedEnvelope, + Guid SourceMessageId, + string TargetTopic); -```csharp // src/Processing.Translator/FieldMapping.cs public sealed record FieldMapping { - public required string SourcePath { get; init; } // e.g. "order.id" - public required string TargetPath { get; init; } // e.g. "orderId" - public string? StaticValue { get; init; } // inject fixed values + public required string SourcePath { get; init; } + public required string TargetPath { get; init; } + public string? StaticValue { get; init; } } ``` -Source paths use dot notation (`customer.address.city`). The transform reads from the source JSON document at `SourcePath` and writes to the target document at `TargetPath`. When `StaticValue` is set, it is injected regardless of source content. +--- -### TranslationResult +## Exercises + +### 1. Basic translation — string to string ```csharp -// src/Processing.Translator/TranslationResult.cs -public sealed record TranslationResult( - IntegrationEnvelope TranslatedEnvelope, - Guid SourceMessageId, - string TargetTopic); -``` +var transform = new FuncPayloadTransform(s => s.ToUpperInvariant()); ---- +var options = Options.Create(new TranslatorOptions +{ + TargetTopic = "translated-topic", +}); -## Scalability Dimension +var translator = new MessageTranslator( + transform, producer, options, + NullLogger>.Instance); -Translators are **stateless pure functions** — given the same input, they produce the same output. This makes them ideal for horizontal scaling. Deploy as many translator replicas as needed behind a competing-consumer group. CPU-intensive transformations (large XML parsing) benefit directly from additional replicas. +var source = IntegrationEnvelope.Create( + "hello world", "SourceService", "greeting.event"); ---- +var result = await translator.TranslateAsync(source); -## Atomicity Dimension +Assert.That(result.TranslatedEnvelope.Payload, Is.EqualTo("HELLO WORLD")); +Assert.That(result.TargetTopic, Is.EqualTo("translated-topic")); +Assert.That(result.SourceMessageId, Is.EqualTo(source.MessageId)); +``` -The translator publishes the translated envelope to the target topic **before** acknowledging the source. If the publish fails, the source message is Nacked and redelivered. The `SourceMessageId` in `TranslationResult` enables end-to-end tracing from the original message through the translation to the target topic. +### 2. CorrelationId is preserved across translation ---- +```csharp +var transform = new FuncPayloadTransform(s => s); -## Lab +var translator = new MessageTranslator( + transform, producer, options, + NullLogger>.Instance); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial15/Lab.cs`](../tests/TutorialLabs/Tutorial15/Lab.cs) +var source = IntegrationEnvelope.Create( + "data", "Service", "event.type"); + +var result = await translator.TranslateAsync(source); + +Assert.That(result.TranslatedEnvelope.CorrelationId, Is.EqualTo(source.CorrelationId)); +``` -**Objective:** Build field mappings for cross-system data transformation, analyze how the Message Translator pattern preserves message **atomicity** through immutable transformations, and design a multi-format translation strategy. +### 3. CausationId set to source MessageId -### Step 1: Build a Field Mapping Configuration +```csharp +var source = IntegrationEnvelope.Create( + "data", "Service", "event.type"); -Write a `FieldMapping` list that transforms this input: +var result = await translator.TranslateAsync(source); -```json -{ "first_name": "Alice", "last_name": "Smith", "email": "alice@example.com" } +Assert.That(result.TranslatedEnvelope.CausationId, Is.EqualTo(source.MessageId)); +Assert.That(result.TranslatedEnvelope.MessageId, Is.Not.EqualTo(source.MessageId)); ``` -Into this output: +### 4. TargetMessageType override changes MessageType + +```csharp +var options = Options.Create(new TranslatorOptions +{ + TargetTopic = "output-topic", + TargetMessageType = "translated.event", +}); + +var translator = new MessageTranslator( + transform, producer, options, + NullLogger>.Instance); + +var source = IntegrationEnvelope.Create( + "data", "Service", "original.event"); + +var result = await translator.TranslateAsync(source); -```json -{ "fullName": "Alice Smith", "contactEmail": "alice@example.com", "source": "CRM" } +Assert.That(result.TranslatedEnvelope.MessageType, Is.EqualTo("translated.event")); ``` -Identify: which mapping uses `SourcePath`, which uses `StaticValue`, and how would you combine `first_name` + `last_name` into `fullName`? Open `src/Processing.Translator/JsonFieldMappingTransform.cs` to verify the mapping mechanics. +### 5. No TargetTopic configured — throws -### Step 2: Trace Immutability Through Translation +```csharp +var options = Options.Create(new TranslatorOptions +{ + TargetTopic = "", +}); -Open `src/Processing.Translator/MessageTranslator.cs`. When a message is translated: +var translator = new MessageTranslator( + transform, producer, options, + NullLogger>.Instance); -1. Is the original `IntegrationEnvelope` mutated, or is a new envelope created? -2. How does the `CausationId` of the translated message link back to the original? -3. If translation fails (e.g., missing required field), what happens to the original message? +var source = IntegrationEnvelope.Create( + "data", "Service", "event.type"); -Explain why **immutable transformation** is critical for atomicity: if translation fails, the original message is untouched and can be retried or routed to the DLQ. +Assert.ThrowsAsync( + () => translator.TranslateAsync(source)); +``` -### Step 3: Design a Multi-Format Translation Pipeline +--- -A partner sends data in XML, but your downstream systems expect JSON. Another partner sends CSV. Design a translation strategy: +## Lab -| Source Format | Translator Step | Output | -|--------------|----------------|--------| -| XML → JSON | `XmlToJsonStep` | Canonical JSON | -| CSV → JSON | Custom `IPayloadTransform` | Canonical JSON | -| JSON → Canonical | `JsonFieldMappingTransform` | Normalized envelope | +> 💻 [`tests/TutorialLabs/Tutorial15/Lab.cs`](../tests/TutorialLabs/Tutorial15/Lab.cs) -How does the **Canonical Data Model** (Tutorial 17 — Normalizer) relate to the Message Translator? Why is normalizing to a canonical format essential for **scalability** — what happens when you add a 5th source format? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial15.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial15/Exam.cs`](../tests/TutorialLabs/Tutorial15/Exam.cs) +> 💻 [`tests/TutorialLabs/Tutorial15/Exam.cs`](../tests/TutorialLabs/Tutorial15/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial15.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/16-transform-pipeline.md b/EnterpriseIntegrationPlatform/tutorials/16-transform-pipeline.md index e3727c6..be0659a 100644 --- a/EnterpriseIntegrationPlatform/tutorials/16-transform-pipeline.md +++ b/EnterpriseIntegrationPlatform/tutorials/16-transform-pipeline.md @@ -1,37 +1,10 @@ # Tutorial 16 — Transform Pipeline -## What You'll Learn - -- The EIP Pipes and Filters pattern applied to payload transformation -- How `ITransformPipeline` chains `ITransformStep` instances in order -- Built-in steps: `JsonToXmlStep`, `XmlToJsonStep`, `RegexReplaceStep`, `JsonPathFilterStep` -- How `TransformContext` carries payload + content type + metadata through the pipeline -- The `TransformResult` returned after all steps complete +Chain ordered `ITransformStep` instances through an `ITransformPipeline` to transform payloads in sequence. --- -## EIP Pattern: Pipes and Filters (Transformation) - -> *"Use Pipes and Filters to divide a larger processing task into a sequence of smaller, independent processing steps (Filters) that are connected by channels (Pipes)."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌────────────┐ ┌────────────┐ ┌──────────────┐ ┌──────────────┐ - │ JsonToXml │───▶│ RegexReplace│───▶│ XmlToJson │───▶│ JsonPathFilter│ - │ Step │ │ Step │ │ Step │ │ Step │ - └────────────┘ └────────────┘ └──────────────┘ └──────────────┘ - ▲ │ - │ TransformContext flows through │ - └──────────────────────────────────────────────────────────┘ -``` - -Each step receives a `TransformContext`, performs one transformation, and returns an updated context. Steps are composed sequentially — the output of one step is the input to the next. - ---- - -## Platform Implementation - -### ITransformPipeline +## Key Types ```csharp // src/Processing.Transform/ITransformPipeline.cs @@ -44,8 +17,6 @@ public interface ITransformPipeline } ``` -### ITransformStep - ```csharp // src/Processing.Transform/ITransformStep.cs public interface ITransformStep @@ -57,8 +28,6 @@ public interface ITransformStep } ``` -### TransformContext - ```csharp // src/Processing.Transform/TransformContext.cs public sealed class TransformContext @@ -72,19 +41,6 @@ public sealed class TransformContext } ``` -The context is **immutable per step** — each step creates a new context via `WithPayload`, preserving metadata across the pipeline. - -### Built-in Steps - -| Step | Description | -|------|-------------| -| `JsonToXmlStep` | Converts JSON payload to XML, updates `ContentType` to `application/xml` | -| `XmlToJsonStep` | Converts XML payload to JSON, updates `ContentType` to `application/json` | -| `RegexReplaceStep` | Applies a regex find-and-replace on the raw payload string | -| `JsonPathFilterStep` | Filters a JSON payload to keep only specified JSONPath expressions | - -### TransformResult - ```csharp // src/Processing.Transform/TransformResult.cs public sealed record TransformResult( @@ -96,59 +52,124 @@ public sealed record TransformResult( --- -## Scalability Dimension +## Exercises -The pipeline runs entirely **in-process** — all steps execute sequentially within a single service instance. To scale, run multiple replicas of the service; each replica processes different messages through its own pipeline instance. Steps are stateless, so there is no coordination between replicas. For CPU-intensive pipelines (large XML conversions), vertical scaling (more CPU per replica) complements horizontal scaling. +### Exercise 1: Single step transforms payload ---- +```csharp +var step = Substitute.For(); +step.Name.Returns("Upper"); +step.ExecuteAsync(Arg.Any(), Arg.Any()) + .Returns(ci => + { + var ctx = ci.Arg(); + return ctx.WithPayload(ctx.Payload.ToUpperInvariant()); + }); + +var options = Options.Create(new TransformOptions()); +var pipeline = new TransformPipeline( + new[] { step }, options, NullLogger.Instance); + +var result = await pipeline.ExecuteAsync("hello", "text/plain"); + +Assert.That(result.Payload, Is.EqualTo("HELLO")); +Assert.That(result.StepsApplied, Is.EqualTo(1)); +Assert.That(result.ContentType, Is.EqualTo("text/plain")); +``` -## Atomicity Dimension +### Exercise 2: Multiple steps applied in order -The pipeline is **all-or-nothing** within a single invocation. If any step throws, the entire pipeline fails and the calling code can Nack the source message. Partial results are not published. The `StepsApplied` count in `TransformResult` tells you exactly how far the pipeline progressed before completion (or failure during diagnostics). +```csharp +var step1 = Substitute.For(); +step1.Name.Returns("Append-A"); +step1.ExecuteAsync(Arg.Any(), Arg.Any()) + .Returns(ci => ci.Arg().WithPayload(ci.Arg().Payload + "A")); ---- +var step2 = Substitute.For(); +step2.Name.Returns("Append-B"); +step2.ExecuteAsync(Arg.Any(), Arg.Any()) + .Returns(ci => ci.Arg().WithPayload(ci.Arg().Payload + "B")); -## Lab +var options = Options.Create(new TransformOptions()); +var pipeline = new TransformPipeline( + new[] { step1, step2 }, options, NullLogger.Instance); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial16/Lab.cs`](../tests/TutorialLabs/Tutorial16/Lab.cs) +var result = await pipeline.ExecuteAsync("X", "text/plain"); -**Objective:** Design a multi-step transform pipeline, trace how immutable `TransformContext` preserves **atomicity** through each stage, and analyze pipeline **scalability** under failure conditions. +Assert.That(result.Payload, Is.EqualTo("XAB")); +Assert.That(result.StepsApplied, Is.EqualTo(2)); +``` -### Step 1: Design a Transform Pipeline +### Exercise 3: Disabled pipeline returns input unchanged -Design a 3-step pipeline for PCI-compliant order processing: +```csharp +var step = Substitute.For(); -| Step | Transform | Class | Purpose | -|------|-----------|-------|---------| -| 1 | XML → JSON | `XmlToJsonStep` | Convert partner XML to canonical JSON | -| 2 | Redact PII | `RegexReplaceStep` | Mask email addresses with `***@***` | -| 3 | Filter fields | `JsonPathFilterStep` | Keep only `$.order.id` and `$.order.total` | +var options = Options.Create(new TransformOptions { Enabled = false }); +var pipeline = new TransformPipeline( + new[] { step }, options, NullLogger.Instance); -Open `src/Processing.Transform/` and verify each step class exists. Write the `TransformOptions` configuration for this pipeline. +var result = await pipeline.ExecuteAsync("{\"id\":1}", "application/json"); -### Step 2: Trace Failure Recovery with StepsApplied +Assert.That(result.Payload, Is.EqualTo("{\"id\":1}")); +Assert.That(result.StepsApplied, Is.EqualTo(0)); +await step.DidNotReceive() + .ExecuteAsync(Arg.Any(), Arg.Any()); +``` -After step 2 of 4, the pipeline fails (e.g., `JsonPathFilterStep` encounters malformed JSON): +### Exercise 4: Payload exceeding max size throws -1. What is `TransformPipelineResult.StepsApplied`? (answer: 2) -2. Is the original source message modified? (hint: `TransformContext.WithPayload` creates copies) -3. How does the pipeline decide whether to retry vs. route to DLQ? +```csharp +var options = Options.Create(new TransformOptions { MaxPayloadSizeBytes = 10 }); +var pipeline = new TransformPipeline( + Array.Empty(), options, NullLogger.Instance); -Explain why `TransformContext` uses `WithPayload` (immutable updates) instead of mutable setters — what **concurrency** benefit does this provide when multiple messages are being transformed in parallel? +var largePayload = new string('x', 50); -### Step 3: Evaluate Pipeline Scalability +Assert.ThrowsAsync( + () => pipeline.ExecuteAsync(largePayload, "text/plain")); +``` -A pipeline processes 10,000 messages/second. Step 2 (regex redaction) is 5x slower than the other steps: +### Exercise 5: Step failure with StopOnStepFailure = false continues -- Can you scale Step 2 independently? (hint: in Temporal, each step is an activity) -- What happens to pipeline throughput if you add a 4th step? -- How does the Pipes and Filters architecture prevent a slow step from blocking the entire system? +```csharp +var failingStep = Substitute.For(); +failingStep.Name.Returns("Failing"); +failingStep.ExecuteAsync(Arg.Any(), Arg.Any()) + .ThrowsAsync(new InvalidOperationException("step error")); + +var goodStep = Substitute.For(); +goodStep.Name.Returns("Good"); +goodStep.ExecuteAsync(Arg.Any(), Arg.Any()) + .Returns(ci => ci.Arg().WithPayload("done")); + +var options = Options.Create(new TransformOptions { StopOnStepFailure = false }); +var pipeline = new TransformPipeline( + new[] { failingStep, goodStep }, options, NullLogger.Instance); + +var result = await pipeline.ExecuteAsync("input", "text/plain"); + +Assert.That(result.Payload, Is.EqualTo("done")); +Assert.That(result.StepsApplied, Is.EqualTo(1)); +``` + +--- + +## Lab + +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial16/Lab.cs`](../tests/TutorialLabs/Tutorial16/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial16.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial16/Exam.cs`](../tests/TutorialLabs/Tutorial16/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial16.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/17-normalizer.md b/EnterpriseIntegrationPlatform/tutorials/17-normalizer.md index 9be2207..56098d5 100644 --- a/EnterpriseIntegrationPlatform/tutorials/17-normalizer.md +++ b/EnterpriseIntegrationPlatform/tutorials/17-normalizer.md @@ -1,33 +1,10 @@ # Tutorial 17 — Normalizer -## What You'll Learn - -- The EIP Normalizer pattern for converting diverse formats to a canonical model -- How `INormalizer` / `MessageNormalizer` auto-detects JSON, XML, and CSV -- The `NormalizationResult` with `DetectedFormat` and `WasTransformed` -- `NormalizerOptions` for CSV delimiter, header mode, and strict content-type handling -- Why a canonical model simplifies downstream processing - ---- - -## EIP Pattern: Normalizer - -> *"Use a Normalizer to route each message type through a custom Message Translator so that the resulting messages match a common format."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - JSON ──────▶ ┌──────────────┐ - XML ──────▶ │ Normalizer │──▶ Canonical JSON - CSV ──────▶ └──────────────┘ -``` - -External systems send data in many formats. The Normalizer detects the incoming format and converts it to the platform's canonical representation (JSON), so every downstream component only needs to understand one format. +Auto-detect JSON, XML, and CSV payloads and convert them to canonical JSON using `INormalizer`. --- -## Platform Implementation - -### INormalizer +## Key Types ```csharp // src/Processing.Transform/INormalizer.cs @@ -40,16 +17,6 @@ public interface INormalizer } ``` -### MessageNormalizer (concrete) - -The `MessageNormalizer` class implements `INormalizer`. It: -1. Inspects the `contentType` parameter (or payload content if `StrictContentType = false`) -2. Detects whether the payload is JSON, XML, or CSV -3. Applies the appropriate conversion to produce canonical JSON -4. Returns the result with the detected format - -### NormalizationResult - ```csharp // src/Processing.Transform/NormalizationResult.cs public sealed record NormalizationResult( @@ -59,10 +26,6 @@ public sealed record NormalizationResult( bool WasTransformed); ``` -When the payload is already JSON, `WasTransformed = false` and the payload passes through unchanged. - -### NormalizerOptions - ```csharp // src/Processing.Transform/NormalizerOptions.cs public sealed class NormalizerOptions @@ -74,74 +37,112 @@ public sealed class NormalizerOptions } ``` -| Option | Purpose | -|--------|---------| -| `StrictContentType` | When `true`, unknown content types throw. When `false`, the normalizer sniffs the payload. | -| `CsvDelimiter` | Delimiter character for CSV parsing (default `,`). | -| `CsvHasHeaders` | When `true`, the first CSV row becomes JSON property names. | -| `XmlRootName` | Root element name used when converting non-XML formats to XML (not used for XML→JSON). | - --- -## Scalability Dimension +## Exercises -The normalizer is **stateless and CPU-bound** — each invocation depends only on the payload and options. Horizontal scaling is straightforward: add more consumer replicas. XML parsing and CSV parsing are the most CPU-intensive paths; JSON pass-through is near zero-cost. Profiling with `Processing.Profiling` can identify which format conversions dominate and guide replica sizing. +### Exercise 1: JSON payload passes through unchanged ---- +```csharp +var options = Options.Create(new NormalizerOptions()); +var normalizer = new MessageNormalizer(options, NullLogger.Instance); -## Atomicity Dimension +var json = """{"name":"Alice","age":30}"""; -Normalization happens **before** any downstream processing. If normalization fails (e.g. malformed XML), the message is Nacked and can be routed to the DLQ with `DeadLetterReason.ValidationFailed`. The `OriginalContentType` and `DetectedFormat` fields in the result provide full traceability of what the normalizer received and what it detected — essential for diagnosing format mismatches in production. +var result = await normalizer.NormalizeAsync(json, "application/json"); ---- +Assert.That(result.DetectedFormat, Is.EqualTo("JSON")); +Assert.That(result.WasTransformed, Is.False); +Assert.That(result.OriginalContentType, Is.EqualTo("application/json")); -## Lab +using var doc = JsonDocument.Parse(result.Payload); +Assert.That(doc.RootElement.GetProperty("name").GetString(), Is.EqualTo("Alice")); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial17/Lab.cs`](../tests/TutorialLabs/Tutorial17/Lab.cs) +### Exercise 2: XML payload converts to JSON + +```csharp +var options = Options.Create(new NormalizerOptions()); +var normalizer = new MessageNormalizer(options, NullLogger.Instance); -**Objective:** Configure the Normalizer for multi-format input handling, analyze how the Canonical Data Model pattern enables **scalable** integration with diverse source systems, and design normalization strategies for edge cases. +var xml = "ORD-199.50"; -### Step 1: Configure a CSV Normalizer +var result = await normalizer.NormalizeAsync(xml, "application/xml"); -A partner sends CSV files with `|` as delimiter and no header row. Open `src/Processing.Transform/` and configure `NormalizerOptions`: +Assert.That(result.DetectedFormat, Is.EqualTo("XML")); +Assert.That(result.WasTransformed, Is.True); -```csharp -var options = new NormalizerOptions -{ - CsvDelimiter = '|', - CsvHasHeaders = false, - StrictContentType = true -}; +using var doc = JsonDocument.Parse(result.Payload); +Assert.That(doc.RootElement.GetProperty("id").GetString(), Is.EqualTo("ORD-1")); +Assert.That(doc.RootElement.GetProperty("total").GetString(), Is.EqualTo("99.50")); ``` -Trace what happens when: (a) a valid CSV arrives, (b) JSON arrives with `contentType = "text/csv"` but `StrictContentType = true`. +### Exercise 3: CSV payload converts to JSON array + +```csharp +var options = Options.Create(new NormalizerOptions()); +var normalizer = new MessageNormalizer(options, NullLogger.Instance); + +var csv = "name,age\nAlice,30\nBob,25"; -### Step 2: Map the Canonical Data Model +var result = await normalizer.NormalizeAsync(csv, "text/csv"); -The platform normalizes all inputs to JSON. Draw a diagram showing 4 source systems and how they funnel through the Normalizer: +Assert.That(result.DetectedFormat, Is.EqualTo("CSV")); +Assert.That(result.WasTransformed, Is.True); +using var doc = JsonDocument.Parse(result.Payload); +var array = doc.RootElement.GetProperty("Root"); +Assert.That(array.GetArrayLength(), Is.EqualTo(2)); +Assert.That(array[0].GetProperty("name").GetString(), Is.EqualTo("Alice")); +Assert.That(array[1].GetProperty("name").GetString(), Is.EqualTo("Bob")); ``` -Partner A (XML) ─────┐ -Partner B (CSV) ─────┤ -Partner C (JSON) ────┼──→ Normalizer ──→ Canonical JSON ──→ Router ──→ N consumers -Internal API (JSON) ─┘ + +### Exercise 4: Unknown content type in strict mode throws + +```csharp +var options = Options.Create(new NormalizerOptions { StrictContentType = true }); +var normalizer = new MessageNormalizer(options, NullLogger.Instance); + +Assert.ThrowsAsync( + () => normalizer.NormalizeAsync("{}", "application/octet-stream")); ``` -How many translators are needed for 4 sources and 6 consumers? With a canonical model: **4** (one per source). Without: **24** (4×6). This is the **scalability** argument for normalization. +### Exercise 5: Custom CSV delimiter parses correctly + +```csharp +var options = Options.Create(new NormalizerOptions { CsvDelimiter = ';' }); +var normalizer = new MessageNormalizer(options, NullLogger.Instance); -### Step 3: Handle Format Detection Failures +var csv = "name;age\nAlice;30"; -A payload arrives with `contentType = "application/json"` but contains invalid JSON. Analyze: +var result = await normalizer.NormalizeAsync(csv, "text/csv"); -- What happens when `StrictContentType = true`? (exception → DLQ) -- What happens when `StrictContentType = false`? (format sniffing attempt) -- Why is strict mode recommended for production **atomicity** — what risks does lenient mode introduce? +Assert.That(result.DetectedFormat, Is.EqualTo("CSV")); +Assert.That(result.WasTransformed, Is.True); + +using var doc = JsonDocument.Parse(result.Payload); +var array = doc.RootElement.GetProperty("Root"); +Assert.That(array[0].GetProperty("name").GetString(), Is.EqualTo("Alice")); +Assert.That(array[0].GetProperty("age").GetString(), Is.EqualTo("30")); +``` + +--- + +## Lab + +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial17/Lab.cs`](../tests/TutorialLabs/Tutorial17/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial17.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial17/Exam.cs`](../tests/TutorialLabs/Tutorial17/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial17.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/18-content-enricher.md b/EnterpriseIntegrationPlatform/tutorials/18-content-enricher.md index cb74e57..20708f7 100644 --- a/EnterpriseIntegrationPlatform/tutorials/18-content-enricher.md +++ b/EnterpriseIntegrationPlatform/tutorials/18-content-enricher.md @@ -1,41 +1,10 @@ # Tutorial 18 — Content Enricher -## What You'll Learn - -- The EIP Content Enricher pattern for augmenting messages with external data -- How `IContentEnricher` / `ContentEnricher` merges external data into the payload -- Enrichment sources: HTTP lookups, database queries, cache -- The merge strategy that preserves existing fields -- How correlation IDs enable tracing through enrichment - ---- - -## EIP Pattern: Content Enricher - -> *"Use a Content Enricher to access an external data source in order to augment a message with missing information."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────┐ ┌─────────────────┐ ┌──────────┐ - │ Incoming │───▶│ Content Enricher│───▶│ Enriched │ - │ Message │ │ │ │ Message │ - └──────────┘ └───────┬─────────┘ └──────────┘ - │ - ▼ - ┌───────────────┐ - │ External Data │ - │ (HTTP / DB / │ - │ Cache) │ - └───────────────┘ -``` - -The enricher takes an incomplete message and supplements it with data fetched from an external source. The original payload fields are **never overwritten** — external data is merged in alongside them. +Augment messages with external data via `IContentEnricher`, merging fetched fields without overwriting existing payload. --- -## Platform Implementation - -### IContentEnricher +## Key Types ```csharp // src/Processing.Transform/IContentEnricher.cs @@ -48,88 +17,186 @@ public interface IContentEnricher } ``` -### ContentEnricher (concrete) +```csharp +// src/Processing.Transform/ContentEnricherOptions.cs +public sealed class ContentEnricherOptions +{ + public string EndpointUrlTemplate { get; init; } + public string LookupKeyPath { get; init; } + public string MergeTargetPath { get; init; } + public bool FallbackOnFailure { get; init; } + public string? FallbackValue { get; init; } +} +``` -The `ContentEnricher` class: -1. Parses the incoming JSON payload -2. Fetches supplementary data from the configured external source -3. Merges the external data into the JSON document (additive — existing fields preserved) -4. Returns the enriched JSON string +```csharp +// src/Processing.Transform/IEnrichmentSource.cs +public interface IEnrichmentSource +{ + Task FetchAsync(string key, CancellationToken cancellationToken = default); +} +``` -The `correlationId` parameter enables distributed tracing — the enricher passes it to external HTTP calls as a request header, so the entire enrichment chain is traceable. +--- + +## Exercises -### Merge Strategy +### Exercise 1: Merge external data at target path -The enricher performs a **shallow merge** by default: -- New properties from the external source are added to the root object -- Existing properties in the original payload are **not overwritten** -- Nested objects can be merged at configurable depth +```csharp +var source = Substitute.For(); +source.FetchAsync("CUST-1", Arg.Any()) + .Returns(JsonNode.Parse("""{"name":"Alice","tier":"Gold"}""")); +var options = Options.Create(new ContentEnricherOptions +{ + EndpointUrlTemplate = "https://api.example.com/customers/{key}", + LookupKeyPath = "customerId", + MergeTargetPath = "customer", +}); + +var enricher = new ContentEnricher( + source, options, NullLogger.Instance); + +var payload = """{"orderId":"ORD-1","customerId":"CUST-1","total":100}"""; + +var result = await enricher.EnrichAsync(payload, Guid.NewGuid()); + +using var doc = JsonDocument.Parse(result); +Assert.That(doc.RootElement.GetProperty("orderId").GetString(), Is.EqualTo("ORD-1")); +Assert.That( + doc.RootElement.GetProperty("customer").GetProperty("name").GetString(), + Is.EqualTo("Alice")); +Assert.That( + doc.RootElement.GetProperty("customer").GetProperty("tier").GetString(), + Is.EqualTo("Gold")); ``` -Original: { "orderId": 123, "customer": "Alice" } -External: { "customerTier": "Gold", "region": "EU" } -Enriched: { "orderId": 123, "customer": "Alice", "customerTier": "Gold", "region": "EU" } + +### Exercise 2: Nested lookup key path extracts correct value + +```csharp +var source = Substitute.For(); +source.FetchAsync("ADDR-7", Arg.Any()) + .Returns(JsonNode.Parse("""{"city":"Seattle","zip":"98101"}""")); + +var options = Options.Create(new ContentEnricherOptions +{ + EndpointUrlTemplate = "https://api.example.com/addresses/{key}", + LookupKeyPath = "order.addressId", + MergeTargetPath = "shippingAddress", +}); + +var enricher = new ContentEnricher( + source, options, NullLogger.Instance); + +var payload = """{"order":{"id":"ORD-2","addressId":"ADDR-7"}}"""; + +var result = await enricher.EnrichAsync(payload, Guid.NewGuid()); + +using var doc = JsonDocument.Parse(result); +Assert.That( + doc.RootElement.GetProperty("shippingAddress").GetProperty("city").GetString(), + Is.EqualTo("Seattle")); ``` ---- +### Exercise 3: Missing lookup key with fallback returns original -## Scalability Dimension +```csharp +var source = Substitute.For(); -The enricher's scalability depends on the **external data source**. The enricher itself is stateless and can be replicated freely, but each replica makes external calls (HTTP, DB). Scaling out enricher replicas increases load on the external service. Mitigate with: -- **Caching**: Cache frequent lookups (e.g. customer tier rarely changes) -- **Bulkheading**: Limit concurrent external calls per replica -- **Circuit breaking**: Fail fast when the external source is down +var options = Options.Create(new ContentEnricherOptions +{ + EndpointUrlTemplate = "https://api.example.com/{key}", + LookupKeyPath = "nonExistentField", + MergeTargetPath = "extra", + FallbackOnFailure = true, +}); ---- +var enricher = new ContentEnricher( + source, options, NullLogger.Instance); -## Atomicity Dimension +var payload = """{"id":"X"}"""; -Enrichment is **not idempotent by default** if the external data changes between retries. However, because the enricher only *adds* data and never *removes* existing fields, a retry that fetches slightly different external data produces a superset of the original enrichment. The enriched message is published before the source is Acked. If the external call fails, the enricher throws, the source message is Nacked, and the retry policy handles redelivery. +var result = await enricher.EnrichAsync(payload, Guid.NewGuid()); ---- +using var doc = JsonDocument.Parse(result); +Assert.That(doc.RootElement.GetProperty("id").GetString(), Is.EqualTo("X")); +await source.DidNotReceive().FetchAsync(Arg.Any(), Arg.Any()); +``` -## Lab +### Exercise 4: Source returns null — fallback value merged -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial18/Lab.cs`](../tests/TutorialLabs/Tutorial18/Lab.cs) +```csharp +var source = Substitute.For(); +source.FetchAsync("KEY-1", Arg.Any()) + .Returns((JsonNode?)null); -**Objective:** Design enrichment strategies using external data sources, analyze **atomicity** when enrichment depends on external service availability, and evaluate caching for **scalable** enrichment. +var options = Options.Create(new ContentEnricherOptions +{ + EndpointUrlTemplate = "https://api.example.com/{key}", + LookupKeyPath = "key", + MergeTargetPath = "extra", + FallbackOnFailure = true, + FallbackValue = """{"status":"unknown"}""", +}); -### Step 1: Design a Two-Step Enrichment +var enricher = new ContentEnricher( + source, options, NullLogger.Instance); -An order message `{ "orderId": 42 }` needs customer data, but only contains `orderId` — not `customerId`. Design the enrichment flow: +var payload = """{"key":"KEY-1"}"""; -1. Step 1: Look up `customerId` from `GET /api/orders/42` → returns `{ "customerId": "CUST-7" }` -2. Step 2: Enrich with customer data from `GET /api/customers/CUST-7` → returns `{ "name": "Alice", "tier": "gold" }` +var result = await enricher.EnrichAsync(payload, Guid.NewGuid()); -Open `src/Processing.Transform/ContentEnricher.cs` and identify how the enricher merges external data into the envelope. Does it mutate the original or create a new enriched envelope? +using var doc = JsonDocument.Parse(result); +Assert.That( + doc.RootElement.GetProperty("extra").GetProperty("status").GetString(), + Is.EqualTo("unknown")); +``` -### Step 2: Analyze Enrichment Failure Atomicity +### Exercise 5: Enrichment preserves all existing fields -The external HTTP service is down during enrichment. Trace what happens: +```csharp +var source = Substitute.For(); +source.FetchAsync("C-1", Arg.Any()) + .Returns(JsonNode.Parse("""{"loyalty":true}""")); -1. Does the enricher retry? What retry policy applies? -2. If all retries fail, where does the message go? -3. Is the original message preserved untouched for retry later? +var options = Options.Create(new ContentEnricherOptions +{ + EndpointUrlTemplate = "https://api.example.com/{key}", + LookupKeyPath = "cid", + MergeTargetPath = "loyalty", +}); -Now consider: the enricher calls two services. Service A succeeds but Service B fails. Is the partial enrichment from Service A committed? How does this affect **atomicity**? Design a strategy: should partial enrichment be discarded or preserved? +var enricher = new ContentEnricher( + source, options, NullLogger.Instance); -### Step 3: Design a Caching Strategy for Scalability +var payload = """{"cid":"C-1","amount":50,"currency":"USD"}"""; -At 10,000 messages/second, each enrichment requires an HTTP call to an external CRM. Without caching, that's 10,000 HTTP calls/second. Design a caching strategy: +var result = await enricher.EnrichAsync(payload, Guid.NewGuid()); -| Cache Level | TTL | Hit Rate | Scalability Impact | -|-------------|-----|----------|-------------------| -| In-memory (per-worker) | 60s | ~80% | Reduces to 2,000 calls/second | -| Distributed (Redis) | 5min | ~95% | Reduces to 500 calls/second | -| Database fallback | 1hr | ~99% | ? | +using var doc = JsonDocument.Parse(result); +Assert.That(doc.RootElement.GetProperty("cid").GetString(), Is.EqualTo("C-1")); +Assert.That(doc.RootElement.GetProperty("amount").GetInt32(), Is.EqualTo(50)); +Assert.That(doc.RootElement.GetProperty("currency").GetString(), Is.EqualTo("USD")); +``` -Open `src/Processing.Transform/` and check if the platform implements caching. How does cache invalidation interact with message **consistency**? +--- + +## Lab + +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial18/Lab.cs`](../tests/TutorialLabs/Tutorial18/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial18.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial18/Exam.cs`](../tests/TutorialLabs/Tutorial18/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial18.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/19-content-filter.md b/EnterpriseIntegrationPlatform/tutorials/19-content-filter.md index 4ef42ef..953e2d0 100644 --- a/EnterpriseIntegrationPlatform/tutorials/19-content-filter.md +++ b/EnterpriseIntegrationPlatform/tutorials/19-content-filter.md @@ -1,34 +1,10 @@ # Tutorial 19 — Content Filter -## What You'll Learn - -- The EIP Content Filter pattern for removing unwanted fields -- How `IContentFilter` / `ContentFilter` strips payloads via JSONPath -- The keep-paths approach: specify what to retain, everything else is removed -- Why smaller payloads improve downstream performance and security -- The complementary relationship with the Content Enricher +Strip payloads down to only the fields downstream consumers need using `IContentFilter` and `JsonPathFilterStep`. --- -## EIP Pattern: Content Filter - -> *"Use a Content Filter to remove unimportant data items from a message, leaving only the items that are important."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────────────┐ ┌────────────────┐ ┌───────────────┐ - │ Full Payload │───▶│ Content Filter │───▶│ Slim Payload │ - │ (20 fields) │ │ (keepPaths) │ │ (3 fields) │ - └──────────────────┘ └────────────────┘ └───────────────┘ -``` - -The Content Filter is the inverse of the Content Enricher. Instead of adding data, it **removes everything except** the specified fields. This produces smaller, focused payloads for downstream consumers that only need a subset of the data. - ---- - -## Platform Implementation - -### IContentFilter +## Key Types ```csharp // src/Processing.Transform/IContentFilter.cs @@ -41,87 +17,135 @@ public interface IContentFilter } ``` -### ContentFilter (concrete) - -The `ContentFilter` class: -1. Parses the incoming JSON payload -2. Evaluates each `keepPaths` entry as a dot-separated property path (e.g. `order.id`, `customer.address.city`) -3. Builds a new JSON document containing **only** the specified paths -4. Paths that don't exist in the payload are silently ignored -5. Returns the filtered JSON string - -### Example - -``` -Input: { "order": { "id": 1, "total": 99.50, "notes": "..." }, - "customer": { "name": "Alice", "ssn": "123-45-6789" }, - "internal": { "debugTrace": "..." } } - -keepPaths: ["order.id", "order.total", "customer.name"] - -Output: { "order": { "id": 1, "total": 99.50 }, - "customer": { "name": "Alice" } } +```csharp +// src/Processing.Transform/JsonPathFilterStep.cs (implements ITransformStep) +public class JsonPathFilterStep : ITransformStep +{ + public JsonPathFilterStep(IEnumerable keepPaths); + public string Name => "JsonPathFilter"; + public Task ExecuteAsync( + TransformContext context, + CancellationToken cancellationToken = default); +} ``` -Notice that `customer.ssn` and `internal.debugTrace` are stripped — this is important for **data minimisation** and **security**. - --- -## Scalability Dimension +## Exercises -The content filter is **stateless and CPU-light** — JSON parsing and selective copying are fast operations. It scales horizontally without limitation. Filtering *reduces* payload size, which **decreases** downstream broker storage, network bandwidth, and consumer memory usage. In high-throughput pipelines, filtering early in the chain has a multiplier effect on overall system capacity. +### Exercise 1: Retain only specified top-level paths ---- - -## Atomicity Dimension +```csharp +var step = new JsonPathFilterStep(new[] { "name", "age" }); +var context = new TransformContext( + """{"name":"Alice","age":30,"email":"a@b.com","role":"admin"}""", + "application/json"); + +var result = await step.ExecuteAsync(context); + +using var doc = JsonDocument.Parse(result.Payload); +Assert.That(doc.RootElement.TryGetProperty("name", out _), Is.True); +Assert.That(doc.RootElement.TryGetProperty("age", out _), Is.True); +Assert.That(doc.RootElement.TryGetProperty("email", out _), Is.False); +Assert.That(doc.RootElement.TryGetProperty("role", out _), Is.False); +``` -Filtering is a **pure, deterministic function** — the same input and keep-paths always produce the same output. This makes it fully idempotent and safe for retries. The filtered message is published before the source is Acked. If any step fails, the source message is Nacked and redelivered. Since filtering never adds data, there is no risk of inconsistency between retries. +### Exercise 2: Extract nested properties ---- +```csharp +var step = new JsonPathFilterStep(new[] { "order.id", "customer.name" }); +var payload = """ + { + "order": {"id": "ORD-1", "total": 100}, + "customer": {"name": "Bob", "email": "bob@test.com"}, + "internal": "secret" + } + """; + +var context = new TransformContext(payload, "application/json"); +var result = await step.ExecuteAsync(context); + +using var doc = JsonDocument.Parse(result.Payload); +Assert.That(doc.RootElement.GetProperty("order").GetProperty("id").GetString(), + Is.EqualTo("ORD-1")); +Assert.That(doc.RootElement.GetProperty("customer").GetProperty("name").GetString(), + Is.EqualTo("Bob")); +Assert.That(doc.RootElement.TryGetProperty("internal", out _), Is.False); +``` -## Lab +### Exercise 3: Missing paths are silently skipped -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial19/Lab.cs`](../tests/TutorialLabs/Tutorial19/Lab.cs) +```csharp +var step = new JsonPathFilterStep(new[] { "name", "nonexistent" }); +var context = new TransformContext( + """{"name":"Alice","age":30}""", "application/json"); -**Objective:** Apply the Content Filter pattern to remove unnecessary data, analyze data minimization for **security** and **scalability**, and design a filter-then-route pipeline. +var result = await step.ExecuteAsync(context); -### Step 1: Configure a Content Filter +using var doc = JsonDocument.Parse(result.Payload); +Assert.That(doc.RootElement.TryGetProperty("name", out _), Is.True); +Assert.That(doc.RootElement.TryGetProperty("nonexistent", out _), Is.False); +``` -A message has fields: `order.id`, `order.items[]`, `customer.email`, `customer.phone`, `customer.ssn`, `audit.createdBy`. You need only `order.id` and `customer.email` for the downstream billing system. Write the `keepPaths` configuration: +### Exercise 4: Filter step inside a pipeline ```csharp -var keepPaths = new[] { "order.id", "customer.email" }; +var filterStep = new JsonPathFilterStep(new[] { "order.id", "order.total" }); +var options = Options.Create(new TransformOptions()); +var pipeline = new TransformPipeline( + new ITransformStep[] { filterStep }, options, + NullLogger.Instance); + +var payload = """ + {"order":{"id":"ORD-5","total":250,"items":3},"customer":{"name":"Eve"}} + """.Trim(); + +var result = await pipeline.ExecuteAsync(payload, "application/json"); + +using var doc = JsonDocument.Parse(result.Payload); +Assert.That(doc.RootElement.GetProperty("order").GetProperty("id").GetString(), + Is.EqualTo("ORD-5")); +Assert.That(doc.RootElement.GetProperty("order").GetProperty("total").GetInt32(), + Is.EqualTo(250)); +Assert.That(doc.RootElement.TryGetProperty("customer", out _), Is.False); +Assert.That(result.StepsApplied, Is.EqualTo(1)); ``` -Open `src/Processing.Transform/JsonPathFilterStep.cs` and trace: What happens to `customer.ssn`? What happens if `keepPaths` references a field that doesn't exist in the message (e.g., `customer.address.zipCode`)? +### Exercise 5: ContentFilter retains only keep paths -### Step 2: Design for Security and Data Minimization +```csharp +var filter = new ContentFilter(NullLogger.Instance); -The Content Filter is a key tool for **data minimization** (GDPR, PCI-DSS). Design a pipeline: +var payload = """ + {"user":"Alice","age":30,"email":"a@b.com","role":"admin","secret":"x"} + """.Trim(); -| Consumer | Allowed Fields | Filtered Fields | -|----------|---------------|----------------| -| Billing | `order.id`, `customer.email`, `order.total` | PII, items, audit | -| Analytics | `order.id`, `order.items[]`, `order.total` | All customer PII | -| Audit | All fields | None (full record) | +var result = await filter.FilterAsync(payload, new[] { "user", "age" }); -How does the Content Filter ensure that the billing system **never** receives `customer.ssn`? Why is this an **atomicity** concern — what happens if the filter is accidentally misconfigured? +using var doc = JsonDocument.Parse(result); +Assert.That(doc.RootElement.TryGetProperty("user", out _), Is.True); +Assert.That(doc.RootElement.TryGetProperty("age", out _), Is.True); +Assert.That(doc.RootElement.TryGetProperty("email", out _), Is.False); +Assert.That(doc.RootElement.TryGetProperty("secret", out _), Is.False); +``` + +--- -### Step 3: Design an Enrich-Then-Filter Pipeline +## Lab -Explain why the order matters: first Enrich (Tutorial 18) then Filter. Draw a pipeline: +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial19/Lab.cs`](../tests/TutorialLabs/Tutorial19/Lab.cs) -``` -Raw message → Content Enricher (add customer data) → Content Filter (remove sensitive fields) → Route to consumer +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial19.Lab" ``` -If you reverse the order (filter first, then enrich), what goes wrong? How does the pipeline order preserve both data completeness and data minimization? - ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial19/Exam.cs`](../tests/TutorialLabs/Tutorial19/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial19.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/20-splitter.md b/EnterpriseIntegrationPlatform/tutorials/20-splitter.md index f701eba..13ecf18 100644 --- a/EnterpriseIntegrationPlatform/tutorials/20-splitter.md +++ b/EnterpriseIntegrationPlatform/tutorials/20-splitter.md @@ -1,36 +1,10 @@ # Tutorial 20 — Splitter -## What You'll Learn - -- The EIP Splitter pattern for breaking composite messages into individual items -- How `IMessageSplitter` and `ISplitStrategy` separate concerns -- `JsonArraySplitStrategy` for splitting JSON arrays -- How each split item gets a shared `CorrelationId`, unique `SequenceNumber`, and `TotalCount` -- The `SplitResult` with item count and published envelopes - ---- - -## EIP Pattern: Splitter - -> *"Use a Splitter to break out the composite message into a series of individual messages, each containing data related to one item."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌───────────────────┐ ┌──────────┐ ┌─────┐ - │ Composite Message │───▶│ Splitter │───▶│ A │ (seq 0, total 3) - │ [A, B, C] │ │ │───▶│ B │ (seq 1, total 3) - └───────────────────┘ │ │───▶│ C │ (seq 2, total 3) - └──────────┘ - All items share the same CorrelationId -``` - -A batch message arrives containing multiple items. The Splitter publishes each item as an independent message, tagged with correlation metadata so they can be reassembled later by an Aggregator (Tutorial 21). +Break composite messages into individual items using `IMessageSplitter` with pluggable `ISplitStrategy`. --- -## Platform Implementation - -### IMessageSplitter +## Key Types ```csharp // src/Processing.Splitter/IMessageSplitter.cs @@ -42,8 +16,6 @@ public interface IMessageSplitter } ``` -### ISplitStrategy - ```csharp // src/Processing.Splitter/ISplitStrategy.cs public interface ISplitStrategy @@ -52,95 +24,156 @@ public interface ISplitStrategy } ``` -The splitter delegates to a strategy for the actual splitting logic. This separation allows different strategies (JSON array, XML child elements, line-based) to be swapped without changing the splitter. - -### JsonArraySplitStrategy +```csharp +// src/Processing.Splitter/SplitResult.cs +public sealed record SplitResult( + IReadOnlyList> SplitEnvelopes, + Guid SourceMessageId, + string TargetTopic, + int ItemCount); +``` ```csharp -// src/Processing.Splitter/JsonArraySplitStrategy.cs -public IReadOnlyList Split(JsonElement composite) +// src/Processing.Splitter/SplitterOptions.cs +public sealed class SplitterOptions { - // Resolves the target array (root array or named property) - // Clones each element to decouple from the source JsonDocument - // Returns individual items as a list + public string TargetTopic { get; init; } + public string? ArrayPropertyName { get; init; } } ``` -If the root payload is not an array, set `SplitterOptions.ArrayPropertyName` to specify which property holds the array (e.g. `"items"` for `{ "items": [...] }`). +--- -### SplitResult +## Exercises + +### Exercise 1: Split comma-separated string into individual envelopes ```csharp -// src/Processing.Splitter/SplitResult.cs -public sealed record SplitResult( - IReadOnlyList> SplitEnvelopes, - Guid SourceMessageId, - string TargetTopic, - int ItemCount); +var producer = Substitute.For(); +var strategy = new FuncSplitStrategy( + composite => composite.Split(',').ToList()); + +var options = Options.Create(new SplitterOptions { TargetTopic = "items-topic" }); +var splitter = new MessageSplitter( + strategy, producer, options, + NullLogger>.Instance); + +var source = IntegrationEnvelope.Create( + "apple,banana,cherry", "InventoryService", "batch.items"); + +var result = await splitter.SplitAsync(source); + +Assert.That(result.ItemCount, Is.EqualTo(3)); +Assert.That(result.TargetTopic, Is.EqualTo("items-topic")); +Assert.That(result.SourceMessageId, Is.EqualTo(source.MessageId)); +Assert.That(result.SplitEnvelopes[0].Payload, Is.EqualTo("apple")); +Assert.That(result.SplitEnvelopes[1].Payload, Is.EqualTo("banana")); +Assert.That(result.SplitEnvelopes[2].Payload, Is.EqualTo("cherry")); ``` -Each split envelope receives: -- **Same `CorrelationId`** as the source — links all items to the original batch -- **Unique `SequenceNumber`** (0, 1, 2, …) — ordering within the batch -- **`TotalCount`** in metadata — how many items the batch contained +### Exercise 2: Split preserves CorrelationId and sets CausationId ---- +```csharp +var producer = Substitute.For(); +var strategy = new FuncSplitStrategy(s => new[] { s }); -## Scalability Dimension +var options = Options.Create(new SplitterOptions { TargetTopic = "topic" }); +var splitter = new MessageSplitter( + strategy, producer, options, + NullLogger>.Instance); -The splitter is **stateless** — it reads one composite message and produces N individual messages. Horizontal scaling via competing consumers is straightforward. Note that splitting **amplifies** message count: a batch of 100 items produces 100 messages. Capacity planning must account for this amplification factor on the target topic and downstream consumers. +var source = IntegrationEnvelope.Create( + "payload", "Service", "event.type"); ---- +var result = await splitter.SplitAsync(source); -## Atomicity Dimension +var splitEnv = result.SplitEnvelopes[0]; +Assert.That(splitEnv.CorrelationId, Is.EqualTo(source.CorrelationId)); +Assert.That(splitEnv.CausationId, Is.EqualTo(source.MessageId)); +Assert.That(splitEnv.MessageId, Is.Not.EqualTo(source.MessageId)); +``` -All split items are published to the target topic before the source message is Acked. If any publish fails, the source is Nacked and redelivered. On retry, the entire batch is re-split, producing the same items (splitting is deterministic). Downstream consumers use `MessageId` for deduplication. The `CorrelationId` + `SequenceNumber` + `TotalCount` triplet enables the Aggregator to know exactly when all items have arrived. +### Exercise 3: No target topic configured throws ---- +```csharp +var producer = Substitute.For(); +var strategy = new FuncSplitStrategy(s => new[] { s }); -## Lab +var options = Options.Create(new SplitterOptions { TargetTopic = "" }); +var splitter = new MessageSplitter( + strategy, producer, options, + NullLogger>.Instance); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial20/Lab.cs`](../tests/TutorialLabs/Tutorial20/Lab.cs) +var source = IntegrationEnvelope.Create("data", "Svc", "evt"); + +Assert.ThrowsAsync( + () => splitter.SplitAsync(source)); +``` + +### Exercise 4: Zero items returns empty result, no publish + +```csharp +var producer = Substitute.For(); +var strategy = new FuncSplitStrategy(_ => Array.Empty()); -**Objective:** Split composite messages into individual items, trace how `SequenceNumber` and `TotalCount` enable the Aggregator to reassemble split messages, and analyze **atomicity** when a split item fails. +var options = Options.Create(new SplitterOptions { TargetTopic = "topic" }); +var splitter = new MessageSplitter( + strategy, producer, options, + NullLogger>.Instance); -### Step 1: Split a Composite Message +var source = IntegrationEnvelope.Create("empty", "Svc", "evt"); -A message `{ "orders": [{ "id": 1, "total": 50 }, { "id": 2, "total": 150 }, { "id": 3, "total": 75 }] }` is split using `JsonArraySplitStrategy` with `ArrayPropertyName = "orders"`. Open `src/Processing.Splitter/` and trace: +var result = await splitter.SplitAsync(source); -1. How many envelopes are in `SplitResult.SplitEnvelopes`? -2. What is `ItemCount`? -3. What `SequenceNumber` and `TotalCount` does each split envelope carry? -4. Do all split envelopes share the same `CorrelationId` as the original? +Assert.That(result.ItemCount, Is.EqualTo(0)); +Assert.That(result.SplitEnvelopes, Is.Empty); +await producer.DidNotReceive() + .PublishAsync(Arg.Any>(), + Arg.Any(), Arg.Any()); +``` -### Step 2: Trace Atomicity When a Split Item Fails +### Exercise 5: JsonArraySplitStrategy splits top-level array -After splitting, the 3 items are processed independently. Item 2 (sequence 1) fails delivery: +```csharp +var producer = Substitute.For(); +var splitOptions = Options.Create(new SplitterOptions { TargetTopic = "json-items" }); +var strategy = new JsonArraySplitStrategy(splitOptions); -| Item | SequenceNumber | Status | -|------|---------------|--------| -| `{ "id": 1 }` | 0 | ✅ Delivered | -| `{ "id": 2 }` | 1 | ❌ Failed | -| `{ "id": 3 }` | 2 | ✅ Delivered | +var splitter = new MessageSplitter( + strategy, producer, splitOptions, + NullLogger>.Instance); -Questions: -- How does the Aggregator (Tutorial 21) detect that item 2 is missing? (hint: `TotalCount = 3` but only 2 arrived) -- Should the Aggregator wait indefinitely or timeout? What timeout strategy preserves **atomicity**? -- Should items 1 and 3 be rolled back (saga compensation), or should only item 2 be retried? +var jsonArray = JsonSerializer.Deserialize( + """[{"id":1},{"id":2},{"id":3}]"""); -### Step 3: Evaluate Splitter Scalability +var source = IntegrationEnvelope.Create( + jsonArray, "BatchService", "batch.created"); -Splitting a message with 1,000 items creates 1,000 individual messages. Analyze: +var result = await splitter.SplitAsync(source); -- Each split message is independently processed — what parallelism level is achievable? -- What is the memory impact of cloning 1,000 JSON elements? (hint: `JsonSerializer.SerializeToElement` creates deep copies) -- Why does `JsonArraySplitStrategy` clone each element rather than using references? What **concurrency** bug would occur without cloning? +Assert.That(result.ItemCount, Is.EqualTo(3)); +Assert.That(result.SplitEnvelopes[0].Payload.GetProperty("id").GetInt32(), Is.EqualTo(1)); +Assert.That(result.SplitEnvelopes[1].Payload.GetProperty("id").GetInt32(), Is.EqualTo(2)); +Assert.That(result.SplitEnvelopes[2].Payload.GetProperty("id").GetInt32(), Is.EqualTo(3)); +``` + +--- + +## Lab + +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial20/Lab.cs`](../tests/TutorialLabs/Tutorial20/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial20.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial20/Exam.cs`](../tests/TutorialLabs/Tutorial20/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial20.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/21-aggregator.md b/EnterpriseIntegrationPlatform/tutorials/21-aggregator.md index db9109e..938255a 100644 --- a/EnterpriseIntegrationPlatform/tutorials/21-aggregator.md +++ b/EnterpriseIntegrationPlatform/tutorials/21-aggregator.md @@ -1,34 +1,10 @@ # Tutorial 21 — Aggregator -## What You'll Learn - -- The EIP Aggregator pattern for combining related messages into one -- How `IMessageAggregator` collects and releases groups -- `ICompletionStrategy` and `CountCompletionStrategy` for deciding when a group is ready -- `IAggregationStrategy` for combining items into the aggregate payload -- `IMessageAggregateStore` for persisting in-flight groups -- The `AggregateResult` with `IsComplete`, `ReceivedCount`, and `CorrelationId` +Collect related messages by `CorrelationId` and combine them into a single aggregate when the group is complete. --- -## EIP Pattern: Aggregator - -> *"Use a stateful filter, an Aggregator, to collect and store individual messages until a complete set of related messages has been received. Then, the Aggregator publishes a single message distilled from the individual messages."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - Item A (corr=X, seq=0) ──▶ ┌────────────┐ - Item B (corr=X, seq=1) ──▶ │ Aggregator │──▶ Aggregate [A,B,C] - Item C (corr=X, seq=2) ──▶ └────────────┘ (published when complete) -``` - -The Aggregator is the counterpart to the Splitter (Tutorial 20). It collects individual messages sharing the same `CorrelationId` and, when the group is complete, combines them into a single aggregate message. - ---- - -## Platform Implementation - -### IMessageAggregator +## Key Types ```csharp // src/Processing.Aggregator/IMessageAggregator.cs @@ -38,21 +14,13 @@ public interface IMessageAggregator IntegrationEnvelope envelope, CancellationToken cancellationToken = default); } -``` - -### ICompletionStrategy -```csharp // src/Processing.Aggregator/ICompletionStrategy.cs public interface ICompletionStrategy { bool IsComplete(IReadOnlyList> group); } -``` - -### CountCompletionStrategy -```csharp // src/Processing.Aggregator/CountCompletionStrategy.cs public sealed class CountCompletionStrategy : ICompletionStrategy { @@ -60,37 +28,22 @@ public sealed class CountCompletionStrategy : ICompletionStrategy public bool IsComplete(IReadOnlyList> group) => group.Count >= _expectedCount; } -``` - -Reads `TotalCount` from the envelope metadata (set by the Splitter) to know the expected group size. - -### IAggregationStrategy -```csharp // src/Processing.Aggregator/IAggregationStrategy.cs public interface IAggregationStrategy { TAggregate Aggregate(IReadOnlyList items); } -``` -### IMessageAggregateStore - -```csharp // src/Processing.Aggregator/IMessageAggregateStore.cs public interface IMessageAggregateStore { Task>> AddAsync( IntegrationEnvelope envelope, CancellationToken cancellationToken = default); - Task RemoveGroupAsync(Guid correlationId, CancellationToken cancellationToken = default); } -``` -### AggregateResult - -```csharp // src/Processing.Aggregator/AggregateResult.cs public sealed record AggregateResult( bool IsComplete, @@ -101,56 +54,167 @@ public sealed record AggregateResult( --- -## Scalability Dimension +## Exercises -The Aggregator is **stateful** — it must collect items across multiple messages. The `IMessageAggregateStore` abstracts the storage (in-memory, Redis, Cassandra). For horizontal scaling, the store must be **shared** across replicas or messages must be **partitioned by CorrelationId** so all items in a group land on the same replica. Partition-based routing is the preferred approach as it avoids distributed locking. +### Exercise 1: Store groups items by CorrelationId ---- +```csharp +var store = new InMemoryMessageAggregateStore(); +var correlationId = Guid.NewGuid(); -## Atomicity Dimension +var e1 = IntegrationEnvelope.Create( + "item-1", "Svc", "line", correlationId: correlationId); +var e2 = IntegrationEnvelope.Create( + "item-2", "Svc", "line", correlationId: correlationId); -Each `AggregateAsync` call atomically adds the item to the store and checks completion. If the group becomes complete, the aggregate is published and the group is removed from the store — all within a single logical operation. If the process crashes after adding but before publishing, the item is re-added on redelivery (the store must be idempotent on `MessageId`). The aggregate is published before the final item's source message is Acked. +await store.AddAsync(e1); +var group = await store.AddAsync(e2); ---- +Assert.That(group.Count, Is.EqualTo(2)); +Assert.That(group[0].Payload, Is.EqualTo("item-1")); +Assert.That(group[1].Payload, Is.EqualTo("item-2")); +``` -## Lab +### Exercise 2: CountCompletionStrategy fires when count reached -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial21/Lab.cs`](../tests/TutorialLabs/Tutorial21/Lab.cs) +```csharp +var strategy = new CountCompletionStrategy(2); +var envelopes = new[] +{ + IntegrationEnvelope.Create("a", "Svc", "t"), + IntegrationEnvelope.Create("b", "Svc", "t"), +}; + +Assert.That(strategy.IsComplete(envelopes), Is.True); +``` + +### Exercise 3: Aggregator returns incomplete when group not ready + +```csharp +var store = new InMemoryMessageAggregateStore(); +var completion = new CountCompletionStrategy(3); +var aggregation = Substitute.For>(); +var producer = Substitute.For(); + +var options = Options.Create(new AggregatorOptions +{ + TargetTopic = "aggregated-topic", + ExpectedCount = 3, +}); + +var aggregator = new MessageAggregator( + store, completion, aggregation, producer, options, + NullLogger>.Instance); + +var correlationId = Guid.NewGuid(); +var envelope = IntegrationEnvelope.Create( + "item-1", "Svc", "line", correlationId: correlationId); + +var result = await aggregator.AggregateAsync(envelope); -**Objective:** Trace the Aggregator's completion logic, design timeout strategies, and analyze how **idempotent** aggregation ensures **atomic** reassembly of split messages. +Assert.That(result.IsComplete, Is.False); +Assert.That(result.AggregateEnvelope, Is.Null); +Assert.That(result.ReceivedCount, Is.EqualTo(1)); +Assert.That(result.CorrelationId, Is.EqualTo(correlationId)); +``` -### Step 1: Trace Aggregation Completion +### Exercise 4: Aggregator completes and publishes when count reached -A Splitter produces 5 items with `TotalCount = 5`. Items arrive out of order: 3, 0, 4, 1, 2. Open `src/Processing.Aggregator/MessageAggregator.cs` and trace: +```csharp +var store = new InMemoryMessageAggregateStore(); +var completion = new CountCompletionStrategy(2); +var aggregation = Substitute.For>(); +aggregation + .Aggregate(Arg.Any>()) + .Returns(ci => + { + var items = ci.Arg>(); + return string.Join(",", items); + }); + +var producer = Substitute.For(); +var options = Options.Create(new AggregatorOptions +{ + TargetTopic = "agg-out", + TargetMessageType = "order.batch", + ExpectedCount = 2, +}); + +var aggregator = new MessageAggregator( + store, completion, aggregation, producer, options, + NullLogger>.Instance); + +var correlationId = Guid.NewGuid(); +var e1 = IntegrationEnvelope.Create( + "A", "Svc", "line", correlationId: correlationId); +var e2 = IntegrationEnvelope.Create( + "B", "Svc", "line", correlationId: correlationId); + +await aggregator.AggregateAsync(e1); +var result = await aggregator.AggregateAsync(e2); + +Assert.That(result.IsComplete, Is.True); +Assert.That(result.ReceivedCount, Is.EqualTo(2)); +Assert.That(result.AggregateEnvelope, Is.Not.Null); +Assert.That(result.AggregateEnvelope!.Payload, Is.EqualTo("A,B")); +Assert.That(result.AggregateEnvelope.MessageType, Is.EqualTo("order.batch")); +Assert.That(result.AggregateEnvelope.CorrelationId, Is.EqualTo(correlationId)); +``` -1. After receiving items 0, 1, 2, 3 — what does `AggregateResult.ReceivedCount` return? What is `IsComplete`? -2. When item 4 arrives, how does the Aggregator know the group is complete? -3. What `CorrelationId` links all 5 items to the same aggregate group? +### Exercise 5: Aggregator merges metadata from all envelopes -### Step 2: Design a Timeout Completion Strategy +```csharp +var store = new InMemoryMessageAggregateStore(); +var completion = new CountCompletionStrategy(2); +var aggregation = Substitute.For>(); +aggregation.Aggregate(Arg.Any>()).Returns("merged"); +var producer = Substitute.For(); +var options = Options.Create(new AggregatorOptions +{ + TargetTopic = "merged-topic", + ExpectedCount = 2, +}); -Not all split items may arrive (e.g., item 2 fails permanently). Design a timeout strategy: +var aggregator = new MessageAggregator( + store, completion, aggregation, producer, options, + NullLogger>.Instance); -- After 30 seconds from the first item, complete the aggregate with whatever has arrived -- Mark the result as `IsPartial = true` -- Route the partial aggregate to a `review.incomplete-batches` topic +var correlationId = Guid.NewGuid(); +var e1 = IntegrationEnvelope.Create( + "A", "Svc", "line", correlationId: correlationId) with +{ + Metadata = new Dictionary { ["key1"] = "val1" }, +}; +var e2 = IntegrationEnvelope.Create( + "B", "Svc", "line", correlationId: correlationId) with +{ + Metadata = new Dictionary { ["key2"] = "val2" }, +}; -What **atomicity** decision must you make: should a partial aggregate be considered "successful" or should it trigger compensation for already-delivered items? +await aggregator.AggregateAsync(e1); +var result = await aggregator.AggregateAsync(e2); -### Step 3: Analyze Idempotent Aggregation +Assert.That(result.AggregateEnvelope!.Metadata, Contains.Key("key1")); +Assert.That(result.AggregateEnvelope.Metadata, Contains.Key("key2")); +``` -A message with `SequenceNumber = 2` is delivered twice (broker redelivery). Without idempotency: +--- -- The aggregate would count 6 items instead of 5 -- `IsComplete` would never be true (6 > 5) or would fire prematurely +## Lab -Open `src/Processing.Aggregator/` and verify: How does `IMessageAggregateStore` handle duplicate `MessageId`s? Why is idempotency critical for **scalable** at-least-once delivery systems? +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial21/Lab.cs`](../tests/TutorialLabs/Tutorial21/Lab.cs) + +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial21.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial21/Exam.cs`](../tests/TutorialLabs/Tutorial21/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial21.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/22-scatter-gather.md b/EnterpriseIntegrationPlatform/tutorials/22-scatter-gather.md index a6abea4..3e6a6d2 100644 --- a/EnterpriseIntegrationPlatform/tutorials/22-scatter-gather.md +++ b/EnterpriseIntegrationPlatform/tutorials/22-scatter-gather.md @@ -1,40 +1,10 @@ # Tutorial 22 — Scatter-Gather -## What You'll Learn - -- The EIP Scatter-Gather pattern for broadcasting a request and collecting responses -- How `IScatterGatherer` scatters to a recipient list with a timeout -- `ScatterRequest` with payload, correlation ID, and recipient list -- `GatherResponse` per recipient with success/error status -- `ScatterGatherResult` with collected responses and timeout flag +Broadcast a request to multiple recipients in parallel and collect their responses within a timeout window. --- -## EIP Pattern: Scatter-Gather - -> *"Use a Scatter-Gather that broadcasts a message to multiple recipients and re-aggregates the responses back into a single message."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────┐ - ┌───▶│ Service A│───┐ - ┌──────────────┐ │ └──────────┘ │ ┌──────────────┐ - │ Scatter │───┤ ┌──────────┐ ├───▶│ Gather │ - │ (broadcast) │ ├───▶│ Service B│───┤ │ (aggregate) │ - └──────────────┘ │ └──────────┘ │ └──────────────┘ - │ ┌──────────┐ │ - └───▶│ Service C│───┘ - └──────────┘ - ←──── timeout window ────▶ -``` - -Scatter-Gather combines the Recipient List (fan-out) and Aggregator (collection) into a single operation with a timeout window. Responses that arrive after the timeout are discarded. - ---- - -## Platform Implementation - -### IScatterGatherer +## Key Types ```csharp // src/Processing.ScatterGather/IScatterGatherer.cs @@ -44,21 +14,13 @@ public interface IScatterGatherer ScatterRequest request, CancellationToken cancellationToken = default); } -``` - -### ScatterRequest -```csharp // src/Processing.ScatterGather/ScatterRequest.cs public sealed record ScatterRequest( Guid CorrelationId, TRequest Payload, IReadOnlyList Recipients); -``` - -### GatherResponse -```csharp // src/Processing.ScatterGather/GatherResponse.cs public sealed record GatherResponse( string Recipient, @@ -66,11 +28,7 @@ public sealed record GatherResponse( DateTimeOffset ReceivedAt, bool IsSuccess, string? ErrorMessage); -``` -### ScatterGatherResult - -```csharp // src/Processing.ScatterGather/ScatterGatherResult.cs public sealed record ScatterGatherResult( Guid CorrelationId, @@ -79,71 +37,133 @@ public sealed record ScatterGatherResult( TimeSpan Duration); ``` -`TimedOut = true` when the gather phase ended because `ScatterGatherOptions.TimeoutMs` elapsed before all recipients responded. The `Responses` list contains whatever was collected before the timeout. - --- -## Scalability Dimension +## Exercises -The scatter phase is **stateless fan-out** — identical to the Recipient List. The gather phase is **stateful** — it holds a `TaskCompletionSource` per recipient and waits for replies correlated by `CorrelationId`. Each scatter-gather operation is independent, so multiple operations can run concurrently. The timeout prevents unbounded resource holding. For high-volume scatter-gather, partition operations across replicas by `CorrelationId`. +### Exercise 1: Empty recipients returns immediately ---- +```csharp +var producer = Substitute.For(); +var options = Options.Create(new ScatterGatherOptions { TimeoutMs = 5000 }); -## Atomicity Dimension +var sg = new ScatterGatherer( + producer, options, + NullLogger>.Instance); -Scatter-Gather has **best-effort semantics** within the timeout window. If a recipient fails to respond, its `GatherResponse` is absent from the result and `TimedOut = true`. The caller decides how to handle partial results — it may proceed with available responses or retry the entire operation. The `Duration` field provides observability into how long the operation took, enabling SLA monitoring. +var request = new ScatterRequest( + Guid.NewGuid(), "ping", new List()); ---- +var result = await sg.ScatterGatherAsync(request); -## Lab +Assert.That(result.Responses, Is.Empty); +Assert.That(result.TimedOut, Is.False); +Assert.That(result.Duration, Is.LessThanOrEqualTo(TimeSpan.FromSeconds(1))); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial22/Lab.cs`](../tests/TutorialLabs/Tutorial22/Lab.cs) +### Exercise 2: Exceeding max recipients throws -**Objective:** Trace the Scatter-Gather pattern's parallel request-response flow, analyze timeout behavior for **partial results**, and design a "best-of-N" selection strategy. +```csharp +var producer = Substitute.For(); +var options = Options.Create(new ScatterGatherOptions +{ + MaxRecipients = 2, + TimeoutMs = 5000, +}); -### Step 1: Trace a Scatter-Gather with Timeout +var sg = new ScatterGatherer( + producer, options, + NullLogger>.Instance); -You scatter a pricing request to 3 suppliers with `TimeoutMs = 5000`: +var request = new ScatterRequest( + Guid.NewGuid(), "payload", + new List { "t1", "t2", "t3" }); -| Supplier | Response Time | Price | -|----------|--------------|-------| -| A | 1 second | $120 | -| B | 3 seconds | $95 | -| C | Never responds | — | +Assert.ThrowsAsync(() => sg.ScatterGatherAsync(request)); +``` -Open `src/Processing.ScatterGather/ScatterGatherer.cs` and trace: +### Exercise 3: Scatter publishes to each recipient topic -1. How does `ScatterGatherResult.Responses` look? (2 responses) -2. Is `TimedOut = true`? (yes — only 2 of 3 responded) -3. What is `Duration`? (≈5 seconds — the timeout) +```csharp +var producer = Substitute.For(); +var options = Options.Create(new ScatterGatherOptions { TimeoutMs = 500 }); -### Step 2: Design a "Best-of-N" Selection Strategy +var sg = new ScatterGatherer( + producer, options, + NullLogger>.Instance); -Using the partial results above, implement a selection strategy that picks the lowest price: +var recipients = new List { "svc-a", "svc-b" }; +var request = new ScatterRequest( + Guid.NewGuid(), "hello", recipients); +await sg.ScatterGatherAsync(request); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + "svc-a", + Arg.Any()); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + "svc-b", + Arg.Any()); ``` -1. Scatter to all suppliers (parallel) -2. Gather responses until timeout -3. From gathered responses, select the one with lowest price -4. If no responses arrived, route to DLQ with reason "no-supplier-response" + +### Exercise 4: Full scatter-gather completes before timeout + +```csharp +var producer = Substitute.For(); +var options = Options.Create(new ScatterGatherOptions { TimeoutMs = 10_000 }); + +var sg = new ScatterGatherer( + producer, options, + NullLogger>.Instance); + +var correlationId = Guid.NewGuid(); +var request = new ScatterRequest( + correlationId, "query", new List { "svc-a" }); + +var scatterTask = sg.ScatterGatherAsync(request); + +await Task.Delay(100); +var submitted = await sg.SubmitResponseAsync( + correlationId, + new GatherResponse("svc-a", "answer", DateTimeOffset.UtcNow, true, null)); + +var result = await scatterTask; + +Assert.That(submitted, Is.True); +Assert.That(result.Responses.Count, Is.EqualTo(1)); +Assert.That(result.Responses[0].Payload, Is.EqualTo("answer")); +Assert.That(result.TimedOut, Is.False); +``` + +### Exercise 5: Default option values + +```csharp +var opts = new ScatterGatherOptions(); + +Assert.That(opts.TimeoutMs, Is.EqualTo(30_000)); +Assert.That(opts.MaxRecipients, Is.EqualTo(50)); ``` -What is the **atomicity** guarantee? The selected best price must be committed as a single decision — if the commit fails, no supplier should be charged. +--- -### Step 3: Compare Scatter-Gather Latency vs. Sequential Calls +## Lab -| Approach | 3 services × 2s avg | 10 services × 2s avg | -|----------|---------------------|----------------------| -| Sequential | 6 seconds total | 20 seconds total | -| Scatter-Gather | ~2 seconds (parallel) | ~2 seconds (parallel) | +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial22/Lab.cs`](../tests/TutorialLabs/Tutorial22/Lab.cs) -How does the Scatter-Gather pattern enable **scalable** multi-supplier/multi-service integration? What happens to latency as you add more recipients? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial22.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial22/Exam.cs`](../tests/TutorialLabs/Tutorial22/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial22.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/23-request-reply.md b/EnterpriseIntegrationPlatform/tutorials/23-request-reply.md index 1401355..718f00f 100644 --- a/EnterpriseIntegrationPlatform/tutorials/23-request-reply.md +++ b/EnterpriseIntegrationPlatform/tutorials/23-request-reply.md @@ -1,38 +1,10 @@ # Tutorial 23 — Request-Reply -## What You'll Learn - -- The EIP Request-Reply pattern for synchronous-style messaging over async channels -- How `IRequestReplyCorrelator` sends and waits for a reply -- How `IntegrationEnvelope.ReplyTo` and `CorrelationId` link request to response -- `RequestReplyMessage` for describing the request -- `RequestReplyResult` with timeout handling and duration tracking +Send a request over an async channel and correlate the response by `CorrelationId` with timeout support. --- -## EIP Pattern: Request-Reply - -> *"Send a pair of Request-Reply messages, each on its own channel."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────┐ Request (ReplyTo=reply-topic) ┌──────────┐ - │ Requester│──────────────────────────────────▶ │ Responder│ - │ │◀────────────────────────────────── │ │ - └──────────┘ Reply (CorrelationId=X) └──────────┘ - │ │ - │ request-topic │ - │ reply-topic │ - └───── correlated by CorrelationId ──────────────┘ -``` - -The requester publishes a message with `ReplyTo` set to a reply topic and a unique `CorrelationId`. The responder processes the request and publishes the reply to the `ReplyTo` topic with the same `CorrelationId`. The requester subscribes to the reply topic and matches by `CorrelationId`. - ---- - -## Platform Implementation - -### IRequestReplyCorrelator +## Key Types ```csharp // src/Processing.RequestReply/IRequestReplyCorrelator.cs @@ -42,11 +14,7 @@ public interface IRequestReplyCorrelator RequestReplyMessage request, CancellationToken cancellationToken = default); } -``` - -### RequestReplyMessage -```csharp // src/Processing.RequestReply/RequestReplyMessage.cs public record RequestReplyMessage( TRequest Payload, @@ -55,85 +23,141 @@ public record RequestReplyMessage( string Source, string MessageType, Guid? CorrelationId = null); -``` - -When `CorrelationId` is `null`, the correlator generates a new one. - -### RequestReplyResult -```csharp // src/Processing.RequestReply/RequestReplyResult.cs public record RequestReplyResult( Guid CorrelationId, IntegrationEnvelope? Reply, bool TimedOut, TimeSpan Duration); -``` - -### RequestReplyOptions -```csharp // src/Processing.RequestReply/RequestReplyOptions.cs public sealed class RequestReplyOptions { - public int TimeoutMs { get; set; } = 30_000; // 30 seconds default + public int TimeoutMs { get; set; } = 30_000; public string ConsumerGroup { get; set; } = "request-reply"; } ``` --- -## Scalability Dimension +## Exercises -The requester is **stateful for the duration of the request** — it holds a pending `TaskCompletionSource` keyed by `CorrelationId`. Multiple concurrent request-reply operations are supported within a single instance. For horizontal scaling, each replica subscribes to the reply topic with its own consumer group and filters replies by `CorrelationId`. Replies not matching any pending request are ignored. The timeout ensures resources are released even if the responder never replies. +### Exercise 1: RequestReplyMessage record properties ---- +```csharp +var correlationId = Guid.NewGuid(); +var msg = new RequestReplyMessage( + "payload", "req-topic", "reply-topic", "TestSvc", "cmd.ping", correlationId); + +Assert.That(msg.Payload, Is.EqualTo("payload")); +Assert.That(msg.RequestTopic, Is.EqualTo("req-topic")); +Assert.That(msg.ReplyTopic, Is.EqualTo("reply-topic")); +Assert.That(msg.Source, Is.EqualTo("TestSvc")); +Assert.That(msg.MessageType, Is.EqualTo("cmd.ping")); +Assert.That(msg.CorrelationId, Is.EqualTo(correlationId)); +``` -## Atomicity Dimension +### Exercise 2: Correlator publishes request with ReplyTo set -The request is published to the request topic and the correlator subscribes to the reply topic **before** sending. This avoids a race condition where the reply arrives before the subscription is active. If the requester crashes after sending the request, the reply may arrive with no listener — the responder should be idempotent. On timeout, `RequestReplyResult.TimedOut = true` and `Reply = null`, allowing the caller to decide whether to retry or escalate. +```csharp +var producer = Substitute.For(); +var consumer = Substitute.For(); +var options = Options.Create(new RequestReplyOptions { TimeoutMs = 500 }); ---- +var correlator = new RequestReplyCorrelator( + producer, consumer, options, + NullLogger>.Instance); -## Lab +var msg = new RequestReplyMessage( + "ping", "commands", "replies", "TestSvc", "cmd.ping"); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial23/Lab.cs`](../tests/TutorialLabs/Tutorial23/Lab.cs) +await correlator.SendAndReceiveAsync(msg); + +await producer.Received(1).PublishAsync( + Arg.Is>(e => e.ReplyTo == "replies"), + "commands", + Arg.Any()); +``` + +### Exercise 3: Correlator sets intent to Command + +```csharp +var producer = Substitute.For(); +var consumer = Substitute.For(); +var options = Options.Create(new RequestReplyOptions { TimeoutMs = 500 }); + +var correlator = new RequestReplyCorrelator( + producer, consumer, options, + NullLogger>.Instance); + +var msg = new RequestReplyMessage( + "data", "req", "rep", "Svc", "cmd.do"); + +await correlator.SendAndReceiveAsync(msg); + +await producer.Received(1).PublishAsync( + Arg.Is>(e => e.Intent == MessageIntent.Command), + "req", + Arg.Any()); +``` -**Objective:** Trace the Request-Reply correlation mechanism, analyze timeout behavior, and design for **scalable** request-reply across distributed services. +### Exercise 4: Timeout returns TimedOut result -### Step 1: Trace Request-Reply Correlation +```csharp +var producer = Substitute.For(); +var consumer = Substitute.For(); +var options = Options.Create(new RequestReplyOptions { TimeoutMs = 300 }); + +var correlator = new RequestReplyCorrelator( + producer, consumer, options, + NullLogger>.Instance); + +var msg = new RequestReplyMessage( + "request-data", "cmd-topic", "reply-topic", "Svc", "cmd.type"); -A request is sent with `TimeoutMs = 5000`. The responder takes 7 seconds. Open `src/Processing.RequestReply/RequestReplyCorrelator.cs` and trace: +var result = await correlator.SendAndReceiveAsync(msg); -1. What does `RequestReplyResult` look like? (`TimedOut = true`, no response) -2. If the responder takes 3 seconds, what does the result contain? -3. How does the `CorrelationId` in the request envelope match to the response? +Assert.That(result.TimedOut, Is.True); +Assert.That(result.Reply, Is.Null); +Assert.That(result.Duration, Is.GreaterThan(TimeSpan.Zero)); +``` + +### Exercise 5: Empty RequestTopic throws -Now: Two requesters send requests with different `CorrelationId` values to the same request topic. How does each requester receive its own correct reply? +```csharp +var producer = Substitute.For(); +var consumer = Substitute.For(); +var options = Options.Create(new RequestReplyOptions { TimeoutMs = 500 }); -### Step 2: Prevent the Subscribe-Before-Publish Race Condition +var correlator = new RequestReplyCorrelator( + producer, consumer, options, + NullLogger>.Instance); -The correlator subscribes to the reply topic **before** publishing the request. Explain: +var msg = new RequestReplyMessage( + "data", "", "reply-topic", "Svc", "type"); -1. What race condition occurs if you publish first, then subscribe? -2. How does pre-subscribing ensure the reply is never lost? -3. Draw the timeline: Subscribe → Publish → Responder processes → Reply arrives → Correlator matches +Assert.ThrowsAsync( + () => correlator.SendAndReceiveAsync(msg)); +``` -This is an **atomicity** concern: without pre-subscription, fast responders could publish replies before the requester is listening, causing permanent message loss. +--- -### Step 3: Design for Request-Reply Scalability +## Lab -At high throughput, many concurrent request-reply operations share the same reply topic: +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial23/Lab.cs`](../tests/TutorialLabs/Tutorial23/Lab.cs) -- How does the correlator isolate concurrent requests? (hint: `CorrelationId` matching) -- What happens if 1,000 requests are in flight simultaneously? Memory implications? -- How does the `TimeoutMs` prevent resource leaks from requests that never receive replies? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial23.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial23/Exam.cs`](../tests/TutorialLabs/Tutorial23/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial23.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/24-retry-framework.md b/EnterpriseIntegrationPlatform/tutorials/24-retry-framework.md index dd11b19..7c0e6af 100644 --- a/EnterpriseIntegrationPlatform/tutorials/24-retry-framework.md +++ b/EnterpriseIntegrationPlatform/tutorials/24-retry-framework.md @@ -1,32 +1,10 @@ # Tutorial 24 — Retry Framework -## What You'll Learn - -- How retries protect against transient failures in distributed systems -- How `IRetryPolicy` and `ExponentialBackoffRetryPolicy` implement retry logic -- `RetryOptions`: MaxAttempts, InitialDelayMs, BackoffMultiplier, MaxDelayMs, UseJitter -- `RetryResult` with success/failure, attempt count, and last exception -- Why non-retryable errors should bypass retries and go straight to the DLQ - ---- - -## EIP Pattern: Retry - -> In distributed integration, transient failures (network blips, temporary overloads, leader elections) are the norm, not the exception. A retry policy wraps any operation and re-executes it with increasing delays until it succeeds or the maximum attempts are exhausted. - -``` - Attempt 1 ──▶ FAIL ──▶ wait 1 s - Attempt 2 ──▶ FAIL ──▶ wait 2 s - Attempt 3 ──▶ FAIL ──▶ wait 4 s - Attempt 4 ──▶ SUCCESS ✓ - └── exponential backoff with jitter -``` +Wrap operations with exponential backoff retry logic, tracking attempts and surfacing the last exception on exhaustion. --- -## Platform Implementation - -### IRetryPolicy +## Key Types ```csharp // src/Processing.Retry/IRetryPolicy.cs @@ -40,41 +18,7 @@ public interface IRetryPolicy Func operation, CancellationToken ct); } -``` - -### ExponentialBackoffRetryPolicy - -```csharp -// src/Processing.Retry/ExponentialBackoffRetryPolicy.cs (key logic) -public async Task> ExecuteAsync( - Func> operation, CancellationToken ct) -{ - for (int attempt = 1; attempt <= _options.MaxAttempts; attempt++) - { - try - { - var result = await operation(ct); - return new RetryResult - { IsSucceeded = true, Attempts = attempt, Result = result }; - } - catch (OperationCanceledException) when (ct.IsCancellationRequested) - { - throw; // honour cancellation immediately - } - catch (Exception ex) - { - if (attempt < _options.MaxAttempts) - await Task.Delay(ComputeDelay(attempt), ct); - } - } - return new RetryResult - { IsSucceeded = false, Attempts = _options.MaxAttempts, LastException = ... }; -} -``` - -### RetryOptions -```csharp // src/Processing.Retry/RetryOptions.cs public sealed class RetryOptions { @@ -84,11 +28,7 @@ public sealed class RetryOptions public double BackoffMultiplier { get; set; } = 2.0; public bool UseJitter { get; set; } = true; } -``` - -### RetryResult -```csharp // src/Processing.Retry/RetryResult.cs public record RetryResult { @@ -99,82 +39,103 @@ public record RetryResult } ``` -### Delay Calculation +--- -``` -delay = min(InitialDelayMs × BackoffMultiplier^(attempt-1), MaxDelayMs) -jitter = ±20% random variation (when UseJitter = true) -``` +## Exercises -Jitter prevents the **thundering herd** problem where many retrying clients hit the same service simultaneously at the exact same backoff intervals. +### Exercise 1: Success on first attempt ---- +```csharp +var policy = CreatePolicy(); -## Scalability Dimension +var result = await policy.ExecuteAsync( + _ => Task.FromResult(42), CancellationToken.None); -The retry policy is **per-operation, in-process** — each replica independently retries its own failed operations. No coordination between replicas is needed. However, aggressive retry settings (high `MaxAttempts`, low `InitialDelayMs`) can amplify load on a struggling downstream service. The `MaxDelayMs` cap and jitter work together to spread retry traffic and protect downstream services. +Assert.That(result.IsSucceeded, Is.True); +Assert.That(result.Attempts, Is.EqualTo(1)); +Assert.That(result.Result, Is.EqualTo(42)); +Assert.That(result.LastException, Is.Null); +``` ---- +### Exercise 2: Retry succeeds after transient failure -## Atomicity Dimension +```csharp +var policy = CreatePolicy(maxAttempts: 5); +var callCount = 0; -When all retry attempts are exhausted (`IsSucceeded = false`), the message should be routed to the **Dead Letter Queue** (Tutorial 25) with `DeadLetterReason.MaxRetriesExceeded`. Non-retryable errors (e.g. `ValidationFailed`, deserialization errors, schema mismatches) should be detected early and sent immediately to the DLQ without consuming retry attempts. The `LastException` and `Attempts` fields in `RetryResult` provide full diagnostic context. +var result = await policy.ExecuteAsync( + _ => + { + callCount++; + if (callCount < 3) + throw new InvalidOperationException("transient"); + return Task.FromResult("ok"); + }, + CancellationToken.None); + +Assert.That(result.IsSucceeded, Is.True); +Assert.That(result.Attempts, Is.EqualTo(3)); +Assert.That(result.Result, Is.EqualTo("ok")); +``` ---- +### Exercise 3: All attempts exhausted returns failure with exception -## Lab +```csharp +var policy = CreatePolicy(maxAttempts: 3); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial24/Lab.cs`](../tests/TutorialLabs/Tutorial24/Lab.cs) +var result = await policy.ExecuteAsync( + _ => throw new TimeoutException("always fails"), + CancellationToken.None); -**Objective:** Calculate exponential backoff delays, analyze why jitter is critical for **scalable** retry under thundering-herd conditions, and design a retry classification strategy. +Assert.That(result.IsSucceeded, Is.False); +Assert.That(result.Attempts, Is.EqualTo(3)); +Assert.That(result.LastException, Is.TypeOf()); +Assert.That(result.Result, Is.Null); +``` -### Step 1: Calculate Backoff Delays +### Exercise 4: Void overload retries and fails -With `MaxAttempts = 4`, `InitialDelayMs = 500`, `BackoffMultiplier = 2.0`, and `UseJitter = false`, calculate the delay before each retry: +```csharp +var policy = CreatePolicy(maxAttempts: 2); -| Attempt | Delay Formula | Delay | -|---------|--------------|-------| -| 1 (first retry) | 500 × 2⁰ | 500ms | -| 2 | 500 × 2¹ | ? | -| 3 | 500 × 2² | ? | -| 4 | 500 × 2³ | ? | +var result = await policy.ExecuteAsync( + _ => throw new IOException("disk full"), + CancellationToken.None); -What is the total maximum wait time across all retries? Open `src/Processing.Retry/ExponentialBackoffRetryPolicy.cs` to verify the formula. +Assert.That(result.IsSucceeded, Is.False); +Assert.That(result.Attempts, Is.EqualTo(2)); +Assert.That(result.LastException, Is.TypeOf()); +``` -### Step 2: Analyze the Thundering Herd Problem +### Exercise 5: Cancellation is propagated -100 consumers lose connection to a database. All retry at the same exponential intervals (no jitter). Draw what happens: +```csharp +var policy = CreatePolicy(maxAttempts: 5); +using var cts = new CancellationTokenSource(); +cts.Cancel(); +Assert.ThrowsAsync( + () => policy.ExecuteAsync( + _ => Task.FromResult(1), cts.Token)); ``` -t=0s: [100 consumers all fail] -t=500ms: [100 consumers all retry simultaneously] → database overwhelmed again -t=1000ms: [100 consumers all retry simultaneously] → database overwhelmed again -``` - -Now add jitter: each consumer randomizes its delay within ±50%. Explain: -- How does jitter spread the retry load over time? -- Why is this critical for **system-level scalability** during recovery? -- What is the relationship between jitter and the database's recovery time? -### Step 3: Design Retry Classification +--- -Not all errors are retryable. Design a classification strategy: +## Lab -| Error Type | Retryable? | Action | -|-----------|-----------|--------| -| HTTP 503 (Service Unavailable) | Yes | Exponential backoff | -| HTTP 400 (Bad Request) | No | Immediate DLQ | -| `JsonException` (deserialization) | No | Immediate DLQ | -| `TimeoutException` (network) | Yes | ? | -| Schema validation failure | No | ? | +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial24/Lab.cs`](../tests/TutorialLabs/Tutorial24/Lab.cs) -Why is fast-failing non-retryable errors critical for **pipeline throughput**? What happens if you retry a `JsonException` 4 times before giving up? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial24.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial24/Exam.cs`](../tests/TutorialLabs/Tutorial24/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial24.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/25-dead-letter-queue.md b/EnterpriseIntegrationPlatform/tutorials/25-dead-letter-queue.md index 6682e94..9860331 100644 --- a/EnterpriseIntegrationPlatform/tutorials/25-dead-letter-queue.md +++ b/EnterpriseIntegrationPlatform/tutorials/25-dead-letter-queue.md @@ -1,44 +1,10 @@ # Tutorial 25 — Dead Letter Queue -## What You'll Learn - -- The EIP Dead Letter Channel pattern as the safety net for unprocessable messages -- How `IDeadLetterPublisher` wraps failed messages in a `DeadLetterEnvelope` -- The `DeadLetterReason` enum: MaxRetriesExceeded, PoisonMessage, ProcessingTimeout, ValidationFailed, UnroutableMessage, MessageExpired -- How `MessageExpirationChecker` detects and routes expired messages -- Admin API capabilities: inspect, replay, and discard dead-lettered messages +Capture unprocessable messages with full diagnostic context so they can be inspected, replayed, or discarded. --- -## EIP Pattern: Dead Letter Channel - -> *"When a messaging system determines that it cannot or should not deliver a message, it may elect to move the message to a Dead Letter Channel."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌────────────┐ ┌─────────────────┐ - │ Processing │──X──│ Dead Letter │ - │ Pipeline │ │ Publisher │ - └────────────┘ └────────┬────────┘ - │ (failed) │ - │ ▼ - │ ┌───────────────────┐ - │ │ Dead Letter Queue │ - │ │ (inspect / replay │ - │ │ / discard) │ - │ └───────────────────┘ - │ - ▼ (success) - Downstream Topic -``` - -Every message that cannot be processed — after retries, due to validation failure, expiration, or poison content — is captured with full diagnostic context so it can be inspected, fixed, and replayed. - ---- - -## Platform Implementation - -### IDeadLetterPublisher +## Key Types ```csharp // src/Processing.DeadLetter/IDeadLetterPublisher.cs @@ -51,11 +17,7 @@ public interface IDeadLetterPublisher int attemptCount, CancellationToken ct); } -``` - -### DeadLetterEnvelope -```csharp // src/Processing.DeadLetter/DeadLetterEnvelope.cs public record DeadLetterEnvelope { @@ -65,13 +27,7 @@ public record DeadLetterEnvelope public required DateTimeOffset FailedAt { get; init; } public required int AttemptCount { get; init; } } -``` - -The `DeadLetterEnvelope` preserves the **complete original message** alongside fault details. Nothing is lost — an operator can inspect the exact payload and headers that caused the failure. -### DeadLetterReason - -```csharp // src/Processing.DeadLetter/DeadLetterReason.cs public enum DeadLetterReason { @@ -82,101 +38,167 @@ public enum DeadLetterReason UnroutableMessage, MessageExpired } -``` - -| Reason | Trigger | -|--------|---------| -| `MaxRetriesExceeded` | All retry attempts exhausted (Tutorial 24) | -| `PoisonMessage` | Message causes repeated crashes — immediate DLQ | -| `ProcessingTimeout` | Processing exceeded the allowed time window | -| `ValidationFailed` | Schema or business rule validation failed | -| `UnroutableMessage` | No routing rule matched and no default topic configured | -| `MessageExpired` | `IntegrationEnvelope.ExpiresAt` is in the past | - -### MessageExpirationChecker -```csharp // src/Processing.DeadLetter/MessageExpirationChecker.cs public sealed class MessageExpirationChecker : IMessageExpirationChecker { public async Task CheckAndRouteIfExpiredAsync( IntegrationEnvelope envelope, - CancellationToken cancellationToken = default) - { - if (!envelope.ExpiresAt.HasValue) return false; - if (_timeProvider.GetUtcNow() <= envelope.ExpiresAt.Value) return false; - - await _deadLetterPublisher.PublishAsync( - envelope, DeadLetterReason.MessageExpired, - $"Message expired at {envelope.ExpiresAt.Value:O}. Current time: {now:O}.", - 0, cancellationToken); - return true; - } + CancellationToken cancellationToken = default); } ``` -Uses `TimeProvider` for testability — unit tests can inject a fake clock. - --- -## Scalability Dimension +## Exercises -The DLQ publisher is **stateless** — it wraps the envelope and publishes to the dead-letter topic. Any replica can dead-letter any message. The dead-letter topic itself is a standard broker topic that can be partitioned and replicated for high availability. The Admin API reads from this topic to support **inspect** (view dead-lettered messages), **replay** (re-publish to the original topic), and **discard** (acknowledge and remove). +### Exercise 1: Publish routes to configured dead-letter topic ---- +```csharp +var producer = Substitute.For(); +var options = Options.Create(new DeadLetterOptions +{ + DeadLetterTopic = "dlq-topic", +}); -## Atomicity Dimension +var publisher = new DeadLetterPublisher(producer, options); -Dead-lettering is the **last resort** — it runs only after all retries are exhausted or the error is non-retryable. The publisher writes the `DeadLetterEnvelope` to the DLQ topic **before** Acking the source message. If the DLQ publish itself fails, the source message is Nacked and redelivered. This means a message can only leave the system through two paths: successful processing or the DLQ. **No message is ever silently lost.** +var envelope = IntegrationEnvelope.Create( + "bad-payload", "OrderSvc", "order.created"); ---- +await publisher.PublishAsync( + envelope, + DeadLetterReason.MaxRetriesExceeded, + "Failed after 3 retries", + attemptCount: 3, + CancellationToken.None); -## Lab +await producer.Received(1).PublishAsync( + Arg.Any>>(), + "dlq-topic", + Arg.Any()); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial25/Lab.cs`](../tests/TutorialLabs/Tutorial25/Lab.cs) +### Exercise 2: Empty topic throws InvalidOperationException -**Objective:** Trace the Dead Letter Queue lifecycle from failure to replay, analyze how the DLQ preserves **zero message loss atomicity**, and design an operational replay workflow. +```csharp +var producer = Substitute.For(); +var options = Options.Create(new DeadLetterOptions +{ + DeadLetterTopic = "", +}); + +var publisher = new DeadLetterPublisher(producer, options); +var envelope = IntegrationEnvelope.Create( + "data", "Svc", "type"); + +Assert.ThrowsAsync(() => + publisher.PublishAsync( + envelope, + DeadLetterReason.PoisonMessage, + "error", + 1, + CancellationToken.None)); +``` -### Step 1: Trace an Expired Message to the DLQ +### Exercise 3: DeadLetterEnvelope record construction -A message has `ExpiresAt = 2024-01-15T10:00:00Z` and the current time is `2024-01-15T10:00:01Z`. Open `src/Processing.DeadLetter/MessageExpirationChecker.cs` and trace: +```csharp +var original = IntegrationEnvelope.Create( + "payload", "Svc", "type"); -1. `CheckAndRouteIfExpiredAsync` detects expiration — what `DeadLetterReason` is used? -2. What information is logged? (hint: expiry time and current time) -3. Where does the complete original envelope end up? +var dlEnvelope = new DeadLetterEnvelope +{ + OriginalEnvelope = original, + Reason = DeadLetterReason.ValidationFailed, + ErrorMessage = "Schema mismatch", + FailedAt = DateTimeOffset.UtcNow, + AttemptCount = 2, +}; + +Assert.That(dlEnvelope.OriginalEnvelope.Payload, Is.EqualTo("payload")); +Assert.That(dlEnvelope.Reason, Is.EqualTo(DeadLetterReason.ValidationFailed)); +Assert.That(dlEnvelope.ErrorMessage, Is.EqualTo("Schema mismatch")); +Assert.That(dlEnvelope.AttemptCount, Is.EqualTo(2)); +``` + +### Exercise 4: Publisher preserves CorrelationId on wrapper + +```csharp +IntegrationEnvelope>? captured = null; +var producer = Substitute.For(); +producer + .PublishAsync( + Arg.Do>>(e => captured = e), + Arg.Any(), + Arg.Any()) + .Returns(Task.CompletedTask); + +var options = Options.Create(new DeadLetterOptions +{ + DeadLetterTopic = "dlq", +}); -Verify that the **entire original envelope** is preserved in `DeadLetterEnvelope` — not just error details. +var publisher = new DeadLetterPublisher(producer, options); -### Step 2: Design an Operational Replay Workflow +var originalCorrelationId = Guid.NewGuid(); +var envelope = IntegrationEnvelope.Create( + "data", "Svc", "type", correlationId: originalCorrelationId); -A message fails validation (`DeadLetterReason.ValidationFailed`). An operator fixes the downstream schema. Design the replay flow: +await publisher.PublishAsync( + envelope, DeadLetterReason.MessageExpired, "expired", 0, CancellationToken.None); +Assert.That(captured, Is.Not.Null); +Assert.That(captured!.CorrelationId, Is.EqualTo(originalCorrelationId)); ``` -1. Operator queries DLQ via Admin API: GET /api/deadletter?reason=ValidationFailed -2. Operator reviews the original envelope and error details -3. Operator triggers replay: POST /api/deadletter/{id}/replay -4. Platform re-publishes the original envelope to its original topic -5. Message re-enters the pipeline from the beginning + +### Exercise 5: Publisher uses custom source when configured + +```csharp +IntegrationEnvelope>? captured = null; +var producer = Substitute.For(); +producer + .PublishAsync( + Arg.Do>>(e => captured = e), + Arg.Any(), + Arg.Any()) + .Returns(Task.CompletedTask); + +var options = Options.Create(new DeadLetterOptions +{ + DeadLetterTopic = "dlq", + Source = "DLQ-Publisher", +}); + +var publisher = new DeadLetterPublisher(producer, options); + +var envelope = IntegrationEnvelope.Create( + "data", "OriginalSvc", "type"); + +await publisher.PublishAsync( + envelope, DeadLetterReason.UnroutableMessage, "no route", 1, CancellationToken.None); + +Assert.That(captured, Is.Not.Null); +Assert.That(captured!.Source, Is.EqualTo("DLQ-Publisher")); ``` -What **atomicity** guarantees must the replay provide? (hint: replay must either fully re-publish or fail cleanly — no partial replays) +--- -### Step 3: Categorize DLQ Reasons and Operational Response +## Lab -| DLQ Reason | Cause | Operational Response | Can Auto-Replay? | -|-----------|-------|---------------------|-------------------| -| `MessageExpired` | TTL exceeded | Review TTL settings | No — stale data | -| `ValidationFailed` | Schema mismatch | Fix schema → replay | Yes | -| `MaxRetriesExceeded` | Transient failures | Investigate root cause → replay | Maybe | -| `PoisonMessage` | Non-retryable error | Manual intervention | No | +> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial25/Lab.cs`](../tests/TutorialLabs/Tutorial25/Lab.cs) -Why is preserving the complete original envelope critical for DLQ operations? What would an operator lose if only the error message was stored? +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial25.Lab" +``` ## Exam > 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial25/Exam.cs`](../tests/TutorialLabs/Tutorial25/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test --filter "FullyQualifiedName~TutorialLabs.Tutorial25.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/26-message-replay.md b/EnterpriseIntegrationPlatform/tutorials/26-message-replay.md index 293e3e2..6e19c6f 100644 --- a/EnterpriseIntegrationPlatform/tutorials/26-message-replay.md +++ b/EnterpriseIntegrationPlatform/tutorials/26-message-replay.md @@ -1,43 +1,8 @@ # Tutorial 26 — Message Replay -## What You'll Learn +Replay previously processed messages from the replay store with filtering and deduplication. -- How `IMessageReplayer` enables selective re-processing of historical messages -- How `IMessageReplayStore` persists replay-eligible messages for audit and reprocessing -- The `ReplayFilter` value object for targeting messages by timestamp range, CorrelationId, or MessageType -- The `ReplayResult` record with replayed, skipped, and failed counts -- The `ReplayId` header added to every replayed message for audit-trail separation - ---- - -## EIP Pattern: Message Replay - -> *"Replay allows an operator to re-inject previously processed messages into the pipeline — essential for disaster recovery, reprocessing after bug fixes, and audit verification."* - -``` - ┌────────────────┐ ┌──────────────────┐ - │ Replay Store │◀──────│ Original │ - │ (persisted │ │ Processing │ - │ messages) │ └──────────────────┘ - └───────┬────────┘ - │ ReplayFilter - ▼ - ┌────────────────┐ ┌──────────────────┐ - │ Message │──────▶│ Pipeline │ - │ Replayer │ │ (re-ingested) │ - └────────────────┘ └──────────────────┘ - │ - ▼ - ReplayId header injected -``` - -Messages are stored as they flow through the pipeline. When a replay is requested, the `ReplayFilter` selects a subset, and each message is re-published with a unique `ReplayId` header so downstream consumers can distinguish replayed messages from originals. - ---- - -## Platform Implementation - -### IMessageReplayer +## Key Types ```csharp // src/Processing.Replay/IMessageReplayer.cs @@ -47,8 +12,6 @@ public interface IMessageReplayer } ``` -### IMessageReplayStore - ```csharp // src/Processing.Replay/IMessageReplayStore.cs public interface IMessageReplayStore @@ -58,8 +21,6 @@ public interface IMessageReplayStore } ``` -### ReplayFilter - ```csharp // src/Processing.Replay/ReplayFilter.cs public record ReplayFilter @@ -71,14 +32,6 @@ public record ReplayFilter } ``` -| Filter Property | Usage | -|-----------------|-------| -| `FromTimestamp` / `ToTimestamp` | Date-range replay — e.g. replay all messages from the last hour | -| `CorrelationId` | Replay a single business transaction (typed as `Guid?`) | -| `MessageType` | Replay all messages of a specific type after a schema fix | - -### ReplayResult - ```csharp // src/Processing.Replay/ReplayResult.cs public record ReplayResult @@ -91,73 +44,165 @@ public record ReplayResult } ``` -Every replayed message receives a `ReplayId` header (a GUID) linking it back to the replay operation. This separates replayed traffic from live traffic in dashboards and audit logs. +## Exercises ---- +### 1. Replay — AllMessagesReplayed CountsAreCorrect -## Scalability Dimension +```csharp +var store = new InMemoryMessageReplayStore(); +var producer = Substitute.For(); -The replay store is **read-heavy** — writes happen once per message, but replays can query millions of records. Production stores should support indexed queries on `CorrelationId`, `MessageType`, and `CreatedAt`. The replayer itself is stateless: it reads from the store, publishes to the broker, and records the `ReplayResult`. Multiple replay operations can run concurrently because each gets a unique `ReplayId`. +var options = Options.Create(new ReplayOptions +{ + SourceTopic = "orders", + TargetTopic = "orders-replay", + MaxMessages = 100, +}); ---- +var replayer = new MessageReplayer( + store, producer, options, NullLogger.Instance); -## Atomicity Dimension +var env1 = IntegrationEnvelope.Create("p1", "Svc", "order.created"); +var env2 = IntegrationEnvelope.Create("p2", "Svc", "order.created"); +await store.StoreForReplayAsync(env1, "orders", CancellationToken.None); +await store.StoreForReplayAsync(env2, "orders", CancellationToken.None); -Replay re-publishes messages to the **same ingress topic** they originally entered. This means all validation, routing, and transformation rules apply again — the message is not injected halfway through the pipeline. If a replay fails mid-batch, `ReplayResult.ReplayedCount`, `SkippedCount`, and `FailedCount` together account for every message matched by the filter. The `ReplayId` header ensures idempotent consumers can detect and deduplicate replayed messages. +var result = await replayer.ReplayAsync(new ReplayFilter(), CancellationToken.None); ---- +Assert.That(result.ReplayedCount, Is.EqualTo(2)); +Assert.That(result.SkippedCount, Is.EqualTo(0)); +Assert.That(result.FailedCount, Is.EqualTo(0)); +``` -## Lab +### 2. Replay — PublishesToConfiguredTargetTopic + +```csharp +var store = new InMemoryMessageReplayStore(); +var producer = Substitute.For(); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial26/Lab.cs`](../tests/TutorialLabs/Tutorial26/Lab.cs) +var options = Options.Create(new ReplayOptions +{ + SourceTopic = "events", + TargetTopic = "events-replay", + MaxMessages = 10, +}); -**Objective:** Design a message replay operation for a production incident, analyze how the `ReplayId` header prevents duplicate processing, and evaluate replay store **scalability** requirements. +var replayer = new MessageReplayer( + store, producer, options, NullLogger.Instance); -### Step 1: Design a Time-Window Replay +var env = IntegrationEnvelope.Create("data", "Svc", "event.fired"); +await store.StoreForReplayAsync(env, "events", CancellationToken.None); -An operator discovers a bug in the content enricher that corrupted messages between 09:00 and 09:30 UTC on January 15th. Write the `ReplayFilter`: +await replayer.ReplayAsync(new ReplayFilter(), CancellationToken.None); + +await producer.Received(1).PublishAsync( + Arg.Any>(), + "events-replay", + Arg.Any()); +``` + +### 3. Replay — FilterByMessageType OnlyMatchingMessagesReplayed ```csharp -var filter = new ReplayFilter +var store = new InMemoryMessageReplayStore(); +var producer = Substitute.For(); + +var options = Options.Create(new ReplayOptions { - FromTimestamp = DateTimeOffset.Parse("2024-01-15T09:00:00Z"), - ToTimestamp = DateTimeOffset.Parse("2024-01-15T09:30:00Z") -}; -// Topic is passed as a separate parameter to the replay store: -// await replayStore.GetMessagesForReplayAsync("eip.orders.enriched", filter, ...); + SourceTopic = "topic", + TargetTopic = "topic-replay", + MaxMessages = 100, +}); + +var replayer = new MessageReplayer( + store, producer, options, NullLogger.Instance); + +var match = IntegrationEnvelope.Create("m", "Svc", "order.created"); +var noMatch = IntegrationEnvelope.Create("n", "Svc", "invoice.created"); +await store.StoreForReplayAsync(match, "topic", CancellationToken.None); +await store.StoreForReplayAsync(noMatch, "topic", CancellationToken.None); + +var filter = new ReplayFilter { MessageType = "order.created" }; +var result = await replayer.ReplayAsync(filter, CancellationToken.None); + +Assert.That(result.ReplayedCount, Is.EqualTo(1)); ``` -Open `src/Processing.Replay/MessageReplayer.cs` and trace: How does the replayer iterate over stored messages? What happens to messages that don't match the filter? +### 4. Replay — SkipAlreadyReplayed SkipsMessagesWithReplayIdHeader + +```csharp +var store = new InMemoryMessageReplayStore(); +var producer = Substitute.For(); + +var options = Options.Create(new ReplayOptions +{ + SourceTopic = "src", + TargetTopic = "tgt", + MaxMessages = 100, + SkipAlreadyReplayed = true, +}); + +var replayer = new MessageReplayer( + store, producer, options, NullLogger.Instance); -### Step 2: Analyze the ReplayId Header for Atomicity +var alreadyReplayed = new IntegrationEnvelope +{ + MessageId = Guid.NewGuid(), + CorrelationId = Guid.NewGuid(), + Timestamp = DateTimeOffset.UtcNow, + Source = "Svc", + MessageType = "type", + Payload = "data", + Metadata = new Dictionary + { + [MessageHeaders.ReplayId] = Guid.NewGuid().ToString(), + }, +}; +var fresh = IntegrationEnvelope.Create("fresh", "Svc", "type"); -The platform injects a `ReplayId` header into replayed messages. Explain why: +await store.StoreForReplayAsync(alreadyReplayed, "src", CancellationToken.None); +await store.StoreForReplayAsync(fresh, "src", CancellationToken.None); -1. Without `ReplayId` — downstream consumers process the message as if it's new → **duplicate side effects** (e.g., double billing) -2. With `ReplayId` — consumers can detect replays and apply **idempotent** processing -3. How does `ReplayId` interact with `MessageId`? (the original `MessageId` is preserved for correlation) +var result = await replayer.ReplayAsync(new ReplayFilter(), CancellationToken.None); -Design a consumer that checks for `ReplayId` and skips already-processed messages using a deduplication store. +Assert.That(result.ReplayedCount, Is.EqualTo(1)); +Assert.That(result.SkippedCount, Is.EqualTo(1)); +``` -### Step 3: Evaluate Replay Store Scalability +### 5. Replay — EmptySourceTopic ThrowsInvalidOperationException -A production system processes 10 million messages/day. Design the replay store requirements: +```csharp +var store = new InMemoryMessageReplayStore(); +var producer = Substitute.For(); -| Requirement | Value | Justification | -|------------|-------|---------------| -| Storage per message | ~2KB (envelope) | Full envelope for accurate replay | -| Daily storage | ~20GB | 10M × 2KB | -| Retention period | 30 days | Regulatory and operational needs | -| Total storage | ~600GB | 30 × 20GB | -| Query performance | < 100ms for time-range | Fast incident response | +var options = Options.Create(new ReplayOptions +{ + SourceTopic = "", + TargetTopic = "tgt", +}); + +var replayer = new MessageReplayer( + store, producer, options, NullLogger.Instance); -What storage technology would you recommend? (hint: time-series databases, object storage with indexing) +Assert.ThrowsAsync( + () => replayer.ReplayAsync(new ReplayFilter(), CancellationToken.None)); +``` + +## Lab + +Run the full lab: [`tests/TutorialLabs/Tutorial26/Lab.cs`](../tests/TutorialLabs/Tutorial26/Lab.cs) + +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial26.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial26/Exam.cs`](../tests/TutorialLabs/Tutorial26/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial26/Exam.cs`](../tests/TutorialLabs/Tutorial26/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial26.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/27-resequencer.md b/EnterpriseIntegrationPlatform/tutorials/27-resequencer.md index 499b435..061e639 100644 --- a/EnterpriseIntegrationPlatform/tutorials/27-resequencer.md +++ b/EnterpriseIntegrationPlatform/tutorials/27-resequencer.md @@ -1,39 +1,8 @@ # Tutorial 27 — Resequencer -## What You'll Learn +Reorder out-of-sequence messages back into their original sequence. -- The EIP Resequencer pattern for restoring message order from out-of-order delivery -- How `IResequencer` and `MessageResequencer` buffer and release messages in sequence -- Grouping by `CorrelationId` with ordering by `SequenceNumber` -- Timeout-based release to prevent indefinite buffering -- The `ActiveSequenceCount` metric for monitoring open sequences -- `ResequencerOptions` for concurrent-sequence limits and release timeout - ---- - -## EIP Pattern: Resequencer - -> *"A Resequencer can receive a stream of messages that may not arrive in order and reorder them so that they are published to the output channel in order."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - Msg #3 ──▶ ┌──────────────────┐ - Msg #1 ──▶ │ Resequencer │──▶ Msg #1, Msg #2, Msg #3 - Msg #2 ──▶ │ (buffer + │ - │ sort + release)│ - └──────────────────┘ - │ - Timeout triggers - partial release -``` - -Distributed systems deliver messages out of order. The Resequencer collects messages sharing a `CorrelationId`, sorts them by `SequenceNumber`, and releases them in order once the sequence is complete — or when a timeout fires. - ---- - -## Platform Implementation - -### IResequencer +## Key Types ```csharp // src/Processing.Resequencer/IResequencer.cs @@ -45,20 +14,6 @@ public interface IResequencer } ``` -`Accept` is synchronous — it accepts a single message and returns zero or more messages ready for release. If the submitted message completes a contiguous run, the entire run is returned. If gaps remain, an empty list is returned and the message is buffered. `ReleaseOnTimeout` forces release of all buffered messages for a given `CorrelationId` when the timeout fires. - -### MessageResequencer (concrete) - -The `MessageResequencer` class implements `IResequencer`. Internally it maintains a `ConcurrentDictionary` keyed by `CorrelationId`. `SequenceBuffer` is a private inner class that uses a `ConcurrentDictionary` for thread-safe storage and `OrderBy` for sequenced release. Each entry tracks: - -1. **Expected next sequence number** — starts at 1 -2. **Buffered messages** — sorted by `SequenceNumber` -3. **Last activity timestamp** — for timeout detection - -When a message arrives whose `SequenceNumber` equals the expected value, the resequencer releases it and any contiguous successors. - -### ResequencerOptions - ```csharp // src/Processing.Resequencer/ResequencerOptions.cs public sealed class ResequencerOptions @@ -68,70 +23,103 @@ public sealed class ResequencerOptions } ``` -| Option | Purpose | -|--------|---------| -| `ReleaseTimeout` | How long to wait for missing sequence numbers before releasing buffered messages (default 30 s) | -| `MaxConcurrentSequences` | Maximum number of distinct sequences tracked concurrently (default 10,000) | - -### Timeout-Based Release +## Exercises -A background timer scans active sequences. When `ReleaseTimeout` elapses since the last message arrived for a correlation, the resequencer calls `ReleaseOnTimeout` to release whatever is buffered in sequence order. +### 1. Accept — CompleteSequenceInOrder ReleasesAllMessages ---- - -## Scalability Dimension +```csharp +var resequencer = CreateResequencer(); +var correlationId = Guid.NewGuid(); + +var r1 = resequencer.Accept(MakeSequenced(correlationId, 0, 3)); +var r2 = resequencer.Accept(MakeSequenced(correlationId, 1, 3)); +var r3 = resequencer.Accept(MakeSequenced(correlationId, 2, 3)); + +// Only the last accept should release all 3 +Assert.That(r1, Is.Empty); +Assert.That(r2, Is.Empty); +Assert.That(r3, Has.Count.EqualTo(3)); +Assert.That(r3[0].Payload, Is.EqualTo("msg-0")); +Assert.That(r3[1].Payload, Is.EqualTo("msg-1")); +Assert.That(r3[2].Payload, Is.EqualTo("msg-2")); +``` -The resequencer is **stateful** — it must buffer messages until a sequence is complete. This limits horizontal scaling because all messages for a given `CorrelationId` must reach the **same instance**. Use broker-level partition affinity (partition by `CorrelationId`) to ensure this. `ActiveSequenceCount` exposes the current memory pressure so auto-scalers can react before buffers overflow. +### 2. Accept — OutOfOrder ReleasesInCorrectOrder ---- +```csharp +var resequencer = CreateResequencer(); +var correlationId = Guid.NewGuid(); + +var r1 = resequencer.Accept(MakeSequenced(correlationId, 2, 3)); +var r2 = resequencer.Accept(MakeSequenced(correlationId, 0, 3)); +var r3 = resequencer.Accept(MakeSequenced(correlationId, 1, 3)); + +Assert.That(r1, Is.Empty); +Assert.That(r2, Is.Empty); +Assert.That(r3, Has.Count.EqualTo(3)); +Assert.That(r3[0].Payload, Is.EqualTo("msg-0")); +Assert.That(r3[1].Payload, Is.EqualTo("msg-1")); +Assert.That(r3[2].Payload, Is.EqualTo("msg-2")); +``` -## Atomicity Dimension +### 3. Accept — IncompleteSequence BuffersAndReturnsEmpty -Messages are **Acked only after successful release** to the downstream topic. If the resequencer crashes, buffered messages are redelivered by the broker (they were never Acked). On restart, the resequencer rebuilds its buffer from redelivered messages. The `ReleaseTimeout` acts as a safety valve — it ensures no message is buffered indefinitely, preventing memory leaks and silent message loss. +```csharp +var resequencer = CreateResequencer(); +var correlationId = Guid.NewGuid(); ---- +var result = resequencer.Accept(MakeSequenced(correlationId, 1, 3)); -## Lab - -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial27/Lab.cs`](../tests/TutorialLabs/Tutorial27/Lab.cs) +Assert.That(result, Is.Empty); +Assert.That(resequencer.ActiveSequenceCount, Is.EqualTo(1)); +``` -**Objective:** Trace the Resequencer's buffering and release logic, analyze ordering guarantees for **atomic** batch processing, and design for partition-aware scaling. +### 4. Accept — DuplicateSequenceNumber IsIgnored -### Step 1: Trace Out-of-Order Arrival +```csharp +var resequencer = CreateResequencer(); +var correlationId = Guid.NewGuid(); -Three messages arrive for `CorrelationId = "order-42"` in this order: #3, #1, #2. Open `src/Processing.Resequencer/` and trace each `Accept` call: +resequencer.Accept(MakeSequenced(correlationId, 0, 2)); +var dup = resequencer.Accept(MakeSequenced(correlationId, 0, 2)); -| Arrival | SequenceNumber | Buffered? | Released? | Why? | -|---------|---------------|-----------|-----------|------| -| 1st | 3 | Yes | No | Waiting for #1 | -| 2nd | 1 | — | Released: #1 | Next expected | -| 3rd | 2 | — | Released: #2, then #3 | Completes the sequence | +Assert.That(dup, Is.Empty); +// Still waiting for seq 1 +Assert.That(resequencer.ActiveSequenceCount, Is.EqualTo(1)); +``` -Verify your trace against the actual implementation. +### 5. ReleaseOnTimeout — IncompleteSequence ReturnsBufferedInOrder -### Step 2: Handle Gaps with Timeout +```csharp +var resequencer = CreateResequencer(); +var correlationId = Guid.NewGuid(); -A sequence has messages #1, #2, #4 buffered, but #3 never arrives. After `ReleaseTimeout` fires: +resequencer.Accept(MakeSequenced(correlationId, 2, 5)); +resequencer.Accept(MakeSequenced(correlationId, 0, 5)); -1. What does `ReleaseOnTimeout` return? (hint: #1 and #2 are released, #4 is released with a gap marker) -2. Is the gap reported for downstream awareness? -3. How does this design prevent indefinite buffering — critical for **system scalability** under high message volumes? +var released = resequencer.ReleaseOnTimeout(correlationId); -Design an alerting strategy for gap detection: when should the operations team be notified? +Assert.That(released, Has.Count.EqualTo(2)); +Assert.That(released[0].Payload, Is.EqualTo("msg-0")); +Assert.That(released[1].Payload, Is.EqualTo("msg-2")); +Assert.That(resequencer.ActiveSequenceCount, Is.EqualTo(0)); +``` -### Step 3: Partition-Aware Resequencing +## Lab -All messages for a `CorrelationId` must be routed to the same resequencer instance. Explain: +Run the full lab: [`tests/TutorialLabs/Tutorial27/Lab.cs`](../tests/TutorialLabs/Tutorial27/Lab.cs) -- What broker feature enables this? (hint: Kafka partition keys, NATS subject-based routing) -- What happens if messages for the same `CorrelationId` land on different resequencer instances? -- How does partition-key routing enable **horizontal scaling** — each instance handles a subset of `CorrelationId`s independently? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial27.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial27/Exam.cs`](../tests/TutorialLabs/Tutorial27/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial27/Exam.cs`](../tests/TutorialLabs/Tutorial27/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial27.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/28-competing-consumers.md b/EnterpriseIntegrationPlatform/tutorials/28-competing-consumers.md index 8ea804f..8b87ff2 100644 --- a/EnterpriseIntegrationPlatform/tutorials/28-competing-consumers.md +++ b/EnterpriseIntegrationPlatform/tutorials/28-competing-consumers.md @@ -1,44 +1,8 @@ # Tutorial 28 — Competing Consumers -## What You'll Learn +Scale message processing horizontally with competing consumer instances. -- The EIP Competing Consumers pattern for parallel message processing -- How `CompetingConsumerOrchestrator` (a `BackgroundService`) manages consumer lifecycles -- `IConsumerLagMonitor` for measuring how far behind consumers are -- `IConsumerScaler` for adding or removing consumer instances -- `IBackpressureSignal` for communicating overload upstream -- Auto-scaling based on consumer lag with cooldown to prevent flapping -- Why this is the **Scalability pillar centerpiece** of the platform - ---- - -## EIP Pattern: Competing Consumers - -> *"Create multiple consumers on a single Point-to-Point Channel so that the consumers can process multiple messages concurrently."* -> — Gregor Hohpe & Bobby Woolf, *Enterprise Integration Patterns* - -``` - ┌──────────────┐ - ┌───▶│ Consumer 1 │ - ┌──────────┐ │ └──────────────┘ - │ Broker │──────┤ ┌──────────────┐ - │ Topic │ ├───▶│ Consumer 2 │ - └──────────┘ │ └──────────────┘ - │ ┌──────────────┐ - └───▶│ Consumer N │ - └──────────────┘ - │ - Orchestrator scales N - based on lag + backpressure -``` - -Multiple consumers read from the same topic. The broker ensures each message is delivered to exactly one consumer in the group. The orchestrator monitors lag and adjusts the consumer count dynamically. - ---- - -## Platform Implementation - -### CompetingConsumerOrchestrator +## Key Types ```csharp // src/Processing.CompetingConsumers/CompetingConsumerOrchestrator.cs @@ -109,14 +73,6 @@ public sealed class CompetingConsumerOrchestrator : BackgroundService } ``` -The orchestrator runs as a hosted `BackgroundService`. On each evaluation cycle it reads the consumer lag via `GetLagAsync`, compares against thresholds, and calls `ScaleAsync` with the desired consumer count. Key features: - -- **Backpressure signaling** — when at max capacity, signals backpressure to upstream producers -- **Cooldown guard** — prevents scaling flapping with a configurable cooldown period -- **Backpressure-aware scale-down** — won't scale down while backpressure is active - -### IConsumerLagMonitor - ```csharp // src/Processing.CompetingConsumers/IConsumerLagMonitor.cs public interface IConsumerLagMonitor @@ -134,10 +90,6 @@ public record ConsumerLagInfo( DateTimeOffset Timestamp); ``` -> **Note:** `ConsumerLagInfo` tracks lag per consumer group and topic. The active consumer count is available separately via `IConsumerScaler.CurrentCount`. - -### IConsumerScaler - ```csharp // src/Processing.CompetingConsumers/IConsumerScaler.cs public interface IConsumerScaler @@ -147,8 +99,6 @@ public interface IConsumerScaler } ``` -### IBackpressureSignal - ```csharp // src/Processing.CompetingConsumers/IBackpressureSignal.cs public interface IBackpressureSignal @@ -159,87 +109,106 @@ public interface IBackpressureSignal } ``` -When `IsBackpressured` is true, the orchestrator pauses scale-down and can signal upstream producers (via broker flow control or HTTP 429) to slow ingestion. +## Exercises -### CompetingConsumerOptions +### 1. BackpressureSignal — SignalAndRelease TogglesCorrectly ```csharp -// src/Processing.CompetingConsumers/CompetingConsumerOptions.cs -public sealed class CompetingConsumerOptions -{ - public int MinConsumers { get; set; } = 1; - public int MaxConsumers { get; set; } = 10; - public long ScaleUpThreshold { get; set; } = 1000; - public long ScaleDownThreshold { get; set; } = 100; - public int CooldownMs { get; set; } = 30_000; - public string TargetTopic { get; set; } = string.Empty; - public string ConsumerGroup { get; set; } = string.Empty; -} -``` +var bp = new BackpressureSignal(); -The `CooldownMs` prevents flapping — after a scale event, no further scaling occurs until the cooldown expires. This avoids rapid oscillation when lag hovers near a threshold. +Assert.That(bp.IsBackpressured, Is.False); ---- +bp.Signal(); +Assert.That(bp.IsBackpressured, Is.True); -## Scalability Dimension +bp.Release(); +Assert.That(bp.IsBackpressured, Is.False); +``` -This is the **Scalability pillar centerpiece**. Competing consumers turn a single-threaded message pipeline into a horizontally scalable processing tier. The broker partitions the topic; each consumer claims one or more partitions. Adding consumers increases throughput linearly — up to the partition count. The lag monitor provides the feedback loop, and the scaler adjusts capacity without human intervention. +### 2. InMemoryConsumerScaler — ScaleUp IncreasesCount ---- - -## Atomicity Dimension +```csharp +var scaler = new InMemoryConsumerScaler( + NullLogger.Instance, initialCount: 1); -Each consumer processes messages independently and Acks them individually. If a consumer crashes, its partitions are rebalanced to surviving consumers, and unacked messages are redelivered. The orchestrator only adjusts the consumer count — it never touches message processing itself. This separation ensures that scaling events cannot corrupt in-flight message handling. +Assert.That(scaler.CurrentCount, Is.EqualTo(1)); ---- +await scaler.ScaleAsync(3, CancellationToken.None); -## Lab +Assert.That(scaler.CurrentCount, Is.EqualTo(3)); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial28/Lab.cs`](../tests/TutorialLabs/Tutorial28/Lab.cs) +### 3. ConsumerLagInfo — RecordProperties AreCorrect -**Objective:** Trace the auto-scaling orchestrator with backpressure signaling, analyze cooldown to prevent scaling flap, and design a production backpressure integration. +```csharp +var now = DateTimeOffset.UtcNow; +var info = new ConsumerLagInfo("group-1", "orders", 500, now); -### Step 1: Trace the Scaling Decision Path +Assert.That(info.ConsumerGroup, Is.EqualTo("group-1")); +Assert.That(info.Topic, Is.EqualTo("orders")); +Assert.That(info.CurrentLag, Is.EqualTo(500)); +Assert.That(info.Timestamp, Is.EqualTo(now)); +``` -A topic has 8 partitions, `MaxConsumers = 12`, and current consumer lag is 5,000. Open `src/Processing.CompetingConsumers/CompetingConsumerOrchestrator.cs` and trace `EvaluateAndScaleAsync`: +### 4. InMemoryLagMonitor — ReportAndGet ReturnsReportedLag -1. Lag exceeds `ScaleUpThreshold` → what happens if current consumers = 8? -2. Lag exceeds threshold but `currentCount >= MaxConsumers` → what signal is emitted? -3. After scaling up, what prevents another scale-up in the next cycle? (hint: cooldown) +```csharp +var monitor = new InMemoryConsumerLagMonitor(); +var lag = new ConsumerLagInfo("grp", "topic", 1234, DateTimeOffset.UtcNow); -Now: with `MaxConsumers = 12` and 8 Kafka partitions, what happens when the orchestrator scales to 9 consumers? (hint: one consumer will be idle — Kafka can't assign more consumers than partitions) +await monitor.ReportLagAsync(lag); +var retrieved = await monitor.GetLagAsync("topic", "grp", CancellationToken.None); -### Step 2: Analyze Cooldown for Scaling Stability +Assert.That(retrieved.CurrentLag, Is.EqualTo(1234)); +``` -Consumer lag oscillates between 900 and 1100 with `ScaleUpThreshold = 1000`. Without cooldown: +### 5. EvaluateAndScale — HighLag ScalesUp -``` -Cycle 1: lag=1100 → scale up (3→4) -Cycle 2: lag=900 → scale down (4→3) -Cycle 3: lag=1100 → scale up (3→4) -... flapping forever -``` +```csharp +var lagMonitor = Substitute.For(); +var scaler = Substitute.For(); +var backpressure = new BackpressureSignal(); +var timeProvider = new FakeTimeProvider(DateTimeOffset.UtcNow); -How does `CooldownMs` break this cycle? What is the relationship between cooldown duration and scaling stability? What value would you set for a production system? +scaler.CurrentCount.Returns(1); +lagMonitor.GetLagAsync(Arg.Any(), Arg.Any(), Arg.Any()) + .Returns(new ConsumerLagInfo("grp", "topic", 5000, DateTimeOffset.UtcNow)); -### Step 3: Design a Backpressure Integration +var options = Options.Create(new CompetingConsumerOptions +{ + MinConsumers = 1, + MaxConsumers = 10, + ScaleUpThreshold = 1000, + ScaleDownThreshold = 100, + CooldownMs = 1000, + TargetTopic = "topic", + ConsumerGroup = "grp", +}); + +var orchestrator = new CompetingConsumerOrchestrator( + lagMonitor, scaler, backpressure, options, + NullLogger.Instance, timeProvider); + +await orchestrator.EvaluateAndScaleAsync(CancellationToken.None); + +await scaler.Received(1).ScaleAsync(2, Arg.Any()); +``` -When the consumer pool is at maximum capacity and lag keeps growing, the orchestrator signals backpressure. Design a system-wide response: +## Lab -| Component | Backpressure Action | -|-----------|-------------------| -| Gateway API | Return HTTP 429 to upstream senders | -| Ingestion producers | Pause or slow message publishing | -| Dashboard (OpenClaw) | Show backpressure warning to operators | -| Monitoring (OpenTelemetry) | Emit backpressure metrics and alerts | +Run the full lab: [`tests/TutorialLabs/Tutorial28/Lab.cs`](../tests/TutorialLabs/Tutorial28/Lab.cs) -How does backpressure prevent **cascade failures** in a scalable system? What happens without it? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial28.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial28/Exam.cs`](../tests/TutorialLabs/Tutorial28/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial28/Exam.cs`](../tests/TutorialLabs/Tutorial28/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial28.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/29-throttle-rate-limiting.md b/EnterpriseIntegrationPlatform/tutorials/29-throttle-rate-limiting.md index 54086b1..1c1d95e 100644 --- a/EnterpriseIntegrationPlatform/tutorials/29-throttle-rate-limiting.md +++ b/EnterpriseIntegrationPlatform/tutorials/29-throttle-rate-limiting.md @@ -1,41 +1,8 @@ # Tutorial 29 — Throttle & Rate Limiting -## What You'll Learn +Control message throughput with token-bucket rate limiting and throttling. -- How `IMessageThrottle` controls message flow rate inside the pipeline -- The `TokenBucketThrottle` algorithm: steady refill rate with burst capacity -- `IThrottleRegistry` for managing multiple throttle policies per partition key -- `ThrottlePolicy` with partition strategies: by TenantId, Queue, Endpoint, or Global -- `ThrottleMetrics` for monitoring throttle pressure and wait times -- The difference between **rate limiting** (HTTP 429) and **throttling** (pipeline delays) -- Per-tenant partitioning for fair resource sharing - ---- - -## EIP Pattern: Throttler - -> *"A Throttler limits the rate at which messages flow through a channel, protecting downstream systems from overload."* - -``` - Producer ──▶ ┌──────────────────┐ ──▶ Consumer - │ Token Bucket │ - │ ┌─────────────┐ │ - │ │ ●●●○○○○○○○ │ │ ← tokens (● = available) - │ └─────────────┘ │ - │ refill: 100/sec │ - │ burst: 200 │ - └──────────────────┘ - │ - No token? → Delay or 429 -``` - -The token bucket allows short bursts above the steady-state rate while enforcing a long-term average. Tokens are consumed per message and refilled at a constant rate. - ---- - -## Platform Implementation - -### IMessageThrottle +## Key Types ```csharp // src/Processing.Throttle/IMessageThrottle.cs @@ -51,10 +18,6 @@ public interface IMessageThrottle } ``` -`AcquireAsync` returns a `ThrottleResult` indicating whether the message can proceed immediately, must wait, or should be rejected. - -### TokenBucketThrottle (concrete) - ```csharp // src/Processing.Throttle/TokenBucketThrottle.cs public sealed class TokenBucketThrottle : IMessageThrottle, IDisposable @@ -65,8 +28,6 @@ public sealed class TokenBucketThrottle : IMessageThrottle, IDisposable } ``` -### ThrottleResult - ```csharp // src/Processing.Throttle/ThrottleResult.cs public sealed record ThrottleResult @@ -78,8 +39,6 @@ public sealed record ThrottleResult } ``` -### ThrottlePartitionKey - ```csharp // src/Processing.Throttle/ThrottlePartitionKey.cs public sealed record ThrottlePartitionKey @@ -90,113 +49,116 @@ public sealed record ThrottlePartitionKey } ``` -| Key Property | Use Case | -|-------------|----------| -| `TenantId` | Fair share per tenant | -| `Queue` | Protect a specific queue | -| `Endpoint` | Throttle per downstream endpoint | +## Exercises -### IThrottleRegistry +### 1. AcquireAsync — WithAvailableTokens ReturnsPermitted ```csharp -// src/Processing.Throttle/IThrottleRegistry.cs -public interface IThrottleRegistry +var options = Options.Create(new ThrottleOptions { - IMessageThrottle Resolve(ThrottlePartitionKey key); - void SetPolicy(ThrottlePolicy policy); - bool RemovePolicy(string policyId); - IReadOnlyList GetAllPolicies(); - ThrottlePolicyStatus? GetPolicy(string policyId); -} -``` + MaxMessagesPerSecond = 100, + BurstCapacity = 10, + MaxWaitTime = TimeSpan.FromSeconds(5), +}); -### ThrottleMetrics +using var throttle = new TokenBucketThrottle(options, NullLogger.Instance); -```csharp -// src/Processing.Throttle/ThrottleMetrics.cs -public sealed record ThrottleMetrics -{ - public required long TotalAcquired { get; init; } - public required long TotalRejected { get; init; } - public required int AvailableTokens { get; init; } - public required int BurstCapacity { get; init; } - public required int RefillRate { get; init; } - public required TimeSpan TotalWaitTime { get; init; } -} -``` +var envelope = IntegrationEnvelope.Create("data", "TestService", "test.event"); -### Rate Limiting vs Throttling +var result = await throttle.AcquireAsync(envelope); -| Aspect | Rate Limiting (Gateway) | Throttling (Pipeline) | -|--------|------------------------|----------------------| -| Location | HTTP ingress (Gateway.Api) | Internal pipeline stage | -| Response | HTTP 429 Too Many Requests | Delay + delivery | -| Effect | Rejects message | Slows message down | -| Visibility | Client sees rejection | Client sees slower response | +Assert.That(result.Permitted, Is.True); +Assert.That(result.RejectionReason, Is.Null); +``` -Rate limiting protects the **platform** from external overload. Throttling protects **downstream systems** from internal overload. +### 2. AcquireAsync — ConsumesToken DecreasesAvailableCount ---- +```csharp +var options = Options.Create(new ThrottleOptions +{ + MaxMessagesPerSecond = 100, + BurstCapacity = 5, +}); -## Scalability Dimension +using var throttle = new TokenBucketThrottle(options, NullLogger.Instance); -Per-tenant partitioning (via `ThrottlePartitionKey.TenantId`) ensures one noisy tenant cannot consume all throughput. Each tenant's bucket is independent. The `ThrottleMetrics` feed into the competing consumers orchestrator (Tutorial 28): if wait times climb, the orchestrator adds consumers. +var before = throttle.AvailableTokens; ---- +var envelope = IntegrationEnvelope.Create("data", "TestService", "test.event"); +await throttle.AcquireAsync(envelope); -## Atomicity Dimension +Assert.That(throttle.AvailableTokens, Is.LessThan(before)); +``` -When `AcquireAsync` delays a message, the message remains **uncommitted** — not Acked until it passes the throttle. If the service restarts during a wait, the broker redelivers. The `MaxWait` timeout prevents indefinite blocking; if exceeded, the message is Nacked and retried (Tutorial 24). +### 3. AcquireAsync — NoTokensWithRejectOnBackpressure RejectsImmediately ---- +```csharp +var options = Options.Create(new ThrottleOptions +{ + MaxMessagesPerSecond = 1, + BurstCapacity = 1, + RejectOnBackpressure = true, +}); -## Lab +using var throttle = new TokenBucketThrottle(options, NullLogger.Instance); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial29/Lab.cs`](../tests/TutorialLabs/Tutorial29/Lab.cs) +var envelope = IntegrationEnvelope.Create("data", "TestService", "test.event"); -**Objective:** Design throttle policies for multi-tenant rate limiting, trace the token bucket algorithm, and analyze why per-tenant throttling is essential for **fair scalability**. +// Consume the only token. +await throttle.AcquireAsync(envelope); -### Step 1: Design a Multi-Tenant Throttle Policy +// Next acquire should be rejected (no tokens, reject mode). +var result = await throttle.AcquireAsync(envelope); -Design a `ThrottlePolicy` for a partner system: +Assert.That(result.Permitted, Is.False); +Assert.That(result.RejectionReason, Is.Not.Null.And.Not.Empty); +``` -| Parameter | Value | Purpose | -|-----------|-------|---------| -| Rate | 50 messages/second | Sustained throughput | -| Burst | 200 messages | Peak absorption | -| `PartitionKey.TenantId` | `"partner-x"` | Per-tenant isolation | -| `MaxWait` | 5 seconds | Max time to wait for a token | +### 4. ThrottleOptions — Defaults AreReasonable -Open `src/Processing.Throttle/` and verify: How does the `TokenBucketThrottle` implement this? What happens when all 200 burst tokens are consumed? +```csharp +var opts = new ThrottleOptions(); -### Step 2: Trace Token Exhaustion +Assert.That(opts.MaxMessagesPerSecond, Is.EqualTo(100)); +Assert.That(opts.BurstCapacity, Is.EqualTo(200)); +Assert.That(opts.MaxWaitTime, Is.EqualTo(TimeSpan.FromSeconds(30))); +Assert.That(opts.RejectOnBackpressure, Is.False); +``` -The `TokenBucketThrottle` has 0 available tokens and `MaxWait = 5s`. A message arrives: +### 5. ThrottleResult — ContainsExpectedFields + +```csharp +var options = Options.Create(new ThrottleOptions +{ + MaxMessagesPerSecond = 100, + BurstCapacity = 10, +}); -1. The throttle checks: 0 tokens available -2. Waits for token replenishment (50 tokens/second = 1 token every 20ms) -3. After ~20ms, 1 token becomes available → message proceeds -4. If no token is available after 5 seconds → what happens? +using var throttle = new TokenBucketThrottle(options, NullLogger.Instance); -What is the maximum queuing depth during a sustained burst? How does `MaxWait` prevent unbounded queue growth? +var envelope = IntegrationEnvelope.Create("data", "TestService", "test.event"); +var result = await throttle.AcquireAsync(envelope); -### Step 3: Analyze Per-Tenant vs. Global Throttling +Assert.That(result.Permitted, Is.True); +Assert.That(result.WaitTime, Is.GreaterThanOrEqualTo(TimeSpan.Zero)); +Assert.That(result.RemainingTokens, Is.GreaterThanOrEqualTo(0)); +``` -Why does the platform use per-`TenantId` throttling by default? +## Lab -| Scenario | Global Throttle | Per-Tenant Throttle | -|----------|----------------|-------------------| -| Tenant A sends 10,000 msg/s | Blocks Tenant B too | Only Tenant A is throttled | -| Tenant B sends 10 msg/s | May be blocked by A | Always gets through | -| Fair resource allocation | No guarantee | Each tenant gets its quota | +Run the full lab: [`tests/TutorialLabs/Tutorial29/Lab.cs`](../tests/TutorialLabs/Tutorial29/Lab.cs) -How does per-tenant throttling prevent the **noisy neighbor** problem? Why is this critical for **multi-tenant scalability**? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial29.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial29/Exam.cs`](../tests/TutorialLabs/Tutorial29/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial29/Exam.cs`](../tests/TutorialLabs/Tutorial29/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial29.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/30-rule-engine.md b/EnterpriseIntegrationPlatform/tutorials/30-rule-engine.md index dc6400b..e695ca1 100644 --- a/EnterpriseIntegrationPlatform/tutorials/30-rule-engine.md +++ b/EnterpriseIntegrationPlatform/tutorials/30-rule-engine.md @@ -1,38 +1,8 @@ # Tutorial 30 — Rule Engine -## What You'll Learn +Evaluate business rules with AND/OR condition logic against message fields. -- How `IRuleEngine` evaluates business rules against integration messages -- `BusinessRuleEngine` with priority-ordered evaluation -- `IRuleStore` for persisting and querying rules at runtime -- `BusinessRule` with conditions (AND/OR) and actions (Route, Transform, Reject, DeadLetter) -- `RuleConditionOperator` enum for flexible condition matching -- `RuleEvaluationResult` and `InMemoryRuleStore` - ---- - -## EIP Pattern: Message Router (Rule-Based) - -> *"A Message Router inspects an incoming message, determines what channel to send it to, and forwards it to that channel. A rule-based router externalizes routing logic into a set of configurable rules."* - -``` - ┌──────────────┐ ┌───────────────┐ - │ Incoming │────▶│ Rule Engine │ - │ Message │ │ │ - └──────────────┘ │ Rule 1 (P=1) │──▶ Route to Topic A - │ Rule 2 (P=2) │──▶ Transform - │ Rule 3 (P=3) │──▶ Reject - │ Default │──▶ DeadLetter - └───────────────┘ -``` - -Rules are evaluated in priority order. The first match determines the action, decoupling business logic from code. - ---- - -## Platform Implementation - -### IRuleEngine +## Key Types ```csharp // src/RuleEngine/IRuleEngine.cs @@ -44,8 +14,6 @@ public interface IRuleEngine } ``` -### BusinessRule - ```csharp // src/RuleEngine/BusinessRule.cs public sealed record BusinessRule @@ -62,8 +30,6 @@ public sealed record BusinessRule public enum RuleLogicOperator { And, Or } ``` -### RuleCondition and RuleConditionOperator - ```csharp // src/RuleEngine/RuleCondition.cs public sealed record RuleCondition @@ -79,8 +45,6 @@ public enum RuleConditionOperator } ``` -### RuleAction - ```csharp // src/RuleEngine/RuleAction.cs public sealed record RuleAction @@ -100,99 +64,131 @@ public enum RuleActionType } ``` -### RuleEvaluationResult - -```csharp -// src/RuleEngine/RuleEvaluationResult.cs -public sealed record RuleEvaluationResult( - IReadOnlyList MatchedRules, - IReadOnlyList Actions, - bool HasMatch, - int RulesEvaluated); -``` +## Exercises -### IRuleStore +### 1. Evaluate — SingleEqualsRule MatchesByMessageType ```csharp -// src/RuleEngine/IRuleStore.cs -public interface IRuleStore +await _store.AddOrUpdateAsync(new BusinessRule { - Task> GetAllAsync(CancellationToken ct = default); - Task GetByNameAsync(string name, CancellationToken ct = default); - Task AddOrUpdateAsync(BusinessRule rule, CancellationToken ct = default); - Task RemoveAsync(string name, CancellationToken ct = default); - Task CountAsync(CancellationToken ct = default); -} -``` - -`InMemoryRuleStore` holds rules in a `ConcurrentDictionary` sorted by `Priority`, suitable for development and tutorials. - ---- + Name = "RouteOrders", + Priority = 1, + Conditions = [new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Equals, Value = "order.created" }], + Action = new RuleAction { ActionType = RuleActionType.Route, TargetTopic = "orders-topic" }, +}); -## Scalability Dimension +var envelope = IntegrationEnvelope.Create("data", "OrderService", "order.created"); +var result = await _engine.EvaluateAsync(envelope); -The rule engine is **stateless** — it loads rules from the store and evaluates each message independently. Rules are cached in memory and refreshed periodically. Horizontal scaling is straightforward: add more consumer replicas sharing the same cached rule set. +Assert.That(result.HasMatch, Is.True); +Assert.That(result.MatchedRules, Has.Count.EqualTo(1)); +Assert.That(result.Actions[0].TargetTopic, Is.EqualTo("orders-topic")); +``` ---- +### 2. Evaluate — NoMatchingRule ReturnsNoMatch -## Atomicity Dimension +```csharp +await _store.AddOrUpdateAsync(new BusinessRule +{ + Name = "RouteOrders", + Priority = 1, + Conditions = [new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Equals, Value = "order.created" }], + Action = new RuleAction { ActionType = RuleActionType.Route, TargetTopic = "orders-topic" }, +}); -Rule evaluation happens **within the pipeline transaction**. If the selected action fails, the message is Nacked and retried. Rules are versioned — updating a rule does not affect in-flight messages. The `RulesEvaluated` counter provides observability into evaluation depth. +var envelope = IntegrationEnvelope.Create("data", "PaymentService", "payment.received"); +var result = await _engine.EvaluateAsync(envelope); ---- +Assert.That(result.HasMatch, Is.False); +Assert.That(result.MatchedRules, Is.Empty); +Assert.That(result.Actions, Is.Empty); +``` -## Lab +### 3. Evaluate — ContainsOperator MatchesSubstring -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial30/Lab.cs`](../tests/TutorialLabs/Tutorial30/Lab.cs) +```csharp +await _store.AddOrUpdateAsync(new BusinessRule +{ + Name = "AllOrders", + Priority = 1, + Conditions = [new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Contains, Value = "order" }], + Action = new RuleAction { ActionType = RuleActionType.Route, TargetTopic = "all-orders" }, +}); -**Objective:** Write business rules with conditions and logic operators, trace priority-based evaluation, and analyze rule caching for **scalable** high-throughput routing decisions. +var envelope = IntegrationEnvelope.Create("data", "Service", "order.shipped"); +var result = await _engine.EvaluateAsync(envelope); -### Step 1: Write a Priority-Based Business Rule +Assert.That(result.HasMatch, Is.True); +Assert.That(result.Actions[0].TargetTopic, Is.EqualTo("all-orders")); +``` -Write a `BusinessRule` that routes all messages from source `"PartnerX"` with `MessageType` containing `"order"` to topic `"orders-priority"`: +### 4. Evaluate — AndLogic AllConditionsMustMatch ```csharp -var rule = new BusinessRule +await _store.AddOrUpdateAsync(new BusinessRule { - Name = "PartnerX-Orders", + Name = "HighPriorityOrders", Priority = 1, LogicOperator = RuleLogicOperator.And, - Conditions = [ - new RuleCondition { FieldName = "Source", Operator = RuleConditionOperator.Equals, Value = "PartnerX" }, - new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Contains, Value = "order" } + Conditions = + [ + new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Equals, Value = "order.created" }, + new RuleCondition { FieldName = "Source", Operator = RuleConditionOperator.Equals, Value = "PremiumService" }, ], - Action = new RuleAction { ActionType = RuleActionType.Route, TargetTopic = "orders-priority" } -}; + Action = new RuleAction { ActionType = RuleActionType.Route, TargetTopic = "premium-orders" }, +}); + +// Only MessageType matches, Source doesn't → no match. +var envelope1 = IntegrationEnvelope.Create("data", "BasicService", "order.created"); +var result1 = await _engine.EvaluateAsync(envelope1); +Assert.That(result1.HasMatch, Is.False); + +// Both match → match. +var envelope2 = IntegrationEnvelope.Create("data", "PremiumService", "order.created"); +var result2 = await _engine.EvaluateAsync(envelope2); +Assert.That(result2.HasMatch, Is.True); ``` -Open `src/RuleEngine/BusinessRuleEngine.cs` and trace: How does `And` vs. `Or` logic change the evaluation? - -### Step 2: Trace Priority-Based Evaluation +### 5. Evaluate — OrLogic AnyConditionMatches -Rules are evaluated in priority order (lowest number = highest priority): +```csharp +await _store.AddOrUpdateAsync(new BusinessRule +{ + Name = "OrderOrPayment", + Priority = 1, + LogicOperator = RuleLogicOperator.Or, + Conditions = + [ + new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Equals, Value = "order.created" }, + new RuleCondition { FieldName = "MessageType", Operator = RuleConditionOperator.Equals, Value = "payment.received" }, + ], + Action = new RuleAction { ActionType = RuleActionType.Route, TargetTopic = "finance" }, +}); -| Priority | Rule | Conditions | -|----------|------|-----------| -| 1 | Premium orders | Source = "PartnerX" AND Type contains "order" | -| 5 | All orders | Type contains "order" | -| 10 | Default | Always matches | +var orderEnvelope = IntegrationEnvelope.Create("data", "Service", "order.created"); +var orderResult = await _engine.EvaluateAsync(orderEnvelope); +Assert.That(orderResult.HasMatch, Is.True); -A message from `PartnerX` with type `"order.created"` matches rules at priorities 1 and 5. Which rule wins? Why does the engine stop at the first match? (hint: deterministic routing for **atomicity**) +var paymentEnvelope = IntegrationEnvelope.Create("data", "Service", "payment.received"); +var paymentResult = await _engine.EvaluateAsync(paymentEnvelope); +Assert.That(paymentResult.HasMatch, Is.True); +``` -### Step 3: Design Rule Caching for Scalability +## Lab -At 50,000 messages/second with 100 rules, each message evaluates up to 100 conditions. Design a caching strategy: +Run the full lab: [`tests/TutorialLabs/Tutorial30/Lab.cs`](../tests/TutorialLabs/Tutorial30/Lab.cs) -- Rules change infrequently (hourly) but messages arrive constantly -- How does the platform cache compiled rules? (Open `src/RuleEngine/` to check) -- What is the cache invalidation strategy when rules are updated? -- What is the performance difference between cached vs. uncached rule evaluation? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial30.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial30/Exam.cs`](../tests/TutorialLabs/Tutorial30/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial30/Exam.cs`](../tests/TutorialLabs/Tutorial30/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial30.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/31-event-sourcing.md b/EnterpriseIntegrationPlatform/tutorials/31-event-sourcing.md index 33ffd87..bdcc4fb 100644 --- a/EnterpriseIntegrationPlatform/tutorials/31-event-sourcing.md +++ b/EnterpriseIntegrationPlatform/tutorials/31-event-sourcing.md @@ -1,44 +1,8 @@ # Tutorial 31 — Event Sourcing -## What You'll Learn +Store domain events as an append-only log and rebuild state by replaying them. -- How `IEventStore` provides an append-only log of domain events -- `ISnapshotStore` for periodic snapshots to speed up replay -- `IEventProjection` and `EventProjectionEngine` for building read-side views -- `EventEnvelope` as the standard wrapper for stored events -- Optimistic concurrency via `OptimisticConcurrencyException` -- `TemporalQuery` for point-in-time replay and `InMemoryEventStore` - ---- - -## EIP Pattern: Event-Driven Consumer (Event Sourcing) - -> *"Instead of storing just the current state, store the full history of state-changing events. Reconstruct current state by replaying events."* - -``` - Command ──▶ ┌──────────────┐ ┌─────────────────┐ - │ Aggregate │────▶│ Event Store │ - │ (validate) │ │ (append-only) │ - └──────────────┘ └────────┬─────────┘ - │ - ┌───────────────────────┤ - ▼ ▼ - ┌───────────────┐ ┌───────────────────┐ - │ Snapshot │ │ Projection │ - │ Store │ │ Engine │ - └───────────────┘ └───────────────────┘ - │ - ▼ - Read-side views -``` - -The event store is the source of truth. Projections build queryable read models. Snapshots cache state at known positions. - ---- - -## Platform Implementation - -### IEventStore +## Key Types ```csharp // src/EventSourcing/IEventStore.cs @@ -64,8 +28,6 @@ public interface IEventStore } ``` -### EventEnvelope - ```csharp // src/EventSourcing/EventEnvelope.cs public sealed record EventEnvelope( @@ -78,8 +40,6 @@ public sealed record EventEnvelope( Dictionary Metadata); ``` -### Optimistic Concurrency - ```csharp // src/EventSourcing/OptimisticConcurrencyException.cs public sealed class OptimisticConcurrencyException : InvalidOperationException @@ -90,12 +50,6 @@ public sealed class OptimisticConcurrencyException : InvalidOperationException } ``` -When `AppendAsync` is called with an `expectedVersion` that does not match the stream's current version, the store throws `OptimisticConcurrencyException`. On success, `AppendAsync` returns the new stream version as a `long`. The caller must reload the stream, re-apply the command, and retry on conflict. This prevents lost updates without pessimistic locks. - -### TemporalQuery - -`TemporalQuery` is a static helper class that replays a stream's events up to a specific point in time, producing the projected state at that moment: - ```csharp // src/EventSourcing/TemporalQuery.cs public static class TemporalQuery @@ -111,88 +65,106 @@ public static class TemporalQuery } ``` -### ISnapshotStore - -```csharp -// src/EventSourcing/ISnapshotStore.cs -public interface ISnapshotStore -{ - Task SaveAsync(string streamId, TState state, long version, CancellationToken cancellationToken = default); - Task<(TState? State, long Version)> LoadAsync(string streamId, CancellationToken cancellationToken = default); -} -``` +## Exercises -### IEventProjection and EventProjectionEngine +### 1. AppendAsync — AndReadStreamAsync Roundtrip ```csharp -// src/EventSourcing/IEventProjection.cs -public interface IEventProjection -{ - Task ProjectAsync(TState state, EventEnvelope envelope, CancellationToken cancellationToken = default); -} -``` +var envelope = new EventEnvelope( + Guid.NewGuid(), "stream-1", "OrderCreated", + """{"total":42}""", 0, DateTimeOffset.UtcNow, []); -`IEventProjection` is an async function: given a current state and an event, it returns the new state. The `EventProjectionEngine` reads new events from the store, applies each to the appropriate `IEventProjection` implementation, and tracks the last processed version per projection. `InMemoryEventStore` implements `IEventStore` using a `ConcurrentDictionary` with full optimistic concurrency support. +await _store.AppendAsync("stream-1", [envelope], expectedVersion: 0); ---- +var events = await _store.ReadStreamAsync("stream-1", fromVersion: 1, count: 100); -## Scalability Dimension +Assert.That(events, Has.Count.EqualTo(1)); +Assert.That(events[0].StreamId, Is.EqualTo("stream-1")); +Assert.That(events[0].EventType, Is.EqualTo("OrderCreated")); +Assert.That(events[0].Version, Is.EqualTo(1)); +``` -The event store is **append-only** — writes never conflict with reads. Multiple projections run in parallel, each maintaining its own checkpoint. Snapshots reduce replay time from O(n) to O(1). The store can be partitioned by `StreamId`. +### 2. AppendMultiple — ReadAllBack InOrder ---- +```csharp +var e1 = new EventEnvelope(Guid.NewGuid(), "s", "A", "d1", 0, DateTimeOffset.UtcNow, []); +var e2 = new EventEnvelope(Guid.NewGuid(), "s", "B", "d2", 0, DateTimeOffset.UtcNow, []); +var e3 = new EventEnvelope(Guid.NewGuid(), "s", "C", "d3", 0, DateTimeOffset.UtcNow, []); + +await _store.AppendAsync("s", [e1], expectedVersion: 0); +await _store.AppendAsync("s", [e2], expectedVersion: 1); +await _store.AppendAsync("s", [e3], expectedVersion: 2); + +var events = await _store.ReadStreamAsync("s", fromVersion: 1, count: 100); + +Assert.That(events, Has.Count.EqualTo(3)); +Assert.That(events[0].Version, Is.EqualTo(1)); +Assert.That(events[1].Version, Is.EqualTo(2)); +Assert.That(events[2].Version, Is.EqualTo(3)); +Assert.That(events[0].EventType, Is.EqualTo("A")); +Assert.That(events[2].EventType, Is.EqualTo("C")); +``` -## Atomicity Dimension +### 3. AppendAsync — VersionConflict ThrowsOptimisticConcurrencyException -Optimistic concurrency ensures **consistency without locks**. The `expectedVersion` acts as a compare-and-swap: if two writers race, one succeeds and the other gets `OptimisticConcurrencyException`. Events are immutable once appended — never modified or deleted — providing a tamper-evident audit trail. +```csharp +var e = new EventEnvelope(Guid.NewGuid(), "s", "E", "d", 0, DateTimeOffset.UtcNow, []); +await _store.AppendAsync("s", [e], expectedVersion: 0); ---- +var e2 = new EventEnvelope(Guid.NewGuid(), "s", "E2", "d2", 0, DateTimeOffset.UtcNow, []); -## Lab +var ex = Assert.ThrowsAsync( + () => _store.AppendAsync("s", [e2], expectedVersion: 0)); -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial31/Lab.cs`](../tests/TutorialLabs/Tutorial31/Lab.cs) +Assert.That(ex!.StreamId, Is.EqualTo("s")); +Assert.That(ex.ExpectedVersion, Is.EqualTo(0)); +Assert.That(ex.ActualVersion, Is.EqualTo(1)); +``` -**Objective:** Analyze event sourcing's append-only model for **audit-complete atomicity**, trace optimistic concurrency conflict resolution, and design snapshot strategies for **scalable** aggregate reconstruction. +### 4. ReadStreamBackwardAsync — ReturnsReversedOrder -### Step 1: Calculate Aggregate Reconstruction Cost +```csharp +var e1 = new EventEnvelope(Guid.NewGuid(), "s", "A", "d1", 0, DateTimeOffset.UtcNow, []); +var e2 = new EventEnvelope(Guid.NewGuid(), "s", "B", "d2", 0, DateTimeOffset.UtcNow, []); +var e3 = new EventEnvelope(Guid.NewGuid(), "s", "C", "d3", 0, DateTimeOffset.UtcNow, []); -An aggregate has 10,000 events. Compare reconstruction approaches: +await _store.AppendAsync("s", [e1, e2, e3], expectedVersion: 0); -| Approach | Events Replayed | Cost | Time (est.) | -|----------|----------------|------|-------------| -| Full replay (no snapshots) | 10,000 | High CPU + memory | ~100ms | -| Snapshot at version 9,900 | 100 | Low | ~1ms | -| Snapshot at version 9,999 | 1 | Minimal | ~0.1ms | +var events = await _store.ReadStreamBackwardAsync("s", fromVersion: 3, count: 100); -Open `src/EventSourcing/` and trace: How does the event store load a snapshot, then replay only subsequent events? What is the **scalability** trade-off between snapshot frequency and storage cost? +Assert.That(events, Has.Count.EqualTo(3)); +Assert.That(events[0].Version, Is.EqualTo(3)); +Assert.That(events[1].Version, Is.EqualTo(2)); +Assert.That(events[2].Version, Is.EqualTo(1)); +``` -### Step 2: Trace Optimistic Concurrency Conflict +### 5. SnapshotStore — SaveAndLoad Roundtrip -Two commands arrive simultaneously for the same stream at version 5. Both expect version 5: +```csharp +var snapshots = new InMemorySnapshotStore(); -``` -Command A: Append event at version 5 → succeeds (stream now at version 6) -Command B: Append event at version 5 → CONFLICT (expected 5, actual 6) -``` +await snapshots.SaveAsync("stream-1", 42, 5); +var (state, version) = await snapshots.LoadAsync("stream-1"); -Trace the conflict resolution: -1. What exception is thrown? -2. Does Command B retry? With what strategy? -3. How does optimistic concurrency ensure **atomic** state transitions without distributed locks? +Assert.That(state, Is.EqualTo(42)); +Assert.That(version, Is.EqualTo(5)); +``` -### Step 3: Design a Temporal Query for Audit +## Lab -Use `TemporalQuery.ReplayToPointInTimeAsync` to reconstruct an order aggregate's state as of yesterday at noon: +Run the full lab: [`tests/TutorialLabs/Tutorial31/Lab.cs`](../tests/TutorialLabs/Tutorial31/Lab.cs) -- What parameters do you supply? (stream ID, point-in-time) -- How does this differ from loading current state? -- Why is this capability essential for **regulatory compliance** and audit trails? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial31.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial31/Exam.cs`](../tests/TutorialLabs/Tutorial31/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial31/Exam.cs`](../tests/TutorialLabs/Tutorial31/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial31.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/32-multi-tenancy.md b/EnterpriseIntegrationPlatform/tutorials/32-multi-tenancy.md index 3bb6447..1fe3fd3 100644 --- a/EnterpriseIntegrationPlatform/tutorials/32-multi-tenancy.md +++ b/EnterpriseIntegrationPlatform/tutorials/32-multi-tenancy.md @@ -1,44 +1,8 @@ # Tutorial 32 — Multi-Tenancy -## What You'll Learn +Isolate message processing per tenant with tenant resolution and context propagation. -- How `ITenantResolver` identifies the current tenant from message metadata or HTTP headers -- `ITenantIsolationGuard` enforces cross-tenant data boundaries -- `TenantContext` carries tenant identity through the pipeline -- `TenantIsolationException` when a message crosses tenant boundaries -- Anonymous tenant handling for unauthenticated or system messages -- `MultiTenancy.Onboarding` for self-service tenant provisioning - ---- - -## EIP Pattern: Message Metadata (Tenant Header) - -> *"Attach tenant identity as message metadata so every component in the pipeline can enforce data isolation without coupling to the resolution mechanism."* - -``` - ┌──────────────┐ ┌──────────────────┐ - │ Ingress │────▶│ Tenant Resolver │ - │ (header or │ │ (metadata / │ - │ metadata) │ │ header lookup) │ - └──────────────┘ └───────┬──────────┘ - │ - TenantContext flows - through pipeline - │ - ┌──────────▼──────────┐ - │ Isolation Guard │ - │ (cross-tenant │ - │ boundary check) │ - └─────────────────────┘ -``` - -Every message entering the platform passes through tenant resolution. The resolved `TenantContext` propagates through all pipeline stages, and the isolation guard rejects any operation that would cross tenant boundaries. - ---- - -## Platform Implementation - -### ITenantResolver +## Key Types ```csharp // src/MultiTenancy/ITenantResolver.cs @@ -49,10 +13,6 @@ public interface ITenantResolver } ``` -Both `Resolve` overloads are synchronous. The metadata overload checks (in order): the `X-Tenant-Id` header, the `TenantId` metadata key, and the authenticated identity's tenant claim. The string overload resolves directly from a tenant ID. If no tenant is found, either overload returns the **anonymous tenant context** (`TenantContext.Anonymous`). - -### TenantContext - ```csharp // src/MultiTenancy/TenantContext.cs public sealed class TenantContext @@ -69,8 +29,6 @@ public sealed class TenantContext } ``` -### ITenantIsolationGuard - ```csharp // src/MultiTenancy/ITenantIsolationGuard.cs public interface ITenantIsolationGuard @@ -79,8 +37,6 @@ public interface ITenantIsolationGuard } ``` -### TenantIsolationException - ```csharp // src/MultiTenancy/TenantIsolationException.cs public sealed class TenantIsolationException : Exception @@ -91,95 +47,91 @@ public sealed class TenantIsolationException : Exception } ``` -When a message's actual tenant does not match the expected tenant, the guard throws `TenantIsolationException` with the `MessageId`, `ActualTenantId`, and `ExpectedTenantId`. This is **non-retryable** — the message goes directly to the DLQ. +## Exercises -### MultiTenancy.Onboarding +### 1. Resolve — FromMetadata WithTenantIdKey ```csharp -// src/MultiTenancy.Onboarding/ITenantOnboardingService.cs -public interface ITenantOnboardingService +var metadata = new Dictionary { - Task ProvisionAsync( - TenantOnboardingRequest request, - CancellationToken cancellationToken = default); + [TenantResolver.TenantMetadataKey] = "tenant-abc", +}; - Task DeprovisionAsync( - string tenantId, - CancellationToken cancellationToken = default); +var context = _resolver.Resolve(metadata); - Task GetStatusAsync( - string tenantId, - CancellationToken cancellationToken = default); -} - -public sealed record TenantOnboardingRequest( - string TenantId, - string TenantName, - TenantPlan Plan, - string AdminEmail, - IReadOnlyDictionary? Metadata = null); +Assert.That(context.TenantId, Is.EqualTo("tenant-abc")); +Assert.That(context.IsResolved, Is.True); ``` -Onboarding provisions: tenant-specific broker topics, throttle policies (Tutorial 29), configuration namespaces, and an admin user. Deprovisioning archives data and removes access. - ---- - -## Scalability Dimension +### 2. Resolve — MissingTenantId ReturnsAnonymous -Tenant isolation enables **horizontal scaling by tenant**. High-volume tenants can be assigned dedicated consumer groups and broker partitions while small tenants share resources. The `TenantContext.TenantId` and `TenantName` identify each tenant, while `IsResolved` distinguishes resolved tenants from anonymous ones. Per-tenant quotas and feature flags can be managed through external configuration, enabling per-tenant scaling decisions in the competing consumers orchestrator (Tutorial 28). - ---- - -## Atomicity Dimension - -The isolation guard runs **before any processing** — a cross-tenant message is rejected atomically before it can corrupt another tenant's data. The `TenantContext` is resolved once at ingress and propagated immutably through the pipeline. This ensures that every component sees the same tenant identity, preventing time-of-check/time-of-use races. +```csharp +var metadata = new Dictionary(); ---- +var context = _resolver.Resolve(metadata); -## Lab +Assert.That(context.IsResolved, Is.False); +Assert.That(context, Is.SameAs(TenantContext.Anonymous)); +``` -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial32/Lab.cs`](../tests/TutorialLabs/Tutorial32/Lab.cs) +### 3. Resolve — String WithExplicitTenantId -**Objective:** Trace tenant resolution and isolation enforcement, design the onboarding resource provisioning pipeline, and analyze why tenant isolation is non-negotiable for **multi-tenant scalability**. +```csharp +var context = _resolver.Resolve("my-tenant"); -### Step 1: Resolve a Tenant Identity Conflict +Assert.That(context.TenantId, Is.EqualTo("my-tenant")); +Assert.That(context.IsResolved, Is.True); +``` -A message arrives with `X-Tenant-Id: tenant-a` but the JWT claim says `tenant-b`. Open `src/MultiTenancy/` and trace: +### 4. IsolationGuard — Enforce PassesWhenTenantMatches -1. How does the tenant resolver prioritize these conflicting signals? -2. Should the resolver trust the header or the JWT? (hint: JWT is cryptographically signed) -3. What exception is thrown for the conflict? Is it retryable? +```csharp +var guard = new TenantIsolationGuard(_resolver); +var envelope = IntegrationEnvelope.Create("data", "Svc", "event") with +{ + Metadata = new Dictionary + { + [TenantResolver.TenantMetadataKey] = "tenant-x", + }, +}; -Design a resolution policy: When is a conflict legitimate (e.g., admin impersonation) vs. a security violation? +Assert.DoesNotThrow(() => guard.Enforce(envelope, "tenant-x")); +``` -### Step 2: Design the Onboarding Pipeline +### 5. IsolationGuard — Enforce ThrowsOnMismatch -When a new tenant onboards, trace the self-service provisioning flow: +```csharp +var guard = new TenantIsolationGuard(_resolver); +var envelope = IntegrationEnvelope.Create("data", "Svc", "event") with +{ + Metadata = new Dictionary + { + [TenantResolver.TenantMetadataKey] = "tenant-a", + }, +}; -| Step | Resource | Class | Atomic? | -|------|----------|-------|---------| -| 1 | Create tenant record | `InMemoryTenantOnboardingService` | Yes | -| 2 | Provision broker namespace | `InMemoryBrokerNamespaceProvisioner` | Yes | -| 3 | Set quota limits | `InMemoryTenantQuotaManager` | Yes | -| 4 | Initialize configuration | ConfigurationStore | Yes | +var ex = Assert.Throws( + () => guard.Enforce(envelope, "tenant-b")); -If Step 3 fails, what compensation is needed for Steps 1-2? How does this relate to the Saga pattern? +Assert.That(ex!.ActualTenantId, Is.EqualTo("tenant-a")); +Assert.That(ex.ExpectedTenantId, Is.EqualTo("tenant-b")); +``` -### Step 3: Analyze Tenant Isolation for Scalability +## Lab -| Without Isolation | With Isolation | -|------------------|----------------| -| Tenant A's traffic spike affects Tenant B | Each tenant has dedicated queues and quotas | -| One tenant's DLQ overflow blocks all tenants | Isolated DLQ per tenant | -| Security breach in one tenant exposes all | `TenantIsolationGuard` prevents cross-tenant access | +Run the full lab: [`tests/TutorialLabs/Tutorial32/Lab.cs`](../tests/TutorialLabs/Tutorial32/Lab.cs) -Why is `TenantIsolationException` non-retryable? Under what circumstances could a cross-tenant message be legitimate? +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial32.Lab" +``` ## Exam -> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial32/Exam.cs`](../tests/TutorialLabs/Tutorial32/Exam.cs) +Coding challenges: [`tests/TutorialLabs/Tutorial32/Exam.cs`](../tests/TutorialLabs/Tutorial32/Exam.cs) -Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline. +```bash +dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial32.Exam" +``` --- diff --git a/EnterpriseIntegrationPlatform/tutorials/33-security.md b/EnterpriseIntegrationPlatform/tutorials/33-security.md index a20a411..55b4dc2 100644 --- a/EnterpriseIntegrationPlatform/tutorials/33-security.md +++ b/EnterpriseIntegrationPlatform/tutorials/33-security.md @@ -1,40 +1,8 @@ # Tutorial 33 — Security -## What You'll Learn +Sanitize input, encrypt payloads, and validate messages against security threats. -- How `IInputSanitizer` strips dangerous content from message payloads -- `IPayloadSizeGuard` enforces maximum payload size limits -- Script tag removal, SQL injection pattern detection, and HTML entity sanitization -- `PayloadTooLargeException` for oversized payloads -- `JwtOptions` for configuring JWT authentication and validation -- `Security.Secrets` for integrating with Azure Key Vault and HashiCorp Vault via `ISecretProvider` - ---- - -## EIP Pattern: Message Filter (Security) - -> *"A security-focused Message Filter removes or rejects messages that contain dangerous content before they reach the processing pipeline."* - -``` - ┌──────────────┐ ┌──────────────────┐ ┌──────────────┐ - │ Ingress │────▶│ Input Sanitizer │────▶│ Payload │ - │ (raw) │ │ (strip scripts, │ │ Size Guard │ - └──────────────┘ │ SQL, HTML) │ └──────┬───────┘ - └──────────────────┘ │ - ▼ - ┌──────────────────┐ - │ Pipeline │ - │ (safe payload) │ - └──────────────────┘ -``` - -The sanitizer and size guard act as a security gateway. They run before any business logic, ensuring that only clean, appropriately-sized payloads enter the pipeline. - ---- - -## Platform Implementation - -### IInputSanitizer +## Key Types ```csharp // src/Security/IInputSanitizer.cs @@ -45,16 +13,6 @@ public interface IInputSanitizer } ``` -`Sanitize` returns a cleaned copy of the input with dangerous content removed. `IsClean` returns `true` if the input contains no dangerous patterns (i.e., `Sanitize` would not modify it). Both methods are synchronous. - -The sanitizer detects and removes: -- **Script tags**: ``, inline event handlers (`onclick`, `onerror`) -- **SQL injection patterns**: `'; DROP TABLE`, `OR 1=1`, `UNION SELECT`, comment sequences -- **HTML entities**: encoded characters used to bypass text-based filters (`<`, `<`) -- **Control characters**: null bytes, Unicode direction overrides - -### IPayloadSizeGuard - ```csharp // src/Security/IPayloadSizeGuard.cs public interface IPayloadSizeGuard @@ -64,8 +22,6 @@ public interface IPayloadSizeGuard } ``` -### PayloadTooLargeException - ```csharp // src/Security/PayloadTooLargeException.cs public sealed class PayloadTooLargeException : Exception @@ -75,10 +31,6 @@ public sealed class PayloadTooLargeException : Exception } ``` -When the payload exceeds the configured limit, the guard throws `PayloadTooLargeException`. This is a non-retryable error — the message is dead-lettered with `DeadLetterReason.ValidationFailed`. - -### JwtOptions - ```csharp // src/Security/JwtOptions.cs public sealed class JwtOptions @@ -92,104 +44,78 @@ public sealed class JwtOptions } ``` -JWT authentication is used at the Gateway API layer. Tokens carry tenant identity (Tutorial 32) and scopes. The `JwtOptions` configure validation parameters and are bound from the `"Jwt"` configuration section via `SectionName`. The `ClockSkew` property controls how much clock drift is tolerated when validating token expiration. +## Exercises -### Security.Secrets +### 1. InputSanitizer — Sanitize RemovesScriptTags ```csharp -// src/Security.Secrets/ISecretProvider.cs -public interface ISecretProvider -{ - Task GetSecretAsync( - string key, string? version = null, CancellationToken ct = default); - Task SetSecretAsync( - string key, string value, - IReadOnlyDictionary? metadata = null, - CancellationToken ct = default); - Task DeleteSecretAsync(string key, CancellationToken ct = default); - Task> ListSecretKeysAsync( - string? prefix = null, CancellationToken ct = default); -} - -public sealed record SecretEntry( - string Key, - string Value, - string Version, - DateTimeOffset CreatedAt, - DateTimeOffset? ExpiresAt = null, - IReadOnlyDictionary? Metadata = null) -{ - public bool IsExpired => ExpiresAt.HasValue && ExpiresAt.Value <= DateTimeOffset.UtcNow; -} -``` - -`GetSecretAsync` returns a `SecretEntry?` containing the value along with version, creation timestamp, and metadata — or `null` if the key does not exist. The optional `version` parameter allows retrieving a specific version of a secret. `SetSecretAsync` returns the newly created `SecretEntry` (with version and timestamp) and accepts optional metadata. `DeleteSecretAsync` returns `true` if the key was deleted, `false` if it did not exist. `ListSecretKeysAsync` returns all known key names, optionally filtered by prefix. +var input = "Hello World"; -Two implementations are provided: -- `AzureKeyVaultSecretProvider` — integrates with Azure Key Vault using managed identity -- `VaultSecretProvider` — integrates with HashiCorp Vault using AppRole or token auth - -Secrets are never stored in configuration files or environment variables in production. The `ISecretProvider` abstraction allows swapping providers without code changes. - ---- +var result = _sanitizer.Sanitize(input); -## Scalability Dimension - -The sanitizer and size guard are **stateless and CPU-bound** — they inspect each payload independently. They run as early pipeline stages, rejecting bad messages before expensive processing occurs. This protects downstream systems from wasted work. In high-throughput deployments, sanitization can be the bottleneck; profiling guides replica count. +Assert.That(result, Does.Not.Contain(""; -Sanitization runs **before the message is Acked**. Callers can use `IsClean` to check whether the input would be modified, and `Sanitize` to obtain the cleaned payload. If the size guard rejects a message, it is Nacked and dead-lettered in a single atomic operation — the original message is never lost, just quarantined for inspection. +Assert.That(_sanitizer.IsClean(dirty), Is.False); +``` ---- +### 3. InputSanitizer — IsClean ReturnsTrueForClean -## Lab +```csharp +var clean = "Hello, this is perfectly safe text."; -> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial33/Lab.cs`](../tests/TutorialLabs/Tutorial33/Lab.cs) +Assert.That(_sanitizer.IsClean(clean), Is.True); +``` -**Objective:** Trace the input sanitization pipeline, analyze how defense-in-depth protects **message atomicity** from injection attacks, and evaluate secret management for **scalable** multi-environment deployments. +### 4. PayloadSizeGuard — Enforce PassesForSmallPayload -### Step 1: Trace XSS Sanitization +```csharp +var guard = new PayloadSizeGuard( + Options.Create(new PayloadSizeOptions { MaxPayloadBytes = 1024 })); -A payload contains `` embedded in a JSON string value. Open `src/Security/InputSanitizer.cs` and trace: +var smallPayload = new string('x', 100); -1. How does the sanitizer detect the `