Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 89 additions & 157 deletions EnterpriseIntegrationPlatform/tutorials/01-introduction.md
Original file line number Diff line number Diff line change
@@ -1,188 +1,120 @@
# Tutorial 01 — Introduction to Enterprise Integration

## What You'll Learn

- What enterprise integration is and why it matters
- The messaging approach to integration
- The Enterprise Integration Patterns (EIP) book and its relevance
- How this platform implements EIP patterns in a modern .NET 10 architecture

---

## What Is Enterprise Integration?

Enterprise integration is the challenge of connecting multiple applications, services, and systems so they can work together and share data. In any organization, different systems need to communicate:

- An **order system** creates an order → the **warehouse system** needs to know about it
- A **CRM** updates a customer record → the **billing system** needs the change
- An **external partner** sends an invoice → your **ERP** needs to process it

### The Four Integration Styles

The EIP book identifies four fundamental integration styles:

| Style | Description | Example |
|-------|-------------|---------|
| **File Transfer** | Systems share data by writing and reading files | Nightly CSV export/import |
| **Shared Database** | Systems read/write to the same database | Two apps sharing a SQL table |
| **Remote Procedure Invocation** | Systems call each other's APIs directly | REST API calls, gRPC |
| **Messaging** | Systems communicate through an intermediary message broker | Kafka topics, NATS subjects |

### Why Messaging Wins

While all four styles work, **messaging** has unique advantages for enterprise integration:

- **Loose Coupling** — Sender and receiver don't need to know about each other
- **Reliability** — Messages persist in the broker even if the receiver is offline
- **Scalability** — Add more consumers to handle increased load
- **Flexibility** — Route, transform, filter, and enrich messages in transit
- **Resilience** — If a system crashes, messages wait in the queue until it recovers

---

## The Enterprise Integration Patterns Book

Published in 2003 by Gregor Hohpe and Bobby Woolf, *Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions* catalogs 65 patterns for messaging-based integration. These patterns are timeless — they apply whether you use Kafka, RabbitMQ, NATS, or any other message broker.

The patterns are organized into categories:

```
Enterprise Integration Patterns
├── Integration Styles (File Transfer, Shared DB, RPC, Messaging)
├── Messaging Systems (Channel, Message, Pipes & Filters, Router, Translator)
├── Messaging Channels (Point-to-Point, Pub-Sub, Dead Letter, ...)
├── Message Construction (Command, Document, Event, Request-Reply, ...)
├── Message Routing (Content-Based Router, Filter, Splitter, Aggregator, ...)
├── Message Transformation (Envelope Wrapper, Enricher, Normalizer, ...)
├── Messaging Endpoints (Gateway, Consumer types, Dispatcher, ...)
└── System Management (Control Bus, Wire Tap, Message History, ...)
Enterprise integration connects applications through messaging. This platform implements 65+ EIP patterns in .NET 10.

## Key Types

```csharp
// The universal message wrapper — every message in the platform is an IntegrationEnvelope<T>
// src/Contracts/IntegrationEnvelope.cs
public record IntegrationEnvelope<T>
{
public Guid MessageId { get; init; }
public Guid CorrelationId { get; init; }
public Guid? CausationId { get; init; }
public string Source { get; init; }
public string MessageType { get; init; }
public T Payload { get; init; }
public MessagePriority Priority { get; init; }
public string SchemaVersion { get; init; }
public MessageIntent? Intent { get; init; }
public IReadOnlyDictionary<string, string> Metadata { get; init; }
}

// Publish and consume via broker abstractions
// src/Ingestion/IMessageBrokerProducer.cs
public interface IMessageBrokerProducer
{
Task PublishAsync<T>(IntegrationEnvelope<T> envelope, string topic, CancellationToken ct = default);
}

// src/Ingestion/IMessageBrokerConsumer.cs
public interface IMessageBrokerConsumer : IAsyncDisposable
{
Task SubscribeAsync<T>(string topic, Func<IntegrationEnvelope<T>, Task> handler, CancellationToken ct = default);
}
```

### Why These Patterns Still Matter

Despite being 20+ years old, these patterns are the foundation of every modern integration platform:

- **Apache Camel** implements them in Java
- **MuleSoft** and **Azure Integration Services** implement them as cloud services
- **Microsoft BizTalk Server** implemented them (now legacy)
- **This platform** implements them in modern .NET 10 with cloud-native architecture
## Exercises

---

## This Platform: A Modern EIP Implementation
### 1. Create an envelope and verify auto-generated fields

The Enterprise Integration Platform replaces legacy middleware (like BizTalk Server) with a modern, cloud-native architecture:
```csharp
var envelope = IntegrationEnvelope<string>.Create(
payload: "Hello, EIP!",
source: "Tutorial01",
messageType: "greeting.created");

```
Enterprise Integration Platform
┌──────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────┐ ┌───────────┐ ┌───────────┐ ┌──────────────────┐ │
│ │ Ingress │───▶│ Broker │───▶│ Temporal │───▶│ Activities │ │
│ │ Adapters │ │ Layer │ │ Workflows │ │ (Transform/Route)│ │
│ └──────────┘ └───────────┘ └───────────┘ └────────┬─────────┘ │
│ ▲ │ │ │
│ │ ▼ ▼ │
│ ┌────┴─────┐ ┌───────────┐ ┌──────────────────┐ │
│ │ External │ │ DLQ │ │ Connectors │ │
│ │ Systems │ │ Topics │ │ (HTTP/SFTP/Email)│ │
│ └──────────┘ └───────────┘ └──────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Assert.That(envelope.MessageId, Is.Not.EqualTo(Guid.Empty));
Assert.That(envelope.CorrelationId, Is.Not.EqualTo(Guid.Empty));
Assert.That(envelope.Timestamp, Is.Not.EqualTo(default(DateTimeOffset)));
Assert.That(envelope.Source, Is.EqualTo("Tutorial01"));
Assert.That(envelope.Payload, Is.EqualTo("Hello, EIP!"));
```

### Key Technology Choices
### 2. Check default values on a new envelope

| Component | Technology | Why |
|-----------|-----------|-----|
| **Runtime** | .NET 10 (C# 14) | Latest LTS, high performance, cross-platform |
| **Orchestration** | .NET Aspire | Single-command local dev with all dependencies |
| **Event Streaming** | Apache Kafka | Best for ordered event streams and audit logs |
| **Task Delivery** | NATS JetStream / Pulsar | No head-of-line blocking for task queues |
| **Workflows** | Temporal.io | Durable execution with saga compensation |
| **Storage** | Apache Cassandra | Write-optimized distributed storage |
| **Observability** | OpenTelemetry + Grafana | End-to-end distributed tracing and metrics |
| **AI** | Ollama + RagFlow | Self-hosted RAG for developer assistance |
```csharp
var envelope = IntegrationEnvelope<string>.Create("payload", "source", "type");

### Design Principles

1. **Zero Message Loss** — Every accepted message is either delivered or routed to a Dead Letter Queue. No silent drops.
2. **Ack/Nack Loopback** — Every integration publishes an acknowledgment (Ack) on success or negative acknowledgment (Nack) on failure.
3. **Atomic Processing** — Temporal workflows ensure all-or-nothing execution with compensation.
4. **Multi-Tenant Isolation** — Tenant A's messages never mix with Tenant B's.
5. **AI-Driven Generation** — Write a minimal spec, let AI generate production-ready integrations.

---
Assert.That(envelope.SchemaVersion, Is.EqualTo("1.0"));
Assert.That(envelope.Priority, Is.EqualTo(MessagePriority.Normal));
Assert.That(envelope.CausationId, Is.Null);
Assert.That(envelope.ReplyTo, Is.Null);
Assert.That(envelope.Metadata, Is.Empty);
```

## How the Platform Maps to EIP Patterns
### 3. Set message intent using `with` expression

Every EIP pattern has a corresponding platform component:
```csharp
var command = IntegrationEnvelope<string>.Create(
"PlaceOrder", "OrderService", "order.place") with
{
Intent = MessageIntent.Command,
};

| EIP Pattern | Platform Component | Project |
|-------------|-------------------|---------|
| Message | `IntegrationEnvelope<T>` | `src/Contracts/` |
| Message Channel | Broker abstraction | `src/Ingestion/` |
| Content-Based Router | `IContentBasedRouter` | `src/Processing.Routing/` |
| Message Translator | `IMessageTranslator<TIn, TOut>` | `src/Processing.Translator/` |
| Splitter | `IMessageSplitter<T>` | `src/Processing.Splitter/` |
| Aggregator | `IMessageAggregator<TItem, TAggregate>` | `src/Processing.Aggregator/` |
| Dead Letter Channel | `IDeadLetterPublisher<T>` | `src/Processing.DeadLetter/` |
| Process Manager | Temporal Workflows | `src/Workflow.Temporal/` |
| Channel Adapter | `IConnector` | `src/Connector.*` |
| Wire Tap | OpenTelemetry | `src/Observability/` |
Assert.That(command.Intent, Is.EqualTo(MessageIntent.Command));
```

You'll explore each of these in detail throughout this course.
### 4. Verify platform types exist (EIP pattern mapping)

---
```csharp
// EIP: Message Channel → IMessageBrokerProducer
var producerType = typeof(IMessageBrokerProducer);
Assert.That(producerType.IsInterface, Is.True);
Assert.That(producerType.GetMethod("PublishAsync"), Is.Not.Null);

## What You'll Build
// EIP: Message Endpoint → IMessageBrokerConsumer
var consumerType = typeof(IMessageBrokerConsumer);
Assert.That(consumerType.IsInterface, Is.True);
Assert.That(consumerType.GetMethod("SubscribeAsync"), Is.Not.Null);
```

By the end of this course, you'll understand how to:
### 5. Verify IntegrationEnvelope is a C# record with value equality

1. **Design** integration solutions using EIP patterns
2. **Build** message-driven pipelines with routing, transformation, and delivery
3. **Configure** message brokers (Kafka, NATS, Pulsar) for different workloads
4. **Orchestrate** complex workflows with Temporal and saga compensation
5. **Monitor** message flow with OpenTelemetry and the OpenClaw UI
6. **Deploy** to Kubernetes with Helm and Kustomize
7. **Secure** integrations with input sanitization, encryption, and multi-tenancy
8. **Scale** with competing consumers, throttling, and backpressure
9. **Test** integrations with unit, contract, integration, and load tests
10. **Operate** in production with disaster recovery and performance profiling
```csharp
var envelopeType = typeof(IntegrationEnvelope<string>);
Assert.That(envelopeType.IsClass, Is.True);

---
var equatable = typeof(IEquatable<IntegrationEnvelope<string>>);
Assert.That(equatable.IsAssignableFrom(envelopeType), Is.True);
```

## Lab

> 💻 **Runnable lab:** [`tests/TutorialLabs/Tutorial01/Lab.cs`](../tests/TutorialLabs/Tutorial01/Lab.cs)

**Objective:** Map EIP pattern categories to concrete platform components and trace how the Pipes and Filters architecture enables scalable message processing.

### Step 1: Map Patterns to Projects

Open [`docs/eip-mapping.md`](../docs/eip-mapping.md). For each of the following EIP categories, identify the `src/` project that implements it and the primary interface it exposes:

| Category | Project | Interface |
|----------|---------|-----------|
| Message Construction | `src/Contracts/` | ? |
| Content-Based Router | `src/Processing.Routing/` | ? |
| Message Translator | `src/Processing.Translator/` | ? |
| Splitter | `src/Processing.Splitter/` | ? |
| Dead Letter Channel | `src/Processing.DeadLetter/` | ? |
Run the full lab: [`tests/TutorialLabs/Tutorial01/Lab.cs`](../tests/TutorialLabs/Tutorial01/Lab.cs)

### Step 2: Trace the Pipes and Filters Chain

Open [`docs/architecture-overview.md`](../docs/architecture-overview.md) and trace how a single message flows through the platform: Ingress → Broker → Workflow → Activities → Connectors. For each stage, write down which EIP pattern it implements and how the platform guarantees **atomicity** (hint: look at Temporal workflows and Ack/Nack).

### Step 3: Evaluate Scalability Points

Identify three places in the architecture where **horizontal scaling** is possible without code changes. Consider: broker partitions, Competing Consumers (`src/Processing.CompetingConsumers/`), and workflow workers. For each, explain what happens to in-flight messages when a new instance is added.
```bash
dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial01.Lab"
```

## Exam

> 💻 **Coding exam:** [`tests/TutorialLabs/Tutorial01/Exam.cs`](../tests/TutorialLabs/Tutorial01/Exam.cs)
Coding challenges: [`tests/TutorialLabs/Tutorial01/Exam.cs`](../tests/TutorialLabs/Tutorial01/Exam.cs)

Complete the coding challenges in the exam file. Each challenge is a failing test — make it pass by writing the correct implementation inline.
```bash
dotnet test tests/TutorialLabs/TutorialLabs.csproj --filter "FullyQualifiedName~Tutorial01.Exam"
```

---

Expand Down
Loading
Loading