Realm Sync Server

Real-time, offline-first sync for Realm-backed apps. Socket.IO + MongoDB with timestamp-based conflict resolution. Built for mobile and web.

✨ Highlights

🔄 Real-time bi-directional sync over WebSockets
📱 Offline-first with durable outbox queue
⚡ Optimistic updates; instant UI then server-verify
🔀 Conflict resolution via last-write-wins timestamps
🎯 Flexible subscriptions (server-side filtering)
🔐 JWT auth, 🛡️ rate limiting, 📝 audit log
🌐 Optional Azure Web PubSub for horizontal scale

🚀 Quick Start

Prerequisites

Node.js 18+
MongoDB 5.0+ (local, Atlas, or Azure Cosmos DB vCore)
Azure subscription (optional; for Web PubSub scaling or Cosmos DB vCore)

💡 Using Azure Cosmos DB vCore? See AZURE_COSMOS_VCORE.md for detailed setup, tuning, and migration guide.

Installation

npm install

Configuration

Create a .env file:

# MongoDB Configuration
MONGODB_URI=mongodb://localhost:27017/realm-sync

# Azure Web PubSub (Optional - for horizontal scaling)
WEB_PUBSUB_CONNECTION_STRING=Endpoint=https://your-pubsub.webpubsub.azure.com;AccessKey=YOUR_KEY;Version=1.0;
WEB_PUBSUB_HUB_NAME=Hub

# Server Configuration
PORT=3000
NODE_ENV=development

# Security (Production)
AUTH_JWT_SECRET=your-secret-key-here

# Performance Tuning
MAX_CONNECTIONS_PER_USER=10
SYNC_RATE_LIMIT_MAX=50
SYNC_RATE_LIMIT_WINDOW_MS=10000

Start Development Server

npm run dev:server

Server starts on http://localhost:3000

Health check: http://localhost:3000/health
Metrics: http://localhost:3000/stats

Try It: Example Clients

Open multiple terminals to simulate real-time sync:

# Terminal 1 - First client
npm run dev:client

# Terminal 2 - Second client (simulates another device)
npx ts-node client/example.ts demo-user-2

# Terminal 3 - Third client (observe real-time sync)
npx ts-node client/example.ts demo-user-3

Watch as changes made in one client instantly appear in all others! 🎉

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        Client Layer                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐          │
│  │ Flutter  │  │   Web    │  │  React   │  │ Node.js  │          │
│  │  Mobile  │  │ Browser  │  │  Native  │  │  Client  │          │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘          │
│       │             │               │             │              │
│       └─────────────┴───────────────┴─────────────┘              │
│                          │                                       │
│                   Socket.IO (WebSocket/HTTP)                     │
└──────────────────────────┼────────────────────────────────────── ┘
                           │
┌──────────────────────────▼──────────────────────────────────────┐
│                     Sync Server Layer                            │
│  ┌────────────────────────────────────────────────────────┐    │
│  │ Socket.IO Server + TypeScript + Express               │    │
│  ├────────────────────────────────────────────────────────┤    │
│  │ • Authentication & Authorization                       │    │
│  │ • Rate Limiting & Security                            │    │
│  │ • Connection Management (per-user rooms)              │    │
│  │ • Subscription Management (FLX filtering)             │    │
│  │ • Conflict Resolution (timestamp-based)               │    │
│  │ • Change Broadcasting                                  │    │
│  │ • Query Translation (RQL → MongoDB)                   │    │
│  └────────────────────────────────────────────────────────┘    │
└──────────────────────────┬──────────────────────────────────────┘
                           │
┌──────────────────────────▼──────────────────────────────────────┐
│                    Persistence Layer                             │
│  ┌──────────────┐        ┌──────────────┐                      │
│  │   MongoDB    │        │   Change     │                      │
│  │              │        │   Audit Log  │                      │
│  │ • Documents  │        │ • Timestamps │                      │
│  │ • Collections│◄───────┤ • Operations │                      │
│  │ • Indexes    │        │ • User IDs   │                      │
│  └──────────────┘        └──────────────┘                      │
└──────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────┐
│              Optional: Azure Web PubSub                          │
│  (Horizontal Scaling for 1000+ Concurrent Connections)          │
│                                                                   │
│  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐               │
│  │ Server │  │ Server │  │ Server │  │ Server │               │
│  │ Node 1 │  │ Node 2 │  │ Node 3 │  │ Node N │               │
│  └────┬───┘  └────┬───┘  └────┬───┘  └────┬───┘               │
│       └───────────┴───────────┴───────────┘                     │
│                        │                                         │
│              Azure Web PubSub Hub                               │
└──────────────────────────────────────────────────────────────────┘

Data Flow

Client makes change (optimistic) → sends to server
Server validates → resolves conflicts via sync_updated_at
Server persists (MongoDB) → writes audit log
Server broadcasts to subscribers
Clients apply + update UI

🧩 Extension System

Extend the sync server with custom plugins! Add authentication, validation, analytics, notifications, and more without modifying core code.

import { SyncServer } from "./server/sync-server";
import { SyncServerPlugin } from "./extensions";

const myPlugin: SyncServerPlugin = {
  name: "my-plugin",
  version: "1.0.0",

  hooks: {
    beforeChange: async (socket, change) => {
      // Validate data before applying
      if (!isValid(change.data)) {
        throw new Error("Invalid data");
      }
    },

    afterChange: async (socket, change) => {
      // Send notification after change
      await sendNotification(change);
    },
  },

  customEvents: [
    {
      event: "custom:action",
      handler: async (socket, data, callback) => {
        // Handle custom event from client
        callback?.({ success: true });
      },
    },
  ],
};

const server = new SyncServer(...);
server.registerPlugin(myPlugin);
server.start();

Available hooks:

beforeJoin, afterJoin - User authentication lifecycle
beforeChange, afterChange - Data validation and processing
beforeUpdateSubscriptions, afterUpdateSubscriptions - Subscription management
onDisconnect - Cleanup on disconnect
onServerStart, onServerStop - Server lifecycle

📚 Full documentation: extensions/README.md

Example plugins: Audit logging, analytics, permissions, presence tracking, activity feeds, and more!

Project Structure

sync-implementation/
├── server/
│   ├── index.ts                 # Server entry point
│   ├── index-with-plugins.ts    # Example with plugins
│   ├── sync-server.ts           # Main sync server logic
│   └── database.ts              # MongoDB operations
├── extensions/                   # 🆕 Plugin system
│   ├── README.md                # Plugin development guide
│   ├── plugin-types.ts          # Plugin interfaces
│   ├── plugin-manager.ts        # Plugin orchestration
│   └── examples/                # Example plugins
│       ├── audit-logger.ts
│       ├── analytics.ts
│       ├── data-validation.ts
│       └── notifications.ts
├── client/
│   ├── sync-client.ts           # Client SDK
│   └── example.ts               # Example usage
├── shared/
│   ├── types.ts                 # Shared TypeScript types
│   └── conflict-resolver.ts     # Conflict resolution strategies
├── tests/
│   ├── database.test.ts
│   └── integration.test.ts
└── package.json

📋 Use Cases

Offline-first mobile (Flutter/React Native)
Collaborative, multi-user apps
IoT device-to-cloud sync
Field service with spotty connectivity
Chats and activity feeds
Multi-tenant SaaS with scoped access

🎯 Key Concepts

Conflict Resolution

Last-write-wins via sync_updated_at (UTC ms):

if (local.sync_updated_at >= remote.sync_updated_at) {
  // Keep local version (newer or equal)
} else {
  // Apply remote version (remote is newer)
}

Subscriptions (Flexible Sync)

Filter server data with MongoDB-style queries:

// Client subscribes to only their own tasks
socket.emit("sync:subscribe", {
  collection: "tasks",
  filter: "userId == $0",
  args: [currentUserId],
});

Historical Sync

Catch up after offline periods:

socket.emit("sync:get_changes", {
  collection: "tasks",
  since: lastSyncTimestamp,
  limit: 500,
});

📡 API Reference

REST

Endpoint	Method	Description
`/health`	GET	Health check
`/ready`	GET	Readiness probe (MongoDB)
`/stats`	GET	Metrics & counters
`/api/negotiate?userId=<id>`	GET	Web PubSub token

Example Response: GET /stats

{
  "totalChanges": 15420,
  "syncedChanges": 15380,
  "pendingChanges": 40,
  "activeConnections": 23,
  "activeUsers": ["user-1", "user-2", "user-3"]
}

Socket.IO

🔼 Client → Server

sync:join - Join sync session

socket.emit(
  "sync:join",
  { userId: "user-123", token: "jwt-token" },
  (response) => {
    console.log(response); // { success: true, timestamp: 1234567890 }
  }
);

sync:change - Send single change

const change = {
  operation: "update",
  collection: "tasks",
  documentId: "task-1",
  data: { title: "Updated", completed: true },
  timestamp: Date.now(),
};

socket.emit("sync:change", change, (ack) => {
  console.log(ack); // { success: true, changeId: '...' }
});

sync:changeBatch - Send multiple changes efficiently

socket.emit('sync:changeBatch', {
  changes: [
    { operation: 'update', collection: 'tasks', documentId: 'task-1', data: {...} },
    { operation: 'delete', collection: 'tasks', documentId: 'task-2' }
  ]
}, (response) => {
  console.log(response.results); // Array of results per change
});

sync:subscribe - Subscribe to filtered data

socket.emit("sync:subscribe", {
  collection: "tasks",
  filter: "userId == $0 AND status == $1",
  args: ["user-123", "active"],
});

sync:get_changes - Fetch historical changes

socket.emit(
  "sync:get_changes",
  {
    userId: "user-123",
    collection: "tasks",
    since: 1234567890,
    limit: 500,
  },
  (response) => {
    console.log(response.changes.length);
    console.log(response.latestTimestamp);
    console.log(response.hasMore);
  }
);

🔽 Server → Client

sync:bootstrap - Initial data load

socket.on("sync:bootstrap", (payload) => {
  console.log(payload.collection); // 'tasks'
  console.log(payload.data); // Array of documents
});

sync:changes - Real-time change notifications

socket.on("sync:changes", (changes) => {
  changes.forEach((change) => {
    console.log(change.operation); // 'update' | 'delete'
    console.log(change.collection); // 'tasks'
    console.log(change.documentId);
    console.log(change.data);
  });
});

💻 Client SDK

Basic Example

import { SyncClient } from "./client/sync-client";

// Initialize client
const client = new SyncClient(
  "http://localhost:3000", // Server URL
  "user-123", // User ID
  "mongodb://localhost:27017/myapp" // Local MongoDB (for offline queue)
);

await client.initialize();
await client.connect();

// Make changes (automatically synced)
await client.makeChange("insert", "tasks", "task-1", {
  title: "Buy groceries",
  completed: false,
  userId: "user-123",
  sync_updated_at: Date.now(),
});

await client.makeChange("update", "tasks", "task-1", {
  completed: true,
  sync_updated_at: Date.now(),
});

await client.makeChange("delete", "tasks", "task-1");

// Monitor connection
console.log("Online:", client.isOnline());
console.log("Pending:", client.getPendingChangesCount());
console.log("Last sync:", new Date(client.getLastSyncTimestamp()));

// Cleanup
await client.disconnect();

Flutter/Dart

See the Dart client SDK for Flutter integration.

final realmSync = RealmSync(
  realm: realm,
  socket: socket,
  userId: userId,
  configs: [
    SyncCollectionConfig<ChatMessage>(
      results: realm.all<ChatMessage>(),
      collectionName: 'chat_messages',
      idSelector: (m) => m.id,
      needsSync: (m) => m.syncUpdateDb == true,
      fromServerMap: (map) => ChatMessage(
        map['_id'] as String,
        map['message'] as String,
        map['senderId'] as String,
        map['timestamp'] as int,
      ),
    ),
  ],
);

realmSync.start();

🧪 Testing

# Run all tests
npm test

# Watch mode (for development)
npm run test:watch

# Integration tests only
npm run test:integration

# Load testing
npm run test:load

# Coverage report
npm run test:coverage

Test Structure

tests/
├── database.test.ts       # MongoDB operations
├── integration.test.ts    # End-to-end sync scenarios
├── crud-operations.test.ts # Create, read, update, delete
├── optimistic-updates.test.ts # Optimistic UI patterns
├── benchmarks/            # Performance tests
├── e2e/                   # Full system tests
└── load/                  # Stress testing

⚙️ Configuration

Environment Variables

Variable	Description	Required	Default
`MONGODB_URI`	MongoDB connection string	✅ Yes	-
`WEB_PUBSUB_CONNECTION_STRING`	Azure Web PubSub connection	⚠️ Production	-
`WEB_PUBSUB_HUB_NAME`	Web PubSub hub name	⚠️ Production	`Hub`
`PORT`	Server HTTP port	❌ No	`3000`
`NODE_ENV`	Environment mode	❌ No	`development`
`AUTH_JWT_SECRET`	JWT signing secret	⚠️ Production	-
`MAX_CONNECTIONS_PER_USER`	Connection limit per user	❌ No	`10` (prod), `100` (dev)
`MAX_CONNECTIONS_PER_IP`	Connection limit per IP	❌ No	`50` (prod), `500` (dev)
`SYNC_RATE_LIMIT_MAX`	Max changes per window	❌ No	`50`
`SYNC_RATE_LIMIT_WINDOW_MS`	Rate limit window (ms)	❌ No	`10000`
`RATE_LIMIT_DISABLED`	Disable rate limiting	❌ No	`false`
`LOG_LEVEL`	Logging verbosity	❌ No	`info`
`ALLOWED_ORIGINS`	CORS allowed origins (comma-separated)	⚠️ Production	`*` (dev)

Advanced Configuration

Create config/production.json for environment-specific settings:

{
  "server": {
    "port": 3000,
    "corsOrigins": ["https://app.example.com"]
  },
  "sync": {
    "maxBatchSize": 100,
    "changeRetentionDays": 30,
    "enableOptimisticLocking": true
  },
  "mongodb": {
    "poolSize": 10,
    "socketTimeoutMS": 45000
  }
}

🚀 Deployment

Build for Production

npm run build
npm start

Docker Deployment

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY dist ./dist
EXPOSE 3000
CMD ["node", "dist/server/index.js"]

docker build -t realm-sync-server .
docker run -p 3000:3000 \
  -e MONGODB_URI="mongodb://..." \
  -e AUTH_JWT_SECRET="your-secret" \
  realm-sync-server

Azure App Service

Deploy to Azure App Service

# Login to Azure
az login

# Create resource group
az group create --name realm-sync-rg --location eastus

# Create App Service plan
az appservice plan create \
  --name realm-sync-plan \
  --resource-group realm-sync-rg \
  --sku B1 \
  --is-linux

# Create web app
az webapp create \
  --name realm-sync-server \
  --resource-group realm-sync-rg \
  --plan realm-sync-plan \
  --runtime "NODE:18-lts"

# Configure environment variables
az webapp config appsettings set \
  --name realm-sync-server \
  --resource-group realm-sync-rg \
  --settings \
    MONGODB_URI="..." \
    AUTH_JWT_SECRET="..." \
    NODE_ENV="production"

# Deploy code
az webapp deployment source config-zip \
  --name realm-sync-server \
  --resource-group realm-sync-rg \
  --src dist.zip

Kubernetes

Kubernetes manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: realm-sync-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: realm-sync
  template:
    metadata:
      labels:
        app: realm-sync
    spec:
      containers:
        - name: server
          image: realm-sync-server:latest
          ports:
            - containerPort: 3000
          env:
            - name: MONGODB_URI
              valueFrom:
                secretKeyRef:
                  name: sync-secrets
                  key: mongodb-uri
            - name: AUTH_JWT_SECRET
              valueFrom:
                secretKeyRef:
                  name: sync-secrets
                  key: jwt-secret
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: realm-sync-service
spec:
  selector:
    app: realm-sync
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
  type: LoadBalancer

Azure Web PubSub (scaling)

# Create Web PubSub resource
az webpubsub create \
  --name realm-sync-pubsub \
  --resource-group realm-sync-rg \
  --location eastus \
  --sku Standard_S1

# Get connection string
az webpubsub key show \
  --name realm-sync-pubsub \
  --resource-group realm-sync-rg \
  --query primaryConnectionString

🐛 Troubleshooting

Common Issues

Connection refused / Cannot connect to server

Check server is running: npm run dev:server
Verify port is not blocked: lsof -i :3000
Check firewall settings
Ensure MongoDB is accessible: mongosh $MONGODB_URI

Changes not syncing between clients

Verify both clients are connected: Check client.isOnline()
Check server logs for errors
Verify sync_updated_at timestamps are being set
Confirm subscription filters match the data
Check network tab in browser DevTools for WebSocket errors

High latency or slow sync

Enable MongoDB indexes on sync_updated_at and _id
Reduce batch size if memory is constrained
Check network latency: ping your-server.com
Monitor server metrics at /stats
Consider enabling Azure Web PubSub for horizontal scaling

Authentication failures

Verify AUTH_JWT_SECRET is set in production
Check JWT token expiration
Ensure token is passed in sync:join event
Validate token format (should be Bearer <token>)

Debug Mode

Enable verbose logging:

LOG_LEVEL=debug npm run dev:server

Health Check Commands

# Server health
curl http://localhost:3000/health

# MongoDB connection
mongosh $MONGODB_URI --eval "db.adminCommand('ping')"

# Active connections
curl http://localhost:3000/stats | jq '.activeConnections'

📊 Performance & Scaling

Optimization Tips

Database Indexes: Create compound indexes on sync_updated_at and collection-specific fields
```
db.tasks.createIndex({ sync_updated_at: 1, userId: 1 });
```
Batch Operations: Use sync:changeBatch for bulk updates (10x faster than individual changes)
Connection Pooling: Configure MongoDB pool size based on concurrent users
```
MONGODB_URI=mongodb://...?maxPoolSize=50
```

Rate Limiting: Adjust limits based on your use case

SYNC_RATE_LIMIT_MAX=100          # Higher for power users
SYNC_RATE_LIMIT_WINDOW_MS=5000   # Shorter window for stricter limits

Change Retention: Clean up old change logs automatically
```
// Runs daily by default, keeps last 30 days
```

Benchmarks

Tested on Azure Standard_B2s (2 vCPU, 4 GB RAM):

Metric	Value
Concurrent connections	1,000+
Changes per second	5,000+
Average latency	<50ms
Memory per connection	~1MB
MongoDB write throughput	10,000 ops/s

Scaling Strategy

Vertical: Increase server resources (CPU/RAM)
Horizontal: Deploy multiple instances behind load balancer + Azure Web PubSub
Database: Use MongoDB Atlas auto-scaling or Azure Cosmos DB vCore sharding

Azure Cosmos DB MongoDB vCore

For Azure-native deployments, Cosmos DB vCore provides:

Native MongoDB compatibility (wire protocol 5.0+)
Full feature support: Change streams, TTL indexes, transactions
Integrated scaling: Auto-scale compute and storage independently
Cost-effective: Pay per vCore + storage (no RU complexity)
Performance tiers: M25 (2 vCores, $108/mo) → M50 (8 vCores, $434/mo) → M80+ for enterprise

When to use vCore:

Need Azure-native integration (VNets, Private Link, Azure AD)
Want predictable pricing (vCore-hours vs. RU/s spikes)
Require 99.995% SLA with zone redundancy
Multi-region reads with local latency (<10ms)

Connection string format:

mongodb://username:password@cluster-name.mongocluster.cosmos.azure.com:10255/?tls=true&authMechanism=SCRAM-SHA-256&retrywrites=false&maxIdleTimeMS=120000

Recommended starting tier: M25 (2 vCores) handles 1-5K concurrent users (~$108/month)

🔒 Security

Production Checklist

✅ Enable JWT authentication (AUTH_JWT_SECRET)
✅ Use HTTPS/WSS in production
✅ Implement CORS whitelist (ALLOWED_ORIGINS)
✅ Enable rate limiting (default: enabled in production)
✅ Validate all user inputs server-side
✅ Use environment variables for secrets (never commit)
✅ Rotate JWT secrets regularly
✅ Monitor failed authentication attempts
✅ Set up MongoDB authentication and network rules
✅ Use Azure Private Endpoints for Web PubSub

Example Secure Configuration

# Production .env
NODE_ENV=production
AUTH_JWT_SECRET=<256-bit-random-secret>
MONGODB_URI=mongodb+srv://user:pass@cluster.mongodb.net/?authSource=admin&ssl=true
ALLOWED_ORIGINS=https://app.example.com,https://mobile.example.com
MAX_CONNECTIONS_PER_USER=5
SYNC_RATE_LIMIT_MAX=30

🤝 Contributing

We welcome contributions! Please see our Contributing Guide.

# Fork the repository
git clone https://github.com/mohit67890/realm-sync-server.git
cd realm-sync-server

# Create feature branch
git checkout -b feature/amazing-feature

# Make changes and test
npm test
npm run test:integration

# Commit with conventional commits
git commit -m "feat: add amazing feature"

# Push and create PR
git push origin feature/amazing-feature

📄 License

MIT — see LICENSE

🙏 Acknowledgments

Inspired by MongoDB Realm Sync
Built with Socket.IO for WebSocket communications
Powered by Azure Web PubSub for scalability
TypeScript SDK patterns from Realm JavaScript

📞 Support & Community

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: See IMPLEMENTATION_GUIDE.md for detailed setup
Examples: Check client/example.ts for usage examples

Built with ❤️ for real-time applications

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
.github/workflows		.github/workflows
docs		docs
examples		examples
extensions		extensions
server		server
shared		shared
tests		tests
.deployment		.deployment
.env.azure.example		.env.azure.example
.env.example		.env.example
.env.test.example		.env.test.example
.gitignore		.gitignore
AZURE_DEPLOYMENT.md		AZURE_DEPLOYMENT.md
README.md		README.md
deploy.cmd		deploy.cmd
jest.config.js		jest.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
web.config		web.config

Folders and files

Latest commit

History

Repository files navigation

Realm Sync Server

✨ Highlights

🚀 Quick Start

Prerequisites

Installation

Configuration

Start Development Server

Try It: Example Clients

🏗️ Architecture

Data Flow

🧩 Extension System

Project Structure

📋 Use Cases

🎯 Key Concepts

Conflict Resolution

Subscriptions (Flexible Sync)

Historical Sync

📡 API Reference

REST

Socket.IO

🔼 Client → Server

🔽 Server → Client

💻 Client SDK

Basic Example

Flutter/Dart

🧪 Testing

Test Structure

⚙️ Configuration

Environment Variables

Advanced Configuration

🚀 Deployment

Build for Production

Docker Deployment

Azure App Service

Kubernetes

Azure Web PubSub (scaling)

🐛 Troubleshooting

Common Issues

Debug Mode

Health Check Commands

📊 Performance & Scaling

Optimization Tips

Benchmarks

Scaling Strategy

Azure Cosmos DB MongoDB vCore

🔒 Security

Production Checklist

Example Secure Configuration

🤝 Contributing

📄 License

🙏 Acknowledgments

📞 Support & Community

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages