Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 18 additions & 39 deletions resources/self_hosting/sharding/configure_replication.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ sidebarTitle: Configure replication
description: Set up replicated shards across multiple Meilisearch instances to ensure high availability and distribute search load.
---

Replication assigns the same shard to multiple remotes in your Meilisearch network. If one remote goes down, another remote holding the same shard continues serving results. This guide covers how to configure replication, common patterns, and what to expect during failover.
Replication assigns the same shard to multiple remotes in your Meilisearch network. This guide covers how to configure replication, common patterns, and scaling read throughput.

<Note>
Replication requires the Meilisearch Enterprise Edition v1.37 or later and a [configured network](/resources/self_hosting/sharding/setup_sharded_cluster).
</Note>

## How replication works

When you configure shards, each shard can be assigned to one or more remotes. If a shard is assigned to multiple remotes, Meilisearch replicates the data to each of them. During a search with `useNetwork: true`, Meilisearch queries each shard exactly once, picking one of the available remotes for each shard. This avoids duplicate results and provides automatic failover.
When you configure shards, each shard can be assigned to one or more remotes. If a shard is assigned to multiple remotes, Meilisearch replicates the data to each of them. During a search, Meilisearch queries each shard exactly once, picking one of the available remotes for each shard (prioritizing the `self`/local remote). This avoids duplicate results.

## Assign shards to multiple remotes

Expand All @@ -36,7 +36,7 @@ curl \

</CodeGroup>

In this configuration, every shard exists on two remotes. If any single instance goes down, all shards remain available.
In this configuration, every shard exists on two remotes. If any single instance goes down, all shard data still exists on another instance.

## Common replication patterns

Expand Down Expand Up @@ -83,26 +83,19 @@ Place replicas in different regions to reduce latency for geographically distrib
}
```

Route search requests to the closest cluster. Both regions hold all data, so either can serve a full result set.
Route search requests to the closest cluster. Both regions hold all data, so either can serve a full result set. By default, Meilisearch prioritizes local search requests and will not transfer the request to a remote server. Make sure your search requests are made on the closest remote instance to ensure this setup is efficient.

## Failover behavior
## Remote availability

When a remote becomes unavailable during a network search:
When a network search runs, Meilisearch builds an internal set of remotes to query: it assigns each shard to a remote, then sends one query per remote with a shard filter. This guarantees that no shard is queried twice and that results are never duplicated.

1. Meilisearch detects the remote is unreachable
2. If another remote holds the same shard, Meilisearch queries that remote instead
3. The search completes with results from all shards, using the available replicas
4. If no remote for a given shard is reachable, results from that shard are missing from the response

Meilisearch does not require manual intervention for failover. When the failed remote comes back online, it automatically rejoins the network and starts serving searches again.
The downside is that there is no automatic fallback. If the remote assigned to a shard is unreachable, that shard's results are missing from the response — Meilisearch does not yet retry using another replica that holds the same shard.

## Scaling read throughput

Replication is the primary way to scale search throughput in Meilisearch. Each replica can independently handle search requests, so adding more replicas increases the total number of concurrent searches your cluster can handle.

To add a new replica for an existing shard:

1. Add the new remote to the network:
To add a new replica for an existing shard, add the new remote and use `addRemotes` to append it to the shard without rewriting the full assignment:

<CodeGroup>

Expand All @@ -112,45 +105,31 @@ curl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
"addRemotes": {
"remotes": {
"ms-03": {
"url": "http://ms-03.example.com:7703",
"searchApiKey": "SEARCH_KEY_03"
"searchApiKey": "SEARCH_KEY_03",
"writeApiKey": "WRITE_KEY_03"
}
}
}'
```

</CodeGroup>

2. Update the shard assignment to include the new remote:

<CodeGroup>

```bash
curl \
-X PATCH 'MEILISEARCH_URL/network' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
},
"shards": {
"shard-a": { "remotes": ["ms-00", "ms-01", "ms-03"] },
"shard-b": { "remotes": ["ms-01", "ms-02"] },
"shard-c": { "remotes": ["ms-02", "ms-00"] }
"shard-a": { "addRemotes": ["ms-03"] }
}
}'
```

</CodeGroup>

This triggers a `NetworkTopologyChange` task that replicates the shard's documents to `ms-03`.

## The leader instance

The leader is responsible for all write operations (document additions, settings changes, index management). Non-leader instances reject writes with a `not_a_leader` error.
The leader is responsible for all write operations (document additions, settings changes, index management). Non-leader instances reject writes with a `not_leader` error.

If the leader goes down:

- **Search continues**: replicas still serve search results for all replicated shards
- **Writes are blocked**: no documents can be added or updated until a leader is available
- **Search may be affected**: if search requests are routed to the downed leader, they will fail
- **Writes are blocked**: no documents can be added or updated until a leader is available. Note that alive remote instances continue to process tasks
- **Manual promotion**: you must designate a new leader by updating the network topology with `PATCH /network` and setting `"leader"` to another instance

<Warning>
Expand Down
70 changes: 51 additions & 19 deletions resources/self_hosting/sharding/manage_network.mdx
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
title: Manage the network
sidebarTitle: Manage the network
description: Add and remove remotes, update shard assignments, and manage your Meilisearch network topology dynamically.
description: Add remotes, update shard assignments, and manage your Meilisearch network topology dynamically. Includes rebalancing behavior and validation rules.
---

Once your [sharded cluster is set up](/resources/self_hosting/sharding/setup_sharded_cluster), you can modify the topology without restarting instances. All topology changes go through `PATCH /network` on the leader instance.

## Add a remote

Use `addRemotes` to add a new instance to the network without rewriting the entire `remotes` configuration:
Include the new remote in the `remotes` object. To assign it to an existing shard, either send the full `remotes` list for that shard, or use `addRemotes` as a convenience to append without rewriting the full list:

<CodeGroup>

Expand All @@ -18,22 +18,36 @@ curl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
"addRemotes": {
"remotes": {
"ms-03": {
"url": "http://ms-03.example.com:7703",
"searchApiKey": "SEARCH_KEY_03"
"searchApiKey": "SEARCH_KEY_03",
"writeApiKey": "WRITE_KEY_03"
}
},
"shards": {
"shard-a": { "addRemotes": ["ms-03"] }
}
}'
```

</CodeGroup>

After adding the remote, update your shard configuration to assign shards to it.
<Note>
`addRemotes` and `removeRemotes` are write-only convenience fields. They are applied on top of the existing shard configuration and are never returned by `GET /network`, which always returns the full `remotes` list for each shard.
</Note>

## Update shard assignments

Each shard object in a `PATCH /network` request accepts three fields:

## Remove a remote
| Field | Type | Behavior |
|-------|------|----------|
| `remotes` | array | Full replacement of the shard's remote list |
| `addRemotes` | array | Adds remotes to the existing list |
| `removeRemotes` | array | Removes remotes from the existing list, applied after `addRemotes` |

Use `removeRemotes` to take an instance out of the network:
Shards not included in the request are left unchanged. To remove a remote from a specific shard:

<CodeGroup>

Expand All @@ -43,19 +57,15 @@ curl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
"removeRemotes": ["ms-03"]
"shards": {
"shard-a": { "removeRemotes": ["ms-03"] }
}
}'
```

</CodeGroup>

<Warning>
Before removing a remote, make sure its shards are replicated on other remotes. Removing the only remote holding a shard makes that data unavailable for network searches.
</Warning>

## Update shard assignments

Reassign shards to different remotes by sending a new `shards` configuration:
To fully replace a shard's assignment, use `remotes`:

<CodeGroup>

Expand All @@ -75,6 +85,29 @@ curl \

</CodeGroup>

## Topology changes and rebalancing

When you modify shard assignments, Meilisearch triggers a `NetworkTopologyChange` task on all remotes. This task runs in three steps:

1. **Compute new shards**: each instance uses rendezvous hashing on document IDs to determine which documents belong to which shard under the new topology.
2. **Export and import**: documents are sent to remotes that now own them.
3. **Delete stale data**: once all remotes confirm their imports are complete, each instance deletes the documents it no longer owns. Search switches to the new shard definitions at this point.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's actually how it behaves, but I prefer to summon @dureuill on that point.


Cancelling a topology change at step 3 only results in stale documents being retained temporarily. It does not cause data loss.

<Warning>
Search requests may return incomplete results during a topology change. Wait for all `NetworkTopologyChange` tasks to complete before resuming normal search traffic.
</Warning>

## Validation

`PATCH /network` rejects requests with a `400 invalid_network_shards` error in the following cases:

- The shard list would become empty after applying the patch
- A shard's `remotes` list would become empty after applying `removeRemotes`
- A shard references a remote that is not in the `remotes` object
- A remote is removed from `remotes` and this leaves a shard with no remotes

## Filter searches by shard

Target specific shards using the `_shard` filter in search requests:
Expand All @@ -88,7 +121,6 @@ curl \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
"q": "batman",
"useNetwork": true,
"filter": "_shard = \"shard-a\""
}'
```
Expand All @@ -99,9 +131,9 @@ Supported `_shard` filter operators:

| Syntax | Behavior |
|--------|----------|
| `_shard = "shard-a"` | Results from `shard-a` only |
| `_shard != "shard-a"` | Results from all shards except `shard-a` |
| `_shard IN ["shard-a", "shard-b"]` | Results from both `shard-a` and `shard-b` |
| `_shard = "shard-a"` | Documents associated to `shard-a` |
| `_shard != "shard-a"` | Documents associated to all shards except `shard-a` |
| `_shard IN ["shard-a", "shard-b"]` | Documents associated to both `shard-a` and `shard-b` |

## Private network security

Expand Down
57 changes: 31 additions & 26 deletions resources/self_hosting/sharding/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,30 +14,29 @@ Replication and sharding require the Meilisearch Enterprise Edition v1.37 or lat

Sharding distributes documents from a single index across multiple Meilisearch instances, called "remotes." Each remote holds one or more named shards containing a subset of your documents.

When a user searches, Meilisearch queries all remotes in the network, collects results from each shard, and merges them into a single ranked response, as if the data lived on one machine.
When a user searches, Meilisearch queries the necessary remotes in the network, collects results from each shard, and merges them into a single ranked response, as if the data lived on a single machine.

## What is replication?

Replication assigns the same shard to more than one remote. If one remote becomes unavailable, another remote holding the same shard continues serving results. Meilisearch automatically queries each shard exactly once, avoiding duplicate results even when shards are replicated.
Replication assigns the same shard to more than one remote. This ensures your data is stored redundantly across instances. During a network search, Meilisearch ensures each shard is queried exactly once, either from a remote shard or from the local one (chosen randomly, favoring the local one). This guarantees each shard is queried exactly once, so results are never duplicated regardless of how many replicas exist.

## How it works

```mermaid
graph TD
Client[Client application] -->|search with useNetwork: true| Leader[Leader instance]
Leader -->|fan out| R1[Remote ms-00<br/>shard-a, shard-c]
Leader -->|fan out| R2[Remote ms-01<br/>shard-a, shard-b]
Leader -->|fan out| R3[Remote ms-02<br/>shard-b, shard-c]
R1 -->|partial results| Leader
R2 -->|partial results| Leader
R3 -->|partial results| Leader
Leader -->|merged results| Client
Client[Client application] -->|search with useNetwork: true| Any[Any instance]
Any -->|fan out| R1[Remote ms-00<br/>shard-a, shard-c]
Any -->|fan out| R2[Remote ms-01<br/>shard-a, shard-b]
Any -->|fan out| R3[Remote ms-02<br/>shard-b, shard-c]
R1 -->|partial results| Any
R2 -->|partial results| Any
R3 -->|partial results| Any
Any -->|merged results| Client
```

1. **Network**: all instances register with each other through the `/network` endpoint, forming a topology with a designated leader
2. **Shards**: the leader distributes document subsets across remotes based on shard assignments
3. **Search**: when `useNetwork: true` is set, the leader fans out the search to all remotes, then merges and ranks the combined results
4. **Failover**: if a remote is down, another remote holding the same shard serves those results
1. **Network**: the user configures the topology via `/network` on the leader, and this instance propagates it to all remotes
2. **Shards**: Remotes distribute the subsets of documents across themselves based on shard assignments
3. **Search**: when `useNetwork: true` is set or not defined (defaults to `true`), the instance receiving the request fans out the search to all remotes, then merges and ranks the combined results

## When to use sharding and replication

Expand All @@ -54,14 +53,18 @@ All instances in a Meilisearch network share a topology configuration that defin

- **`self`**: the identity of the current instance
- **`leader`**: the instance coordinating writes and topology changes
- **`remotes`**: all instances in the network with their URLs and search API keys
- **`remotes`**: all instances in the network with their URLs, search API keys, and write API keys
- **`shards`**: how document subsets are distributed across remotes

The leader instance is responsible for write operations. Non-leader instances reject write requests (document additions, settings changes, index creation) with a `not_a_leader` error.
The leader instance is responsible for write operations and topology changes. Non-leader instances reject write requests (document additions, settings changes, index creation) with a `not_leader` error. Search requests can be sent to any instance in the network.

## Searching across the network

To search across all instances, add `useNetwork: true` to your search request:
To search across all instances:

<Note>
`useNetwork` defaults to `true` when a network topology is defined.
</Note>

<CodeGroup>

Expand All @@ -71,8 +74,7 @@ curl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
"q": "batman",
"useNetwork": true
"q": "batman"
}'
```

Expand All @@ -83,7 +85,6 @@ The response includes `_federation` metadata showing which remote each result ca
```json
{
"q": "batman",
"useNetwork": true,
"filter": "_shard = \"shard-a\""
}
```
Expand All @@ -101,8 +102,8 @@ curl \
-H 'Authorization: Bearer MEILISEARCH_KEY' \
--data-binary '{
"queries": [
{ "indexUid": "movies", "q": "batman", "useNetwork": true },
{ "indexUid": "comics", "q": "batman", "useNetwork": true }
{ "indexUid": "movies", "q": "batman" },
{ "indexUid": "comics", "q": "batman" }
]
}'
```
Expand All @@ -117,16 +118,20 @@ Most Meilisearch features work transparently across a sharded network. The follo
|---------|---------------------|-------|
| Full-text search | Yes | Results merged and ranked across all remotes |
| Filtering and sorting | Yes | Filters applied on each remote before merging |
| Faceted search | Yes | Facet counts aggregated across remotes |
| Faceted search | Partial | Facet distribution in search results works across remotes, but the `/facet-search` endpoint does not support `useNetwork` |
| Hybrid/semantic search | Yes | Each remote runs its own vector search, results merged |
| Geo search | Yes | Geographic filters and sorting work across remotes |
| Multi-search | Yes | Use `useNetwork: true` per query |
| Multi-search | Yes | Works per query; `useNetwork` defaults to `true` when a network is configured |
| Federated search | Yes | Federation merges results from both indexes and remotes |
| Analytics | Partial | Events are tracked on the instance that receives the search request |
| Tenant tokens | Yes | Token filters apply on each remote |
| Document operations | Leader only | Writes must go through the leader instance |
| Settings changes | Leader only | Settings updates must go through the leader |
| Conversational search | Yes | Chat queries can use network search |
| Conversational search | No | Chat completions do not support `useNetwork` |

<Warning>
Search requests may return errors during a network topology change if they reference shards that are being added or removed. Wait for all `NetworkTopologyChange` tasks to complete before searching.
</Warning>

## Prerequisites

Expand All @@ -147,7 +152,7 @@ Before setting up sharding and replication, you need:
Set up replicated shards for high availability and read scaling.
</Card>
<Card title="Manage the network" href="/resources/self_hosting/sharding/manage_network">
Add and remove remotes, update topology, and handle failover.
Add and remove remotes, update shard assignments.
</Card>
<Card title="Enterprise Edition" href="/resources/self_hosting/enterprise_edition">
Learn about the differences between Community and Enterprise editions.
Expand Down
Loading