Improvements for the Index basics overview page

[yetanothertw]

Summary

Fixes #5221

The PR touches 19 files, but the core structural changes are across four pages (the rest are updated links):

• The Data store page
• The Index basics page
• The Manage and perform operations on indices page
• The Templates page

My main aim was to remove duplication and split relevant info into dedicated pages (one concept at a time). I've also reworked some of the introductions on the parent pages to better follow progressive disclosure patterns and linked out to relevant pages for depth.

The Index basics overview page said nothing about how data is physically stored, which I think was a knowledge gap. Introducing the new How an index stores data section ties together the data store and deploy and manage sections (or rather bridges important concepts).

Changes

• The Elasticsearch data store parent landing page now includes an overview/wayfinding layer. The updated conceptual intro progressively discloses what the data store model is and introduces concepts like documents in indices, shards distributed across nodes for scalability and fault tolerance, multiple independent datasets side by side. There are now three organised sections: Understand data storage, Configure how data is stored, Manage data, each with a brief description and links to child pages. The page now serves two audiences at once. Someone new gets a mental model of what the ES data store is before choosing where to dive in. Someone experienced can scan the headings and jump directly to the child page they need.

• Index basics page aims to be more user-focused and to serve one purpose: explain what an index is. The rewritten intro that frames the index as the level at which you interact with your data (store to it, search against it). The Index components section (documents, metadata, mappings) has been tightened but it's structurally the same. A new How an index stores data section explains shards, Lucene, primary/replica shards, and links out to existing detailed pages (clusters-nodes-shards.md, near-real-time-search.md) rather than duplicating their content. A new Common index design decisions section touches on naming conventions, mapping strategy, index vs data stream, shard sizing.

• Moved all the index management info into dedicated pages where it logically belongs. A chunk of it also went into the Manage and perform operations on indices including the required security permissions, a tab-by-tab overview of the Index Management page in the {{kib}} UI (Indices, Data Streams, Index Templates, Component Templates, Enrich Policies) and instructions for investigating indices. All the "how to use the Index Management UI" content now lives on a single dedicated page rather than being split across. The mapped_pages URLs for the old index management content moved here too, so existing links don't break.

• The Templates page absorbed the {{kib}} template creation wizard tutorial that was embedded in the Index basics page, added in the {{kib}}/API table split. The same applies for the component templates info.

Problem statement

At present the Index basics page includes a collection of content types. It performs the following tasks:

• Defines what an index is; explains documents, metadata, mappings (Conceptual overview)
• Introduces the Index Management UI and lists required permissions (How-to)
• Includes screenshots + instructions for the Indices view (stack & serverless variants) (How-to)
• Includes screenshots + instructions for data streams, index templates views (How-to)
• Includes a step-by-step tutorial: create an index template (Tutorial)
• Includes screenshots + instructions for component templates and enrich policies (How-to)

Another problematic page is the parent Data store overview page that needs to be edited in tandem as it should be providing more wayfinding guidance for the other related topics in this section (templates, mappings, aliases, data streams, and so on).

There is also a significant amount of duplication of information happening across the entire section (and an additional layer of duplication when compared to the Data lifecycle section). For example, the Manage data streams section (lines 77-99) duplicates data-streams/manage-data-stream.md.
This makes content drift more likely to occur over time. The manage-data-stream.md page is more detailed and better-placed. The section here (Index management) is a condensed copy.
Another example of duplication: component template definition appears three times on this page, and the ]dedicated templates page](https://www.elastic.co/docs/manage-data/data-store/templates) does it all over again.
The section about enrich policies repeats the same pattern. Lines 250-256 re-define what an enrich policy contains (policy type, source indices, match fields, enrich fields, query). This is already covered on the dedicated enrich page that is linked in the first line of the section.

Issues with the existing [Index management](https://www.elastic.co/docs/manage-data/data-store/index-basics#index-management section

This page is really long and difficult to scan, all while trying to be three things at once:

• A UI tour of the Index Management feature in Kibana
• A conceptual explainer for data streams, index templates, component templates, and enrich policies
• A tutorial for creating an index template via the wizard

This violates progressive disclosure. A user arriving at this page expecting to learn how to use the Index Management UI instead gets re-taught concepts that are already covered in depth on dedicated pages, and then pulled into a step-by-step tutorial that takes up 42% of the page (lines 117-233).

Generative AI disclosure

1. Did you use a generative AI (GenAI) tool to assist in creating this contribution?

• Yes
• No

Used Claude 4.6 Opus High to check and update links

[github-actions]

Vale Linting Results

Summary: 5 warnings, 8 suggestions found

⚠️ Warnings (5)

File	Line	Rule	Message
deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md	83	Elastic.Spelling	'Kopf' is a possible misspelling.
manage-data/data-store/templates.md	182	Elastic.Spelling	'api' is a possible misspelling.
manage-data/data-store/templates.md	284	Elastic.Spelling	'api' is a possible misspelling.
manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md	204	Elastic.Spelling	'api' is a possible misspelling.
solutions/observability/apm/explore-data-in-elasticsearch.md	21	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.

💡 Suggestions (8)

File	Line	Rule	Message
deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md	82	Elastic.Wordiness	Consider using 'because' instead of 'since'.
deploy-manage/monitor/orchestrators/ece-monitoring-ece-set-retention.md	48	Elastic.Clone	Use Clone only when referring to cloning a GitHub repository or creating a copy that is linked to the original. Often confused with 'copy' and 'duplicate'.
manage-data/data-store.md	39	Elastic.WordChoice	Consider using 'efficiently' instead of 'easily', unless the term is in the UI.
manage-data/data-store/index-operations-reference.md	49	Elastic.Wordiness	Consider using 'all' instead of 'all of '.
manage-data/data-store/templates/index-template-management.md	15	Elastic.Wordiness	Consider using 'all' instead of 'all of '.
manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md	62	Elastic.WordChoice	Consider using 'run, start' instead of 'Execute', unless the term is in the UI.
manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md	64	Elastic.WordChoice	Consider using 'run, start' instead of 'execute', unless the term is in the UI.
troubleshoot/observability/troubleshoot-mapping-issues.md	32	Elastic.Wordiness	Consider using 'remove' instead of 'eliminate'.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[github-actions]

🔍 Preview links for changed docs

• deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md
• deploy-manage/monitor/orchestrators/ece-monitoring-ece-set-retention.md
• deploy-manage/tools/snapshot-and-restore/restore-snapshot.md
• explore-analyze/alerting/watcher/watcher-ui.md
• manage-data/data-store.md
• manage-data/data-store/data-streams/manage-data-stream.md
• manage-data/data-store/index-basics.md
• manage-data/data-store/index-operations-reference.md
• manage-data/data-store/manage-data-from-the-command-line.md
• manage-data/data-store/perform-index-operations.md

More links …

• manage-data/data-store/templates.md
• manage-data/data-store/templates/index-template-management.md
• manage-data/ingest/transform-enrich/ingest-pipelines.md
• manage-data/ingest/transform-enrich/set-up-an-enrich-processor.md
• manage-data/lifecycle/index-lifecycle-management.md
• manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md
• manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md
• solutions/observability/apm/explore-data-in-elasticsearch.md
• solutions/observability/apm/reduce-storage.md
• solutions/observability/synthetics/manage-data-retention.md
• troubleshoot/observability/troubleshoot-mapping-issues.md

[shainaraskas]

went a little too deep in some spots - sorry!

I think overall this looks miles better. I put some comments on the "manage and perform operations on indices" page in particular - I think it could be broken up and relocated for optimal usefulness.

fwiw I was going to mention adding crosslinks from the API page to equivalent operations on the other pages. one thing looking at this page flagged for me was that we don't tell someone how to actually index into an index. would be nice to hint "you can test index docs at any time using [endpoint], but you want to go look at ingest for the whole story"

also, the API story is much more end to end ... could consider providing paths forward even further afield (now that you shaped and ingested your data, go learn how to search it! deletion is actually probably also happy path for this section!)

I know I'm scope creeping a bit, but I think we're very close to a downright useful story and can put a nice bow on it :)

[shainaraskas]

nicenicenice. my only blocking comment is exposing equivalent serverless permissions for managing indices in kib. everything else simply opinion

[shainaraskas]

I think that now that this page has changed so much we might want to change the title : Indices? we don't append basics to any of our other overview topics

[yetanothertw]

I see your point! Personally, I'd prefer something like Understanding indices
or Indices overview.

It seems to me that people are more likely to search for the term index rather than indices (or even indexes), so I'd like to keep index in the title.
Maybe something like:

• Index fundamentals
• Index concepts
• What is an index?

Wdyt, @shainaraskas ?

[shainaraskas]

I see what you mean. I don't have a better idea so we can leave it for now / follow your heart

[mdbirnstiehl]

Looks good on the solutions side of things.

[shainaraskas]

🚢

[szabosteve]

🚢 Thanks for this! LGTM!