research: optimal LLM prompt structure for network traffic analysis

[NotYuSheng]

Background

The story feature currently sends the LLM a list of the top N conversations (sorted by bytes) plus a protocol/category breakdown. The conversation cap (STORY_MAX_CONVERSATIONS=20) was questioned as potentially deceptive — the LLM only sees a slice of traffic and has no sense of what it's missing.

A packet-budget approach was considered but rejected: the LLM never sees raw packets anyway (it sees one summary line per conversation), so a packet count is a poor proxy for prompt quality. More importantly, LLMs don't process long repetitive lists well — attention degrades over distance and the model anchors on the first and last entries, treating the middle as noise. Feeding 50 nearly-identical conversation lines may actually produce worse narratives than a well-structured 20.

Core question

What prompt structure actually helps an LLM reason well about network traffic?

The hypothesis is that the LLM needs the shape of the traffic — not an enumeration of flows. Specifically:

• What protocols/apps dominate (covered)
• What the outliers are — risky, unusual, or anomalous flows (partially covered)
• Aggregate structure: top destinations, unique external hosts, protocol × risk distribution
• A sense of coverage — what fraction of traffic is represented

Research tasks

• Review existing literature or blog posts on LLM prompting for structured/tabular data (network logs, security events)
• Investigate how SIEM/security tools (Splunk, Elastic, Darktrace, etc.) summarise traffic for human analysts — what abstractions do they use?
• Look at how other network analysis AI tools (if any) structure their prompts
• Identify what aggregate views would give the LLM the most signal per token:
  • Top-N destination hosts (grouped, not per-flow)
  • Protocol × risk matrix
  • Unique external ASNs/orgs seen
  • Time distribution of traffic (bursty vs. steady)
• Evaluate whether the current per-conversation line format is optimal or if a different schema would be clearer to the model

Goal

Produce a concrete recommendation for how to restructure the story prompt — what to add, what to remove or compress, and what the ideal conversation-list size is given typical LLM attention behaviour.

[NotYuSheng]

Research findings

1. Lost in the Middle — the problem is quantified and architectural

Liu et al. (TACL 2024) measured accuracy on a 20-document QA task:

• Position 1 (first): ~75% accuracy
• Position 20 (last): ~72% accuracy
• Positions 10–15 (middle): collapsed to below the no-context baseline of 56.1%

The U-shaped curve (primacy + recency bias) is not a tuning artefact — it is baked into Rotary Position Embedding (RoPE), which underlies Qwen2.5, LLaMA, Mistral, and most modern LLMs. It cannot be prompted away. The practical ceiling before middle items are ignored is 5–7 distinct list entries.

Implication: the current 20-conversation list means entries 6–14 are systematically lost. The risk-first ordering helps (highest priority items are at position 1), but the structural problem remains.

2. The flat conversation list is the wrong abstraction

How Darktrace, Vectra, Corelight, Splunk, and Sentinel present network data to analysts — and now to LLMs — is consistent:

• None of them present a byte-sorted flow list. They present aggregate blocks, anomaly clusters, and population-level statistics.
• Darktrace Cyber AI Analyst groups weak signals into a single incident narrative with a MITRE ATT&CK chain. The analyst sees "these 6 anomalies form a coherent lateral movement pattern" — not "here are 20 flows."
• Corelight uses a map-reduce pattern: summarise chunks (per log type or time window) separately, then synthesise. Each sub-summary is narrow and scoped.
• CrowdStrike Charlotte AI uses 12+ specialised sub-agents, each receiving a narrow input. No single agent sees the full dataset.

The core insight: a large language model and a human analyst have the same limitation — they cannot hold 847 conversations in working memory and reason about them. The right move is to pre-compute the reasoning-relevant aggregates and give those to the LLM, with the flow list as supporting evidence, not as the primary input.

3. What aggregates give the most signal per token

From analyst methodology (CyberDefenders SOC guide, TrustedSec DFIR guide, Elastic Security Labs, NDR vendor documentation):

Aggregate	Signal	In current prompt?
Unique external destination count (full dataset)	High cardinality = DGA, scanning	Only implicitly
Top 5 external ASNs by outbound bytes	Bulk traffic to one ASN/country = exfil/C2	No
Protocol × risk matrix	Which protocols are flagged and how many	Partially (separate sections)
Unique internal hosts with external connections	Lateral movement, compromised host count	No
TLS anomaly summary (N self-signed, N expired, N unknown CA)	C2 infra rarely uses valid certs	Per-conversation only
Unknown app % of all conversations	High unknown% = encrypted/obfuscated C2	No
Beacon candidates (inter-arrival jitter CV < 15%)	Strongest single C2 keepalive signal in NDR tools	No
Top talker: packet count vs. byte count ratio	High packets/low bytes = scanning or beacon; high bytes = exfil	No

The population-level statistics (computed over all 847 conversations, not just the top 20) are especially important — they let the LLM know what the unseen 80%+ looks like, fixing the representativeness problem.

4. Chain-of-thought decomposition improves grounding

ProvSEEK (arXiv 2025) and the Corelight approach both show that explicit step-by-step reasoning instructions reduce hallucination. Instead of asking the LLM to do everything at once, decompose:

1. Identify the 1–3 most significant findings and state them in one sentence each
2. For each finding, identify which specific conversations or aggregates support it
3. Construct the timeline from those grounded events
4. Write the narrative, referencing the findings

Each claim is anchored to evidence before prose is generated. This mirrors how Darktrace's AI Analyst works.

Recommended direction

The conversation list should not grow — it should shrink to ~5–7 entries and be surrounded by rich aggregates computed over the full dataset.

• The LLM reasons about the full shape of traffic from the aggregates (population-level, lossless)
• The short conversation list provides specific flows to cite as evidence
• The bookend rule: put the highest-priority item at position 1 and explicitly reference it in prose before the list, exploiting both primacy bias and explicit attention

This means the representativeness concern is addressed not by including more conversations, but by computing aggregate views over all conversations and injecting those instead.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2024)
• Corelight: Leveraging Map-Reduce & LLMs for Network Detection
• Corelight: LLM Prompts for Network Security Alert Summaries
• CrowdStrike Charlotte AI Multi-Agent Architecture
• Table Meets LLM: Can LLMs Understand Structured Table Data? (WSDM 2024)
• Elastic Security Labs: Identifying Beaconing Malware
• Global Analysis with Aggregation-based Beaconing Detection (ACM 2023)
• ProvSEEK: LLM-driven Provenance Forensics (arXiv 2025)
• Darktrace: Autonomous Threat Investigation with AI Analyst
• Morph: Lost in the Middle explainer with quantitative detail

[NotYuSheng]

Revised direction: separation of concerns between backend analysis and LLM interpretation

After reviewing the full frontend rendering and thinking about what LLMs actually do well, the right architecture is clearer.

What LLMs excel at

• Natural language reasoning — explaining why something is suspicious, connecting heterogeneous signals into a coherent hypothesis
• Contextualisation — translating technical findings into plain language an analyst can act on
• Generating hypotheses — "this pattern is consistent with C2 beaconing because X, Y, Z"
• Narrative synthesis — weaving multiple independent signals into a coherent story

What LLMs are bad at

• Precise counting and arithmetic — ask an LLM to identify the top talker from 20 rows and it often gets it wrong
• Exhaustive enumeration — "list all flows to China" from a big table is unreliable
• Detecting statistical patterns — beaconing, jitter, outlier detection require computation, not language modelling
• Retaining information from the middle of a long list (established by Liu et al. 2024)

The implication: the backend should do the analysis, the LLM should do the interpretation

Currently the prompt hands the LLM raw data and asks it to do both analysis and interpretation. This is backwards. The hard analytical work — identifying the top ASNs, computing beacon candidates, counting TLS anomalies, building the protocol×risk matrix — should be done by the backend in deterministic SQL/Java code. Those pre-computed facts are then handed to the LLM as concise, accurate inputs. The LLM's job becomes purely interpreting and explaining pre-digested findings, which is what it's actually good at.

This also means the LLM gets a much shorter, higher-signal prompt — no long flow lists, just structured facts it needs to reason about.

What this means for the frontend

Currently the frontend renders whatever the LLM writes as free-form prose sections. But if the backend pre-computes structured findings, some of those should be rendered as visual components — not just prose:

• Top external ASNs by bytes → mini bar chart or table (backend-computed, rendered directly)
• Protocol × risk matrix → small table (backend-computed, rendered directly)
• TLS anomaly summary → count badges (backend-computed, rendered directly)
• Beacon candidates → flagged flow list (backend-computed, rendered directly)
• LLM narrative → prose sections explaining and connecting the above (LLM-generated, rendered as text)

The LLM narrative would then reference these visual anchors rather than trying to reproduce their content in prose. This mirrors how commercial security tools like Darktrace and Vectra work — the AI produces a short analytical narrative that cites specific pre-computed detections, rather than generating the detections themselves from raw data.

Proposed new story response structure

StoryResponse
├── aggregates (backend-computed, always accurate)
│   ├── topExternalAsns: [{asn, org, country, bytes, pct}]
│   ├── protocolRiskMatrix: [{protocol, total, atRisk}]
│   ├── tlsAnomalySummary: {selfSigned, expired, unknownCa}
│   ├── uniqueExternalHosts: number
│   ├── unknownAppPct: number
│   └── beaconCandidates: [{srcIp, dstIp, protocol, periodMs, cv}]
└── llmOutput (LLM-generated, interprets the aggregates)
    ├── narrative: NarrativeSection[]   (kept, but shorter/focused on interpretation)
    ├── highlights: Highlight[]         (kept)
    ├── timeline: StoryTimelineEvent[]  (kept)
    └── suggestedQuestions: string[]    (kept)

The LLM prompt would include the pre-computed aggregates as its primary input, with a small number of representative conversations (5–7, not 20) as supporting evidence for citation.

Summary of what changes

Backend:

• New service/queries to compute the aggregate views over the full conversation dataset
• LLM prompt shrinks significantly — aggregates replace the long conversation list
• LLM is explicitly instructed to interpret the provided aggregates, not re-derive them

Frontend:

• New visual components to render each aggregate block directly (not via LLM prose)
• LLM narrative sections become shorter and interpretation-focused
• Story page redesigned to show aggregate panels + LLM interpretation side by side