Skip to content

fix: chunk oversized string fields to prevent Loki rejection#154

Merged
vik-nullify merged 2 commits intomainfrom
fix/log-entry-size-chunking
Feb 11, 2026
Merged

fix: chunk oversized string fields to prevent Loki rejection#154
vik-nullify merged 2 commits intomainfrom
fix/log-entry-size-chunking

Conversation

@vik-nullify
Copy link
Member

Summary

  • Log entries with large string fields (HTTP response bodies, Lambda payloads) were exceeding Grafana Loki's 256KB per-entry limit and being silently dropped
  • Instead of truncating and losing data, oversized string fields (>200KB) are now automatically split across multiple log entries
  • Each chunk is annotated with {key}_chunk and {key}_total_chunks metadata for reassembly in Grafana

Changes

  • pkg/logger/chunking.gochunkOversizedFields() helper that detects oversized string fields and splits them into multiple field sets
  • pkg/logger/logger.goDebug, Info, Warn, Error methods now emit multiple log lines when chunking is needed
  • pkg/logger/chunking_test.go — Tests for chunking, reassembly, multiple oversized fields, and edge cases

Test plan

  • Unit tests pass for chunking logic
  • Full test suite passes
  • Deploy to UAT and verify Loki no longer drops entries
  • Verify chunks can be reassembled in Grafana by filtering {key}_total_chunks > 0

🤖 Generated with Claude Code

Log entries with large string fields (e.g. HTTP response bodies) were
exceeding Grafana Loki's 256KB per-entry limit and being dropped.

Instead of truncating and losing data, oversized string fields are now
automatically split across multiple log entries, each annotated with
chunk metadata ({key}_chunk, {key}_total_chunks) for reassembly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vik-nullify vik-nullify force-pushed the fix/log-entry-size-chunking branch from 2fb9086 to 42cb544 Compare February 11, 2026 21:35
@vik-nullify vik-nullify requested a review from dnjg February 11, 2026 21:45
@vik-nullify vik-nullify marked this pull request as ready for review February 11, 2026 21:45
@vik-nullify vik-nullify added this pull request to the merge queue Feb 11, 2026
Merged via the queue into main with commit 0d0e434 Feb 11, 2026
3 checks passed
@vik-nullify vik-nullify deleted the fix/log-entry-size-chunking branch February 11, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants