Add `experimental/ast/printer` package by emcfarlane · Pull Request #657 · bufbuild/protocompile

emcfarlane · 2026-01-22T19:00:41Z

This adds experimental/ast/printer, an AST printer for protobuf files. It does not yet implement formatting, but uses the dom library to produce correct indentation for synthetic edits. Formatting support will be added later.

The printer preserves comments and whitespace using a trivia index. Each comment is classified as either attached (bound to a specific token) or detached (bound to a positional slot between declarations). Attached comments travel with their token when declarations are moved; detached comments stay in place. At print time, the printer zips slot trivia with children within each scope, and looks up attached trivia when emitting individual tokens. This works identically for natural and synthetic tokens (no special-casing needed). Synthetic tokens won't have a trivia and fallback to the declared gap. This gap is also what will be used to inform how to format the trivia between tokens when formatting.

doriable

The API looks very reasonable to me, I just had a few nitpicks/discussion points for clarifying a few things. Thank you!

mcy · 2026-02-17T21:55:01Z

+)
+
+// PrintFile renders an AST file to protobuf source text.
+func PrintFile(file *ast.File, opts Options) string {


You want this to take function options right?

Do you mean type Option func(...)? Was following dom and report options

mcy · 2026-02-17T21:56:06Z

+// the pending buffer so that adjacent pure-newline runs are combined into a
+// single kindBreak dom tag, preventing the dom from merging them and


Could this be dealt with by making kindBreak a bit more sophisticated?

Add TestBufFormat that reads buf's bufformat testdata (54 test cases) and compares printer output against golden files. Currently 35/54 pass. Fixes: - Empty files produce empty output instead of trailing newline - New gapInline style keeps punctuation on same line as preceding comments: `value /* comment */;` instead of breaking to new line - New gapGlue style for path separators glues comments without spaces: `header/*comment*/.v1` - Preserve source blank lines after detached comments using actual newline count from pending trivia tokens - Body declarations at file level only get blank lines when source had them, instead of unconditionally - Compound string first element on new indented line after `=` - Strip trailing whitespace from comments in format mode

Add test cases documenting 8 remaining formatting issues: - Issue 1: trailing // before ] should become /* */ on single-line - Issue 2: trailing comment after , should stay inline - Issue 3: comment after [ opener should expand to multi-line - Issue 5: comment before } in enum (already passes) - Issue 6: EOF comment after blank line should preserve blank line - Issue 7: block comments in RPC parens should not add extra spaces - Issue 8: extension path comments should preserve spaces - Issue 9: message literal with block comments should expand

Two formatting fixes: - Extract inline trailing comments after commas in walkDecl so they stay on the same line as the comma instead of becoming leading trivia on the next token - Preserve blank lines before EOF comments by checking trivia.blankBeforeClose in printFile

…sion - Negative prefix (-) with block comments uses gapSpace for proper spacing (e.g., "- /* comment */ 32") - Revert compound string // to /* */ conversion attempt as it caused trailing comment conversion on the following semicolon

This ensures block comments in glued contexts (RPC parens, path separators, generics) always get a space after them before the next word token. Without comments, behavior is unchanged.

…rsion Three fixes for comment preservation in the printer: 1. Generalize inline trailing comment extraction in walkDecl to all tokens, not just commas. A comment on the same line as any token (e.g., "bar: 2 // comment") is now correctly attached as trailing trivia on that token. Guarded by firstNewline < len(leading) to avoid reclassifying block comments between same-line tokens. 2. Add emitCommaTrivia to printDict so that comments attached to comma tokens (which are removed during message literal formatting) are never silently dropped. 3. Manage convertLineToBlock in printCompoundString: clear it for intermediate parts (// comments between string parts on their own lines are fine), restore the caller's value for the last part's trailing (a // there would eat the following ; or ]). Add withLineToBlock helper for scoped save/restore. Set it in printOption since ; follows the value inline.

…ia only Set convertLineToBlock in printPath since path components are glued inline (gapGlue) and a trailing // comment between components would eat the next identifier. Remove the convertLineToBlock check from emitTrivia (leading trivia). After the generalized inline trailing extraction in walkDecl, all comments remaining in leading trivia are on their own lines and never eat following tokens. Only emitTrailing needs the conversion. This prevents over-conversion of leading // comments that are safe as-is.

Categorize and explain the stylistic differences between our printer output and the old buf format golden files. All remaining differences are intentional formatting choices, not correctness issues.

emcfarlane · 2026-04-21T14:17:58Z

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package printer_test


This code only exists to validate against the current to produce the above doc.

doriable

Sorry for the slow turnaround time on the review :< Left some comments -- I think the primary feedback is around the handling of trivia and the edge-cases/invariants that can come up. I think once those are ironed out, we can finalise the behaviour through the test results.

doriable · 2026-04-30T21:27:26Z

+//   - Consecutive newline tags accumulate, but are capped at two newlines
+//     (one blank line) in the output.


So this doc is a little bit confusing/misleading on the context of the behaviour implemented in print.go:

if len(tag.text) == 1 { // Single-newline breaks increment by 1, capped at 2 // (one blank line maximum). p.newlines = min(p.newlines+1, 2) } else { // Multi-newline breaks set the floor directly. p.newlines = max(p.newlines, len(tag.text)) }

In the case of Text("\n\n\n"), this would actually just jump to the else condition... this behaviour only applies if new lines are accumulated one tag at a time... which I don't think we can assume to be the case. I believe the fix should actually be in print.go to check for p.newlines?

doriable · 2026-04-30T22:07:00Z

+					// Convert // comment to /* comment */ for inline contexts.
+					body := strings.TrimPrefix(strings.TrimRight(t.Text(), " \t"), "//")
+					p.push(dom.Text("/*" + body + " */"))


This doesn't work if the comment is something like:

message Foo { // i am an agent of chaos */

Since you'll get /* i am an agent of chaos */*/

doriable · 2026-04-30T22:27:52Z

+			}
+		}
+	} else {
+		p.pending = append(p.pending, trailing...)


There is an edge case here... so, only non-synthetic tokens can have trivia, e.g. trailing, and if a non-synthetic token with trailing is followed by synthetic token, which will always be hasTrivia == false, then pending is only emitted at the next point with trivia?

doriable · 2026-04-30T22:35:34Z

+		p.printToken(expr.Colon(), gapInline, ctx)
+	} else if p.options.Format && !expr.Key().IsZero() && !expr.Value().IsZero() {
+		// Insert colon in format mode when missing (e.g. "e []" -> "e: []").
+		p.push(dom.Text(":"))


Might be worth commenting that in the case where p.pending is not empty, the pending is flushed after the :, since we're just pushing the : without flushing pending first... it seems like a behaviour that might be okay (but also, for trailing, it's a little weird...), so it's at least something we should comment (or maybe fix... o_o)

doriable · 2026-04-30T22:36:24Z

+}
+
+// emitGap pushes whitespace tags for the given gap style.
+func (p *printer) emitGap(gap gapStyle) {


Worth noting that this doesn't handle if things exist in p.pending.

doriable · 2026-04-30T22:47:54Z

+// withIndent runs fn with an indented printer, swapping the sink temporarily.
+func (p *printer) withIndent(fn func(p *printer)) {
+	originalPush := p.push
+	p.push(dom.Indent(strings.Repeat(" ", p.options.TabstopWidth), func(indentSink dom.Sink) {


There is a strings.Repeat allocation per indent... perhaps it's worth caching this? o_o

emcfarlane force-pushed the ed/printer2 branch from dbf4d2f to 4422e60 Compare January 22, 2026 19:03

Add experimental/printer package

c170def

emcfarlane force-pushed the ed/printer2 branch from 4422e60 to c170def Compare January 22, 2026 19:06

emcfarlane added 5 commits January 27, 2026 17:10

Fix eol

35190b9

Add edit tests

cdd02b3

Handle comments on deletes

4a8252b

Cleanup

84e2e94

Fix synthetic tokens patch commas

5ac99d8

doriable reviewed Feb 3, 2026

View reviewed changes

Comment thread experimental/printer/printer.go Outdated

Comment thread experimental/printer/printer.go Outdated

Comment thread experimental/printer/printer.go Outdated

Comment thread experimental/ast/printer/printer_test.go

emcfarlane added 4 commits February 4, 2026 00:43

Integrate synthetic gap

a6f993c

Fix synthetic groups

be09ae5

Merge branch 'main' into ed/printer2

ba19f96

Move to ast

5d0deec

emcfarlane changed the title ~~Add experimental/printer package~~ Add experimental/ast/printer package Feb 6, 2026

Trivia index

67878ec

emcfarlane force-pushed the ed/printer2 branch from ee5138a to 67878ec Compare February 7, 2026 07:48

emcfarlane added 4 commits February 16, 2026 21:03

Cleanup trivia implementation

e4e8145

Fix trivia between tokens

9b3b444

Fix lint

d9ef7c0

Fix options

c233a2b

emcfarlane marked this pull request as ready for review February 17, 2026 16:52

mcy reviewed Feb 17, 2026

View reviewed changes

Doc trivia example

f78d0ab

emcfarlane marked this pull request as draft March 11, 2026 19:23

emcfarlane added 6 commits March 11, 2026 20:24

Merge branch 'main' into ed/printer2

2b03df5

Add format support

9657f96

Add buf block comment formatting

28e9fe9

emcfarlane added 27 commits March 19, 2026 14:41

Fix gap on nested comments between elements.

7f60149

This ensures block comments in glued contexts (RPC parens, path separators, generics) always get a space after them before the next word token. Without comments, behavior is unchanged.

Fix lint issues in printer package

64358c2

Document remaining bufformat golden test differences

187a180

Categorize and explain the stylistic differences between our printer output and the old buf format golden files. All remaining differences are intentional formatting choices, not correctness issues.

Simplify

5c803c5

Add gapPreserve fix inline comments

10351ee

Add idempotency checks

fd37275

Fix leading scope comments

7597558

Indent compound strings

2be32f6

Fix single line dicts

70e5bc0

Update bufformat-diff summary table

d18a6ed

Fix trailing comments on options

9c49fe6

Add an immutable ctx param

9e3e50e

Cleanup dom tags

ef5b26d

Drop indent flag

fe7257d

Drop bufformat cmp

360940b

Merge branch 'main' into ed/printer2

eb1b52e

Fix syntax issues

23d96c6

Fix names for synthetic types

687f085

Support invalid decl printing

d4a6ed8

Add back bufformat compare

1c27a8d

Cleanup comments

e43f8c7

Line comment handling

5ef1f40

Fix empty-decl blank-line bug and single-option compact-option

a639803

Fix lint

98d59e8

emcfarlane commented Apr 21, 2026

View reviewed changes

Merge branch 'main' into ed/printer2

b6b02a6

doriable reviewed Apr 30, 2026

View reviewed changes

		// the pending buffer so that adjacent pure-newline runs are combined into a
		// single kindBreak dom tag, preventing the dom from merging them and

		// - Consecutive newline tags accumulate, but are capped at two newlines
		// (one blank line) in the output.

Conversation

emcfarlane commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doriable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doriable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

emcfarlane commented Jan 22, 2026 •

edited

Loading