Profiling - `exp/trace`

[vaind]

I've been pinged on an old issue for adding APIs to go runtime that would have enabled us to profile efficiently multiple goroutines - the proposal got some attention and is being considered.
I've responded to some clarification requests.

The most recent response indicates there's a new/experimental alternative approach - using golang.org/x/exp/trace. I'm bringing the information here since I'm not currently actively working on the Go SDK so I don't know what the plans are for profiling.

cc @giortzisg @cleptric

[vaind]

Sadly it looks like a likely decline as per latest comment

That relates to the originally proposed changes, because the way forward (which wasn't available back then) should be exp/trace.

[giortzisg]

Will investigate to see if using exp/trace would work and get back to you.

[linear-code]

GO-132

[giortzisg]

I revisited this with a small POC and confirmed that the current Go runtime options still don’t give us a good path for trace-mode profiling. The only approach that gave us enough information to build something useful was combining runtime/trace with CPU profiling (which was also the proposal from the Go maintainers), and that was enough to build a useful profile chunk.

However, the main problem is profiling lifecycle and overhead. For trace mode, profiling would need to be follow the trace activity, which means constantly turning profiling on/off. In the POC, the only setup that produced usable stacks depended on pprof.StartCPUProfiler and that makes it a poor fit for trace mode. The start/stop behavior then becomes really expensive for the traced request flow.

I will be closing this, since we can't really pursue this approach any further. Would be happy to revisit this in the future if there's a solution that fits well with trace mode.