Provide a (limited) openapi-like REST API

[lars-t-hansen]

For use by the dashboard, Sonalyze should export an API compatible with the Slurm-monitor API, albeit probably a subset.

@ashenr will come up with a priority list of APIs.

[PubuduSaneth]

@ashenr discussed with me in F2F meeting about the plans for VM monitoring. Following are the stats/data I'd like to have in a VM monitoring dashboard.

1. CPU usage %
2. RAM Memory and GPU memory usage
3. overall GPU usage and usage per GPU (when multiple GPUs are avail in the VM)
4. Disk I/O usage

I gathered above stats per-process via JobAnalyser + Sonar CLI in my VMs. For example, a single run I submit will execute 2-3 processes (different PIDs), so I could gather stats of each process via JobAnalyser + Sonar CLI. Therefore, it would be ideal If VM monitoring dashboard provides an option to view/extract stats per process.

[ashenr]

Below are the list of API endpoints ordered by the priority.

'Existing' means, this is already available in current REST API implementation.

#	New?	Endpoint	Description	Query Params	Response Type
1	Existing	GET /cluster/{vm-id}/nodes/info	Hardware specs (CPU, memory, GPUs)	—	Record<string, NodeInfoResponse>
2	Existing	GET /cluster/{vm-id}/nodes/last-probe-timestamp	Last Sonar data timestamp	—	Record<string, Date>
3	Existing	GET /cluster/{vm-id}/error-messages	Error messages	—	Record<string, ErrorMessageResponse>
4	Existing	GET /cluster/{vm-id}/nodes/cpu/timeseries	CPU utilization over time	timeseries params	Record<string, SampleProcessAccResponse[]>
5	Existing	GET /cluster/{vm-id}/nodes/memory/timeseries	Memory utilization over time	timeseries params	Record<string, SampleProcessAccResponse[]>
6	Existing	GET /cluster/{vm-id}/nodes/gpu/timeseries	Per-GPU utilization over time	timeseries params	Record<string, SampleGpuTimeseriesResponse[]>
7	Existing	GET /cluster/{vm-id}/nodes/diskstats/timeseries	Disk I/O over time	timeseries params	NodeDiskTimeseriesResponse
8	Existing	GET /cluster/{vm-id}/nodes/process-gpu-util	Current per-GPU utilization	—	NodeSampleProcessGpuAccResponse
9	NEW	GET /cluster/{vm-id}/processes	Running processes with CPU/mem/GPU per PID	—	ProcessSnapshot[]
10	NEW	GET /cluster/{vm-id}/processes/timeseries	Per-process usage over time	timeseries params + pid (optional)	Record<string, ProcessTimeseriesEntry[]>
11	NEW	GET /cluster/{vm-id}/processes/gpu	Which process uses which GPU	—	ProcessGpuSnapshot[]

Where {vm-id} is a stable identifier for the VM (e.g., hostname like my-gpu-vm).

Below are the response types.

Existing response types in the API

NodeInfoResponse {
  time: Date
  cluster: string
  node: string
  os_name: string
  os_release: string
  memory?: number
  sockets?: number
  cores_per_socket?: number
  threads_per_core?: number
  cards?: GpuCardInfo[]    // { uuid, model, mem_size }
}

NodeStateResponse {
  time: Date
  cluster: string
  node: string
  states?: string[]        // ["UP"] or ["DOWN"] for VMs
}

SampleProcessAccResponse {
  time: Date               // ISO 8601 string
  memory_util?: number     // percentage
  cpu_util?: number        // percentage
}

SampleGpuTimeseriesResponse {
  uuid: string
  index: number
  data: Array<{
    time: Date
    ce_util?: number       // GPU compute utilization %
    mem_util?: number      // GPU memory utilization %
  }>
}

ErrorMessageResponse {
  cluster: string
  node: string
  time: Date
  details: string
}

Per-process response types (NEW)

ProcessSnapshot {
  pid: number
  user: string
  cmd: string              // command/program name (e.g., "python", "bwa")
  cpu_pct: number          // CPU usage % (100 = 1 full core)
  mem_kb: number           // resident memory in KB
  gpu_pct: number          // GPU compute utilization %
  gpu_mem_pct: number      // GPU memory utilization %
  gpus: string[]           // GPU UUIDs this process is using
  time: Date               // sample timestamp
}

ProcessTimeseriesEntry {
  time: Date
  pid: number
  cmd: string
  cpu_pct: number
  mem_kb: number
  gpu_pct: number
  gpu_mem_pct: number
}

ProcessGpuSnapshot {
  pid: number
  cmd: string
  user: string
  gpu_uuid: string         // which GPU this process is on
  gpu_index: number        // GPU index (0, 1, 2, ...)
  gpu_pct: number          // compute utilization %
  gpu_mem_pct: number      // memory utilization %
  gpu_model: string        // e.g., "NVIDIA A100"
  time: Date
}

[lars-t-hansen]

(Notes to self)

It looks like what we want is Huma, which is the server side (in contrast to client libraries, eg https://openapi-generator.tech/). Alas this will take us to Go 1.25 but that's probably OK on the server side.

Also the notation Record<string, ...> above is a map: an object where the host name is the key and the payload is the value, for all host names.

[lars-t-hansen]

Well, the framework Just Works (on larstha-846-rest-api-take2), so that's fine. Now to somehow import the model from the OpenAPI spec, generate code from that, and hook that up to the actual query logic...

[lars-t-hansen]

Huma can't generate that code. The closest we come might be oapi-codegen but that barfs on the OpenAPI 3.1 used by our current yaml. Various sources say "anyOf" may be awkward, it's used quite a bit in our spec, sometimes for no obvious reason as in anyof(integer, number), sometimes for nullability, sometimes for more complex purposes.

Given that the needs outlined above are fairly limited it may be expedient to implement the API by hand. It's a shame to have to write the model code again but probably affordable.

[lars-t-hansen]

More notes to self. The description above is fine as far as it goes for simple one-node use cases where a VM doesn't live very long. But since there are no time limits and since "all" data are retrieved then it scales neither to long time frames nor to large systems.

For this use case, I will arbitrarily assume that the time window of interest is up to the latest time in the database and (for now) starting at most 2 weeks before that.

[lars-t-hansen]

@ashenr looking at the table above, and at the yaml for the API, i have a question about query params. the yaml shows that node name is an optional parameter to many of these operations, but node name is not mentioned in the table above. is this (a) an oversight or (b) deliberate since for the VM case there is only one node and it's just as easy to ask for all nodes as for a single one?

[ashenr]

This was deliberate. In the mapping, a vm-id corresponds directly to a specific cluster.
Since each VM environment effectively acts as a single-node entity in this context, I assumed that specifying a node name is redundant.

[lars-t-hansen]

This was deliberate. In the mapping, a vm-id corresponds directly to a specific cluster. Since each VM environment effectively acts as a single-node entity in this context, I assumed that specifying a node name is redundant.

OK. Per discussions, the vm-id will be the cluster name, in the normal way. The cluster name is always a configuration parameter to the sonar daemon, it is never inferred from host name or domain, and it can be a fairly arbitrary string (the charset is limited to the charset for host names). Then, in a VM setting, for every API in the table above that returns a mapping from node name to data, there will usually be only one member in the mapping, for the single node that is the VM itself. Since the cluster name and node names in the cluster are in different namespaces, the two names can be identical strings if we like (though I probably would not do that w/o good reason).

To illustrate this, my test setup has a cluster on naic-monitor called 'my.cluster.no' with naic-monitor itself being the only node in that cluster. Then we get this type of interaction with Sonalyze for the rest API listening on localhost:8888:

$ wget -q -O- http://localhost:8888/cluster
[{"time":"2026-03-12T05:26:41Z","cluster":"my.cluster.no","slurm":0,"partitions":null,"nodes":["naic-monitor"]}]
$ wget -q -O- http://localhost:8888/cluster/my.cluster.no/nodes/last-probe-timestamp
{"naic-monitor":"2026-03-11T09:23:00Z"}
$ wget -q -O- http://localhost:8888/cluster/my.cluster.no/nodes/info
{"naic-monitor":{"time":"2026-03-11T10:21:00+01:00","cluster":"my.cluster.no","node":"naic-monitor","os_name":"Linux","os_release":"5.14.0-611.13.1.el9_7.x86_64","memory":49034032,"sockets":4,"cores_per_socket":1,"threads_per_core":1,"cards":[]}}

[lars-t-hansen]

Row 8 is supposed to be .../process/gpu/util not .../process-gpu-util.