`ocm describe cluster` fails for Archived/Deprovisioned clusters

[cben]

here is an example Deprovisioned subscription

> ocm get sub 2QaydcSjzvQGWHqxRpSReT7mWW4
{
  ...
  "cluster_id": "24301dug094uiivh57o0hfdhotp1cbom",
  "display_name": "celia-imdsv2",
  "external_cluster_id": "89ffcb09-33bd-47ce-aa21-e793383b136d",
  "href": "/api/accounts_mgmt/v1/subscriptions/2QaydcSjzvQGWHqxRpSReT7mWW4",
  "id": "2QaydcSjzvQGWHqxRpSReT7mWW4",
  "kind": "Subscription",
  "managed": true,
  ...
  "organization_id": "1wuVGGV6SCmD8ya6yRGEJzvmVuC",
  "plan": {
    "href": "/api/accounts_mgmt/v1/plans/MOA",
    ...
  },
  "provenance": "Provisioning",
  "status": "Deprovisioned",
}

and now let's try to ocm describe cluster it:

> ocm describe cluster 89ffcb09-33bd-47ce-aa21-e793383b136d
Error: Can't retrieve cluster for key '89ffcb09-33bd-47ce-aa21-e793383b136d': There are no subscriptions or clusters with identifier or name '89ffcb09-33bd-47ce-aa21-e793383b136d'

> ocm describe cluster celia-imdsv2
Error: Can't retrieve cluster for key 'celia-imdsv2': There are no subscriptions or clusters with identifier or name 'celia-imdsv2'

> ocm describe cluster 2QaydcSjzvQGWHqxRpSReT7mWW4
Error: Can't retrieve cluster for key '2QaydcSjzvQGWHqxRpSReT7mWW4': There are no subscriptions or clusters with identifier or name '2QaydcSjzvQGWHqxRpSReT7mWW4'

> ocm describe cluster 24301dug094uiivh57o0hfdhotp1cbom
Error: Can't retrieve cluster for key '24301dug094uiivh57o0hfdhotp1cbom': There are no subscriptions or clusters with identifier or name '24301dug094uiivh57o0hfdhotp1cbom'

These messages are both misleading technically (there is a subscription with these ids) and unfortunate ☹️.

Here are the accessed APIs:

Request URL is 'https://api.stage.openshift.com/api/accounts_mgmt/v1/subscriptions?search=%28display_name+%3D+%2724301dug094uiivh57o0hfdhotp1cbom%27+or+cluster_id+%3D+%2724301dug094uiivh57o0hfdhotp1cbom%27+or+external_cluster_id+%3D+%2724301dug094uiivh57o0hfdhotp1cbom%27%29+and+status+in+%28%27Reserved%27%2C+%27Active%27%29&size=1'

Request URL is 'https://api.stage.openshift.com/api/clusters_mgmt/v1/clusters?search=id+%3D+%2724301dug094uiivh57o0hfdhotp1cbom%27+or+name+%3D+%2724301dug094uiivh57o0hfdhotp1cbom%27+or+external_id+%3D+%2724301dug094uiivh57o0hfdhotp1cbom%27&size=1'

clusters API indeed stops returning data on Archived/Deprovisioned clusters, but subscriptions API retains a lot of info that could be shown (as UI does on https://console.redhat.com/openshift/archived).

Peeking at GetCluster() helper code, it does query subscriptions but appends status in ('Reserved', 'Active') clause.

• It's used by many commands, most of which still want to only deal with active managed clusters. For those, the error message should be adjusted to say a subscription exists but is inactive.
  => improve GetCluster message when Subscription exists but is inactive #518
• rename GetCluster -> GetActiveCluster?
• GetCluster helper returns a cmv1.Cluster so would be unsuitable anyway when we only have subscription data.
  => Add a new GetSubscription / GetSubscriptionAndCluster helper?
• PrintClusterDescription would need many adjustments to deal with partial data...
• Accept subscription IDs (currently only looks up by cluster ID | external UUID | display_name)
• --output and --json flags now dump Cluster JSON. How can we extend them to include Subscription too / support cases of having only Subscription?

[cben]

So my #518 caused some complaints, at least from 2 internal users, when reusing a display_name of previously deleted clusters 👎

Scenarios:

1. The key matches zero clusters — no change.
2. The key matches exactly 1 deleted cluster 🗑️.
   This is what improve GetCluster message when Subscription exists but is inactive #518 wanted to improve, by telling you e.g. Cluster was Deprovisioned, see `ocm get subscription 2S434jffVG4fSgDdK4cbkDTxVNp` for details
3. The key matches >1 cluster, all deleted 🗑️🗑️🗑️.
   The error message that There are %d subscriptions with cluster identifier or name '%s' is (1) technically correct but hardly actionable, because it lists no ids, and ocm list clusters won't even show deleted clusters.
4. The key matches exactly 1 active cluster — no change.
5. The key matches 1 active cluster + 1 or more deleted ✔️🗑️🗑️
   This is the UX regression 👎: Previously ocm commands would just silently use the active one, now they exit with error.
   User can still proceed by doing ocm list clusters NAME, and passing exact ID instead of name.
6. The key matches >1 active clusters ✔️✔️?! (A) Unlikely but technically possible if you set one cluster's display_name to equal another's ID (B) If you use undocumented percents e.g. %foo%.
   No change. (This is the ambiguity subsTotal > 1 check was intended to guard from.)

=> So I'm gonna revert it (maybe tweaking the "not found" wording a little).

Luckily, both old and new GetCluster() logic only allow "Reserved" and "Active" cluster:

ocm-cli/pkg/cluster/cluster.go

Lines 199 to 200 in 6d866ac

	if !ok || (status != "Reserved" && status != "Active") {
	err = fmt.Errorf("Cluster was %s, see `ocm get subscription %s` for details", status, subID)

There is no functionality yet to actually operate on archived/deprovisioned clusters, so it won't break any command.
Only the message for "matches exactly 1 deleted cluster" will become less informative, but the "1 active + 1 or more deleted" use case is more important.

[cben]

Huh, #518 may have also broken handling of clusters in some other states. When zero subscriptions matched, it falls back to querying clusters, and if it finds exactly 1 it returns it successfully.

For example IIUC in old logic, a Stale subscription would not match so it'd fall back and the cluster could be returned(?), while with #518 it will find the subscription and error out, not give the fallback a chance 👎
This further supports reverting.

• need test case for "1 active + 1 or more deleted"
• need test case for fallback to Cluster lookup

[cben]

7. 1 Stale
8. 1 Stale + 1 or more deleted
9. >1 Stale
10. 1 Active + 1 or more Stale
11. >1 Active + 1 or more Stale
12. 1 Active + 1 Stale + 1 deleted

[cben]

Yep, current code refuses to describe a Stale cluster (with unambiguous ID!):

> ocm describe cluster 27p17lo7mpk3ju97ngej0su85stlr6ga
Error: Can't retrieve cluster for key '27p17lo7mpk3ju97ngej0su85stlr6ga': Cluster was Stale, see `ocm get subscription 2YklT9XVqpQfRQ0zVNgtd2icIYj` for details

and works with #518 reverted:

> ocm describe cluster 27p17lo7mpk3ju97ngej0su85stlr6ga

ID:			27p17lo7mpk3ju97ngej0su85stlr6ga
External ID:		f6b170a4-13bf-4a89-9f3e-5f977e4109e9
Name:			f6b170a4-13bf-4a89-9f3e-5f977e4109e9
Display Name:		f6b170a4-13bf-4a89-9f3e-5f977e4109e9
State:			ready 
API URL:		
API Listening:		
Console URL:		https://console-openshift-console.apps.spoke-0.qe.lab.redhat.com
Masters:		0
Infra:			0
Computes:		0
Product:		ocp
Subscription type:	
Provider:		baremetal
Version:		4.14.4
Region:			
Multi-az:		false
CCS:			false
HCP:			false
Subnet IDs:		[]
PrivateLink:		false
STS:			false
Existing VPC:		unsupported
Channel Group:		
Cluster Admin:		false
Organization:		Red Hat Inc.
Creator:		mcornea@redhat.com
Email:			mcornea@redhat.com
AccountNumber:          1460290
Created:		2023-11-27T09:31:03Z
Expiration:		0001-01-01T00:00:00Z

[cben]

• There are also some weird subscriptions without subscription.status field! 🤷‍♂️
  These now give confusing & contradictory errors:

  > ocm describe cluster 2CUX7WsBdoXKI9xMSRPXS4s5oLe
  Error: Can't retrieve cluster for key '2CUX7WsBdoXKI9xMSRPXS4s5oLe': There are no subscriptions or clusters with identifier or name '2CUX7WsBdoXKI9xMSRPXS4s5oLe'
  > ocm describe cluster 1tmtnutmah2arlel7ul65o1efgll2m6i
  Error: Can't retrieve cluster for key '1tmtnutmah2arlel7ul65o1efgll2m6i': Cluster was , see `ocm get subscription 2CUX7WsBdoXKI9xMSRPXS4s5oLe` for details
  > ocm describe cluster 8f736269-9518-4460-95c3-9a14abc7ebe4
  Error: Can't retrieve cluster for key '8f736269-9518-4460-95c3-9a14abc7ebe4': Cluster was , see `ocm get subscription 2CUX7WsBdoXKI9xMSRPXS4s5oLe` for details
  

  (that subscription does exist! "provenance":"Telemetry", but "last_telemetry_date":"0001-01-01T00:00:00Z". The cluster record also exists, "state":"ready". Hard to say from API what happened to it or how come it has no subscription.status, but the CLI errors are not helping... As a rule, I'd like CLI to reflect API reality, even when that reality is borked.)