Error thrown on GKE cluster, permission issue

[EricVS]

Hi,

I just deployed your application, created a Helm chart for it, but I get this error with your ClusterRole:

I0114 09:24:30.253282 10558 round_trippers.go:405] GET https://10.92.128.1:443/api/v1/nodes/gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv 403 Forbidden in 22 milliseconds I0114 09:24:30.253309 10558 round_trippers.go:411] Response Headers: I0114 09:24:30.253314 10558 round_trippers.go:414] Content-Length: 409 I0114 09:24:30.253318 10558 round_trippers.go:414] Date: Tue, 14 Jan 2020 09:24:30 GMT I0114 09:24:30.253322 10558 round_trippers.go:414] Audit-Id: 58e9a508-c00c-4d00-a71e-7a55fbfa1e24 I0114 09:24:30.253326 10558 round_trippers.go:414] Content-Type: application/json I0114 09:24:30.253329 10558 round_trippers.go:414] X-Content-Type-Options: nosniff I0114 09:24:30.253355 10558 request.go:874] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes \"gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv\" is forbidden: User \"system:serviceaccount:kube-system:node-terrmination-handler\" cannot get resource \"nodes\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"name":"gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv","kind":"nodes"},"code":403} I0114 09:24:30.253923 10558 taint.go:81] Failed to remove taint: nodes "gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv" is forbidden: User "system:serviceaccount:kube-system:node-terrmination-handler" cannot get resource "nodes" in API group "" at the cluster scope I0114 09:24:30.253947 10558 handler.go:90] Failed to process initial node state - nodes "gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv" is forbidden: User "system:serviceaccount:kube-system:node-terrmination-handler" cannot get resource "nodes" in API group "" at the cluster scope F0114 09:24:30.253957 10558 main.go:87] nodes "gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv" is forbidden: User "system:serviceaccount:kube-system:node-terrmination-handler" cannot get resource "nodes" in API group "" at the cluster scope

Looks like the ClusterRole is still missing something to access the node pool. Any idea what needs to be added?

Kind regards,

Eric V.

[rohitagarwal003]

Can you verify that https://github.com/GoogleCloudPlatform/k8s-node-termination-handler/blob/master/deploy/rbac.yaml#L35 is correctly applied on your cluster?

[eric-aops]

Hello,

Thanks for your reply, this is what's current in place on my cluster:

╰─$ (⎈ gke_dashur-dev_asia-southeast1-a_dashur-dev:kube-system) k get clusterroles node-termination-handler -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2020-01-14T09:13:27Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: node-termination-handler-1.0.0
    k8s-app: node-termination-handler
    name: node-termination-handler
  name: node-termination-handler
  resourceVersion: "77819"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/node-termination-handler
  uid: 262267b3-66d4-4acc-9576-8fa84929afbe
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - update
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - delete

and the clusterrolebinding:

╰─$ (⎈ gke_dashur-dev_asia-southeast1-a_dashur-dev:kube-system) k get clusterrolebindings node-termination-handler -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2020-01-14T09:13:27Z"
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: node-termination-handler-1.0.0
    k8s-app: node-termination-handler
    name: node-termination-handler
  name: node-termination-handler
  resourceVersion: "77822"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/node-termination-handler
  uid: 91d1143b-ee48-48a4-8c24-dabb0205fed8
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: node-termination-handler
subjects:
- kind: ServiceAccount
  name: node-termination-handler
  namespace: kube-system

I temporarily assigned cluster-admin role to the service account to get the Pods working.

Kind regards,

Eric V.