Web-GUI not working after Reboot

[discostur]

Hi,

after rebooting CoaL i'm not able to login via web-GUI. I'm getting the following error:

"Services required for authentication are currently unavailable. Signin will be re-enabled when service is restored"

It seems to be kind of a timing issue: after reboot adminui checks if authentication service is available - but not all zones are finished booting up. So adminui needs to check again after some time if authentication is available now.

I can fix the problem via "svcadm restart adminui" when i'm logged in to the adminui zone.

Logfile gives me the following:

[2016-04-02T15:37:35.054Z]  INFO: adminui/audit/15389 on 002ef8fe-783c-4a1c-b2fd-977ade7eed20: handled: 503 (req_id=d263dfd0-f8e8-11e5-a021-b7edb026542f, route=getapiauth, audit=true, remoteAddress=192.168.2.130, remotePort=60730, latency=2, _audit=true)
    GET /api/auth HTTP/1.1
    host: 192.168.9.101
    accept-language: de-de
    accept-encoding: gzip, deflate
    connection: keep-alive
    accept: */*
    user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/601.5.17 (KHTML, like Gecko) Version/9.1 Safari/601.5.17
    x-adminui-token: cd20668f3d5592a8934026d69cdc2a5dfd15d294
    referer: https://192.168.9.101/
    x-requested-with: XMLHttpRequest
    --
    HTTP/1.1 503 Service Unavailable
    content-encoding: gzip
    content-type: application/json

    {
      "code": "ServiceUnavailableError",
      "message": "service unavailable (moray)"
    }
    --
     ServiceUnavailableError: service unavailable (moray)
        at Server.requireAuth (/opt/smartdc/adminui/lib/auth.js:76:21)
        at next (/opt/smartdc/adminui/node_modules/restify/lib/server.js:745:30)
        at f (/opt/smartdc/adminui/node_modules/restify/node_modules/once/once.js:17:25)
        at attachOtherClients (/opt/smartdc/adminui/lib/adminui.js:180:16)
        at next (/opt/smartdc/adminui/node_modules/restify/lib/server.js:745:30)
        at f (/opt/smartdc/adminui/node_modules/restify/node_modules/once/once.js:17:25)
        at Server.attachSdcClients (/opt/smartdc/adminui/lib/sdc-clients.js:36:9)
        at next (/opt/smartdc/adminui/node_modules/restify/lib/server.js:745:30)
        at f (/opt/smartdc/adminui/node_modules/restify/node_modules/once/once.js:17:25)
        at Server.<anonymous> (/opt/smartdc/adminui/lib/trace-event.js:24:13)
    --
    req.timers: {
      "bunyan": 42,
      "parseQueryString": 18,
      "gzip": 162,
      "parseAccept": 59,
      "handler-4": 18,
      "attachSdcClients": 10,
      "handler-6": 11,
      "requireAuth": 522
    }

In Addition, i'm seeing an unhandled expection right after the services has started (not quite shure if it is related to my problem):

[2016-04-02T15:25:11.851Z]  INFO: adminui/15389 on 002ef8fe-783c-4a1c-b2fd-977ade7eed20: HTTP Server started on 0.0.0.0:80
[2016-04-02T15:25:11.852Z]  INFO: adminui/15389 on 002ef8fe-783c-4a1c-b2fd-977ade7eed20: HTTPS Server started on 0.0.0.0:443
[ Apr  2 15:25:12 Rereading configuration. ]
[ Apr  2 15:25:12 No 'refresh' method defined.  Treating as :true. ]
[2016-04-02T15:26:11.928Z] FATAL: adminui/15389 on 002ef8fe-783c-4a1c-b2fd-977ade7eed20: Uncaught Exception
    AssertionError: connection state change after close
        at Object.module.exports.(anonymous function) [as notEqual] (/opt/smartdc/adminui/node_modules/assert-plus/assert.js:242:35)
        at Client._onConnection (/opt/smartdc/adminui/node_modules/moray/node_modules/fast/lib/client.js:430:12)
        at /opt/smartdc/adminui/node_modules/moray/node_modules/fast/lib/client.js:196:14
        at FunctionCall.doCallback_ (/opt/smartdc/adminui/node_modules/moray/node_modules/backoff/lib/function_call.js:162:20)
        at FunctionCall.abort (/opt/smartdc/adminui/node_modules/moray/node_modules/backoff/lib/function_call.js:121:10)
        at Client.close (/opt/smartdc/adminui/node_modules/moray/node_modules/fast/lib/client.js:180:30)
        at /opt/smartdc/adminui/node_modules/moray/lib/connection_pool.js:134:16
        at Array.forEach (native)
        at ConnectionPool.removeHost (/opt/smartdc/adminui/node_modules/moray/lib/connection_pool.js:132:19)
        at /opt/smartdc/adminui/node_modules/moray/lib/connection_pool.js:76:14

Thanks,
Kilian

[bahamat]

It looks like your Moray zone(s) may be unhealthy. Use the following commands to start diagnosing why.

moray=$(vmadm lookup tags.smartdc_role=moray)
vmadm get $moray
svcs -z $moray -xv
tail -50 $(svcs -L moray)

[discostur]

The moray zone looks ok:

vmadm get $moary:

{
  "zonename": "7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
  "autoboot": true,
  "brand": "joyent-minimal",
  "limit_priv": "default",
  "v": 1,
  "create_timestamp": "2016-04-01T13:26:22.559Z",
  "cpu_shares": 8192,
  "max_lwps": 1000,
  "max_msg_ids": 4096,
  "max_sem_ids": 4096,
  "max_shm_ids": 4096,
  "max_shm_memory": 8192,
  "zfs_io_priority": 20,
  "max_physical_memory": 8192,
  "max_locked_memory": 8192,
  "max_swap": 16384,
  "cpu_cap": 400,
  "billing_id": "16425b42-b818-11e2-90d4-87477f18a688",
  "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
  "package_name": "sdc_8192",
  "package_version": "1.0.0",
  "tmpfs": 8192,
  "dns_domain": "local",
  "archive_on_delete": true,
  "maintain_resolvers": true,
  "resolvers": [
    "10.99.99.11"
  ],
  "alias": "moray0",
  "nics": [
    {
      "interface": "net0",
      "mac": "82:cb:5b:1c:bd:4b",
      "vlan_id": 0,
      "nic_tag": "admin",
      "netmask": "255.255.255.0",
      "ip": "10.99.99.17",
      "ips": [
        "10.99.99.17/24"
      ],
      "primary": true
    }
  ],
  "datasets": [
    "zones/7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef/data"
  ],
  "image_uuid": "3c3d2d86-f7cb-11e5-b32d-d768ace12ffa",
  "uuid": "7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
  "zone_state": "running",
  "zonepath": "/zones/7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
  "zoneid": 2,
  "last_modified": "2016-04-02T15:23:28.000Z",
  "firewall_enabled": false,
  "server_uuid": "564d9ee7-4a17-cd23-73b6-cb15390be0f2",
  "datacenter_name": "coal-1",
  "platform_buildstamp": "20160330T234300Z",
  "state": "running",
  "boot_timestamp": "2016-04-02T15:24:27.000Z",
  "pid": 4184,
  "customer_metadata": {
    "sapi-url": "http://10.99.99.32",
    "user-script": "#!/usr/bin/bash\n#\n# This Source Code Form is subject to the terms of the Mozilla Public\n# License, v. 2.0. If a copy of the MPL was not distributed with this\n# file, You can obtain one at http://mozilla.org/MPL/2.0/.\n#\n\n#\n# Copyright (c) 2014, Joyent, Inc.\n#\n\nexport PS4='[\\D{%FT%TZ}] ${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'\n\nset -o xtrace\nset -o errexit\nset -o pipefail\n\n#\n# The presence of the /var/svc/.ran-user-script file indicates that the\n# instance has already been setup (i.e. the instance has booted previously).\n#\n# Upon first boot, run the setup.sh script if present. On all boots including\n# the first one, run the configure.sh script if present.\n#\n\nSENTINEL=/var/svc/.ran-user-script\n\nDIR=/opt/smartdc/boot\n\nif [[ ! -e ${SENTINEL} ]]; then\n    if [[ -f ${DIR}/setup.sh ]]; then\n        ${DIR}/setup.sh 2>&1 | tee /var/svc/setup.log\n    fi\n\n    touch ${SENTINEL}\nfi\n\nif [[ ! -f ${DIR}/configure.sh ]]; then\n    echo \"Missing ${DIR}/configure.sh cannot configure.\"\n    exit 1\nfi\n\nexec ${DIR}/configure.sh\n",
    "assets-ip": "10.99.99.8"
  },
  "internal_metadata": {},
  "routes": {},
  "tags": {
    "smartdc_role": "moray",
    "smartdc_type": "core"
  },
  "quota": 25,
  "zfs_root_recsize": 131072,
  "zfs_filesystem": "zones/7d7c4fd3-c3c8-4ad1-80e3-29fb06f48bef",
  "zpool": "zones",
  "zfs_data_recsize": 131072,
  "snapshots": []
}

svcs -z $moray -xv

-> nothing, but svcs -z $moray says everything is online

tail -50 $(svcs -L moray)

-> empty; think it is beacause i have 4x moray logfiles:

(coal-1:moray0) ~]# svcs -L moray
/var/svc/log/smartdc-application-moray:moray-2024.log
/var/svc/log/smartdc-application-moray:moray-2023.log
/var/svc/log/smartdc-application-moray:moray-2022.log
/var/svc/log/smartdc-application-moray:moray-2021.log

tail -50 /var/svc/log/smartdc-application-moray:moray-2024.log
[ Apr  2 15:12:28 Enabled. ]
[ Apr  2 15:12:28 Rereading configuration. ]
[ Apr  2 15:12:36 Executing start method ("node main.js -f etc/config.json -p 2024 &"). ]
[ Apr  2 15:12:36 Method "start" exited with status 0. ]
[ Apr  2 15:23:20 Stopping because service disabled. ]
[ Apr  2 15:23:20 Executing stop method (:kill). ]
[ Apr  2 15:24:38 Enabled. ]
[ Apr  2 15:25:18 Executing start method ("node main.js -f etc/config.json -p 2024 &"). ]
[ Apr  2 15:25:18 Method "start" exited with status 0. ]

[askfongjojo]

Hi @discostur, I believe what @bahamat meant was that moray was likely unavailable when AdminUI wasn't working for you. To confirm the issue, you can bring up the developer console (or Firebug if you are on FF) to view the ping API response. You will see that the UI is polling for availability of the dependent services, specifically moray and ufds. When moray is unavailable, you'll see something like this in the http response:
{"services":{"moray":false,"ufds":true},"time":"2016-04-01T00:01:14.986Z"}

This should be helpful for your troubleshooting.

[bahamat]

@discostur Based on your feedback, I think you just need to restart the adminui service.

Log into your headnode and run the following commands:

sdc-login adminui
svcadm restart adminui
exit; exit

[discostur]

@askfongjojo you were right, just after the reboot i can see the following response:

{"services":{"moray":false,"ufds":true},"time":"2016-04-06T19:49:51.693Z"}

@bahamat after doing an

sdc-login adminui svcadm restart adminui

i am able to login to the web gui

[bahamat]

Filed internally as ADMINUI-2293.