Consider adding built-in metric for scrape failure reason

[bwplotka]

Proposal

Discussed in prometheus/proposals#76 (comment)

We propose with @dashpole to add a built-in (like up) scrape metric with limited cardinality that gives a clue where the problem is (e.g. scrapes_failed_total{area=[client / network / timeout / scrape_limit / memory_limit / storage]).

Main motivation is prometheus/proposals#76 where we propose adding one more, non trivial reason

Please do not create PRs for it yet, we need to first agree if we want this.

[bwplotka]

We do not have any metric for failures atm

prometheus/scrape/scrape.go

Line 2010 in 026d284

scrapeHealthMetric = reportSample{

[dashpole]

If we add it, it could be behind extra_scrape_metrics config so we don't increase the default cardinality of most users.

[krajorama]

Hello from the bug scrub!

We're not opposed to the idea of adding a new limited cardinality metric gated by the extra_scrape_metrics configuration.

[dashpole]

I looked through all issues on the prometheus/prometheus repo for scrape failure logs, and this is what I found:

Scrape Errors

Error Text	Go Error Type	Occurrences
context deadline exceeded	context.DeadlineExceeded	14
connect: connection refused	syscall.ECONNREFUSED	9
local error: tls: unexpected message	tls.AlertError	5
context canceled	context.Canceled	4
server returned HTTP status 503 Service Unavailable	(http StatusCode)	2
server returned HTTP status 403 Forbidden	(http StatusCode)	2
server returned HTTP status 406	(http StatusCode)	2
bind: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.		2
read: connection reset by peer	syscall.ECONNRESET	2
connect: no route to host	*net.OpError(syscall.EHOSTUNREACH)	1
x509: cannot validate certificate for xxx.xx.xx.xx because it doesn't contain any IP SANs	x509.HostnameError	1
connect: invalid argument	*os.SyscallError(EINVAL)	1
unexpected EOF	io.ErrUnexpectedEOF	1
net/http: HTTP/1.x transport connection broken: malformed HTTP response		1
server returned HTTP status 400 Bad Request	(http StatusCode)	1
tls: failed to verify certificate: x509: certificate signed by unknown authority	x509.UnknownAuthorityError	1
remote error: tls: bad certificate	tls.AlertError	1
server returned HTTP status 500 Internal Server Error	(http StatusCode)	1
lookup RBIALIMONP004V.b2b.regn.net on 127.0.0.11:53: no such host	*net.DNSError	1
server returned HTTP status 404 Not Found	(http StatusCode)	1

Proposed mapping to string error categories

Go Error Type	String in Implementation	Occurrences
context.DeadlineExceeded	"timeout"	14
syscall.ECONNREFUSED	"connection_refused"	9
tls.AlertError	"tls_error"	6
(http StatusCode) [4xx]	"client_http_error"	6
context.Canceled	"other"	4
(http StatusCode) [5xx]	"server_http_error"	3
syscall.ECONNRESET	"network_error"	2
x509.HostnameError	"tls_error"	1
x509.UnknownAuthorityError	"tls_error"	1
*net.DNSError	"dns_error"	1
*net.OpError(syscall.EHOSTUNREACH)	"network_error"	1
*os.SyscallError(EINVAL)	"network_error"	1
io.ErrUnexpectedEOF	"other"	1
(Not specified / Empty)	"other"	3

The overall categories of areas would be: "dns_error", "connection_refused", "tls_error", "network_error", "client_http_error", "server_http_error", "body_size_limit", "other".

Open Questions

How could we reduce cardinality (and should we?):

• Move "connection_refused" and "tls_error" into "network_error". But connection_refused seems very common, and tls also seems useful.
• Move "body_size_limit" into "other". No reported cases of this, so maybe not important
• combine http client and server errors?