error-central.js/index.html at master · error-central/error-central.js · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
<html>

<!--
			Github issue: https://github.com/facebook/react/issues/16501
			SO question: https://stackoverflow.com/questions/47292727/react-uncaught-rangeerror-maximum-call-stack-size-exceeded
		-->

<head>

	<!-- Basic Page Needs
		  –––––––––––––––––––––––––––––––––––––––––––––––––– -->
	<meta charset="utf-8">
	<title>Error Central: Fresh thinking on error messages</title>
	<meta name="description" content="">
	<meta name="author" content="">

	<!-- Mobile Specific Metas
		  –––––––––––––––––––––––––––––––––––––––––––––––––– -->
	<meta name="viewport" content="width=device-width, initial-scale=1">

	<!-- FONT
		  –––––––––––––––––––––––––––––––––––––––––––––––––– -->
	<link href="//fonts.googleapis.com/css?family=Raleway:400,300,600" rel="stylesheet" type="text/css">

	<!-- CSS
		  –––––––––––––––––––––––––––––––––––––––––––––––––– -->
	<link rel="stylesheet" href="css/normalize.css">
	<link rel="stylesheet" href="css/skeleton.css">

	<!-- Favicon
		  –––––––––––––––––––––––––––––––––––––––––––––––––– -->
	<link rel="icon" type="image/png" href="images/favicon.png">


	<!-- This is Error Central -->
	<script src="./error-central.js" charset="utf-8"></script>

	<style>
		textarea {
			font-family: monospace;
			width: 500px;
			height: 50;
			text-align: left;
			background-color: #222;
			color: yellow;
		}

		/* Larger than phablet */
		@media (min-width: 550px) {
			h1 {
				font-size: 4.0rem;
			}

			h2 {
				font-size: 3.6rem;
			}

			h3 {
				font-size: 2.8rem;
			}

			h4 {
				font-size: 2.4rem;
			}

			h5 {
				font-size: 2.0rem;
			}

			h6 {
				font-size: 1.5rem;
			}
		}

		img {
			margin-bottom: 2rem;
		}
	</style>

	<script>
		function fd(x) {
			x(5);
		}
		function fc(x) {
			fd(x);
		}
		function fb(x) {
			fc(x);
		}
		function fa(x) {
			fb(x);
		}

		function formEvalForError(f) {
			try {
				eval(f.elements["code"].value);
			} catch (e) {
				throw e; // intentionally re-throw (caught by window.onerror)
			}
		}

		function jsonTest() {
			try {
				return JSON.parse({ "balance": 0, "count": 0, "time": 1323973673061, "firstname": "howard", "userId": 5383, "localid": 1, "freeExpiration": 0, "status": false });
			} catch (e) {
				throw e; // intentionally re-throw (caught by window.onerror)
			}
		}


	</script>
</head>

<body>

	<div class="container">

		<script>
			doGithub = false;
			doEc = false;
			doSoGeneric = false;
			doDependencies = false;
			doSo = false;
		</script>

		<h1>Re-thinking Error messages and automating good engineers
			<!-- : Demo of Error Central for javascript -->
		</h1>

		<p>
			<em>The Github repo for this demo is <a href="https://github.com/error-central/error-central.js">here</a>.</em>
		</p>

		<p>
			<em>tl;dr error messages need <strong>smarter search</strong>, <strong>social analytics</strong>, and
				<strong>a move to real machine learning</strong></em>.
			This is an interactive demo imaging a better experience for error messages.
			I hope it will spark discussion, spur some interest, and perhaps lead to some great new tools.
		</p>

		<p>
			To view the demo <strong>open the javascript console in your browser</strong> by pressing
			<code>Option-Command-J</code>. Please experience it in Chrome or Firefox.
			(I'm showing this off in the browser so that you can play along in the console, but these same principles could be
			extended to any IDE or terminal session. At the end of this article I show some exploratory work I've done in
			those areas.)
		</p>

		<h2>Smarter Search</h2>

		<!-- <h3>Errors should link to docs</h3> -->
		<p>
			Let's begin with a small improvement to simply connect errors to the web. Just give the
			user a link to the official docs for the error. Better yet, link to the docs in the user's language.
		</p>
		<form>
			<textarea name="code">
undefinedFunction(4)
			</textarea>
			<input type="button" onclick="formEvalForError(this.form);" value="Execute" />
		</form>

		<img src="images/ec-docs.png" />

		<h3>Automatically search Stack Overflow</h3>
		<p>It's an open secret that the first thing you do when you get an error is to search for it. So why are developers
			manually selecting text and copy-pasting it into Google? We know that 90% of the time we're going to end up on
			Stack
			Overflow, so let's automatically search there.
		</p>
		<form>
			<textarea name="code">
if (3 > Math.PI {
	console.log("wait what?");
}
		</textarea>
			<input type="button" onclick="doSo=true; formEvalForError(this.form);" value="Execute" />
		</form>

		<img src="images/ec-search-so.png" />

		<h3>...But be smart when searching</h3>
		<p>
			Any developer knows when to strip out details that are specific to their code. Our system should be smart enough
			to do
			this too. Of course, there are times when these little details are important, for example when the error happens
			in a
			specific line of a dependency. So the right thing to do is to search for both the exact text and for a "generic"
			version.
		</p>

		<form>
			<textarea name="code">
undefinedFunction(4)
			</textarea>
			<input type="button" onclick="doSo=true; doSoGeneric=true; formEvalForError(this.form);" value="Execute" />
		</form>

		<img src="images/ec-search-generic-so.png" />


		<h3>Recognize what a “good” result is</h3>

		<p>Our smarter error handler should also have the smarts to know what a good result is. A "good" result is one that
			is
			recent, that contains a substantial match to the query.
		</p>

		<p><em>Not yet implemented :(</em></p>


		<h3>Search known issues</h3>
		<p>
			Sometimes the error you're having is a known issue with your repo. Shouldn't you be notified of that right away?
			So let's also search the repo issues and wiki!
		</p>

		<form>
			<textarea name="code">
undefinedFunction(4)
					</textarea>
			<input type="button" onclick="doSo=true; doSoGeneric=true; doGithub=true; formEvalForError(this.form);"
				value="Execute" />
		</form>

		<img src="images/ec-search-issues.png" />

		<p>
			And while we're at it: If there is no open issue for our error, let's provide a quick link to create one!
		</p>

		<form>
			<textarea name="code">
aDifferentUndefinedFunction(4)
						</textarea>
			<input type="button" onclick="doSo=true; doSoGeneric=true; doGithub=true; formEvalForError(this.form);"
				value="Execute" />
		</form>

		<h3>Search with dependencies</h3>
		<p>
			Any good developer knows that sometimes you need to "massage" your error search query with details about your
			environment. The first plan of attack here is to try adding dependency names to your search queries. So if your
			project
			uses react, you might add "react" to the query. Rather than try to guess which dependency to add, why not just try
			all
			of them?
		</p>

		<p>
			Doing this in the demo would exhaust our quota for Stack Overflow, so we'll just link to the searches. (But in a
			perfect solution, all of these searches would happen in the background and you'd be alerted to any good looking
			results.)
		</p>

		<p>
			Run this code, and then select the "React" search from the dependencies.
		</p>
		<form>
			<textarea name="code">
Array.apply(null, new Array(1000000)).map(Math.random);
								</textarea>
			<input type="button"
				onclick="doSo=true; doSoGeneric=true; doGithub=true; doDependencies=true; formEvalForError(this.form);"
				value="Execute" />
		</form>

		<img src="images/ec-search-dependencies.png" />

		<p>
			All this searching is great, but sometimes the problem isn't in our repo at all, but in one of our dependencies.
		</p>
		<p>
			So let's search issues for our dependencies.
		</p>


		<p><em>...not yet implemented...</em></p>

		<h2>Social Analytics</h2>

		<p>All that automatic searching is great, but sometimes the solution doesn't live in a web result</p>

		<p><strong>Who of my co-workers has seen this error lately?</strong> Errors are often very specific to the codebase.
			Knowledge of “How to fix error X?” often lives only in the head of someone else. So when you see an error, you
			should instantly see who of co-workers has seen this error lately.
		</p>

		<p><strong>How often is this error happening?</strong> My engineering team once lost an entire mornign trying to fix
			a "bug" that turned out to be very subtly caused by an overnight update in an npm package. Little did we know that
			that literally thousads of other developers aroudn the world were getting the same error that morning! We should
			have seen immediately that this error message has spiked in the last 24 hours.</p>

		<form>
			<textarea name="code">
undefinedFunction(4)
								</textarea>
			<input type="button"
				onclick="doSo=true; doSoGeneric=true; doGithub=true; doDependencies=true; doEc=true; formEvalForError(this.form);"
				value="Execute" />
		</form>

		<img src="images/ec-analytics.png" />


		<h2>Closing thoughts</h2>
		<p>
			While this shows off error messagse in a browser, the same concepts apply in any programming envitonment. My
			friend Gabe and I also built similar functionality as a
			<a href="https://github.com/error-central/error-central/tree/master/vscode-extension">
				plugin for VS Code</a>, and directly in the terminal (bash).
		</p>


		<hr style="margin-top:600px;">

		<!--
	<p>Buttons on this page produce errors. Open the browser console Command-Option-J to view console.</p>

	<button onclick="x=undefinedFunction(4)"><code>undefinedFunction(4)</code></button>


	<button onclick="x=undefinedFunction(4)"><code>undefinedFunction(4)</code></button>

	<button onclick="x=undefinedFunctionNoIssue(4)"><code>undefinedFunctionNoIssue()</code></button>

	<button onclick="Array.apply(null, new Array(1000000)).map(Math.random);"><code>RangeError</code></button>

	<button onclick='jsonTest()'>
		<code>JSON.parse()</code>
	</button>

	<button onclick="fa(4)"><code>Stack Function</code></button>

	<form>
		<textarea name="code">randomFunction();</textarea>
		<input type="button" onclick="formEvalForError(this.form);" value="Execute" />
	</form>
 -->

		<div style="color: lightgrey">
			<h1>Old outline:</h1>
			<h3>Background</h3>
			<p>
				Its an open secret these days that software development consists of a lot of
				"run code", "get error", "copy-paste-error to google", "open top Stack Overflow link".
			</p>
			<p>
				<em>Good</em> developers know meta-search skills like removing code-specific
				function and variable names or recognizing when the results are too general.
				They know when to add a dependency to the search, or perhaps a bit of the stack trace.
			</p>
			<p>
				<em>Experienced</em> developers know when they are out of options, and will
				"escalate" to asking co-workers/co-contributers on Slack. "Has anyone seen
				this error before?"
			</p>

			<h3>What is this?</h3>

			<p>
				This repot is a "concept car" of what error messages could be. We hope it will spur
				interest and perhaps this can become a real tool someday. This works for
				javascript errors in Chrome or Firefox, but these ideas could be extended to
				anywhere errors appear
			</p>

			<p>
				We have also built basic versions of this functionality as a
				<a href="https://github.com/error-central/error-central/tree/master/vscode-extension">
					plugin for VS Code</a>, and directly in the terminal (bash).
			</p>

			<h3>Fundamentals</h3>
			<pre>
		- Deeper description of error from official docs
			- Ideal world there'd be some standard so that any package (e.g. react)
			  could publish error info. But for now we do it by hand.
		- Where possible, error description in developers native language (esp
		  important for beginnerin developers)
			- '<em>"0"-prefixed octal literals and octal escape seq. are deprecated</em>' is a
			  tough English sentance for a non-native speaker.
	</pre>

			<h3>Automate actions good devs do already</h3>
			<pre>
		- Search google. 80% of the time land on stackoverflow
			- Including long-shot searches that a coder would only do
			- Remove code-specific parts of error that would be irrelevant,
			  e.g. "myFunnyVariableName"
		- Search open issues for the project
			- If some other dev has reported the problem, know it immediately.
		- Search dependencies
			- Use stack trace to infer what dependency might be culpre
			- Serach open issues for dependencies
			- Extend serach to issues for dependencies, even going down entire
			  dependency textarea
		- Search with stack trace
		- OVERALL: know what a "good" result looks like.
			- A match with stack trace is better than one just for general type of error
	</pre>

			<h3>Analytics and sociality</h3>
			<pre>
		- Record errors and context.
		- Has this error been spiking worldwide in the last 24 hours? (Maybe a
		  dependency had a botched upgrade.)
		- Basic analytics on error freqency and first occurance. (a la Sentry)
		- Who of my co-workers/co-contributors has seen this error?
			- Especially if with same/similar stack trace / branch
	</pre>

			<h3>Machine Learning</h3>
			<pre>
		- We can also go <em>beyond</em> automating developer behavior.
		- Instead of naive text matching search, use the full context of an error
			when searching for relevant answers, issues, people.
			- Repo / branch
			- Stack trace
			- Dependencies
			- Input/launch configurations
		- Use the logging info to record the total context of an error. Then when
			the error happens later for someone else, they can know who to talk to, or
			even automatically see what worked to fix it.
	</pre>
		</div>
	</div>

</body>

</html>