From d01c0856e15d5ff639a30f71c149addcf146d0af Mon Sep 17 00:00:00 2001
From: saroup <saroup@microsoft.com>
Date: Wed, 12 Feb 2020 13:59:21 -0500
Subject: [PATCH 1/4] Adding design doc that explains how we calculate
 confusion matrix numbers

---
 docs/ConfusionMatrixCalculations.md   | 108 ++++++++++++++++++++++++++
 docs/images/accuracy_table_output.PNG | Bin 0 -> 34710 bytes
 2 files changed, 108 insertions(+)
 create mode 100644 docs/ConfusionMatrixCalculations.md
 create mode 100644 docs/images/accuracy_table_output.PNG

diff --git a/docs/ConfusionMatrixCalculations.md b/docs/ConfusionMatrixCalculations.md
new file mode 100644
index 0000000..48396c7
--- /dev/null
+++ b/docs/ConfusionMatrixCalculations.md
@@ -0,0 +1,108 @@
+# NLU.DevOps confusion matrix calculations
+
+This document describes how NLU.DevOps processes the results returned by NLU provider and describes how the confusion matrix metrics are computed and reported.
+
+The compare sub-command compares the results returned from NLU provider with the labeled test set. It runs the comparison for 4 kinds of results:  
+- Intents
+- Entities
+- Text
+-  Entity Values
+  
+For each test utterance one or more confusion matrix results can be computed:
+- True Positive (TP)
+- True Negative (TN)
+- False Positive (FP)
+- False Negative (FN)
+
+ The cases where more than one confusion matrix result is returned are as follows:
+-	When actual and expected intents mismatch it returns a FP result for the intent that was matched and a FN result for the intent that was not matched.
+-	When actual and expected entities mismatch, a FP result is returned for the entity that was matched and a FN result is returned for the entity that was not matched.
+  
+## Confusion matrix output for intent comparison
+ 
+|Utterance text        | Actual Intent  | ExpectedIntent   | 
+|:---------------------: | :------------: |:---------------:|
+|who is bob goodermuth? | ContactInfo | ContactInfo|
+**True Positive for ContactInfo:** The expected and actual intents are equal
+
+
+
+| Utterance text        | Actual Intent  | ExpectedIntent   | 
+|:---------------------: | :------------: |:---------------:|
+|who is bob goodermuth? | Greeting | Greeting|
+**True Negative for ContactInfo:** Actual intent is not ContactInfo, and the expected result it not ContactInfo
+
+### Producing two results
+| Utterance text        | Actual Intent  | ExpectedIntent   | 
+|:---------------------: | :------------: |:---------------:|
+|who is bob goodermuth? | Greeting | ContactInfo|
+**False Positive** for Greeting and **False Negative** for ContactInfo since intents mismatch
+
+## Confusion matrix output for entity comparison
+When the expected list of entities is not empty, we check that each entity matches by type, value and by the occurence index of the matching text in the utterance.
+| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+|:---------------------: | :------------: |:---------------:|
+|Schedule meeting with bob tomorrow| 	26, 33, datetime, tomorrow <br> 23, 25, personName, bob| 23, 25, personName, bob <br>26, 33, datetime, tomorrow|
+**True Positive** for datetime and personName since they have a corresponding match in the actual entities based on entity type, text match and index match.
+
+| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+|:---------------------: | :------------: |:---------------:|
+|Schedule meeting with bob tomorrow| 	26, 33, datetime, tomorrow <br> 23, 25, personName, bob| |
+
+**False Positive** for datetime and personName since they don’t have a matching entity in the expected entities list.
+
+| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+|:---------------------: | :------------: |:---------------:|
+|Schedule meeting with bob | 	17, 24, personName, bob| 22, 24, personName, bob|
+**False Positive** for personName since the start position doesn’t match with the expected one.
+
+### Producing two results
+
+| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+|:---------------------: | :------------: |:---------------:|
+|Schedule meeting with bob | 	22, 24, userName, bob| 22, 24, personName, bob|
+
+**False Positive** for userName since it doesn’t have a matching entity in the expected entities list 
+**FalseNegative** for personName since it doesn’t have a corresponding match in the actual entities
+
+
+| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+|:---------------------: | :------------: |:---------------:|
+|Schedule meeting with bob tomorrow| 	| 23, 25, personName, bob <br> 26, 33, datetime, tomorrow 
+
+**False Negative** for personName and datetime since they don’t have a corresponding match in the actual entities26, 33, datetime, tomorrow 
+
+
+| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+|:---------------------: | :------------: |:---------------:|
+|Good Morning|  |
+
+**True Negative:** There are no expected entities and no entities identified
+
+
+## Confusion matrix output for entity value comparison 
+
+Besides comparing entity values for the entities comparison we also compute True positive and False Negative metrics by comparing entity values only. 
+| Utterance text        | Actual entity value	 | Expected entity value  | 
+|:---------------------: | :------------: |:---------------:|
+|Do I have unpaid bills?| "bills",<br>"invoice", <br>"invoices"| bills|
+
+
+**True Positive:** for bills since the entity matches by type, startPos and endPos and the expected value has match in the actual entity value subtree.
+
+**True Negative:** Not calculated
+
+**False Positive:** Not calculated
+
+| Utterance text        | Actual entity value	 | Expected entity value  | 
+|:---------------------: | :------------: |:---------------:|
+|Do I have unpaid bills?| "bills",<br>"invoice", <br>"invoices"| checks|
+**False Negative:** for checks since the entity matches by type, startPos, endPos, but the value does not have the expected subtree in the entity resolution value
+
+
+## Reporting F-measure
+After computing the confusion matrix numbers, we compute the precision, recall and f1 score for each of intent and entity and the micro-average F-measure for intents and entities
+
+We print out to the user the following tables that can inform the user on the performance of the model and help them make decisions on which intents or entities to improve.
+
+![accuracy table](images/accuracy_table_output.PNG)
diff --git a/docs/images/accuracy_table_output.PNG b/docs/images/accuracy_table_output.PNG
new file mode 100644
index 0000000000000000000000000000000000000000..abd3604db96e83f0cb3517102b442792e923159a
GIT binary patch
literal 34710
zcmeFZcT|(x)-H|>ww0}-fHb!S6cj`Rqy;-70aQRhN)(VDP(p_!Dk@4<q7Z6Sn$!qL
zVyJ>bjG;&kkU&&Qgb)IR5JE_}FS^e@XYccU_jiAH?C%?QjC=n82d{5gYpycqdY)&_
zM;9$jHgDLqK}<|+v+4OWm&L@^SOGr`zlZ}@<aSPX1OKdoT{bx-R@x>%4*ap!<D~gX
zF|o45jZ4?p0e`Q*dEOo-CZ-U*^0O*&bmoSbShSkynUmI`&aA<WxWcM(vCmyf)h<?(
z4>y-7ReLB-uUVz`Rqbuyl;&79yZh*D(LJf>q7ZJo@$wSqpTGQPjhmh#MSKS4k6-jS
zWX!=o-PB$$yDVk(r<*WHEs@YxShOq&G_LC&&{VNcjk|&0H?IFGCU(30$S-1IC+ua`
ziitfxv}v`&@ht4JNVqWrOZ?6i6N^kb@{8|D&2?haDfb%^gA7f!x4+}d;2@JXtFqct
znDylG@P{JNBhfeg<-)wTX&1qIbz))>>!Xx>*e(+{3zbW_g5?*Fm=H$>MmrlF5V=Y$
zc)R#=T;AJoO6DuYCMo}1^lGsMv&Ju&7wBE?<-+I#d17Lsq$3L+aT<ABC|$V-)@Zms
zZCMan=cs{y#2g&GY<|rjX(n%xN!g@o?kP!R7T{ZuDP(S>q1!;@ls=>_pE6qd)gF9|
zDdh{RoOJ0p5@xKT?9b-N*f4|i*CMgtgO8hQ<~5WcVz=4bfVPY`b=Ncbfi+oOBv)d^
z%`iK!Z{e?$Nlm<h*q15623QU!!NkHV31oW}{0!&XBV6({@QGOV)2t-VtP@*M{<?4C
ztbh3sc%k`wrBi=~%f$HQ$}o$al0>(=S3|M(iPrm3bTb>NT$Vm+Ib{?vroS9=+%YlV
zyEc02*Ve5gSE`o6kw(J|S1fB%=$KktYanH1gdAHCC+e_aM0v5>=L}`X1$8c}X$uLd
zPc1NS6VSf8+o__u0iR4YEoE-e_mL}44M?DIdCq<T;b=sJ$)S)>5$$$~#;`~OV(>>s
zf}WVzBfy?52lJA_;VPDn>FXoCItM_Flv-+emp<h}i26`Lf7a5y<A!6g^ROH9^P&zH
zZ|v|!v1un@SY~FLtT|c3ffOjwrLc378ogMC?b57J^hv!0rX3n+58>c=i94;HIVEU_
zy$e|%g?!*Lv6-3i5&q(|(F}r#=d&(?d-D9*^uu}H=_@w(@5G#G0`WK-k@@H}J9*k9
z1gxC}xiD>qsn(UJ5(GXSh%IqLL-kHzZQSv?R$TlH!F7w+iM?5IbMUDTW5clpV-*Ks
z^Y0~gSQU(LblwZ#FSk;+Uwwm-kiv&7$Y=|eQLzQx)d}pRTdt-KVz+rfo0|Qay1$eB
zW(5~fDtQqloYynJQ?gAT29tz_hw5utnlZ`gh-ZRV(6llka$;e%SY!%dF<&QeQ0V}q
z8D5%`IoS&><beWHke&#Nit!~nB9qB2aE>sXn+mpGb~?=ABl@{%KIb%Njv<XTs6)Yo
z6efYcIy&5*v`CBV%hRJSr~GaprJ{pb4{C~z6i88i<Gkvbp<(PKK%SWf_f0L_9i&sO
z!28Bnj9h9uZAZi{LROlR?UcV&fUVERcXc8yqO{nT>U#}|-1SY?tY9uE?H6g~=h5$H
z!V8rJtxor%$)k}*_v^V841G$-0SgKzx4y0iNnr{~v7epofMH-W1{zr<$@X~Go1%be
zT=7_2haje40COBzwDkR(R*Q*U+$p|FOzgnMC}4khdjJ1pulO&$Jt5$@Qz@_$k@b~^
zgSDa6Z=hb(T&NjU9qKR;>_zRwnK{@w$;h9LW_99np((h-f4aqNpz*?gzRCS94gca6
z-8H(%rq?s;Ihsob)6R>#>1RNwy}BQt$W89sWTAZLI^_SyrFzm#Bps{KMsb~6>`ua|
zCCQC^+#I}3Tm|*+wFdm=#%SjFQeS(@VU@=bheM2K`Q}nqk(z`Ci^eZ*4*REPmgH+B
zYJ7~g<TD9b-HsNN({nt_IcZ8N2{#_qlOyV6`oxpDm(AxaiSK!1_Wno1^c22PN}$-$
zDbHotCnHQ^v<97G?$>(YVe%WNYZdEZxX{U6Q;E^4?S+xyHKNEk=l8p$tXBFpuhwK;
zk%@eDgdfXn38c<7T-JQ{*8EiVJTlY3%7LE)`!t%a;vQ2_)6`^bb0)LRd@DBx9gb-Y
zSW^YGHDdow*dCfEp*2`FZ{Y~=G!7k}dP0867UIv`O}Fzbf?b%G4><1JPP!k3%@kA1
zn@wX?J6dkn*rHH~OYsoVrus0n)PZ;IgjG)0Z<g0?v(wa`)F{>Jf_6!4Z0bH;p|}Gj
zJ~l9saTyeCy{>0(3Lb=qp3{7$TaEVS<kc008L$<ltQ`6d={{H6WP!6?ov4wQg?h($
z-Uoj`2u6bHDC)9eCI@#MiXLWN%4`!jsrS4~6U31uinWl<I`Ers(G%c(i`Xy=i?oc5
z0gHx@6H$Uol&`_W-#n4w;PAz?^)~c->)qTlnb20y7D8hHhC1K{8R#Xrw#h}F)9e`&
zRJle_Mly<f$i+&(r3Xf?G28HRKU8{Q$)(Vl$YCw$YBqH%|8D7!m;&$JMHa_edzz(d
zPr`B=M}o=5;X<ngy@zm0YW?W;*n*TGpeYj>YNO+Y!tVt<8JFD|BgeKE*4^6vaqR8L
z$Rhpa=oWuwA@FtIAPZS1@Qq&Alni;T=0pWbx}jW3yMoz(`s@D4?EbnPvrd|ok7~zW
zF4Y`4-tWBLFdrT6?N@KRg~Vye<$2>^7gi;0dYLk`<9*wsg&PEeQY~Z~V34bt&uX=v
zofn)LRLYu3Sx=lq1XAL51PMe6FNlm<3s+^mR{aF@iJiTCT+3ox<LAt``RmEhfUh?Q
z(G&ipx^UU;y|S?0*FAc<p<Bk~>vz+6pGe^Zn<v4ADgEgq`Q%n!$A_npQx2WMcS-`X
z2SH8UkTQF}6SguLzSy-{4&bvYsqS(c&Jxj-z<6||PLwb5-H=ZvKNgtx@A<T1lVhvA
zlvPse^Y!Tc3HyZygbpu+c2>%deJpf{brdwzJmGYR3%(6q@c=rjhuktQmGBj`{5)ea
zH83uD<a-+5sNo9X$vmpYQ-0$<qJt(Els+mRc)x)(f8mQ>Q}^`Ry+{{yU{ay>8qaaJ
zE@x7^xv!3shbRZSc<ncp2>O`xm@f*;Uw^(YUz(deVnq4A2KaX7dKkN{+j?&bb>!qs
zkZDIZDJ7^mLmL<%uYkSz2(ZSd9?MUpEK^re(lFtY`W*C`9BcmBfjkiDk6FI}_`yq2
zw~DI$Vh%P_^CXo~(|3DRzNvAuZRT4)K)+k|GKqu<C7lFkhhqT*SK^0-pqK*d#fQVO
zgRt2dwCD211D9h8Zltl~?2iC%*7>JNuYhZ>XYx^i`AGX-V?{dUo%pVvF^jsP!Nfdt
z+2+jl_I=_!TknwoV#1@Dx35<$SJHN{Lo^|erM>N+jJ|cHmE`8)oP)la*Z=3i5*ajg
zXP{awm@Sw$=1Pvo*Jn_bR^Dn?uHLSZ8Cu*v`0>%e$I%3Z?ZcE3t+C?kz+T{o)F!vH
zVh01tp$-fS1{1HO9M3nyxn?_lr>g1>u&lts9se8=U@fPu3T(Sp)1}W*0XBpb@M<-f
z3M6*%(ou`z4vF5h6L}%h&GHpb$O;rXjCd<k(_^zkrqwLtaWLc_waV5)ISgfR4%NK0
zp9uSpzc3h6(4mqzg1@TFPOt4vt@lg^9p9s3=5AS+JeU1OviQgo`&+B^5KDz|PE%-8
z2adDQ^rcH-c7LxYa`GwaG?6#I=3OH9GWalxrK0sGyAM^NFGze?Jfl6v*)nd*5w%_%
z992rAr;q!;M=)FH73NB`BJ|bHL~qgue40#;@p)>h8)%?ZXkdk>2NoJj2N}*OSN0%n
z*__Fg?D^4qqdLRi$T-k^hbsX~c!{3y=9#>y6^yubmTG135?I6Rn1Xd_|7t%S<C)E2
z_K95O^-Vz^Gin|nMnz-qfiB)i2fc-8+N4tw6+6}DWwEG?z@tXHznf%|f-U-%lMnNB
zzAAbuA*}|(UE40FW2^$mUO3&Xwf-2?=`foe?(IS0r1<?|mB&;l6*hivh=_1a?eNm_
zb1%-hR?u#j0$a_PO5JZ7%(AI{UxRJ*^vk3y_C!y72rIM*$W)7dHkL988JMhZTT%dg
z#a2E0qrX95dA#iI;e?B9W)5??)c7N0A}Xp8VXeWoHb)DX4#JSyx_<0=i0ijSTJIE-
zsBWvteJUT@Qa7Zd!gfxCRxWwVZQ*`c3X5~@4KZ~td4ghI1V70}yQo~hd360IL8k;?
zc{Wv^a58~*?eD$x9~`&-mPq73JIwuG5<=k|Rx7pRN3#+e=wG6bKbkfl(b5ZulK&YS
z33ghgFvovg+KhOJ`sGjY8P0yS5)dQv1C2nj`3TJ*Am&KnaM8FsO0W5*ITJqloYC-t
z6#mVE#i8-ghmiX6_cMVIQLYoB1esFRud(-R2!VMcQ)3V+v|SF--+5Lsq>b#jDdr=i
zTGP_$ke5K36ki>}Xzs)L#FhE<8|r>h2jUHz)%eW$%RwIjw;oeOuCEH!cT6;mt`i>i
zl%#u;Q6E2(IJ0d*uQXoGSxgo5Sk6VQyJWPJI}qpRU<E!q!g0FwjgZ4qQr4XsN|tj-
zEK?ges=IruxC-^y9VOdGe&lM}GB0e@ZTf;{WI^lVwadG^g{;qW`1jm}$rLW#!vF2_
zOPM3-Mel8ZNHgsX8S=b8?@MVg7cY3*IheIr2#v_qhv_cy>r%Xp&mvXKK$4-hkcBlP
z&9^OEiaKSg_cTcvtdBymlHS?m8|!A)O*}Ua*mLKj4t)BikrCF;JsD<bEUH^H<@C+-
zy!)9uR9HIZSyI~D@3jo>Z|e4)C5$-1zxqFx+mAupZfx#*^fS!Cd97ATDms-SSRP`7
zok|&9>`c4oI8!Cx(yKca&CN=EOZmr5D&RiS%4VhLEiB2EH|epV@UMxF<-(LwZRG}^
z6+)Bbrd5B~SC>rKVG&Kdb|vKV98MtY4eVB|>)8pvx!@Mzsnk$X9I~Ulwwy4<nJ1T`
zal@C~lb)l~%t4HrB8?@5fW3_EoI*G~gbuv%iQKQhM4_O`?}BEYBGe~z>`hDMZIC=D
zV(I1K#cLcpv)6G`9uH@+=?wmY=~vYm`3fvrv6R?FAOvJ)Mjps>N44wuPi#a!q<!SS
zs~MOswKmsmS_#*y_us5X-&Nv@K<MhmKEaXT%B8V%ZHnriG@@7G_F0Zkj1sf+5O1D9
zI%o@`vScgT2ii^c6|&N~J>OHjg;5%n&-Y`r_)E!k)A*0^x+BdJzy!MpE<*u(X<bt{
z8#E<m838_0W$UTY$}T;|Ebm>r*Y@e!bH*1sYZz~1HLq)OJu&d&(3Ws)Wnx`iWL*S_
z5i)n*_?#cDBu$R}v1rT)U;RZWqTRkIlL21+z7CgpO&(JEIdA(Tyu}f9o)USohW!=2
zr2YxLoj0qN7u|L=-|=%nz4u!`>Xh;Dm>Pd+A;K7A%if4gq=hcQTWJgLG=md(4DowF
z@3+a!SDsahtTnL8v@SyV%?J6En898eOJ`MME#?!syJ`XkYUr~h_lgoNWh<8G5R~Gd
z)SepYT&=NmGGKT!p)A;0n~jf*%WQOfnt88*wC~8IPb{eKP<DV|EBTA|U{RHt`A8ar
zZ>BO<W_!xE&PCsQ+cqB_1Ghb){shQlHn68|T9>q9HkHJ(?Y=)p+4Nc;MHzmxrKfo;
zr)JDUma6uW&yx>Z4H3kQt2TAd;pi41#%x$q-LwfQJ_N8h@F+UA9I$xm+6QnT(x7hl
z^V4f})i|A4yTAPl1G@x@KmI5&t*_XK<T#6XFPKdsF&shI9%+#}VJ3lQr-nUO(=M0#
z&sX$Fq>oxxDK$?apwplCl`f~=n`!^TOK~W%S~Pdaebnl03zo#+un?S>w;G)1%h_m$
z<nEc}&p2NKVLPNnM+h^a*h-jde2w>ztDyRFWl6xA@ZXJnLkX<1?TIEv32oT$DhHMR
zIoBH9LU2iYLn^4}t7>_D_aBZ8hlikVF0Gp-T>i+o0W49J+%w(L=JJ{WUa2*dOuK#N
z_lYnS#eMdvjS^uyWjrBMFKWs$fm|hw)<r(^rx&Z`5nKvIC&_}N!@(ifHAf{POrEr_
z;XB%Q&TF9L*sbyX(2zPc8!hP@f3H{l51jV@bF_bfaQ-f=f`4WOZK`uDP0`mi0HBKL
z#d@&PNcu;QfDv)P=l)!em^>2cvC9TfYNWD>-{ZZyeWt)!EoCi89-9waE<Meq18z3Z
z@B?%{@VHZZ&!VV0qcr+E8{^rmuhc$1b0E~C)S5Xsa<+1cczQZbqoUn(<e~x_Cz+Ml
z;d8C3m7{kYQzLK;IDTc=XSpp*NN3PxD#jM?(?V|<;`J`K!YuZ3vfIk2Rgi%o?B>Lt
z?Zo-E#nCtjoAA50fv%<-gyB(ji*Ft|RV0coYHE9A8oxk$_X;+$PmKBwNmw$b_Pu_I
zb~)@Y1~YK}pw707ZXhwvB0nuG0w1Z#G9W8ldfF|0Dq@|rD#k&J7~CXPe;DvRPOB;R
zt*Lqt>=y+;45|S8;Xq!pgKK8`aOKx8IO0(r&PK`5TBUVNmBE_ujgbC4l33{*Y+<T=
zhsz)NoMBld%~H>Q#y-_8{2nZtsPl)byvY10YHNCaunwfFzlCC@H3VWyZI;@fsiHMO
zxM-xY!I@Iw>5H5y%4F#Neg@J>|7etA6~-3B)wvJuu=3AG$3Q9^BksSc+Y>>4H1eSY
z=oNJYcJxx~q5RrV{PpWy;BR!~{$-8DT$(4Y()pZLZ}RLOLM&H4+r<7jPAzW%3V<vn
zDgkE6oTUeNs=%<v*586~hoSchB=K5cQ!w_)G8chZ9F^tGO?;LioP1T+b%h>`dDG_f
z6+dv6MY+oOu+Zo}hS<__I4_UU%;=lg7w{^etgf^FgOpWGgEHUHJ)p`2k!X)qn%5$9
zAeXl?yC>6S&$hlSbJA?`-v~J1q<5`7RfILRyxz!HS;Z2pEZbKTra`>^QEUI(X5ud|
zy*%mJ5+Nml{R*9$By0B)Jb(knE#rDaOVA{@bc{w@kgv`>lb3Pxdd}h5u+VSz@usA=
zuY0<FZ4D1=(U_AXbYw}UvwWKe95tV1oTkXc=Et#O4k8?YIos6$9b2CVg-Aw@XFfR1
z#gD8>(TF;E^pz3SmdcMbXUOsz=OrcE?t~vH!Z?*u@VcQxS^kJpOZVOWN~d2@2t0oI
z``X*bcZiehnKZ3@v^OQJ1>x)rZf%cm@9=&91>v0P5~JI<2V>@H#j7ymQ%Z_<q0(pa
zM8#%vyAZ$WHp1A~G`WV-V<ONKmkrGu0AuLCZ5=NrJ%3z#_h~JsCv(Jv_5Kdlqs1pr
zwT^*?D!y*kZ$F$jHpLiNxN&9YXfwlezO`57$w3uInQ<YXmY_R`px2r?K|br19@#Q>
z%+;pwY6A9LyI_&#+1%}X>1k~JQY}JWG!rx>-1ro4GHTK$kr&4z>6K@TLQhVgRb!V^
z!yZGVfq-+*ofIjn*2Nuc@BJu#Tkwc@?-{MfBlLRv=p7P+oe}D)Af;YA1^|&H2_4gc
z^DcR%%fw!0KKk3{Axt}V*qoPe8cq>yg<O|!Ssz=FzS!2Sv;A}ZST!vBrlP!RUcfr@
z-4O}qC6~atL!Xh0!3L7EOu(Z#?3+&dc5QM*A2FzpTH;4d4R<7pr1$1H55Y1?%ez!!
zX70D@3kW)hR5RExY%+M;CGd<myPXkOqjyDEtpMpF#`^fgMY#^vuVvIJYQrabbQ$Mn
zgn3d{T;iI-yI785Chb#=iT6w&V!0r#Ek(bUzclwV6!MK->!)0?X8ZWZzDG9=AG7Sh
z`=jhT$Po#4jIHh^u=a&LBNed)+Vf2Y&<=^yF$agP#YXzIAGBmCL{A6F&6l{RD(MeR
zj{Cq|U)k+)0zYd@WO>9%FvQgoSS(Qn2$}AJ=j^UuIVGH3txJ}Hoz3-_ye(bthjl4*
z@~ds_*sH|nS%c#p#yWvH?J(Ey_~+zFzhUiX{ApK>dr4Vf`4iVcjm2Kt`4m0pqe{C}
zaehf*O`k;$w|xEa(8VQa%;A7FtAQ15zU}yUpH)g%F{JFk$IWp}@GP<9^q&49T7Vh5
zA1soVI)$R>O#{qWIq<30U|;wTGp3Kgw6dg>LnB7<E~-oiSayG@X(!i*tv^cf0O-D2
ze$78-hw7bl)7kDeBln9Rw`8eFTe(W0UIbZSkXts~3A~nN@N2227=Qy*9?}I7{ZyVj
zQ<1tzRmM<B9zYCbcQo+iLD@u3$kFDsBVUCljnm~??a~b-<3FY!9Oinv3a0BE$$pHJ
z2q$2LR;;zq4`Iz;|LpwL3i+i>bm23C;lHtL!oQZj%^OM0o-N<3W<J{$gcYZM#nzdN
zKW}43Fa-+&rdZ}1WdGO&06pmuJVkhTFY4DXF1QpPHP!`1yTtgH+kkY*1%dUB&nE;V
zX~|8a!3GZuhkSY!CMTAH49x{ex0UA7375Q@2TnO3V|*0p+fq@8y<ab`vU4!RQTQ2c
zHe-;d3O@fBPp{Rs3N=RkBelVz`H71b$Z0~v`+dgSHu;TT6BVW{7)#92-l2zTKcU0-
zeE%icArd~Z-*#RtZ)|15ER;+D)hXXS^m<fpf2NE{GZ-NdzN9G=q8?iz<<dpl7pG_k
zWURn8pEwUL2pz0WR_}KeT%%Ii4nfqK_sN7+kAPFb{Q9MuyA-8kQ^vna!<iP;Qc>E>
z8*}ZqMs??oQQS*R5r?me%=31K!{N2@Js;{6(OKlf&NssSU?Gs6WhZ1o6MFUaPx^_9
zk0Aj!+Fc5DRDf~20>4RGFWo$_n6WL>a?1GTt{>qbA)v`xxVH%DTGQVZ!Z_FP(WNli
zfuWBNAVX@=g~<?q_7&qk;4HgxSXl|3$D<;*^T7$U-6npo|K3Y~VzvKEVV0k?nTcM8
zIt95bx{;Z9t2zzmG=O{a7iQ{kvHGTjm+ya)SJdQxy`^M7^!on)h>rgo8-BC={dj<o
z$EI`Xd>N5Ks0RG8{~iqK$)9+p|AkBSxq3q>UEE)nYTZm7X6H`UJ^Gh854?Igc`Q^d
z5A*=^QTItZKw$xq;R<F@YkWr%p!36*!)0Adj@kR;4&Nj=0%!dhPVC)bLd6inbvkW*
z8od0f=IDYta5V0N9e1{5khz`x5C(Gve_7s=SySg|8U00DGJ1GCyfne1s&08=8H6b%
zo~r(QZ?u;Hf8`8eT&9i@CROoy>H-trtajaKrY31TcX%Ir#>dskD0ncLZeX&L1G^uT
zRY^-1E_Q&1#XmHVhA{6idi{PqsHx2=Pqr(x*K(agGt%I*qfOS#ibQ{w5dK>GP5Z>2
z-@Ngq!(5uZ--}n{PT)`*1$u|d(t2_U!~jjPN){U6S{T$3w)KvhTG!#Xk$D(d<`(X{
z=b?m3M})dFBb6+?hj_z!yDI(%ze1B*@9*eW0o6e97Gq!`p9zs8_XK+UB4v1M&TOR7
z1FV8dz2yUQFXlo5-d+j30kv3a5#ce4&<;q4Ka--j4(0`D2Gy_Wamrs@z^RBd;VuIK
z)L5CI25Y<d3)13sq$~W$dY=WuE!<~<`L_Uj&}A7HWOK4BL6SG)+U+!%%QS=$FrA4y
z<jvoK(1P{soxlQz>8w`GE$TK1+fs_>`eO?<WWPM%nGQY0C9p#Q0M-rSmfW@TSMrAW
zBzFhqjbkdysl>e0$W3?*j<yvJZ7fH&>hNib+0?x$L5EVos=!Bii9t~3Ua}%G!KtbZ
zmLkv-dZyEEOaI{RLfhN~H(%+CM)eMczrwzA1IcyyI(W@uHpvszKvpT$$qZ0<DJ2KK
z(=f!zaw%<DeG0V>sw7XJ(HW15T6bo2P^Cj8NVLySDr2c=;4`ne@lIQTYq8TyML7L@
zc?-tG!ajHvG6O}Km92vp%LQ4)9CBpzlS?8m7!P>UZfE@%2SaH2U~Q{HRgHyr^X4O@
z9WPQStqxQAS=+1JZYt>A@xfK^tFTOdW3E>>4EXhEo=qht30@`|)@XP}(cJ#<u8yr4
z(Q_yueW`>l7}aiw=mY|~(+1|NW!crls$fTS8j{=_CJF!o0)*L5(&}ixUvF8!IPLIp
zYor<ISijuE2NJKU=Y`d%ast`uEMsfEN93m+9Pi8`I5o_D@v)R-hrWgvb$B6xQcRQ+
zTpJG!G)6u}4sQEemgWSuA@s>(7AsbD_)Lvy;8{NS8-}yl9Q&Hh8(8(8v5{L2=7EMn
z5`!AneSVT$!dV)9AZ4|b{Ilob<(Yw#lRI*9emQS|JpN?EF|3(H_ByiDd)V?^iqnuk
zYeykXpTQ#~;jn<w)3^~1N-YZ1sZ2uEZIH$B=4@?2RY-Yz(;9P7Zeh?Ng!baM(O|52
zep<Wq^fw_t<H_9kVU;^^ydGC46p}kQNz9&Y->A_(+ADo69FkWNn(3jT>za>vLw(zT
z1tNW52D++HTNk-gs)3bJmlhG`R9O^jKiR4DW^O_17YSz1(qUuMmMPSJ=o(-;wpP_Z
zWw9#Esh&3Pdc&+#rD{uJgj29<;3d?-JVmI0jtjvz2Xq-+bE-fLPngsgW3+<P)*c<Q
zO$;J6;eQjp^y<)WclLYW5<>@AcB<MPq6IFc#&7ake~o&o9kv#Mdb%O6@r15>1Pp86
zsq<~c`gCgYldR&z9vvf|376h;G=Ho?s)?6=5zHN|KQ>agxzZGVy*r?bhXjpLYA+jl
z)FK-M9KF+;+?l$tVy2XKiXd?^cnGiMp6g0%49cpdZ351Y`+<nhb~V;0HC{ZmEb2x@
z9MJ1BGWuoBWbhZ(GQ6c~#JM5|`e8-^Yv25?>QKj2RQ^QHE?S3dHRPPE>eZc6R+g=c
zc|?zy02F9vt#YdGJ!wM(Z6{y;LPD&|gch7-XRGG47HWvmFW8tLoVl}vb9eMX@PtWq
zyxdaN&xsJc+hMlKof@<K>)Ye@5yEC4iQ4Of4X~E^v44EJKjColC4z9b+jQF0j?O#V
zr+yhfN+j<%F57U%WhQ_KR!%BvP@Zyf7=fK%>1|MPwbv~6*n=iU_E~1lHNtOy^fNIt
zM~A5!(3r91X!d}+@>#Rw>ht9w*Ff>=1~qHa?E!k}AH!QAsUvuCv(bsaZ@`zbPw67q
zoMy{ziLduWMJ7xhTklrmufUud*!Y6a1Gi<5gY4e4ZA_NyFHmCX_)V{z6(aUee^(Q^
z>Ib{;_QoC7y<UphkEr31eRdw=ot<q75EXU}z7;0P%EU4qkwe2Bc{*MZ5(DChi3N3l
zp9IgJb=Pavl+&-<K1+C|Cl2GW@2x-O4Ja3GX^b?Zm-v^e&12+IiM&DA!G|*k(zDHa
z!$<*Z47k~Y&6|G&r`WZMM*XG7uqDdI?eqx3atROWPzw<0tQ&g<my=6FFKdSo;4A@c
z6zhgxGx2$MuP;DxUOxJ&cZQ*}U*+C8uM=6c`5p}gdM<7v3fk7aJ!)uPSt(rcQ|%1j
z@VTCs<>9<7c>ii|PJoM`VN`j0`l5DiySjYgJgXcQ5`d&!;V-%5o9-Tl3++-IABOQ~
zAI`}w-i3XC8RYz$CjGD5QJuNp;`YTXk=SK3iTq+JTO*KKId&NDKX;q9>ET{wPeKuc
zbfPcBf#Ol^dC{p8$4E(FUv`*pn9GruwUU9_f^*j)&IwsIRvQWiz67Y*lrk1i`M7N5
z?Ztg!AQZ>+EHK}so2wc)m*Ey026~c<IkZs(um%6lU8vp3H^=z^;I{YOu5`y9<8wfa
zYJS+@t)qI+K7uF{t%-U!PKImFUb^Fsi6ewRbv9zN-&1FkpjgrJ7(VpDihg3daZsf6
zK$YYWdm@lGC^S(Rrqs)%wnw>3TX7RFs~-os0y-zo7KWMM9NR5vKow(rW825j`lG$0
z$qtG3r`ct0nT$iWnoWtoJ_LZlDSwGx{>DrH&BCgiPk%;zkY2tFzUZ$Ksvl8k<3!_^
z1Ny#yl~Apytgd$Myt3`TSP`aT@H23BTFy-TO25PV`}o(TaX>-JzZIv>U}{EgMyCE{
zP+3pj0chrx-HsEZ%IY>{H}w@uqo228f&f%9NHb8cm0CJXs7|zqUI<#UPv<o}Up5jr
zFD$N0gUXFZPS%=eEV}~C;%6VCU!5MtD%w>W2kBfY?fn{Ar8w%+>pyZ%JG!8`#7sq3
zwFe0)WP##7CBEmPi+SX?GFQuK9@Lz{!x&;IJ0POv?28o#qyq%VR{0gpbp_Wv=h-xu
z!Xb|!&EN>hki5E%WcX|QE9c;|aRaT|Z)yNp1)bN*kJOD+l+)!4jp!ITV&S`l6iSuU
zY_{r43!6(_K3W-;8<QP&WQ^jdaae~j@qNI4NZo(4=Q?W3E~lB$p)!2{-1{w^+mLgl
z0Q(#-WFvOG;Zu8sSbOXjmj;giVmoGr#g8<k(|I60^Py@GQX6%79b&`-mVKE{@{P&4
zX2_(;Rt*~YK#Z-ytq{Lj<1A!RGku?jKNAJT*RjT4eqJFamG5+^h1CZodCg-<yJZ10
zwge1ppG^)qLFH~x(TOQAW)yMwM~*cpIAbC6gRV{y4aXgpj-7QzDt6;)ojgljqX_eF
z+z8L8CR-KBlU@lFaSjd|u*Ml+esoN~<n3****16dMHR%!yELtv?EryK;LX;xTTt4!
zQEYFCt9V%rD6jc)pktqAKg8+unq|(uS^0&#!tUxtsW9Q86K>@7ybx-Rq)wRq@q-DV
zFzhQ57YwSMgM$Wj8L$fI{YYPD*SN?TV2^+La!gxyOFjo`(edFNcDQ_WXYCE+CLPk|
z&>cVEFlorT3o6#848#+?h9Ey%lMR@&X*Ap$#1p)fRnNN+Z;3uzJ_;OB{Gi1!xu)->
zqz?I`#p)tj*zmAMnGU3aY$ia=g@D?C&uW2^XUp^?q?9?v6I5bVn<Hbi4?jHkiiNrO
zvABx%)Fp-uAaD`L)f(SFTAsXht03k+<AlAmVf%tvw552|o7M^^tA#U6Tn_8~^gGb0
z3OJw@${%}j=K`aD{90yuPzg1c%B&FcuWT~S(d$1)G-{2_*DW=31v?u<veAoLz;w>v
zVa3MW7HQLsdvm2VHSn=%L0L4+eCUUb(*6d@qa871b8Xi-LA$_VM0_nUv)GnjaHx96
zRo?pEw)~(=`E5IVp@tFM0@>G#?CKn7QGY!kYwieVP*m`em<_}yXNTw#Faqrd_c^?a
zum9;{osnDfLjgtD3pssQ+PL+H^_xgH46dl=RUur9i+X>vzoJ_ER?jowQe9_&97#qv
z&uQ36SoH!3V5r-V%l_)D{1p{ju(bYA&90t9)z<giEF#SYl+%ROZb#BCfomt<=TAmd
zLWiNUGea%q#I@NG_i+=pi5^^A5=Hae+<1D6Oh;M6qpX=a0}NHOzOEBFA;sbPbN7X;
zR*fkrrOPl28xJppa|VQ~FLo;*$wptjL6B?}75F|OvfDTR0w}_UvfkwaQlBICN>1Rp
zxYURH^~p}1I%S`38-)u@qzwCc-qdEom$vTen&o{-V@)%o8=z?^MI+4TFC67An_rc*
z0p&rGW^r($H3ZY1-M)E^^ajLd;Pk|O8whcRFR>G;LF}0Lwpi1q?ig}Oj#6gdMC9^M
zWO7otKEsiuroLBUZA4~Bpk{~GVonIfekuv~$VPm!g9_=3w!F{t?&bD^^kx=Xx*edm
zws}p_bOzMkd}R)Wo$?8|8Kz!I)xPEX1&Y{%5i~`z_}{%?DrYR_;URweq#wO3JA5$V
z+-UQ67w^H*y!+h_?Q4wwIrKT%(i>w!c3Np!c>gfDKq98V;;UR?{gyiVgIwJ4N5{yo
zMv&*Ooe>>Mj+0RH^{;j!Z`A6Yap22`CX^Ko%`9wzWXie5V8)gVzWH_axp+5?&TlL@
z%n6C5-97=FnWzmV$G#ri<kz(N`Eyw}ckmnFIChLG0zSidR<*dcyLMu6={1taEp|nr
z&ckRM`G)f&wI9%2^m7FV<%G>Ot(K#(=dvdFLtjJ>v2F2>$O<iqWz4so0lY&w-svOa
zH7n3M5M?MN(sCG^0j;>nroCnZxGSlX@k63@P1ag{>UNgy{WX&A$z6u?N39p=inprj
zb&#XFons7rYL$i34X1ikJNAk!yS~qsaqe=QTP)__S=xw;w<v%Y{zjwH-0xj1FjL*v
zX9V{mxx+5eQ}3Q`xmvWtYMfMZdAaQryXdK850YNDN~KS<A1X)MeMOR1F*Ox8B+`y4
zsAA3Fpp_O5LoP)i^calyzHd-cRT2<#Jo?C~4IuPVMus}8FLUvEx%8{raK|rjgfd1c
zjq{Fks^Qy`$eErgXS&svae@uI8a*LTMVNkgsOjUYS$UJ-^_xuwxu*XLVU-bZ9zVSO
zt--2K4i)Iap!yEwy%j>U0T5SWBq2r3mDD5n?LH1zUr)%jYtQZ{`}Rvj&xW;}2^boI
z4LKFK&<h4<HT}q-IwK5um=Jl0zgXtiEw7@ElOR@|`;3?m=sM-wcX5Vr&n0GsBg9#L
zVc66Ro!~7f=X`evN#1Eyxmfg+sq}^^c|7RW%?tGe_18J9u9C-bG0zf#!yL1(CZw{q
zl7ZxPhwSXawcvx4YL<6e;FkwuM;5pmV}?6>Mw|?vd|a<0lUR1@(sC-$4v2DJPkH<7
zt*00nrt_3FC`GXJY4*AKjZ1)(t<tec%2;DXqD}BpuD*Y&090n^%(?ne^WWn>?^c_#
zJA1q)nKG9DMs!xrZH*F0Uz~HcGCuX#FbXsl25av4t9q=(e6b{IO%*-i!{vWdgL~bm
zlh3)*6#0GG=+8LX@5wlSBl`D0r^K`6Er4Vm<Z^i%@vW2jzi90LfGKxgkrp@Uxc(o=
z@Tt8?xYt0M(7$F8{p}4u-v2d_AQJG{i|oiv%Uf#xeW3nVVs<zjP>S(b5x43TvnlVJ
zGtMmhN$J-lHAyzT_a{&k;JMOotjOu7dDZW^fd$T1wNn8=77@hM%JE>{!X`n+nJtuI
zen2g4u@T53lBpHXqZ(p>tfYga(Sgx`x~v3$!^F8D^#P~S{L|CB>Wp>ErTwk2{Eo?#
zC!dD78(alj_%-cezkXX`wr*Aca6nDp{JzVXo|cTZk(YTtN$85=II=Q)BGf1(xd{D<
zzeww=L|^ssrK@1@Et}7;tf%u|++|}9hL-gJT=}SIw1X%bEN#n5cB;Mi&K>pA?OV8>
zEhE{zF{J(|eTkLHK#kOJ*|Rlr&wsovR2_eul3H*i#NOYRu7T10i3dWTylXuwWi{=R
zjyfMvSq2cu%xH-wDJDSFpiCj|dPL4J+h3lU4lPOPm{V!8Zlf@osWAneOP;l5@b<_d
zW!*1lfvp?h%#zFApf_Y8OR-rQiP1A_DBoQs2%z>mBeymR_0HQ)cwiGP(hj}((RBYo
z+cfkFt+fwB1kVj+(uF*F=qz6v4)?1mLhnN3i;v>i8SM_45-(JK=gI6Yf(QLv>H5Lv
zociVl?kP(H(N=)IbGzx*p3x7aqX73ir2XPcKD=2kcUL`q@f9Qj;faF{lNgXWE@39w
zrO;4!gSjG=*UlJhi~50wcx>{kY`pzs)25Wd*QWGrE4TRm2eM7wU96ppu#0;8lsX-Q
zzKl?QrxpRggM+^EI_~sr*puc@%xzg+QdUs_VLx;U9G_LOPdR~wdE8!|DZvj~*<rW6
zwQoK@=;HmsCiP$@!d-KRdUcqFiscxUweRImVw3<o#U`&S996<F%g07^LhAZ|@BcB_
z-K<5jd|N(q=5gwKxBZ;Q2L-;L`b59|lXY8}t8v}3Yty$lsUild&qS>u#<{3Ed@#3H
zcLSQ68qj$E$vpZ<2>ZA-s-DO&Nei-vHaZ^H@HONw#nu_<suEmOwQq#MH2M-0sm3ap
zkbiQnZ%?Xq6%aRBJW5&h!oSDGyT#|ZFCO&}7^+YtD1UfrW;Ee+c!F%mlq_}u5mRuI
zo*@?&&3qxUdzb5!a*Im$BaaBR1ALcrA`wWx69;~bU83BZqxT=V#Ax8Dre`#p<b0pB
zvuquVD?{WPDJ5m<r%`52-OsVD@lOVV<kFDVV?)mh<7KJFhyJBqH!I@KHg!L2@BkK$
zk~<#t5xrd<{AOG)H$ZTy_lveqVmp9RS=r^*Y3nw7Q=kVl2Ds^!*mH3#Omh40@k{y^
zKUhpbB{reh9-JCies|Df&uq$%3Fue}(W_5qrJd1y22iRpMXfLH@QG;X0ogJ*@c2K%
zhLvS9CS^rRF)S`mP}^yR^6Nj{?5DC7$ZdRW4wA%3r_B&ThYDz*_5dVQWYPSKIr8#}
z(IsC$T3jaiP>q_6gR3w7_@Db^-(=k>npd>>+9?R{00c9{F8=(gu49U6R^Xl}PknSZ
z*$mX@z30<5V0N#_D6a4`vZSjZ>PMiFei|`GLu$NzjlTlMqqoGL=JAfu;kJc&=(}Y=
zpuV!n#HBV_A65{naK`;cXw(|HG1y$03IBFKy2N6S@X^UE(ryk^@8B+djW}7%4l62u
zo+cyj>MtENj0+*vxHf{q$$h2hZ@T~3b~eToTorO(=6m1v#ZDVKLpEQ`x8nI&vWINI
zM#$S5aHE!ro|WO<iOZ7eso&ep9~i#?w~#RcMYQ;9(X-^f$`}ArF^{2Vm%$CbD+=rk
zEk|0zu0Q)jK!(UTkk|KDQpxm-e3KOQ^Iu(V1|a5^`!{C`ld|$X{U;y(-<{re{L@b-
zrooJiqxSrj9{iEpb{*Izq!e&wfB4;h=C&R7+Nievzc?=srSLNjC|zzNPP5-d{BO~-
z{kp&;+x&wf6jBtn5Sjb{g7XBt8}PbOl#0VDTmQruPReb2+y4N2XeHa1Pk?vDV%+KX
zq^vOS{_2TU?<6w3Ww`+lTI>?t<v?X<{ftsZl}uW-9}(uU?6`u{Jn`;t9em}@UlES&
zh@&psTnbGs+iZLdgZlVl0>v8q#L*1j4916nloIe8Eygo>D+!Yd6eSM;(=n}|SRA8y
zDwIY~)W8XI0l5hMoPnv+GlomP>2CzOc}|u7i9L?l%WFSDQ`ay=L-Kor30XtOkf8$%
z!(VOzBB^Z)wy_1HOr7O6>xxNT`kh_)_?Uu>lEp7hKo(pfihEe&sxrLz207A?_}0#7
zef&=-M_9;AhYgn^azM_`%x#P&)~%)QT+91+-CytibO>fmO>>FCt^f>bchbLk-yhYX
zPCskvj`W!tPP#>Xh5poEC>dNDI#k+*3sE;;v?IGSp1N(bgTVc<G&;LV2T2XD@#-bJ
zcEf>{#yUG(6*^uQ0%yD;JmdzH7E_pfysCdv<EW0G2i2$>OBlSPdPplJu^nE?S@QZl
zJHvT>F*@8BK%8EiD;YYsbB|64A%_{k8A-$wJFJRtqyybS(E<h@v69cm-zpse<U0zk
z(d^ahLc;x&)`!Nm8nq(QVbQPNw9K&q$l2;A1E$>}2?PWZ4;zYfDqpci5!?fQ9QntX
zH5YkW%9V@#%oL9&UF`?sx0b-P7P#he3q<zt$cWaF=Q`}vno`mW7wT42T#M$e$gyf2
z{=yY!i#-jRrC9HIYKd<VDD-${wXNBXAL=sNh~M=9f(b&`m6yDZMxPlZXy&5R&`M^#
zA)d$sKh<ru+d{oHdx||w-H&{yDJR~;ob^^&W>^`VubbF>k@Q^m!kf0%+f>_Kx6PO`
z0EN?fbfe1i7Sc;S9bo5c7<LF{hY|5fS!XT~OWrz>S;5=F$oK9-@Ml4iiF!SuFhAl`
z=+w6Ox7X%|>x^{uLu#DcvY`ktaIAC<sxS`VM9&$@2;qBwcSZnXk>jFzP~Ew%XY8@z
z8imc`q}_o9F)!!XnWhWm@O^9O74X{0Jm^ury6;bLN9cbjFKM2DGV{Brh>ySj=-gMC
z!32Nd{q+QcGXQnHpH!4TXEZV2zF-g=54g6UswDMH<0{N{QoUbLcJnxWUPyS76|I5a
zSjOe&IQwx%w-I6^_M7`XQxBduYRha7BE`?N^R9qr(g6nov~a=2d%+u0u~{Rk;U+cs
zRKHSARg-l$sGjD9l|9DUXH;;l<QP_xqFt{+gt~sUYstEINU!dhJ7FaWZ@O7jW~*ck
zp4kFuvK;Fz)r)(OMaf7<*CmA;>1E%rf_U1ipHR<k1Lq`%HOzsG_f|IP_K^4?Ll=x5
zzSQ?0ni?Q;GwZvqf)Vm;UjGBpi7xO?Gt}HdN9EG<44F%7-6+tQ)TZWcmqNQxsrk{j
z&_k2$ZkLw>JPO@9W=i5dmXHKD9{~0sA#A~A$vg&!Y1dghe38o_daSXPMG|E+lc&`P
zPlmQm?Beq2S?)$^r-uXWz=Ji2>TB7g!Mh<m%qIbC&8eR>_T;{xmfI=tR`kg|9ra0E
zSd(>mt57G`P!=CQD(p92->s6_pmfDCFtTgDYo0l9K`>vk;<f>5P{Ayrr*Fp2SqPAc
zU7DWHW<o9^XgN#)Y#6<CrJJK!Kjl<*a9zmatFU0j%QwFZgL@WkiKac%i+Sm=h<}<w
zEl(M(H;!=z+ce%V;+>7$CWDWEFVNzh-PT_e7MtKd;0(-rK+rk@TmEJiPW&x$&_HlS
zE8(r0Hn3p#9xTjuJl{;!WMsT~e6=c}71xH2ek0)2GD!ajiGgg;;N3pG{+9r;?5_(Q
zE~?>&roy&;^nL#pDqm)d$%5#2WT3qpNM3+(4?*jbY8Rio#DoAuI3P|dhgfLcXh}Vs
zdcuDYx%?c!Uo?lkU~LBz3Bps}^R%yC4TRD`47=9u^;y3YyMJ(@z{pB&Sxq7p8>jk<
z<MpE6J!}a6Al5TUHwNhau@4-0KoruTFS!>x-JhE@=k1I%*r`DOba#78TCl;R19_oQ
z=Cg$H>ic!|X8~oY<SqbQ+Mbf0Zd>epz9)^Bfhd&=K|uWeX+7X`(h{|c_9H|=xATVN
z;MK-(!4>tq<V|O^BpL}#*46DyR3l7jXR^`Qhht-BP*3Oy`-%ASFKa9EfGNaAU!3OP
zNQ|#_^f<ZG(&b6*&qmm8XK2sTCW>I$dKjtZt~;6`ttNu2X0w4`^@8vIi%}6TF`L$M
zb9j1N+3?r1$}%$PtZHGcAN_FLmR7b5m*Se~qN-rX<fbp0gLJi2gn+&<kgB9~;;`|C
zCyLo3ytqOY=KhA>(dh++0@t3>h@~oMb`TmFsiwta4*YJ5l36{LU{FK)JK_uS06zWJ
zEZ@1QOuZvcY3f2ecaZQzN-@Y{4wP@Rd?eDqBg}}AbKg3&F0B}Z%p+$c<=1lN#Tg&}
z(c~-ZX-ZGaUvkfHOv<93;LG1TYMWX`&q2+U)1Ji?T)0Jr^y(c3i*ItYNr!-KlJ3B}
zcRL<G7vHj&r&3qkh-)*bdj{a8gP>B$H2Ly($*7kv>WuS_r0HBoTxcHOjN<^(H^0CQ
zk25?Y%{P`^AKY{EL&n#DwuJF0#(nD9Pj6XixSt1Jkl&sZheQ!2@X9h0CgJuRUDyNR
z*e^eJMn13Zjsh=G&oOscRZ?xTJl_er0Ta?BlDoob>n2K;p-K|;yzX%4lT0PdM7;!c
z$lSPYN0T+60^kDysu3~aSTqx19KQPa=%vzp_sP$;c?*wdzto>7aGv56bw=s<AIxjc
zG=9%H1y?V{>>M{5pRtHJ_%lua!$8B}-fQe!$=N?;plw5+hgEvdIUOAPH)acLB7Ez}
z`FPvJ4)^@uo?0O9mAdTkZ+H?Ik1jM04X;s@x}5&!|NNgrJbyp$_<wwQ;eVDF`F~WM
zi~p_0<m&;+6hM9=9{|L#mxR2PFjn$bdi?}YmZu0R4AFCv?4TR>eGRQr$~Ozo5IMCI
z7l3L$CoajzkKZ)vy?^$oW@2QQoA741FB_toH?jJ)?TK6|pw7_|Ga308>Nx}J`J(`j
zDLGiLq!>~xWo1yfp<pcGYc6Of_ttu62{AG74si+XvuB;oXVy(#s=Hfn16IH?JbnuT
zi3r4PtHi#%zu&NG&@_B0?K(HJh?C1hUxV{To~AD-mF5$Nz57>Bg(7Mw{P4*oR{wl}
z(cn}IP3aQYm|=r#E{X^#NgC>xb$$J@uy=UI(+#f6&Ro}qiQKmps0@^F#I9HB5X{(L
z>s!T@pzGV;H~>D6>A1tkTyx=@y{}QRBRDr?&YN9bl;VqRKeI~gox#0^m$=;zH)LHO
zG+EnKeG#ZSz!=d22^-xqfm%IDhKpB_T}~CW#z9<eNCc4akhRVgdmO-Dd$)<lrfFoM
ztuF2h34gHhrO4?SH>Ti~5PizaOEbtJk>Nz>XKS9Sz!V9mmnIewnqd*eGfqwNK+UAu
zBX)0c(BABpr&Heh?^`m|JM>p_$Vx_=y3Zxu4Ll!=1Qz4|>#sdcHGAGDEPRVuNzU)+
zbeWhLCS`JK;e6sSTXEf5vD<o$*DhIomtw0oHSe^#)$tO&3;F?<L;v)BN(vZ)o#L?r
zCfq!F$4w9Ky{#U37l^M0RZbJvh>3-66(@IMH)4*jSF(*8*?{?a_toJ!L=F1Fl>TV_
zE+?Rzli0?44JTRIi0w$?frB5)S!plxkhZyoeB)y*Z-4c?4cLQ4y{;%Fmcpe2@p?Of
znPVX%pFEUVNJCT<*e=K`LP&#{SNQvOTs)AosnA7FjoFZwhc2M=?4^2GSu^=ODXUCY
zT}CPKQKBA>Y5ZE<=X?x#PPfuKlistqQ0*%wHvOdW%W<3Q9f-Y-LrkZ{x%3eiAqkle
zonEXabGrd;q-vh(xD!Eo&3#f{lXY%=jWpNad>sp@SatIH&Aabe>O_7g<_*Lzv8)F|
z%JdJ}f;_K7ANx9AIW#^yal<GE`?I*s)i+bRQ6XUi#3fwSUMU3(12N>n2fm*B_g06w
zQ<lJ30nk?iQiE_KpqkE}O|v>c$(BfhB~C+`3i(k&tElk{<CBGBwR-+(8lvDmAYzSu
z@uPqiC<iafrohGn`Ym8_DpFP}rGg+*;>MJ=rtYtEHATllIt_q|5woM^sj^K^S4uOf
z+;3>tW)v>R%%{zz`7Zw!b5QIyWUIJwipwWALDxZ`h!*0k%#tGzFRY-}Q~-T`FRV`l
z^t1sH*lIIj-yqBX(fa~U+Q|;LxAu&>Mx8@fr(h0(8CxRfEyaA6HZ*1w1&1rz4Sfxb
zkT0*0R8L{@LuH)X!PM|n4I(dZ5YkUPoj)>&rj$OhwM5~i#cn^l*U(=17QGl^gh`;l
z^k*hRcRENS%h83$$dgb@lWL_l;&--|QHm4R%FE#By2cmY5ZZjBM{Amaq2AY{KtU~S
z|A<v$c4v>Tis(pxtcz;Wyny1%=7GHplB%^OQ#fU*o{9;i5nN0?4q4+DPb_oyeLpvf
zyMMX#gr~|kE8ghs-Nx!RtEf}VdA7z`$IEUx&@5IA@zv+~`uAVM4Q*KO4O~`=YgNp8
z&(Ev>Do?GE+@&ifHuPK8$2qu*rp<hKelAc>1U06)eW2b7yxFKx_vLML5iIWIFh)Ms
z;?gmf%(T^eLg}&s?i5h*ssbN!r>#t3_T;d**t-az@7e*U0J1t6j9$L1X>-Uht-kj9
zo*`cd;uK9P+)Gp2J99&#a`IHSln^MdH9x;v4B$)_2<~8OHsUq<0*<h3=g2|aH`(Qp
z9ub!oJbZ>&cCa%GtA{DgcO47W6?tmfq~dGE#ESQALX;A=b1{eVRKJ#I3BH*pUT!ry
zXI_0PfT+n^x0COW7gRS{Ll;-R5F=W9X#OMyj@G<XEBG0;^F#ZRpa)VT;@p*}ag3BB
zh6w_vr-?|QIu1~DYem~K9M!y8>~_dQm7{Q_V+Ye#p6qc^^%8{Uu=$>SOva}xM|m>Z
zm{sBL&*<|bJDAz=?+G1~?=`7!W@a|;0F2^JZN3OzB?i&>+0e?^0utHsP^xdcy<h(M
zu%UvhuSKWbm9pp62Mj>u9f<KJ^K`k6fG%u+Ql+_r5<PC|58GIF+lD_?tNn&d<)=Q?
zqrUbekZ_PG)chpK?){|e)15oG!B<$4gIvR7^%yBBqeQxmzM^DpwE<8CgE>b-cHr>j
zTF3TQ>Eqqx3%v~Y4?57m$P`}<F|ndwqA*3NuaavmcH08mjL`ogGc*jWMqp~Xec6ij
zBD2OP@w&#RwDZyD<=cg;m4Dr7wRhkK52(X6dA71XO(|%~zY9SO>0fE-U0@VJKma;e
zx@3a-o<d@VqgVqAqx1Wl8mm?MHpUhVe7#wdRrs=y;J~>_$iU`L`m6SYxUE#@qL(hR
zs|G*YKA<F?08P=fS%oiIe-!0<GhBUhWj&~FnhhFonSeqJfx=vXR&@48n#so@xGhki
z5Md%S;?SK))=rG)S9ytv*<F$u(J-r*Eq(ELHLxD<pA?l_XmS^}t-BAni{iuo-Sxd1
zE^WytbLsU+dxk5mH1FSPbgc-?7AtWz=0Fv%)geDhdi@*hc>x-h{@+V2zE*bK@<*Kr
zrM;Uh#%`~6I6C%6Rf!o*00r4{;Pw9h>^4rid^=CL_$|`t9M<x%SM{EiN+Js~0JJR;
zSPbmOh{+-{Zq3oi?XGP*=XYTd{uxi({Q>qTmK2}3qL`Yz?Jj{YC6E7#hpdEiOw`h7
zpwStmQm4am2a^y;n7)CNg%z3#b6HrY6lQMPcT7Swn}vcAi*o0$P9@;<qZQj(2v!w?
zGc!FZs3MifuIos`ZCtA5_q_O2p^8z&8Y^W}UW+$%6M^~$!V8)`zKqG2?n65$qvZSS
z@>aB!YTh=ysb{wc3-j<+bI3f+0n#95TeoX9Sr2{V+x)4fSJZC+0>Uy7fK>Y24RqDL
z=dY?q7DAgd{;2B39)40sItsib-KDTT14tgWP?wKnt+ms9M)9SOhNp$z`hKyp){w)v
zOfNGgXjR^64fy??)h-(l{W`7|FVNxtqrEc?OET>neltzks?*qFW9m$cv}(BK-l~~f
zW~JsriD@oii94c|IZftLn)^bfxhI(kB7&oh5+RxkqJWaApdykYAmZ{~HqAV9-_QL#
zAMW>fp5y)Se)Kpluj}MI|L1xB*Wdp}Z=qM$Z|=ZbYFc{iE}b_EmIZ)(0mN8;%f(zn
zu?LN{YxHRrv)KpV^a#kZkmRE#u1fN@nkhV((k0f)MW2Xal-ozoU7Xb8vz_~XIYWhk
zZl7v#MtEH+nkEP8ayRX8%71WrS-*qCnZ;aX8!62a-%d;3x%rybxtFhhU)SNU1`pqL
zS5iVQDhX<jpl7=BqTFgyPF1~P^RQveOl;^><j`jG!mJ-7r}~UR0b(7#)jLf!&To@O
zr>|SH)|fYDl{2QJO_uCz!L{qZPJF&s`9vT6H<mjJ*{OPZ$s2rWr~ZDdyy+?#B6`r$
zL#;QJUmLKnK^g1bBAOX16PL>7BA*XJf4%dH)3Fot1ML=tV%N3fNO*+XFiKz4@OdpT
zJytuMvc-M-K;%3nL?oRt(q9)n<pPki1U%<MBLpvhWuQ*@6d=E>a7kI5YjA4B4Y<_`
z;EVVI1_VVXe^Ue8<=vSq`%S-SI4~u@uL|h%Y0JKBdtGR|Pm6MCHp**0vC;NeD&;Jb
z+Yi6d;IB#hJLt=)wW8UWli|v7RjF5&>RrPNTzVl>&*t4;V?0|sLfbb6^l74jq@DKZ
zjWZ}rUS+1+HouanY@&5cJv&;%&X&R>0VK4(zj2+uj`cO8on_`{83(V~9DnzETfkbH
z$|K%koYAd#SgY+1LkJkKgwix0v*+v2`ZgqJ?x)oC^e1mwtjVtYPH3Wcpup>Wgg!Ko
z`lH+viq$t#96-D^=Eds=rgg}?%AJn=B43CLe(l=@zLHuPz?FXIHfqUqn6cb`g)$Am
zusXg9m2apd7_1-NOjCN2wpNh0bM;K+XtHrN40+*pEg(W$TayNHviRY1-3*cPqnp7d
z<*{6i?jw-1**D%t(6f(9r)=%?gYkCiTOpZS?ee2tc@3`saMrW&foJEZbiL3|(#+D@
zQ*4rXmiP}ifRn^>ex5bI_wUQque9B%fBikhcO0CZA_!G_Rs3MC)Y`6sun^sdzW&Kz
z7Q`11yNi`ntuRM^#Is+iZ2leNWc#AaY8Pa(AT5P_S`In>7^uYU4&=PMTXq@{L<IiJ
zfkUcsP;2Wo@-tw`K!6jcu98P`x%S(=eBf8@P5X^B&w6)`WAB`6;;AoarP#UGKnJ!W
zS}V$Fh~4QwvLQf9keZuq3v}$ul!huMNt}5RqwQc?jEXNk&sLLr<_l-+E2!%);p;aB
z(dO=%b|%8I8rdJKAM$>}d8=OaWhEMI1l<A7vi@Or*s@po^g`DI@;o~!uQbWM%EYOA
zMB-etz$Wz_0kMEh7eb1WU+a?HqJ#xL*@yzxjetrhYxJsNUp))N8RYHep-0_#MMi!f
z+5x$?$zWL+bflGiliV1X3--NsO(z@>;N2okecHXqq>Y+2WOl*qRC5;8AQ(T~+RDD=
zzOqEH8T`fQ;xjg{ULr@mQw{M@z43**kqG#I`8|Br6`id1l|x35_Z>RCe0!xu^ljio
z%ugsANuJ3Q1dtzY)6}WK4PA=at)W6-gVV5F;(XfamWx(3FFTj|4uI|_3k3?dHTW;G
zu4u9xe9qg#{_z)$7iDkyUi_QcyllBA59acQ?G3mD2lMs8_mqIsyT%zz&yOB#y!Mi?
za7sk5>y&_$IsRT3mw$r!rpj&*BtqMPnC`MA_nzqwxjmPcJ%}BMO{%rppu@NW-+N-k
z)hX&!6b*2RITI|N3kE=-w6J^xMA&D(RF!q*>7PiqqW}mJi}KY*nt`jry8<29cQTty
z&5Smy9NS{+26ymne0lVIb31TR0cEWI<IWi0QLF{g3q5U2aW(J^)npkBxzvW_?HcTz
z%bn;o(LP*DH+tg^cnlF|^wtlWgYM6iA8ae&&!(!{%=r7+SNY0i^jV{mw_>`kEmar)
zbwwciO#uB$a4<mc11a*jdLx@OMJI+UF1!NHg}g1pmXv$oeZ2E_#~tY2)rd7DPc&5)
z_2a^k60+K}L54L0C~bYTTr23BF83TP%Q-4#;O2!<s2a#)QtL6<zhs9k!gA=ZCmzT|
zD#S`*YQvR1AjY$--Bw`n7o}jVB;erz99y=wb=O}ZPIF6@ytZ!^18J8E4IwAuUaBDe
z$~pQ&Czw4&Pl4*C9IPX@OOO7unlYwe-6+F6d-RF$pZ_l|1dymx^_>AjDgUUR{BOIx
zc8%VwcVAwVXNH=$3jDvqS^rU!dkG1kvhFoG?!Ec1stmfY6+Ica|J~U3s(^o-wEteG
z`vWNUMfD@7eHfNu*&^_q>gU?OWst3`=4;f$GO9Ra-O}3VzlXV3WFW9+breuo!0Q&z
z9WIy$%=!p32mnxFuAMW^MV~IO9t{Bi*J2$FExHTBSp5@2TU^KYyj=e%q)xqDTIXJm
zX}l=sAN};Hd&niJHj``)XKF%)F8P9+Lx*^Lk<5pjmxm{ITrU<Z(U;9iwA*Gk;=#Ta
zhVtu0nV_ugDb;e8so^`I4i^@BoYjsfGq$8xSA3e7!DtmIGN4`6@Ys#a)tqZ91Tt$v
z!6r+=cGuwNjUE>hJeMzOHQTpfozyF;dJDTRgVRM~Sp(F*eQ>bJ+@<5x-LONty)E*{
z8y<2Xq~!?w(C*dqb^gI_gTn{uqH~N4=ZCi!0}Z%)LF}k#Kg?5&Dl+fb#bv>pnKgO5
zU~RfiVWTgd9||dn6?@Ekn>Gw=i>M5mLgiv{>uq1#iR)OlH^{a}Q`g@_w73G_9$5Ld
zR{vns0Fbpa#hD=N7pOgvTC5jsgbuJIh#I>Savn!Br5X*?+^y_M!mbbVj~m@UWhSS>
zZ?7e>0Dbbp_dpG+{p{s}i${z)IqLV#e-Y#?ZnrZJ3ZECUlp(h?1~nW6!4uUo83~Br
zV>);q!81uLmTeaY-3>vj?gJp%1UtN!Ofr~oAY%nk5_MdOaw)YvA6WY61ciW)gg_l`
z;(>306n$MH|1`vf<AdIw(ce(4_iREN+7`2L3&`)@Cwxx{hYPc$o)6ercy#W`!@8Fb
z6uUS8^H;H&shATw6cQ2groONU_Y9fx?J9`xLvV?!rDTO+W>i%<KSxz;xri9WziDft
z*xigrQtpjeMWf6!8xDXtcp?oMbTgBiiGVJ}z{$@h>kt0%;+oZ0lyY#s0UFCF{wB(*
zD~kg-S(B9I)#j%e+@w?AA>u&{Sp@sp4(a9%xqixwxrvZt2dK7TeyFc+t|8v%PtPvo
zIirRQqG<Iox`x}Qc$y^}L`~sI)vC<EDcXwm4Of`$-I@1vh|v(+M)ByMM5z$2#E9&k
z2NXQXr5tw<g*G9oHsM%XUk21TJcRnoH>0$b^V~a&vVNQu%ksokdG0~T+>?7Q5yL7v
z{FcwbD{<(wyn})oTw*mVDi(h90x(Co)$?PU+}zc^H7BbpCr)i+`UguR^6FeO^+76l
zWt9J?nH+KD?@*taOI)$=eTb&AIOsci_>x9!Y-9@iUM1j2UHR5DY2{mQt(Ry98VBam
z*)A{(Ln!aJ2+=R#umB*d-x5d*H+uS8#-IE%P+z$H(fV@Aus|Bc4Bbgs6g!_c<a4yc
zE<iO^3<n;ZyQQ(ec<rK>d7~kSgCk1K=iIq+ZSIm6ya!-<hq-YD7#Va@eGsjhxOFWE
zas18^7?|7Oc>sp_@C$}@b0L5Mo6~(hUKHA63|!i^yh3f-HoiChR7<Lt{|C&dh=q7P
zGbMmAiG(q|-hoUb9}C0C+UQC@DTNoK$YIN&Ku*@K^H*)$5{+m&2gOB4%paFHhi-Ic
zIQLe$MRt8aztJj~HREV!_Z7+g8b;IdgJx`G4QK3o2T4R)SVQ>hMH~E<ABpJ9!E^u@
zO5r760@4x+zrHjVc&T;6O&!=&eZ%rC^hM<J4u_+c(QhsnI0tl7E)uky_7`7hOzm(R
z9ib6$xX9&3h%IWy#W2burT>@pt%=o0)?P!Eul6+4r_Bhf5z}$lcXrQyifCDOs1dpu
z!x$l#T4tHyKWA~CLf-KNh~O3hs(Q-fQ7T8B-#W#sO(B0?n){A5vUbm(zck56;Qg{d
zrO3I908up!d4bLT)sz#^3)K<hGR<O<FShT{A1|u)<SlVZj_oQJRK^NNjU2O`zCTdB
z7KJnyrl$qKf-06b-xYR@bJX~nlRVAF4E2;-(8&_O3{m-Wm5jJW$`9UUQJQ<!TP};W
zCp17xpv{zR^eywsg`a|JsJos50tFmRLH6ye!4Z3_c0mPGF6fmG;Hz6>3c{WJmsI3N
zZA6PNy1*Lj8*vt*@f<V%hI?^dAjo|Fhk`LQ$Mf<6m=X%A%Jk3XHC<OcZ@YPZDtgcY
zR1y>fPlbq^qJk|(!&9~cxDS_W799<T^kWkjcX0>*9S_CdZnOW+k@44K|6if_kNy~c
z%WVJ0WVwG^p-|i(%bbqx;OzJkIw0u&vkthwdc|op(mEQqQMsiD4Q(6xleaefWy60E
zh8)`!)binlIW}pcu>;5tzPXMSGMn&(>UTQV;X@_PYr+X;&ixKG6l?@>Aj5*^Z<-Ed
z#`<MKOf-fWCD<FRT`7-S56d$bE}H|lvv>b+@X-0uPxvcj&w@+Ks4_|a(D$|DtN=W9
z!4hsNMG)HL?$9>#g2==iQ8Z;VYNB?zs2VO}{{&=Yb8pAwz7_Z6ni8Ybyge5mn>e9p
zKUHMSUvx`0ItA*Ozo%B~)-n`gwqxFF`ZVzx%OU6L!kNa$9hbm}a!n#}bThBHI_qJ0
zX0v><mK_Qt16*uzi?qn({<tyuu~&k4W!HZY2Af(a-1Yz7Sd5wMalSM0pf~a~SZuX|
zIq-qn2RFYIzQdx1V$umRmdRB8<u!J?-!$|n-z^s=2@uHIBb}JnBr~*0IN;GSV(wDS
z2@GquA`<q9@uf7nKQgs(V42hr%$Oa(J2pxY7MF)l>a??y2cX$_RKu8x8BVH3#`I?j
z)%bsI=f&NW(-R%088~dx+LK(CPn-4B2cS%-ppDU<J}iz#a^$Z|@+YPKZe3GUw3_pV
zmHM*0db6-g<W!L@zSNK917RdG776PBtiK1uqW2PcWWm(jjE220j~D_+Ey2WI{)FoG
zKTdb7V6wx0x-N8i*Q$x%1s?tD&Y)LEd7_d$A;|F%=^_1bB)e*4mi0o1N{n6FAAx$l
zG}(oZf^|=IO2Qbx-*uRIelE+I_E9(FHQ>YvPhb0yH$MnLoK>gXSX(%O%I%|)JP#Na
z<iPd=K&!k4FRWc_p3kRk`R_C=9omciutBX4Vldb<i1{8R*;2GQr?pwrAj-%|vkil<
zWdPGAz&l=a=QcxU7g!qVe4skAb%mjdIL`8K)Y?7MLE$RIwZn$bnhhy3LybMU@DOj~
z>ou{#uk&S|-^IfUnnR}q6*0M%uP|MLT!d-vog$3htGz8fHgbQpZkAYS;E*{LQ3(BX
z^_U4J3-(1PdKPPdUSA1dU3DltN?Hbynej%*uevx%5E&i4<~4}lGourO!|VHV1hwon
zF<OQfR+9rd!RL6Pi;c8iEyP3An=|mE=K!4|cXduk;(gXjxvxUGaL{ylduVczlV?<T
zh;iD`g%X=0p(8G~6=EEH4Zp0W9)4aYyf$ZXzDsg`@rWxgQ$U-nAahJQeo$bs6wjw0
z2oKp!BOTbbl_*A4@=<15DJHg3@vjKjHTZ1O72f@Qku8D1Ep2D9E~9h&ltaZ!sxHK8
z`?4A!pIKe#n4WMR;_6Aqw81;22$AfxC9$0rUsJuK67cVSYmup1<}>*&$$DdQr-;Zn
zTx_7u$G+QaxuV-6juVu*uO>Re8n7^l{|Vi9YxA*@oHkB(C|?+d&#%z6?x;k(_8d@t
z`<djQ?0W1`!*)=OhO27r;`j`cq0e{m!wO|F`C|3K!7wvR+pSg_HDUOgnS}m93+_lV
zObWl{@kZkz%I_CV-|_{dxjB=cA|+pr%VSl*E_WH`f36`JqxVo`u)5N+OMG;t??K}E
zV>d9PTAva4&a<t_pJsg23%!?mI-?i7-XC+e>6OCB<@`l_(^<fDf+^mpdRxM**qVPw
zCx35<C;I)Hmq}T_`eG*c#!0F9Wp@EBnqkC<w3@8gvgQ<T!rrUP+WX?QA22~#&K*+f
zR;M(?9v%PuLpGv>eo5BV5X^_m(5B%U6<U3ze$$PapgtRtkZw>C>99lj+{!R@6LQA*
zv;e4iIJ++Zzj47GZmLHuf09CPgX;M-<i5@~Neum}0|N6(w$0MTWFNIIB{sx%$6O80
z@7#K&Ly9q;`=InO9Zo3u#7OFMcEXm$?3#`q-+XzGLZTzpk71~$hmPE)PNlT;WHC$=
z2MlTa)}8KJg~Zr4Ytd)~b(wqYGS$}M6@W4+%&KR{9<C9Q#L$YAP~`Z1WbNi!`3DMV
zvO;8}Q&u|89awa%n(|2N<P{P#2)}iVTJBpMJ1uL#OTgu=4Xk6CZ!~H-(w+RT+9iv7
zyI3ve)4j;?BAlNECaZjuRP<B$sCOWEn-YAFgkA|N$E<o0*Yhj3Fr&K)<COr$svzcf
zoPdKyKkk0lM>1xXE>zghrHD7k$-S^WG&wQ0S;jzxnp;`!&==IHaUEyO<Ay_l1|nLY
z1MrWxUl<*2kgC+my-&09TkDSm$2qRJnRk6yqs(<oyLb16z)9>;FrN{kyROI6<YV87
zLooxtqH3?^<_vr6Y^XQ%HxkS*(Y*ZQ8?L4vY0cB?;6GHKPVZYEsef5_{r>zc)@Ycd
z$rA=iM5)zc+IeSnI*$~bYHg_6dEY88Wo275K2cpfdC)WEzM)287<$vTv+|_Sui7it
z=;<Eg;3|av^fUX%(^b}p*Q7-hW3u~`V3uvZvl^2T*qwD^(yu3&DYoMX@k++5&(V!#
zEj<%_ch?-}HY_Epd*sB>oIRzYlS#0UNdplif|A6arR&4#oSTYPO!R2~J+wV}*#S7g
z??5O?*6|kwG;(Rb@RaB_m_MzVJF<4=q6UWXR++m1>kmsk8laZju2c9~<q>hUzR$<^
zVV}0Nw4}mvbRrV3*b_+OSaVT%&E^hiu8S>=aNS~-84sKnfNS(E8Aqx3bSCB8Ol09g
zI;UjI8k?}_z6zNC@A41R1F~Hz=~n4JhqVBR4U+z4QXZ)eYY1!_#44MGDFhU`9u^$8
z<&SkS@C#DY%m3Yi^wSVlF}<`~kMFXPG${QXy`aVXr8AB1jN`<W_e%SgkA~Npjq^&M
z?|yn@6_eu-ffYm}X;fp7+#iUU3rl=Z^7t-<hFPYGUTLqTfDnf2hDuF+(h|yc;-|fu
zr#+tlGf2zoHX{uvK^>?SSZ#~JLkw+n&aO#q+?A5qT<ml6I%if1*9_ZuLGBEgn$o0#
z5zREFF**u8To!w}aq-)`TX?(`naA-gubib*lRTA?Wbi~gbuCL^+6y$t*tAl<i#VQk
zXw{k*$+?Q_TssZ))O_ZjW-Fd|?SEj6j0cri4B_1_+rAK@w^fPipdsCUjh^63KL{V<
z`Rts0V1%RA-eYr$cH8|Cn&>!Gwbo!<7!oU(1sPc|aXiGbII^kejV_+{08|4kvK>%S
z>vuwL^2H;hFPyEFOu_~TEP7$ts#}3FaA28VSPMl~z0bou#0<~Dn8epob#aPgfpK{3
z582Joy!mrhsQ3|aMN1HM(!(xNTz8?1B?)GQL9Q#jZZ@FvRTr=FKBx54j@UB8_&c0B
zLSsnSpef^U+WSL3togk&lW!NHCO&QWY?;eNI6jo45tZV>G#Gv_2W4|cSfWTrM<?Xq
zkT8j!>91Pv01E-`J$;~x`|#mn{{XHfz1Wm&9sQdNE)@P~rLKT`J9+>2bB@8ojK7id
zzc{)Q4oLPH#w7iXn!jRe040f#|GS6(zcukcYfb#uk?Fq-I>6aErV2!<QCGgOm-Uy`
zs!FJf`kCE7&zVG*(8w!sT4P96rZ2h@z!L$vg`6H?*AhAuU!XV;MvK`A@K3NppmPDm
zAQO_?*`Lf1k52JLM!O9yAFJMiR~q!bd_@ExeM{;ovc8Wx&aWfkkCay&9-OC!*$E~u
z0OHo0(JbANE>Z(jE$m2f`R<93DO;@>%n>8cj(q9SYwfc6Xn=p)nyM;1RJ=V%2v`i#
z4W&=KCVanG0EQ9v#abzcG=~Nb09Yx%8ZhX9zZS1t$p9pld*pJ10;G+_%e<*`+8+YD
zfgHm{!TH6P`1&Y<G15Pw+)p_!EYQF8s4}+_@a-w$!SQUDN2>AF@M+2A_#Q-HB?D+B
zxK~)c86jMWjsb)I*xy+}=^2j}zQUyKT=`%w7-UrKn}LX)rG54$p6*&^94S5rwM3q&
z1FB@*;5yf`F4*dmO9r)R$gHD0z%K_HY1Ipc!r@O^>sLCvnN=_`F--wQUr%GVa7p0M
z`T7mhTTL~Pgl~h*FXbCXpbmCR!p`iDLJ6*42x**nOd<=)&z21!&PQguY@SSuLh;F=
z<)P79{G|fAG5Yy8A7wGp#qhK0V!y%0D;Uk~FQwgis5N_FfGg4{+YsX_dz>7;&1p5%
zdw|C`Xa3F&gf!B!7UzaZ=4kXtt9LNeXJvHpEY_|xx^|dd4UL`q^`-zIh&2U9@yWN4
z3;X;<A(X8*$BXul=V{MV?OFR+gMyxT)P*a(-BgJt@Srxj`~)a+gm=JrdyRBk;+vK%
z-RS1Zd1@B<1b1w9`J=EVQRS3;gK~lkA(7jcm|eF}2dsiB9(<pjm8L3?$0MGcVX;V3
zw}kiW|3j;#<_{MiSar^M_k@H<(7FG1aWw;PawHy`-~2OkDf4-eTs>6o+t3Y`r<L(y
z!!sQFKw@JCI<hpAuKH_NX#3_LI<C{HB+eAc)ZE5Vz?xWmO^&9J?mh-oo*c62-RnJ^
z!Jd(s?i_xXa5*4?<jeJP+CSa_)l?kX`yx_FnY-czPGDzQ!Q0*?-KmE6OUyN<Sr)#y
zztrv>+no@aKN#-GaGRK=1?$E-Y|IK-He>0>+4S78s&unF76D{<hFR!iUIM+jGR1r>
zx)yV_NSb1Vcb`HX!(UkfPugq@ZB05YIy>`bX>gq7BysJASZ#H}lU{Zx&Cf??sMb)D
z%J5*l-XrWmA7Z&tmuPrX$EX>r*MVcJURZ+s>KFx#82+jU7n*uPZr7uv$<~g{ThALg
zZ!|2Uvv2D0pJ;sFT(S2=^-Mk5YBDX6OEix2T8J5%1`5?)$TwI<EoX_CJL;VeS1c{e
z1v;`bx4#FPEFPe$f)(RaZ`H6N8=;OObtll7#$kfU6@K_<Lrre$M;F{pBsr0$q|JKs
z8tCz`=Z`xu-d@_XH_|jaC?Vti{08Air5d}jW3`!~sxBp!zJ&It^hOi?HxBA4+LFfG
zyXle3lEtnE-J-d$j4dxYU4Y=OsL1l(>CuGi#*f!I)CD=lisAZeixwk$Bsi8%WNgym
z;sp6`_aZ+S@@m*FmT+525*RwrEU@Dl;m|2#sIp#IV$8xKEfVNsakU6=OB*QY307rW
z&;8oGPdd6zYUO*^z-HsUE&Q!fSpH_60hC7{h_bh|RJ0_J1BLzGfOWR`P#dQVy*d42
zc43k5hHv2I?gxP6mE~Dp{x`ay&8A}kE0dsUs5#R5t{UDcF`({GA_|m`3G;YBTZzOK
zrW5yemD5UlOXE)W6Qyg^ZoWraa<Q>kKBRhl(@{LCtCi0PB~8pdGx7oI3VfyxsrY@d
zY4hd%=x<cOTEr6s_$lQi@55O&n4D=%i0uo&Z~IKVbcB=B5ezlLKR!6xI=Axa<byNY
zlL>o(o+V#Ds~A3ljkC>b11bNK3Clu+?yU_2UyAfRQWoTxX(9(SZy#`N2&CyO;MKL`
z92r4$F<ytm2d~b$82YH5vkZg)?p&2U7ykN8CFjYP&&b$-H;Md*GCMkA+vpYDp8t43
zz{d`^F-bne-b2Mq`@xupwpCdhOpx=o!km_#mBn1i&j7ea@Y9QW>9hES^bggolA3My
zV*$2LQ>WkWHImfG4jZloRuB&kt#o@$+>=q;T4&U3fNxp{ScWeZZ`9<KBWhEs=qY@&
zVFygS%_4Ovlx%)kFt4^}nP3wx2tL&0?5SRmvdNOQoa}W_&<11$+OJJ0N6B4Zk)t#P
z^}@zCSy$?SJtrXNMR`4nIfj2fJQ7De1#08|z7UPMnzyr8tzs}{rB0gpdk`0K2EHj(
zbx$y~)dhD0nL8?GT^8oX=w3Y*>(+4@-}Drar!WP*13!3P%rCY6RGQ^)Vs-iF>F)OQ
zZLrwx@m`=LuhC*wp>jrYxn&H21Zs)=znq4Embo7Q8*DGOKRS<zy+t^`1xL({dLe<|
z#3!v!gD7)Pa0)tHgQ1VuD({mbp&l@bB!lOY9+jxbeCbDJKV7MtAOD6paMC`lZfOgZ
zoPMWRbjI2pLN^;=4nor2#6}aZOU0N8%s$fMakml_eP}{bGDUyTN=PURSyjgPd2AL}
z(MJi$L0wX^9g+j-A#@)BAawk+cZkU`UNHWtwQcn5zQg_#)fFVdAc3Xo-<t22cCn>r
zh3rx@()tq~{76-oTwk(}l^fvJ0m5Ys2j;7K16zJ2?(m<&-fOVl|Lm>&cOUzIkBEX5
z2dpF~9!vOc>R)vrWrn3QXAJ&#na)@3{@o?GzS&OQV+C^(9`T>!;sG~i&VpN3{ZqoD
zWNCF&U4a%|&Y-FUyjUMu1G_^V4F;0GH4LUXL69T#N&XA|Ulix|iAL=&AZOO(_)3wA
zc+u(1NISnlX1(`a8BH`E0nhN@iN?QgD82<mycrN<=!x1T`jYWh7QFz5ghIreVJ1f{
zxkPTxPqN}c7v>?7jWT!MTYUR7?`&zFu}8`t#01cCtBz*pYuk1pDcz=(5=JDza8w=*
z5tj#SSh>=e?{=q`^`-Mqtmn0NINj%07krfBFuZO}1QmxwA~7O4Ij{GF%6W7|VHh7k
zh~$rM*IY;U9&_Kd`dr$>XV_Z*hC||K^&hkv?t4wp89dR)qVS(eVzSu#M_PdpUFR_)
z(+=kD6(~yrU}Pg#5(OUDePuki&L>!gciog_-JPIsm;0A300<hbqEzT~aA>KqXUZWs
zI;_hRJ0R~v<{PRaQqw~7odopv@S&wa5<0Igw*m)*HM^9wc=j4~9(Pbch`g>ARXPP4
zfR&R8&m^Hg(@ESVb*RvL@im~U7)#;M;GavDF3EBhl1%W9?~-`N4(3Wkzr0GG3N7!|
z#l)9D^Sv)gQkd1uy|b#BtUxzxO|kR-K~5x>=><8W-+&S2VACD6{P);kk=jTMqGkcd
zMRehGseTUS!rXOOya##~t(@5Gf<S(!8?#V{!jz(r(tx>hH}NQ9<I*{;*t?MTyd5oh
zC*eKRwvpCJfUX~&XRD>sGCZ<^@c(+a6BWnuzI{eA=|r{GUVxmSMN7*1u@`*pjGf`T
zEkf}Dxeb_!Lx)S|T|Z^>7wR_lc)Hf(h>$xQJRt{!e0V2Q?f3e$IyhmvSfh|<Y^K1L
zZ)*t2<@?egSVKytV*-AMkjGNHGbTpAYCqKH-R6hIAEl_8R%`XdE`#|OFSv8C8rN|z
zve^yxrItzQ_W*L%9vj3wm6FsPb>u+txeY91pS%E15w{tU#=e>eh>@p4$dx-Eh?msL
zL63=HP-A<O6G@(iLBL4gU{dew4d^+Z`q(d*!x%U@as50@GXkL?!;?!rF!diNVv6_C
z-FRTRVLA&!58J`vt!6|X0Thk5s77X{^<ZVJl^#XT3sbby`em!;`r39nFH@@23z}p3
zD8JO6QI+iq=nbHnr~@6v0D%J%0CX)gGVmdJ>V6N|SD*Ybv~zbpguRdY>8wz1S9#+o
z`mvh2_f&9x%v3PL0H0N^`Kcrdg<O5Y_>Pi7kN5Wz@SsIp`HeChK1SUn$>Mi;J;`m>
zMSAmYCPxaTSy)m=X8K9Y#tueEflO;s8bMIlDnXRIi*lVx<~Ka|k;R9)m-qO7NgTkP
zu1m=*D9YWagqQf;85m4|iNlTGzxH@3P&!C?i<^&+ju9WzU}rF5nYTeJ0tpx+@wZ<0
zJHNQD$>W7{#oO^s`+>V25QnjEpXpAwxjXt9I~dB|uD5#{>A{q3KO7D#l+xF=pt=G(
zW}CTURNBe%SIbtL;YJUW9dSn=FjFY#YNKBN%v=xNWDyb*>IozY;*yN;C^qq->yGE-
z@X3elX{tH{oC<}wGbTc_O=|sy20AO9QBA|sV}7Z=IBe*aUil-Va3DQHiRO}Do{@$G
z+QT@uC5^DzFzLeCro8E5Q=n~Z;jy*29YD35#m~TX%=kvl?WZ>sNX|!)+WyGKV!>}8
zw}id|pw%p&mWw6FD7Fnc#zME)nDJ@zQ}P&VaP6eZsQt+YTUxH)inZ)|5HKsvWnU$A
z8)GV6mss;{{Fn*PXD{bv!)|%KoE`DdH&3(t5vZ~6(4O0--p6pkhGpx$MN{`t;pVvJ
z^&;9LqQEg10y5ly2m7$I>#E)kdm9@XIS=2ghu>$BY#K*h!D8h<&Z6oy&8dteHB+Wy
zpZ#%$XXCzN?P2h<iP)b{9jtN4xo*~@{0eL_$C%=eY}`^VR{L>X?kG*=W2RbR{)V*m
z%8V=F17TNq>H5@IH?+01yV`gOW--RecSS<Ba)Nxomn`P{D`ThXID5n6_||QLyJw{v
zLhB~=K>YQVbdCu>Mjo7bQmz3|OWlK)yU_Beo<XDOg^H2ZfjM?*Yk?ap%%pXS&^bIZ
zhWLym*MGGmB*ssvn$4K!cf2M)b{>XWf@@n=>4$m1QwcU_;>9$SQFC9A$V6X`h&Y-c
zS#+TSLBffABe$-@v9{(NJzTm|`opF(cdStX@6K226=t0Sts4yuMC^%PZm;`5P1<6V
z=Gl2<M@S<tKonWzlWmd&LfgVkGmQ2Cjb(o<P)*bTY0R&26UZX2A$)@(@HT&H_59n(
zf<7BhMxa~u9#Gn&A_0?+`+e?u2<T#`KzvI+G9-JUOhXQG40G0nad4y`m!eVnKW)=Y
zwgW<^8HY_Qkvp6aYyDe#TwtD;<mJXET3JS3vG$L&5Dn3TQqZ~<X!=&0{)*oy@=#CR
z@^=r()+9r;m788M$`B8>0LFH(W3y&b8q|~{r18jKUG2baQ>{GYvzYKeO=;^2SX%`E
zYhycGF6umc|FWY_u%XBq8z+uHalB_P{diVk5a-<w{_)nNW!^eXVTYqKBQL)<@;YsX
zW5m^yKYN8vW4jG+dErCQe832r`R>{$NtmdF$bZ67vW|`6-IB+V^wCzTVD7@fFC&_M
zb5ko|uu;b+Z`Y6!=Y0Xx`L#9t=t|ZbX0>t2@pp>fW;Ye}W7$R3B)U98n8`ToGWBpo
zHnfml*g%(Dn%^Z#d=TPi_6e0g#aX!B*AR5Q>v-ds%1igGM03}@a_?X?D<GdyOUu%2
zK<757PHfyL4=SWEW+K3{$LUwPG+8@dFJ3?WOs^QTSc<0cRL_JxVDHB`4}LbF{RmPB
zjD)i_4ORZwtw-$I<V=c0bc){5C??^{rJgjbSm6<GIG`gen?hrcN@|4&LvaFm4wb&?
zINbo3!Vu%6OVXv|k#dh>|5auZi`$;H7zH3D^~Rz|UY~#Oy28w?emYCR7a8xc_^fCC
zP_A4VFC*W`Z!T&?wP5a|ZYWIF+ldE>nA>KiB2ip#q@7!m(Rj>Mj0%C-NKvT0pAh$A
z5!X2>jRvRNfA5(RYuzQ~ONC~ikfj*|?q?qa7vk7&&|xIC!;Jf}eSmR-vOkL`kG}KK
znUchO!X9ha59U?NhDR#6(S%!CL4Wwe1);~uJJ!`o{H^U{zddiGkX6a@#<HsPgYYOD
zh0+7^@&?+ars-WSan_x6i`yKEbFiHpWSaD5Y^WwO5(A3ElW)9eF6y!%sbvf79B#U5
zlaZWIPih9V{X>j;_l`^nr}Zj0ef#ORp7SSte@IP3@82X@ZXOr~AilCZw#U1&#&3e-
z0La>2!x_AzsRI2qqUQRrw~Xzaqf`~})s$g;1N$LrHdw>?a8y2S2}~Kw4e;-$G>P2Y
zqQbxP>dusV4ZReV$tOZle8I~N2~m$C?6Ggxi$n<*X2jeM$!ZpBlFvT9fM<6Fq>kdK
zRRbx`F)~DQXH>lPN<*7xM<WMee23dX`~f^w@@Cl>bBzxMy)TQ>NHO=T5|2>{K=Ls4
zQZHfN_lNp$I24b%BrOc*{{H1B<Q`Q*D?d`bMIgK?E$bzE!Xx3>z`RLqseyr_76RqQ
zTkE<~(X+KL6@33sV&Iqhot3n%o&TKV`|mymlzi>q_&>~i0&0<;#Q%Z9`ndmo_vpNk
z(^x42!=1bTC5CG8irb*(rsUms{uK%mXlnCfrJ(EIwbmYqIpWxz!}P<|3aj|Qz*bqG
Mu|HjP^6K6H1JkcDtpET3

literal 0
HcmV?d00001


From 04f6b128900a7f522d2f5ca85b8bbc041686883a Mon Sep 17 00:00:00 2001
From: saroup <saroup@microsoft.com>
Date: Thu, 13 Feb 2020 08:39:22 -0500
Subject: [PATCH 2/4] adjusting spaces

---
 docs/ConfusionMatrixCalculations.md | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/docs/ConfusionMatrixCalculations.md b/docs/ConfusionMatrixCalculations.md
index 48396c7..b76953e 100644
--- a/docs/ConfusionMatrixCalculations.md
+++ b/docs/ConfusionMatrixCalculations.md
@@ -23,19 +23,21 @@ For each test utterance one or more confusion matrix results can be computed:
 |Utterance text        | Actual Intent  | ExpectedIntent   | 
 |:---------------------: | :------------: |:---------------:|
 |who is bob goodermuth? | ContactInfo | ContactInfo|
-**True Positive for ContactInfo:** The expected and actual intents are equal
 
+**True Positive for ContactInfo:** The expected and actual intents are equal
 
 
 | Utterance text        | Actual Intent  | ExpectedIntent   | 
 |:---------------------: | :------------: |:---------------:|
 |who is bob goodermuth? | Greeting | Greeting|
+
 **True Negative for ContactInfo:** Actual intent is not ContactInfo, and the expected result it not ContactInfo
 
 ### Producing two results
 | Utterance text        | Actual Intent  | ExpectedIntent   | 
 |:---------------------: | :------------: |:---------------:|
 |who is bob goodermuth? | Greeting | ContactInfo|
+
 **False Positive** for Greeting and **False Negative** for ContactInfo since intents mismatch
 
 ## Confusion matrix output for entity comparison
@@ -43,6 +45,7 @@ When the expected list of entities is not empty, we check that each entity match
 | Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob tomorrow| 	26, 33, datetime, tomorrow <br> 23, 25, personName, bob| 23, 25, personName, bob <br>26, 33, datetime, tomorrow|
+
 **True Positive** for datetime and personName since they have a corresponding match in the actual entities based on entity type, text match and index match.
 
 | Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
@@ -97,6 +100,7 @@ Besides comparing entity values for the entities comparison we also compute True
 | Utterance text        | Actual entity value	 | Expected entity value  | 
 |:---------------------: | :------------: |:---------------:|
 |Do I have unpaid bills?| "bills",<br>"invoice", <br>"invoices"| checks|
+
 **False Negative:** for checks since the entity matches by type, startPos, endPos, but the value does not have the expected subtree in the entity resolution value
 
 

From 10d3c1f468bf83c70ab427ec44fdee63c12a5f64 Mon Sep 17 00:00:00 2001
From: saroup <saroup@microsoft.com>
Date: Thu, 13 Feb 2020 08:51:24 -0500
Subject: [PATCH 3/4] adding text confusion matrix calculations

---
 docs/ConfusionMatrixCalculations.md | 40 ++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 6 deletions(-)

diff --git a/docs/ConfusionMatrixCalculations.md b/docs/ConfusionMatrixCalculations.md
index b76953e..92389c9 100644
--- a/docs/ConfusionMatrixCalculations.md
+++ b/docs/ConfusionMatrixCalculations.md
@@ -42,26 +42,26 @@ For each test utterance one or more confusion matrix results can be computed:
 
 ## Confusion matrix output for entity comparison
 When the expected list of entities is not empty, we check that each entity matches by type, value and by the occurence index of the matching text in the utterance.
-| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+| Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob tomorrow| 	26, 33, datetime, tomorrow <br> 23, 25, personName, bob| 23, 25, personName, bob <br>26, 33, datetime, tomorrow|
 
 **True Positive** for datetime and personName since they have a corresponding match in the actual entities based on entity type, text match and index match.
 
-| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+| Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob tomorrow| 	26, 33, datetime, tomorrow <br> 23, 25, personName, bob| |
 
 **False Positive** for datetime and personName since they don’t have a matching entity in the expected entities list.
 
-| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+| Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob | 	17, 24, personName, bob| 22, 24, personName, bob|
 **False Positive** for personName since the start position doesn’t match with the expected one.
 
 ### Producing two results
 
-| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+| Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob | 	22, 24, userName, bob| 22, 24, personName, bob|
 
@@ -69,19 +69,47 @@ When the expected list of entities is not empty, we check that each entity match
 **FalseNegative** for personName since it doesn’t have a corresponding match in the actual entities
 
 
-| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+| Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob tomorrow| 	| 23, 25, personName, bob <br> 26, 33, datetime, tomorrow 
 
 **False Negative** for personName and datetime since they don’t have a corresponding match in the actual entities26, 33, datetime, tomorrow 
 
 
-| Utterance text        | Actual entities (startPos, endPos, type, value)	 | Expected entities (startPos, endPos, type, value)   | 
+| Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Good Morning|  |
 
 **True Negative:** There are no expected entities and no entities identified
 
+## Confusion matrix output for text comparison 
+We compute the text results for speech tests only.
+
+|Actual utterance text  | Expected utterance text  | 
+|:---------------------:| :-----------------------:|
+|Good morning, Cortana  | Good morning, Cortana    | 
+
+**True Positive:** Text equality between normalized actual text and expected. Where consecutive spaces have been normalized to one space, punctuation has been removed and case is ignored
+
+|Actual utterance text  | Expected utterance text  | 
+|:---------------------:| :-----------------------:|
+|Good morning, Courtney | 	Good morning cortana   | 
+
+**False Positive:** texts are not equal
+
+|Actual utterance text  | Expected utterance text  | 
+|:---------------------:| :-----------------------:|
+|empty                  | 	empty                  | 
+
+
+**True Negative:** Both utterances are empty
+
+|Actual utterance text  | Expected utterance text  | 
+|:---------------------:| :-----------------------:|
+|empty                  | Good morning, Cortana   | 
+
+**False Negative:** Only the actual utterance text is empty 
+
 
 ## Confusion matrix output for entity value comparison 
 

From f9f99c4c2dde8d52c5c3c2935e5ccad5fee9c179 Mon Sep 17 00:00:00 2001
From: saroup <saroup@microsoft.com>
Date: Thu, 13 Feb 2020 20:10:34 -0500
Subject: [PATCH 4/4] minor modifications

---
 docs/ConfusionMatrixCalculations.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/ConfusionMatrixCalculations.md b/docs/ConfusionMatrixCalculations.md
index 92389c9..56450ea 100644
--- a/docs/ConfusionMatrixCalculations.md
+++ b/docs/ConfusionMatrixCalculations.md
@@ -57,6 +57,7 @@ When the expected list of entities is not empty, we check that each entity match
 | Utterance text        | Actual entities<br>(startPos, endPos, type, value)	 | Expected entities<br>(startPos, endPos, type, value)   | 
 |:---------------------: | :------------: |:---------------:|
 |Schedule meeting with bob | 	17, 24, personName, bob| 22, 24, personName, bob|
+
 **False Positive** for personName since the start position doesn’t match with the expected one.
 
 ### Producing two results