Naive Benchmarking

[grahamschock]

We should think about how to show our algorithm is optimal.

For right now I think this means

• Deciding what client workloads would look like
• Code to actually profile the systems performance
• Who decides what is good performance, the server or the client? If it's the client we have to deal with the synchronous headache of having a process for each request 🙃
• Generations of data and visualizations
• What is optimal?

[grahamschock]

Deliverables:

• Graph similar to Yao
• Working with just 1 client (how many requests we answer within x time)
• Looking at how many stages
• Looking at what deadline miss rate is
• Actually measuring accuracy, determining what accuracy would be and if our workload helps with this

[kylevitale2]

Accuracy should be measured in statistical confidence as opposed to how close the current calculation is to our expected result (our matrix calculation by chance could be close to the answer at a certain stage, but we are more confident in our answer with additional stages)

[grahamschock]

I don't really understand. Couldn't we also have an ML that will identify something correctly accidentally? @kylevitale2

[kylevitale2]

I may have misunderstood the graphs I was looking at in the Yao paper. In the OPT algorithm, the computed confidence of each level is known beforehand, so I thought maybe this means that we look at the confidence increase as our deadlines increase for not just that function, but all of them.

I do think that using the confidence value could resolve this issue of accidentally correctly identifying something, however, because as the confidence increases, we can say with more certainty that the ML has correctly identified something (let's say a stage earlier on thinks something is 55% dog, 45% cat, and then as it advances to the next stage the ML think something is 70% dog 30% cat. Both would say that we have found a dog, but our later stage would have a higher confidence value from continuing calculations).

In a sense I'm using confidence and accuracy interchangeably.

[grahamschock]

@kylevitale2 Let me know if you want me to run anything on the GPU machine to test.