Planning on adding an evaluation metric that can be used to benchmark trained alpaca models.
Going to focus on these two datasets for evaluation:
- SquaD Dataset - F1 Score
- WikiText Dataset - Perplexity
I'm not so sure the Wikitext perplexity score will give us much useful information, but seems to be a popular metric for these foundation models. I'm more interested in the Squad F1 score, which will give us a standard benchmark for a Q/A task. Even though alpaca is not trained on Squad style dataset, I think with the right prompt, it can be done.
Planning on adding an evaluation metric that can be used to benchmark trained alpaca models.
Going to focus on these two datasets for evaluation:
I'm not so sure the Wikitext perplexity score will give us much useful information, but seems to be a popular metric for these foundation models. I'm more interested in the Squad F1 score, which will give us a standard benchmark for a Q/A task. Even though alpaca is not trained on Squad style dataset, I think with the right prompt, it can be done.