Model Performance Parameters for Custom Models in Generative AI
When you create a custom model, OCI Generative AI fine-tunes a pretrained model with the dataset. After fine-tuning is complete, you can access the model performance parameters on the model's details page to determine if your model has improved for your use case after fine-tuning on the dataset. The model performance parameters are computed on the validation dataset.
Model Performance Parameters for Large Language Models
The following metrics apply to custom models.
- Accuracy
- Measures whether the generated tokens match the annotated tokens. For example, an accuracy of 0.9 means that 90 percent of the output tokens matched the tokens in the dataset.
- Loss
- Accuracy tells you how many predictions the model got wrong, and loss measures how wrong the generated outputs of a model are. A loss of 0 means that all outputs were perfect, while a large number indicates highly random outputs. Loss decreases as a model improves.