Model Performance Parameters for Custom Models in Generative AI

When you create a custom model, OCI Generative AI fine-tunes a pretrained model with the dataset. After fine-tuning is complete, you can access the model performance parameters on the model's details page to determine if your model has improved for your use case after fine-tuning on the dataset. The model performance parameters are computed on the validation dataset.

Model Performance Parameters for Generation Models

The following metrics apply to custom models.

Accuracy
Measures whether the generated tokens match the annotated tokens. For example, an accuracy of 0.9 means that 90 percent of the output tokens matched the tokens in the dataset.
Loss
Accuracy tells you how many predictions the model got wrong, and loss measures how wrong the generated outputs of a model are. A loss of 0 means that all outputs were perfect, while a large number indicates highly random outputs. Loss decreases as a model improves.