Oracle Cloud Infrastructure Documentation

Managing Imported Models

In addition to using the hosted pretrained models in OCI Generative AI, you can import validated open source and third-party models (for example, from Hugging Face) into OCI Generative AI, host them, create endpoints, and use them like any other model.

Learn about Compatible Models for Import.

Hugging Face Prerequisites

Before you import a model directly from Hugging Face:

  • Decide which validated model from Hugging Face to import and note its recommended dedicated AI cluster hardware unit shape.
  • To access and use some models, you need a Hugging Face token, especially the more recent and gated versions such as Llama 3 and Llama 3.1 For these models, generate an access token from your Hugging Face account settings under Access Tokens. Ensure it has the necessary permissions (at least "read" access).

Object Storage Prerequisites

Before you import a model from an Object Storage bucket:

  • If you're not an OCI admin, ask one to give you IAM permission to manage Object Storage in your compartment:
    allow group <your-group-name> to manage object-family in compartment <compartment-with-bucket>
  • Decide on a model that works with the /v1/chat/completions endpoint—only these models are supported.
  • Ensure the model supports only one of these capabilities:
    • TEXT_TO_TEXT: text in, text out
    • IMAGE_TEXT_TO_TEXT: image or text or both in, text out
    • AUDIO_TO_TEXT: audio in, text out
    • EMBEDDING: text in, vector embeddings out
    • RERANK: query and candidate documents in, relevance scores and a reordered list out
  • Save model artifacts in an Object Storage bucket.
  • Important: The configuration file must be called config.json for a successful import, similar to most Hugging Face models.

Dedicated AI Cluster Hardware Unit Shapes

When you create a hosting dedicated AI cluster for an imported model, select the imported model and a compatible hardware unit shape that's available in the target region. Imported model unit shape names don't include a model-provider prefix. For example, select H100_X2 or A100_80G_X1, rather than a provider-specific value such as Cohere_H100_X2.

Important

Imported models don't require the minimum hosting commitment of 744 unit-hours that applies when you host pretrained models available in OCI Generative AI on dedicated AI clusters.

Before you create the cluster, see Imported Model Hardware Unit Shapes, Service Limits, and AI Unit Counts to:

  • Identify the hardware shapes available in the target region.
  • Select an imported-model hardware unit shape.
  • Find its service limit name and required hardware units.
  • Find its AI unit count for calculating the hourly price.