Oracle Cloud Infrastructure Documentation

Search Engines

The traditional approach for candidate generation using k-NN search (k-nearest neighbors) is accurate but very slow, especially for large scale systems where very low latency is required.

ANN (approximate nearest neighbor) search uses tools that restructure indexes more efficiently and reduce the dimensionality of searchable vectors. Using this approach is a tradeoff between accuracy and low latency. By leveraging ANN with k-NN, search engines can approximate the nearest neighbors of specific query documents and retrieve relevant candidates with very low latency, improving search latency for large datasets. OpenSearch 2.11 supports the NMSLIB, FAISS, and LUCENE search engines, which all implement ANN. For more information, see k-NN search with filters.

Lucene Engine

The Lucene K-NN filter uses the HNSW graph in the implementation.

The following example shows creating an index using the Lucene engine:

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100,
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "properties": {
      "location": {
        "type": "knn_vector",
        "dimension": 2,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "lucene",
          "parameters": {
            "ef_construction": 100,
            "m": 16
          }
        }
      }
    }
  }
}

To use an ingestion pipeline on the index, add the pipeline in the API call, ensuring that the model deployed on the ingestion pipeline has the same dimensionality of the KNN vector specified in the API call, as follows: 

PUT /test-lucene-index
{
  "settings": {
    "default_pipeline": "test-nlp-pipeline",
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100,
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "properties": {
      "location": {
        "type": "knn_vector",
        "dimension": 2,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "lucene",
          "parameters": {
            "ef_construction": 100,
            "m": 16
          }
        }
      }
    }
  }
}

Faiss Engine

For the Faiss engine, you can use the HNSW algorithm or the IVF algorithm to implement the ANN filter. See Faiss k-NN filter implementation for details about the Faiss configuration parameters.

The following examples shows how to create an index using the Faiss engine.

Without an ingestion pipeline: 

PUT /test-faiss-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "item_vector": {
        "type": "knn_vector",
        "dimension": 3,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "faiss"
        }
      }
    }
  }
}

With an ingestion pipeline

PUT /test-faiss-index
{
  "settings": {
    "index": {
      "knn": true
    },
   "default_pipeline": "test-nlp-pipeline"
  },
  "mappings": {
    "properties": {
      "item_vector": {
        "type": "knn_vector",
        "dimension": 3,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "faiss"
        }
      }
    }
  }
}

NMSLIB Engine

The following example shows how to create an index using the nmslib engine: 

PUT /nmslib-index
{
    "settings": {
        "index.knn": true,
        "default_pipeline": "test-nlp-pipeline"
    },
    "mappings": {
        "properties": {
            "passage_embedding": {
                "type": "knn_vector",
                "dimension": 1024,
                "method": {
                    "name":"hnsw",
                    "engine":"nmslib",
                    "space_type": "cosinesimil"
                }
            },
            "passage_text": {
                "type": "text"
            }
        }
    }
}

Spaces

Each of the supported engines use some heuristic to compare document embedding and retrieve the most relevant documents by measuring the similarity. You have the choice of several objective metrics to use. The metrics you choose depends on the custom use case, model, and data. For more information, including a table showing the available options to choose from, see Spaces.