Getting Started

You can run the Kimchi Proxy in your Kubernetes cluster or use the one on the Kimchi platform. In both cases, the Proxy expects the request to follow the OpenAI API contract described in the OpenAI API Reference documentation. The response will also follow the OpenAI API contract.

The only supported endpoint is the /openai/v1/chat/completions, which mimics OpenAI's /v1/chat/completions endpoint.

Installation options

Choose from three installation methods based on your infrastructure management approach:

Console installation

Install Kimchi through the Kimchi console interface:

1. Navigate to Kimchi > Model Deployments in the Kimchi console
2. Click Install Kimchi
3. Select your cluster from the list (only eligible clusters will appear)
4. Run the provided script in your terminal or cloud shell
5. Wait for the installation to complete

Terraform installation

Kimchi can be installed automatically using our official Terraform modules. Set the install_ai_optimizer variable to true in your configuration.

EKS example

module "castai-eks-cluster" {
  source = "castai/eks-cluster/castai"

  aws_account_id     = var.aws_account_id
  aws_cluster_region = var.aws_cluster_region
  aws_cluster_name   = var.aws_cluster_name

  castai_api_token = var.castai_api_token

  # Enable Kimchi
  install_ai_optimizer = true

  # Other configuration...
}

Available Terraform modules:

• EKS: terraform-castai-eks-cluster
• GKE: terraform-castai-gke-cluster
• AKS: terraform-castai-aks

Helm installation

For direct installation without using Terraform modules or the console:

# Add the Kimchi Helm repository
helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update

# Install the Kimchi Proxy
helm upgrade -i castai-ai-optimizer-proxy castai-helm/castai-ai-optimizer-proxy -n castai-agent \
  --set castai.apiKey=<CASTAI_API_TOKEN> \
  --set castai.clusterID=<CASTAI_CLUSTER_ID> \
  --set castai.apiURL=<API_URL> \
  --set createNamespace=true

Replace the environment variables with your actual values:

• <CASTAI_API_TOKEN>: Your Kimchi API key
• <CASTAI_CLUSTER_ID>: Your Kimchi cluster ID
• <API_URL>: Kimchi API URL (https://api.cast.ai or https://api.eu.cast.ai)

Streaming

The API fully supports both streaming and non-streaming responses.

To enable streaming, add "stream": true to your request body. When streaming is enabled, you'll receive the response as a data stream, following the same format as OpenAI's streaming responses.

Example request with streaming enabled:

curl https://llm.kimchi.dev/openai/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer $CASTAI_API_KEY' \
-X POST -d '{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": "What kind of instance types to use in GCP for running an AI training model?"
    }
  ],
  "stream": true
}'

Supported providers

You can find the list of supported LLM providers and their supported models here. Kimchi can proxy requests to any provider and model combination from this list once they are registered.

Model quantization and precision

When working with Large Language Models (LLMs), model quantization - the process of reducing model precision to decrease memory usage and increase inference speed - plays an important role in balancing performance and resource utilization.

Understanding model precision

Models can be quantized to different precision levels:

• 16-bit (FP16): Full precision, offering the highest accuracy
• 8-bit (INT8): Reduced precision with good accuracy-performance balance
• 4-bit (INT4): Lowest supported precision, maximizing performance at the cost of some accuracy

When deploying self-hosted models, Kimchi uses optimized quantization (typically 4-bit or 8-bit) by default to balance performance and resource usage.

Viewing model quantization

The Kimchi /ai-optimizer/v1beta/hosted-model-specs API endpoint returns the quantization format for each model using the GGUF standard (e.g., Q8_0, Q4_K_M). The API response lets you view the specific quantization being used for any model. For example:

{
  "items": [
    {
      "model": "llama3.1:8b",
      "description": "Llama 3.1 8B is a compact 8 billion parameter model balancing performance and efficiency. It features a 128K token context window, multilingual support, and optimized low-latency inference. Ideal for startups and mobile apps, it handles content generation, summarization, and basic language tasks effectively.",
      "cpu": 6,
      "memoryMib": 16384,
      "provider": "ollama",
      "tokensPerSecond": 50,
      "createTime": "2024-11-08T12:36:45.258213Z",
      "quantization": "Q4_K_M",
      "regions": [
        {
          "name": "us-west1",
          "pricePerHour": "0.24082",
          "cloud": "GCP",
          "instanceType": "n1-standard-8",
          "gpuCount": 1,
          "gpuName": "nvidia-tesla-t4"
        }
      ]
    }
  ]
}

Register LLM providers

To enable the Kimchi Proxy to route your requests to the appropriate LLM provider, you must register the providers you want to use (e.g., OpenAI, Gemini, Groq, Azure).

To register the LLM providers, make a POST request to the relevant Kimchi API endpoint. Below is an example of OpenAI, Azure, Gemini, and VertexAI providers being registered, specifying authentication, available models, and provider-specific parameters.

curl https://api.cast.ai/v1/llm/providers \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'X-API-Key: $CASTAI_API_KEY' \
  -X POST -d '{
  "providers": [
    {
      "name": "openai-gpt3.5",
      "supportedProvider": "OPENAI",
      "apiKey": "<openai-api-key-1>",
      "models": ["gpt-3.5-turbo-0125"]
    },
    {
      "name": "openai-gpt4+",
      "supportedProvider": "OPENAI",
      "apiKey": "<openai-api-key-2>",
      "models": ["gpt-4o-2024-05-13", "gpt-4-0613"]
    },
    // Azure OpenAI configuration
    {
      "name": "azure-provider",
      "supportedProvider": "AZURE",
      "url": "https://something-azure-openai.openai.azure.com",
      "apiKey": "<azure-api-key>",
      "apiVersion": "2024-02-01",
      "models": ["gpt-3.5-turbo-0125", "gpt-3.5-turbo-0301", "gpt-4o"],
      "isHosted": true
    },
    // Google's Gemini API configuration
    {
      "name": "gemini-api-provider",
      "supportedProvider": "GEMINI",
      "apiKey": "<gemini-api-key>",
      "models": ["gemini-1.5-flash", "gemini-1.5-pro"]
    },
    // Google Cloud Vertex AI Gemini configuration
    {
      "name": "vertex-ai-gemini-provider",
      "supportedProvider": "VERTEXAIGEMINI",
      "apiKey": "<gcloud-access-token>",
      "models": ["gemini-1.5-flash", "gemini-1.5-pro"],
      "url": "https://us-central1-aiplatform.googleapis.com/v1/projects/some-project/locations/us-central1",
      "isHosted": true
    }
  ]
}'

Note that you may register a single Provider multiple times. For instance, you can have an OpenAI Provider per OpenAI API Key to limit the models that can be used by each API Key.

Make requests to the Proxy

To start making requests to the Kimchi Proxy running on the Kimchi platform, follow these steps:

1. Go to Kimchi → API Keys and click Create API key. Copy the key and store it securely.
2. Include the API Access Key in the X-API-Key header or the Authorization header with the Bearer schema when making requests to the Proxy endpoint.
3. Make a POST request to the Proxy endpoint with the desired payload:

Authorization Bearer

curl https://llm.kimchi.dev/openai/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer $CASTAI_API_KEY' \
-X POST -d '{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": "What kind of instance types to use in GCP for running an AI training model?"
    }
  ]
}'

X-API-Key Header

curl https://llm.kimchi.dev/openai/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'X-API-Key: $CASTAI_API_KEY' \
-X POST -d '{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": "What kind of instance types to use in GCP for running an AI training model?"
    }
  ]
}'

Modify the request payload as needed, following the OpenAI API Reference documentation.

Supported endpoints

Different tools and integrations may require different base URLs for the Kimchi Proxy. The default endpoint works with most standard OpenAI SDK implementations and tools like Azure Prompt Flow. Here's a table of known endpoint requirements:

If you use a tool or SDK not listed here and encounter connectivity issues, try the default endpoint first. For tools requiring a different endpoint configuration, contact our team on the Slack community channel or Kimchi support.

We regularly update this list as we verify endpoint requirements for different tools and SDKs.

Run the Kimchi Proxy in-cluster

If you prefer to run the Kimchi Proxy in your own Kubernetes cluster, follow these steps:

1. Install the Kimchi Proxy using Helm:

helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update

helm upgrade -i castai-ai-optimizer-proxy castai-helm/castai-ai-optimizer-proxy -n castai-agent \
  --set castai.apiKey=$CASTAI_API_TOKEN \
  --set castai.clusterID=$CASTAI_CLUSTER_ID \
  --set castai.apiURL=https://api.cast.ai

Replace the following values:

The in-cluster proxy supports the following API key configuration:

2. Make requests to the in-cluster proxy endpoint:

curl http://castai-ai-optimizer-proxy.castai-agent.svc.cluster.local:443/openai/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'X-API-Key: $CASTAI_API_KEY' \
  -X POST -d '{
    "model": "gpt-4",
    "messages": [
      {
        "role": "user",
        "content": "What instance types should I use for AI training?"
      }
    ]
  }'

Viewing your generative AI savings report

After setting up the Kimchi Proxy, you'll want to see how many tokens you're using and what it costs you. The Analytics report becomes available once you make requests through the proxy.

Requirements for the savings report

To see your savings data:

1. Ensure you have properly registered your LLM providers
2. Make at least a few successful requests through the proxy
3. Wait a short time for the data to be processed (usually just a few minutes)

The report will automatically appear in your Kimchi console once there is actual usage data to analyze. This helps ensure the savings calculations are based on real traffic patterns rather than estimates.