Deploy custom model with Kimchi

Want to run your own fine-tuned model on your own infrastructure, but still keep things simple for your developers?

This tutorial walks you through how to take a model you've stored in Google Cloud Storage (GCS) and deploy it with Kimchi, so you can call it via an OpenAI-compatible API just like any other model.

Introduction

In this tutorial:

• Base model – a general-purpose model like LLaMA 3, Mistral, etc.
• Fine-tuned model – that base model, further trained on your data (for your domain, tone of voice, tasks, etc.).
• Custom model in Kimchi – any model whose weights live in your storage and are deployed through Kimchi into your Kubernetes cluster.

The idea is simple: you train the model wherever you like, store the artifacts in GCS, and then Kimchi takes over the last mile (infra, GPUs, and deployment).

Prerequisites

Before you follow the steps, you'll need:

1. Kimchi set up and connected to a cluster
2. A fine-tuned model saved in a GCS bucket

Model artifacts are already exported and uploaded to GCS.

A typical layout might look like:

gs://my-ml-bucket/models/llama3-support-bot-v1/
├── model.safetensors  # or model.bin / similar
├── tokenizer.json
├── config.json
└── metadata.json      # optional, e.g. task, dataset, tags

Step 1: Prepare your model artifacts in GCS

First, make sure your model artifacts in GCS are:

Complete – you'll typically need:

• model.safetensors or pytorch_model.bin (or equivalent)
• config.json
• tokenizer.json (or tokenizer.model, depending on the stack)

In a single folder:

gs://my-ml-bucket/models/llama3-custom-model/

Step 2: Connect your repository in Kimchi

Now we jump into Kimchi.

1. Log in to the Cast AI console.
2. Navigate to Kimchi → Hosted Models → Repositories
3. Click Connect to start the flow.
4. Copy and paste your bucket path.
5. Click Next.
6. Run the script in your console
7. Click I ran the script and wait for the script to finish.
8. The repository was successfully connected.
9. Click Register model

Step 3: Register your custom model

You'll now go through a short form.

Select a model: Choose a model from the list provided in the dropdown. These are the models that were detected in the bucket.

Base model: Pick the base model you fine-tuned from the dropdown.

Description: Write a short description of your model.

Using LoRA adapter: Check this box if your custom model is using LoRA adapter. This helps Kimchi understand what runtime/config to use for deployment and validation.

Click Register model, then click Deploy model:

Step 4: Configure and deploy the model

Now that the model is registered, it's time to actually deploy it. You'll be guided through a short deployment flow.

Kimchi utilizes a specialized node template based on model size and other configuration settings.

Select the cluster your model will get deployed on

Configure the automation and scaling features as desired

Click Deploy:

Wait for the model to get deployed. This step may take up to a couple of minutes, depending on the model size.

Behind the scenes, Kimchi will:

• Deploy the Helm chart
• Mount or pull the model from GCS
• Start the model server
• Wait for the model to load

Once the deployment status says Running, your custom model is live.

Step 5: Call your custom model

With the deployment healthy, you can now call the model like any other Kimchi model.

Since the model is running in your own Kubernetes cluster, we need to run:

kubectl run test-client --rm -i --tty --image=alpine/curl -- /bin/sh

to create a temporary pod to which we can send cURL requests.

Here's a minimal example (adjust the model name as needed):

curl http://castai-ai-optimizer-proxy.castai-agent.svc.cluster.local:443/openai/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer $CASTAI_API_KEY' \
  -v -X POST -d '{
    "model": "Llama-3.2-3B-Instruct-abliterated",
    "messages": [
      {
        "role": "user",
        "content": "What kind of instance types to use in GCP for running an AI training model?"
      }
    ]
  }'

From here, your application can treat this model like any other OpenAI-style model, just point to Kimchi, use the custom model name, and you're good to go.

That's it! You've just:

1. Registered a custom model in Kimchi
2. Deployed it onto your own cluster
3. Called it via an OpenAI-compatible endpoint