LlamaIndex with a custom endpoint

By the aitokensai editors · Verified 2026-11-14 Sources: LlamaIndex docs · OpenAI API docs

LlamaIndex splits routing across two independent objects (the LLM and the embedding model) and most people only change the first one. That is why a migration that looks finished leaves half the traffic on the old provider, and half the bill on the old account.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

What you need before you start

• A gateway API key and base URL (https://aicomp.ai/v1)
• pip install llama-index-core llama-index-llms-openai-like llama-index-embeddings-openai-like
• The embedding model name the endpoint actually serves — not the one you assume it serves

Both objects, in one place

Since v0.10 the model classes live in separate packages, and the OpenAI-compatible variants carry the -like suffix. The distinction matters: the plain OpenAI class validates the model name against OpenAI's own list, so a gateway-served ID is rejected before any request is made.

import os
from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai_like import OpenAILikeEmbedding

# OpenAILike accepts any model string. The plain OpenAI class validates the
# name against OpenAI's own list and rejects gateway-served IDs.
Settings.llm = OpenAILike(
    model="gpt-5.6-luna",
    api_key=os.environ["GATEWAY_API_KEY"],
    api_base="https://aicomp.ai/v1",
    is_chat_model=True,          # use /chat/completions, not /completions
    temperature=0.1,
    max_tokens=1024,
)

# The embedding model is a SEPARATE object. Leaving it unset sends embeddings
# to whatever the default is -- usually straight to OpenAI, on a second bill.
Settings.embed_model = OpenAILikeEmbedding(
    model_name="text-embedding-3-small",
    api_key=os.environ["GATEWAY_API_KEY"],
    api_base="https://aicomp.ai/v1",
    embed_batch_size=64,
)

Two lines in that block are the ones people omit. is_chat_model=True selects /chat/completions over the legacy completions route. Without it you get a request the endpoint was never going to answer well. And Settings.embed_model is the one that decides where your indexing bill goes.

The cost shape nobody budgets for

Indexing and querying are not the same expense wearing different hats. Indexing is one large embedding bill paid once per rebuild; querying is one small embedding call plus one generation call, paid per question. A pipeline that looks cheap per query can be dominated by re-indexing if the document set churns.

from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
from llama_index.core import Settings

counter = TokenCountingHandler()
Settings.callback_manager = CallbackManager([counter])

# ... build the index, run queries ...

# Indexing and querying have completely different cost shapes:
#   indexing = one large embedding bill, paid once per rebuild
#   querying = one small embedding call + one generation call, paid per question
print("embedding tokens:", counter.total_embedding_token_count)
print("prompt tokens   :", counter.prompt_llm_token_count)
print("completion tok. :", counter.completion_llm_token_count)
print("resets:", counter.reset_counts())

Separating the two counters is the whole point. If the embedding token count is large relative to the prompt count, your lever is chunking and rebuild frequency, not model choice. Swapping to a cheaper generation model barely moves a bill that is mostly embeddings.

If you serve more than one tenant

Settings is global mutable state. Two concurrent requests that each assign Settings.llm will read each other's model, and the failure is intermittent enough to survive code review.

from llama_index.core import Settings, VectorStoreIndex

def build_and_query(docs, question, llm, embed_model):
    # Per-request scope instead of mutating the global Settings.
    return Settings.withLLM(llm, lambda: Settings.withEmbedModel(
        embed_model,
        lambda: VectorStoreIndex.from_documents(docs).as_query_engine().query(question),
    ))

# Global Settings is mutable process state. Two concurrent requests that each
# assign Settings.llm will read each other's model.

What it costs

Rates checked 2026-11-14. Gateway rates move with upstream promotions (verify the current number in your dashboard before committing to a budget).

Rates checked 2026-11-14. Embedding models are billed separately from generation and are usually priced per token at a fraction of generation rates, which is exactly why they are easy to overlook until the index is rebuilt.

LlamaIndex failures and what each one is actually telling you

When this is the wrong move

Cases where routing LlamaIndex through a gateway is the wrong move

How to confirm the change actually took effect

"I set it and nothing happened" is the most common failure here, and it is almost never the endpoint. It is usually the client: a desktop app that reads its own settings file rather than your shell environment, a process started before you exported the variable, or an SDK client instance built once at import time and reused after the config changed. Verify before you debug anything else.

Layer 1, is the endpoint reachable, and does your key work?

curl -s https://aicomp.ai/v1/models \
  -H "Authorization: Bearer $KEY" | head -c 400

A JSON list of model IDs means both are true. An Invalid token body means the endpoint answered but rejected the credential The URL is right, the key is not. An HTML page means something between you and the endpoint (a corporate proxy, a captive portal) answered instead, and no amount of config change will fix it from inside your app.

Layer 2 — who actually answered?

from openai import OpenAI
client = OpenAI(base_url="https://aicomp.ai/v1", api_key=KEY)

r = client.chat.completions.create(
    model="gpt-5.6-luna",
    messages=[{"role": "user", "content": "Reply with the single word: ok"}],
    max_tokens=8,
)
print(r.model)          # 回显的 model 是谁 —— 这是端点身份的唯一硬证据
print(r.usage)          # prompt_tokens / completion_tokens

The model field echoed back is the only hard evidence of identity. A request that silently fell through to a default provider still returns HTTP 200 and still returns sensible text, which is exactly why this layer exists. If the echoed model is not the one you asked for, you are talking to someone else.

Layer 3 — did it land in the usage log?

Reachability is not billing. Open the provider's usage view and look for the model ID you just called, on today's date. If the call succeeded but does not appear, you either hit a cached response or you are still on the old path. This layer is also the one that catches the case where two environments share a key and you are reading the wrong one.

Three ways this check lies to you

• A 200 that is not from the API. Proxies and error pages return 200 with HTML. Check that the body parses as JSON before you trust the status code.
• A stale process. Long-running servers and editors capture configuration at startup. Restart the process, not just the shell.
• A client built before the change. SDK clients snapshot base_url at construction. If yours is a module-level singleton, the new value never reaches it.

FAQ

Get API access