Skip to main content

Connect model providers

A provider defines an upstream API endpoint and its authentication method. Declare providers in spec.providers on the AIGateway resource and store credentials in Kubernetes Secrets.

Supported providers​

Providerschemacredentials.typeSecret key
OpenAIOpenAIAPIKeyapiKey
AnthropicAnthropicAnthropicAPIKeyapiKey
AWS BedrockAWSBedrockAWSCredentialsnone, uses IAM
Claude on AWS BedrockAWSAnthropicAWSCredentialsnone, uses IAM
Azure OpenAIAzureOpenAIAzureAPIKeyapiKey
Google Vertex AIGCPVertexAIGCPCredentialsservice_account.json
Google AI StudioGeminiAIStudioAPIKeyapiKey
Google Gemini APIGoogleGenerativeLanguageAPIKeyapiKey

For Azure deployments that authenticate by role, use credentials.type: AzureCredentials with region.

AWSBedrock uses the Bedrock Converse API. AWSAnthropic uses the Bedrock InvokeModel API with Anthropic's native request format and the awsanthropic pricing family.

GeminiAIStudio and GoogleGenerativeLanguage require APIKey. GCPVertexAI requires GCPCredentials, region, projectName, and secretRef. Resource validation rejects unsupported schema and credential type combinations.

Add a provider​

  1. Create the credential Secret:

    kubectl create secret generic openai-key \
      -n <NAMESPACE> \
      --from-literal=apiKey='<OPENAI_API_KEY>'

  2. Declare the provider on the AIGateway resource:

    aigateway.yaml
    spec:
      providers:
        - name: openai
          schema: OpenAI
          endpoint:
            hostname: api.openai.com
            port: 443
          credentials:
            type: APIKey
            secretRef:
              name: openai-key
              key: apiKey

  3. Add a route that references the provider. Resource validation requires every route to reference a declared provider:

    aigateway.yaml
    spec:
      routes:
        - name: gpt4o
          match:
            model: 'gpt-4o'
          backendRefs:
            - provider: openai

  4. Apply the resource and confirm the operator reconciled it:

    kubectl apply -f aigateway.yaml
    kubectl get aigw -n <NAMESPACE>

    The Providers column reports ready providers out of total. For detail, read the ProvidersReady status condition, whose message names the provider that failed:

    kubectl get aigw <NAME> -n <NAMESPACE> \
      -o jsonpath='{.status.conditions}' | jq .

  5. Send a request through the gateway to confirm the credential works end to end:

    curl -sk https://<GATEWAY_ENDPOINT>/v1/chat/completions \
      -H "Authorization: Bearer <TOKEN>" \
      -H "Content-Type: application/json" \
      -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "ping"}]}'

Provider specifics​

Anthropic​

providers:
  - name: anthropic
    schema: Anthropic
    endpoint:
      hostname: api.anthropic.com
      port: 443
    credentials:
      type: AnthropicAPIKey
      secretRef:
        name: anthropic-key
        key: apiKey

AWS Bedrock​

Assign the gateway proxy pods an IAM role through IAM roles for service accounts (IRSA) or EKS Pod Identity. Grant the role the required Bedrock actions.

providers:
  - name: bedrock
    schema: AWSBedrock
    endpoint:
      hostname: bedrock-runtime.us-east-1.amazonaws.com
      port: 443
    credentials:
      type: AWSCredentials
      region: us-east-1

Google Vertex AI​

Vertex AI uses a regional endpoint and Google service account key. Match credentials.region to the region prefix in the hostname.

Name the Secret data key service_account.json.

kubectl create secret generic vertex-sa \
  -n <NAMESPACE> \
  --from-file=service_account.json=<PATH_TO_SERVICE_ACCOUNT_JSON>
providers:
  - name: vertex
    schema: GCPVertexAI
    endpoint:
      hostname: us-central1-aiplatform.googleapis.com
      port: 443
    credentials:
      type: GCPCredentials
      region: us-central1
      projectName: <GCP_PROJECT_NAME>
      secretRef:
        name: vertex-sa

Service-account keys are the only supported Vertex credential. Workload Identity Federation is not supported.

Google AI Studio​

GeminiAIStudio uses Google AI Studio's OpenAI-compatible API and requires only an API key.

OpenAI-compatible providers​

Many providers expose an OpenAI-compatible API at a non-standard path. OpenRouter, for example, serves chat completions under /api/v1. Set schema: OpenAI and add pathPrefix; the prefix replaces the client's /v1 segment, so the upstream path becomes <PATH_PREFIX>/chat/completions.

spec:
  providers:
    - name: openrouter
      schema: OpenAI
      pathPrefix: /api/v1
      endpoint:
        hostname: openrouter.ai
        port: 443
      credentials:
        type: APIKey
        secretRef:
          name: openrouter-key
          key: apiKey

pathPrefix is valid only with schema: OpenAI. Providers that already carry their own prefix, such as GeminiAIStudio, must not set it.

Audit and journaling events identify an OpenAI-compatible provider by its configured name.

Name an OpenRouter provider exactly openrouter

The pricing service resolves OpenRouter rates through the provider name. Use openrouter so the gateway can price and admit its model slugs. Spend tracking supports one OpenRouter provider.

Rotate a credential​

Update the Secret in place. No resource edit and no pod restart is needed; the gateway picks up Secret changes on its own.

kubectl create secret generic openai-key \
  -n <NAMESPACE> \
  --from-literal=apiKey='<NEW_OPENAI_API_KEY>' \
  --dry-run=client -o yaml | kubectl apply -f -

Send a test request afterwards to confirm the new credential is in use.

Remove a provider​

Remove routes that reference the provider, remove the provider, and apply the resource.

kubectl apply -f aigateway.yaml
kubectl delete secret openai-key -n <NAMESPACE>

The gateway cleans up the infrastructure it created for that provider automatically.

Next steps​