Using the Lynio AI Gateway & AI Compute Credits

The Lynio AI Gateway provides a zero-egress, high-performance OpenAI-compatible API layer powered by regionally hosted GPU accelerator nodes. It enables cloud tenants to run private Large Language Model (LLM) inferencing with strict regional data residency, real-time Server-Sent Events (SSE) streaming with reasoning trace visibility, asynchronous batch job processing at a 50% discount, and transparent AI Compute Credit metering.

Key Capabilities & Highlights

• OpenAI Drop-in Compatibility: Works out of the box with standard OpenAI SDKs (openai Python package, OpenAI Node.js SDK, LangChain, LlamaIndex, LiteLLM, Cursor, and vLLM clients).
• Real-Time Streaming & Deep Reasoning: Stream tokens live word-by-word with separate, inspectable reasoning/thinking traces (reasoning_content).
• Asynchronous Batch Processing (50% Discount): Offload bulk document processing and agent workflows to run during idle GPU cycles at half the credit cost.
• Priority-Aware GPU Scheduler: Core-side Least-Connections load balancer ensures interactive chat streams always receive 100% immediate priority over background batch jobs.
• Dynamic Model Multipliers: Transparent compute-weighted pricing reflecting the GPU compute requirements configured for each model in your selected region.
• Fair-Use Pacing Protections: 5-hour anti-burst and weekly pacing rate limits protect against accidental quota exhaustion.
• Interactive AI Playground: Discover regional models, test prompts, tune hyperparameters, optimize context windows, and track background batch jobs directly in the Lynio Console.

Understanding AI Compute Credits

How are AI Compute Credits calculated?

When interacting with an AI model, text is broken down into tokens (roughly 3 to 4 characters per token). Smaller and larger models generate approximately the same number of raw text tokens for a given prompt and response. However, larger models require substantially more GPU memory (VRAM), compute power, and energy.

To ensure fair compute accounting, Lynio uses AI Compute Credits:

1. Interactive & Streaming Completions

$\text{AI Compute Credits} = \text{round}\big(\text{Total Raw Tokens} \times \text{Model Multiplier}\big)$

2. Tiered Asynchronous Batch Discounts

Background batch jobs run during available GPU cycles and receive tiered credit discounts based on priority:

• Low Priority (0): 60% Discount ($\text{Credits} = \text{round}(\text{Tokens} \times \text{Multiplier} \times 0.40)$) — Maximum savings for non-urgent bulk workloads.
• Normal Priority (10): 50% Discount ($\text{Credits} = \text{round}(\text{Tokens} \times \text{Multiplier} \times 0.50)$) — Standard balanced batch processing.
• High Priority (20): 40% Discount ($\text{Credits} = \text{round}(\text{Tokens} \times \text{Multiplier} \times 0.60)$) — Accelerated queue dispatching.

Discovering Regional Models & Multipliers

Because each region hosts its own library of GPU-accelerated models and multipliers:

• View all available models and their active credit multipliers in the Console > AI Playground model selector.
• Query them programmatically via GET /api/v1/ai/models.
• Every chat completion response provides the exact multiplier and credits consumed in the x-lynio-model-multiplier and x-lynio-credits-used HTTP response headers.

GPU Scheduling, Concurrency & Weighted Fair Queueing

To ensure high availability, interactive responsiveness, and fair batch throughput across multi-node clusters:

1. Interactive Priority Preemption: Interactive streaming and chat requests always receive 100% immediate priority over background batch jobs.
2. Least-Connections Load Balancing: Interactive requests are routed automatically to the GPU node with the lowest active workload in your region.
3. Admission Queue with 30s Wait: If all GPU nodes in a region are operating at maximum interactive capacity, requests wait in Core's queue for up to 30 seconds before returning HTTP 429 Too Many Requests (Retry-After: 10).
4. Weighted Fair Queueing (WFQ 3:2:1): When batch queues contain a mix of priorities, jobs are dispatched according to a balanced 3 High : 2 Normal : 1 Low ratio. This accelerates high-priority tasks while mathematically preventing starvation of normal and low-priority jobs.

Rate Limiting & Pacing Windows

To safeguard tenant budgets and ensure smooth GPU availability throughout the billing cycle:

1. 5-Hour Anti-Burst Limit (10% of monthly quota): Protects against runaway scripts or infinite loops by limiting burst consumption in a rolling 5-hour window.
2. Weekly Pacing Limit (35% of monthly quota): Ensures your credit balance is paced evenly across the billing cycle.
3. Monthly Hard Quota: The total monthly AI Compute Credit allowance allocated to your tenant (e.g. 1,000,000 Credits/mo).

When a rolling limit is reached, the API returns an HTTP 429 status code indicating the cooldown duration and refresh time.

Authentication & Base URL

All requests to the AI Gateway require authentication using a valid Lynio API Key or JWT Bearer Token.

• Base URLs:
  • https://api.lynio.cloud/api/v1/ai
  • https://api.lynio.cloud/api/v1/ai/v1 (supported for OpenAI SDK drop-in compatibility)
• Headers:
  • Authorization: Bearer <YOUR_LYNIO_API_KEY>
  • Content-Type: application/json

API Reference

1. Discover Models & Multipliers (GET /models)

Returns the list of AI models currently available in the region alongside their active credit multipliers.

GET /api/v1/ai/models

Response Example

{
  "object": "list",
  "data": [
    {
      "id": "<model-tag>",
      "object": "model",
      "created": 1786600000,
      "owned_by": "ollama",
      "credit_multiplier": 1.0
    }
  ]
}

2. Interactive Chat Completions (POST /chat/completions)

Submits a prompt or conversation history for synchronous or Server-Sent Events (SSE) streaming execution.

POST /api/v1/ai/chat/completions

Request Parameters

Streaming SSE Chunks with Reasoning

When stream: true is requested, the server streams chunks using standard SSE format:

1. Thinking / Reasoning Phase:

   data: {"choices":[{"delta":{"reasoning_content":"Analyzing the request..."}}]}
   

2. Content Generation Phase:

   data: {"choices":[{"delta":{"content":"Here is the solution..."}}]}
   

3. Completion & Usage:

   data: {"choices":[{"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":25,"completion_tokens":85,"total_tokens":110}}
   data: [DONE]
   

3. Asynchronous Batch Jobs (POST /jobs & POST /v1/batches)

Submits an asynchronous inference task to be processed on the background queue with a 50% discount on compute credits.

POST /api/v1/ai/jobs

Request Parameters

Request Example (cURL)

curl -X POST https://api.lynio.cloud/api/v1/ai/jobs \
  -H "Authorization: Bearer $LYNIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-tag>",
    "region": "NL-Lynio-MSP1",
    "prompt": "Summarize this 50-page technical specification...",
    "temperature": 0.3,
    "webhook_url": "https://api.example.com/webhooks/ai-batch-complete"
  }'

Response (HTTP 202 Accepted)

{
  "id": "773e7098-9411-4475-8025-0676451ec9ea",
  "job_id": "773e7098-9411-4475-8025-0676451ec9ea",
  "object": "ai.job",
  "status": "queued",
  "model": "<model-tag>",
  "region": "NL-Lynio-MSP1",
  "batch_discount": 0.5,
  "created_at": "2026-08-15T08:29:53Z"
}

4. Inspect Batch Job Status & Output (GET /jobs/:id)

Fetches the live status, execution runtime, generated output, reasoning steps, and token/credit breakdown for a batch job.

GET /api/v1/ai/jobs/773e7098-9411-4475-8025-0676451ec9ea

Response Example (Completed)

{
  "id": "773e7098-9411-4475-8025-0676451ec9ea",
  "object": "ai.job",
  "status": "completed",
  "model": "<model-tag>",
  "region": "NL-Lynio-MSP1",
  "response_payload": {
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "Summary of technical specification:\n1. Architecture...",
          "reasoning_content": "1. Analyze system requirements..."
        },
        "finish_reason": "stop"
      }
    ]
  },
  "prompt_tokens": 1200,
  "completion_tokens": 433,
  "total_tokens": 1633,
  "model_multiplier": 1.0,
  "batch_discount": 0.5,
  "total_credits": 817,
  "webhook_status": "delivered",
  "started_at": "2026-08-15T08:29:55Z",
  "completed_at": "2026-08-15T08:30:37Z",
  "created_at": "2026-08-15T08:29:53Z"
}

5. Cancel or Delete Batch Jobs

• Cancel a Queued Job:

  POST /api/v1/ai/jobs/:id/cancel
  

• Delete a Job Record:

  DELETE /api/v1/ai/jobs/:id
  

6. Check Token Usage & Limits (GET /usage)

Returns monthly credit consumption, token statistics, and live pacing limits for your tenant.

GET /api/v1/ai/usage?region=NL-Lynio-MSP1

SDK Integration Examples

Python: Real-Time Streaming with Reasoning

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.lynio.cloud/api/v1/ai/v1",
    api_key=os.environ.get("LYNIO_API_KEY"),
)

stream = client.chat.completions.create(
    model="<model-tag>",
    messages=[
        {"role": "system", "content": "You are a cloud architect assistant."},
        {"role": "user", "content": "Explain multi-region VPC peering."},
    ],
    stream=True,
    extra_body={"region": "NL-Lynio-MSP1"}
)

for chunk in stream:
    delta = chunk.choices[0].delta
    # Check for reasoning/thinking tokens
    reasoning = getattr(delta, "reasoning_content", None) or getattr(delta, "thinking", None)
    if reasoning:
        print(f"[Thinking]: {reasoning}", end="", flush=True)
    if delta.content:
        print(delta.content, end="", flush=True)

Python: Asynchronous Batch Job Submission & Polling

import os
import time
import requests

API_KEY = os.environ.get("LYNIO_API_KEY")
BASE_URL = "https://api.lynio.cloud/api/v1/ai"

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

# 1. Submit Batch Job (50% Discount)
submit_resp = requests.post(f"{BASE_URL}/jobs", headers=headers, json={
    "model": "<model-tag>",
    "region": "NL-Lynio-MSP1",
    "prompt": "Extract all key entities and summarize the following logs...",
    "temperature": 0.2,
})
job = submit_resp.json()
job_id = job["id"]
print(f"Job submitted successfully. Job ID: {job_id} (Status: {job['status']})")

# 2. Poll for Completion
while True:
    status_resp = requests.get(f"{BASE_URL}/jobs/{job_id}", headers=headers).json()
    status = status_resp["status"]
    print(f"Checking Job {job_id}... Status: {status}")

    if status == "completed":
        message = status_resp["response_payload"]["choices"][0]["message"]
        print("\n--- Output ---")
        print(message["content"])
        print(f"\nTotal Credits Billed (50% OFF): {status_resp['total_credits']} credits")
        break
    elif status in ["failed", "cancelled"]:
        print(f"Job terminated with status: {status}. Error: {status_resp.get('error_message')}")
        break

    time.sleep(3)

Best Practices for Optimizing AI Credit Usage

1. Use Batch Processing for Non-Interactive Tasks: Route bulk data pipelines, document summarization, and agent swarms through POST /api/v1/ai/jobs to automatically receive the 50% credit discount.
2. Check Multipliers in your Region: View the credit multiplier for each model in the AI Playground dropdown or via GET /api/v1/ai/models to choose the most cost-effective model for your task.
3. Utilize Context Summarization: In multi-turn chat applications, summarize older history or use sliding context windows to avoid resending large message blocks repeatedly.
4. Implement Webhook Callbacks: Provide a webhook_url when submitting batch jobs to receive push notifications rather than polling the status endpoint.
5. Handle HTTP 429 with Retry-After: Respect the Retry-After: 10 header on capacity limit responses to ensure clean back-off during peak cluster load.

Troubleshooting & Status Codes