Skip to main content

Rate Limiter Routing

Enforce usage quotas and route requests accordingly. Block users who exceed their limits while providing clear error messages.

Use Case​

  • Prevent API abuse
  • Manage costs with usage quotas
  • Provide clear feedback on rate limits

Configuration​

{
  "model": "router/dynamic",
  "router": {
    "type": "conditional",
    "pre_request": [
      {
        "name": "daily_user_limit",
        "type": "rate_limiter",
        "limit": 2,
        "period": "hour",
        "entity": "user_id",
        "target": "requests"
      }
    ],
    "routes": [
      {
        "conditions": {
          "all": [
            {
              "pre_request.daily_user_limit.allowed": {
                "$eq": true
              }
            }
          ]
        },
        "name": "main_requests",
        "targets": {
          "$any": [
            "anthropic/*"
          ],
          "sort_by": "price",
          "sort_order": "min"
        }
      },
      {
        "name": "other_requests",
        "targets": {
          "$any": [
            "openai/gpt-4.1-nano"
          ],
          "sort_by": "price",
          "sort_order": "min"
        }
      }
    ]
  }
}

How It Works​

  1. Rate Limiter: Checks if user has exceeded 2 requests per hour
  2. Main Requests: Routes allowed requests to any Anthropic model, sorted by lowest price
  3. Other Requests: Routes remaining requests to GPT-4.1-nano for cost optimization

Variables Used​

  • pre_request.daily_user_limit.allowed: Result of rate limit check (true if within limit)

Customization​

  • Adjust limits (daily, weekly, monthly)
  • Different limits per user tier
  • Custom error messages
  • Graceful degradation instead of blocking