ibm-watsonx-ai: Initial IAM token POST times out behind HTTP proxy unless a custom httpx.Client(proxy=..., trust_env=False) is injected

[kmizutalw]

Bug Report

Summary

When running behind a corporate HTTP proxy (no auth, no TLS inspection), initializing ibm-watsonx-ai’s APIClient(Credentials(...)) consistently times out during the initial IAM token request (POST https://iam.cloud.ibm.com/identity/token).
The same request succeeds instantly with curl and with a manually created httpx.Client(proxy=..., trust_env=False). Injecting that client into APIClient(httpx_client=...) makes the SDK work reliably.
This strongly suggests that ibm-watsonx-ai’s internal httpx client is not reliably honoring proxy environment variables and/or NO_PROXY logic in some environments, leading to a direct (non‑proxy) egress attempt → ConnectTimeout.

Environment

Runtime: Python 3.12 on Linux (host: HOST_A, user: USER_A)
Network: Corporate HTTP proxy, no authentication, no TLS interception

Proxy FQDN: proxy.example.co.jp:8080
iam.cloud.ibm.com and .ml.cloud.ibm.com are allowed

Proxy env vars: HTTP_PROXY/HTTPS_PROXY set to http://proxy.example.co.jp:8080
NO_PROXY contains only internal hosts (e.g., localhost,127.0.0.1,::1,.svc,.cluster.local)
Observed with: ibm-watsonx-ai used directly and via LangChain wrappers
Models: ibm/granite-4-h-small (also reproduced when Granite 3 was tested; Granite 3 is deprecated)

Note: In this environment, the same IAM token POST works via curl and via explicit httpx.Client(proxy=...). The failure occurs when the SDK constructs its own httpx client during APIClient initialization.

Steps to Reproduce (failure)

# failure_repro.py
import os
from ibm_watsonx_ai import APIClient, Credentials

# Env set before launch:
#   HTTPS_PROXY=http://proxy.example.co.jp:8080
#   HTTP_PROXY=http://proxy.example.co.jp:8080
#   NO_PROXY=localhost,127.0.0.1,::1,.svc,.cluster.local
#   WATSONX_APIKEY=***
#   WATSONX_URL=https://jp-tok.ml.cloud.ibm.com

creds = Credentials(
    api_key=os.getenv("WATSONX_APIKEY"),
    url=os.getenv("WATSONX_URL"),
)

# This line – during APIClient init – triggers IAM token POST internally
client = APIClient(
    creds,
    verify=True,
    request_timeout=60,  # even long timeouts do not help in this environment
)

# Expected: returns with a valid client
# Actual: httpx.ConnectTimeout raised during token retrieval

Actual result (stack excerpt):

httpx.ConnectTimeout: timed out
  ...
  File ".../ibm_watsonx_ai/utils/auth/iam_auth.py", line 85, in _generate_token
    response = self._api_client.httpx_client.post(...)
  ...
  File ".../httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: timed out

Control Tests (success)

A) curl via proxy (POST /identity/token) — responds immediately (400 for dummy key):

curl -x http://proxy.example.co.jp:8080 \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -d "grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=DUMMY" \
     https://iam.cloud.ibm.com/identity/token
# → 400 with JSON error (as expected for dummy key)

B) httpx (explicit proxy) — responds immediately (400 for dummy key):

import httpx
with httpx.Client(proxy="http://proxy.example.co.jp:8080", timeout=15, trust_env=False) as c:
    r = c.post(
        "https://iam.cloud.ibm.com/identity/token",
        headers={"Content-Type":"application/x-www-form-urlencoded"},
        data={"grant_type":"urn:ibm:params:oauth:grant-type:apikey","apikey":"DUMMY"},
    )
    print(r.status_code)  # → 400 (expected for dummy key)

Conclusion: Network path & proxy are fine; issue is SDK’s internal client initialization path.

Workaround (reliable)

Inject a pre‑configured httpx client with forced proxy and trust_env=False into APIClient:

# success_workaround.py
import os, httpx
from ibm_watsonx_ai import APIClient, Credentials
from ibm_watsonx_ai.foundation_models import ModelInference

proxy_url = "http://proxy.example.co.jp:8080"

httpx_client = httpx.Client(
    proxy=proxy_url,     # important: single 'proxy=' param (httpx modern API)
    timeout=60,
    trust_env=False,     # do NOT rely on env var parsing; always use this proxy
)

client = APIClient(
    Credentials(api_key=os.getenv("WATSONX_APIKEY"), url=os.getenv("WATSONX_URL")),
    verify=True,
    httpx_client=httpx_client,    # ← injected client
    request_timeout=60,
)

mi = ModelInference(
    model_id="ibm/granite-4-h-small",
    project_id=os.getenv("WATSONX_PROJECT_ID"),
    api_client=client,
    params={"decoding_method":"greedy","max_new_tokens":64,"temperature":0.2},
)

print(mi.generate_text("Return 'OK' only."))
# → Works consistently behind the proxy

This workaround also lets LangChain users succeed by wrapping ModelInference in a Runnable / custom LLM class, while keeping LangChain features (prompt templating, chains, etc.).

Expected Behavior

• APIClient(Credentials(...)) should be able to reliably obtain the IAM token behind a standard HTTP proxy without requiring a custom httpx client injection.
• Proxy environment variables (HTTP(S)_PROXY, NO_PROXY) should be interpreted consistently, and the initial IAM POST should honor the configured proxy.

Actual Behavior

• In some environments, APIClient’s internal httpx client attempts to connect without the proxy (or misinterprets NO_PROXY), causing httpx.ConnectTimeout during the initial IAM token POST.
• The same request succeeds via curl and via explicit httpx.Client(proxy=..., trust_env=False).

Additional Context

• Proxy requires no authentication and performs no TLS interception.
• Problem reproduced on a clean virtualenv.
• Works fine once httpx_client with proxy= and trust_env=False is injected.
• LangChain users hit the same issue via ChatWatsonx / WatsonxLLM because those wrappers internally construct ibm-watsonx-ai clients.

Versions

python 3.12.3
ibm_watsonx_ai 1.5.2
httpx 0.28.1
langchain 1.2.10
langchain-ibm 1.0.4

Logs (sanitized excerpt)

httpx.ConnectTimeout: timed out
  ...
  File ".../ibm_watsonx_ai/utils/auth/iam_auth.py", line 85, in _generate_token
    response = self._api_client.httpx_client.post(...)
  ...
  File ".../httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: timed out

If maintainers need additional traces, I can provide an HTTPX_LOG_LEVEL=trace capture showing whether CONNECT is attempted via the proxy during the IAM token POST. Happy to run any further diagnostics.