Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
339 changes: 339 additions & 0 deletions product/observability/logs-export.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Logs Export"
description: "Easily access your Portkey logs data for further analysis and reporting"

Check warning on line 3 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L3

Did you really mean 'Portkey'?
---

<Info>
Expand All @@ -11,7 +11,7 @@
[Data Service](/changelog/data-service) must be enabled to use the Logs Export feature for Self Hosted Enterprise customers.
</Note>

At Portkey, we understand the importance of data analysis and reporting for businesses and teams. That's why we provide a comprehensive logs export feature that allows you to download your Portkey logs data in a **structured format**, enabling you to gain valuable insights into your LLM usage, performance, costs, and more.

Check warning on line 14 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L14

Did you really mean 'Portkey'?

## Exporting Logs In-App

Expand Down Expand Up @@ -63,7 +63,7 @@
- **Success**: Export completed successfully
- **Failure**: Export job failed. Click on the `Start Again` button to retry the job.

6. Clik on the **Start** button the dashboard to start the logs-export job

Check warning on line 66 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L66

Did you really mean 'Clik'?

7. For completed exports, click the **Download** button to get your logs data file. You can

Expand All @@ -72,7 +72,7 @@
</Frame>

<Note>
Currently we only support exporting 50k logs per job. For more help reach out to the Portkey team at [support@portkey.ai](mailto:support@portkey.ai)

Check warning on line 75 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L75

Did you really mean 'Portkey'?
</Note>

## Export File Details
Expand All @@ -93,6 +93,345 @@

You can analyze your API usage patterns, monitor performance, optimize costs, and make data-driven decisions for your business or team.

## Exporting logs via API

You can programmatically export logs using the Log Export API. This follows an asynchronous workflow:

1. **Create an export job** — Define your export parameters
2. **Start the export** — Begin processing the export
3. **Poll for status** — Check until the job completes
4. **Download the file** — Retrieve the JSONL file via signed URL

### Prerequisites

- API key with `logs.export` scope enabled
- For completion logs, the `completion` scope may also be required

### Step 1: Create a log export

Create an export job by specifying the time range and fields you want to export.

<CodeGroup>
```python Python
from portkey_ai import Portkey

Check warning on line 116 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L116

Did you really mean 'portkey_ai'?

client = Portkey(api_key="YOUR_PORTKEY_API_KEY")

export = client.logs.exports.create(
filters={
"created_at": {

Check warning on line 122 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L122

Did you really mean 'created_at'?
"gte": "2024-01-01T00:00:00Z",
"lte": "2024-01-31T23:59:59Z"
}
},
requested_data=[
"id",
"trace_id",

Check warning on line 129 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L129

Did you really mean 'trace_id'?
"created_at",
"request",
"response",
"ai_provider",

Check warning on line 133 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L133

Did you really mean 'ai_provider'?
"ai_model",

Check warning on line 134 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L134

Did you really mean 'ai_model'?
"request_tokens",

Check warning on line 135 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L135

Did you really mean 'request_tokens'?
"response_tokens",

Check warning on line 136 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L136

Did you really mean 'response_tokens'?
"total_tokens",

Check warning on line 137 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L137

Did you really mean 'total_tokens'?
"cost",
"response_time",

Check warning on line 139 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L139

Did you really mean 'response_time'?
"status_code"

Check warning on line 140 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L140

Did you really mean 'status_code'?
],
workspace_id="YOUR_WORKSPACE_ID" # Optional
)

export_id = export.id

Check warning on line 145 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L145

Did you really mean 'export_id'?
print(f"Export created with ID: {export_id}")

Check warning on line 146 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L146

Did you really mean 'export_id'?
```

```typescript Node.js
import Portkey from "portkey-ai";

const client = new Portkey({ apiKey: "YOUR_PORTKEY_API_KEY" });

const exportJob = await client.logs.exports.create({
filters: {
created_at: {

Check warning on line 156 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L156

Did you really mean 'created_at'?
gte: "2024-01-01T00:00:00Z",
lte: "2024-01-31T23:59:59Z",
},
},
requested_data: [
"id",
"trace_id",
"created_at",
"request",
"response",
"ai_provider",
"ai_model",
"request_tokens",
"response_tokens",
"total_tokens",
"cost",
"response_time",
"status_code",
],
workspace_id: "YOUR_WORKSPACE_ID", // Optional
});

const exportId = exportJob.id;
console.log(`Export created with ID: ${exportId}`);
```

```bash cURL
curl -X POST "https://api.portkey.ai/v1/logs/exports" \
-H "x-portkey-api-key: YOUR_PORTKEY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"filters": {
"created_at": {
"gte": "2024-01-01T00:00:00Z",
"lte": "2024-01-31T23:59:59Z"
}
},
"requested_data": [
"id",
"trace_id",
"created_at",
"request",
"response",
"ai_provider",
"ai_model",
"request_tokens",
"response_tokens",
"total_tokens",
"cost",
"response_time",
"status_code"
]
}'
```
</CodeGroup>

### Step 2: Start the export

Once the export job is created, start processing it.

<CodeGroup>
```python Python
client.logs.exports.start(export_id)
print("Export job started")
```

```typescript Node.js
await client.logs.exports.start(exportId);
console.log("Export job started");
```

```bash cURL
curl -X POST "https://api.portkey.ai/v1/logs/exports/{export_id}/start" \
-H "x-portkey-api-key: YOUR_PORTKEY_API_KEY"
```
</CodeGroup>

### Step 3: Poll for completion

Check the export status until it returns `success`.

<CodeGroup>
```python Python
import time

while True:
status = client.logs.exports.retrieve(export_id)
print(f"Status: {status.status}")

Check warning on line 244 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L244

'status' is repeated!

if status.status == "success":
print("Export completed!")
break
elif status.status == "failure":
print(f"Export failed: {status.error}")
break

time.sleep(5) # Wait 5 seconds before polling again
```

```typescript Node.js
const pollForCompletion = async (exportId: string) => {
while (true) {
const status = await client.logs.exports.retrieve(exportId);
console.log(`Status: ${status.status}`);

if (status.status === "success") {
console.log("Export completed!");
break;
} else if (status.status === "failure") {
console.log(`Export failed: ${status.error}`);
break;
}

await new Promise((resolve) => setTimeout(resolve, 5000)); // Wait 5 seconds
}
};

await pollForCompletion(exportId);
```

```bash cURL
curl "https://api.portkey.ai/v1/logs/exports/{export_id}" \
-H "x-portkey-api-key: YOUR_PORTKEY_API_KEY"
```
</CodeGroup>

### Step 4: Download the export

Once the export is complete, retrieve the signed download URL.

<CodeGroup>
```python Python
download_response = client.logs.exports.download(export_id)

Check warning on line 289 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L289

Did you really mean 'download_response'?
download_url = download_response.url

Check warning on line 290 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L290

Did you really mean 'download_url'?

print(f"Download URL: {download_url}")

Check warning on line 292 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L292

Did you really mean 'download_url'?

# Download the file
import requests

response = requests.get(download_url)
with open("logs_export.jsonl", "wb") as f:

Check warning on line 298 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L298

Did you really mean 'wb'?
f.write(response.content)

print("Logs exported to logs_export.jsonl")
```

```typescript Node.js
const downloadResponse = await client.logs.exports.download(exportId);
const downloadUrl = downloadResponse.url;

console.log(`Download URL: ${downloadUrl}`);

// Download the file using fetch
const fileResponse = await fetch(downloadUrl);
const fileContent = await fileResponse.text();

// Save to file (Node.js)
import fs from "fs";
fs.writeFileSync("logs_export.jsonl", fileContent);

console.log("Logs exported to logs_export.jsonl");
```

```bash cURL
# Get the download URL
curl "https://api.portkey.ai/v1/logs/exports/{export_id}/download" \
-H "x-portkey-api-key: YOUR_PORTKEY_API_KEY"

# Then download using the returned URL
curl -o logs_export.jsonl "{signed_download_url}"
```
</CodeGroup>

### Complete example

Here's a complete script that creates, starts, monitors, and downloads a log export:

<CodeGroup>
```python Python
import time
import requests
from portkey_ai import Portkey

Check warning on line 339 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L339

Did you really mean 'portkey_ai'?

client = Portkey(api_key="YOUR_PORTKEY_API_KEY")

# Step 1: Create export
export = client.logs.exports.create(
filters={
"created_at": {
"gte": "2024-01-01T00:00:00Z",
"lte": "2024-01-31T23:59:59Z"
}
},
requested_data=["id", "created_at", "ai_model", "total_tokens", "cost"]

Check warning on line 351 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L351

Did you really mean 'ai_model'?

Check warning on line 351 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L351

Did you really mean 'total_tokens'?
)
export_id = export.id
print(f"Created export: {export_id}")

# Step 2: Start export
client.logs.exports.start(export_id)
print("Export started")

# Step 3: Poll until complete
while True:
status = client.logs.exports.retrieve(export_id)
if status.status == "success":

Check warning on line 363 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L363

'status' is repeated!
break
elif status.status == "failure":

Check warning on line 365 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L365

Did you really mean 'elif'?
raise Exception(f"Export failed: {status.error}")
time.sleep(5)

# Step 4: Download
download_url = client.logs.exports.download(export_id).url
response = requests.get(download_url)
with open("logs_export.jsonl", "wb") as f:

Check warning on line 372 in product/observability/logs-export.mdx

View check run for this annotation

Mintlify / Mintlify Validation (portkey-docs) - vale-spellcheck

product/observability/logs-export.mdx#L372

Did you really mean 'wb'?
f.write(response.content)

print("Export complete! File saved to logs_export.jsonl")
```

```typescript Node.js
import Portkey from "portkey-ai";
import fs from "fs";

const client = new Portkey({ apiKey: "YOUR_PORTKEY_API_KEY" });

async function exportLogs() {
// Step 1: Create export
const exportJob = await client.logs.exports.create({
filters: {
created_at: {
gte: "2024-01-01T00:00:00Z",
lte: "2024-01-31T23:59:59Z",
},
},
requested_data: ["id", "created_at", "ai_model", "total_tokens", "cost"],
});
const exportId = exportJob.id;
console.log(`Created export: ${exportId}`);

// Step 2: Start export
await client.logs.exports.start(exportId);
console.log("Export started");

// Step 3: Poll until complete
while (true) {
const status = await client.logs.exports.retrieve(exportId);
if (status.status === "success") break;
if (status.status === "failure") throw new Error(`Export failed: ${status.error}`);
await new Promise((r) => setTimeout(r, 5000));
}

// Step 4: Download
const downloadUrl = (await client.logs.exports.download(exportId)).url;
const response = await fetch(downloadUrl);
const content = await response.text();
fs.writeFileSync("logs_export.jsonl", content);

console.log("Export complete! File saved to logs_export.jsonl");
}

exportLogs();
```
</CodeGroup>

### API reference

For detailed API specifications, see the [Log Exports API reference](/api-reference/admin-api/data-plane/logs/log-exports-beta/create-a-log-export).

| Endpoint | Description |
|----------|-------------|
| `POST /v1/logs/exports` | Create a new export job |
| `POST /v1/logs/exports/{id}/start` | Start processing an export |
| `GET /v1/logs/exports/{id}` | Retrieve export status |
| `GET /v1/logs/exports/{id}/download` | Get signed download URL |
| `GET /v1/logs/{id}` | Fetch a single log entry |

## Support

If you have any questions or need assistance with the logs export feature, reach out to the Portkey team at [support@portkey.ai](mailto:support@portkey.ai) or join our [Discord community](https://portkey.ai/community).
Expand Down
Loading