Openflow Sight

A powerful Streamlit-based log analysis tool for searching, filtering, and analyzing Snowflake Openflow telemetry data. Quickly identify errors, warnings and other events, trace issues across processors, and export findings for further investigation.

---

Features

Runtime Selection & Filtering

• Multi-Runtime Support: Select one or multiple runtimes to analyze simultaneously
• Internal Runtime Toggle: Show or hide internal system runtimes (prefixed with runtime-)
• Processor Filtering: Filter logs by specific processor(s) — displays the top 20 processors with the most activity
• Log Level Filtering: Filter by severity level:
  • ERROR — Critical issues requiring immediate attention
  • WARN — Potential problems that may need investigation
  • INFO — Informational messages for context

---

Advanced Search Capabilities

• Multi-Term Search: Enter comma-separated terms to search for multiple keywords at once
  • Example: timeout, connection, failed finds logs containing any of these terms
• Search Modes:
  • Contains (default): Case-insensitive partial matching with multi-term support
  • Regex: Full regular expression support for complex pattern matching
  • Exact Match: Find logs with an exact message match

---

Log Frequency Graph & Timeline

• Interactive Timeline Histogram: Visualize log distribution over time with Plotly
  • Stacked bars showing ERROR, WARN, and INFO logs by severity level
  • Color-coded by log level (red for errors, orange for warnings, blue for info)
  • Hover for detailed counts at each time bucket
  • Zoom and pan to focus on specific time periods
• Configurable Time Buckets: Aggregate data by:
  • 1 minute
  • 5 minutes
  • 15 minutes
  • 1 hour

---

Flexible Time Window Selection

• Preset Time Windows:
  • Last 1 hour
  • Last 6 hours
  • Last 24 hours
  • Last 7 days
• Custom Time Range: Define exact start and end dates/times (UTC) for precise analysis
• Time Range Slider: Fine-tune the selected window with an interactive slider

---

Individual Logs View

• AG Grid Table: Enterprise-grade data grid with Excel-like functionality
  • Column Sorting: Click column headers to sort ascending/descending
  • Column Filtering: Filter each column independently
  • Column Resizing: Drag column borders to resize
  • Pagination: Navigate through results (50 rows per page)
  • Multi-Row Selection: Checkbox selection for bulk operations
  • Sidebar Panel: Access additional filtering and column visibility options
• Real-Time Metrics Bar: Displays:
  • Total log count
  • Error count
  • Warning count
  • Info count
  • Unique processor count

---

Grouped Patterns View

• Automatic Log Clustering: Similar logs are automatically grouped using fuzzy matching
• Pattern Analysis: Each group shows:
  • Total occurrences
  • First and last seen timestamps
  • Affected processors
  • Sample log message
• Drill-Down Capability: Expand any group to view all individual occurrences
• Normalized Matching: Dynamic values (timestamps, UUIDs, line numbers) are normalized for accurate grouping

---

Export to CSV

• One-Click Export: Download filtered results as a CSV file
• Smart Naming: Exported files include runtime name and timestamp
  • Example: logs_runtime-prod_20260120_143022.csv
• Full Data Export: Exports all columns (Timestamp, Processor, Level, Message)

---

Technology Stack

Component	Technology
Frontend	Streamlit with custom CSS theming
Data Source	Snowflake via Snowpark
Visualization	Plotly (timeline), Altair
Data Grid	AG Grid (st-aggrid)
Fuzzy Matching	RapidFuzz
Data Processing	Pandas
Caching	Streamlit (60-second TTL)

---

Prerequisites

• Python 3.7+
• Snowflake account with access to Openflow telemetry data
• Snowflake Streamlit environment

---

Data Requirements

The application queries telemetry data from:

snowflake.telemetry.events

Update the EVENTS_TABLE constant in OpenflowSight.py if your table path differs.

---

Usage Workflow

Quick Investigation

1. Select Runtime(s) in the sidebar
2. Choose Log Level(s) (ERROR, WARN, INFO)
3. Set Time Window (preset or custom)
4. Review Timeline to identify activity spikes
5. Drill into Details using Individual Logs or Grouped Patterns tabs

Deep Dive Analysis

1. Use Multi-Term Search to find related logs (e.g., timeout, connection)
2. Filter by Processor to isolate specific components
3. Switch to Grouped Patterns to identify recurring issues
4. Export to CSV for offline analysis or sharing

Pattern Recognition

1. Open Grouped Patterns tab to see clustered logs
2. Review occurrence counts to prioritize high-frequency issues
3. Check affected processors to understand scope
4. Expand groups to trace individual occurrences

---

Customization

Theme Customization

The app uses a professional light theme. Modify the custom_theme CSS in OpenflowSight.py:

custom_theme = """
<style>
    .stApp {
        background-color: #F0F2F6;  /* Main background */
    }
    /* Add your custom styles here */
</style>
"""

Query Customization

Modify SQL queries to match your telemetry schema or add custom filters.

---

Performance Tips

• Use Specific Time Windows: Smaller windows = faster queries
• Filter by Processor: Reduces data volume significantly
• Clear Cache: Click "Clear Cache" in sidebar if data appears stale
• Pagination: AG Grid loads 50 rows per page for performance

---

Troubleshooting

Issue	Solution
Snowflake Connection Error	Verify EVENTS_TABLE path and permissions
No Data Displayed	Check time window and runtime filters
Slow Performance	Use shorter time windows or filter by processor
Stale Data	Click "Clear Cache" or "Refresh Data" button

---

Dependencies

streamlit
snowflake-snowpark-python
pandas
altair
plotly
st-aggrid
rapidfuzz

---

Built with Streamlit for Snowflake