SQL: Query topics

modules/reference/pages/sql/sql-statements/create-table.adoc

@@ -31,8 +31,8 @@ WITH (option = 'value' [, ...]);
 
 |`schema_subject`
 |STRING
-|No
-|Schema Registry subject name to use for deserializing topic data.
+|Yes
+|Schema Registry subject name to use for deserializing topic data. Redpanda SQL requires a schema to query a topic.
 
 |`schema_lookup_policy`
 |STRING
@@ -51,17 +51,25 @@ a|How to handle records that fail deserialization.
 |`struct_mapping_policy`
 |STRING
 |No
-a|How to map nested structures to SQL columns.
+a|How to map nested structures from the topic schema to SQL columns.
 
-* `JSON` (default): Stores nested data as JSON.
-* `FLATTEN`: Expands nested fields into top-level columns.
-* `COMPOUND`: Maps to ROW types.
-* `VARIANT`: Stores as a variant type.
+* `COMPOUND` (default): Maps each nested structure to a SQL xref:reference:sql/sql-data-types/row.adoc[ROW] value with named fields, queryable using `(column).field_name` syntax. Cyclic types are not supported in `COMPOUND` mode — use `JSON` for recursive schemas.
+* `JSON`: Stores each nested structure as a JSON value. Required for recursive (cyclic) types.
 
 |`output_schema_message_full_name`
 |STRING
 |No
 |Full Protobuf message name. Required when the schema contains multiple message definitions.
+
+|`confluent_wire_protocol`
+|STRING
+|No
+a|Whether records on the topic are encoded with the https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#wire-format[Confluent Schema Registry wire format^] (a magic byte followed by a 4-byte schema ID before the payload).
+
+* `'true'` (default): Records carry the Confluent wire-format prefix. Use this for topics whose values were produced by a Schema-Registry-aware client.
+* `'false'`: Records are raw Protobuf or Avro without the wire-format prefix.
+
+Only valid when `schema_lookup_policy = 'LATEST'`.
 |===
 
 == Examples
@@ -73,20 +81,23 @@ Map the `transactions` topic to a table through `default_redpanda_catalog`:
 [source,sql]
 ----
 CREATE TABLE default_redpanda_catalog=>transactions
-WITH (topic = 'transactions');
+WITH (
+  topic = 'transactions',
+  schema_subject = 'transactions-value'
+);
 ----
 
-=== Specify a Schema Registry subject
+=== Create a table from a multi-message Protobuf schema
 
-Map a topic and specify the Schema Registry subject:
+When the Protobuf schema for the topic defines more than one message, specify the message to use with `output_schema_message_full_name`:
 
 [source,sql]
 ----
-CREATE TABLE default_redpanda_catalog=>user_events
+CREATE TABLE default_redpanda_catalog=>orders
 WITH (
-  topic = 'user-events',
-  schema_subject = 'user-events-value',
-  schema_lookup_policy = 'LATEST'
+  topic = 'orders',
+  schema_subject = 'orders-value',
+  output_schema_message_full_name = 'com.example.orders.Order'
 );
 ----
 

modules/sql/pages/query-data/index.adoc

@@ -0,0 +1,3 @@
+= Query data
+:description: Query live and historical data in your Redpanda topics using standard PostgreSQL syntax.
+:page-layout: index

modules/sql/pages/query-data/query-streaming-topics.adoc

@@ -0,0 +1,80 @@
+= Query streaming topics
+:description: Map a Redpanda topic to a SQL table and run analytical queries directly against live streaming data.
+:page-topic-type: how-to
+:personas: app_developer, data_engineer
+:learning-objective-1: Map a Redpanda topic to a SQL table using the default Redpanda catalog
+:learning-objective-2: Run analytical SQL queries against live topic data
+
+Map a Redpanda topic to a SQL table to run analytical queries directly against live streaming data without building ETL pipelines. Redpanda SQL reads each record's fields from the topic's registered schema.
+
+To extend queries past your Redpanda retention window by reading from Iceberg-translated history, see xref:sql:query-data/query-iceberg-topics.adoc[].
+
+After completing these steps, you will be able to:
+
+* [ ] {learning-objective-1}
+* [ ] {learning-objective-2}
+
+== Prerequisites
+
+Before you query a topic with SQL:
+
+* Enable the Redpanda SQL engine on your Redpanda Bring Your Own Cloud (BYOC) cluster. See xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL].
+* A Redpanda Cloud user with the *SQL: Access* (or *SQL: Manage*) data-plane RBAC permission. For a *SQL: Access* user to query a topic, a *SQL: Manage* user must first `GRANT SELECT` on the topic to that user. See xref:sql:manage/manage-access.adoc[Manage access to Redpanda SQL].
+* Connect to Redpanda SQL with `psql` or another PostgreSQL client. See xref:sql:connect-to-sql/index.adoc[Connect to Redpanda SQL].
+* Confirm that the Redpanda topic you want to query has a schema registered in Schema Registry. Redpanda SQL supports Protobuf, JSON, and Avro schemas.
+
+== Map the topic to a SQL table
+
+Each Redpanda topic appears as a SQL table inside a Redpanda catalog. When Redpanda SQL is enabled, a catalog named `default_redpanda_catalog` is created automatically and points at your cluster.
+
+Define a table against the topic with `CREATE TABLE`:
+
+[source,sql]
+----
+CREATE TABLE default_redpanda_catalog=>orders WITH (
+  topic = 'orders',
+  schema_subject = 'orders-value'
+);
+----
+
+Replace `orders` with your topic name and `orders-value` with the Schema Registry subject that holds the topic's value schema. `schema_subject` is required: Redpanda SQL needs a schema to deserialize and query the topic's records.
+
+If the topic uses a Protobuf schema that defines more than one message, also set `output_schema_message_full_name` to the fully-qualified name of the message to use:
+
+[source,sql]
+----
+CREATE TABLE default_redpanda_catalog=>orders WITH (
+  topic = 'orders',
+  schema_subject = 'orders-value',
+  output_schema_message_full_name = 'com.example.orders.Order'
+);
+----
+
+The table inherits its column definitions from the registered schema. Each top-level field in the schema becomes a SQL column. For querying nested fields in struct types, see xref:sql:query-data/query-nested-fields.adoc[].
+
+== Run queries
+
+Query the table with standard `SELECT` syntax. The following query returns the first 10 records:
+
+[source,sql]
+----
+SELECT * FROM default_redpanda_catalog=>orders LIMIT 10;
+----
+
+Aggregate and filter records using familiar PostgreSQL constructs:
+
+[source,sql]
+----
+SELECT customer_id, SUM(amount) AS total
+FROM default_redpanda_catalog=>orders
+WHERE status = 'completed'
+GROUP BY customer_id
+ORDER BY total DESC
+LIMIT 10;
+----
+
+== Next steps
+
+* xref:sql:query-data/query-iceberg-topics.adoc[Query Iceberg-enabled topics]: run queries against historical data retained beyond your Redpanda retention window.
+* xref:reference:sql/sql-statements/create-table.adoc[CREATE TABLE]: full reference for the table-against-topic syntax, including all options.
+* xref:reference:sql/index.adoc[Redpanda SQL Reference]: supported SQL statements, clauses, data types, and functions.

modules/sql/pages/query-data/redpanda-catalogs.adoc

@@ -1,81 +1,2 @@
 = Redpanda Catalogs
-:description: Redpanda catalogs are named connections that map Redpanda topics to queryable SQL tables.
-:page-topic-type: reference
-
-Redpanda catalogs are named connections that let you query Redpanda topics using standard SQL. The catalog model consists of three core concepts:
-
-* Catalogs: Named connections to a Redpanda cluster, created with xref:reference:sql/sql-statements/create-redpanda-catalog.adoc[CREATE REDPANDA CATALOG].
-* Tables: Redpanda topics mapped as queryable SQL tables using the `catalog_name\=>table_name` syntax, created with xref:reference:sql/sql-statements/create-table.adoc[CREATE TABLE].
-* Storage connections: Named connections to external object storage such as Amazon S3, created with xref:reference:sql/sql-statements/create-storage.adoc[CREATE STORAGE].
-
-NOTE: Redpanda SQL operates in read-only mode. Data mutation operations such as `INSERT`, `UPDATE`, and `DELETE` are not available. Data is ingested into Redpanda topics and made queryable through catalog mappings.
-
-== Typical workflow
-
-To query Redpanda topic data with SQL:
-
-. Create a catalog connection:
-+
-[source,sql]
-----
-CREATE REDPANDA CATALOG production_redpanda
-WITH (
-  initial_brokers = 'broker1:9092',
-  schema_registry_url = 'http://schema-registry:8081'
-);
-----
-
-. Map a topic as a table:
-+
-[source,sql]
-----
-CREATE TABLE production_redpanda=>user_events
-WITH (topic = 'user-events');
-----
-
-. Query the data:
-+
-[source,sql]
-----
-SELECT * FROM production_redpanda=>user_events LIMIT 10;
-----
-
-== Related statements
-
-[cols="<40%,<60%",options="header"]
-|===
-|Statement |Description
-
-|xref:reference:sql/sql-statements/create-redpanda-catalog.adoc[CREATE REDPANDA CATALOG]
-|Create a catalog connection to a Redpanda cluster.
-
-|xref:reference:sql/sql-statements/alter-redpanda-catalog.adoc[ALTER REDPANDA CATALOG]
-|Modify connection properties of an existing catalog.
-
-|xref:reference:sql/sql-statements/create-table.adoc[CREATE TABLE]
-|Map a Redpanda topic to a SQL table through a catalog.
-
-|xref:reference:sql/sql-statements/alter-table.adoc[ALTER TABLE]
-|Modify options of an existing catalog table.
-
-|xref:reference:sql/sql-statements/drop-table.adoc[DROP TABLE]
-|Remove a catalog table mapping.
-
-|xref:reference:sql/sql-statements/drop-redpanda-catalog.adoc[DROP REDPANDA CATALOG]
-|Remove a Redpanda catalog connection.
-
-|xref:reference:sql/sql-statements/drop-storage.adoc[DROP STORAGE]
-|Remove a named storage definition.
-
-|xref:reference:sql/sql-statements/show-tables.adoc[SHOW TABLES]
-|List tables within a catalog.
-
-|xref:reference:sql/sql-statements/describe.adoc[DESCRIBE]
-|Show details about a catalog or catalog table.
-
-|xref:reference:sql/sql-statements/create-storage.adoc[CREATE STORAGE]
-|Create a connection to external object storage.
-
-|xref:reference:sql/sql-statements/alter-storage.adoc[ALTER STORAGE]
-|Modify an existing storage connection.
-|===
+// stub