Agentically coded SCIP backend for the LSP

Cargo.toml

@@ -50,5 +50,7 @@ subprocess = "1.0"
 thiserror = "2.0"
 urlencoding = "2.1"
 utf8-read = "0.4"
+scip = "0.6.1"
+protobuf = "3"
 walkdir = "2"
 heck = "0.5"

USAGE.md

@@ -117,3 +117,133 @@ method now_we_can_declare_this_method_with_a_really_really_really_really_long na
 ```
 Will allow 'long_lines' globally, 'nsp_unary' and 'indent_no_tabs' on the
 `param p = (1 ++ *` line, and 'indent_paren_expr' on the `'4);` line.
+
+## SCIP Export
+
+The DLS can export a [SCIP index](https://sourcegraph.com/docs/code-search/code-navigation/scip)
+of analyzed DML devices. SCIP (Source Code Intelligence Protocol) is a
+language-agnostic format for code intelligence data, used by tools such as
+Sourcegraph for cross-repository navigation and code search.
+
+### Invocation
+
+SCIP export is available through the DFA (DML File Analyzer) binary via the
+`--scip-output <path>` flag:
+```
+dfa --compile-info <compile_commands.json> --workspace <project root> --scip-output <scip_file_name> [list of devices to analyze, ]
+```
+
+It is worth noting that SCIP format specifies that symbols from documents that are not under the project root (which we define as the workspace) get slotted under external symbols with no occurances tracked.
+
+### SCIP schema details
+Here we list how we have mapped DML specifically to the SCIP format.
+
+#### SCIP symbol kind mappings
+
+DML symbol kinds are mapped to SCIP `SymbolInformation.Kind` as follows:
+
+- `Constant` — Parameter, Constant, Loggroup
+- `Variable` — Extern, Saved, Session, Local
+- `Parameter` — MethodArg
+- `Event` — Hook
+- `Method` — Method
+- `Class` — Template
+- `TypeAlias` — Typedef
+- `Object` — All composite objects (Device, Bank, Register, Field, Group, Port, Connect, Attribute, Event, Subdevice, Implement)
+- `Interface` — Interface
+
+Note: SCIP's `Object` kind is used for DML composite objects because they are
+instantiated structural components in the device hierarchy, not types or
+namespaces. `Event` is used for DML hooks because they represent named event
+points that can be sent or listened to.
+
+Since SCIP's `Kind` enum is too coarse to distinguish between the various DML
+composite object kinds (e.g. `register` vs `bank` vs `attribute`), the
+`SymbolInformation.documentation` field carries a short-form declaration
+signature that disambiguates:
+
+- **Composite objects:** the DML keyword for the object kind, e.g. `register`,
+  `bank`, `attribute`, `group`, `field`, `device`, etc.
+- **Methods:** the DML declaration modifiers, e.g. `method`,
+  `independent method default`, `shared method throws`.
+- **Other symbol kinds:** no documentation is emitted.
+
+#### Symbol Naming Scheme
+
+SCIP symbols follow the format:
+`<scheme> ' ' <manager> ' ' <package> ' ' <version> ' ' <descriptors>`
+
+For DML, the scheme is `dml`, the manager is `simics`, version is `.` (currently we cannot extract simics version here), and the
+package is the device name. Descriptors are built from the fully qualified path
+through the device hierarchy:
+
+```
+dml simics sample_device . sample_device.regs.r1.offset.
+                                                       ^ term (parameter)
+dml simics sample_device . sample_device.regs.r1.read().
+                                                      ^ method
+dml simics sample_device . bank#
+                              ^ 'type' (template)
+```
+
+Descriptor suffixes follow the SCIP standard:
+- `.` (term) — used for composite objects, parameters, and other named values
+- `#` (type) — used only for templates
+- `().` (method) — used for methods
+
+#### Local Symbols
+
+Method arguments and method-local variables use SCIP local symbols of the form
+`local <name>_<id>`, where `<id>` is the internal symbol identifier. Local
+symbols are scoped to a single document and are not navigable across files.
+
+#### Occurrence Roles
+
+DML definitions (including the primary symbol location) are emitted with the
+SCIP `Definition` role. Declarations that also appear as definitions share
+this role. Declarations that do _not_ define a value (e.g. abstract method
+declarations, or `default` parameter declarations that are overridden) are
+emitted with the `ForwardDefinition` role.
+
+References (including template instantiation sites from `is` statements) are
+emitted as plain references with no additional role flags. Access-kind
+refinement (`ReadAccess` / `WriteAccess`) is not yet tracked.
+
+#### Enclosing Ranges
+
+For composite object definitions and method declarations, each `Definition`
+or `ForwardDefinition` occurrence includes an `enclosing_range` that spans
+the full AST node (e.g. the complete `register r1 is ... { ... }` block or
+the full method body). This allows consumers to associate the definition site
+with the extent of the construct it names.
+
+#### Deduplication and Determinism
+
+When multiple device analyses share source files (e.g. common library code),
+the SCIP export deduplicates occurrences and symbol information so that each
+(symbol, range, role) triple and each symbol entry appears at most once.
+All output is sorted deterministically: documents by relative path,
+occurrences by range, symbols by symbol string, and relationships by symbol.
+
+#### Relationships
+
+Composite objects that instantiate templates (via `is some_template`) emit
+SCIP `Relationship` entries with `is_implementation = true` pointing to the
+template symbol.
+
+#### File Symbols and Imports
+
+Each source file involved in the analysis gets a dedicated SCIP symbol of kind
+`File`. A `Definition` occurrence is emitted at line 0 of each file so that
+navigation to the file symbol opens the file itself.
+
+For each `import "..."` statement, an `Import` occurrence is emitted at the
+import statement's span, referencing the imported file's symbol. This lets
+consumers navigate from import statements to the imported file and visualize
+file-level dependency graphs.
+
+File symbols use the format:
+```
+dml simics . . path/to/file_dml.
+```
+where path segments are separated by term descriptors (`.`).

src/actions/requests.rs

@@ -959,6 +959,153 @@ impl RequestAction for GetKnownContextsRequest {
     }
 }
 
+// ---- SCIP Export Request ----
+
+#[derive(Debug, Clone)]
+pub struct ExportScipRequest;
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ExportScipParams {
+    /// Device paths to export SCIP for. If empty, exports all known devices.
+    pub devices: Option<Vec<lsp_types::Uri>>,
+    /// The file path where the SCIP index should be written.
+    pub output_path: String,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct ExportScipResult {
+    /// Whether the export succeeded.
+    pub success: bool,
+    /// Number of documents in the exported index.
+    pub document_count: usize,
+    /// Error message, if any.
+    pub error: Option<String>,
+}
+
+impl LSPRequest for ExportScipRequest {
+    type Params = ExportScipParams;
+    type Result = ExportScipResult;
+
+    const METHOD: &'static str = "$/exportScip";
+}
+
+impl RequestAction for ExportScipRequest {
+    type Response = ExportScipResult;
+
+    fn timeout() -> std::time::Duration {
+        crate::server::dispatch::DEFAULT_REQUEST_TIMEOUT * 30
+    }
+
+    fn fallback_response() -> Result<Self::Response, ResponseError> {
+        Ok(ExportScipResult {
+            success: false,
+            document_count: 0,
+            error: Some("Request timed out".to_string()),
+        })
+    }
+
+    fn get_identifier(params: &Self::Params) -> String {
+        Self::request_identifier(&params.output_path)
+    }
+
+    fn handle<O: Output>(
+        ctx: InitActionContext<O>,
+        params: Self::Params,
+    ) -> Result<Self::Response, ResponseError> {
+        info!("Handling SCIP export request to {}", params.output_path);
+
+        // Determine which device paths to export
+        let device_paths: Vec<CanonPath> =
+            if let Some(devices) = params.devices {
+                devices.iter().filter_map(
+                    |uri| parse_file_path!(&uri, "ExportScip")
+                        .ok()
+                        .and_then(CanonPath::from_path_buf))
+                    .collect()
+            } else {
+                vec![]
+            };
+
+        // Wait for device analyses to be ready
+        if !device_paths.is_empty() {
+            ctx.wait_for_state(
+                AnalysisProgressKind::Device,
+                AnalysisWaitKind::Work,
+                AnalysisCoverageSpec::Paths(device_paths.clone())).ok();
+        } else {
+            ctx.wait_for_state(
+                AnalysisProgressKind::Device,
+                AnalysisWaitKind::Work,
+                AnalysisCoverageSpec::All).ok();
+        }
+
+        let analysis = ctx.analysis.lock().unwrap();
+
+        // Collect device analyses
+        let devices: Vec<&crate::analysis::DeviceAnalysis> =
+            if device_paths.is_empty() {
+                // Export all device analyses
+                analysis.device_analysis.values()
+                    .map(|ts| &ts.stored)
+                    .collect()
+            } else {
+                device_paths.iter().filter_map(|path| {
+                    analysis.get_device_analysis(path).ok()
+                }).collect()
+            };
+
+        if devices.is_empty() {
+            return Ok(ExportScipResult {
+                success: false,
+                document_count: 0,
+                error: Some("No device analyses found".to_string()),
+            });
+        }
+
+        info!("Exporting SCIP for {} device(s)", devices.len());
+
+        // Extract import resolution data for the SCIP export
+        let import_data = crate::scip::extract_import_data(
+            &analysis.isolated_analysis,
+            &analysis.import_map,
+            &devices,
+        );
+
+        // Determine project root from workspaces
+        let project_root = ctx.workspace_roots
+            .lock()
+            .unwrap()
+            .first()
+            .and_then(|ws| parse_file_path!(&ws.uri, "ExportScip").ok())
+            .unwrap_or_else(|| std::path::PathBuf::from("."));
+
+        let index = crate::scip::build_scip_index(&devices, &project_root,
+                                                   &import_data);
+        let doc_count = index.documents.len();
+
+        let output = std::path::Path::new(&params.output_path);
+        match crate::scip::write_scip_to_file(index, output) {
+            Ok(()) => {
+                info!("SCIP export complete: {} documents written to {}",
+                      doc_count, params.output_path);
+                Ok(ExportScipResult {
+                    success: true,
+                    document_count: doc_count,
+                    error: None,
+                })
+            },
+            Err(e) => {
+                error!("SCIP export failed: {}", e);
+                Ok(ExportScipResult {
+                    success: false,
+                    document_count: 0,
+                    error: Some(e),
+                })
+            }
+        }
+    }
+}
+
 /// Server-to-client requests
 impl SentRequest for RegisterCapability {
     type Response = <Self as lsp_data::request::Request>::Result;

src/cmd.rs

@@ -322,6 +322,24 @@ pub fn set_contexts(paths: Vec<String>) -> Notification<notifications::ChangeAct
     }
 }
 
+pub fn export_scip(devices: Vec<String>, output_path: String) -> Request<requests::ExportScipRequest> {
+    Request {
+        params: requests::ExportScipParams {
+            devices: if devices.is_empty() {
+                None
+            } else {
+                Some(devices.into_iter()
+                    .map(|p| parse_uri(&p).unwrap())
+                    .collect())
+            },
+            output_path,
+        },
+        action: PhantomData,
+        id: next_id(),
+        received: Instant::now(),
+    }
+}
+
 fn next_id() -> RequestId {
     static ID: AtomicU64 = AtomicU64::new(1);
     RequestId::Num(ID.fetch_add(1, Ordering::SeqCst))

src/dfa/client.rs

@@ -412,4 +412,37 @@ impl ClientInterface {
         self.server.wait_timeout(Duration::from_millis(1000))?;
         Ok(())
     }
+
+    pub fn export_scip(&mut self,
+                       device_paths: Vec<String>,
+                       output_path: String)
+                       -> anyhow::Result<crate::actions::requests::ExportScipResult> {
+        debug!("Sending SCIP export request for {:?} -> {}", device_paths, output_path);
+        self.send(
+            cmd::export_scip(device_paths, output_path).to_string()
+        )?;
+        // Wait for the response
+        loop {
+            match self.receive_maybe() {
+                Ok(ServerMessage::Response(value)) => {
+                    let result: crate::actions::requests::ExportScipResult
+                        = serde_json::from_value(value)
+                        .map_err(|e| RpcErrorKind::from(e.to_string()))?;
+                    return Ok(result);
+                },
+                Ok(ServerMessage::Error(e)) => {
+                    return Err(anyhow::anyhow!(
+                        "Server exited during SCIP export: {:?}", e));
+                },
+                Ok(_) => {
+                    // Skip other messages (diagnostics, progress, etc.)
+                    continue;
+                },
+                Err(e) => {
+                    trace!("Skipping message during SCIP export wait: {:?}", e);
+                    continue;
+                }
+            }
+        }
+    }
 }