Skip to content

Content hash based resources#4148

Open
timon-schelling wants to merge 5 commits into
masterfrom
content-hash-based-resources
Open

Content hash based resources#4148
timon-schelling wants to merge 5 commits into
masterfrom
content-hash-based-resources

Conversation

@timon-schelling
Copy link
Copy Markdown
Member

No description provided.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a persistent resource storage system for Graphite, replacing the practice of embedding raw image data directly within the node graph. It implements a ResourceStorage trait with specialized backends: IndexedDbResourceStorage for web environments and MmapResourceStorage for desktop, utilizing Blake3 for content hashing. Key architectural changes include a preprocessor that automatically swaps resource hashes for dedicated resource nodes and updates to the document save/load flow to handle externalized assets. Feedback focuses on scalability and stability concerns, specifically the memory-intensive hydration of the entire IndexedDB store on startup, the use of expect() and panic! which could lead to application crashes, and the inefficient re-initialization of HTTP clients during resource loading.

Comment thread node-graph/graph-craft/src/application_io/resource/indexed_db.rs Outdated
Comment thread desktop/src/app.rs Outdated
self.resize();

self.desktop_wrapper.init(self.wgpu_context.clone());
let resource_storage = MmapResourceStorage::new(dirs::app_resources_dir()).expect("Failed to initialize on-disk resource storage");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using .expect() here will cause the application to crash immediately if the resources directory cannot be created or accessed (e.g., due to permission issues). While this occurs at startup, it would be better to handle this error gracefully and show a message to the user, especially since this directory is critical for the new resource system.

async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };

let response = match reqwest::Client::new().get(&url).send().await {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Creating a new reqwest::Client for every execution of the load_resource node is inefficient. reqwest clients are intended to be reused to benefit from internal connection pooling and to avoid the overhead of re-initializing the client state for every request.

Suggested change
let response = match reqwest::Client::new().get(&url).send().await {
let response = match _editor.application_io.as_ref().map(|io| io.http_client()).unwrap_or_else(reqwest::Client::new).get(&url).send().await {

Comment on lines +254 to +256
application_io.load_resource(&hash).unwrap_or_else(|| {
panic!("Resource {hash} not found");
})
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using panic! inside a node is generally discouraged as it will crash the entire node graph executor and potentially the application. If a resource is missing (e.g., if it was manually deleted from the disk), it would be better to return an empty Resource or a placeholder image and log a warning instead of crashing.

application_io.load_resource(&hash).unwrap_or_else(|| {
		log::error!("Resource {hash} not found");
		Resource::new(Vec::new())
	})

@timon-schelling timon-schelling force-pushed the content-hash-based-resources branch 3 times, most recently from c902384 to 31b7b33 Compare May 16, 2026 00:15
@Keavon Keavon force-pushed the master branch 2 times, most recently from 4b7a823 to 847b8e9 Compare May 17, 2026 14:37
@timon-schelling timon-schelling force-pushed the master branch 2 times, most recently from 15fcaac to d5f0140 Compare May 17, 2026 15:37
@timon-schelling timon-schelling force-pushed the content-hash-based-resources branch from 058da30 to 8200b15 Compare May 17, 2026 22:15
@timon-schelling timon-schelling force-pushed the content-hash-based-resources branch 2 times, most recently from 35eec4d to 3c27e65 Compare May 18, 2026 01:19
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

19 issues found across 44 files

Confidence score: 2/5

  • Merge risk is high because several high-confidence 8/10 issues are user-impacting: panic paths in node-graph/nodes/gstd/src/platform_application_io.rs and node-graph/graph-craft/src/application_io.rs can crash resource loading/render flows when resources or hashes are missing.
  • There is concrete document integrity risk across node-graph/graph-craft/src/application_io/resource/indexed_db.rs and editor/src/messages/resource/resource_message_handler.rs: reads/exports depend on in-memory cache and can omit resources that still exist only in IndexedDB, leading to broken saves.
  • Migration and persistence paths also look fragile in editor/src/messages/portfolio/document_migration.rs and editor/src/messages/portfolio/portfolio_message_handler.rs, where legacy shapes/state cleanup can be skipped and leave incompatible or stale persisted data.
  • Pay close attention to node-graph/nodes/gstd/src/platform_application_io.rs, node-graph/graph-craft/src/application_io/resource/indexed_db.rs, and editor/src/messages/resource/resource_message_handler.rs - crash and data-loss behaviors are the main merge blockers.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="node-graph/graph-craft/src/application_io/wasm.rs">

<violation number="1" location="node-graph/graph-craft/src/application_io/wasm.rs:1">
P1: Custom agent: **PR title enforcement**

PR title is not in imperative mood and lacks a leading action verb</violation>
</file>

<file name="editor/src/messages/portfolio/document/utility_types/embedded_resources.rs">

<violation number="1" location="editor/src/messages/portfolio/document/utility_types/embedded_resources.rs:47">
P2: Serializing the `HashMap` directly makes saved document output nondeterministic. Sort the resource entries before writing them so identical documents serialize identically.</violation>
</file>

<file name="node-graph/libraries/application-io/src/lib.rs">

<violation number="1" location="node-graph/libraries/application-io/src/lib.rs:155">
P2: Hash the shared `Io` allocation here, not the `Arc` handle's address. As written, `Arc::clone` of the same `application_io` produces different `EditorApi` hashes and can cause avoidable cache misses.

(Based on your team's feedback about avoiding misleading or inconsistent hashing implementations.) [FEEDBACK_USED]</violation>
</file>

<file name="node-graph/graph-craft/src/application_io.rs">

<violation number="1" location="node-graph/graph-craft/src/application_io.rs:68">
P1: Don't `expect` on `resources` here. `new()`/`default()` leave it unset, and some call sites (for example the CLI) construct `PlatformApplicationIo` without calling `inject_resources()`, so loading a content-hash resource can abort the whole process.

(Based on your team's feedback about avoiding panics in application code.) [FEEDBACK_USED]</violation>
</file>

<file name="editor/src/dispatcher.rs">

<violation number="1" location="editor/src/dispatcher.rs:42">
P2: `with_executor()` now leaves the resource subsystem uninitialized, so non-test callers built through this constructor will drop `ResourceMessage`s and can panic when a resource handle is requested.

(Based on your team's feedback about making invalid states unrepresentable in data models.) [FEEDBACK_USED]</violation>
</file>

<file name="node-graph/graph-craft/src/application_io/resource/mmap.rs">

<violation number="1" location="node-graph/graph-craft/src/application_io/resource/mmap.rs:56">
P2: Verify the mapped bytes still hash to the requested `ResourceHash` before returning them.</violation>

<violation number="2" location="node-graph/graph-craft/src/application_io/resource/mmap.rs:107">
P1: Don't report a successful resource hash after the filesystem write failed. Otherwise callers can store a hash that this backend can never load.</violation>
</file>

<file name="node-graph/nodes/gstd/src/platform_application_io.rs">

<violation number="1" location="node-graph/nodes/gstd/src/platform_application_io.rs:140">
P2: Use an actual transparent image placeholder here; empty bytes make failed loads disappear instead of preserving the transparent fallback promised by this node.</violation>

<violation number="2" location="node-graph/nodes/gstd/src/platform_application_io.rs:142">
P2: This now treats every resource as an HTTP URL, so local asset paths documented for this node will always fail.</violation>

<violation number="3" location="node-graph/nodes/gstd/src/platform_application_io.rs:265">
P1: Don't panic when the resource backend or hash is missing; this node can crash the whole render path on ordinary load failures.

(Based on your team's feedback about avoiding panics in application code.) [FEEDBACK_USED]</violation>
</file>

<file name="node-graph/graph-craft/src/application_io/resource.rs">

<violation number="1" location="node-graph/graph-craft/src/application_io/resource.rs:12">
P2: This `Mutex` is unnecessary for the in-memory store; it only adds contention and panic-on-poison paths to code that could use a plain `HashMap`.

(Based on your team's feedback about reusing existing abstractions instead of adding mutexes.) [FEEDBACK_USED]</violation>
</file>

<file name="editor/src/application.rs">

<violation number="1" location="editor/src/application.rs:12">
P2: `Editor::new()` now returns a partially initialized editor and depends on a separate `replace_application_io()` call before graph execution is safe.

(Based on your team's feedback about placing integrity-enforcing initialization where the state is created.) [FEEDBACK_USED]</violation>
</file>

<file name="editor/src/messages/portfolio/document_migration.rs">

<violation number="1" location="editor/src/messages/portfolio/document_migration.rs:1594">
P1: Handle the legacy 1-input image node shape here too; after the alias remap, older embedded-image documents skip this migration and keep an incompatible input layout.</violation>
</file>

<file name="editor/src/messages/resource/resource_message_handler.rs">

<violation number="1" location="editor/src/messages/resource/resource_message_handler.rs:35">
P1: `export()` silently drops resources that are missing from the in-memory cache, which can save broken documents on the IndexedDB backend.</violation>

<violation number="2" location="editor/src/messages/resource/resource_message_handler.rs:54">
P1: The non-test default creates an unusable `ResourceMessageHandler`: it can panic in `resources()` or silently export no resources during save.

(Based on your team's feedback about avoiding panics and making invalid states unrepresentable.) [FEEDBACK_USED]</violation>
</file>

<file name="editor/src/messages/portfolio/document/graph_operation/utility_types.rs">

<violation number="1" location="editor/src/messages/portfolio/document/graph_operation/utility_types.rs:304">
P1: This rewires the bitmap layer to a hash before the resource write can fail safely. If `ResourceMessage::Write` hits an I/O error, the node still keeps the hash-only reference and later resource loading panics when that hash is missing.

(Based on your team's feedback about avoiding panics in application code.) [FEEDBACK_USED]</violation>
</file>

<file name="node-graph/graph-craft/src/application_io/resource/indexed_db.rs">

<violation number="1" location="node-graph/graph-craft/src/application_io/resource/indexed_db.rs:177">
P1: `read()` only checks the cache, so saving a reopened autosaved document can omit resources that still live only in IndexedDB.</violation>
</file>

<file name="editor/src/messages/portfolio/portfolio_message_handler.rs">

<violation number="1" location="editor/src/messages/portfolio/portfolio_message_handler.rs:452">
P1: This GC fallback loads and selects another document, then aborts the cleanup pass for legacy persisted state entries.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

@@ -1,105 +0,0 @@
use dyn_any::StaticType;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Custom agent: PR title enforcement

PR title is not in imperative mood and lacks a leading action verb

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/wasm.rs:

<comment>PR title is not in imperative mood and lacks a leading action verb</comment>

<file context>
@@ -1,105 +0,0 @@
-use dyn_any::StaticType;
-use graphene_application_io::{ApplicationError, ApplicationIo, ResourceFuture};
-use std::collections::HashMap;
-use std::sync::Arc;
-#[cfg(feature = "tokio")]
-use tokio::io::AsyncReadExt;
-#[cfg(target_family = "wasm")]
-use wasm_bindgen::JsCast;
-#[cfg(feature = "wgpu")]
</file context>

}

fn load_resource(&self, hash: ResourceHash) -> graphene_application_io::ResourceFuture {
self.resources.as_ref().expect("Resource storage not initialized").load(hash)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Don't expect on resources here. new()/default() leave it unset, and some call sites (for example the CLI) construct PlatformApplicationIo without calling inject_resources(), so loading a content-hash resource can abort the whole process.

(Based on your team's feedback about avoiding panics in application code.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io.rs, line 68:

<comment>Don't `expect` on `resources` here. `new()`/`default()` leave it unset, and some call sites (for example the CLI) construct `PlatformApplicationIo` without calling `inject_resources()`, so loading a content-hash resource can abort the whole process.

(Based on your team's feedback about avoiding panics in application code.) </comment>

<file context>
@@ -1,14 +1,93 @@
+	}
+
+	fn load_resource(&self, hash: ResourceHash) -> graphene_application_io::ResourceFuture {
+		self.resources.as_ref().expect("Resource storage not initialized").load(hash)
+	}
+}
</file context>

})();

if let Err(error) = write_result {
log::error!("Failed to write resource to {path:?}: {error}");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Don't report a successful resource hash after the filesystem write failed. Otherwise callers can store a hash that this backend can never load.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/resource/mmap.rs, line 107:

<comment>Don't report a successful resource hash after the filesystem write failed. Otherwise callers can store a hash that this backend can never load.</comment>

<file context>
@@ -0,0 +1,160 @@
+		})();
+
+		if let Err(error) = write_result {
+			log::error!("Failed to write resource to {path:?}: {error}");
+			let _ = fs::remove_file(&tmp);
+		}
</file context>


#[node_macro::node(category(""))]
pub async fn resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] editor_api: &'a PlatformEditorApi, hash: ResourceHash) -> Resource {
let application_io = editor_api.application_io.as_ref().expect("ApplicationIo must be available when using resources");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Don't panic when the resource backend or hash is missing; this node can crash the whole render path on ordinary load failures.

(Based on your team's feedback about avoiding panics in application code.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/nodes/gstd/src/platform_application_io.rs, line 265:

<comment>Don't panic when the resource backend or hash is missing; this node can crash the whole render path on ordinary load failures.

(Based on your team's feedback about avoiding panics in application code.) </comment>

<file context>
@@ -254,3 +259,34 @@ where
+
+#[node_macro::node(category(""))]
+pub async fn resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] editor_api: &'a PlatformEditorApi, hash: ResourceHash) -> Resource {
+	let application_io = editor_api.application_io.as_ref().expect("ApplicationIo must be available when using resources");
+	application_io.load_resource(hash).await.unwrap_or_else(|| {
+		panic!("Resource {hash} not found");
</file context>

document.network_interface.replace_implementation(node_id, network_path, &mut node_template);
// Upgrade `image` nodes that stored image data as `Image<Color>`` to `image` nodes that store a reference to the image data as a resource.
// Encodes the image data as PNG and stores it in the document's embedded resources and rewires the node to reference that resource.
if reference == DefinitionIdentifier::ProtoNode(graphene_std::platform_application_io::image::IDENTIFIER) && inputs_count == 2 {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Handle the legacy 1-input image node shape here too; after the alias remap, older embedded-image documents skip this migration and keep an incompatible input layout.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At editor/src/messages/portfolio/document_migration.rs, line 1594:

<comment>Handle the legacy 1-input image node shape here too; after the alias remap, older embedded-image documents skip this migration and keep an incompatible input layout.</comment>

<file context>
@@ -1588,12 +1589,24 @@ fn migrate_node(node_id: &NodeId, node: &DocumentNode, network_path: &[NodeId],
-		document.network_interface.replace_implementation(node_id, network_path, &mut node_template);
+	// Upgrade `image` nodes that stored image data as `Image<Color>`` to `image` nodes that store a reference to the image data as a resource.
+	// Encodes the image data as PNG and stores it in the document's embedded resources and rewires the node to reference that resource.
+	if reference == DefinitionIdentifier::ProtoNode(graphene_std::platform_application_io::image::IDENTIFIER) && inputs_count == 2 {
+		let image = node.inputs.iter().find_map(|input| match input.as_value()? {
+			TaggedValue::ImageData(image) => Some(image.clone()),
</file context>


let path = self.path_for(hash);
let mmap = Self::open_mmap(&path)?;
let resource = Resource::new(MmappedBytes(mmap));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Verify the mapped bytes still hash to the requested ResourceHash before returning them.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/resource/mmap.rs, line 56:

<comment>Verify the mapped bytes still hash to the requested `ResourceHash` before returning them.</comment>

<file context>
@@ -0,0 +1,160 @@
+
+		let path = self.path_for(hash);
+		let mmap = Self::open_mmap(&path)?;
+		let resource = Resource::new(MmappedBytes(mmap));
+
+		self.cache.write().unwrap_or_else(|poisoned| poisoned.into_inner()).insert(*hash, resource.clone());
</file context>

async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };

let response = match reqwest::Client::new().get(&url).send().await {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This now treats every resource as an HTTP URL, so local asset paths documented for this node will always fail.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/nodes/gstd/src/platform_application_io.rs, line 142:

<comment>This now treats every resource as an HTTP URL, so local asset paths documented for this node will always fail.</comment>

<file context>
@@ -137,26 +136,32 @@ fn image_to_bytes(_: impl Ctx, image: List<Raster<CPU>>) -> List<u8> {
+async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
+	let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };
+
+	let response = match reqwest::Client::new().get(&url).send().await {
+		Ok(response) => response,
+		Err(error) => {
</file context>

let Ok(data) = data.await else {
return Arc::from(include_bytes!("../../../graph-craft/src/null.png").to_vec());
async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Use an actual transparent image placeholder here; empty bytes make failed loads disappear instead of preserving the transparent fallback promised by this node.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/nodes/gstd/src/platform_application_io.rs, line 140:

<comment>Use an actual transparent image placeholder here; empty bytes make failed loads disappear instead of preserving the transparent fallback promised by this node.</comment>

<file context>
@@ -137,26 +136,32 @@ fn image_to_bytes(_: impl Ctx, image: List<Raster<CPU>>) -> List<u8> {
-	let Ok(data) = data.await else {
-		return Arc::from(include_bytes!("../../../graph-craft/src/null.png").to_vec());
+async fn load_resource<'a: 'n>(_: impl Ctx, _primary: (), #[scope("editor-api")] _editor: &'a PlatformEditorApi, #[name("URL")] url: String) -> Arc<[u8]> {
+	let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };
+
+	let response = match reqwest::Client::new().get(&url).send().await {
</file context>
Suggested change
let placeholder = || -> Arc<[u8]> { Arc::from(Vec::<u8>::new()) };
let placeholder = || -> Arc<[u8]> { Arc::from(include_bytes!("../../../graph-craft/src/null.png").to_vec()) };

@@ -0,0 +1,47 @@
#[cfg(target_family = "wasm")]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This Mutex is unnecessary for the in-memory store; it only adds contention and panic-on-poison paths to code that could use a plain HashMap.

(Based on your team's feedback about reusing existing abstractions instead of adding mutexes.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At node-graph/graph-craft/src/application_io/resource.rs, line 12:

<comment>This `Mutex` is unnecessary for the in-memory store; it only adds contention and panic-on-poison paths to code that could use a plain `HashMap`.

(Based on your team's feedback about reusing existing abstractions instead of adding mutexes.) </comment>

<file context>
@@ -0,0 +1,47 @@
+
+#[derive(Debug, Default)]
+pub struct HashMapResourceStorage {
+	resources: Mutex<HashMap<ResourceHash, Resource>>,
+}
+
</file context>

Comment thread editor/src/application.rs

impl Editor {
pub fn new(environment: Environment, uuid_random_seed: u64) -> Self {
pub fn new(environment: Environment, uuid_random_seed: u64, resource_storage: Box<dyn ResourceStorage>) -> Self {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Editor::new() now returns a partially initialized editor and depends on a separate replace_application_io() call before graph execution is safe.

(Based on your team's feedback about placing integrity-enforcing initialization where the state is created.)

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At editor/src/application.rs, line 12:

<comment>`Editor::new()` now returns a partially initialized editor and depends on a separate `replace_application_io()` call before graph execution is safe.

(Based on your team's feedback about placing integrity-enforcing initialization where the state is created.) </comment>

<file context>
@@ -8,23 +9,29 @@ pub struct Editor {
 
 impl Editor {
-	pub fn new(environment: Environment, uuid_random_seed: u64) -> Self {
+	pub fn new(environment: Environment, uuid_random_seed: u64, resource_storage: Box<dyn ResourceStorage>) -> Self {
 		ENVIRONMENT.set(environment).expect("Editor shoud only be initialized once");
 		graphene_std::uuid::set_uuid_seed(uuid_random_seed);
</file context>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant