Make it easier to deserialize data from a TypedValue

[dgulotta]

Right now, it's not very convenient to transmit nontrivial data structures via pods. For example, in FrogCrypto, I'd like the server to be able to send a Vec<(i64, RawValue)> to the client. There is a lot of packing and unpacking that needs to be done to convert this type into a Value and then back into its original form.

It seems like the most convenient thing would be to implement a serde::Serializer and serde::Deseralizer that can handle converting to and from TypedValue, similar to the approach taken by serde_json::Value.

[dgulotta]

The Serializer approach does lead to a few less-than-ideal behaviors. The Vec and HashSet types look the same to the serializer, so there is no way to map both Vec to Array and HashSet to Set.

Another approach would be to make our own trait for converting to/from Value, and provide a procedural macro for deriving this trait. I guess the "from" trait could just be TryFrom, but the "to" trait needs to take a max_depth_containers parameter. This approach would offer the user fewer customization options than the serde approach, but maybe it would be good enough.

[robknight]

In the existing serialization code we map the Set and Array variants of TypedValue to JSON objects that looks like this:

{
  set: [a, b, c],
  max_depth: 32
}

{
  array: [a, b, c],
  max_depth: 32
}

Would that work for this case?

See

pod2/src/frontend/serialization.rs

Line 134 in fe8c4ec

fn test_value_serialization() {

[dgulotta]

It's not quite the same thing - I'm thinking about serializing an arbitrary Rust data structure to a TypedValue and this example is about serializing a TypedValue to existing formats like JSON.

[dgulotta]

I tested out the serde approach, and it seems to work in enough cases to be useful. For example, here is a test on a relatively complicated struct:

pod2/src/middleware/serialization.rs

Lines 703 to 757 in 44f58d6

	mod test {
	use serde::{de::DeserializeOwned, Deserialize, Serialize};
	use crate::{
	backends::plonky2::primitives::ec::{curve::Point, schnorr::SecretKey},
	middleware::{serialization::ValueSerializer, Params, RawValue},
	};
	#[derive(Serialize, Deserialize, PartialEq, Eq, Debug)]
	enum Method {
	Search,
	Mine,
	}
	#[derive(Serialize, Deserialize, PartialEq, Eq, Debug)]
	struct FrogDesc {
	frog_id: i64,
	name: String,
	seed_range: Vec<(Method, RawValue)>,
	option1: Option<i64>,
	option2: Option<i64>,
	unit: (),
	sk: SecretKey,
	pk: Point,
	}
	fn test_roundtrip<T: Serialize + DeserializeOwned + Eq + std::fmt::Debug>(t: T) {
	let depth = Params::default().max_depth_mt_containers;
	let val = t.serialize(ValueSerializer::new(depth)).unwrap();
	println!("{val:?}");
	let out: T = Deserialize::deserialize(val.typed()).unwrap();
	assert_eq!(t, out);
	}
	#[test]
	fn test_frog_desc() {
	let seed_range = vec![
	(Method::Search, RawValue::default()),
	(Method::Mine, RawValue::default()),
	];
	let sk = SecretKey::new_rand();
	let pk = sk.public_key();
	let desc = FrogDesc {
	frog_id: 1,
	name: "a frog".to_string(),
	seed_range,
	option1: Some(2),
	option2: None,
	unit: (),
	sk,
	pk,
	};
	test_roundtrip(desc);
	}
	}

Most of this code could go in a separate crate if we don't want to support an official serializer for pods. However, in order for public and secret keys to be handled correctly, their serialization methods need to tell the serializer that they are newtype structs:

pod2/src/backends/plonky2/primitives/ec/curve.rs

Line 153 in 44f58d6

serializer.serialize_newtype_struct("Point", &point_b58)

pod2/src/backends/plonky2/primitives/ec/schnorr.rs

Line 236 in 44f58d6

serializer.serialize_newtype_struct("SecretKey", &sk_b64)

[robknight]

Oh, I see what you mean now!

This looks neat, and easier than manually converting each data structure, even if we had better helper functions for data conversion/access. We used to have something similar in POD1, for converting JavaScript types to PODValues.