characteristic update_values didn't validate typedStruct fields

[matt-beanland]

When using DiffoExample, invalidStruct fields were provided to Characteristic.update_values via Nbn.define_cvc. Invalid values should have caused the operation to fail, however they were not persisted but returned in the result as if they were valid.

Likely that Feature.update_values has similar issues.

[matt-beanland]

The typedStruct value is being persisted correctly in the datalayer, as an embedded structure.
However when the typedStruct is given a field which doesn't exist it is being returned as a map, not a struct in the result. Interestingly another characteristic with value = %AssignableValue{} is returned correctly.

field should be invalid:

       value: %{
         __struct__: "DiffoExample.Nbn.CvcValue",
         svlan: nil,
         bandwidth: 10000,
         cvc_id: "CVC-POI-SYD-001"
       },

valid:

value: %DiffoExample.Nbn.CvcValue{svlan: 1, bandwidth: 10000},

[matt-beanland]

The type of the field isn't being validated either, we put in an atom and it was persisted. Interestingly the return type is the correct struct (structure doesn't validate values, just keys)

value: %DiffoExample.Nbn.CvcValue{svlan: :fred, bandwidth: 10000},

[matt-beanland]

The Diffo.Provider.Characteristic value field is weakly typed, it is just a term. The provider instance extension 'characteristic' dsl sets the type, this can be a simple type, or an embedded resource or TypedStruct.

It might be necessary to persist the type of the value in another field, value_type. We might need to create our own AshType.NewType to handle validation of this.

[matt-beanland]

term is a datatype used for complex elixir structures which a datalayer typically serialises using erlang:binary_to_term/1

If we serialise to our string format with embedded structure names we have the type information needed to deserialise, which we can't introspect from the resource, since the attribute is just term.

[matt-beanland]

AshNeo4j has been refactored, see diffo-dev/ash_neo4j#196 and while this improves typing with Cast/Dump using Ash.Type.cast_stored/dump_to_native wherever possible it removes support for term, which are attempting to use as a dynamic type.

Diffo requires a dynamic type for things such as Characteristic.value, where the exact type of the value won't be known until a Service/Resource is specified (using DIffo Provider Instance Extension DSL). Typically this is defined in the domain, which merely uses Diffo and doesn't recompile it or its dependencies.

The idea is to have an Ash.Type.Dynamic (This probably belongs in Ash but we can experiment in Diffo.). It could simply use the %Ash.Union struct, since the Dynamic typing comes from the resource (or typedStruct) DSL.
The actual running type comes from the type field in the union struct. We'd attempt to apply Ash.Type cast_stored/dump_to_native on this.

The %AshUnion struct is %Ash.Union{value: nil, type: nil}. This represents a nil value of unknown type.
A nil value of a known type is represented %Ash.Union{type: DogTypedStruct, value: nil}.

[matt-beanland]

Since refactoring potential for Dynamic Typing is now stronger but Characteristic.update_values is silent (just logs) if it can't find the characteristic name or the keys don't match the structure. Also Dynamic.apply constraints isn't actually checking anything!

[matt-beanland]

This might be fixed now with improvements to Diffo.Type.Dynamic.

[matt-beanland]

Added a test case which showed that the bug is fixed.