ISO 15118-20 Ed. 1 Amd 1 (AC-DER) fails to create code

[barsnick]

Describe the bug

The schema for Amendment 1 to the ISO 15118-20 has been released:
https://standards.iso.org/iso/15118/-20/ed-1/en/Amd/1/

Yet cbExiGen fails to create code for it.

Even if a potential bug is fixed (or worked around), the created code does not compile.

To Reproduce

Place V2G_CI_AC_DER.xsd into the same directory as the ISO 15118-20 schema files (e.g. ISO_15118-20/FDIS/).

Add this section to src/config.py:

diff --git a/src/config.py b/src/config.py
index 65884f4..a27330c 100644
--- a/src/config.py
+++ b/src/config.py
@@ -726,4 +726,61 @@ c_files_to_generate = {
                               'iso20_ACDP_Datatypes.h', 'iso20_ACDP_Encoder.h']
         }
     },
+    'iso20_AC_DER_Datatypes': {
+        'schema': 'ISO_15118-20/FDIS/V2G_CI_AC_DER.xsd',
+        'prefix': 'iso20_ac_der_',
+        'type': 'converter',
+        'folder': 'iso-20',
+        'h': {
+            'filename': 'iso20_AC_DER_Datatypes.h',
+            'identifier': 'ISO20_AC_DER_DATATYPES_H',
+            'include_std_lib': ['stdint.h'],
+            'include_other': ['exi_basetypes.h']
+        },
+        'c': {
+            'filename': 'iso20_AC_DER_Datatypes.c',
+            'identifier': 'ISO20_AC_DER_DATATYPES_C',
+            'include_std_lib': [],
+            'include_other': ['iso20_AC_DER_Datatypes.h']
+        }
+    },
+    'iso20_AC_DER_Decoder': {
+        'schema': 'ISO_15118-20/FDIS/V2G_CI_AC_DER.xsd',
+        'prefix': 'iso20_ac_der_',
+        'type': 'decoder',
+        'folder': 'iso-20',
+        'h': {
+            'filename': 'iso20_AC_DER_Decoder.h',
+            'identifier': 'ISO20_AC_DER_DECODER_H',
+            'include_std_lib': [],
+            'include_other': ['exi_bitstream.h', 'iso20_AC_DER_Datatypes.h']
+        },
+        'c': {
+            'filename': 'iso20_AC_DER_Decoder.c',
+            'identifier': 'ISO20_AC_DER_DECODER_C',
+            'include_std_lib': ['stdint.h'],
+            'include_other': ['exi_basetypes.h', 'exi_types_decoder.h', 'exi_basetypes_decoder.h',
+                              'exi_error_codes.h', 'exi_header.h', 'iso20_AC_DER_Datatypes.h',
+                              'iso20_AC_DER_Decoder.h']
+        }
+    },
+    'iso20_AC_DER_Encoder': {
+        'schema': 'ISO_15118-20/FDIS/V2G_CI_AC_DER.xsd',
+        'prefix': 'iso20_ac_der_',
+        'type': 'encoder',
+        'folder': 'iso-20',
+        'h': {
+            'filename': 'iso20_AC_DER_Encoder.h',
+            'identifier': 'ISO20_AC_DER_ENCODER_H',
+            'include_std_lib': [],
+            'include_other': ['exi_bitstream.h', 'iso20_AC_DER_Datatypes.h']
+        },
+        'c': {
+            'filename': 'iso20_AC_DER_Encoder.c',
+            'identifier': 'ISO20_AC_DER_ENCODER_C',
+            'include_std_lib': ['stdint.h'],
+            'include_other': ['exi_basetypes.h', 'exi_basetypes_encoder.h', 'exi_error_codes.h', 'exi_header.h',
+                              'iso20_AC_DER_Datatypes.h', 'iso20_AC_DER_Encoder.h']
+        }
+    },
 }

Run cbExiGen, and note this failure:

*** Generator info: ISO_15118-20/FDIS/V2G_CI_AC_DER.xsd ***
Traceback (most recent call last):
  File "src/main.py", line 58, in <module>
    analyze_schema(sys.argv)
  File "src/main.py", line 54, in analyze_schema
    gen.generate_files()
  File "/home/intech/Development/bitbucket/v2g_exi_codec_gen/src/cbexigen/FileGenerator.py", line 240, in generate_files
    self.__generate(True, params)
  File "/home/intech/Development/bitbucket/v2g_exi_codec_gen/src/cbexigen/FileGenerator.py", line 190, in __generate
    self.__generate_converter_h(current_schema, parameters, self.__analyzer_data)
  File "/home/intech/Development/bitbucket/v2g_exi_codec_gen/src/cbexigen/FileGenerator.py", line 144, in __generate_converter_h
    header.generate_file()
  File "/home/intech/Development/bitbucket/v2g_exi_codec_gen/src/cbexigen/datatype_classes.py", line 569, in generate_file
    element = self.__generate[curr_idx]
IndexError: list index out of range

If you change this line:

cbexigen/src/cbexigen/encoder_classes.py

Line 968 in cc1470c

if curr_idx > len(self.elements_to_generate):

and the corresponding ones in database_classes.py and decoder_classes.py, which may have off-by-one errors, as such:

diff --git a/src/cbexigen/datatype_classes.py b/src/cbexigen/datatype_classes.py
index de5ae06..a4fd47d 100644
--- a/src/cbexigen/datatype_classes.py
+++ b/src/cbexigen/datatype_classes.py
@@ -580,7 +580,7 @@ class DatatypeHeader:
 
             if skip_element:
                 curr_idx += 1
-                if curr_idx > len(self.__generate):
+                if curr_idx >= len(self.__generate):
                     log_write_error('Module datatypes: Generator loop aborted! Index larger than existing elements.')
                     break
             else:
diff --git a/src/cbexigen/decoder_classes.py b/src/cbexigen/decoder_classes.py
index e2dabc8..1d08f55 100644
--- a/src/cbexigen/decoder_classes.py
+++ b/src/cbexigen/decoder_classes.py
@@ -973,7 +973,7 @@ class ExiDecoderCode(ExiBaseCoderCode):
 
             if skip_element:
                 curr_idx += 1
-                if curr_idx > len(self.elements_to_generate):
+                if curr_idx >= len(self.elements_to_generate):
                     log_write_error('Module decoder: Generator loop aborted! Index larger than existing elements.')
                     break
             else:
diff --git a/src/cbexigen/encoder_classes.py b/src/cbexigen/encoder_classes.py
index d707145..3cc35a5 100644
--- a/src/cbexigen/encoder_classes.py
+++ b/src/cbexigen/encoder_classes.py
@@ -970,7 +970,7 @@ class ExiEncoderCode(ExiBaseCoderCode):
 
             if skip_element:
                 curr_idx += 1
-                if curr_idx > len(self.elements_to_generate):
+                if curr_idx >= len(self.elements_to_generate):
                     log_write_error('Module encoder: Generator loop aborted! Index larger than existing elements.')
                     break
             else:

then code for AC-DER is created, but with three warnings while generating
Generator loop aborted! Index larger than existing elements.
The generated code is incomplete, it does not compile, due to incomplete types, forward declared structs:

In file included from src/output/c/iso-20/iso20_AC_DER_Datatypes.c:21:
src/output/c/iso-20/iso20_AC_DER_Datatypes.h:2721:64: error: field has incomplete type 'struct iso20_ac_der_AC_ChargeParameterDiscoveryReqType'
        struct iso20_ac_der_AC_ChargeParameterDiscoveryReqType AC_ChargeParameterDiscoveryReq;
                                                               ^
src/output/c/iso-20/iso20_AC_DER_Datatypes.h:2721:16: note: forward declaration of 'struct iso20_ac_der_AC_ChargeParameterDiscoveryReqType'
        struct iso20_ac_der_AC_ChargeParameterDiscoveryReqType AC_ChargeParameterDiscoveryReq;

Anything else?

No response

[barsnick]

This may be related to this xmlschema issue:
xmlschema does not handle substitution groups correctly (#xmlschema/452)
(Substitutions of substitutions not working.)

[SebaLukas]

I did some research and was able to generate an initial AC DER c code using a hack.

Hoping that xmlschema version 4.1.0 would solve the above problem, I updated xmlschema. I describe the procedure here: #123

Unfortunately, the code generation still did not work.

Then I had an idea. I merged the two AC and AC_DER xsd files together. Maybe xmlschema and/or cbexigen have a problem with the file structure.
You can find the diff here: der_diff.txt

With the merged xsd, there are no problems during generation. But the new DER parameters were not included in the AC_ChargeParameterDiscovery and AC_ChargeLoop messages.
I was also able to solve this problem and push this in this branch: https://github.com/EVerest/cbexigen/tree/feat/adding-ac-der-support

Explanation: Before the fix, cbexigen assumed that there was only one particle when types inherit parameters from other types. For example, with AC_CPDReqEnergyTransferMode and BPT_AC_CPDReqEnergyTransferMode. However, DER adds DER_AC_CPDReqEnergyTransferMode, which inherits parameters from AC_CPDReqEnergyTransferMode. With my fix, both particles are now added to the AC_ChargeParameterDiscoveryReq struct.

I also added the generated code to libcbv2g: https://github.com/EVerest/libcbv2g/tree/feat/adding-ac-der-messages
libcbv2g also builds with the new ac files. However, I haven't been able to test the changes against reference streams yet, as I don't have any.