Benchmark MV2/3 inference on Alif E8 with CMSIS-NN on Cortex-M55

[rascani]

Run MobileNetV2 and/or MV3 inference on Alif E8 board using the CMSIS-NN backend (CPU-only, no Ethos) and benchmark against tflite-micro. This is a key KPI target for demonstrating the CMSIS-NN on real hardware.

• MV2 or MV3 model exported and quantized (INT8) for Cortex-M55
• Inference runs successfully on Alif E8 hardware
• Latency and memory measured and compared to tflite-micro on same hardware

cc @digantdesai @SS-JIA @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @psiddh @AdrianLundell

[rascani]

All experiments done using the Alif fork of the MLEK and using the Alif E8 HP Cortex-M55. Memory arenas were all located in SRAM, and model binaries were stored in MRAM. Minor modifications were made to ensure both frameworks used the same tick source.

Versions
tflite-micro: 6ce10eed9ada8e7a7eb4a6210235b14d241f2a36
executorch: v1.1.0
cmsis-nn: v7.0.0

Export

• For tflite-micro, tf.keras.applications.MobileNetV2 was converted with the TFLite converter, with int8 ptq and input/outputs.
• For executorch, the torchvision.models.mobilenet_v2 was exported with the aot_arm_compiler.py. input/output remained in floating point, which is evidenced by the quant/dequant layers bookending the model.

RAM (bytes)

Memory	ExecuTorch	tflite-micro	Delta %
Total	1,563,213	1,668,468	-6.3%
Arena	1,505,280	1,509,744	-0.3%
Runtime	53,469	158,724	-66.3%
Temp	4,464	0

Flash (bytes)

Memory	ExecuTorch	tflite-micro	Delta %
Model	3,995,776	4,013,552	-0.4%
TFLM Optimized Model*	3,995,776	3,707,856	+7.8%

• Created using tflm-model-transforms tool.

Load/Inference Stages

ExecuTorch Event	ExecuTorch cycles	tflite-micro Event	tflite-micro cycles	Delta %
Program::load_method	1,064,785	AllocateTensors	7,827,316	-86.4%
Method::execute	249,819,842	Invoke	271,781,600	-8.1%

Per-Op Comparison

ExecuTorch Event	ExecuTorch cycles	tflite-micro Event	tflite-micro cycles	Delta %
quantize_per_tensor.out	1,365,831
quantized_conv2d.out	13,754,047	CONV_2D	13,738,232	+0.1%
quantized_depthwise_conv2d.out	13,715,766	DEPTHWISE_CONV_2D	13,858,176	-1.0%
quantized_conv2d.out	4,915,937	CONV_2D	5,026,592	-2.2%
quantized_conv2d.out	24,249,462	CONV_2D	24,643,117	-1.6%
quantized_depthwise_conv2d.out	10,286,711	DEPTHWISE_CONV_2D	10,540,465	-2.4%
quantized_conv2d.out	2,791,555	CONV_2D	2,914,111	-4.2%
quantized_conv2d.out	10,329,014	CONV_2D	10,582,216	-2.4%
quantized_depthwise_conv2d.out	15,319,785	DEPTHWISE_CONV_2D	15,415,662	-0.6%
quantized_conv2d.out	3,501,427	CONV_2D	3,692,389	-5.2%
quantized_add.out	1,295,313	ADD	1,278,669	+1.3%
quantized_conv2d.out	10,328,587	CONV_2D	10,570,862	-2.3%
quantized_depthwise_conv2d.out	3,837,222	DEPTHWISE_CONV_2D	4,355,305	-11.9%
quantized_conv2d.out	1,136,532	CONV_2D	1,202,194	-5.5%
quantized_conv2d.out	3,463,769	CONV_2D	3,568,789	-2.9%
quantized_depthwise_conv2d.out	5,001,666	DEPTHWISE_CONV_2D	5,116,081	-2.2%
quantized_conv2d.out	1,359,732	CONV_2D	1,451,392	-6.3%
quantized_add.out	442,361	ADD	426,794	+3.6%
quantized_conv2d.out	3,461,586	CONV_2D	3,564,670	-2.9%
quantized_depthwise_conv2d.out	5,001,462	DEPTHWISE_CONV_2D	5,674,069	-11.9%
quantized_conv2d.out	1,362,463	CONV_2D	1,457,320	-6.5%
quantized_add.out	441,165	ADD	426,540	+3.4%
quantized_conv2d.out	3,461,450	CONV_2D	3,566,137	-2.9%
quantized_depthwise_conv2d.out	1,417,981	DEPTHWISE_CONV_2D	1,458,106	-2.8%
quantized_conv2d.out	664,555	CONV_2D	715,256	-7.1%
quantized_conv2d.out	2,204,116	CONV_2D	2,397,930	-8.1%
quantized_depthwise_conv2d.out	2,513,281	DEPTHWISE_CONV_2D	2,631,590	-4.5%
quantized_conv2d.out	1,170,751	CONV_2D	1,246,720	-6.1%
quantized_add.out	229,342	ADD	214,149	+7.1%
quantized_conv2d.out	2,206,123	CONV_2D	2,283,776	-3.4%
quantized_depthwise_conv2d.out	2,515,314	DEPTHWISE_CONV_2D	2,613,703	-3.8%
quantized_conv2d.out	1,174,027	CONV_2D	1,267,180	-7.3%
quantized_add.out	229,144	ADD	214,190	+7.0%
quantized_conv2d.out	2,206,855	CONV_2D	2,427,902	-9.1%
quantized_depthwise_conv2d.out	2,514,307	DEPTHWISE_CONV_2D	2,747,833	-8.5%
quantized_conv2d.out	1,178,809	CONV_2D	1,252,872	-5.9%
quantized_add.out	229,246	ADD	214,243	+7.0%
quantized_conv2d.out	2,204,248	CONV_2D	2,354,402	-6.4%
quantized_depthwise_conv2d.out	2,514,216	DEPTHWISE_CONV_2D	2,845,567	-11.6%
quantized_conv2d.out	2,631,631	CONV_2D	2,731,635	-3.7%
quantized_conv2d.out	6,197,251	CONV_2D	6,868,700	-9.8%
quantized_depthwise_conv2d.out	3,760,252	DEPTHWISE_CONV_2D	4,014,247	-6.3%
quantized_conv2d.out	5,206,315	CONV_2D	5,409,370	-3.8%
quantized_add.out	335,621	ADD	320,427	+4.7%
quantized_conv2d.out	6,220,624	CONV_2D	6,882,596	-9.6%
quantized_depthwise_conv2d.out	3,762,025	DEPTHWISE_CONV_2D	4,224,862	-11.0%
quantized_conv2d.out	5,213,275	CONV_2D	5,372,752	-3.0%
quantized_add.out	335,854	ADD	320,467	+4.8%
quantized_conv2d.out	6,200,389	CONV_2D	6,824,852	-9.1%
quantized_depthwise_conv2d.out	1,067,135	DEPTHWISE_CONV_2D	1,114,904	-4.3%
quantized_conv2d.out	2,147,021	CONV_2D	2,261,670	-5.1%
quantized_conv2d.out	3,196,400	CONV_2D	4,070,726	-21.5%
quantized_depthwise_conv2d.out	1,538,803	DEPTHWISE_CONV_2D	1,666,569	-7.7%
quantized_conv2d.out	3,855,691	CONV_2D	4,128,122	-6.6%
quantized_add.out	149,998	ADD	134,591	+11.4%
quantized_conv2d.out	3,183,487	CONV_2D	4,095,884	-22.3%
quantized_depthwise_conv2d.out	1,540,446	DEPTHWISE_CONV_2D	1,673,116	-7.9%
quantized_conv2d.out	3,855,349	CONV_2D	4,131,670	-6.7%
quantized_add.out	149,608	ADD	134,485	+11.2%
quantized_conv2d.out	3,203,353	CONV_2D	4,088,336	-21.6%
quantized_depthwise_conv2d.out	1,533,768	DEPTHWISE_CONV_2D	1,675,183	-8.4%
quantized_conv2d.out	7,658,326	CONV_2D	8,228,792	-6.9%
quantized_conv2d.out	7,967,278	CONV_2D	9,148,418	-12.9%
quantized_avg_pool2d.out	217,082	MEAN	9,872,444	-97.8%
_clone_dim_order.out	2,937
quantized_linear.out	2,265,887	FULLY_CONNECTED	2,243,810	+1.0%
dequantize_per_tensor.out	4,545
		SOFTMAX	174,653

Findings

• Model load time is significantly faster with ET due to AOT memory planning.
• Inference time is about 8% faster with ET for two reasons:
  • ET exported with a CMSIS optimized avg_pool2d operator when TFLM used a reference mean operator.
  • Convolutional layers ~14m cycles faster with ET than TFLM, despite using the same kernels. This was due to cache friendliness of ET's memory planning buffer selection combined with the E8 HP M55's 32 KiB DCache.
• There does appear to be more framework overhead with ET (400k cycles) than TFLM (13k cycles). This was measured by total inference time - sum(per operator time).
• Memory arena usage is identical when you add in temp allocator usage (which is planned work).
• Runtime memory allocation (ie, persistent object allocations) are significantly reduced for ET.
• The model binary sizes were quite similar, which is to be expected since they are dominated by the size of weights, but the tflm model optimization tool got measurable savings.

[AdrianLundell]

Very nice! Have you considered also measuring the accuracy difference with this setup? Even if pretty much the same operators are used I imagine the scales will differ somewhat as they are computed for the portable kernels in executorch vs. tflite reference kernels (which CMSIS-NN are bit accurate to) in tflite micro.

[rascani]

Very nice! Have you considered also measuring the accuracy difference with this setup? Even if pretty much the same operators are used I imagine the scales will differ somewhat as they are computed for the portable kernels in executorch vs. tflite reference kernels (which CMSIS-NN are bit accurate to) in tflite micro.

I was planning on leaning on the FVP for that. Would there be any reason to use real hardware instead?