Build Data-Driven Testing & Quality Assurance Infrastructure

[pizofreude]

Overview

This issue proposes the development of a comprehensive, data-driven testing and quality assurance infrastructure for FEAScript-core, focused on scientific computing reliability, performance, and maintainability.

Goals

• Implement automated performance benchmarking with systematic data collection
• Establish regression testing pipelines with numerical result validation
• Design and integrate memory usage monitoring and reporting systems
• Develop continuous integration analytics for tracking code quality and test coverage

Proposed Tasks

1. Performance Benchmarking

   • Create benchmarks for solvers, mesh generators, and visualization routines
   • Collect and store performance metrics (e.g., runtime, peak memory usage, result accuracy)
   • Automate benchmark runs in CI

2. Regression Testing

   • Setup test suites for key algorithms (e.g., solvers, mesh importers)
   • Validate numerical results against known benchmarks and previous releases
   • Report and visualize regression trends in CI

3. Memory Usage Monitoring

   • Integrate lightweight memory profiling tools for both CPU and WebGPU workflows
   • Store historical memory metrics for large mesh runs
   • Alert on excessive memory growth or leaks

4. Continuous Integration Data Analytics

   • Collect and visualize code coverage, test pass rates, and performance trends
   • Integrate analytics dashboards (e.g., with GitHub Actions, Codecov)
   • Use data insights to guide future development and refactoring

Impact

This initiative will greatly improve the reliability, efficiency, and scalability of FEAScript-core, helping contributors maintain high scientific and technical standards as the project evolves.

[nikoscham]

Hey @pizofreude! This is a really solid roadmap. Thanks for that. I think we will need to break this into smaller pieces. Then, I think we should start with the low-hanging fruits. Do you want to be assigned to this issue?

[pizofreude]

@nikoscham agreed, would you kindly add my issue as roadmap instead and create new issues broken into smaller pieces? Then, I'll close this issue and look up those optimized issues. Feel free to assign me to them as well.

[nikoscham]

@nikoscham agreed, would you kindly add my issue as roadmap instead and create new issues broken into smaller pieces? Then, I'll close this issue and look up those optimized issues. Feel free to assign me to them as well.

Thanks for your response @pizofreude. I will create sub-issues for every task, one by one.

[nikoscham]

Important (help wanted)
I’m running verification cases based on The Standard NAFEMS Benchmarks (NAFEMS P18).
Does anyone know of an open-source equivalent dataset that captures the same benchmark cases (e.g. 2D heat transfer with convection)?