JS-Remote-Test Requirement

[glistening]

By @zherczeg's request, I would like to write the requirement for js test framework.

Unifying the test framework

• Both of IoT.js and JerryScript
  ( It is already supported from very initial version )
• Both of remote and local
  ( This unifying idea is raised by @yichoi Testrunner refactor to support multiple targets iotjs#1212 )
• Multiple board/os (e.g. RPI/Raspbean, Artik053/TizenRT, ...)
  ( Improving remote test runner to cover new os/boards iotjs#1073 )

Minimizing the maintenance cost

• Minimize duplicated code to IoT.js and JerryScript repository

System I/O

• Cover system I/O also after related sensors are delivered, this item can be started

Etc

• Remove circular dependencies that exists on current structure
• We would like to choose IoT.js by commit ID or by indicating path to IoT.js/JerryScript

[glistening]

@hs0225 @daeyeon Could you check the requirement ? (missing requirement or invalid requirement, ... )

[zherczeg]

@rtakacs please explain how #36 will satisfy these requirements

[rtakacs]

Status of the main points (with a minimal description):

• Support IoT.js and JerryScript as main applications.
• Support multiple targets (device-os pairs).
• Add js-remote-test to IoT.js as a submodule.
  Not supported yet. First, the circular dependencies should be removed. This can be done by the new architecture (introduced in Change the current architecture #36). What's more, there are other cross dependencies (discussed in #28) that also could be eliminated with the new architecture.
• System I/O testing.
  Not implemented yet.
• Minimize duplicated code in IoT.js and JerrScript.
  Since the js-remote-test will be used as a submodule, I think there shouldn't be any duplicated codes.
• Remove circular dependencies.
  The new architecture could remove circular dependencies.
• Define commit hash (or tag) for the testrunner.
  I think, the user should change the branch or the commit in iotjs (or jerry) that should be tested.

Bonus:

• Download dynamically the dependencies.
  Not implemented yet. Currently, there is a init.sh script that downloads all the projects:

  • iotjs
  • jerryscript
  • nuttx
  • nuttx-apps
  • stlink
  • freya
  • tizenrt

  If the user want to test only one target, it's unnecessary to have all the dependencies. This also can be eliminated by using the new builder structure (introduced in Change the current architecture #36). After the target is specified (by a --target option), only the necessary dependencies should be downloaded.

@zherczeg @glistening @hs0225 What do you think? Can I start to modify the architecture in js-remote-test?

[glistening]

@rtakacs Yes, please. I've read your new architecture proposal (#36).
@hs0225 Do you have any objection or other idea?

[hs0225]

It looks good. In addition, I suggest one more thing.

• How authorized people publish test results from anywhere.

[rtakacs]

@hs0225 There are more tables in the Firebase database for all the targets. If someone wants to maintain a target (run tests on device, and upload that), we can send an email - password pair for him or her. (Please note that the email doesn't need to be valid.) These data should be set by environment variables:

$ export FIREBASE_USER=<e-mail>
$ export FIREBASE_PWD=<password>

After that, the js-remote-test is able to upload the data (--public option is needed).

[glistening]

@rtakacs For system I/O tests, there are 2 items:

1. test automation with system I/O
2. coverage measuring automation with system I/O

Please contact @hs0225 before beginning system I/O test automation. Hosung may have an idea or design.