iOS/MacOS/Android Tests infrastructure Steps

[CedricGuillemet]

Related : test infrastructure

I have a plan consisting in individual steps. Each step can be done by a different person. Most steps depend on the previous one.
Platform can be treated in parrallel: Linux vs MacOS/iOS vs Android.
Steps tagged as (local) can also all be done before replicating on self hosted agent.
We can work on (local) tasks until we receive the HW. Or Sergio will have to open a VNC access to his MacMini.
For steps regarding self hosted agent: document any change in the configuration so we can duplicate it on newer HW.

• Step 1:

• BN Linux validation is possible but CI test was unreliable. It should be put back in. Test it as an optional build test or find why it's dead locking (if the bug is still there).
  Success: Linux Validation tests running reliably on the CI.

• Step 2:

• MacOS validation tests (local)
  The project is done and IIRC it ran and did the tests correctly. I don't remember if the return status was possible or correct. Make sure it runs locally as expected (same as win32) and check the return value.
  Success: Local MacOS validation tests return a value indicating success or failure.

• Step 3:

• Add MacOS/iOS self hosted agent : https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser
  Only take care of building. Compare build duration between selfhosted and AzurePipelines.
  Success: Any CI build triggers MacOS and iOS building on the self hosted agent.

• Step 4:

• MacOS validation test on the self hosted agent
  If the return value is correct (failling or succeeding) and runs locally, it should be fine to run it on the self hosted agent.
  Change Azure Pipelines YML to enable validation tests
  Success: MacOS builds also trigger Validation tests on the self hosted Mac

• Step 5:

• MacOS validation test images artifact
  Unlike Win32/Linux, rendered images cannot be written anywhere. It can be outputed to Home folder. Check output directory.
  Success: Rendered images written in Home are uploaded as artifacts on the CI

• Step 6:

• iOS validation test (local)
  There is a bit of unknown here. IIRC, local VT test displays a popup indicating success or failure. More over, apps (on iOS or Android) can't be shut down. Only the system can do that. So, we can't check a return status code. I have no good solution here. Maybe a simple NSLog is enough.
  Success: With a script running locally on MacOS, upload and launch the VT app on the device, run it and get the result

• Step 7:

• iOS crash and Metal Validation Error detection (local)
  How to detect the app crashed or Xcode find a Metal Validation error? In case of a crash, we won't find a log (see previous step) and we can deduce something went wrong. When there is a MTL validation error, does it stop the app? Will Xcode freeze everything waiting for a user action? Can we have a stack trace of the MTL validation error?
  Force the app to crash to test it properly detects it ( (int*)*nullptr = 1337; somewhere in the code). Find a fix in the closed PR to repro a Metal Validation Error.
  Success: A crash or a Metal Validation Error is properly detected when running on the device

• Step 8:

• iOS Access to camera (local)
  Make sure accessing the camera is allowed automatically and won't open a blocking popup on the device. Create an AR session and close it after a few seconds then go to the next test. Create a Playground for that taks and add it to config.json. This is a temporary playground. A production test playground will have to be created in a later step. Goal is in only to check camera access is allowed.
  Success: getting access to camera is allowed and won't block CI

• Step 9:

• iOS Validation tests on the CI
  Everything should be in place to run VT on the self hosted agent. Update azure pipelines YML accordingly
  Success: iOS Validationt tests are performed and result is checked by the CI

• Step 10:

• iOS validation test images and logs (optional)
  Rendered and compared images are uploaded as artifacts on the CI. It's usefull to understand what's failing. Find a command to copy application folder from the iPhone to a local MacOS folder
  Success: Rendered images can be downloaded from the iPhone and uploaded as artifacts on the CI

• Step 11:

• Android build self hosted
  Same as MacOS and iOS, register Android Build to be performed on the self hosted agent. Compare build time. If it's not faster on the self hosted agent, then only build the ValidationTests app with it. Keep the rest on the CI to not overload the self hosted agent.
  Success: Android Validation Tests is triggerd by the CI to be built on the self hosted agent

• Step 12:

• Android validation tests locally with command lines (local)
  It's possible and easy to run the simulator/device with command lines and retrieve the log and generated image https://github.com/CedricGuillemet/BabylonNative/blob/AndroidValidationTestsCI/azure-pipelines.yml
  Write a small script that runs the built VT app on the device, get the end result (Same as iOS with a log?) and retrieve logcat and generated images.
  Success: A small bash script will run app and get results from the device

• Step 13:

• Android Access to the camera (local)
  Same as iOS. Check camera access is not blocking. IIRC it can be enabled automatically when running the simulator by command parameters. IDK if it's the same with adb. Maybe with some special entries in the manifest?
  Success: Android Camera access is allowed

• Step 14:

• Android Validation tests triggered by CI is tested on self hosted agent
  Simply update the Azure Pipelines YML to run the validation tests on the self hosted agent
  Success: Self hosted agent runs VT when a build is done on CI

• Step 15:

• XR playgrounds (local)
  Create a set of playgrounds that will be tested. I don't think image comparison is needed at this point. There are properties we can check (tracking is enabled, object is detected, ...). IMHO if we arrive at this point 99% of bugs and regressions will be detected. One playground per XR feature. Check with Raanan and Alex for list of features.
  Success: Multiple Playground are written, 1 per major feature.

• Step 16:

• XR playgrounds added in config.json
  Finish playgrounds so they can be properly added in config.json. Maybe the ValidationTests script will need to be modified so Tracking for example is compared with expectation.
  Success: XR playground runs on iOS and Android self hosted agent

[bghgary]

There are a bunch of different items in this issue that may have different priorities. We need to probably break this issue up into smaller manageable issues.