Implement validation test infrastructure

[bghgary]

• Be able to see the tests passing and what tests are currently passing
• CI to run tests for as many as platforms as possible

[CedricGuillemet]

I did some tests with the Linux port and it went pretty well.
It's possible to create a window and render using opengl thanks to the on-demand offscreen window manager. To start it in the CI:

  - script: |
      export DISPLAY=:99
      Xvfb :99 -screen 0 1600x900x24 &
      sleep 3

Then run the executable. That's basically what is done with bjs validation tests when starting a browser.

Publishing the result is easy as well.

  - task: PublishBuildArtifacts@1
    inputs:
      artifactName: 'Rendered Pictures'
      pathtoPublish: 'pathToGeneratedImages'
    displayName: 'Publish Tests Results'

the CI pathtoPublish supports files and directories.

It will zip the content and list it the artefact list that you can download.

ValidationTests App can save generated images and error images in a particular. We can publish both.

For the Direct3D WARP, it needs some changes in bgfx for creating a special device and related swapchain. I think we need to try that on a smaller sampler (just create the device/swapchain, ensure everything properly initialized) to run on the CI. I don't know if it works without a window. And the CI doesn't provide a WindowManager. So ensuring it works before starting a PR on bgfx is a good thing.

For Metal, I'm still investigating...

[CedricGuillemet]

Android

CI can run Android Simulator with a valid opengles 3.0 context.

Renderer: Android Emulator OpenGL ES Translator (Google SwiftShader)
Version: OpenGL ES 3.0 (OpenGL ES 3.0 SwiftShader 4.0.0.1)
GLSL version: OpenGL ES GLSL ES 3.00

The emulator runs with the validation tests mandatory resolution (600x400). Tested for now with the Playground and experience.js to render the cube. Screenshot is made with ADB.

Next step is to create a new ValidationTests project. Save screenshots to the emulator folder (/local/tmp) then use ADB to move the rendered and error pictures to a publishable directory.
It's possible to take advantage of GetWorkingDirectory in Testutils for special folder.
Then, in the pipeline, poll still using ADB to know when the activity is closed. Appropriate functions commented in .yml.
Is it possible to get an errorcode from Activity or app when exiting?

MacOS

I did some tests on Metal and it's not possible to get a default device on the CI.

iOS

iOS simulator are installed with the MacOS CI images. I didn't find any documentation on how to do some tests with it. Not sure it's possible to render something with it considering the impossibility to create a Metal Device.
It seems the best way to test it on iOS is thru appCenter. Appsigning will have to be resolved before testing it

Get temp folder on simulator using

func NSTemporaryDirectory() -> String

on CI, get simulator files using:

lastUsedSimulatorAndApplication=`ls -td -- ~/Library/Developer/CoreSimulator/Devices/*/data/Containers/Data/Application/*/ | head -n1`

'tmp' directory is a sub folder.
Copy everything to 'Results' folder and let the publish do the rest.

So far, I can run an ios 13.3 simulator, install the freshly compiled app and run it(?). I don't get anything when doing a screenshot. I retrieved the logs and it looks like it's not behaving properly with background (?)

2020-04-23 13:12:28.702221+0000 0x7774     Default     0xc7d9               2752   0    runningboardd: (RunningBoard) [com.apple.runningboard:assertion] Acquiring assertion targeting application<com.BabylonNative.Playground.iOS> from originator [application<com.BabylonNative.Playground.iOS>:2885] with description <RBSAssertionDescriptor; "Shared Background Assertion 0 for com.BabylonNative.Playground.iOS"; ID: 2752-2885-73; target: 2885> attributes = {
    <RBSLegacyAttribute: 0x7fdd5943b3c0; requestedReason: FinishTask; reason: FinishTask; flags: PreventTaskSuspend>;
    <RBSAcquisitionCompletionAttribute: 0x7fdd5962e6b0; policy: 0>;
}

I didn't see any "metal not present" error so keeping fingers crossed.
I didn't manage to get the application log (from NSLog), might be a good thing to do it.
Main change to the CI is that scrip. Also take care of sdk for the xcode task

  - task: Xcode@5
    inputs:
      xcWorkspacePath: 'buildiOS/BabylonNative.xcodeproj'
      scheme: 'Playground'
      sdk: 'iphonesimulator'
      useXcpretty: false
    displayName: 'Build iOS'
  - script: |
      mkdir Results
      xcrun simctl list
      xcrun simctl boot A73F7BD9-4151-48AF-B290-0B054DCB5286 # iphone 11max 
      xcrun simctl spawn booted log stream > Results/log.txt &
      xcrun simctl install booted buildiOS/Apps/Playground/Debug-iphonesimulator/Playground.app
      xcrun simctl launch booted com.BabylonNative.Playground.iOS
      xcrun simctl io booted screenshot Results/screen0.png
      sleep 1
      xcrun simctl io booted screenshot Results/screen1.png
    displayName: 'Test iOS Simulator'

Win32

WIP. I'm doing a win32 console app, without window and testing on CI in debug build to get bgfx debug output. So far, it looks like it's possible to get a context with these backends:

D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp : 1544 D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1544): BGFX Supported renderer backends (6):
D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1547): BGFX 	 - Noop
D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1547): BGFX 	 - Direct3D 9
D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1547): BGFX 	 - Direct3D 11
D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1547): BGFX 	 - Direct3D 12
D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1547): BGFX 	 - OpenGL 2.1
D:\a\1\s\Dependencies\bgfx.cmake\bgfx\src\bgfx.cpp (1547): BGFX 	 - Vulkan

Some warnings on compressed texture formats but nothing serious.
bgfx init and validation tests work fine.

Linux

Headless OpenGL initialization works fine and images are renderer and saved. Remaining issue is a proper OpenGL shader compilation pipeline.

[CedricGuillemet]

Android Update

I managed to get result images exported as an artifact
https://dev.azure.com/babylonjs/ContinousIntegration/_build/results?buildId=5432&view=artifacts&type=publishedArtifacts
2 problems remaining:

• I can't find a way to check when the activity is terminated. I have a little bash script that polls the app pid but it doesn't seem to work
• Get the error code to b exposed to the CI. An app doesn't have an errorcode. I have to find another way to get the comparison result. On the same trend, I can't tell when the app crashes with less than expected images ouput.

[CedricGuillemet]

Current State:

• Win32 : Works as expected
• Linux : Works but disabled because of dead lock
• MacOS: Project done for local testing. Screenshot feature is unusable (bgfx?)
• iOS: Works for local testing as CI doesn't support Metal
• Android: Project done. A second PR is needed to get it running on CI Simulator

[CedricGuillemet]

I'm adding a note here to check for sharing more code with bjs. the validation_native.js should be factored to use the same logic script.

[thomlucc]

related to #1073