Automate post-deployment smoke tests for releases

[Matt-Yorkley]

We should investigate/discuss the best way to automate smoke-tests.

[Matt-Yorkley]

@sauloperez we need to create RachelBot before the real @RachL goes on holiday

[Matt-Yorkley]

Rough ideas for this:

1. We can add a task in the deploy playbook that runs when the environment is production that tests the live homepage of the site after the deployment has finished. It can check that the response code is 200 and that some content we expect to be there is present in the HTML response. We can also include a message with the results of that check in the Slack notification, which would be quite nice. Obviously that's pretty basic, but it would take 5 minutes to implement.

2. I think Selenium can be configured to point to a live server instead of a local test server. We could potentially set up a Travis type build with a customised config, that runs through a subset of our existing feature tests (so we don't have to create a new set of tests from scratch, and they stay up-to-date). I'm not sure how difficult this would be.

[RachL]

Pleaaaase don't call it that wayy 🙀 🤣

We've spoken about other tools as well that are complementary to Selenium like Fitnesse here:
https://community.openfoodnetwork.org/t/seed-data-development-provisioning-deployment/910/5?u=rachel

I don't think we need to rush things before I leave for holidays. I can dedicate sometime on it Thursday and Friday, but I would really love to take the time to choose the right tool.

We need to have tools that enables us to grow this kind of tests coverage at a regular rhythm. And I would love it to be a tester task rather than a dev task. So this would be one of my criteria to choose the tool.

What do you think @lin-d-hop

[filipefurtad0]

Hey @Matt-Yorkley ,

I'm wondering if it would be possible to run some already existing specs directly on our staging or live servers. For example, if we set Capybara.app_host = pointing out to our servers perhaps we could cherry-pick some specs and adapt them for smoke tests.

Would this be an approach to address point 2 you mentioned?

[sauloperez]

theoretically yes @filipefurtad0 . Something that needs to explored. The adaptation work I see already is around the dummy data we create such as enterprises, users, orders, etc. Our specs already handle cleaning that up after the test execution but I'm sure there are issues we haven't anticipated.