DEVPROD-9461: Move Query-related tests to replica-* variants instead of standalones and single-replicas #1257
Conversation
…of standalones and single-replicas
There was a problem hiding this comment.
Thanks @wqian94. This is changing a lot of tasks. From the perf perspective this looks good (and is what we want) but if possible can we get someone from query to also review this to make sure you're happy to shift all testing to replica sets?
Also there is no variant that runs with replica-80-feature-flag so I think you can remove all references to that.
| - standalone-all-feature-flags | ||
| - standalone-classic-query-engine | ||
| - standalone-sbe | ||
| - replica-80-feature-flags |
There was a problem hiding this comment.
I don't think we currently have any variants in sys-perf that run with replica-80-feature-flags
There was a problem hiding this comment.
I saw a bunch of tests with replica-80-feature-flags in the repo already. Do you know what's going on with those?
There was a problem hiding this comment.
I'm not sure - the only PR I could find related to it was https://github.com/10gen/dsi/pull/1637 but I couldn't find any that added/removed the variant and it wasn't removed as part of SERVER-89499. I can follow-up more on this if you think having a variant with the 8.0 FFs is necessary.
There was a problem hiding this comment.
I think we do want an 8.0 FFs variant. Lemme ask around Query and see who might know more.
There was a problem hiding this comment.
Now that we've committed to releasing 8.0 with the feature flags we ended up choosing, having an 8.0 feature flags variant isn't that useful. However, I'd advocate to leave the specifications intact for now and have the Product Performance team do the work to wholesale remove these and ensure that we have good substitutes intact.
I've created a ticket to track that effort here.
There was a problem hiding this comment.
Thanks @BlakeIsBlake for opening that ticket. As Blake said I'm happy for you to leave tests that have replica-80-feature-flags as is, and when our team gets to PERF-5857 we can easily switch them to a 9.0/all FF variant. Just bear in mind, that it may be slightly confusing that the Genny AutoRuns are implying it's running on an 8.0 FF variant when it actually isn't.
There was a problem hiding this comment.
Code changes look good but can we do some perf/noise analysis to validate that moving to replica sets still gives useful signal. Maybe you could use the variant to variant feature in the perf analyser to compare these tests x5 running on standalone vs x5 running on replica? (I think configuring and running all the FF, SBE, classic engine, etc. variants will be a pain so just the plain standalone and replica should be fine).
Also can we get someone from query on this PR?
|
Looks like SERVER-88741 is the ticket that added the |
|
Also, @alicedoherty the variant-to-variant comparisons seemed to spawn an excessive number of tasks. Since many of them failed, the perf analyzer is unable to create a comparison. I'm going to retry this week with fewer tasks and hopefully that's enough to convince us that the CoVs are acceptable. |
|
@BlakeIsBlake this is the PR in question. Do you think we should leave the |
|
Just a quick update, since I neglected to do so earlier: I'm rerunning the v2v analysis, now that a crash with the v2v analyses I was doing has been fixed. |
|
Analysis complete for classic. CoV seems slightly high for some tests, though? |
|
I don't think the SBE variant is available on master right now, so I can either compare older checkouts or just let it be for now. |
Thank you! The tests with very high CV on the 3-node replica sets look to be already very noisy on standalones. Although the percentages seem to have gone up for some, clicking into the actual values makes it look like the values are in the range of noise previously seen on standalones so I'm not too worried about them.
We have SBE Standalone ARM AWS 2023-11 and Query Engine (SBE) 3-Node ReplSet ARM AWS 2023-11 running on master right now - what issues were you having when scheduling them? I'd like to see a variant to variant analysis for the SBE 3-Node vs SBE Standalone (just like you did with Classic Engine) if possible. Thanks again @wqian94 for the analysis on this! |
|
The old SBE variant doesn't seem to be available for v2v comparison last I checked. I'll look again today. |
Jira Ticket: DEVPROD-9461
Whats Changed
Replace
standalone-classic-query-engine,standalone-sbe, and the correspondingsingle-replicavariants with the latestreplica-query-engine-classicandreplica-query-engine-sbevariants, respectively. Also clean up other Query-related tests to move away fromstandalones andsingle-replicas where possible.Patch Testing Results
Patch.