-
Notifications
You must be signed in to change notification settings - Fork 367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add 1K and 5K RayCluster/RayJob scalability test results #2218
base: master
Are you sure you want to change the base?
Conversation
- | | ||
#!/bin/sh | ||
|
||
ray job logs $RAY_JOB_SUBMISSION_ID --address=http://$RAY_DASHBOARD_ADDRESS --follow || \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a temporary workaroud until ray job submit
doesn't error on already existing submission IDs
apiVersion: apps/v1 | ||
kind: DaemonSet | ||
metadata: | ||
name: ray-image-preloader |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For 1K and 5K tests, a lot of time was spent on pulling Ray images so I added pre-loader daemonset that runs before the tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! Would you mind adding a README to briefly document the benchmark results we have? Thanks!
@@ -0,0 +1,17 @@ | |||
#!/bin/bash |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both 1000-raycluster
and 5000-raycluster
directories have wait-for-raycluster.sh
. Could we unify them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah we could, can I do it in a follow-up when we introduce the 10K tests?
command: | ||
- "bash" | ||
- "1000-raycluster/wait-for-rayclusters.sh" | ||
- name: Wait for pods to be running |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does this mean? Does it execute after "Wait for RayClusters ready"? If a RayCluster is "ready", all Pods in the RayCluster should be running and ready. It is a bit weird for me to measure "pods to be running" after RayCluster is ready.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, this step is not actually waiting for pdos to be ready, it's measuring the WaitForControlledPodsRunning
metric, which we capture after waitign for all RayClusters to be ready. I updated the name to better reflect what this is doing.
Method: WaitForControlledPodsRunning | ||
Params: | ||
action: gather | ||
- name: Measure pod startup latency |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is PodStartupLatency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The definition is here: https://github.com/kubernetes/community/blob/master/sig-scalability/slos/pod_startup_latency.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that you mention it, we're probably not that interested in startup latency because it excludes image pull times and init containers. I'll remove this in a follow-up PR
kind: RayCluster | ||
labelSelector: app.kubernetes.io/created-by = kuberay-operator | ||
operationTimeout: 120s | ||
- Identifier: JobLifecycleLatency |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does JobLifecycleLatency
mean? Does it refer to the submitter K8s Job?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, it's measuring time for Job to complete. In our case it measures submitter job time to completion, which roughly represents how long the training job took to complete
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
I updated the README with some references on how to understand the results |
Why are these changes needed?
Per #2069, adds 1K and 5K RayCluster / RayJob test results
Related issue number
#2069
Checks