[request] Do we have plan to merge Kubernetes part to kubeflow/pytorch-operator?

[gaocegege]

cc @Jeffwan

I think we share the similar scope between pytorch-operator and elasticjob-operator. I am wondering if we can collaborate to support PyTorch and PyTorch elastic well.

[kiukchung]

Hi @gaocegege thanks for bringing this up! We've been looking into adding support for TorchElastic parameters to the existing Kubeflow pytorch operators and are planning to bundle TorchElastic in the pytorch docker image for the pytorch 1.7 release - which shouldn't be a blocker for the Kubeflow work but certainly makes things smoother for users.

We welcome collaborations! and can certainly use some help from someone who is knowledgeable with Kubeflow in making pytorch operators support elastic parameters.

[gaocegege]

@kiukchung Thanks for your reply! We are glad to support it in PyTorch operator, and we can help implement or propose it in Kubeflow community.

[gaocegege]

/cc @johnugeorge @andreyvelich @kuikuikuizzZ

[kiukchung]

That would be awesome! I think we can approach this from both sides. You from Kubeflow side and us from PyTorch/TorchElastic side.

We can help from torchelastic and pytorch side and review/expedite relevant PRs and prioritize any changes required on TorchElastic or PyTorch (in general Facebook side) user-facing APIs and documentation to make the work on your end smoother (e.g bundling torchelastic in the pytorch docker image for instance). Let us know if you have any specific asks.

Looking forward to seeing elastic support added to Kubeflow pytorch operators!

cc) @drdarshan, @chauhang, @Tierex, @jspisak

[gaocegege]

Yeah, we are going to have some action items these days and maybe we will have some specific asks when we are ready. Thanks for your help!

[jspisak]

+1 @gaocegege - thanks for reaching out on this! KubeFlow is a high priority platform for us so anything we can do to make it easier for users to leverage advanced features would be great, especially if we can collaborate to make it happen..

cc @chauhang

[Jeffwan]

@gaocegege @kiukchung I can write a proposal on it. It needs some interface changes. We can have some discussion by end of month.

[kiukchung]

sounds good @Jeffwan, looking forward to the proposal! Keeping this issue open to continue the conversation on the same thread...

[gaocegege]

@Jeffwan Thanks! Looking forward to it.

[gaocegege]

ping @Jeffwan

Any progress for the issue?

[kiukchung]

@gaocegege did you have some thoughts on how we can merge elasticjob-operator with pytorch-operator? FWIW torchelastic is being bundled with the PT docker (https://github.com/pytorch/pytorch/blob/master/Dockerfile#L55) in the next PT release so it should make things simpler to add an option to the existing pytorch-operator to wrap the entrypoint with python -m torchelastic.distributed.launch and set some launch args via env vars (I'm guessing this is why @kuikuikuizzZ asked for this feature in #128 (I've got an open PR for this, waiting for someone in my team to review it).

cc) @jspisak , @chauhang

[gaocegege]

cc @Jeffwan

[Jeffwan]

FWIW torchelastic is being bundled with the PT docker (https://github.com/pytorch/pytorch/blob/master/Dockerfile#L55) in the next PT release

That's a good news. operator will be only used for orchestration, it's user's responsibility to wrap entrypoint. I think operator side we can determine if that's a torch elastic job by checking args or envs. common logic will be similar, adding/removing pods will be allowed for elastic job. (it's not allowed for general torch jobs).

It's a little bit delayed, I was busy with testing-infra issue in kubeflow org. Currently presubmit jobs blocks all the PRs, it will takes some time to get all issues fixed.

What's the release schedule for torchelastic.0.2.1 and the pytorch version with torchelastic?