Performance difference issue

[Jeffery-MIC]

Hi, thank you so much for this astonishing work. I really appreciate your efforts.
I have a question regarding the performance difference.
After installing the environment by running

pip install -r requirements/dev_3.11.txt

I successfully finish running the rxrx3_core_benchmarks_openphenom.ipynb without doing any modification.

However, I find the performance of gene-gene interaction recall shown in the notebook as

But the official paper reported

I wonder what step did I miss or I should do some corresponding modification to the evaluation steps to obtain the identical performance from OpenPhenome S/16.

Thank you for your time reading this.

[safiyecelik]

Hi Jeffery,

Thanks for reaching out!
Am I understanding correctly that you are seeing ~.02 higher recall in your own runs than what was reported in the paper?

[Jeffery-MIC]

Yes, I found around .02 gap between the performance of the released weight and the reported paper in terms of OpenPhenom S/16. I didn't do any modification to the code so I wonder how can I obtain identical performance or which number should I be following. Thank you so much for your prompt response.