About the Team:

OpenAI puts particular emphasis on large-scale deep learning. Our approach results in a small number of extremely capable models. Towards that end, we need to make a lot of difficult decisions based on incomplete data. The Science of Deep Learning team tries to fundamentally understand what is happening as our models scale and how to evaluate their performance.

About the Role:

Evaluation metrics allow us to understand what is happening under-the-hood and the progress of our models. We are looking for an engineer who wants to create the requisite systems/frameworks for making sense of these evaluations.

As a software engineer on the team, you will be responsible for:

• Creating the API specific to evaluations that will stand the test of time
• Designing the tooling and frameworks needed for cataloging all present/future evaluations
• Designing tooling for evaluation visualizations and analysis

OpenAI is a very small team. You will work directly with our best researchers and engineers to help make key decisions pushing our state of the art models towards previously unexplored regimes of AI capabilities. You’ll closely collaborate with the Science of Deep Learning team to refine our evaluation tools and methodology. This is a 0-1 effort and a huge area of investment for us. The ability to understand/analyze evals systematically will allow us to answer questions like, "How close are our models to AGI?"

About yourself:

• Good taste in API design, thinking through long-term success vs short-term happiness
• Excited by building tooling that accelerates the efficiency of our technical teams.
• Track record of excellent engineering skills across multiple domains
• Interested in understanding what happens under-the-hood of AI models
• Open to technical leadership down the road

Bonus Points if You:

• Have experience in statistical analysis, to draw correct conclusions from the sea of data we swim in
• Have experience in database design 
• Have data visualization experience