OpenAI Launches HealthBench to Benchmark AI in Healthcare Conversations

OpenAI has unveiled HealthBench, a pioneering open-source large language model designed specifically to benchmark AI performance in healthcare-related conversations. This initiative aims to rigorously evaluate whether AI models provide the most accurate and helpful responses to health inquiries, a critical step toward integrating AI safely and effectively in medical contexts.

HealthBench was developed in collaboration with 262 physicians from 60 countries, ensuring a diverse and comprehensive medical perspective. The model incorporates 5,000 realistic health conversations that span 26 medical specialties, including neurological surgery and ophthalmology, reflecting a broad spectrum of healthcare scenarios.

To assess AI responses, HealthBench uses a physician-written rubric where each criterion is weighted according to medical expert judgment. The rubric is then scored by GPT-4.1, an advanced AI model, ensuring an objective and consistent evaluation process. This method allows HealthBench to identify strengths and weaknesses in AI-generated health advice.

For example, in a scenario where a 70-year-old neighbor is found unresponsive but breathing, an AI model is prompted to provide emergency steps. HealthBench evaluates the response, highlighting correct actions such as calling emergency services and checking airway positioning, while also noting areas for improvement. The final score reflects the overall quality of the guidance, with this example receiving a 77% rating.

HealthBench supports 49 languages, including less commonly represented ones like Amharic and Nepali, making it a versatile tool for global healthcare AI evaluation. This multilingual capacity ensures that AI models can be tested for accuracy and cultural relevance across diverse populations.

Currently, OpenAI's own o3 reasoning model leads the benchmark with a 60% score, followed by Elon Musk's Grok at 54%, and Google's Gemini 2.5 Pro at 52%. These results provide valuable insights into the comparative strengths of leading AI models in healthcare applications.

The introduction of HealthBench marks a significant advancement in the responsible deployment of AI in healthcare. By providing a standardized, physician-informed framework for evaluating AI responses, it helps ensure that AI tools meet the high standards required for medical advice and patient safety.

As AI continues to integrate into healthcare workflows, tools like HealthBench will be essential for developers, healthcare providers, and regulators to monitor and improve AI performance. This ensures that AI not only supports clinicians but also safeguards patient wellbeing through reliable and accurate information.