TL;DR
At the American Society for Reproductive Medicine annual meeting, researchers reported that physicians rated ChatGPT’s answers to common IVF questions lower than ASRM educational materials for accuracy, completeness, safety and overall quality. The chatbot scored similarly on clarity, but half of its answers included at least one unsupported claim, and 4.2% contained misinformation.
Physicians rated ChatGPT’s answers about IVF as less accurate, complete and safe than patient education materials from the American Society for Reproductive Medicine, according to a comparative analysis presented at the society’s annual meeting. The chatbot’s clarity scores were comparable to ASRM’s, but half of its responses contained an unsupported claim, and 4.2% contained misinformation.
Nicole Fischer, MD, MPH, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, presented the findings. Her team used the free GPT-5.6 Luna to answer common questions about IVF and assisted reproductive technology, then compared the responses with two ASRM educational documents. Four physicians rated the material on a five-point scale, where five was the highest score.
ASRM materials scored 5.00 for accuracy and quality, 4.88 for completeness and safety, and 4.63 for clarity. ChatGPT scored 3.98 for accuracy, 3.40 for completeness, 2.92 for safety, 3.38 for quality and 4.60 for clarity. The team also checked whether responses covered concepts listed in ASRM’s ReproductiveFacts.org materials: ASRM documents met all checklist items on average, compared with 94.1% adherence for ChatGPT.
The team ran each prompt three times, producing 12 ChatGPT responses for the comparison. In a separate assessment, it asked the chatbot to estimate IVF success rates for ages 20 through 45. Researchers reported that the answers generally aligned with CDC and Society for Assisted Reproductive Technology data and reflected declining birth rates with age, but the model overestimated live birth rates at age 35 and underestimated them at ages 37 and 40.
Clear Answers Can Still Miss Key Details
The findings matter because people may use chatbots to prepare for fertility treatment or interpret information about its risks and likely outcomes. Fluent wording is not the same as a complete or reliable answer: a response can be easy to read while leaving out relevant information or including unsupported claims. That gap may be hard for a patient to spot without medical knowledge or a source to check against.
The study does not establish how patients act on chatbot answers or whether the responses affect treatment choices. It also does not show that every ChatGPT answer about IVF is wrong. Instead, its comparison suggests that, in the tested prompts and model version, clarity alone did not match the stronger ratings the physicians gave ASRM’s reference materials. The authors’ message for clinicians was to ask patients whether they use chatbots, encourage them to request sources, and direct them to trusted educational resources.
How Researchers Compared IVF Answers
The analysis had two parts: a comparison of ChatGPT responses to common questions about the IVF process and risks, and a review of the model’s age-based success estimates against CDC and SART data. Researchers assessed accuracy, completeness, clarity, safety and quality, as well as checklist coverage, red flags and readability. Physician ratings were based on a five-point scale; they were not measurements of patient outcomes.
Readability differed by prompt style. ChatGPT answers to conversational prompts were at about an eighth-grade reading level, while answers to generic prompts were closer to a 10th-grade level. The ASRM documents were roughly at a 12th-grade level. Fischer said this suggests the way a patient phrases a question can affect how readable the response is. The study tested one chatbot version, so it does not represent all AI systems or later versions.
“The key point here is that fluent, confident wording can coexist with omissions, outdated information, and errors.”
— Nicole Fischer, MD, MPH, as quoted in MedPage Today
Limits of One Small Comparison
The researchers described the study as limited by its small sample and use of a single version of one large language model. Some assessments were subjective or descriptive, and the analysis did not measure patient-level outcomes. The results therefore cannot establish how often patients encounter errors in ordinary use, whether other models perform differently, or whether the findings apply to later ChatGPT versions.
The supplied report does not identify the exact questions and full responses used in the analysis, or provide uncertainty ranges for the ratings and estimates. It is also not clear how well the chatbot’s success-rate estimates would perform for individual patients, whose circumstances may differ from population data. The results are a comparison of tested answers, not a clinical evaluation of IVF care or a forecast for any one patient.
Testing Newer and Specialized Models
Fischer said newer and specialized systems, including ChatGPT Health, merit further testing. Larger studies could compare multiple models and versions, use a wider range of patient questions, and assess whether answers remain accurate and safe when prompts are phrased in different ways. Patient-focused research would also be needed to learn whether chatbot information changes understanding or decisions.
For now, the researchers encouraged clinicians to ask patients how they use AI tools, show them how to request citations, and point them to established sources such as ASRM’s patient education materials. The report does not state when follow-up testing will be published. Patients considering IVF can use chatbot responses as a starting point for questions, but should verify medical information with a qualified clinician.
Key Questions
Did ChatGPT score as well as ASRM on IVF answers?
No. In this study, physicians gave ChatGPT lower scores than ASRM materials for accuracy, completeness, safety and quality. Clarity scores were similar: 4.60 for ChatGPT and 4.63 for ASRM on a five-point scale.
How often did researchers find unsupported claims or misinformation?
Half of the ChatGPT answers contained an unsupported claim, and 4.2% contained misinformation. The researchers found no misinformation in the ASRM responses they assessed.
Were ChatGPT’s IVF success estimates accurate?
The estimates generally aligned with CDC and SART data and showed declining birth rates with age, according to the report. The model overestimated live birth rates at age 35 and underestimated them at ages 37 and 40. These findings do not predict an individual patient’s outcome.
Can this study tell patients whether to rely on ChatGPT?
The study tested a small set of responses using one version of one model and did not measure patient outcomes. Its findings support checking chatbot claims against trusted sources and discussing medical questions with a qualified clinician, rather than treating the results as a verdict on every AI response.
Source: medpagetoday.com
