Can AI chatbots give reliable cancer treatment advice?

Many people now type cancer questions into AI chatbots such as ChatGPT. These tools answer in fluent, confident sentences, which can feel reassuring at a frightening time. Researchers have started to test how reliable those answers are.

The short answer from published studies is mixed. For general questions, such as common cancer myths or what a type of cancer is, chatbots were often accurate. For specific treatment recommendations, the picture was weaker. In a 2023 JAMA Oncology study, about one in three ChatGPT answers mixed guideline-based treatments with treatments that did not match the guidelines, and about one in eight included treatments that were not part of any recommended regimen.

The authors of that study concluded that chatbots are not a reliable source of cancer treatment information. Chatbots have changed since then, but no published study we found shows that a general chatbot can safely choose treatment for a real patient. Treatment decisions depend on the full medical record, and your oncologist makes them with you.

Key facts: what studies of AI chatbots and cancer treatment advice found

The main peer-reviewed findings, in brief:

  • JAMA Oncology 2023 (Chen and colleagues): ChatGPT gave at least one guideline-based treatment in every answer with a recommendation, but 34.3% also included non-guideline treatments
  • Same study: 12.5% of answers contained hallucinated treatments not in any recommended regimen
  • JAMA Oncology 2023 (Pan and colleagues): four chatbots answering top cancer searches gave good-quality information with no misinformation found, but at a college reading level and with little practical guidance
  • JNCI Cancer Spectrum 2023: ChatGPT answers on 13 common cancer myths were judged accurate in 96.9% of ratings, versus 100% for the US National Cancer Institute
  • JAMA Network Open 2023: for 10 fictional patients needing precision oncology, chatbot treatment options did not match expert quality, and ChatGPT gave 27 non-existent trial numbers out of 85
  • Nature Medicine 2026: members of the public using chatbots for medical scenarios did no better than those using their usual sources

Key takeaways: AI chatbots and cancer treatment advice

  • Chatbots were often accurate on general cancer facts but less reliable on specific treatment recommendations
  • The most worrying errors were right and wrong advice mixed in one confident answer
  • Chatbots can invent things, including treatments and clinical trial numbers, and can be out of date
  • Most studies were done in English, with US guidelines, on older chatbot versions
  • A chatbot can help you prepare questions; your oncologist decides on treatment using your full records

How does an AI chatbot answer a cancer question?

A chatbot such as ChatGPT is built on a large language model. It learns patterns from huge amounts of text and then writes an answer one word at a time, choosing words that are likely to fit. It does not check a medical record or look at your scans. Unless it is connected to a search tool, it also does not know about anything published after its training data was collected.

This is why an answer can sound expert but still be wrong. In the 2023 JAMA Oncology treatment study, the ChatGPT version tested had a knowledge cut-off of September 2021, so the researchers compared it with the 2021 guidelines rather than newer ones. Cancer treatment changes often, so an older model can miss newer options.

The World Health Organization (WHO) published guidance on these models in January 2024. It warned that they can produce false, inaccurate, biased or incomplete statements, and that people may trust automated answers too much. It named patients asking about symptoms and treatment as one of the uses that needs care.

The JAMA Oncology study: ChatGPT versus NCCN cancer guidelines

In August 2023, researchers led by Shan Chen and Danielle Bitterman published a research letter in JAMA Oncology. They asked ChatGPT, using the GPT-3.5 model, for treatment recommendations on breast, prostate and lung cancer. They wrote 26 descriptions of diagnoses and asked about each in four different ways, giving 104 prompts.

Board-certified oncologists compared the answers with the 2021 guidelines of the US National Comprehensive Cancer Network (NCCN). ChatGPT gave at least one recommendation for 102 of the 104 prompts, and every one of those included at least one treatment that matched the guidelines. But 35 of the 102 answers (34.3%) also recommended one or more treatments that did not match. Thirteen answers (12.5%) contained hallucinations, meaning treatments that were not part of any recommended regimen, such as localised treatment suggested for advanced disease, or targeted therapy and immunotherapy where they did not apply.

The authors noted that the mix of right and wrong advice was hard to spot, even for experts. They advised that clinicians tell patients chatbots are not a reliable source of treatment information.

ChatGPT and NCCN study numbers at a glance

From the JAMA Oncology research letter (Chen and colleagues, 2023):

  • 104 prompts: 26 diagnoses, each asked in 4 ways, for breast, prostate and lung cancer
  • 102 of 104 answers (98%) gave at least one treatment recommendation
  • 35 of 102 (34.3%) included at least one treatment that did not match NCCN guidelines
  • 13 of 104 (12.5%) included hallucinated treatments
  • The 3 oncologist reviewers fully agreed on 61.9% of scores, showing that guidelines themselves need interpretation
  • Asking explicitly for NCCN-based treatment gave fully guideline-matched answers for 73.1% of diagnoses, versus 57.7% to 65.4% for other wordings

How accurate are AI chatbots on common cancer questions?

General cancer questions are a different test from treatment advice. A second 2023 JAMA Oncology study, by Pan and colleagues, took the top five Google search questions for skin, lung, breast, colorectal and prostate cancer. The team put them to four chatbots, ChatGPT 3.5, Perplexity, Chatsonic and Bing AI, giving 100 answers. Quality was rated good overall, with a median score of 5 out of 5, and the reviewers found no misinformation. But the answers were written at about a college reading level, and they scored low on actionability, meaning they gave little clear guidance on what to do next.

A smaller study in JNCI Cancer Spectrum compared ChatGPT with the US National Cancer Institute's answers to 13 common cancer myths. Five experts rated ChatGPT's answers accurate in 96.9% of ratings and the Institute's in 100%. Reviewers noted that ChatGPT often used vague or hedging language.

Both groups of authors said chatbots should be a supplement, not a main source of medical information.

Do AI chatbots answer cancer questions better than doctors?

A 2024 JAMA Oncology study, by David Chen and colleagues, made headlines with this question. The team took 200 cancer questions posted on the Reddit forum r/AskDocs and the replies written there by verified doctors. They then asked three chatbots, GPT-3.5, GPT-4 and Claude, the same questions. Six oncology and palliative care doctors rated the answers without knowing who wrote them.

The chatbot answers were rated higher for quality, empathy and readability. For the best-rated chatbot, average quality was 3.56 out of 5 against 3.00 for doctors, and empathy was 3.62 against 2.43.

The limits matter. The doctors' replies were short forum replies, not answers from a patient's own oncologist who knows the case. No patients rated the answers, the rating tools had not been validated, and most questions were general rather than about specific treatment. The authors said more research is needed and that doctors should oversee chatbot answers for accuracy. A warm, well-written answer is not the same as a correct treatment plan.

Can AI chatbots suggest personalised or targeted cancer treatment?

Precision oncology means choosing treatment based on gene changes found in the tumour. It is one of the hardest areas for any adviser, human or machine. A 2023 study in JAMA Network Open, by Benary and colleagues in Berlin, tested four AI tools, ChatGPT, Galactica, Perplexity and BioMedLM, on 10 fictional patients with advanced cancer and gene changes.

The chatbot suggestions were compared with those of an expert doctor. Agreement was low, with F1 scores, a combined accuracy measure from 0 to 1, between 0.04 and 0.19. A molecular tumour board, the expert team that reviews such cases, could usually tell which options came from AI. ChatGPT also gave 85 clinical trial numbers, and 27 of them did not exist.

The chatbots did produce some helpful ideas, including two useful options the expert had not listed. The authors concluded that the tools were not yet suitable for routine treatment decisions in personalised oncology. They noted that a newer ChatGPT version scored better, with an F1 of 0.26, which is still far from expert level.

Types of errors AI chatbots make in cancer treatment advice

The studies above found these kinds of mistakes:

  • Hallucination: inventing treatments that are not part of any recommended regimen
  • Made-up references: clinical trial numbers or sources that do not exist
  • Mixing correct and incorrect advice in one answer, which is hard to spot
  • Outdated information from a training cut-off date, missing newer treatments
  • Vague or hedging wording that patients may misread
  • Different answers to the same question, depending on wording

Why is it hard to spot a wrong chatbot answer?

A chatbot writes in the same confident tone whether it is right or wrong. In the JAMA Oncology treatment study, wrong recommendations usually sat alongside correct ones, so the answer looked reasonable overall. The authors noted that even experts found these errors hard to identify.

A large 2026 study in Nature Medicine, by Bean and colleagues, looked at what happens when ordinary people use chatbots. It was a randomised study of 1,298 people in the UK, each given a medical scenario. When the researchers tested the chatbots alone, they identified relevant conditions in 94.9% of cases. But when members of the public used the same chatbots, they identified relevant conditions in fewer than 34.5% of cases and chose the right next step in fewer than 44.2%. That was no better than people who used their usual sources, such as internet search.

The authors found that people often gave incomplete information and that chatbots sometimes gave misleading answers. This study was about general medical scenarios, not cancer, but the lesson is similar: a good test score does not mean safe advice for real users.

Limits of the research on AI chatbots and cancer

What the existing studies cannot tell us:

  • Most studies tested older chatbot versions from 2022 to 2024; newer versions may perform differently, better or worse
  • Questions were mostly in English, and none of the studies here tested Indian languages
  • Benchmarks were mainly US guidelines, which may not match treatments available or affordable in India
  • Several studies used short, single questions rather than real back-and-forth conversations
  • Studies were small, with 13 to 200 questions or 10 fictional patients in the cancer studies
  • None tested whether chatbot use changed real patients' treatment or outcomes

How to use AI chatbots safely for cancer information

If you choose to use a chatbot, these habits lower the risk:

  • Use it to understand words and prepare questions, not to choose or change treatment
  • Never stop, start or change a medicine because of a chatbot answer
  • Check important facts with your oncologist or a trusted source such as a cancer centre or national cancer institute
  • Be wary of trial numbers, drug names or study results the chatbot gives; ask for the source and check it exists
  • Avoid typing your name, ID numbers or other personal details
  • For urgent symptoms, contact your hospital or call 112, or 108 for an ambulance

Why your oncologist makes the treatment decision

A cancer treatment plan depends on many details: the exact type of cancer, its stage, the pathology and biomarker results, earlier treatments, kidney and liver function, other illnesses, and what matters to you and your family. Many of these details are only in your full medical record, and some come from examining you. A chatbot sees only what is typed into it.

Oncologists also follow guidelines, but they adapt them to the person and to what is available locally. In the JAMA Oncology treatment study, even three expert oncologists fully agreed on only 61.9% of scores, which shows how much judgement is involved. Complex cases are often discussed at a tumour board, where several specialists review the scans and reports together.

The authors of several studies saw a role for chatbots in helping patients learn and prepare. That is different from deciding treatment. OncoVOIS, for example, explains reports in plain words and does not give treatment decisions. Bring what you have read, and your questions, to your oncologist.

AI chatbots and cancer information in India

The studies in this guide were carried out mainly in the United States, Canada, the United Kingdom and Germany. The questions were in English and the answers were compared with US guidelines such as NCCN, or with US National Cancer Institute information. The authors of the cancer myths study noted that answers to non-English questions may be less accurate.

This matters in India. Many families prefer to ask in Hindi, Telugu, Tamil, Bengali or another language, and we did not find published studies of how well chatbots answer cancer treatment questions in Indian languages. Treatments that a US guideline lists may also differ in availability, approval or cost in India, and a chatbot may not know which drugs are covered under a government scheme or insurance.

So an answer that sounds right for a patient in the US may not fit a patient in India. Your oncologist knows what is available at your hospital and what it may cost, and can explain options that suit your situation.

What happens next for research on AI chatbots in cancer care?

Researchers agree that more testing is needed before chatbots are used to give patients medical advice. The authors of the 2026 Nature Medicine study called for systematic testing with real human users before chatbots are used in public healthcare, because standard exam-style tests did not predict how they performed with people.

The WHO guidance from January 2024 made over 40 recommendations for governments, technology companies and healthcare providers. These include involving patients early in development and checking that models are accurate and safe.

In cancer care, the published authors call for studies on how chatbots fit into clinical work, data security, accuracy and effects on patients and doctors. Newer chatbot versions keep appearing, and a model that scores well today may behave differently after an update. For now, the evidence supports using chatbots, if at all, as a learning aid alongside your care team, not as a source of treatment decisions.

Questions to ask your doctor about AI chatbots and cancer treatment advice

  • I read this answer from an AI chatbot. Is it correct for my type and stage of cancer?
  • Is this treatment the chatbot mentioned approved and available in India?
  • Which guidelines does this hospital follow for my cancer?
  • Is my case discussed at a tumour board?
  • Which trusted websites or leaflets can I read to learn more?
  • Who can I call if something I read online worries me between visits?

Sources

1. Chen S, Kann BH, Foote MB, et al. Use of Artificial Intelligence Chatbots for Cancer Treatment Information. JAMA Oncology, 2023. https://jamanetwork.com/journals/jamaoncology/fullarticle/2808731 2. Pan, et al. Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About Cancer. JAMA Oncology, 2023. https://jamanetwork.com/journals/jamaoncology/fullarticle/2808733 3. Johnson SB, King AJ, Warner EL, et al. Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectrum, 2023. https://academic.oup.com/jncics/article/7/2/pkad015/7078555 4. Chen D, et al. Physician and Artificial Intelligence Chatbot Responses to Cancer Questions From Social Media. JAMA Oncology, 2024. https://pmc.ncbi.nlm.nih.gov/articles/11099835 5. Benary M, et al. Leveraging Large Language Models for Decision Support in Personalized Oncology. JAMA Network Open, 2023. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2812097 6. Bean, et al. Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 2026. https://research.bangor.ac.uk/en/publications/reliability-of-llms-as-medical-assistants-for-the-general-public-/ 7. World Health Organization. WHO releases AI ethics and governance guidance for large multi-modal models. WHO, 2024. https://www.who.int/news/item/18-01-2024-who-releases-ai-ethics-and-governance-guidance-for-large-multi-modal-models 8. SUNY Downstate Health Sciences University. Assessment of AI chatbot responses to top searched queries about cancer (news release). EurekAlert, 2023. https://www.eurekalert.org/news-releases/999276

Frequently asked questions

Can I trust ChatGPT for cancer treatment advice?

Published studies suggest caution. In a 2023 JAMA Oncology study, ChatGPT included at least one guideline-based treatment in every answer with a recommendation, but 34.3% of answers also suggested treatments that did not match NCCN guidelines, and 12.5% contained hallucinated treatments.

The authors concluded that chatbots are not a reliable source of cancer treatment information. A chatbot can help you understand words and prepare questions, but your oncologist decides treatment using your full records.

How accurate are AI chatbots for cancer questions?

It depends on the question. For general questions, studies found good accuracy: four chatbots answering top cancer searches gave no misinformation, and ChatGPT's answers on 13 cancer myths were judged accurate in 96.9% of ratings.

For specific treatment recommendations, accuracy was weaker, with right and wrong advice often mixed. Most studies used older chatbot versions, English questions and US guidelines, so results may differ for newer versions and for patients in India.

What is AI hallucination in cancer advice?

A hallucination is when a chatbot confidently states something that is not true. In the 2023 JAMA Oncology study, 13 of 104 ChatGPT answers included treatments that were not part of any recommended regimen for that cancer.

In another study, ChatGPT gave 85 clinical trial numbers for fictional patients, and 27 did not exist. Because hallucinations sound as confident as correct answers, check any drug, trial or study a chatbot mentions with your oncologist.

Are AI chatbots better than doctors at answering cancer questions?

One 2024 JAMA Oncology study found that chatbot answers to 200 cancer questions from a Reddit forum were rated higher for quality, empathy and readability than short replies written there by doctors.

Those doctors did not know the patients, no patients rated the answers, and the rating tools were not validated. A well-written answer is not the same as a correct treatment plan. Your own oncologist knows your full case and makes treatment decisions with you.

Can a chatbot tell me which cancer treatment is best for me?

No published study we found shows that a general chatbot can safely choose treatment for a real patient. In a study of 10 fictional patients needing precision oncology, chatbot treatment options did not reach the quality of an expert doctor.

Treatment depends on stage, pathology, biomarkers, earlier treatment, your general health and what is available locally. Your oncologist, often with a tumour board, weighs all of this. A chatbot sees only what you type into it.

Can AI chatbots find clinical trials for cancer?

Be careful. In a 2023 JAMA Network Open study, ChatGPT gave 85 clinical trial numbers for fictional cancer patients, and 27 of them did not exist. Another tool tested in the same study gave no made-up trial numbers.

If a chatbot mentions a trial, check that it exists on an official trial registry and ask your oncologist whether you might be eligible. Trial eligibility depends on detailed medical criteria that only your care team can check.

Is it safe to share my medical reports with an AI chatbot?

WHO guidance from January 2024 warned about cybersecurity risks to patient information with these AI models, and about people trusting automated answers too much. It is safer to avoid typing your name, ID numbers or other personal details into a general chatbot.

Even with full reports, a general chatbot can still make errors in reading them, as the studies in this guide show. Your oncologist explains what your results mean for you.

Do AI chatbots give correct cancer information for India?

The studies in this guide were done mainly in English, using US guidelines or US National Cancer Institute information. We did not find published studies testing cancer treatment answers in Indian languages, and the cancer myths study authors noted non-English answers may be less accurate.

A chatbot may also not know which treatments are approved, available or affordable in India, or covered by a scheme. Your oncologist knows what your hospital offers and can explain the likely costs.

This page is general information, not medical advice. OncoVOIS explains and helps you prepare; your oncologist decides about your care.