Do not rely on AI chatbots to independently diagnose medical conditions or resolve legal disputes; verify primary citations before seeking care, litigating, or spending money

When provided with a complete, structured medical record, large language models identify the underlying illness over 90% of the time. However, when everyday users consult AI chatbots regarding their own symptoms, they diagnose conditions less accurately than people searching standard medical websites, and are no better at deciding whether to visit the emergency department. On legal questions, general-purpose LLMs hallucinate inaccurate citations in 58% to 88% of queries and readily adopt erroneous premises introduced by the user.

Zero financial cost. Verification requires a few minutes to half an hour to request the specific statutory section, clinical guideline, or academic citation from the model, and then consult the primary text directly. The habit to break is unquestioning obedience to fluent, confident model outputs.

In a 2026 preregistered randomized study published in Nature Medicine involving 1,298 UK laypeople, participants evaluated clinical scenarios requiring diagnostic and triage decisions. Participants assisted by leading AI models (GPT-4o, Llama 3, Command R+) were compared against a control group using standard web searches (primarily NHS.uk). When the complete medical case was fed directly to AI, the models correctly identified the condition in 94.9% of cases and selected the proper triage level in 56.3%. However, when everyday participants interacted with those same models, diagnostic accuracy plunged below 34.5% and triage accuracy fell below 44.2%. Control participants searching standard resources were 1.76 times more likely to identify the correct condition (95% CI 1.45–2.13) and 1.57 times more likely to identify red-flag conditions (95% CI 1.28–1.92). Chat logs revealed failure points at both ends: lay users frequently omitted critical clinical nuances, and users often dismissed the model when it did propose the correct diagnosis. In the legal domain, a Stanford empirical study (2024) evaluated general-purpose LLMs against federal court case law: ChatGPT-4 exhibited hallucinations in 58% of legal queries, while Llama 2 hallucinated in 88% of cases. When queries incorporated flawed legal assumptions, models sycophantically confirmed the user's premise. A follow-up 2025 study evaluating specialized commercial legal AI tools found hallucinations occurred in 17% to 33% of complex research queries.

Controversy: A 2023 JAMA study found that healthcare professionals rated AI responses to forum questions as higher quality and more empathetic than physician responses (78.6% preference). However, that study evaluated response tone and composition rather than patient comprehension or diagnostic outcomes. Benchmarks on static exam questions do not predict lay user performance. Model hallucinations stem from next-token training dynamics where guessing produces partial reward, while acknowledging uncertainty receives no reward. To audit AI outputs, ask the model to provide the exact title and article number of the statutory code, guideline, or peer-reviewed DOI, then verify the exact sentence in official databases. Prompt objectively ("What does statutory law specify in this situation?") rather than seeking affirmation ("Am I guaranteed to win this case?"). Critical red flags like sudden facial droop, unilateral arm weakness, or crushing chest pain require immediate emergency services, not chatbot triage (see Chapter 13, Rules 3 and 7). Legal aid options are outlined in Chapter 7, Rule 3. Benefit magnitude is categorized as Low due to surrogate accuracy endpoints. Primary beneficiaries are yourself and your family.

Bean AM, Payne RE, Parsons G, et al. (2026). Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 32(2), 609–615. https://doi.org/10.1038/s41591-025-04074-y；Dahl M, Magesh V, Suzgun M, Ho DE (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis, 16(1), 64–93. https://doi.org/10.1093/jla/laae003；Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE (2025). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 22(2), 216–242. https://doi.org/10.1111/jels.12413；Kalai AT, Nachum O, Vempala SS, Zhang E (2025). Why Language Models Hallucinate. arXiv:2509.04664（预印本，备注里那项）. https://arxiv.org/abs/2509.04664；反方：Ayers JW, Poliak A, Dredze M, et al. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine, 183(6), 589. https://doi.org/10.1001/jamainternmed.2023.1838