The Critical Role of Data Quality and Prompt Engineering

If you are new to the world of AI, it's important to understand the impact that data quality and prompting have on outcomes. A recent experiment following last week's San Antonio Breast Cancer Symposium (SABCS24) highlights how different AI systems handle real-world information needs. The results demonstrate that successful AI outcomes depend heavily on data quality and well-engineered prompting.

A Tale of Three AI Responses

When asked “What are the key opinions coming out of SABCS24 last week regarding the PATINA trial?  Please provide source links.”, three different AI systems produced notably different responses:

ChatGPT: The Limitations of Outdated Training Data

ChatGPT immediately acknowledged its knowledge cutoff date and provided general background information about the PATINA trial. While comprehensive in explaining the trial's structure and potential discussion points, it couldn't offer any actual insights from the conference. This illustrates a fundamental challenge with static training data: it becomes outdated for time-sensitive queries.  Here’s an excerpt from that response noting that ChatGPT did give us a detailed response on PATINA based on 2023 data.

“As of my knowledge cutoff in October 2023, I don't have real-time access to current events or conference data, including updates from SABCS (San Antonio Breast Cancer Symposium) 2024 or any specific trial outcomes presented there. However, I can help you find information on the PATINA trial and suggest a general framework for understanding the types of findings that might emerge at such conferences.

PATINA Trial Overview (pre-2024 context)

The PATINA trial (NCT03781063) is a Phase III, randomized clinical trial designed to evaluate the combination of palbociclib, an oral CDK4/6 inhibitor, with letrozole in patients with HR-positive, HER2-negative breast cancer. Specifically, it focuses on the extended use of palbociclib, aiming to determine whether extending the use of palbociclib for longer durations after initial response can improve progression-free survival (PFS) compared to the standard duration of treatment.”

KOLPulse: The Power of Specialized Data

In contrast, KOLPulse delivered specific, recent insights from the conference, including direct quotes from key opinion leaders and their social media engagement. Its response highlighted actual findings presented at SABCS24, demonstrating how specialized data sources can dramatically improve AI output quality.

“The PATINA trial has generated significant discussion at SABCS24. Here are some of the key opinions:

 

1. Paolo Tarantino highlighted that adding palbociclib to maintenance endocrine therapy after first-line trastuzumab, pertuzumab, docetaxel induction for HR+/HER2+ metastatic breast cancer significantly improved progression-free survival (PFS) by 15 months, marking it as "Amidst the excitement at #SABCS24, the practice-defining impact of the #PATINA trial is the highlight."
   ◦ https://x.com/PTarantinoMD/status/1867270082625196056
2. Erika Hamilton, MD suggested that chemo induction doesn’t significantly add to the progression-free survival compared to Palbociclib, advocating for its place in treatment strategy without overdue toxicities.
   ◦ https://x.com/ErikaHamilton9/status/1867234802593849710
3. Ilana Schlam stated that the addition of palbociclib provides an amazing benefit with a significant improvement in PFS and practice-changing results for HR+/HER2+ metastatic breast cancer, with a focus on the safety and efficacy of new treatment regimens.
   ◦ https://x.com/IlanaSchlam/status/1867267431405678599
4. Sara Tolaney emphasized the significant PFS advantage and discussed the enduring quality of the findings and robust results that are changing current treatment practices.
   ◦ https://x.com/stolaney1/status/1867267861036904636

Overall, the consensus at SABCS24 is that the PATINA trial results are practice-changing and demonstrate a substantial improvement in progression-free survival, highlighting the importance of palbociclib in the treatment regimen for metastatic HR+/HER2+ breast cancer.”

Perplexity: The Middle Ground

Perplexity's response showed an interesting middle ground. Perplexity was much more current than ChatGPT and offered deeper findings on the trial results and discussions relative to the conference last week.  

Here’s what perplexity answered:

“The PATINA trial results presented at SABCS24 last week generated significant excitement and discussion among oncologists. Key opinions emerging from the conference include:

1. Practice-changing results: Many experts view the PATINA trial as potentially practice-changing for HR+/HER2+ metastatic breast cancer treatment36.
2. Unprecedented outcomes: The addition of palbociclib to anti-HER2 therapy and endocrine therapy resulted in a remarkable 15.2-month improvement in median progression-free survival (PFS)37.
3. New standard of care: Some oncologists suggest that palbociclib combined with anti-HER2 and endocrine therapy may represent a new standard of care for HR+/HER2+ advanced breast cancer7.
4. First large Phase 3 study: PATINA is noted as the first large Phase 3 study to demonstrate the benefit of CDK4/6 inhibition in HR+/HER2+ metastatic breast cancer15.
5. Toxicity profile: The safety profile was generally consistent with previous palbociclib studies, with manageable side effects56.
6. Future implications: Discussions arose about the potential to bypass chemotherapy for some patients and move directly to combination therapy with palbociclib4.
7. High-performing control arm: Dr. Sara Hurvitz highlighted that the recruitment window excluded patients with rapid progression, leading to a high-performing control arm3.

These opinions reflect the significant impact the PATINA trial results are expected to have on the treatment landscape for HR+/HER2+ metastatic breast cancer.”

When following up on specific KOLs, here’s Perplexity’s response:

“Several key opinion leaders (KOLs) actively discussed the PATINA trial results on social media during and after SABCS24. Notable mentions include:

1. Dr. Otto Metzger: As the Principal Investigator of the PATINA trial, he highlighted the significant 15.2-month improvement in median progression-free survival (PFS) with the addition of palbociclib2.
2. Dr. Erika Hamilton: She expressed excitement about the PATINA results, noting the 15-month improved PFS when palbociclib is added to endocrine therapy and HER2 antibodies in the first-line maintenance setting78.
3. Dr. Sara Hurvitz: While not directly quoted, she was mentioned as having discussed the high-performing control arm in the PATINA trial1.
4. Dr. Giampaolo Bianchini: Although not specifically discussing PATINA, he was noted as one of the most brilliant discussants at SABCS24, indicating his active participation in the conference discussions1.
5. Dr. Danso: Presented a summary and algorithm for ER+ Breast Cancer treatment at the year in review session, which likely incorporated PATINA findings1.

These KOLs were among the most active in discussing the PATINA trial results and their potential impact on HR+/HER2+ metastatic breast cancer treatment strategies.”

At first glance, Perplexity’s response contained good and timely information but if you notice most of the individuals mentioned were presenters with the exception of the remark on Dr. Erika Hamilton.  It didn’t actually address the question in our prompt.  We asked what the key opinions were about those who presented, not a summary of what the presenters shared.  After numerous follow-up prompts, I couldn’t delineate in the response the difference between KOL opinions vs publications and the names were often missing.

Key Lessons for AI Implementation

This comparison reveals several critical insights:

1. Data Recency Matters: The stark contrast between ChatGPT's inability to discuss current events and the other systems' detailed coverage demonstrates why data freshness is crucial for certain applications.
2. Domain Expertise Counts: KOLPulse's ability to surface relevant expert opinions shows the value of specialized, domain-specific data sources.
3. Prompt Engineering Is Critical: The quality and specificity of responses varied significantly, highlighting how different systems interpret and respond to the same prompt based on their underlying architecture and training.

Looking Forward

As organizations implement AI solutions, these examples underscore the importance of carefully considering:

• The freshness and updating mechanism of training data
• The specificity of data sources to your domain
• The design of prompts and interaction patterns

When choosing an AI solution, it’s not just about having the latest model or checking the check box that the solution is AI enabled– it's about having the right data, the right prompting strategy, and the right implementation approach for your specific needs.