Amazon's new Health AI Chatbot is rife with potential for misuse — here's why I wouldn't trust it with my data | Tech Radar

Overview

Amazon's new Health AI Chatbot is rife with potential for misuse — here's why I wouldn't trust it with my data

When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.

Details

Amazon has launched its new Health AI service in the US, a chatbot which can help you understand, treat and diagnose health conditions. Available to all US Amazon Prime subscribers, it can have conversations with you about your health issues, recommend fixes, health products and connect you with doctors.

Let's get this out of the way: I think AI has a place in healthcare. As medical staff battle creaking hospital infrastructure and overwhelming demand, patients suffer long waitlists and (in the US, at least) seemingly ever-rising costs driven by for-profit pharmaceutical and medical industries. AI has the potential to ease all of those problems.

However, its implementation needs to be carefully considered, and it's not necessarily something I'd trust Amazon to do. While we've all ended up trusting Amazon with our data in some form or another (Amazon Web Services is the biggest provider of cloud storage in the world, after all), I'd still be reluctant to hand over my sensitive health information to its chatbot. Let's break down exactly why.

Chat GPT Health gives you a new way to explore medical questions

I asked experts whether I should use Chat GPT for health advice, and I was shocked

Amazon describes Health AI as "an agentic AI health assistant designed to make health care easier". It says Health AI is "designed to be a personalized health agent that knows you and your medical history so it can provide more helpful responses and take meaningful action, including connecting you to the professionals, treatments, and account services you need to get and stay well".

These services include recommending you Amazon Pharmacy products and connecting you to healthcare providers (specifically from Amazon's One Medical group). With permission, Amazon can also access your medical records and have the chatbot discuss them with you.

Amazon insists security is tight: it says Health AI is a "HIPAA-compliant" environment, referring to the US Health Information Portability and Accountability Act. This means all your protected health information (PHI) is treated as it would be at a doctor's office, and subject to all legal privacy requirements.

The HIPAA Journal, in an article about AI published last year, said healthcare providers and vendors using AI run risks as a result of the technology. It stated that "lurking surreptitiously behind the potential benefits of using PHI in AI technology lies a murky mix of risks that could negatively impact [healthcare providers], your vendors, and even your patients, especially when HIPAA compliance and patient PHI are involved."

Some of these risks involve the nature of AI models requiring huge corpuses of data to train. If you're building a health-based chatbot, you need a lot of health data to do so, and Amazon, one of the world's biggest collectors of data, sees the value in your personal information.

Amazon insists that "Protected health information from Amazon One Medical and Amazon Pharmacy is not used in the broader Amazon store to market general merchandise or by Amazon Ads, and Amazon does not sell customers' personal data," and "we only use protected health information for purposes permitted under HIPAA".

Open AI says 40 million people use Chat GPT for healthcare every day

Amazon’s AI agent shopper is roping in brands without asking

Claude for Health brings AI assistance to healthcare

While Amazon will record your data to train future models of its AI, it does say it will anonymize that data. However, that in itself isn't properly private. Meta was found to be able to link Facebook accounts to users of the period-tracking app Flo, even after names and other identifying account information had been removed, thanks to a unique identification number. It was found in court to be spying on Flo users. I'm bringing this up because I wouldn't trust name removal to be an effective way of anonymizing accounts, when it comes to harvesting chat logs to train the next generation of AI.

If there are data breaches, or Amazon uses this data in a way that isn't HIPAA compliant, I also don't think any punishment can be effectively enforced. The Facebook court case I linked above took many years to be resolved, and still no punishment has been handed out. Amazon is simply too big to fine, and too big to discipline, especially as its data centers are used by everyone from governments to grocers.

Big tech's desire to help build a happy, healthy populace is second to its desire for the profits it could wring out of your personal data.

With that in mind, Amazon isn't doing a good enough job at convincing me its new Health AI service is safe, private and in my best interests. The state of New York obviously feels the same way: it's in the process of blocking AI chatbots from giving legal or medical advice, and right now its hard to argue with such policies.

Follow Tech Radar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds. Make sure to click the Follow button!

And of course you can also follow Tech Radar on Tik Tok for news, reviews, unboxings in video form, and get regular updates from us on Whats App too.

Matt is Tech Radar's expert on all things fitness, wellness and wearable tech.

A former staffer at Men's Health, he holds a Master's Degree in journalism from Cardiff and has written for brands like Runner's World, Women's Health, Men's Fitness, Live Science and Fit&Well on everything fitness tech, exercise, nutrition and mental wellbeing.

Matt's a keen runner, ex-kickboxer, not averse to the odd yoga flow, and insists everyone should stretch every morning. When he’s not training or writing about health and fitness, he can be found reading doorstop-thick fantasy books with lots of fictional maps in them.

You must confirm your public display name before commenting

1ICYMI: the week's 7 biggest tech news stories from Sonos' big return to our review of the 'impressively premium' Mac Book Neo

2 Amazon's new Health AI Chatbot is rife with potential for misuse — here's why I wouldn't trust it with my data

3 Express VPN just beat its lowest-ever prices for the second time this year! Perfect if you're traveling for St Patrick's Day

4 The starter you pick in Pokémon Fire Red and Leaf Green means a lot more than you might think — here's why you need to choose carefully

5 How to Make a Killing is a rare misfire for A24 — a serial killer story should not be this boring

Tech Radar is part of Future US Inc, an international media group and leading digital publisher. Visit our corporate site.

© Future US, Inc. Full 7th Floor, 130 West 42nd Street, New York, NY 10036.

Key Takeaways

• Amazon's new Health AI Chatbot is rife with potential for misuse — here's why I wouldn't trust it with my data

• When you purchase through links on our site, we may earn an affiliate commission

• Amazon has launched its new Health AI service in the US, a chatbot which can help you understand, treat and diagnose health conditions

• Let's get this out of the way: I think AI has a place in healthcare

• However, its implementation needs to be carefully considered, and it's not necessarily something I'd trust Amazon to do