Alexa and her friends know you all too well

Amazon has sold hundreds of millions of Alexa devices. You probably know people who place these Amazon-built microphones in their homes to listen for voice commands. The company has a privacy page, complete with cringe-inducing Alegria cartoons, describing the information they do and do not collect and share about your life. But should you trust these assurances? What does Amazon do with their recordings of you?

An independent study published in April looks into this issue. The researchers found that regardless of whether Amazon sells recordings from your home, they do use what you say to build a profile of your interests. Exactly how far this practice goes is unclear. Amazon uses this information, but so do third parties. For what it’s worth, Amazon disagrees with the study’s conclusions. To explain in detail, we need some background on how Alexa works.

Alexa listens, but is she interested?

Is Alexa always listening? Amazon says no, but the company’s explanation is an exercise in semantics. The device must be listening to the goings-on in your home at all times in order to know when to activate. Sometimes it will misinterpret your at-home chatter and do something rash. Amazon is currently in court over this sort of thing. Alexa recordings may also be used by the police. A former executive at the company admitted that he turns Alexa off when he wants a private moment.

Amazon generally describes the Alexa process as a computer algorithm analyzing your voice. In 2019 they were forced to admit that this, too, was not entirely true. In fact, they were paying thousands of human beings to listen to home audio captured by Alexa devices. The reports from those workers make for some interesting — and alarming — reading, with some saying they listened in on what seemed to be criminal activity.

Amazon does explicitly admit that they keep all of their recordings of you. They may have many thousands of these records. Currently, there is no evidence that your raw voice recordings are shared with anyone outside of Amazon itself. Instead, the company processes the recording of your voice to create a data file containing an interpretation of what you said, along with further information, and the company shares that.

Let’s say that you choose to install an Alexa device in your home. Beyond their attempt to discern what you are requesting, what else does Amazon do with its recordings of your voice? The study set out to test this question. The researchers analyzed the information flow between users, Alexa, Alexa apps, and third parties, and they looked for patterns.

To do this, the researchers set up several faux personas that interacted with the Alexa device. These personas asked Alexa about certain interests, installed various Alexa skills — essentially apps — geared toward their interests, read websites related to those interests, and so forth. They also created bland control personas with no specific interests. 

Advertisers paid significantly more to show ads to the personas with specific interests, but only did so at times shortly after a persona started asking Alexa about a specific interest. This strongly suggests that Alexa listens for when, and how often, you ask about certain subjects.

Where does this information go? The study found that both Amazon and outside parties used data from Alexa recordings for advertising and tracking purposes. Those third parties may or may not have properly disclosed this in their privacy statements — when, indeed, they had privacy statements.

Selling to you, or selling you?

The researchers conclude that Amazon uses the information from your Alexa interactions to deduce what your interests are and then market to you, both on its own platforms and on other platforms across the web. The marketing you receive is personalized based on what Alexa hears. This applies not only to web ads, but also to audio ads on platforms such as Spotify, Pandora, and Amazon. The researchers did not test ads on video streaming services.

The researchers point out that while Amazon may not specifically use your recorded voice to target ads to you, it appears that they do target ads to you based on your Alexa usage. It is likely that the metadata and voice analysis, rather than the recording itself, is used to sell targeted ads. Metadata is a fancy word for information beyond the main voice transcription. This could be things like time, date, user preferences, and feature usage. It might include more worrisome things such as user age, sex, accent, speaking patterns, daily habits, and mental condition.

How much further does Amazon go in their analysis and sale of you? The paper points out that Amazon holds a patent called “voice-based determination of physical and emotional characteristics of users.” This appears to be tech that is designed to use your voice commands to infer that you are excited, sad, happy, tired, or hungry, among other things. There is no direct evidence that they currently use this capability.

It will take more empirical studies like this one to test whether things like this are going on, and if so, to force Amazon to admit it. In the meantime, you could continue to use your home-to-cloud microphone system. You could believe that its manufacturer values your privacy. And you could trust that they will be transparent about the extent to which it captures and uses your data. But you’d be betting against the company’s track record.