Oct 23, 2018 3:01 AM

Stepping Into an Amazon Store Helps It Get Inside Your Head

The retailer can learn a lot about shoppers' habits from its growing line of cashier-less Amazon Go convenience stores.

Image may contain Human Person Grocery Store Shop Supermarket Market and Shelf

Infrared light flooded down invisibly as I eyed the pastries in Amazon’s new convenience store in downtown San Francisco. It helped cameras mounted on the store’s ceiling detect that I picked up a croissant, then put it back.

My flirtation with a $3.19 morsel of flaky pastry was recorded during a preview of the Amazon Go store that opened in San Francisco’s financial district this morning. As in the five other such stores in Seattle and Chicago, shoppers gain entry by scanning a QR code in the Amazon Go mobile app to open a subway-style entry gate. Hundreds of cameras on the ceiling, plus sensors in the shelves, then record what each person picks up, so they can walk out without having to visit a checkout.

Amazon Go’s design offers shoppers an eerie freedom. Breezing out of a store without breaking stride feels efficient but also a little like shoplifting. Less perceptibly, the cameras and shelf sensors also log data that provides Amazon a remarkable view of what people do in a physical store. “Anything that you pick up, anything that you put back, is kept track of,” says Dilip Kumar, the vice president responsible for the technology behind Amazon Go.

Tracking shoppers like that puts Amazon in a position to teleport some of the data-driven tactics from its dominant online shopping business to the physical world. Kumar likened the way his system logged my moment of temptation with the croissant to how the company’s online store logs what people click. “It's the equivalent of interest on an Amazon detail page,” he said.

Peter Fader, a professor of marketing at the University of Pennsylvania's Wharton School, compares the data that Amazon Go stores could collect to an earlier retail revolution—the debut of barcode scanners. That technology also streamlined the experience for shoppers while revealing new information to store owners, such as which items were frequently bought together.

When someone visits an Amazon Go store, the Seattle company's technology can see not just what they bought but what they picked up and discarded, and also in what order they handled different items. That information could be used to optimize a store’s selection and design, and to power personalized marketing messages, Fader says. “It becomes possible to figure out what’s the bait to attract and retain and build relationships with the most valuable customers,” he says.

Cashier-less systems can collect impressively detailed data on shoppers. Standard Cognition is one of several startups offering similar technology to existing retailers. The company boasts that, unlike Amazon Go, its system uses only overhead cameras, not shelf sensors, making it easy to deploy in existing stores. Evan Shiue, who leads strategy at Standard Cognition, says the startup’s technology can log things like how often people picking up a can of Pepsi look at its nutrition panel, or whether they put it down and buy a Coke instead.

“You get a lot of great behavioral data in ecommerce—what I added to my basket and then took out and how long I looked at a particular item,” says Shiue. “We can now do this in bricks and mortar through computer vision.”

Visitors to an Amazon Go store are watched closely from above by conventional and depth cameras—aided by infrared illumination—from the moment they scan their app to gain entry. Amazon trained machine-learning algorithms to recognize when items have been picked up using thousands of hours of footage of people grabbing items from shelves, Kumar says. Weight sensors in a store's shelves help the system confirm what item, and how many, a person has taken.

Kumar wouldn’t discuss in detail how Amazon might use the data it collects in Amazon Go stores. He acknowledged that the company could one day combine information from my visit with my record of Amazon.com purchases—potentially helping customer data analysts at both businesses—but said that wasn’t the project’s “primary purpose.” A company spokesperson said that any sensitive data collected by Amazon Go stores is treated in accordance with Amazon's existing data security policies, and directed WIRED to a privacy notice in the convenience stores' app.

The success of other companies that use technology to merge online and offline customer tracking suggests that Amazon could reap considerable benefits. Stacy Smollin Schwartz, a professor at Rutgers Business School, points to Starbucks’ popular mobile app, which uses loyalty points and personalized challenges to lure customers into stores, and Disney’s MagicBand wristbands that track visitors’ perambulations around theme parks. Both have shown that a digital lens on a person’s real-world actions can open up new ways to change their behavior.

For Amazon, that might mean sending people targeted promotions through the Amazon Go app—for example, to encourage someone who grabs a daily breakfast sandwich to also drop by for lunch. “Knowing people’s habits and being able to individually try to manipulate those habits to increase the individual loyalty and profitability of each customer is very valuable,” says Smollin Schwartz.

A few minutes after I walked out of the store’s electronic gate Friday, the Amazon Go app buzzed for my attention. It showed an itemized receipt for the Advil, curry paste, and Amazon Go-branded chocolate bar I had grabbed from the shelves and stuffed into my pockets. The price total and tax were noted. Not visible—what Amazon learned about me.