USC Viterbi’s Gerard Medioni helped create much of the technology behind Amazon Go, a new cashier-free market.
At first glance, the store of the future doesn’t seem all that revolutionary.
At a modest 1,800 square feet, the first Amazon Go store is stocked with ready-to-eat meals prepared by on-premise chefs and local bakeries. The grocery mix includes staples such as bread, butter and milk as well as artisan cheeses and craft beer, making Amazon Go seem like little more than an upscale convenience store that caters to harried, well-heeled office workers in this charming corner of Seattle. Imagine a 7-Eleven for the wine-and-brie crowd.
But take a closer look. Hundreds of sensors and cameras hang from the ceilings. Unlike a typical supermarket, Amazon Go has no carts and baskets. It has no cashiers or checkouts. Shoppers never have to wait in long lines or fish through their wallets, pockets and purses for cash or credit cards to pay for groceries.
Instead, patrons carry shopping bags, fill them with whatever items they want and simply walk out of the store — the overhanging cameras and sensors having recorded their purchases and debited their Amazon accounts. A little later, shoppers receive a receipt through their Amazon Go app, making the grocery shopping experience “more convenient than ever before,” in the words of Amazon Chief Executive Jeff Bezos.
The company’s newest store concept — checkout-free shopping — promises to revolutionize the supermarket industry.
The Technology Behind Amazon Go
Computer vision, sensor fusion and deep learning power what can be described as “Just Walk Out” shopping. Overseeing this complex array of interacting technologies is USC Viterbi Professor Gerard Medioni, who is an expert in computer vision — the science of transforming images into useable information. Medioni is currently on leave from the school while working as director of research for Amazon Go.
Medioni and his team have worked to solve what he calls the “who took what problem.” That is, determining which customers grab what items and then providing an accurate receipt. A gifted undergraduate or graduate student, Medioni said, could cobble together off-the-shelf technologies that would get it right some of the time, but not nearly often enough.
For Amazon Go to work, Medioni and his team had to develop state-of-the-art computer vision, algorithms and technologies that could accurately and consistently track what customers take off the shelves, even when shoppers inadvertently cover labels with their hands or when products next to each other, say, strawberry and raspberry jam, look nearly identical. After “working at the edge of the doable” and beta testing for more than a year, the first Amazon Go store opened in January.
“My research and experience at USC Viterbi have provided the foundation for me to be successful at Amazon,” said Medioni, who has worked in the computer vision field for more than 35 years. “I love to read comments sent by customers talking about how delightful and magical this is. Having helped create a unique customer experience is a wonderful reward for me.”
How it works
Passing through a gate that resembles a subway turnstile, shoppers swipe their Amazon Go app to enter the store. They can then put away their phones — the scanned QR codes make it possible for the system to identify their accounts.
Custom-built overhead cameras and sensors tabulate what people take and put back. So advanced are motion detection and object identification technologies that paper towels, orange juice and other merchandise need no special chips attached to them.
Weight sensors on shelves provide an additional source of information, Medioni explained, detecting when an item is taken or returned. That means that shoppers who try to sneak a chocolate bar into their bag will be billed for it. No cheating. And to protect shoppers’ privacy, the majority of cameras are used only to identify products.
This is one of the most sophisticated uses of computer vision in a retail setting that I’m aware of. Getting all those algorithms to work together in real time is perhaps the biggest challenge they overcame.” — Wael Abd-Almageed, a computer science research team leader at the USC Information Sciences Institute
“Getting all those algorithms to work together in real time is perhaps the biggest challenge they overcame.”
Popular with Shoppers
At noon on a recent Monday, Seattle’s Amazon Go store buzzed with activity. Workers wearing orange T-shirts stocked shelves, checked IDs and offered recommendations in the beer and wine section, and made gourmet sandwiches and salads in the pristine kitchen at the front of the store. Shoppers, many of them hip millennials from nearby businesses, darted in and grabbed curry chicken wraps, Southwestern steak and roasted corn salads, and other freshly made gourmet fare.
Mary Boxley, 24, is among the fast-growing group of regulars. “It’s become my go-to spot for lunch. The only reason I’m in the store for more than 30 seconds is deciding what to eat,” she said. “I love how quick, easy and convenient the experience is.”
The company plans to roll out Amazon Go in San Francisco and New York, and has opened two more stores in Seattle and one in Chicago. Amazon could open up to 3,000 by 2021, according to Forbes. Like the original market, the new outlets will be relatively small, feature a variety of products and be near office buildings to ensure steady traffic.
There appears to be a strong appetite for the concept. Nearly three-quarters of 1,000 U.S. consumers surveyed by marketing firm Digital Third Coast said they would be very or extremely likely to shop at such a store, if one opened in their neighborhood, according to Supermarket News.
A Retail Revolution?
Amazon is entering an offline supermarket industry already undergoing a period of great change, said Jim Lee, former president and chief operating officer of the Stater Bros. supermarket chain and professor in residence at the USC Marshall School of Business.
In recent years, big-box retailers like Target and Walmart have ramped up their grocery businesses. That has put tremendous stress on legacy companies such as Kroger (the parent of Ralphs) and Albertsons (which includes Albertsons, Safeway and Vons). The rise of Sprouts, Whole Foods, ethnic chains and other specialty retailers has further increased competition.
Amazon Go will “add more pressure on retailers to be excellent by getting rid of one of customers’ pet peeves — waiting in line to check out — and eliminating or potentially greatly reducing the cost to staff the check stands ” Lee said.
“Amazon Go is a game changer,” he added. “There’s a number of things that keep supermarket executives awake at night, and this is one more thing.”
Executives’ collective insomnia might worsen if Amazon adds the checkout-free technology to the 465-store Whole Foods chain it acquired last year for $13.7 billion. While Amazon has said it currently has no plans to do so, Lee and others believe the company probably will if it can overcome the technological challenges of embedding such a complex system in much larger stores.
As innovative as the Amazon Go technology is, retailers will likely embrace it slowly, if at all, said Dave Marcotte, senior vice president of retail insights for the Americas at Kantar Consulting. That’s because they won’t add all those sensors and cameras to their stores until executives believe the expensive makeovers will increase profitability.
Some retailers could employ Amazon Go or similar technologies in the next five years when tech costs drop and they open new or remodel existing stores, added Marcotte, who spent more than 25 years in the high-tech and retail sectors. Prime candidates are fast-food restaurants, shops inside airports, possibly some clothing stores and, of course, supermarkets — “places that people want to get in and out of,” he said.
Online to Offline
So why has Amazon moved into the low-margin, offline
supermarket space, e.g. by launching Amazon Go and acquiring Whole Foods? What to make of its opening 16 Amazon Books stores since late 2015? Does the company hope to dominate the brick-and-mortar world?
Anthony Dukes, a professor of marketing at the USC Marshall School of Business, believes that Amazon has far less ambitious goals. Like any other business, he said, the company wants to increase sales and profits. Instead of trying to take over the supermarket and other industries, Dukes said, it simply hopes to gather additional information about customers to better improve its services and make better recommendations.
“To the extent that Amazon can leverage more data to help figure out what, where and when customers want to buy, their tastes, their income levels and what they’re willing to pay for it, the more profitable they’re going to be,” Dukes said.
Medioni, the USC Viterbi professor at Amazon Go, offered this explanation: Amazon’s commitment to customer service drives everything the company does, including its foray into supermarkets and other offline ventures.
“We wanted to create a wonderful customer experience,” Medioni added. “We advanced the state of the art in many areas of computer vision, and we are not done. The store is working well, but we are continuing to make improvements.”