Amazon taps AI, ML to keep bad apples off its shelves

In an interview, Rajeev Rastogi, Vice President, ML at Amazon India, said that the company has developed computer vision programs that detect defects such as cuts and scratches on tomatoes and onions to detect when they have gone bad.

The system uses a mix of Convolutional Neural Networks (CNNs) and Visual Transformer (ViTs) algorithms. CNNs are deep learning algorithms that can take image input and assign importance to different aspects of that image, while VITs are special versions of transformer algorithms, which can weigh the importance of each piece of data.

“In our grocery business, quality produce is the most important customer input and the number one driver of repeat purchases,” said Rastogi. “Currently, quality is processed manually, which doesn’t really scale. . It is also very error-prone, expensive and does not have high repeatability. Therefore, we have developed a computer vision system for grading fresh produce quality by analyzing images of the produce,” he said.

Amazon uses computer vision techniques to detect cuts, cracks and pressure damage and more. The technology is now deployed for tomatoes and onions in Amazon’s stores, but Rastogi said it is building an AI-enabled machine, the Auto Grader, that automatically grades products running on a conveyor belt .

“This will improve product quality and consistency while reducing grading costs by about 78% compared to manual grading,” he said.

Rastogi said, “In the future, we want to use infrared sensors to detect characteristics such as sweetness and ripeness, which we cannot detect in RGB (red, green and blue) images that are detected by your traditional computer vision algorithms.” have been captured.” He added that infrared can help avoid “disastrous methods” of quality assurance, such as the ripeness and sweetness that often require a person to actually eat a fruit or vegetable.

“A near-infrared image for a sweet versus sweet product will be very different. Because the sugar level will be different, and the near-infrared signature will be different,” he said.

The system is implemented in some pilot projects in Europe as well as in stores in India. Rastogi acknowledged that the system is in its initial stages and it will take some time for the company to finalize the timeline when it will be rolled out on a large scale. However, he added that the system is at over 90% accuracy in terms of accuracy in finding faults. “It varies from one product to another. Like we have much better results for tomatoes and we can do it with high precision. Our numbers for onions are a little less and we are improving that.”

He added that the difficulty in determining the quality of fresh produce can also vary from one product to another. “Most of the defects we are seeing (right now) are visually detectable by a human. I think that whatever can be detected visually, can also be easily detected with computer vision,” he said.

In the future, Amazon may use such a system to categorize and categorize products in order to identify the product the seller is offering. For example, if a seller is offering a particular type of tomato, the system can classify what the tomato should look like and find out in advance whether the right product is being delivered.

The company has also had to devise systems for the quality of Indian products. Since the quality of fresh produce in India is not at par with its counterparts in western countries, the approach to determining quality also needs to change. “A tomato in India is not the same as a tomato in the US. Therefore, you cannot simply say that you will have the same approach to dealing with tomatoes and onions in every country. The nature of the product can vary greatly due to agricultural practices. It is,” said Rastogi.

In theory, the company could even bring Auto Grader to the front-end, where customers can just open the Amazon app and find out the quality of the vegetable or fruit themselves.