How Amazon’s delivery robots will navigate your sidewalk

Earlier this year, Amazon announced its Scout sidewalk delivery robot. At the time, details were sparse, except for the fact that the company had started to make deliveries in a neighborhood in Washington State. Today, at Amazon’s re:Mars conference, I sat down with Sean Scott, the VP in charge of Scout, to talk about how his team built the robot, how it finds its way around and what its future looks like.

These relatively small blue robots could be roaming a sidewalk near you soon, though as of now, Amazon isn’t quite ready to talk about when and where it will expand its network from its single neighborhood to other areas.

“For the last decade, we’ve invested billions of dollars in cargo planes and delivery vans, fulfillment center robots, and last holiday period, we shipped over a billion products with Prime free shipping,” Scott told me. “So it’s my job as VP of Amazon Scout to bring another new, innovative, safe and sustainable solution to this delivery network to help us really grow quickly and efficiently to meet customer demand.”

Currently, in Amazon’s trial, the robots are always accompanied by human assistants. Those assistants — and they probably look a bit like robot dog walkers as they trot through the neighborhood — are currently the ones who are taking the packages out of the robot when they arrive at their destination and put it on the customers’ doorsteps. For now, that also means the customers don’t have to be home, though chances are they will have to be once this project rolls out to more users.

As of now, when it’s ready to make deliveries, Amazon drives a large van to the neighborhood and the Scout robots leave from there and return when they are done. Scott wouldn’t say how far the robots can travel, but it seems reasonable to assume that they could easily go for a mile or two.

As we learned earlier this year, Amazon did make a small acquisition to kickstart the program but it’s worth stressing that it now does virtually all of the work in house, including building and assembling the robots and writing the software for it.

“For Scout we’re actually owning the entire development from the industrial design to the actual hardware, mechanical, electrical, the software, the systems, manufacturing and operations,” said Scott. “That really helps us control everything we’re doing.” Having that end-to-end control enables the team to iterate significantly faster.

The team even built a rig to test the Scout’s wheels and in the process, learned that the wheels’ material was actually too soft to survive the rigors of daily sidewalk driving for long.

Inside its labs, the team also built a sidewalk environment for real-world testing, but as is so often the case these days, the team did most of the machine learning training in a simulation. Indeed, since there are basically no maps for navigating sidewalks, the team has to build its own maps of every neighborhood it goes into and it then uses this highly detailed map in its simulation.

That’s important, Scott noted, because simply using a game engine with repeating textures just wouldn’t be good enough to train the algorithms that keep the robot on track. To do that, you need real-world textures, for example.

“We thought about building a synthetic world, but it turns out building a synthetic world is much harder than copying the real world,” Scott said. “So we decided to copy the real world.” He showed me a video of the simulated robot moving through the simulation, using a map that looks a bit like a highly zoomed-in Google Maps 3D view. Not perfect, but perfectly reasonable, down to the gutters on the street and the small bumps where two concrete plates on the sidewalk line up.

This simulation allows Amazon to make thousands of simulated deliveries before the team ever goes out to test the robot on the street. In the demo I saw, the robot had no issues navigating around obstacles, pausing for crossing cats and getting to his destination. That’s possible thanks to a combination of detailed maps and high-resolution imagery of its surroundings, combined with GPS data (when available) and cutting-edge machine-learning techniques.

Once it is out and about, though, the robot will have to face the elements. It’s watertight, something you’d expect from a company that is based in Seattle, and it’s got sensors all around to ensure it can both find its way on sidewalks that are often littered with obstacles (think trash day) and full of curious cats and dogs. Around the robot is an array of cameras and ultrasonic sensors, all of which are then evaluated by a set of machine learning algorithms that help it plot its path.

“We jokingly refer to the sidewalk as the Wild West,” said Scott. “Every sidewalk is a snowflake and every neighborhood is a collection of snowflakes.”

At times, the robot also has to deviate from the sidewalk, simply because it is blocked. In those cases, it will opt for driving on the street. That’s something local laws in many states now allow for, though Scott tells me that the team only considers it when it’s a street where a pedestrian would also feel comfortable. “If you feel safe walking on that road, that’s where we want to be. We want to be viewed as a pedestrian and treated as a pedestrian,” he said. And that’s how the law in Washington State looks at these robots, which, for example, mean that they have to be given the right of way.

Scott also noted that the team designed the robot so it would be visible when necessary, with blinking lights when it crosses a street, for example, but also a bit boring, so that it would blend into the environment. “We really want this to blend into the background and be part of the environment and not be this loud and obnoxious thing that’s always rolling through the neighborhood,” said Scott. So it has the bright blue Amazon Prime color on top to be seen, but is otherwise relatively bland and without any anthropomorphic features. It’s just your average neighborhood delivery robot, in other terms.

As it moves along, it makes very deliberate movements, which Scott believes will make people feel more comfortable around it. Unlikely a drone, there’s no major risk when any parts of the robot break during a mission. Somebody can simply come and pick it up. Still, the team says it did design the robot with safety at the front and center of its process.

One thing that’s currently not clear — and that Amazon didn’t want to talk about yet — is how it will solve the actual handover of the package. Right now, the assistant handles this part, but in Amazon’s photos, the customer walks up to the robot and takes the package out of it. That’s a reasonable scenario, I think. In the long run, Amazon could also outfit the robot with multiple compartments to make multiple deliveries in one go. Right now, the Scout can only handle a single package.

One advantage of the robot has over human delivery people is that if you’re not home, it can just wait for a while, Scott said. So it’s conceivable that you’ll come home one day and there’s a Scout, standing patiently in front of your door, waiting to deliver your latest impulse order. Until then, it’ll likely be a while, though. Amazon won’t commit to any timetable or wider rollout.