How Google Knows Your House Number

Google Street View is great at giving you a private tour of those places you could never get to visit yourself—Machu Picchu, say, or the Large Hadron Collider. But where it starts to feel weird is when it goes to the places that are already all too familiar—like your own home. It’s a little unnerving to think that Google knows where you live.

A new paper from a team led by Google researcher Ian Goodfellow, and covered in MIT's Technology Review, now reveals quite how intimately Google knows your street, down to the decorative plaque on your door that shows your house number. Because while users can zoom into Street View images and interpret that level of detail, so can an automated system used by the tech giant to produce accurate maps.

As the researchers explained in their introduction:

Recognizing multi-digit numbers in photographs captured at street level is an important component of modern-day map making. A classic example of a corpus of such street level photographs is Google’s Street View imagery comprised of hundreds of millions of geo-located 360 degree panoramic images. The ability to automatically transcribe an address number from a geo-located patch of pixels and associate the transcribed number with a known street address helps pinpoint, with a high degree of accuracy, the location of the building it represents.

The point is to reduce the problem you sometimes get when you search for a specific building, but end up with only a vague location that doesn’t tell you which end of the street you need to be. You know, that thing that always happens just when you’re running late for a job interview, and it’s raining, and your phone is about to die.

Technology Review added that it’s a particularly useful tool for Google in places where house numbers don’t follow an ascending order from one end of a street to the other.

It’s a tricky task for an automated system, because everyone’s house numbers are located in different positions, and use different fonts, colours, sizes, and so on. And without being 100 percent correct, a house number is rendered useless. As the paper put it, “There is no ‘partial credit’ for getting individual digits of the sequence correct. This is because for the purpose of making a map, a building can only be found on the map from its address if the whole street number was transcribed correctly.” That’s zero marks for effort.

To transcribe the numbers, the researchers used an artificial neural network—a computational model inspired by the human brain—and trained it over six days using a public database known as the Street View House Numbers dataset, which contains about 200,000 images of house numbers cropped from Street View images.

The system got the numbers right with an accuracy of around 98 percent, which makes it comparable to a human’s abilities in the same task. But, as the authors noted, hiring human operators to go through and transcribe the numbers is more expensive, and not as super-quick as the automated method. “We can for example transcribe all the views we have of street numbers in France in less than an hour using our Google infrastructure,” they wrote.

Overall, the system transcribed a rather impressive 100 million street numbers. It’s a nice example of machines mastering things humans do without thinking twice—like reading a street number—but it couldn’t yet be used for just any old text; the number sequences were limited to only five digits each.

That said, it’s still a good start. “Perhaps our most interesting finding is that neural networks can learn to perform complicated tasks such as simultaneous localization and segmentation of ordered sequences of objects,” the authors concluded. Robot brains are getting clever.