How One Visual Search Startup Plans to One-Up TinEye

When you have a photo and want to find a higher resolution copy online, you can use tools like Google Image Search and TinEye. They work in a very specific way extracting specific patterns and how they contrast with their surroundings. That works well for finding alternate copies of the same image, but it's not so great if you try to go deeper.

Enter Clarifai, which creator Matthew Zeiler claims improves upon the techniques used by Google and Tineye by way of superior artificial intelligence. "We understand images and video automatically with a whole variety of different 'models' as we call them," Zeiler explained to Motherboard. "In our demo, you can throw in an image and get different predictions from the models." By default, the demo on its website uses its "general" model, which is designed for everyday photos that the average person would take with their phone. While Clarifai is primarily a web services company, they have an iPhone app for the average consumer to help get their name out. "If you take your phone out of your pocket and take a picture of the world, it should give you meaningful stuff back."

The tags that Clarifai outputs aren't just naming the objects it sees, but can get pretty descriptive. During my chat with the Clarifai team, I pulled up the demo and fed it the "Colin Powell pretending to play guitar at Bohemian Grove" photo that was leaked with his hacked emails. Being both obscure and low-resolution, it seemed like a pretty solid test of their capabilities. Getting tags like "music," "performance," "guitar," and "stringed instrument" wasn't necessarily surprising. But I found myself really impressed with one tag in particular, "festival," because Clarifai was able to discern that it was looking at an outdoor stage setup.

Last week, Clarifai added two new features, which they've dubbed "Custom Training" and "Visual Search." These go hand in hand: Custom Training allows you to "teach" Clarifai to associated photos, while Visual Search "allows any user to easily organize, access, or recommend their images or products by visual similarity and/or keyword." In a way, the most impressive part may not even but the underlying AI tech in and of itself, but how fast and, once again, specific it is. Everything you do with Clarifai is almost instantaneous, whether it's giving you keywords or using its new features to group photos together.

When you feed Clarifai examples of various photos that fit a given criteria, it immediately finds other similar photos. "This is not the web data, this is your data," Zeiler explained. "So you as a retailer can put in your product photos, or you as a social media site can put in your social media photos, or a consumer can put in their own photos. It indexes all of that and can learn from your data."

To illustrate this, Zeiler pulls up some photos of sneakers. After selecting just a small handful of Converse brand shoes, Clarifai immediately populated the results with all of the other Converse photos in its database, because it could distinguish the company's trademark style. This even extended to some of Converse's non-sneaker footwear, as a few stray pairs of Converse boots showed up in the results. Similarly, but more abstract, when Zeiler started choosing what he called "treeshoe" photos of shoes alongside trees, Clarifai immediately narrowed it down to other "treeshoe" photos.

Clarifai's iPhone app can also do this with your personal photos Instead of tagging each photo to reflect who's in it, you can teach it with a few photos of the same person, and then it will group all photos of that person together. That's a time saver, and even if, in the long run, Clarifai is making its money providing services to other websites, you can see it being what they will become known for by the average person.

"The thing I'm most excited about with this is it opens up this technology for everybody out there," Zeiler explained. "We don't care if it's our own applications that go directly to users, or through the great companies developing on our platform. We want to get it out to everyone on the planet."