• About
    • About the FIA
    • Priorities
    • Our Team
    • Brainstorming Board
    • Partners and Affiliates
    • Contact Us
  • News + Events
    • News
    • Events
    • Videos
    • Newsletters
    • @FIAumd
    • In the Media
  • Spark Grants
    • Spark Grants Overview
    • Spark Grants FAQ
    • 2012-2015 Seed Grants
    • 2012-2015 Seed Grant Winners
  • Special Topics
    • SearchReSearch
    • Curated Topics
FIA

SearchReSearch

Image identification is great, when it works.

Dan Russell • January 7, 2022
 SearchReSearch
Republished with permission from SearchReSearch
Image identification is great, when it works. Dan Russell

Searching by image...

What plant is this? Can search-by-image help?

... is a fundamental skill for SearchResearchers. You should know how to use regular Google Search-by-Image (see this for a refresher), and you should know how to use Google Lens (refresher), and you should know about Bing's Search-by-Image, Tineye, and Yandex's search-by-image tool as well. That's five major search-by-image systems that you should just know how to use.

But the problem with all of these systems is that they do what they can with what they see, but you shouldn't rely on them for proper identification, particularly of plants, dogs, cats, people, and the random marginalia of life.

Here's what I mean.

This is a photo I took of a plant growing in a nice Silicon Valley bit of landscaping. I happen to know what this is (it's an Arbutus unedo, which we've talked about before in SRS (post about these plants back in 2014). I'm curious to see if Google Lens will identify it correctly or not. I know this is a difficult search because this plant looks superficially like several others. Here, I'm interested in the plant with the red berries and now the low-growing plant below it.

Original Photo

As you know, you can select the region of interest in the photo, so I first selected a random bit of leaves, branches, and a corner of one of the fruits. This turns out not to have worked well at all.


Google Lens thinks it's a Toyon bush, which is reasonable guess, except that the berries are all the wrong size (Toyon berries are much smaller and don't have little hairs). Luckily, I also know what a Toyon bush looks like, so I know it's wrong. But unfortunately, if you didn't happen to know what a Toyon was, you might well accept this an an ID. And you'd be wrong.

For a second attempt, I focused the area of interest onto just the berries. This also turns out to not work well either.



Now Lens thinks it's a Lychee, which isn't a terrible guess, but upclose, they look nothing alike. This is a basic problem with identification only from an image: it's hard to get the upclose details that matter! (For what it's worth, this is a problem for humans trying to identify something from just an image. The key difference is that the human will tell you "I can't be sure, given just this photo" whereas the algorithm just gives you an ID without any indication that there's a possibility of error.

I try again with a broader region to search, and the third time is a charm. Here I've changed the region of interest to be the berry, a few leaves, and a couple of small branches. And this time, it gets it right. Well, kind of. It calls it a "strawberry tree" which is correct and informal. I would have preferred the answer to be more like this:
Arbutus unedo, an evergreen shrub or small tree in the family Ericaceae native to the Mediterranean region and western Europe. The plant is known for its fruit which look a great deal like a strawberry, hence the common name "strawberry tree." However, it is not really related to strawberries at all.





SearchResearch Lesson about Search-by-image


Basically, you have to be careful and check your results. It's tempting to do a quick search and be done with it, but if you're trying to figure out if it's a potentially deadly flower (or berry, or insect bite...), you'll want to do as you always do and DOUBLE CHECK.

Note that all of the search-by-image systems (Tineye, Yandex, Bing, Google, etc.) are all sensitive to the are of interest--you will get different answers depending on which region of the image you ask about.

None of them will tell you about the degree of certainty, but will give you a ranked list, rank ordered by a mysterious relevance operation that probably has nothing to do with accuracy, but more to do with image similarity.

And it should go without saying, but NEVER TRUST a search-by-image function to identify a mushroom. To get a complete ID of a mushroom often requires examining the gills and the spores, frequently with a microscope.

Be aware of the limits of the tools you use--the skilled SRS Researcher knows the limits!


Search on! (Cautiously...)



Share

Comments

This post was republished. Comments can be viewed and shared via the original site.
5 comments

About the Author

Dan RussellDan Russell

I study the way people search and research. I guess that makes me an anthropologist of search. While I work at Google, my blog and G+ posts reflects my own thoughts and not those of my employer. I am FIA's Future-ist in Residence. More »

Recent News

  • Deepfakes and the Future of Facts
    Deepfakes and the Future of FactsSeptember 27, 2019
  • Book cover for Joy of Search by Daniel M. Russell
    The Joy of Search: A Google Insider’s Guide to Going Beyond the BasicsSeptember 26, 2019
  • The Future of Facts in a ‘Post-Truth’ World
    The Future of Facts in a ‘Post-Truth’ WorldMay 15, 2018
  • The Future of Virtual and Augmented Reality and Immersive Storytelling
    The Future of Virtual and Augmented Reality and Immersive StorytellingJune 6, 2017

More »

Upcoming Events

There are no upcoming events scheduled. Please check back later.
Event Archive »
Video Archive »

Join Email List

SearchReSearch

  • Answer: Musicians travels–how did they get from A to B?
    Answer: Musicians travels–how did they get from A to B?February 1, 2023
  • A personal note:  Transitions
    A personal note: TransitionsJanuary 24, 2023
  • SearchResearch Challenge (1/18/23): Musicians travels–how did they get from A to B?
    SearchResearch Challenge (1/18/23): Musicians travels–how did they get from A to B?January 18, 2023
  • Modifying a Reverse Image Search Query: Google Lens and Reverse Image Search
    Modifying a Reverse Image Search Query: Google Lens and Reverse Image SearchJanuary 16, 2023

More »

University of Maryland logo
Robert W. Deutsch Foundation logo
Google logo
Barrie School
Library of Congress logo
State of Maryland logo
National Archives logo
National Geographic Society logo
National Park Service logo
Newseum logo
Sesame Workshop logo
Smithsonian logo
WAMU
© 2023 The Future of Information Alliance, University of Maryland | Privacy Policy | Web Accessibility