Cloudinary's guide to extracting metadata to search, sort and classify your images

There are many image tools available to web designers, from image file compression tools to color picking tools and more. These tools help you to utilise images in a way that will provide the best user experience for your site, as well as ensuring images display properly on every screen size whether desktop or mobile.

Manipulating images can be easily accomplished using familiar photo editing applications, or by using a cloud-based option such as Cloudinary. What you may not be aware of is the myriad of data that is embedded in digital imagery, and this woefully-underused semantic information could add value to your web designs.

If you were to realise this data's potential, you could hone in on color-specific user uploads for a specific site page; filter only photos that show your users' faces; select photos taken with newer DSLR cameras or a specific smartphone brand; or even use geo-data to select photos taken in your site visitor's locale.

The exciting news is that Cloudinary's API gives you control over the extraction of the rich data in your website and mobile app's photos, allowing you to search, sort and classify your images in ways you may not have considered. Here's how.

Predominant colors

Cloudinary supports image color search – similar to that used by Google Image Search's search tools – by using a standard palette of 12 main colors and analyzing your site's images to discover the predominant color. And as Cloudinary is a cloud-based service (hence the name!), the image processing is done online with no software installation required.

Click here to set up a free Cloudinary account

This capability is useful for advanced search functionality on stock photo sites (click here to find out how to build a stock photo site with Cloudinary) and for ecommerce sites, such as clothes retailers who want to allow customers to refine a search to a garment's color.

The image above with the public ID 'fashion1' was uploaded to Cloudinary. By using Cloudinary's Admin API, you can extract the prominent colors in the photo by setting the 'colors' parameter to true (see reference documentation). Here are examples for Ruby, PHP, Node.js and Python:

Cloudinary::Api.resource('fashion1', :colors => true)

$api->resource("fashion1", array("colors" => TRUE));

cloudinary.api.resource('fashion1',  
                        function(result)  { console.log(result) }, { colors: true });

cloudinary.api.resource("fashion1", colors = True)

Below is the JSON result of this API call. As you can see, the dominant colors of image 'fashion1' are white (50.7%) and blue (27.8%), with the next prominent being gray and brown.

{
  "public_id": "fashion1",
  "width": 225,
  "height": 380,
  ...
  "predominant": {
    "google": [
      [ "white", 50.7 ],
      [ "blue",  27.8 ],
      [ "gray", 11.2 ],
      [ "brown", 5.1]
    ]
  }
}

Using this data, you can keep the color mapping in your model and allow clothes searches to be refined based on colors. Searching for blue clothes should return the product.

Another result you get as part of the color information API is a histogram of 32 RGB colors that best represent the image. This JSON snippet was included in the result of the API call:

{
  "public_id": "fashion1",
  ...
  "colors": [["#FFFFFF", 50.7], ["#011B43", 5.8], ["#5077A7", 4.9], ["#031235", 4.3], ["#F4CBB4", 3.3], ["#3A6498", 1.9], ["#6284AF", 1.9], ["#2D5E95", 1.9], ["#30578B", 1.8], ["#080918", 1.8], ["#E5B09D", 1.8], ["#36262F", 1.7], ["#264876", 1.6], ["#281A25", 1.5], ["#486A99", 1.4], ["#E3D6CF", 1.4], ["#4D3135", 1.4], ["#07264F", 1.2], ["#664E55", 1.1], ["#6E443C", 1.0]]
}

This represents the RGB format and percentage breakdown of the 32 colors that best represent the image. The hex value '#FFFFFF' is white, and is the same 50.7% as the top JSON result, and the hex codes that immediately follow are mainly shades of blue.

Face detection info

Cloudinary also supports face detection based cropping and pixelation, which means that single and multiple faces can be automatically detected. Simply set the 'faces' parameter to true in the same way demonstrated above for 'colors'.

You can enable multiple flags in a single call for fetching all information at once, and that result includes the exact coordinates of all detected faces in the image, meaning you can discover how many faces are in the photo and where they are.

The following Ruby command asks for the faces information of the 'fashion1' image:

Cloudinary::Api.resource('fashion1', :faces => true)

Here is the JSON result:

{
  "public_id": "fashion1",
  ...
  "faces": [[99, 21, 64, 87]]
}

So a single face was detected in the [99, 21, 64, 87] rectangle of the image, in the standard pattern of [top, left, right, bottom]. The same command works for an image like the one below, which contains many faces, and returns the coordinates of each rectangle containing a face:

{
  ...
  "faces":  [ [513, 19, 38, 52], [409, 26, 40, 54], [79, 31, 43, 59], [232, 32, 40, 54], [321, 33, 41, 57], [160, 37, 43, 59], [211, 153, 43, 59], [503, 151, 43, 59], [113, 162, 40, 54], [427, 160, 45, 61], [307, 172, 48, 65] ]
}

However, face detection is not 100 per cent accurate so it is important to have this process verified by the human eye.

Camera information

All modern digital cameras, including smartphone cameras, store additional metadata in your images via the exchangeable image file format (Exif). Exif metadata includes information on camera model, picture orientation, timestamps, GPS location, exposure, and more depending on your camera model.

Cloudinary's API can return an image's metadata by simply setting the 'exif' parameter to true (see our reference documentation). This can not only be done when requesting information from previously uploaded images, but also while uploading photos, so it is returned as part of an upload response.

The following PHP command uploaded an iPhone photo to Cloudinary that was shot in portrait orientation:

\Cloudinary\Uploader::upload("exif_sample.jpeg", 
   array("public_id" => "exif_sample", "colors" => TRUE, "exif" => TRUE))

And here's the JSON of the upload response including the requested Exif and colors information:

{ 
  "public_id": "exif_sample",
  "width": 2592,
  "height": 1936,
  ...
  "exif": {
    "ApertureValue": "4281/1441",
    "ColorSpace": "1",
    "ComponentsConfiguration": "1, 2, 3, 0",
    "Compression": "6",
    "DateTime": "2010:12:27 11:17:34",
    "DateTimeDigitized": "2010:12:27 11:17:34",
    "DateTimeOriginal": "2010:12:27 11:17:34",
    "ExifImageLength": "1936",
    "ExifImageWidth": "2592",
    "ExifOffset": "204",
    "ExifVersion": "48, 50, 50, 49",
    "ExposureMode": "0",
    "ExposureProgram": "2",
    "ExposureTime": "1/4309",
    "Flash": "24",
    "FlashPixVersion": "48, 49, 48, 48",
    "FNumber": "14/5",
    "FocalLength": "77/20",
    "GPSAltitude": "20723/924",
    "GPSAltitudeRef": "0",
    "GPSImgDirection": "42155/344",
    "GPSImgDirectionRef": "T",
    "GPSInfo": "574",
    "GPSLatitude": "21/1, 768/100, 0/1",
    "GPSLatitudeRef": "N",
    "GPSLongitude": "86/1, 4500/100, 0/1",
    "GPSLongitudeRef": "W",
    "GPSTimeStamp": "17/1, 17/1, 3326/100",
    "ISOSpeedRatings": "80",
    "JPEGInterchangeFormat": "870",
    "JPEGInterchangeFormatLength": "9932",
    "Make": "Apple",
    "MeteringMode": "1",
    "Model": "iPhone 4",
    "Orientation": "6",
    "ResolutionUnit": "2",
    "SceneCaptureType": "0",
    "SensingMethod": "2",
    "Sharpness": "2",
    "ShutterSpeedValue": "4781/396",
    "Software": "4.2.1",
    "SubjectArea": "1295, 967, 699, 696",
    "WhiteBalance": "0",
    "XResolution": "72/1",
    "YCbCrPositioning": "1",
    "YResolution": "72/1"
  },
  "colors":[["#CBC9C5",10.2],["#C4BCB4",9.0],["#1888AB",6.0],["#202618",6.0],["#226391",5.4],["#223A62",4.3],["#B9B4AD",3.8],["#2F88A1",3.5],["#C9C3BA",3.4],["#7492B2",3.4],["#157193",3.1],["#96ABCC",2.9],["#C8B495",2.8],["#4F97AB",2.8],["#484033",2.7],["#669FAD",2.5],["#A0A29E",2.4],["#38A7C8",2.3],["#57A5B7",2.3],["#2D8FAF",2.2],["#ACCADC",2.1],["#073554",2.0],["#60AFC7",2.0],["#1D4A6F",2.0],["#A39477",1.9],["#D1C4A0",1.8],["#296F96",1.7],["#4F6E91",1.5],["#5F5F57",1.4],["#90AECB",1.0]],
  "predominant": {"google":[["teal",41.7],["brown",35.6],["blue",12.1],["green",8.4]]
}

A clever feature here is Cloudinary's Exif-based automatic rotation – simply set the 'angle' parameter ('a' for URLs) to 'exif', and voila:

http://res.cloudinary.com/demo/image/upload/a_exif/exif_sample.jpg

Community

Thanks to these features, you can utilise image metadata and semantic information to enhance your image rich web and mobile applications with little effort – with Cloudinary doing all the heavy lifting for you.

The ability to fetch predominant colors, color histogram, faces and Exif metadata is available to all of Cloudinary's plans, free and paid, allowing Cloudinary to offer its best-ever solution to your online image management and manipulation needs.

Click here to set up a free Cloudinary account