How to build an AI stylist inspired by outfits on Instagram
The architecture
I built this app using a combination ofGoogle Cloud Storage,Firebase, and Cloud Functions for the backend,Reactfor the frontend, and theGoogle Cloud Vision APIfor the ML bits. I divided the architecture into two bits.
First, there’s thebatch process, which runs every hour (or however frequently you like) in the Cloud:
“Batch process” is just a fancy way of saying that I wrote a Python script which runs on a scheduled interval (more on that later). The process:
This is really the beefy part of the app, where all the machine learning magic happens. The process makes outfit recommendations and writes them to Firestore, which is my favorite ever lightweight database for app development (I use it in almost all my projects).
The actualapp(in this case, just a responsive web app) is simple: it just reads the outfit recommendations from Firestore and displays them in a pretty interface:
Let’s take a look!
Grabbing social media data
Ideally, I wanted my app to pull pictures from Instagram automatically, based on which accounts I told it to follow. Unfortunately, Instagram doesn’t have an API (and using ascraperwould violate their TOS). So I specifically asked Laura for permission to use her photos. I downloaded them to my computer and then uploaded them to aGoogle Cloud Storage bucket:
Filtering for fashion pictures
I like Laura’s account for inspiration because she usually posts pictures of herself in head-to-toe outfits (shoes included). But some pics on her account are more like this:
Adorable, yes, but I can’t personally pull off the dressed-in-only-a-collar look. So I needed some way of knowing which pictures contained outfits (worn by people) and which didn’t.
For that, I turned to my trusty steed, theGoogle Cloud Vision API(which I use in many different ways for this project). First, I used its classification feature, which assigns labels to an image. Here’s the labels it gives me for a picture of myself, trying to pose as an influencer:
The labels are ranked by how confident the model is that they’re relevant to the picture. Notice there’s one label called “Fashion” (confidence 90%). To filter Laura’s pictures, I labeled them all with the Vision API and removed any image that didn’t get a “Fashion” label. Here’s the code:
If you want the full code, check it outhere.
Digitizing my closet
Now the goal is to have my app look at Laura’s fashion photos and recommend me items in my closet I can use to recreate them. For that, I had to take a pictures of item of clothing I owned, which would have been a pain except I happen to have a very lean closet.
I hung each item up on my mannequin and snapped a pic.
Using the vision product search API
Once I had all of my fashion inspiration pictures and my closet pictures, I was ready to start making outfit recommendations using theGoogle Vision Product Search API.
This API is designed to power features like “similar product search.” Here’s an example from the Pinterest app:
IKEA also built a nice demo that allows customers to search their products via images with this kind of tech:
I’m going to use the Product Search API in a similar way, but instead of connecting a product catalog, I’ll use my own wardrobe, and instead of recommend similar individualitems, I’ll recommend entire outfits.
To use this API, you’ll want to:
At first I attempted this using theofficial Google Python client library, but it was a bit clunky, so I ended up writing my own Python Product Search wrapper library, which you can findhere(on PyPi). Here’s what it looks like in code:
Note this wrapper libraryhandles uploading photos to a Cloud Storage bucket automatically, so you can upload a new clothing item to your product set from a local image file:
Searching for similar items
Once your product set is done indexing, you can start using it to search for similar items. Woohoo! ?
In code, just run:
The response contains lots of data, including which items were recognized in a the source photo (i.e. “skirt”, “top”) and what items in your project set matched with them. The API also returns a “score” field for each match which tells you how confident the the API is that an item in your product set matched the picture.
From matching items to marching outfits
The Product Search API looks at an inspiration picture (in this case, Laura’s fashion pics) and finds similar items in my wardrobe. But what I really want to do is put together wholeoutfits, which consist of a single top, a single pair of pants, a single set of shoes, etc. Sometimes the Product Search API doesn’t return a logical outfit. For example, if Laura is wearing a long shirt that looks like it couldalmostbe a dress, the API might return both a similar shirt and dress in my wardrobe. To get around that, I had to write my own outfit logic algorithm to build an outfit from the Search API results:
Scoring outfits
Naturally, I couldn’t recreate every one of Laura’s outfits using only items in my limited wardrobe. So I decided my approach would be to look at the outfits I could most accurately recreate (using the confidence scores returned by the Product Search API) and create a “score” to sort the recommended outfits.
Figuring out how to “score” an outfit is a creative problem that has no single answer! Here are a couple of score functions I wrote. They give outfits containing items that have high confidence matches more gravitas, and give a bonus to outfits that matched more items in my closet:
Putting it all together
Once I had written all the logic for making outfits in a Python script, I ran the script and wrote all the results to Firestore. Firestore is a serverless database that’s designed to be used easily in apps, so once I had all my outfit matches written there, it was easy to write a frontend around it that made everything look pretty. I decided to build a React web app, but you could just easily display this data in a Flutter or iOS or Android app!
And that’s pretty much it! Take that, expensive stylist.
Thisarticlewas written byDale Markowitz, an Applied AI Engineer at Google based in Austin, Texas, where she works on applying machine learning to new fields and industries. She also likes solving her own life problems with AI, and talks about it on YouTube.
