Dylan Vassallo 😁

How we built Classyfy

April 23, 2012

Hackathon

At 5 PM last Friday, roughly one hundred Bruins descended upon the Career Center to take part in the Facebook Hackathon at UCLA. With the help of friendly Facebook engineers and ridiculous amounts of food and caffeinated beverages, teams of up to four students worked through the night to build a project that would then be presented in a two-minute demo at the end of the competition. After 24 insane hours of designing, coding, debugging, and polishing, Team Out and About (Angela, Samir, Prasanth, and myself) took first place with our app Classyfy. Classyfy is a fun way to take your party photos from trashy to classy: simply log in with Facebook, choose a photo album, and watch as the red party cups in your images are detected and replaced with juice boxes, sandwiches, Red Bull cans, or even pineapples. It’s not foolproof, but it’s a lot of fun.

Zuck, Classyfied

It was a great project to work on and I’m really proud of my teammates for how well we worked together in executing on our vision. In November we’ll be heading up to Facebook HQ to compete against the winners from other schools, but for now I’d like to share some details about how we built Classyfy.

Classyfy architecture

The Classyfy app can be thought of as two connected systems, which I’ll call the “frontend” and the “backend”. The frontend is a pretty standard web application, handling the Facebook authentication flow and serving rendered templates to users. The backend is where the heavy lifting of processing the images takes place. We chose to host the frontend on Heroku, which was a time saver in several big ways. Not only did we get HTTPS (required for Canvas apps) and git-based continuous deployment for free, but upon creating our app in the Facebook Developers interface, we specified that we wanted to use Heroku and in seconds were able to clone down some skeleton code that was all set for use with the Open Graph. Heroku provides this starter code in several languages including Ruby, our weapon of choice. Even better, the Ruby template is built on the Sinatra microframework, which is perfect for hackathon-sized projects, and the excellent Koala gem for accessing the Facebook API.

Instead of using the PostgreSQL database that comes with every Heroku app, we set up Redis on an Amazon EC2 instance. I had already been in love with Redis for some time, and Sinatra plus Redis is a wicked fast and flexible combination that is perfectly suited for hackathons. Adding to the awesome is Resque, a Redis-backed job queue written in Ruby that comes with a slick admin interface. Together, Sinatra, Resque, and Redis comprise Classyfy’s plumbing. Users interact with the Sinatra frontend, images to be processed are pushed onto the Resque job queue, and Redis is the persistent in-memory datastore that ties everything together. Because Heroku runs in Amazon’s Virginia datacenter, there was minimal latency between our frontend Heroku app running Sinatra and our Redis- and Resque-powered backend running on an EC2 instance in that same datacenter.

As for the image processing itself, we experimented with Camellia and its Ruby bindings but ended up writing a C++ program (executed by the Resque workers) that uses OpenCV and cvBlobsLib to do the red cup detection. Finding red blobs in an image is easy, but figuring out which are actually red cups isn’t; for that we used several heuristics based on the shape, size, and other properties of each blob. The C++ executable outputs information about each detected red cup that is shuttled into ImageMagick for overlaying the juice boxes over the photo. Once that’s finished, the modified image is uploaded to S3 and marked as ready to go in the Redis representation of the photo album. Once all the photos in the album are processed, the user is taken to the results page where they are free to share their newly classyfied images with prospective employers to their heart’s content.