Background Image

Team Project

UX Researcher
User Experience Designer
Visual Designer

Nikita Spirin (HCI Phd Candidate)
Motahhare Eslami (HCI Phd Candidate)
Pooja Jain (Master in Computer Sicence)
Brian Bailey (Project Advisor)

Spring-Fall, 2013

Design Challenge

Examples are very important in design, but existing design search tools still do not cover many search cases. Long tail queries containing subtle and subjective concepts, representing regular design search needs, but at the same time, are poorly supported. The key problem is due to the complexity of the task, which so far has been tackled only algorithmically using general image search techniques.

Subtle and subjective search needs are poorly supported

Proposed Solution

A crowdsourcing design community based on AMT (Amazon Mechanical Turk) that helps designers searching for examples more efficiently with quality control.
Read our paper work: Searching for Design Examples with Crowdsourcing.


"Human performs better then search engines when it comes to subtle queries"

We propose an approach for design search based on crowdsourcing based on the assumption that "Human performs better then search engines when it comes to subtle queries”. To enable crowdsourced design search, we explore multiple strategies for design search need explanation to the crowd, results collection, and quality control. Since it relies on people, queries could be formulated using a combination of natural language and images, which is quite intuitive for designers.


Assumption feedback from users

Seven designers and four architects responded and were interviewed for us to develop a deep understanding of the problem. Participants had a diverse professional experience (2 – 25 years, the average is 8.5 years). The interview questions focused on the ways creative professionals communicate concepts and what expectations do they have for a “magical” search engine that could help them find design examples.

Participants were skeptical about a design search engine at first, but changed their attitude when they were informed that the search process is performed by people. However, most of the participants concerned about the capabilities of non-professional workers online. We got a suggestion to limit the scope to only high quality examples and, hence, guarantee that results will be at least marginally useful. Interestingly, it confirms our choice of a collection-driven approach. The major advantage that professionals saw in such a tool was due to potential time savings.

Proposed Solution

Conceptual Workflow for
Crowdsourcing Design Search

Our idea is to first build a collection of high quality design examples by crawling curated design websites and partition it into disjoint subsets of design examples. Then, at a query time, we will assign workers to subsets and ask them to judge relevance of each example with respect to a particular query. It's done by AMT (Amazon Machanical Turk).

Background Image

We recruited participants on AMT in April-October 2013. We didn’t specify any constraints on workers’ qualifications. 728 workers provided 63450 judgements and completed 1300 tasks.

To come up with the best way to convey a design search need to the crowd.

Since the ultimate goal is to generate high quality design examples with minimal cost. We experimented with different strategies to improve quality, reduce AMT workers’ efforts and query processing cost.

Product Image

A/B testing against Google Images

The goal of this experiment is to understand whether our approach can retrieve more relevant and diverse design examples compared to Google Images. In this experiment, We considered 3 real design needs coming from different design domains: Web design, Typography, Interior design.

Top: Results retrieved by Google Images in a query-by-example mode. The query image is shown in the upper left corner of each screenshoot. The scope is limited to for web design and for interior design. Bottom: Results retrieved with our crowdsourcing-based approach. The results retrieved with our approach are much more relevant. Moreover, our results are diverse.

Product Image

Design Sherlocks

We created a system called “Design. Sherlocks”. It’s a crowdsourcing design platform for designers searching for high quality design examples. Designers can post queries, name price and narrow search pools for quality control on Design Sherlock. The actual tasks run on AMT in the back. We also designed the mobile version to help designers track their tasks on the go.

Background Image
Background Image
Background Image
Background Image
Background Image
Background Image