Design sherlock,
Crowdsourcing search platform

I collaborated with two engineers on an interesting crowdsourcing project. We experimented around the question if collective intelligence can lead to a better search tool for objects detection, feelings, layouts, which are poorly supported by the existing search engines. Together, we built a crowdsourcing design platform based on our findings.

2013-2014, Paper work: Searching for Design Examples with Crowdsourcing.
Background Image

Challenge

Subtle and subjective search needs are poorly supported

Examples are very important in design, but existing design search tools still do not cover many search cases. Poor performance for objects detection, feelings, layouts. Long tail queries containing subtle and subjective concepts, representing regular design search needs, but at the same time, are poorly supported.

Background Image Background Image

Assumption

Human performs better than search engines when it comes to subtle queries

We propose an approach of crowdsourcing search, based on the assumption that "Human performs better than search engines when it comes to subtle queries”, with the help of workers on Amazon Mechanical Turk (AMT) crowdsourcing platform. Since it relies on people, queries could be formulated using a combination of natural language and images, which is quite intuitive for design search.

Validation

Interviewing professionals

Before starting this work, we conducted interviews with creative professionals to learn about their typical design search needs. All together, 11 interviews were conducted. Participants had a diverse professional experience (2 – 25 years, the average is 8.5 years). The interview questions focused on the ways creative professionals communicate concepts and what expectations do they have for a “magical” search engine that could help them find design examples.

Most of the participants concerned about the capabilities of non-professional workers online. We got a suggestion to limit the scope to only high quality examples and, hence, guarantee that results will be at least marginally useful. Interestingly, it confirms our choice of a collection-driven approach. The major advantage that professionals saw in such a tool was due to potential time savings.

Background Image

Proposed Solution

Crowdsourcing design search

Our idea is to first build a collection of high quality design examples by crawling curated design websites and partition it into disjoint subsets of design examples. Then, at a query time, we will assign workers to subsets and ask them to judge relevance of each example with respect to a particular query. It's done by AMT (Amazon Machanical Turk).

Background Image

Experiment-1

Exploring design query formulation

Since the ultimate goal is to generate high-quality design examples with minimal cost. We experimented with different strategies to improve quality, reduce AMT workers’ efforts and query processing cost. We experimented different query combinations with 728 workers, who provided 63450 judgements and completed 1300 tasks.

We found that it is important to use visuals. Importantly, both positive and negative examples are needed since example-driven explanations highlighting contrasting concepts is the most successful educational technique.

Background Image

Experiment-2

A/B testing against Google Images

As an ultimate evaluation, we compare our approach with Google Images in a query-by-example mode and show that the crowd can select more relevant and diverse design exam- ples, according to judgements of expert designers.

Background Image

Design scherlock

Crowdsourcing search platform

We created a system called “Design. Sherlock”. It’s a crowdsourcing design platform for designers searching for high quality design examples. Designers can post queries, name price and narrow search pools for quality control on Design Sherlock. The actual tasks run on AMT in the back. We also designed the mobile version to help designers track their tasks on the go.

Background Image
Background Image
Background Image
Background Image
Background Image
Background Image