Dreamine

About Verizon Connected Futures III:

The Verizon Connected Futures Prototyping and Talent Development program, in partnership with NYC Media Lab, supports the development of new media and technology projects from universities across New York City, linking students and faculty with industry experts. Verizon funds 12 prototyping projects in Virtual & Augmented Reality and Artificial Intelligence to NYC university teams.

Press:

Parsons Students Prototype AR, VR, and AI Projects as Part of Verizon Connected Futures III

VERIZON FUNDS 11 PROTOTYPING PROJECTS FROM NYC UNIVERSITY TEAMS

Our team is the Dreamine team. We prototyped an AR modelling clay game for children aged 5 years and up that brings their creations to life in the digital world. It aims to expand physical play to digital interaction and inspires children’s creativity and hands-on skills.

Year: 2018
Project Type: Team Project
My Role: Team Leader, UX Designer and Programmer
My Deliverables: User Interviews, Prototyping, and User Testing
Tech: Vuforia, Unity and Computer Vision
Team Members: Shuang Cao, Ying Xiao, Fanyun Peng

I found a project by Thomas Romain. He made a workshop named “Father and Sons’ Design.” His sons created original characters, and then Thomas drew illustrations of their creations.

I thought about how we use contemporary technology to provide children with a greater creative space. Also, I think hands-on ability is another important skill for children to have, especially for children born in the digital age. How can digital products stimulate children to explore the world by hand? What if I design a product to combine hands-on work with AR technology to stimulate children with imaginative and creative thinking?

Brainstorming

We brainstormed on how to combine hands-on skills with our contemporary technologies. Then we decided to use play dough. It is a great tool for kids to play and create things with their hands.

Through research, we decided to combine AR technology and play dough to encourage children in think creatively and make with their hands.

Process

We wanted to build an AR modeling clay game for children who age 5 years and up, which can bring their creations to life in the digital world, how to combining AR technology, play dough and storytelling to stimulate children's creativity and collaborative skills.

Persona

Storyboard

Prototype 1

We created a story outline for my application to make kids more engaged with the game. The story is about a warrior Jack's adventure. We also made a map for it. The first prototype focused on the AR interaction.

For the user test, users appreciated the UX design. The thought of having a story to give different missions to kids was a good choice. However, some people thought the story should be more educational. The story should convey a concept that kids can learn from.

Prototype 2

So for my second prototype, I wanted to build the scanning function. It was the main technical challenge for my project. I tried two different ways: 3D Scanning and Image Processing.

Then I did a user test with a 4-year-old girl. I observed her playing with play dough with and without her parents. I found an important thing. Without the help of her parents, it was hard for her to build something or transfer what she was thinking to what she had thought. She was upset if objects she made were different from what her think. So I decided to add cutters and stampers to help kids make their creations.

Techniques

After the user test, we decided to use Computer Vision to recognize shapes made by cutters. We did a lot of research on image recognition. I tried different algorithms to extract image features. I spent four weeks to test those algorithms. I tried SIFT and MSER algorithms to extract image features, which turned out non-ideal results. Then I tried the Find-contours example in openCV library. It could detect certain shapes. But it was hard to recognize different shapes. Finally I found an algorithm named Edge Histogram Detection. I Combined it with another way, matching the corners of shapes to recognize all shapes.

Prototype 3

We laser cut some cutters and stampers. To make the game more meaningful we also changed our story. We created a virtual land named Dreamland. People are all craftsman there. Their abilities of imagination and creativity are all from a magic stone. One day the magic stone is destroyed into pieces. The princess of the Dreamland is on a journey to find the pieces of the stone and bring it back to dreamland. Children need to help her during the journey. For this time, users will firstly view the story introduction. Then they can point the camera at the map at the first place to start the mission. It is a linear successional game. Children need to use cutters and stampers to create different characters to finish missions level by level.

Demo Presentation Day on Verizon Alley

Demo on 2018 Games for Change Festival

Prototype 4

For this demo, we changed some interactions in former scenes. Also, we added a new scene to enrich the game content. For the first scene, to make users familiar with the main operations of our game we added a letter for users to click to read. We added a cave scene. Players should use play-dough and cutters to make an energy bottle to get energy. For the final scene, Players should control the elf created in the second scene to generate the energy boom to defeat the monster.

Dreamine-Content