All posts by rexpiyum

Tracking, Animating, and 3D Printing the Freehand Drawing Process

Our interactive drawing setup(left), 2D visualization(middle) and 3D bas-relief model(right) generated by our system

In order to visualize the techniques, process, and emotions of sketch artists, we have sought to display elements of traditional drawing processes. To do so, we created an interactive system that unobtrusively tracks the freehand drawing process (movement and pressure of artist’s pencil) on a traditional easel. The system outputs recorded information using video renderings and 3D-printed sculptures.

To test our system, we held a user study with 6 experienced artists who created multiple pencil drawings using our easel. The resulting digital and physical outputs from our system revealed vast differences in drawing speeds, styles, and techniques. The easel, video renderings, and bas-relief sculptures will be presented at the ACM Twelfth International Conference on Tangible, Embedded and Embodied Interactions (TEI 2018) in Stockholm, Sweden. You can read the write up here: (TEI 2018 publication)

Link for the youtube video.


Our interactive system is a traditional drawing easel which has been augmented with a pencil tracking system and a pencil pressure sensing system.

Pencil Tracking

To track the movements of the pencil, our system uses two cameras, which are mounted on the top and left sides of the easel. Images captured by the cameras are used to determine the vertical and horizontal location of the pencil. To make the tracking easier, we covered the drawing pencils in a layer of blue ink and mounted green colored background strips along the bottom and right edges of the easel. The horizontal and vertical locations of the drawing pencil is determined by locating the blue color blob created by the pencil against the green background.

Pressure Sensing

The pencil pressure sensing system is based on acoustic sensing, since we observed that the sound created by friction between the pencil and paper can be used to approximate the pencil pressure. While this relationship is not reliable enough to measure subtle variations of pressure, it is sufficient for detecting the major changes. To record sound, we placed 12 modules (each containing a microphone and microcontroller) in a 3 X 4 grid on the back side of the easel. Weighted averages of the three sensors closest to the pencil are used to determine the pencil pressure exerted on the drawing surface.

You can read more about the development of the easel here

Visualizing the Data

To display the recorded data, we chose to render the pencil speed and the pressure as an animation. In the animations, the pencil speed is determined by calculating the distance between data points. The pencil strokes which were drawn in slow, medium, or high speeds are represented distinctly in the visualization using green, yellow, and red colors. The different pressure levels are depicted using different line thicknesses.

In addition, we created another program that generates 3D bas-relief models displaying the drawing data. Bas-relief is a type of sculpture that consists of a projected image with little overall depth, such as Egyptian hieroglyphs or coins. In our models, the thickness of the ridges is based on the speed of the drawing, while the height of the ridges is based on the pressure of the drawing stroke. The height of the ridge can be compounded if several lines are drawn over the same area.


Artists Exploring the System

To explore the possibilities of our system, we conducted a study with six local artists, including MFA students, cartoonists, and a primary school art teacher. Each artist was invited to a drawing session during which they created three sketches, two of objects in the room (a lamp and flower pot) and one of whatever they wanted. In between creating the sketches, the artists were shown the video rendering and the 3D bas-relief rendering of the sketch they had just completed. All artists who took part in our study considered our tracking system to be unobtrusive and were interested in seeing the visualizations of their pencil movements.

Corresponding 2D renderings of pencil drawings by 4 different artists

The video renderings revealed unique characteristics among the drawing styles of participants. For example, they clearly showed that some participants, particularly the cartoonists, tend to use thicker lines in their drawings when compared to the others. The artists felt that the system could be useful both for teaching beginning artists and as a tool to study the evolution of a particular artist’s style.

Tracking Artist Hand Movements

Recently we started a new project to analyze hand movements of artists while they are drawing on a physical drawing canvas. The goal of this project is to uncover the latent elements of the creative process of artists while they draw on a canvas and combine them with the final static art piece to create a novel art experience. To achieve this we capture the hand movements of the artists while they are engaged in the drawing process. Then the captured information is displayed as part of a visualization that will be superposed into the actual art piece to create dynamic form of art.

In our first experiment we used a LEAP motion sensor to capture the finger and palm position of the right hand(drawing hand) of artist while they draw on a paper mounted in easel. Then we create a dynamic visualization by plotting the angle between the right index finger and the center of the palm over the period of drawing.



Hand movement superposed on the original art work

Since the LEAP’s spatial location data is not very accurate, now we are building a new experiment setup. The new system is composed of an easel with two cameras: one to capture the vertical position of the drawing instrument and a second one that captures its horizontal position. A LEAP motion sensor is also used to track the finger and palm during the drawing process. At the end LEAP’s data will be combined with data captured from 2 cameras accurately re-produce the finger and palm movements of the artists. Following image shows our new setup.


Bio workshop at HeatSync

Hi folks, as we are continuing our work in DIY biology with general public and non-professional biology hobbyists, last week Cass, Stacey and Me conducted a DIY bio workshop at HeatSync Labs in Meza, Az. HeatSync is a community driven maker-space, one of the coolest places I’ve ever been in Arizona. Unfortunately Matt couldn’t make it this time, even though he immensely contributed in organising and planning the workshop.DSC_0414.JPG

The first part of the workshop was about yoghurt fermentation. Cass explained the steps of yoghurt fermentation process and worked closely with the participants in the process. Here are some images.

Then we moved to the second part of the workshop – Gram staining. In this activity participants were asked to follow instructions printed on the card given by us, as well as Cass’ guidance. Everyone was so excited to see the microscopic images of the slides they have created, actually results were awesome!


While everyone was partying with bacterias in the downstairs, I was busy connecting the camera of our DIY incubator to the HeatSync WIFI network. Yes, now we have a WIFI camera inside our incubator as we promised in one of our earlier posts!

We left our incubator and some basic materials at heat syncs lab, so that they can play with them in the summer. Hopefully we will get some useful feeds from the camera too!DSC_0411

I’m Piyum, signing off and running to catch the flight to CHI 16 to present our Bio work there. More on that later! Thanks for reading.


Here we were at CHI 16 poster session.




A low cost and accurate incubator for DIY biology

DSC_0958DIYbio, (Do It Yourself) biology is a growing movement that aims to make biology accessible outside of professional contexts. Over the past few years, platforms such as OpenPCR and Pearl Biotech transilluminator have been designed to support biology work in schools and maker spaces for a fraction of the cost of professional lab equipment. These exciting developments pave the way for crowdsourcing biology research amongst students, hobbyists, and non-experts. (Matt wrote a nice article introducing our work on DIYBio at SANDS.)

This post is about our contribution to the growing number of open source biology tools by developing a low-cost, yet relatively precise and easy-to-use incubator. An incubator is an essential tool for a number of biological experiments and is often used in bacterial cell culture experiments. Given the higher price range of the commercially available incubators (upwards of several thousand dollars), non-professional biology enthusiasts might not afford to add an incubator to their stock of DIYbio tools. In our instructable, we describe how to build a low cost (under $70), yet accurate ( +/- 0.25C) DIY incubator using simple materials and some basic electronics components.image

The goal of the incubator is to keep a constant temperature inside. On the most basic level, we are using a temperature sensor and a light bulb as our heating element. The change in temperature inside the incubator is proportional to the amount of heat emitted by the tungsten bulb minus the heat loss through the walls of the incubator. In other words, to increase the temperature, the bulb needs to emit more heat than the heat loss through the walls. Once the desired temperature is reached, the bulb should emit the same amount of heat as what’s lost.The amount of heat emitted by the tungsten bulb is proportional to the power applied to the bulb. Hence, we can control the temperature inside the incubator by controlling the AC input supplied to the bulb. Our design contains a custom built AC phase control circuit, an Arduino Uno, an Adafruit Max31855 thermocouple amplifier and a K-type thermocoupler, an Olimex LCD Shield (16×2 LCD with 4 buttons).

Please refer to our Instructable post for detailed implementation details.

Last week me, Matt and Stacey tried an experiment to culture E.Coli  in our DIY Bio lab and used our incubator at constant 37C temperature for around 24 hours. Happily, everything went super smoothly this time. (Even though we had few hiccups in testing stages because of some firmware bugs!)DSC_0886

In next steps, we plan to build a shaker and add a camera to the incubator to have a “live feed” of what’s happening inside. We will keep you updated about new developments of this project via SANDS blog. Until then, thanks for reading, I’m Piyum Fernando here, signing off.