Archive

I have many fascinations in life, and many in data. One is broadly representing data through the senses (surprise surprise) and the other is autographic data visualizations. I came across this concept from Dietmar Offenhuber (whose book on the subject I impatiently await). In a nutshell, autographic data visualizations are ‘naturally occurring’ data visualizations.

Take the following examples:
The rust on the penny visualizes its age. The pH strip reacts with the substance in the paper to change its color based on acidity.

The tracks that the fly leaves behind on the window reveal its path over time.

It's been a while! Jordan and I have been busy with life and fun workshops that we have been facilitating. If I had to play favorites, I'd choose the Human Synthesizer experience that I mentioned in the last newsletter. Jordan ultimately facilitated three, each one with different themes that the participants pulled out.

What we did

Most of the data that we encounter is far removed from our personal experiences. Our eyes might gloss over, or we might have difficulties connecting a chart's peaking line to actual events. Getting an audience to spend five minutes with a chart is enough of a challenge, let alone cultivating empathy to put faces to the numbers. Jordan and I are always trying to think of novel ways we can interact with data and make it more personal and intuitive - using our own data or employing all the senses to widen our minds as to what the data can mean, represent, and give us a fresh perspective. This is called by many names: Data visceralization, embodied data, somaesthetics, data representations, and ranges by having 'new age-y' connotations to a broader design meaning. It's not new that the body and mind work together to respond to and create an experience - theories rooted in bodily processes argue that language, reasoning, etc. are ultimately rooted in bodily processes. If so, how can we use our bodies to help us learn and assimilate information?

The Human Synthesizer (aka Data Choir) exercise was one such exploration of how to use our bodies to 'experience' non-personal data.

Last time I talked about our Sensory Sketching Workshop and focused primarily on smell, a sense that I am fascinated and puzzled by. I won’t say that I came to a dead end with smell, but I did become frustrated with the logistics of representing data through smell. The fact that smell can’t be shared digitally is one obstacle, and the other is due to the transient nature of smell: it’s difficult to capture and sustain. All the DIY methods of collecting smell either only work for a handful (like flowers) and/or they are not “faithful” replicates of the actual scent (I'm looking at you, Scratch and Sniff). Until I have my own smell-atory, or more realistically, access to a lab, I will be focusing more energy on sound. As the only sense besides sight that can be distributed digitally, sound reaches a wider audience and doesn’t necessarily require physical materials. To top it, off, sound is well within Jordan’s expertise.

Sound is already being leveraged to encode data or sonification. Apple and Highcharts have already integrated an out-of-the-box audio implementation for their charts. For a more creative and musical approach, you can listen to the sonorous and fabulous sonification podcast, Loud Numbers, or check out this animation-sonification from the BBC that visualizes and sonifies COVID cases. While a wide variety of tools can be used, from physical synthesizers to using programming languages, you don’t need fancy tools or knowledge to get started. If you want to learn more about sonification, make sure to check out the Sonification Archive.

Jordan and I tend to focus on ways to create participatory data experiences that are accessible to a wide spectrum of data knowledge. Our next project is no different: we aim to assemble a one-off human data choral, which we have deemed the Human Synthesizer. And some of Jordan’s previous work — her project on data recipes (including a Heartbeat Duet she performed with her mom) and the participatory sonification she conducted at the 2021 Loud Numbers festival — employs some of the key techniques we’ll be using in the Human Synthesizer: engaging the voice and audience participation.

In the last newsletter, I introduced the Sensory Sketching Workshop that Jordan and I led for CHI2022, where our frustration with the lack of vocabulary to describe senses other than sight or sound poses an obstacle to communicating our sensory experience.

I’ve leaned into N. J. Enfield’s theories, which have supported some ideas I have absorbed from my linguistics degree: that language is built for communicating what needs to be done, but not for describing and perceiving reality. I feel this acutely when I try to describe certain odors or taste, yet fall short - I remember tasting guava for the first time, as a 21-year-old in Thailand. The best description I could muster was “apple-y”, which alarmed and frustrated me at the time. My guava experience reinforces my theory that in the western world I grew up in, smell is mostly linked to internal experiences and memories, which are subjective and not very effective in persuading others to take an action, unlike tangible evidence. Although we often employ metaphors (a “delicate” or “balanced” scent) and similises ("this guava tastes like apples"), Sissel Tolaas has gone so far as to propose the adoption of the artificial smell vocabulary, an "alphabet for the nose" or NASALO. Other than the fact that adopting NASALO isn’t realistic or natural, I don’t believe we necessarily need more words to describe odors. Although Sissel Toolas has critiqued the use of metaphors for describing smells, there are many instances of indigenous languages using metaphors to describe colors, e.g. “it’s the color of the sea” or “the color of blood" (you can't deny their vivd imagery and tangibility!) Instead of creating a new vocabulary to express smells, we can repurpose words and leverage our own experience and cultural references to develop metaphors. Due to the intimacy and imagery created by metaphors, they seem to be highly adept at expressing the complexity of smells and the memories and emotional reactions they evoke. During the workshop, we often relied on metaphors to relate our experiences or ideas to the rest of the group.

“Metaphors of sensory experience…”

“...sometimes [we] need explanation. [Some sketches] are not obvious beyond the person giving the example.”

What happens when you organize a workshop with academics in Human-Computer Interaction (HCI)? 🤯 and over-dosing on inspiration — aka our workshop at the CHI 2022 conference, Sketching Across the Senses.

From a participant researching the quantified self to organizers baking with data or specializing in sound and sonification, there were enough dream projects to almost make me consider going back to school to get my Ph.D. in HCI.

You can get a sense of the flow of ideas from our jam-packed workshop agenda:

This issue is a continuation of the last, where we dove into learnings from our four days of data collection on our sensory experiences during meals.

Phase 1: Data collection

Day 1: texture

Day 2: color

How was the last exercise? Any attempts?

Here’s the gist: we took four days of data on our sensory experiences during meals. Although we tend to associate food and eating under the umbrella sense of taste as an experience, each day we focused on collecting data from the different senses we encounter while eating:

Phase 1: Data collection

Day 1: texture

Hi readers, I'm Max (and sometimes Jordan).You're getting this email because you signed up for this newsletter, but you can unsubscribe anytime and I promise not to take it personally. Today we are diving into taste. Grab a snack or drink for a meta experience, and let's go.

I was on a breakfast mission in a market in Trang, Thailand. A few locals were sitting at a 3-seat make-shift bar, with what looked like ravioli in a thin liquid speckled with red and green. As taste descriptions were not part of my limited Thai vocabulary, I took the low-stakes chance of ordering the dish, not knowing if it was savory or sweet. It turns out that it was both and much more: there was bitterness from the herbs, and sourness from what I discerned to be a type of lime. It was as if all my tastebuds threw a big reunion and were dancing the night away. I couldn't tell you what the specific flavors were, just the basic tastes.

Let's get the taste/flavor conundrum out of the way. Chocolate mousse is comprised of tastes that I would describe as being bitter, sweet and fat (and maybe even salty because I'm fond of a sprinkle of sea salt), but it has a rich — yet airy — chocolate-y and perhaps milky flavor. Flavor is a holistic experience, and taste is just one factor. Confused? Let’s explore the concept of taste a bit more.

When Jordan Wirfs-Brock and I started exploring multi-sensory data representations  earlier this year (i.e. using the 5 senses to encode data), we revisited Observe, Collect, Draw, a data visualization sketchbook by Giorgia Lupi and Stefanie Posavec, and expanded the exercises to include the other 4 senses. Being gourmand and all, I immediately took to taste. With each prompt, I started by asking myself: how can I translate this data into taste? I’ll present you with my first foray with four tasty portraits from my favorite artist, Nakajima Tsuzen.

Each subsequent exercise led to more questions, though unlike the show, Lost, it led to insights as well.

I chose to keep the visual cues associated with the prompt not just for the purposes of this exercise, but as I believe in multi-sensory data representations, one sense is not enough. In this sketch, I chose sight & taste.