02 septiembre, 2020

Week Zero: Getting to the Pitch Sometime this summer, the Contentstack marketing team called up Valtech, and asked them to build an Augmented Reality (AR) demo on top of our CMS. Pascal Lagarde (VP Commerce) and Auke van Urk (CTO), were all up for the challenge, in fact, the plan became to do it in four weeks. This is the story of how it was done.

Sometime this summer, the Contentstack marketing team called up Valtech, and asked them to build an Augmented Reality (AR) demo on top of our CMS.

Pascal Lagarde (VP Commerce) and Auke van Urk (CTO), were all up for the challenge, in fact, the plan became to do it in four weeks. This is the story of how it was done.

Week Zero: Getting AR the Brief

Jason Alderman is a Senior Engineer at Valtech who used to work designing interactive exhibits in museums. One of his favorite projects was a donation machine for a museum lobby, which was a giant glass porthole attached to a set of sails. When the machine detected a donation bill, it would suck it up through a snaking tube into the porthole, which would then activate a sensor that would make the sails blow as if in the wind.

He’s excited about the possibilities of Augmented Reality.

I like the connection between the physical and the digital world. Right now, we're holding up these small pieces of metal and glass up to our faces and moving them around like a magic window. The technology is still kind of evolving. I'm really interested to see what the end result will be.

Jason was the first team member to get tasked with responding to “the brief” which was, admittedly, a somewhat rough Google Doc where a few Contentstack people had traded ideas with a few Valtech people along the lines of “could it look like Minority Report?” and “it needs to be interesting for marketers and developers alike.”

Jason is positive about this experience, telling me: “We were given a lot of creative free rein. That's one of the things I love about Valtech—they really invest in the people and let them run with their ideas.” He planned a workshop with a few other developers, UX researchers, and experience designers. “We figured that we probably needed to get as many perspectives inside the company as we could and brainstorm things.”

Identifying the Parameters: Why Contentstack?

Identify How a Headless CMS Will Be Useful in an Augmented Reality Context

Contentstack is a Content Experience Platform (CXP) with a headless CMS at the core. It’s essentially a highly user-friendly database and environment for content creation and storage (text, media, or otherwise) with powerful APIs and integration capabilities that allow that content to easily be delivered to any kind of channel or environment. Traditionally, CMS have been used to power the web, but today the demand for content-rich experiences is significantly more diverse. Beyond web and the mobile web and even app, brands need content to exist in an atomic form, ready to be delivered in an optimized and personalized way to digital billboards, point of sale terminals, social media, marketing automation systems—and yes, Virtual Reality and Augmented Reality experiences.

Valtech is one of the founding partners together with Contentstack of the MACH Alliance, which is a governing and educational body promoting a new standard for enterprise architecture: Microservices, API-first, Cloud-native SaaS, and headless. Says Jason, “It's a way of having an enterprise CMS that can feed all sorts of different front-ends from mobile apps to react apps.”

List Known Strengths of Contenstack CMS

The Valtech team made a list of all the strengths of the Contentstack platform that could be highlighted in an AR demo.

• Detailed content models can be structured easily to feed websites, apps and, of course, AR.
• Internationalization: robust multilingual support, including fallback languages—for instance, if there is no content for a given channel in Mexican Spanish, you can rollback to general Spanish content.
• Robust ability to set up workflows—easily configuring layered steps comprising different actions (approval, commenting, adding elements) that can be set up to automatically push to the next stage.
• Tremendous capability for personalization through powerful integration with tools such as Optimizely or Dynamic Yield.

Isolating the Task: Why AR?

Augmented Reality is hot right now. But the team that took our brief wasn’t a pure AR team. It was a group of people who know how to build experiences and augment them with technology in order to make them either useful or really fun, or both. Given the brief of delivering content-rich experiences pulled from a headless CMS, their first question was "are we sure the best way to accomplish showing off this CMS is through AR?"

What Are the Benefits and Useful Applications of Augmented Reality?

Along with Jason, leading the brainstorm efforts was Danielle Holstine, Delivery Manager—a software engineer turned project manager—who did spend 10 years developing AR and VR technology. She sees potential for AR in everything: “To experience VR, you currently have to put this big thing on your face and it blocks out, you know, it's like blinders; you can't see anything else around you. AR, on the other hand, uses what you're already seeing and just adds information on top of it, so it's additive.”

Especially interesting is the potential of web-based AR and the ability to move away from native apps, which makes these experiences more accessible and easier to engage with. “Phone manufacturers like Apple and Samsung have been investing in the hardware required to do augmented reality functions: improving cameras, sensors, all those kinds of things. And equally on the software side, there's been a lot of development on browser-based AR so it no longer requires a dedicated application to make use of your camera and the sensors on your phone but rather being able to access the information through just a browser.”

But UX researcher and designer Hayley Sikora had questions. “Knowing that we’re working with an amazing CMS and that the brief was to convey information through it, my question was, 'why are we doing it in AR?' Because it’s very difficult to get large amounts of information across in AR.”

Britt Midgette, Sr. Experience Designer, agrees. “We can’t just do AR because it’s cool. It must enhance the experience in a needful way. VR is a different thing—you are creating worlds, there’s "no reason” to do that—but it’s fun, and you can add a lot of stuff in that world. You can still show people a lot of things in an AR world but really, why? Some things should be static. AR can just get in the way of what people are trying to do.”

Frame the Brainstorm To Serve the Opportunity

The resolution came from framing the question in a storytelling narrative:

Since Augmented Reality is layering information on top of the real world, hopefully to make things easier and provide context, there are industries that have complex information, which can be simplified or explained, personalized, and delivered through an AR experience.

The team then broke down this narrative into its component parts and discussed each in turn. The goal was to come up with one-two strong concepts that could be presented to Contentstack in a pitch.

What Are Industries That Have Complex Information?

We started with a brain-dump of all kinds of industries that have complex information that might be difficult to understand, or that people might need some help digging through and figuring out what is relevant.

That was the first part of the narrative: There are industries that have complex information, which can be simplified or explained, personalized and delivered through an AR experience.

The next step was to identify the kinds of information that could be simplified and explained in the two most favoured industries.

Given the Industries “Beauty & Skincare” and “Museums & Education,” What Is Their Complex Information?

The questions that people had around beauty and skincare came naturally to many people in the room, like Hayley, who admits, “I have so many questions about what goes into my own skincare regimen.”

Ideas listed included:

• Ingredients: How can I understand the composition of this product? Are there known allergens in this? How have these ingredients been sourced?
• Benefits: What is actually healthy, versus just a “scam”? What is this product promising to do, and how can I track whether it’s actually working?
• Reviews: Can I see a rating or review? Who recommends this product? Are there influencers that have covered it?

When it came to museums & education, Hayley was inspired by the experience of her aunt, who recently decided to homeschool her children: “I was thinking that it would be a really amazing opportunity to provide kids across the world with some interactive learning tools that could, first of all, give their parents a break from having to be their homeschool teachers 100% of the time — but also give them some fun ways to learn this content."

Ideas for museum & educational complex information included:

• Learning management: Tracking systems for grades, assessments, progress
• Additional context: Who was the creator of an artwork? What are narratives behind certain artifacts which give them context, beyond just the names and dates?
• Details: Virtually dissect a dinosaur skeleton — pull out different bones and see where they were found, what they were for, and how they evolved.
• Media: Sound clips, 3D models, music (instrument types, styles)
  Provenance: How did the artifact get to the museum? Where was it originally created; what hands did it pass through; will it be, or has it been repatriated to the original cultures or people to whom it belongs?

What is the kind of complex information that we could work with?

Here the team had fleshed out the second part of the narrative: There are industries that have complex information, which can be simplified or explained, personalized and delivered through an AR experience.

The final piece of the puzzle was personalization.

How Can We Personalise This Information?

Jason explains that without personalization, any content experience, AR-enhanced or otherwise, is just a bundle of information. The benefit of using technology to represent content in a dynamic format like AR is that it can be personalized, made highly relevant and specific to the person accessing that information.

Adds Hayley, Personalization is only going to continue to get more important. The newest generation is seeking more personalized material than ever because they get instant gratification all day long with personalized content that is sent to them on their social media feeds, so they're expecting that out of other channels as well.

How could personalization be used to de-complexify the types of information that we identified in beauty & skincare and museums & education?

Beauty & Skincare:

• Ingredients: Which of these ingredients will help me achieve my goals?
• Recommendations: Based on your purchase history, preferences; hide products that might cause an allergic reaction or are otherwise incompatible with your personal history. Upload a “shelfie” and get an analysis of how this would fit into your existing routine.
• Face scans: Similar to other Valtech projects showing makeup on someone’s face “live”, can products be recommended based on a scan of your face?
• Phone a friend: Are there reviews I can see from people I know, or from elsewhere online? Can we support or mimic the social buying experience?

Museums & Education:

• Game mechanics: Tour, scavenger hunt, quiz
• Social dynamics: Tether two people virtually to join in a trivia battle, or to share the experience in a personal way
• Responsive content: Dynamically generating a layout of a physical space to match your preferred experience, such as drawing a “map” for you personally to follow through a museum exhibit
• Avatars: To protect kids’ privacy, instead of putting in all of their own personal information into the app, can they create an avatar that represents their preferences and personality traits?
• Text to speech: Keeping in mind that a lot of content stored in Contentstack CMS is text-based, could text-to-speech be implemented to create a personalized audio tour experience using existing written content?

How Can We Personalise This Information?

There are industries that have complex information, which can be simplified or explained, personalized and delivered through an AR experience.

This was the end of the brainstorming session, where two strong concepts had emerged to be taken into the pitch presentation.

From here, the final question was, can this be done in our timeline: 3 weeks from this point on?

Here’s Danielle: “We knew we had three weeks, which is a very short time, to implement something this complex. A traditional two-week sprint process obviously isn't going to cut it for this. This work needs to move so rapidly that we don't have extended periods of time to wait, to have something blocked, those kinds of things.

“So as the brainstorm team was talking, I sketched out a three, one-week-sprint plan with rough goals for each of those weeks.

“The first week is really focused on nailing down the technology we're going to use. So what are the AR libraries that we're going to use? How are we going to track the items? Are we going to do it with fiducial markers, are we going to do it with image-based markers, are we going to do it with object tracking... Each of those has an increasing level of complexity. So we need to make that decision really soon. The next step was nailing down our interaction models and what we want the experience to be.

“Then the second week goal is going to be focused on really hard development: making the application, getting the data into Contentstack, and getting the data back out and visualized the way that we want it in the AR space.

“And then the third week would be really focused on polishing and refining. So, the intention is between the first week and the second week, to actually have our proof of concept — a working thing that we can send around to everybody to test and manipulate, get some feedback on it. And then spend that last week editing, adjusting, and refining. And if we have time, adding in some of the many nice-to-haves that we left on the drawing board."

The Pitch

Lowest Effort, Highest Reward

Based on what they knew they could accomplish in 3 weeks, and that had the highest potential to deliver a “wow”-factor demo, Valtech pitched Contentstack two ideas for an AR proof of concept.

Present in an Easy-To-Imagine Format

Valtech kept the presentation short, and pitched only one slide per concept, complete with hand-drawn illustrations that showed the concept, but made it clear that it was a mere idea, and not a fully living thing.

Knowing that it was possible, and armed with a wealth of ideas, here are the two ideas Valtech presented to us.

Beauty & Skincare

What’s inside the bottle?
Scan a product on the shelf or at home to get personalized recommendations based on the ingredients in the product. See other products that are similar based on some criteria (feel, effect); products that are different (avoiding allergens, discovering other product lines); learn about sustainability and sourcing of the ingredients; or get instructions (see influencer content on tips and tricks, see usage and recommendations from the brand.)

 

Museums & Education

Personal AR Audio Tour.
In a museum gallery or a simulated at-home environment, receive a personalized museum audio tour using text-to-speech technology, including: paths based on how objects in the museum are related to each other; paths that follow a particular preferred narrative thread or subject; synchronize the audio tour with other devices so users can experience the tour together with family or friends.

Offer Recommendations & Guidance

The team also gave some personal guidance on their preference, which was towards the retail app. Says Jason: “I love museums, but we did not think that museum demo would be as effective as one that retailers could translate their business onto more easily.”

Hayley adds: “The opportunities in education are almost endless because there’s so much we could make interactive and gamify. The challenge with education and museums is bureaucracy — who actually takes ownership of it? What school system is going to pay to create an AR learning program for their kids? That's just not feasible. So I think taking this down a route where we could be talking about products that can go to a broader consumer audience makes sense.”

The Decision

On the Contentstack side, me (I’m Varia — Director of Marketing) and my colleague Gal Oppenheimer (Manager, Solutions Architects) immediately gravitated towards the retail and skincare application idea. So that's the application we'll build — and over the next few weeks, we'll share with you exactly what that looks like.

Next week, we will show how Gal and his team helped Valtech to build the content models that will help to power this experience from Contentstack. Plus, Valtech’s software engineers research AR frameworks, interaction design storyboards start to take shape, and we start to wrestle with the surprisingly sticky problem of marker tracking.