There are a number of technological markers that perhaps point to this scenario, some of which I will touch upon. These include the potential impact of AI (Artificial Intelligence), Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR) as well as the blockchain and NFTs (Non-Fungible Tokens). This list is by no means exhaustive, but their applications  demonstrate some of the seismic shifts that are underway.

The inexorable march of technological change continues to speed up. The global pandemic has played a role here (in shifting habits and speeding up the adoption curve in certain instances) but these changes have always been coming. The transformation underway is also being accelerated by changes in audiences. Their expectations have been changed in a digital realm of on-demand products and services from Uber to Netflix that offer instant satisfaction and service in a few clicks. Increasingly realistic and interactive computer games have the potential to transport us to the virtual worlds promised in novels such as Ready Player One. For example, VR offers the possibility where we could visit any period in history without ever needing to set foot in a museum. Digital experiences such as these will become ever more immersive with the continual improvement of graphics and the integration of other elements to the experience through Haptic Technology for example and even Digital Scent technology.

Location-Based Entertainment (LBE) Experiences  have become increasingly popular, providing immersive entertainment in dedicated venues. Even while mainstream adoption still eludes VR, experience economy startups such as Zero Latency VR have helped grow an industry which has seen hundreds of free-roam VR experiences open across the globe over the last few years. How long will it be before these experiences move beyond games related content such as shooting zombies to exploring alternative content? Narrative-driven immersive entertainment experiences such as Jeff Wayne’s The War of The Worlds mentioned earlier already use a combination of VR plus real actors to transport visitors to Victorian England, the setting of the classic H.G. Wells science fiction novel.

• Dig Deeper – REMIX Talk – Tim Ruse, Founder & Managing Director, Zero Latency

The Second Coming of Virtual Reality (VR)

After failed experiments in the 80s, 90s and 2000s such as Google Glasses, both the fully immersive experience promised by Virtual Reality as well as Mixed Reality (overlaid with the real world) have made something of a comeback in recent years. This resurgence has been driven by big tech (which brings its own challenges and opportunities) allied with the promise of the Metaverse. While widespread adoption of VR in the home has not quite panned out as predicted, it may still just be a matter of time as the technology continues to develop and further players throw their hats in the ring (including a rumoured Apple device that is supposedly just around the corner). Headsets such as the Quest 2 and Quest Pro (owned by technology company Meta, formerly Facebook) are now untethered (meaning they do not require a wired connection to a powerful and expensive PC), offering the possibility of a convenient, easy-to-use, high quality VR experience at an affordable price with free roaming. Meta recently revealed they have now sold over 10 million headsets (2021) and generated USD $1.5 billion sales on its Quest App store.

Meta’s purchase of Oculus (the company that developed the tech behind the Quest devices) underpins their belief that the internet is moving into a new phase. This has been called the metaverse, a collection of “virtual environments, experiences and assets” (Herrman and Browning 2021) where we use VR, MR and related technologies to experience, create and collaborate in entirely new ways. Meta recently announced Workrooms which uses the Oculus Quest 2 to allow multiple people to meet and brainstorm in a virtual room as a more immersive and social remote working alternative to Zoom. They believe it could offer a genuine alternative to in-person meetings. To make the interactions as natural and intuitive as possible, participants can also use hand gestures (rather than controllers). Time will tell whether this vision will come to fruition but market research firm IDC predicts that spending in the VR space will grow to USD $72.8 billion by 2024 (up from $12 billion in 2020), driven by potential use cases across multiple industries from construction to medicine.

Companies such as Meta ultimately need to monetise the metaverse to get a return on this investment which poses a serious question mark for many people. The flip side is that the technology can become more accessible if monetisation happens via third-party advertisers. The question is what the trade-offs will be, such as data collection which is the current method of monetisation, or if other models will emerge.

VR still needs the killer apps to truly achieve broader adoption but experimenting with these technologies could be an incredible opportunity for creatives and cultural institutions. The barriers to entry are disappearing and with the tech sector seeking to explore the possibilities of these technologies, they make natural partners. Some players in this space have even set up specific initiatives to collaborate with the cultural sector, such as VIVE Arts (HTC) and Google Arts & Culture which have now been operating for a number of years.

The Holodeck was a popular feature of the fictional Star Trek universe which allowed the cast to visit a convincing recreation of any place or time (real or imagined). The series has famously predicted a number of technological breakthroughs and VR fully evolved could allow us the same experience. Through VR we can even sidestep some of the limitations of the museum experience. For many visitors, the joy of collections is the wonder of seeing the actual object, but with VR, that object can be placed in context by recreating the historical environment around it. We can do things with the object that we could never in an institution, such as getting as close as we want, even picking it up and manipulating it to see it from multiple angles. For many who might never get to visit artefacts in far-flung parts of the world (or who might not want to visit in person for other reasons such as growing environmental concerns over air travel), it is another way (or the only way) of engaging with them. Virtual experiences also offer another solution to the problem of overtourism (which will inevitably return once the pandemic recedes).

Another factor that could drive the growth of virtual worlds is the emergence of low-cost scanning technologies that allow us to capture objects quickly and easily with little or no technical know-how. LiDAR technology (Light Detection and Ranging) can now be found in the latest iPhone and iPad models. This means that millions of people all around the world can now use apps like Polycam to scan objects in 3D, which could include cultural objects, heritage sites, and institutions for example. Think about what the smartphone has done for photography, and the potential of these new scanning technologies could have a similar impact as we march off into the metaverse.

The Tribeca Film Festival has introduced Tribeca Immersive in the last few years to shine a light on creative innovators in this space and VR has proved that it can generate a genuine emotional engagement and response from audiences in the same way as film, television, or real life creative experiences. The VR project, Lovebirds of the Twin Towers, which premiered at the 2021 festival is a powerful example of this.

Mixed Reality (MR) and Augmented Reality (AR)

Unlike VR which seals the user off from the real world, both AR and MR are technologies that blend digital content with the real world. Where they differ in terms of the level of integration and interaction that blur the physical and digital with the latter offering more possibilities. For example, the introduction of wearable MR devices such as the Microsoft HoloLens 2 that overlay digital content onto the real world already has multiple applications in fields like gaming, education, and industrial training. It is also being used to develop immersive experiences such as bringing extinct animals back to life at the National Museum of Natural History in Paris.

Further possibilities exist by bringing all of these technologies together known as Extended Reality (XR) , the collective term for the integration of VR, AR, and mixed reality (MR) to develop multiple realities allowing for seamless transitions between virtual and real environments, enabling even more immersive and interactive experiences.

Screen technology

Rolling Stone described ABBA Voyage as “Ground breaking … [and a] … jaw-dropping spectacle … [saying] … it has to be seen to Be believed”. The Guardian weighed in with “a dazzling retro-futurist extravaganza” ; their review states that it “really does recapture much of the essence of one of the biggest bands in the world in their prime.”

Produced in collaboration with ILM (Industrial Light & Magic) the company that has produced special effects for films ranging from Star Wars to Jurassic Park and countless technology innovations. The new immersive ABBA Voyage production creates Abba’s ‘Abbatars’, using innovative screen technology to provide a seemingly holographic representation of a classic version of the iconic band combined with other 21st century effects and storytelling has received multiple 5* reviews and has been talked about as one possible future for live entertainment in the era of the metaverse. The ABBA Voyage concept took five years and £140 million to develop. It has involved building a dedicated 3,000-capacity arena in the Queen Elizabeth Park, London The popularity of the show has now seen it extended to November 2023 and the concept hopes to tour to other locations around the globe.

We are entering an era where it is now possible to realistically reproduce history which offers huge possibilities for the cultural sector if this technology goes mainstream.

Case Study – Outernet

Outernet is another example of leading edge screen technology allied to other Web3 technologies which in this case provides what could be called a real life manifestation of the metaverse. Outernet claims to be “the world’s most advanced public building” and the largest digital exhibition space in Europe. Located next to Tottenham Court Road station in London it opened in 2022. This enviable location makes it subject to as many as 400,000 eyeballs every day (and over 100 million per year). A £1 billion development that was a decade in gestation it consists of multiple spaces including the Now Building, Now Arcade and the Now Trending space. Outernet claims that the Now Building has the highest resolution screen in the world and could be the ‘world’s largest wraparound video display, with over 2,000 square metres of 16K LED screen’ combined with over 200 speakers.

Outernet has a business model that funds its creative program (Outernet Arts) and experimentation by offering brands a new format to advertise and engage with consumers offering immersion and interaction by utilising metaverse and Web3 technologies such as MR, VR, AI, NFTs and crypto. The screens utilise Unreal Engine’s gaming engine meaning they are able to host “everything from global live streaming to Web3-integrated retail concepts, which basically means putting the metaverse right on Oxford Street” (Forbes). Brand partners include Burberry, Amazon, Netflix, EA Sports, Tommy Hilfiger, NBC Universal, Shiseido and BMW.

Creative partners that tap into the tools offered by Outernet include the British Phonographic Industry (Brit Awards) who recently screened a series of performances at Outernet. They have also signed a multi-year partnership with the Royal College of Arts, the London College of Fashion’s Fashion Innovation Agency (FIA) and the Ridley Scott Creative Group. Outernet is aiming for close to half of its content being purely creative. Outernet Arts is currently supported by BMW.

HERE at Outernet is a 2000 capacity live music venue that has also opened as part of the new creative precinct which is the largest since the 1940’s in the city and also uses immersive technologies to help performers augment the music experience.

Media reaction to this bold new venue has been mixed with some suggesting it is a glimpse of the future while others such as The Guardian saying “Why not just go the full Vegas?” as they referred to “the crass, ad-laden reinvention of central London”. Outernet has ambitions to expand beyond this first outpost with planned future sites announced for New York and Los Angeles.

Projection – Blurring the physical and digital

Immersion projection experiences are proving hugely popular with audiences. Lighthouse Immersive behind the Immersive Van Gogh shows claim 5 million+ tickets sold between February 2021 and May 2022 (one in 90 Americans have bought a ticket) and are the most successful visitor attraction on Ticketmaster. These impressive sales figures have helped them seal a partnership with Disney to deliver their first immersive show Immersive Disney Animation.

They already operate 18 venues across the US that range from USD $4 million to $15 million to set up and have a team of 850 people and are primed for further expansion.

There are many competitors in this space. Perhaps the two most significant are The LUME (by Grande Experiences) that is mentioned elsewhere and CultureSpaces based out of France. CultureSpaces is also expanding rapidly and operates 8 permanent immersive projection experiences including Atelier des Lumières (Paris), Bassins Des Lumières (Bordeaux), Carrières Des Lumières, (Les Baux-De-Provence), Fabrique Des Lumières (Amsterdam), Hall Des Lumières (New York), Bunker Des Lumières (Jeju), Théâtre Des Lumières (Séoul), Phoenix Des Lumières (Dortmund).

Given the relative ease in opening up these experiences the firm has developed and trademarked its own Amiex technology for its digital exhibitions (Art & Music Immersive Experience). They have also specialised in the innovative use of heritage and unusual spaces to provide a point of difference (a Cold War Bunker, Ironworks, former Bank, a Cave complex and even a WW2 Submarine Base). These spaces provide some incredible instagrammable moments and they have just under 1 million social followers as a result (across multiple sites and platforms).

Illumiarium has taken a different approach. While they still provide an immersive projection experience they are focusing on both a broader range of content. They currently offer three experiences that rotate at two venues (Las Vegas and Atlanta) which are Space: A Journey to the Moon & Beyond, Wild: A Safari Experience and Waking Wonderland: An Immersive Adventure inspired by Lewis Carroll’s Wonderland. The company has raised an impressive USD $100 million allowing for significant investment in the immersive content. The Wild experience cost about $10 million for example. They have also added in a range of other technology to amp up the immersion such as scent, interaction with the projected content and haptics.

Perhaps the best evidence of demand is teamLab, a creative collective of interdisciplinary artists and technologists founded in 2001 whose digital artworks have been featured at cultural institutions around the world. In 2018 they decided to cut out the middle man so to speak and opened their own permanent space, teamLab Borderless in Tokyo. In its first full-year over 2.1 million people visited making it the most popular cultural venue in the world by a single artist (or collective in this case) overtaking the Van Gogh Museum in Holland. teamLab has opened a further space in Shanghai with other locations to follow (including in Holland). The level of interaction in their experiences like the Illuminarium is a differentiator as well as the playful sculptural installations that augment the experience. However, what really marks them apart is the incredible original creative content rather than it being derived from the work of artists such as Van Gogh or Monet which dominate (understandably) the output of other players in this space. teamLab’s core creative model means that they are at the leading edge of digital art and may out innovate others in this space.

teamLab have realised that technology alone is not enough however and there is evidence that once the wow moment provided by experiences such as those centred around large scale projection mapping has worn off then they struggle to achieve repeat visitation. The content of course can be changed but there are question marks about the longevity of some of these experiences especially where they are more passive. Industry publication Blooloop sums up the shift that is happening: “In 2023, visitors expect more than an Instagrammable moment. They want to be immersed and active in any experience.”

• Dig Deeper – REMIX Talk – Takashi Kudo, Global Brand Director, teamLab

Holograms

Deceased music stars such as Tupac Shakur and Michael Jackson have famously used the Pepper’s Ghost Effect to appear as holograms at live music events. The Pepper’s Ghost effect was popularised by the English scientist John Henry Pepper in 1862 and is a special effect that creates the illusion of a hologram and has entertained people the world over.

However, developments are afoot in the space that could mean this holographic technology comes more to the forefront of immersive experience design. For example, Universal recently filed a patent for an ‘Interactive Pepper’s Ghost Effect System’ according to Theme Park Insider. This new iteration proffers the tantalising possibility of blurring video game type experiences with the real world through convincing interactive experiences with holograms.

Interaction

The next frontier is true interaction rather than a passive experience. Immersive Gamebox which is a provider of social gaming experiences (a category sometimes called ‘competitive socialising’) using immersive technologies such as projection is one such example. It used technology such as touchscreens and motion tracking within a variety of different games that are a combination of original productions as well as being based on licensed IPs such as Netflix’s Squid Game and the popular children’s TV series Paw Patrol. The first Immersive Gamebox location was in London but it has multiple sites globally across countries (including the US, UK, Canada, UAE and Germany). It has announced USD $90 million of franchise deals and USD $31 million investment raised to date as part of a strategy where it is aiming for 100 venues by the end of 2023. The company was founded by Will Dean, the man behind the well-known Tough Mudder brand. It is also another venture that is trying to capitalise on the struggles racing retail landlords although unlike some of the large scale immersive experiences it is designed to fit in smaller retail units (280-370 sqm).

Audio

Audio is another area where great advances are being made that add to the sense of immersion and these technologies are becoming increasingly affordable and available to consumers and creatives (both for recording and listening) . The first live Dolby Atmos broadcast took place at the Rio Olympic Games in 2016 and in the following 7-years immersive audio has now become mainstream. We can experience it at home through spatial audio devices such as Apple Airpods Pro and numerous VR headsets that use it to increase the level of immersion.

Spatial audio adds realism, and directional audio, motion and tracking technologies allow creators to add interaction and to envelop the audience in a soundscape by providing three-dimensional audio from any angle. Darkfield who are the subject of the next case study deploy this technology to great effect. The premise is that ‘sound engineers record using binaural setups with two microphones capturing audio at the same time, creating a 3D sound (XR Today)’ or the spatial audio effect. For example this allows you to hear the audio but also convincing background and directional noise to make you feel like you are in a real life situation by reproducing the effect of 360 degree sounds you would be exposed to in your everyday life.

Audio Case Study – Darkfield

Some immersive entertainment companies have focused exclusively on using audio to build immersive experiences. Darkfield is a powerful example of this but their story also charts the evolution of London’s immersive sector. I recently interviewed two of the Co-Founders, David Rosenberg and Glen Neath for the REMIX podcast, See Things Differently. They are influential and serial creative entrepreneurs in the UK, and as we find out Darkfield was not their first rodeo but is built on a myriad of experiences that came before.

David was a member of a creative collective that founded Shunt (which Glen was also involved in later on) which was hugely influential in kicking off the immersive movement which is now sweeping the world. One of their projects was the Shunt Vaults, a series of cavernous spaces underneath London Bridge station that were part performance space, part nightclub. They have since been transformed as part of a multi-million dollar redevelopment of the station which forced the closure of Shunt Vaults. Shunt helped pioneer site-specific performance which was what immersive experiences were known as back in those days. The format was extremely entrepreneurial where the team operated one of London’s largest and most edgy hospitality venues that helped underwrite the immersive performances and experiences that were hugely popular with the patrons.

Time Out agreed and described it as follows: “A maze of vaults underneath London Bridge station is the unlikely home of one of the city’s coolest and dampest fringe venues. Officially a members club (non-members have to queue for entry) shunt offers a weekly line-up of Live art, theatre, music and pretty much any avant-garde event they happen to devise. Boundary bending is an understatement here as shunt goes out of its way to make sure every night is unique. Expect themed club nights to morph flamboyantly into performance events and interactive theatre shows (often led by the in-house shunt collective) subjecting you to a sensory overload.” Perhaps the irony now is that given the challenges of the retail and hospitality sectors which have supplanted Shunt Vaults the answer may have been sitting there all along.

Shunt perhaps does not get the credit it deserves as a pioneer in the immersive space, perhaps because they disbanded and others in the scene went onto huge success such as the likes of Punchdrunk, Secret Cinema and more latterly organisations such as Meow Wolf and teamLab who have mainstreamed immersive experiences but as The Guardian put it “Along with Shunt, Punchdrunk led the charge for a wave of immersive, experiential theatre that aims to erase the fourth wall as much as possible.”

David and Glen used these experiences and went on to found Darkfield. This time they have specifically focused on the power of audio to tell incredible immersive stories. Over 250,000 people have bought tickets to step into the strange worlds of Darkfield which unfold behind the doors of shipping containers that they have deposited all around the globe. Once the audience steps inside they might find themselves on a commercial airliner, at a séance or in a bed that could be in a hospital or maybe an asylum. You are asked to put on a set of headphones and from there you enter an immersive audio adventure which utilises binaural 360-degree sound, complete darkness and a variety of sensory effects, where you are situated at the centre of intense, evolving narratives.

During Covid, Darkfield rapidly pivoted, being unable to deliver public experiences. They pivoted to create Darkfield Radio which is an app based immersive audio experiences to audiences at home that provided an innovative new form of entertainment directly responding to the new age we are living in. Darkfield Radio immerses audiences in strange and curious worlds, bringing new meaning to the familiar spaces we inhabit.

The first season of shows launched in July 2020 to critical acclaim, featuring in major film festivals such as Venice International Film Festival, IDFA and Raindance Film Festival, Tribeca Film Festival, and winning Columbia’s Digital Storytelling Lab’s Breakthrough Award in recognition of the year’s most innovative narrative. Season 2 was recently released.

Darkfield Radio was described by The Guardian as “unsettling audio that turns your home into a sci-fi dystopia”

Darkfield experiences are built on an App interface called Wiretapper that David is also a Co-Founder of which delivers immersive audio-based shows directly to you via your Android or iPhone. The events take place in public spaces and at home which fits the requirements of the different Darkfield experiences.

Darkfield’s latest endeavour, Deadhouse is a collaboration with the BBC and is a horror themed trilogy of immersive binaural experiences and they continue to release new standalone immersive experiences with their most recent production being Arcade.

To hear more about the origins and evolution of Darkfield then check out our interview with the founders on REMIX’s See Things Differently Podcast

• Dig Deeper – REMIX Podcast – David Rosenberg, Co-Founder, Darkfield & Shunt Vaults & Glen Neath, Co-Founder, Darkfield