Editor’s Note: The USC Institute for Creative Technologies, a unit of USC Viterbi, celebrated its 25th anniversary in August 2024. Randall Hill, ICT executive director; vice dean, USC Viterbi School of Engineering; and Omar B. Milligan Professor in Computer Science (Games and Interactive Media); shares ICT’s origin story (and its connection to “Star Trek”) and looks ahead into the research that will power its future.

In 1996, an extraordinary workshop took place about 40 miles south of Los Angeles, in Irvine. It was called “Modeling and Simulation: Linking Entertainment and Defense” and had been set up by the National Research Council. About 60 people had been invited. Half the room was from the entertainment industry. Everyone else was from across academia and the branches of the Department of Defense.

Why had the National Research Council brought everyone together? Well, the Department of Defense (DOD) had been watching the emergence of immersive and interactive entertainment for some time and wondered how they could take advantage of those skills for the military.

Back in Orange County, the presentations sparked something. The conversation grew animated as participants found common ground. Much of that connection was born from science fiction that had become science fact. It certainly didn’t hurt that many of the military participants were die-hard Trekkies, and several of the Hollywood folks in the room were from Paramount Studios and had worked on the famed 1960s TV classic.

After the event, the NRC produced a set of academic papers, later published as a book. It could have ended at that, but it didn’t.

People with influence on both sides wanted to make sure something came out of it. Anita Jones, the U.S. Department of Defense’s director of defense research and engineering from 1993 to 1997, and Mike Andrews, the Army’s then-chief scientist, decided to form a university-affiliated research center, known as a UARC. The proposed UARC would provide unique core competencies in the areas of realism, simulation, the behaviors within simulations, and the application of simulation technology into education, training and operations. USC won the competition for the UARC, which was named the Institute for Creative Technologies, or ICT.

USC decided to locate the fledgling institute off campus in Marina del Rey. Hollywood production designer Herman F. Zimmerman, renowned for his futuristic-looking sets on “Star Trek,” got to work, creating highly imaginative interiors for ICT. It resembled the inside of a starship, complete with bulkheads and an automatic door in the front of the building that opened with a “whoosh.”  The legend goes that Zimmerman had his crew mock up and construct everything on a Paramount soundstage before they eventually installed the ICT interiors over the course of a weekend.

ICT’s first staffers came from Disney, Paramount, across USC, and other academic institutions. It was both a visionary and highly experimental group of talented individuals. Everyone had big ideas, and there were several grand plans. The DOD’s official mandate was to make military training better by creating immersive systems to improve decision-making, build better leaders and support the acquisition of other foundational skills.

 

Building the holodeck

But our Army sponsors knew exactly what they wanted. They said our mission, if we chose to accept it, was: “Go build the holodeck.” A few years later they might have said, “Go create the OASIS [Ontologically Anthropocentric Sensory Immersive Simulation] from ‘Ready Player One.’”

Twenty-five years later, that’s exactly what ICT has done. ICT creates highly realistic computer graphics and virtual environments, delivering synthetic and adaptive experiences, which are so compelling that participants respond as if they were real. Today, ICT has 16 labs, with academics working alongside creatives and makers. We build concepts, demos and prototypes that are cost-effective, agile and adaptive.

Some of ICT’s best-known projects include advanced head-mounted displays (HMDs), digital-based human performance apps, dynamic adaptive synthetic characters, virtual reality-based exposure therapy for veterans, simulations to improve interpersonal skills, counter-terrorism strategy games, virtual human therapeutics, ship simulators and AI-powered digital doppelgangers inside our Light Stage.

Along the way, ICT has received two Academy Awards (Science and Technology) for innovations that have been used in more than 50 Hollywood movies, including “Avatar” and “Blade Runner 2049.” Many ICT projects have become DOD programs of record, part of the formal acquisition program that has been approved and authorized by the appropriate level of authority within the DOD. They include DisasterSim, a game-based training tool focused on international disaster relief; Emergent Leader Immersive Training Environment (ELITE), an instructional platform, delivered via digital devices, that provides opportunities to practice skills in realistic and relevant training scenarios with virtual human role-players and real-time data-tracking tools that allow for structured feedback; and One World Terrain (OWT), an Army program designed to assist the DOD in creating the most realistic, accurate and informative representations of the physical and nonphysical environment.

 

The next 25 years

In 2023, the USC Institute for Creative Technologies (ICT) became part of the USC Viterbi School of Engineering. The institute now sits alongside its sister institute, the USC Information Services Institute (ISI), within the USC School of Advanced Computing, a proud part of USC President Carol Folt’s $1 billion Frontiers of Computing initiative.

Here are three initiatives that will form the core of our work in the next 25 years and beyond.

 

Next-gen education

AI has changed education forever. Knowledge is no longer learned by rote, or even valued in the same way it once was. Everything we need to know is at our fingertips, and its delivery is mediated by AI-powered assistants. Education is now moving toward learning how to verify incoming information, interpreting data, comprehending the value of concepts, upskilling to maintain relevance and developing successful collaboration within human-AI teams.

ICT’s Learning Sciences Lab is a pioneer in this area via its Personal Assistant for Life-Long Learning (PAL3), a friendly, informative and humorous virtual agent that is personalized to students and can be accessed from mobile devices. Simply put, PAL3 is your very own AI, which supports lifelong learning, including on-the-job and just-in-time training, as well as ongoing assessment and tracking. PAL3 tracks where learners are (knowledge, past training and experience) and where they want to go (career and learning goals), and uses that information to give personalized, adaptive coaching and resource recommendations to build a more resilient workforce.

Building on its success with PAL3, ICT is leading AIRCOEE (AI Research Center of Excellence for Education), a two-year, $4.5 million dollar research contract through U.S. Army Combat Capabilities Development Command, in collaboration with the USC Rossier School of Education and ISI. AIRCOEE is addressing two fundamental questions through building a suite of tools: “How do we use AI to improve education?” and “How do we upskill our population in AI and prepare them for the jobs of the 21st century?”

 

Future of filmmaking

As part of its basic and applied research to advance technologies for automated storytelling and narrative understanding, ICT’s Narrative Group Lab is now investigating how AI will change the future of filmmaking.

Can a small group of independent filmmakers on a budget rival the digital sorcery of films like “Avatar: The Way of Water”?

Our project “Acting and Interacting in Generated Worlds” is examining how to integrate AI into production workflows, by capturing the real-world performances of professional actors for playback in fully synthetic (computer-generated) environments.

This research moves us beyond current virtual production setups with actors performing on controlled light stages featuring LED volume walls. Instead, our research will allow actors’ performances to be captured in any lighting environment. Then we can retarget their work onto digital doubles, inside computer-generated digital worlds. This will enable us to achieve Hollywood-quality productions, with small teams of filmmakers able to create movies that rival those with much larger production crews.

 

Saving planet Earth

Finally, ICT’s geospatial terrain research could help save our planet. Earlier this year, I was invited to England to meet with the United Nations Environment Programme World Conservation Monitoring Centre (UNEP-WCMC), located at Cambridge University in England.

The U.N. group is very interested in our data capture (using drones), 3D processing and semantic segmentation (computer vision for accurate labeling) to produce high-resolution terrains. This could assist with monitoring and assessing the biodiversity of the planet. We hope to partner with the United Nations on this worthwhile project, providing our AI tools to improve their Protected Planet database.

We can then take action where it is needed, and protect the world we live in.

Learn more about ICT at: ict.usc.edu