AI for art tech in motion, visualization, and experimental media

Project Description

This project builds upon the Theme-based Research Scheme awarded in 2021, which focused on exploring symbiotic creativity between AI and human artists. The project demonstrated the power of AI to collaborate with human creators, blending machine intelligence with human creativity to produce innovative art forms. Building on this foundation, we investigate media capture, reconstruction, and AI-generated content (AIGC) for computational photography, motion, 3D modelling, and experimental media, seeking to revolutionise conventional approaches in performing arts, visualisation, and pushing the boundaries of how art is produced and experienced.

With the establishment of the HKBU Motion Capture and Visualization Laboratory at Hong Kong Baptist University (HKBU). This lab focuses on motion capture and generative AI for the performing arts and choreography, collaborating closely with partners like the Hong Kong Academy for Performing Arts (HKAPA) and the Hong Kong Ballet. Together, we have explored ways to develop motion intelligence and applied these technologies to produce captivating ballet performances, such as the HK Ballet’s show of "Wiz of Oz" show in 2024, where AI-driven motion and expression capture and visualisation enhanced the creative process [11].

In the summer of 2022, we introduced the Turing Orchestra, a symbiotic concert where an AI choir, choreographer, and visual storyteller demonstrated the possibilities of human-AI symbiotic performance for the first time [12][13][14][15][16][17][18]. This marked the debut of the AI Turing Orchestra, showcasing the potential of AI to contribute meaningfully to live artistic productions. Following this, we have created several outstanding concert visualisations, including the Hong Kong Generation Next Arts (HKGNA) 2023 concert visualisation of Arvo Pärt’s famous opus, Wenn Bach Bienen gezüchtet hätte (If Bach Had Been a Beekeeper) [19][20], and the 2024 East Meets West concert, where we demonstrated AI visualisation of ethnic Yao culture display and choreography, attracting wide report and discussion [21][22][23][24].

We have also developed solutions, including MotionGPT and BuVatar, which were showcased at FILMART 2024, gaining attention for their innovative applications in digital media [25] [26][27][28][29]. These tools offer advanced capabilities for generating motion sequences and creating virtual avatars. Our collaborations extend to the Academy of Film and Mei Ah Entertainment, with whom we are investigating how AI can revolutionise movie production, bringing new efficiencies and creative possibilities to the filmmaking process.

This project seeks to push the boundaries of human-AI creative collaboration by advancing media capture technologies, generative AI, and their application across multiple art forms, including performing arts, film, and experimental installations. Its interdisciplinary nature not only fosters innovation within the research community but also provides tools and frameworks that can impact the broader creative industries, enhancing how we create, consume, and experience art.

Professor CHEN Jie (Department of Computer Science)

Funding/Award

Research Grants Council - Theme-based Research Scheme (T45-205/21-N), Building Platform Technologies for Symbiotic Creativity in Hong Kong

Creative and Media Practice

EXPLORE MORE