Multimodal AI Explained

Current AI systems typically process inputs with a single data variation at a time to perceive external environments.

While LLMs can process multiple data types, such as text, images, and documents, they cannot infer context from these data formats to enhance their perception of environments beyond their training modules. 

Data modality refers to different data variations and specific data formats that present information. For example, text and video are two data modalities presented in distinct formats. 

Human perception is enabled by our senses, which allow us to take in real-world information to comprehend, perceive, and interact with our environments.

AI aims to more accurately replicate this by processing different data modalities and formats within a single system to construct a perception of the external physical world and how humans function within it. 

What is Multimodal AI? 

Multimodal AI is a form of artificial intelligence that can represent, interpret, generate, and reason across multiple data formats. Multimodal AI ingests multiple data modalities within a single user interaction and combines them within a unified system to enhance perception. 

Multimodal AI won’t just describe the world through text by ingesting multiple data modalities; it will leverage its enhanced perception, enabled by those modalities, to autonomously act within it, moving from explaining to doing. 

Actions by AI are already being taken in digital environments, where models leverage user input data to perceive screens, files, and APIs to automate tasks. Multimodal AI is headed towards taking actions in the physical world as well, perceiving it through camera and audio footage, and real-world sensor data.  

Multimodal AI is quickly becoming the go-to core architecture for mainstream models. A model that can ingest, process, and generate multiple data modalities means a more human-like understanding of its environment.

This enables greater output accuracy and reduced hallucinations. However, multimodal AI also requires models to have a much larger context window, as multimodal inputs are extremely token-expensive to process. 

Increasing Compute

Multimodal AI inference requires much more compute than primarily outputting text, as current LLMs do. If every mainstream model shifts towards multimodality, compute provisioning costs could skyrocket, and e-waste could increase further as supply chains stretch to produce even more powerful GPUs, replacing existing chip hardware before the end of its lifecycle. 

While multimodal AI, alongside agentic AI, represents a big leap forward in model autonomy and towards a breakthrough in AGI. It also highlights a growing gap between sustainable AI development and progress that outpaces our ability to regulate it ethically.

Significant adoption of multimodal AI will increase computational demand to the point of massively straining power grids. 

Energy consumption in data centers will increase as model training intensifies to process and produce outputs across multiple data modalities.

Model parameters and weights (internal numerical values in models learned from training data that allow them to identify patterns in new datasets) will need to be continuously adjusted to account for diverse inputs with varying data formats, which will require more intensive computing. 

Implications of Multimodality

We also have to ask ourselves, do we really want AI to be able to perceive the physical world? The convergence of multimodal AI with spatial awareness technology—combining real-world data formats such as CCTV footage and traffic sensor data to enable AI to perceive height, width, and length and reconstruct three-dimensional environments—will enable automated systems to influence real-world actions. 

Because of technological determinism, artificial intelligence already influences the physical world through digital environments. However, a barrier exists between the physical and the digital, as models rely on human input for perception, keeping AI systems somewhat in check. 

If that barrier is removed through multimodality, which already implies advanced model reasoning, the result will be ambient intelligence that can freely take action in the physical world, with unprecedented implications.

Ambient intelligence is an AI system fully embedded in physical-world machines, capable of autonomously detecting and responding to human presence without command inputs. 

Final Thoughts

Multimodal AI introduces a form of intelligence that can process, reason, and generate outputs across multiple variations of data formats. By integrating diverse data modalities such as text, audio, video, and eventually real-world sensory data, multimodal systems have a more holistic perception of the physical world. 

Multimodal AI will increase output accuracy and reduce model hallucinations while simultaneously expanding context windows, which will demand more intensive compute, driving up resource costs for consumers and straining energy grids. 

If mainstream LLMs adopt multimodal functionality into their architectures, hardware turnover will skyrocket, and e-waste will grow as GPUs are discarded before the end of their lifecycles in favor of newly produced chips to power greater multimodal inference. 

Multimodal AI poses risks but also massive benefits for applications in robotics and spatial computing. Whether humanity is prepared for such innovations remains yet to be seen, as they raise more existential questions than they answer. 

Technological determinism has already blurred the lines between AI’s isolated digital training environments and the context of real-world events. Further removing model perceptual limits by integrating multimodality could lead to a loss of human oversight. 

Overall, multimodal AI reflects a pivotal step on the path towards a breakthrough in AGI. Nonetheless, the trajectory of multimodal AI development puts it on track to outpace humanity’s ability to legislatively guide its ethical adoption and use.

Multimodal AI demands proactive governance and more cemented global standards for AI models that responsibly serve us rather than increase our anxiety and deplete us.