Why On-Device AI Is the Future for Consumer and Enterprise Apps

If you’re not an accountant, the entire year-end tax return can be a nightmare. While you’re looking forward to returning, you’re probably not too excited about spending money on a tax expert or spending several hours filing your return. What if the entire process could be carried out using a digital assistant on your smartphone? The time and cost savings from using such a productivity app would be significant. This is the potential power of artificial intelligence (AI) installed on your device.

The tax return example is just one of many ways on-device AI can save consumers and businesses time and money. From optimizing smart home devices to automatically drafting customer contracts, on-device generative AI—and the productivity apps it can enable—are key to unlocking an exciting new era in smartphone and PC markets.

AI workloads falling from the cloud

AI for personal and work devices is not a new concept, but the vast majority of applications run in the cloud. While using the cloud is great in terms of resource capacity and storage, the cloud-centric AI model suffers from technical challenges such as high latency and network congestion. As a result, the user experience for many cloud-based AI applications falls short of customer expectations.

To address these technical challenges, smartphone manufacturers are starting to embed AI accelerators into high-end devices to support local AI inference. However, on-device AI applications have been limited primarily to voice control, AI-enhanced imaging, and other “experience-centric” applications. Unlocking the value of AI on devices requires developing a wide range of productivity-enhancing AI applications tailored to specific use cases using compressed generative AI models.

The Value of On-Device AI

ChatGPT started a massive hype cycle about generative AI among consumers and enterprises, leading to testing and implementation in various markets. With most AI models deployed in the public cloud, users experience network congestion, data privacy issues, and rising cloud bills as user bases expand. In turn, local AI workloads – powered by on-device AI – improve user experience by eliminating network latency, reducing various expenses, supporting future AI capabilities, and increasing data security. These benefits are explained in more detail below.

• Improved network latency: AI applications such as digital assistants and Enterprise Extended Reality (XR) require low latency to provide the most natural, personalized, and engaging interactions. Bringing AI inference to the device eliminates the risk of network latency, enabling developers to build a wider range of productivity apps for “mission-critical” applications that would be impossible with a cloud-centric AI architecture.
• Cost savings: As AI deployments continue to grow, the demand for cloud networking and hosting will further increase costs for application developers and enterprises. On-premises AI processing eliminates many of these costs, as well as reducing power consumption in data centers. Optimization tools such as compression and quantization will play a key role in enabling generative AI on-device by developing accurate, low-power AI models with fewer than 15 billion parameters.
• Supporting future AI capabilities: No one wants to invest in a device that will be outdated in a year or two. On-device AI accelerators can be optimized to support generative AI models and applications that have not yet hit the market. In turn, smartphone and PC owners maximize their return on investment (ROI).
• Increased data security: While public cloud service providers implement security measures, they are not bulletproof, as evidenced by cloud breaches at several organizations in recent years. On-device AI stores user and sensor data locally, minimizing the risk of personal data or intellectual property (IP) breaches. It’s also worth noting that low-latency AI models on devices improve threat detection and other cybersecurity features.
• Model Personalization: While AI models can be personalized in the public cloud, this conflicts with end-user demands for greater data privacy and cost optimization. On-device computing enables local tuning of AI models to end-user preferences, behaviors, and applications. This is particularly valuable because it enables efficient personalization of AI models using a variety of incoming sensor/user data sources, including Wi-Fi, GPS, and sensor data. This has significant benefits, including increased AI productivity, improved accessibility, and more intuitive and automated interactions/experiences.

AI on devices makes consumers more productive

Consumers are upgrading their smartphones more slowly than in years past. Perhaps the market has reached a point of diminishing returns. For example, it seems that each new smartphone release offers little or no additional value compared to its predecessor. ABI Research believes that consumer demand for smartphones and tablets can be stimulated by a combination of on-device AI and productivity-focused AI applications.

If device manufacturers can demonstrate a measurable return on investment (cost and time savings) from these AI applications on devices, consumers will be encouraged to upgrade their devices more frequently. Whether it’s saving time by automatically scheduling family gatherings or saving utility costs through optimized energy use, consumers will have a new reason to buy newer smartphone models. Moreover, productivity-enhancing AI applications have the potential to help an artist or producer bring a creative idea to life.

Epitomizing the market trajectory, Qualcomm and Samsung recently partnered to support mobile AI features on the Galaxy S24 series. Not only will AI productivity apps reduce device refresh rates, but the new hardware will justify device makers like Samsung raising the retail prices of their products.

How Enterprises Can Leverage On-Device AI

Similarly, in the enterprise market, a lack of device innovation has caused a stagnation in the growth of desktop and laptop shipments. Implementing AI natively on these devices will attract enterprises due to the value generated from offline productivity, lower latency, increased data privacy, improved user-device communication, and model personalization. On-device AI productivity saves companies time and money by automating administrative tasks (e.g., scheduling, contract drafting, note-taking, etc.) and enabling users to be productive even when their device is offline. Enterprises that leverage these innovative generative AI applications can save thousands of dollars per employee per year and enable employees to use generative AI applications like Microsoft Copilot while on the go (e.g., traveling to a customer site).

ABI Research has observed that the earliest adoption of AI on devices in the enterprise is in back-office operations, offices, and professional services, as early applications (such as Microsoft Copilot) provide a clear return on investment. However, as AI on devices matures with productivity applications and support for different form factors, it is expected to see increased adoption in other industries, such as manufacturing, healthcare, logistics and transportation, and telecommunications.

While smartphones and PCs account for the lion’s share of the AI ​​discussion on enterprise devices, the same benefits can be applied to the automotive, XR, and Internet of Things (IoT)/wearables spaces. Indeed, reduced latency enhances the capabilities of the in-vehicle digital assistant, and data privacy protects sensitive patient or healthcare producer data while eliminating cloud computing costs. Additionally, mining and logistics companies will appreciate the high reliability of on-device AI when using XR and IoT devices in remote areas prone to network outages. Similar to the consumer segment, AI hardware on devices with relevant productivity-enhancing AI apps is expected to reduce device refresh rates among enterprises as they look for the next “killer app.”

The future of artificial intelligence on devices

A wave of recent trends is integral to the on-device AI experience. Heterogeneous chipsets like the Qualcomm Snapdragon X Elite for PCs consolidate the graphics processing unit (GPU), central processing unit (CPU), and neural processing unit (NPU) into a single system-on-chip (SoC). This allows the AI ​​workload to run more efficiently and improves application performance. In addition, there has been a major push to build highly optimized, device-ready small generative AI models capable of matching the accuracy, performance, and insight of much larger models without the high power, memory, and compute requirements. This software innovation has been complemented by increased collaboration among key stakeholders to combine low barriers to entry (via SDKs like the Qualcomm AI Stack and no-code/low-code platforms) and accelerate the development of AI applications that increase productivity.

The fate of the device AI market rests on the shoulders of three key stakeholders:

• Independent software vendors (ISVs) leverage available AI models and tools to create AI applications that are optimized for underlying hardware.
• Chipset vendors ensure that the chipset can support AI on the device and facilitate application development by offering an SDK. It is also important for chipset vendors to provide silicon capabilities to meet the device’s limitations.
• Original equipment manufacturers (OEMs) are consolidating various components into a single device and tailoring applications to consumer/enterprise problems and hardware.

By working closely between these companies, innovation can be further developed to ensure sustainable, long-term revenue streams through AI that enhances productivity on devices. For example, the Ray-Ban Meta collection of smart glasses uses Qualcomm chipsets to provide artificial intelligence in the glasses, reducing network latency and real-time translation capabilities. What were once seen as “entertainment” devices will be recognized as essential “productivity” devices, offering value beyond enhanced photography or simple voice assistants.

Finally, ABI Research predicts that the market will gradually adopt a “hybrid AI” approach. With a hybrid AI architecture, AI workloads reside at the edge, in the cloud, or on-device, depending on commercial and technical priorities. For example, for data-sensitive applications, model training can take place in the cloud, and inference and tuning of these models – which use user data – takes place on the device to ensure maximum privacy. By adopting a hybrid approach to AI, users can distribute power consumption, reduce memory bottlenecks and maximize price-performance

Reece Hayden is a Principal Analyst at ABI Research and leads the firm’s artificial intelligence and machine learning research group.