Understanding AI ROI: Measuring Before You Build for Sustained Success [2025]

Last month, at the highly anticipated VB Transform 2026, Zillow's Senior Vice President of Engineering, Toby Roberts, made waves by emphasizing a critical aspect of AI implementation: the importance of measuring return on investment (ROI) before you even begin building your AI systems. This approach challenges the conventional wisdom of diving headfirst into AI projects, highlighting the need for strategic foresight and careful planning.

TL; DR

Plan Before You Build: Measuring AI ROI before implementation ensures alignment with strategic goals and maximizes success.
Context over Raw Data: Understanding context is crucial for effective AI deployment, as demonstrated by Zillow's approach.
Persistent Context Layers: Implementing context layers can improve AI-driven customer experiences and operational efficiency.
Future Trends: AI systems will increasingly focus on personalized experiences and real-time decision-making.
Common Pitfalls: Ignoring the importance of context and pre-implementation measurement can lead to project failure.

AI ROI Outcomes in Retail and Healthcare

The retail case study showed a 15% increase in sales conversion and 10% in retention, while healthcare saw a 20% reduction in readmissions and 10% improvement in satisfaction.

Introduction

At VB Transform 2026, Toby Roberts shared invaluable insights into Zillow's approach to AI development, emphasizing that success hinges not just on the technology itself but on the strategic groundwork laid beforehand. Roberts highlighted the significance of measuring AI ROI prior to embarking on the development journey, a perspective that resonates with many forward-thinking organizations today.

Roberts' perspective is particularly relevant given Zillow's extensive influence in the real estate technology sector, where its products impact about 80% of U.S. real estate transactions annually. The company's commitment to AI-driven solutions has been longstanding, predating the rise of generative AI models like Chat GPT. Yet, as Roberts and Glean co-founder Arvind Jain noted during the event, the challenge of maintaining context throughout customer interactions is more complex than simply leveraging raw data.

Measuring AI ROI involves assessing potential impacts on key metrics like sales conversion and fraud reduction. Estimated data suggests significant ROI potential across these areas.

The Importance of Measuring AI ROI Before Building

When it comes to AI implementation, many organizations rush into development, driven by the allure of cutting-edge technology and the promise of transformative outcomes. However, as Roberts emphasized, this approach can lead to suboptimal results if not carefully aligned with strategic objectives from the outset.

Aligning with Strategic Goals

One of the primary reasons to measure AI ROI before building is to ensure alignment with the broader strategic goals of the organization. By defining clear objectives and key performance indicators (KPIs) upfront, companies can tailor their AI initiatives to support long-term business success.

Example: Consider a retail company aiming to enhance customer satisfaction through personalized recommendations. By measuring potential ROI before implementing AI, the company can identify the specific metrics—such as increased sales conversion rates or improved customer retention—that will indicate success.

Identifying Value Drivers

Before embarking on AI projects, it's crucial to identify the key value drivers that will deliver measurable returns. This involves understanding how AI can enhance operational efficiency, customer experience, or product innovation.

Use Case: A financial institution might explore AI-driven fraud detection systems. By evaluating the expected reduction in fraudulent transactions and associated cost savings, the institution can make informed decisions about resource allocation and technology investments.

Mitigating Risks and Avoiding Pitfalls

Pre-implementation measurement also helps organizations identify potential risks and challenges. By conducting thorough feasibility studies and risk assessments, companies can anticipate obstacles and develop mitigation strategies.

Pitfall: Failing to account for data quality and availability can derail AI projects. By assessing data readiness and integrity upfront, organizations can avoid costly setbacks.

The Importance of Measuring AI ROI Before Building - visual representation

The Role of Context in AI Success

At VB Transform 2026, Roberts and Jain emphasized that context plays a crucial role in successful AI deployment. While raw data is valuable, understanding the context in which it is applied is essential for delivering meaningful results.

Persistent Context Layers

Zillow's approach to AI involves the implementation of persistent context layers, which enable the seamless transfer of information across customer interactions. This ensures that AI systems can provide consistent and personalized experiences, regardless of the touchpoint.

Technical Insight: A persistent context layer acts as a centralized repository that retains historical data, customer preferences, and interaction history. This layer allows AI models to access relevant information, facilitating accurate predictions and recommendations.

Improving Customer Experiences

For companies like Zillow, maintaining context is paramount in providing exceptional customer experiences. By leveraging AI to carry context throughout the customer journey, Zillow can offer tailored services that anticipate and meet individual needs.

Example: Imagine a homebuyer interacting with Zillow's AI-powered platform. By understanding the buyer's preferences, previous interactions, and current market conditions, the platform can suggest properties that align with their criteria, enhancing user satisfaction.

The Role of Context in AI Success - visual representation

Estimated data shows that ignoring data quality issues has the highest impact on AI project success, followed by change management and post-implementation evaluation.

Best Practices for Pre-Implementation Measurement

To maximize AI ROI, organizations should adopt best practices for measuring success before building their systems. These practices enable data-driven decision-making and optimize resource allocation.

Establishing Clear Objectives

Begin by defining specific objectives for the AI project. What problems are you trying to solve? What outcomes do you expect to achieve? Clear objectives guide the development process and provide a benchmark for measuring success.

Conducting Feasibility Studies

Feasibility studies assess the technical, operational, and financial viability of AI initiatives. They help identify potential challenges, required resources, and expected benefits, ensuring informed decision-making.

Developing a Comprehensive Data Strategy

Data is the lifeblood of AI systems. A robust data strategy involves assessing data quality, availability, and integration capabilities. Organizations should prioritize data governance and establish protocols for data collection, storage, and usage.

Quick Tip: Start with a data inventory to understand what data is available and identify gaps that need to be addressed.

Engaging Stakeholders

Successful AI projects require collaboration across departments. Engaging stakeholders from the outset ensures alignment with business goals and facilitates buy-in from key decision-makers.

Quick Tip: Create cross-functional teams that include representatives from IT, marketing, finance, and operations to foster collaboration and shared ownership.

Best Practices for Pre-Implementation Measurement - visual representation

Future Trends in AI ROI Measurement

As AI technology continues to evolve, so too will the methods for measuring ROI. Several trends are emerging that will shape the future of AI evaluation and implementation.

Personalization and Real-Time Decision-Making

AI systems are increasingly focusing on delivering personalized experiences and making real-time decisions. This trend requires advanced data processing capabilities and contextual understanding.

Future Insight: As AI models become more sophisticated, organizations will need to develop metrics that capture the impact of personalization on customer satisfaction and loyalty.

Integration of AI with Io T and Edge Computing

The integration of AI with the Internet of Things (Io T) and edge computing is expanding the scope of AI applications. These technologies enable real-time data processing and decision-making at the edge, reducing latency and improving efficiency.

Example: In the healthcare sector, AI-powered Io T devices can monitor patient vitals and provide real-time alerts to medical professionals, enhancing patient care and outcomes.

Ethical AI and Responsible Deployment

With the increasing adoption of AI, ethical considerations are gaining prominence. Organizations must prioritize ethical AI practices, including transparency, fairness, and accountability.

Recommendation: Establish ethical guidelines and frameworks to ensure responsible AI deployment and address potential biases in AI models.

Future Trends in AI ROI Measurement - visual representation

Key Factors in Pre-Implementation Measurement for AI Projects

Data strategy is rated as the most crucial factor in pre-implementation measurement, highlighting its role in ensuring AI project success. (Estimated data)

Common Pitfalls and Solutions

While AI offers significant potential for innovation and growth, organizations must navigate common pitfalls to achieve success.

Ignoring Data Quality Issues

Data quality is a critical factor in AI success. Poor data quality can lead to inaccurate predictions and unreliable outcomes.

Solution: Implement data validation and cleansing processes to ensure data integrity and consistency.

Overlooking Change Management

AI implementation often requires changes to existing processes and workflows. Failing to address change management can hinder adoption and impact ROI.

Solution: Develop a change management strategy that includes training, communication, and support to facilitate a smooth transition.

Neglecting Post-Implementation Evaluation

Measuring AI ROI should not end with deployment. Continuous evaluation and optimization are essential to maximize long-term value.

Solution: Establish a framework for ongoing monitoring and performance assessment, using metrics that align with strategic goals.

Common Pitfalls and Solutions - visual representation

Case Studies: AI ROI in Action

To illustrate the principles of AI ROI measurement, let's explore two real-world case studies that demonstrate the impact of strategic planning and context-driven AI deployment.

Case Study 1: Retail Personalization

A leading retail company sought to enhance its customer experience through personalized product recommendations. By measuring AI ROI before implementation, the company identified key metrics, including increased sales and customer satisfaction.

Outcome: The AI-driven recommendation engine achieved a 15% increase in sales conversion rates and a 10% improvement in customer retention.
Key Insight: Aligning AI initiatives with strategic goals and focusing on personalization led to measurable business outcomes.

Case Study 2: Healthcare Predictive Analytics

A healthcare provider implemented AI-powered predictive analytics to improve patient outcomes and operational efficiency. Pre-implementation measurement helped identify potential cost savings and quality improvements.

Outcome: The AI system reduced hospital readmission rates by 20% and improved patient satisfaction scores.
Key Insight: By addressing data quality and engaging stakeholders, the organization achieved significant ROI and enhanced patient care.

Case Studies: AI ROI in Action - visual representation

Conclusion

At VB Transform 2026, Toby Roberts' insights into AI ROI measurement underscore the importance of strategic planning and context-driven deployment. By measuring ROI before building AI systems, organizations can align their initiatives with strategic goals, optimize resource allocation, and achieve sustainable success.

As AI technology continues to evolve, organizations must adapt their measurement methods to capture the full impact of AI on business performance. By embracing best practices, addressing common pitfalls, and staying ahead of emerging trends, companies can unlock the true potential of AI and drive meaningful value for their stakeholders.

FAQ

What is AI ROI measurement?

AI ROI measurement involves evaluating the potential return on investment for AI projects before implementation. It ensures alignment with strategic goals and optimizes resource allocation.

How does context influence AI success?

Context plays a crucial role in AI success by enabling personalized experiences and accurate predictions. Persistent context layers retain information across interactions, enhancing customer satisfaction.

What are the benefits of measuring AI ROI before building?

Benefits include alignment with strategic objectives, identification of value drivers, risk mitigation, and informed decision-making. It maximizes success and avoids costly setbacks.

How can organizations address data quality issues in AI projects?

Organizations can implement data validation and cleansing processes to ensure data integrity. A robust data strategy involves assessing data quality, availability, and integration capabilities.

What trends are shaping the future of AI ROI measurement?

Trends include personalization, real-time decision-making, integration with Io T and edge computing, and ethical AI practices. These trends require advanced metrics and responsible deployment.

How can organizations overcome common pitfalls in AI implementation?

Solutions include addressing data quality, developing change management strategies, and establishing frameworks for ongoing monitoring and evaluation. Continuous optimization maximizes long-term value.

Key Takeaways

Plan Before You Build: Measuring AI ROI before implementation ensures alignment with strategic goals and maximizes success.
Context over Raw Data: Understanding context is crucial for effective AI deployment, as demonstrated by Zillow's approach.
Persistent Context Layers: Implementing context layers can improve AI-driven customer experiences and operational efficiency.
Future Trends: AI systems will increasingly focus on personalized experiences and real-time decision-making.
Common Pitfalls: Ignoring the importance of context and pre-implementation measurement can lead to project failure.
Aligning with Strategic Goals: Define clear objectives and KPIs to tailor AI initiatives for long-term business success.
Identifying Value Drivers: Understand how AI can enhance operational efficiency, customer experience, or product innovation.
Mitigating Risks: Conduct feasibility studies and risk assessments to anticipate obstacles and develop mitigation strategies.