How Do You Know If Your AI/ML Investment Is Worth It?
Thu Jun 20 2024
Author
Noah Xiao
Setting the right benefit realisation framework is essential to find out.
One of the main barriers to investment in AI is the ability to quantify benefits and find a way to measure them. It is always better to face this barrier early so you can clearly understand what you are in for, ideally even at the ideation phase of a use case for AI.
As a solution proposer, gaining buy-in will require a compelling story that builds momentum and drives better adoption. Simply advocating that “AI/ML is the future” isn’t enough. You need to talk about the actual value, such as ROI and measurement. The best way to start is by developing a value hypothesis that quantifies the improvement your AI use case will have on a specific metric or KPI.
As a decision-maker or investor, you want to hear about the numbers and validate the metrics to ensure that every dollar is spent wisely. We know AI/ML is here to stay and play a significant role in the future, but it’s crucial to have a clear understanding of the financial impact and the projected benefits to make informed decisions.
As a solution provider, having comprehensive measurements and conducting them scientifically helps you understand what’s working and what’s not. This is essential for ongoing improvement and ensuring the solution continues to deliver value over time.
Benefit Realisation Framework
Benefit realisation is almost like a culture for any AI/ML initiative, from initiation to production. Organisations are different, so the actual framework may vary. However, it is important that the benefit realisation framework aligns with the organisation’s processes and ways of working. Below are some common steps for a good framework:
- Define Objectives and Metrics: Establish clear objectives and identify the key metrics that will be used to measure success.
- Develop a Value Hypothesis: Create a hypothesis that quantifies the expected improvements and benefits from the AI/ML initiative.
- Align with Organisational Goals: Ensure the AI/ML project aligns with the broader goals and strategic priorities of the organisation.
- Engage Stakeholders: Involve key stakeholders early in the process to gain their support and ensure their needs are considered.
- Implement Pilot Projects: Start with pilot projects to test the AI/ML solutions on a smaller scale and gather initial results.
- Measure and Validate: Continuously measure the performance against the defined metrics and validate the results to ensure accuracy.
- Iterate and Improve: Use the insights gained from measurement to make iterative improvements to the AI/ML solution.
- Scale and Integrate: Once validated, scale the solution across the organisation and integrate it into existing processes.
- Communicate Results: Regularly communicate the results and benefits to all stakeholders to maintain support and demonstrate value.
- Review and Refine: Periodically review the framework and refine it to adapt to changing organisational needs and new insights.
An Example
For example, take the hypothesis: Implementing a more accurate sales forecast will decrease lost sales by 10%, through better on-shelf availability. An AI/ML driven sales forecast for replenishment is the use case, lost sales is the metric and 10% is the quantified amount.
Initiation Phase
First, define clear objectives and identify key metrics to measure success. In this example, the objective is to reduce lost sales, and the metric is the lost sales percentage. Develop a value hypothesis that quantifies the expected improvements. For instance, the hypothesis is that an accurate sales forecast will decrease lost sales by 10%.
Ensure the AI/ML project aligns with the broader goals and strategic priorities of the organisation, such as improving on-shelf availability. Engage key stakeholders early in the process to gain their support and ensure their needs are considered. In this case, involving sales, inventory, and management teams is crucial to ensure comprehensive support.
Implementation Phase
Create a detailed plan that includes development and rollout considerations. With the clear objectives and metrics defined in the initiation phase, evaluate key design decisions to ensure they align with these goals. Additionally, include change management strategies to ensure smooth adoption across the organisation.
Trial Phase
Start with small-scale pilots to test the AI/ML solution and gather initial results. This may involve implementing the sales forecast model in a few test stores and products initially. Organise these pilots with enough variations of solutions to conduct A/B testing, allowing you to compare different approaches and identify the most effective one.
Continuously measure the performance against the defined metrics and validate the results to ensure accuracy. Measure lost sales in test stores and validate the 10% reduction target.
Use the insights gained from these measurements and A/B testing to make iterative improvements to the AI/ML solution. Refine the forecast model based on initial results and feedback, ensuring it becomes more accurate and effective with each iteration.
Scale Phase
Once validated, scale the solution across the organisation and integrate it into existing processes. Roll out the forecast model to more stores and products, and integrate it with existing systems. Ensure that the solution is adaptable and can handle the increased scale.
Production Phase
Regularly communicate the results and benefits to all stakeholders to maintain support and demonstrate value. Share the success metrics and benefits with all relevant stakeholders. Periodically review the framework and refine it to adapt to changing organisational needs and new insights. Continuously review the forecast model’s performance and make necessary adjustments to ensure it continues to deliver value over time.
By following these steps, organisations can create a robust benefit realisation framework that ensures their AI/ML initiatives deliver measurable and sustainable value.
But reality is harder than the example
In reality, there are lots of challenges, from defining the right metrics, collecting the data without noise, gain trust and confidence about the results.
Typical pain points you may encounter:
- Spending a lot of time measuring and analyzing the results.
- Hard to agree on the metrics due to different focuses across different groups of stakeholders.
- Difficulties in handling noise from other business initiatives running in parallel.
- Uncertainty about how large the sample should be to achieve statistical significance.
- Uncertainty about the minimum period required for accurate measurement.
- Unsure about how to design an unbiased and scientifically valid A/B test.
- Doubts within the organisation about the effectiveness of the AI/ML solution.
- Difficulty in explaining the science of measurement to business stakeholders.
At jahan.ai, we have built our products with benefits realisation at the core of our design, from use case definition to implementation to production. Our Measurement Module standardises and simplifies benefit realisation, providing you with the best practice framework. All our offerings are configurable to your requirements and can be measured and A/B tested to quantify true impact.
In addition, our A/B testing suite offers the following capabilities:
- Automated Control Group Setup: Automate the setup of control groups to benchmark performance effectively.
- Performance Isolation and Tracking: Isolate and track the performance of multiple trials and initiatives running simultaneously.
- Customisable Clusters: Configure or upload your own location or product clusters based on financial, customer, or operational attributes.
- Integrated Automation: Fully automate and integrate trials and their outputs into any existing benefits trackers for explainability, or use our own tracker.
Knowing where your experiment has not hit the mark is just as important as knowing it has achieved success. The former gives you an objective to improve, while the latter provides a compelling story to secure future buy-in for similar solutions.
What has worked for you? How have you overcome this challenge?
At jahan.ai, we build end-to-end AI twins for retail and supply chain businesses. We take the concept of a digital twin, a virtual replica of business processes, further with advanced AI, which not only mirrors these processes but also automatically optimises them. In this way, we support businesses in driving efficiency, going beyond their potential.