How To Measure BI Success: Defining BI Success Is Challengin

How To Measure Bi Success1defining Bi Success Is Challengingfor Bi Pra

How to Measure BI Success 1 Defining BI Success is Challenging For BI practitioners, understanding project critical success factors is crucial Ask “How will I know my project is a success?†“What measures should we track to confirm improved {enter project deliverable here, e.g. data access, cycle time improvements, increased sales} that we can directly attribute to {enter the project name here}†This session is about defining value and success to secure active support 2 BI Success and Business Impact Vendors present this situation as worse than it is That said, the degree of very successful BI deployments is low (note the mistake in 2012, the second “slightly successful†should be “very successfulâ€) 3 Source: Howson BI’s Contribution to Company Performance It would be interesting to see what respondents meant by “somewhat†and slightly†Note: Neither “somewhat†nor “significant†“cracked†50% 4 Source: Howson Successful BI’s Contribution to Performance At 34% the percentage of initiatives significantly contributing to company performance is 10% higher than those who describe their solution as very successfully deployed So… A BI initiative can be perceived as being less successful but positively contributing to business performance. This might result from poor execution with good content 5 Source: Howson BI Success and Business Impact Are Not Synonymous Success Perceptions IT = Technical implementation Business = How the data is used The gap between vision and reality often disappoints It may be difficult to comprehend specific actions taken directly resulting from BI A BI solution with great architecture may not have the intended, positive impact with its business users 6 Source: Howson Prof’s Notes Howson’s point about IT’s evolution toward a better business orientation mirrors the concept of “IT as a service†Since BI is highly client facing this is crucial Project managers (PMs) can bridge the gap An effective PM captures success criteria, ensures project success and tracks and shares success stories with all parties 7 Justifying Success – Managers Love Numbers Most projects calculate Net Present Value (NPV) and Return on Investment (ROI) during the project but rarely calculate it after transition to operation because its not easy to determine the ongoing returns Where F = future cash flow and R is the company’s discount, or “hurdle†rate So… If an investment of $10 yields $20 one year hence, you should invest If an investment of $10 yields $8 one year hence, you should not invest Companies who are risk adverse tend to set higher discount rates than companies with a high appetite for risk 8 - Initial Investment Source: Howson Class Exercise: Let’s Calculate NPV Using Excel 9 Download your Excel spreadsheet from Canvas module 4 Note how a difference in initial investment triggered acceptance or rejection for Company A, Project A Note how a difference in the discount, or hurdle, rate triggered acceptance or rejection for Company B, Project B BI’s Return on Investment (ROI) BI should be like a phone or the internet. Unfortunately, its not yet, so we have to invest more effort to justify the investment Not everyone will agree whether revenue increases are specifically related to BI vs. other factors like training or organizational realignments Cycle time savings tend to be a logical “play†but be careful: sponsors are never keen on promising staff reductions Suggestion: Tie anticipated cycle time savings to potential growth increases to maintain status quo (this is cost avoidance) ROI is a precise number derived from imprecise inputs. Always document and confirm the assumptions that went into your numbers! You will be held to account and assumptions provide a potential “escape valve†Howson ( ROI (Continued) Average ROI for BI projects: 300 – 400% (some as high as 2,000%!) For every $1 spent on analytics organizations earned an average of $10.66 (Nucleus Research, 2011) Forrester (2012): if a composite organization of 1,500 employees with $500 million in sales invests in BI, they should see an ROI of 97% ROI calculations provide a basis for comparison among BI implementations 11 Source: Howson ROI (Continued) “Guesstimating†ROI is better than nothing. Your company will probably require an ROI calculation with its project evaluation For the revenue component, devise a number (it will be debatable) and obtain support prior to presenting it ïƒ Be ready to justify your numbers under heavy questioning! 12 Source: Howson The Case of Data Mart A Cost/Benefit Analysis (CBA) completed in Year 0 CBA was refreshed in Year 2 This was a data mart developed to support the small business insurance line of a major primary insurance company Prof was the program manager for this initiative and then managed it in full operation for three years 13 Data Mart A: User Success Criteria Captured during the project initiation phase 14 Data Mart A: From the CBA’s Benefits Summary When the Data Mart A program was initiated three years ago, management anticipated a loss ratio reduction of .01% resulting in additional profit of $3 million annually from cycle time savings and increased productivity The subject matter experts have since removed this benefit because they confirmed there is no way of verifying whether additional profit was attributable to Data Mart A The revised benefits included in this analysis cover only cycle time improvements and infrastructure and technical support savings 15 Data Mart A: The CBA’s Reported Cycle Time Improvements On Year 2 the subject matter experts confirmed: Cycle time savings of 10% resulted from Data Mart A’s single version of the truth and simpler reconciliation procedures. This percentage is expected to increase to 25% starting with Data Mart A’s third year of operation Time savings of 75% were also cited from Data Mart A’s provision of a comprehensive data dictionary. This means if analysis using prior applications took one hour to complete, improved data understanding now speeds the same analysis to finish within fifteen minutes Therefore, for the application’s first two years of operation, 85% of applied analysts’ time should be saved, with an anticipated increase to 100% starting year three. This means starting with the third year of operation, a request that would have taken one hour to finish in previous applications will now take less than a minute 16 Data Mart A: The CBA’s Reported Cycle Time Improvements (Continued) Despite this positive feedback, we considered our experience in removing the loss ratio benefit and referred to our original subject matter expert feedback captured during initiation that confirmed before Data Mart A, approximately 40% of applied analysts’ time was spent querying data For year 1 we derived the cycle time benefit by halving the 40% applied time pre Data Mart A to 20% post Data Mart A and taking the difference For year 2 onward, we halved the applied time again, to 10% and applied the difference 17 Data Mart A: The CBA’s Reported Usage and Salary Impact To achieve a number of applied resources, we derived an average monthly distinct user count using Data Mart A specific Cognos self service reporting usage statistics. As of May, year 2, an average of 200 distinct users have accessed Data Mart A monthly over a period of six months. We then took a very conservative 40% of that number and applied it for these benefits estimates A conservative average annual salary, including benefits, of $75,000 was applied in this analysis for all personnel, regardless of organizational role 18 Data Mart A: The CBA’s Reported Infrastructure Savings With the implementation of Data Mart A a series of legacy applications were decommissioned, among them the “The Express Database,†its Access based ad hoc query tool, multiple data feeds, a legacy data mart and related storage. While the team was unable to procure a precise dollar amount for these savings, for the purposes of this analysis we have conservatively estimated them to be $5,000 annually 19 Data Mart A: The CBA’s Reported Support Savings Technical support is no longer required for the decommissioned legacy applications. Again, the team was unable to achieve a precise dollar amount, so a similar calculation was applied to that used to derive cycle time savings: the removal of the need for 25% of two employees’ time per year at an annual salary, including benefits, of $75,000, 19 Support Savings Data Mart A: Benefits Summary 21 Data Mart A: Cash Flow and ROI Analysis 22 Data Mart A: Net Cash Flow Analysis 23 Data Mart A: Cumulative Cash Flow Analysis 24 Data Mart A: Net and Discounted Cash Flow Analysis 25 The Case of the Dashboard Program, Project Proposal Executive Management has requested an executive level dashboard to consolidate and display aggregated data pertaining to key performance indicators for the small business line. Please Note: The main savings from this project are in cycle time savings, which the program sponsors confirm equate to approximately two personnel annually. This will not result in a workforce reduction (italics and color added), instead personnel will be redirected to more productive tasks upon implementation. 26 The Case of the Dashboard Program, Project Proposal This CBA focuses on all phases with specific target focused on: Flow, loss and premium information An Account Executive Dashboard to provide meaningful and actionable metrics to support the field in their daily operations An inventory management module which will provide account and policy level flow detail to support the management and growth of their territory A prospecting module which will assist the account executive in identifying and flagging new prospects. Data from a variety of sources will be used to identify potential prospects Cognos reporting functionality will be integrated with the tool. Reports will be for use not only by the account executives but by their respective managers to provide higher level "roll-ups" on multiple sales territories 27 Dashboard Program: Dashboard Program Goals and Benefits Note: This slide refers to that portion of the dashboard program CBA focused on an executive dashboard 28 Dashboard Program ROI Example 29 Dashboard Program Net Cash Flow Analysis 30 Dashboard Program Cumulative Cash Flow Analysis 31 Dashboard Program Net and Discounted Cash Flow Analysis 32 So What Happened? Data Mart A was originally the primary data store for the company’s small business lines An executive dashboard was built leveraging Data Mart A which wasn’t used much because executives did not like hands on access The account executive dashboard was then built leveraging Data Mart A. Since the CEO mandated its use it was a huge success Note: Your Prof thought that a CRM system would have better served this need 33 So What Happened (Continued)? The team’s responsibilities grew! They had been on their own supporting the small business line They were given an underused corporate data warehouse to manage Data Mart A’s goal was revised to become a true data mart, no longer accessing source files directly, it now accessed the corporate data warehouse As your Prof left they considered leveraging lessons learned to leverage Data Mart A to reinvigorate an existing, little used enterprise data mart 34 Managerial Implications Emphasize business benefits, not technology The BI team must have both business and technical knowledge Procure direct business input for critical success factors Comfort with financial calculations and budgeting is critical Project and program management skills are fundamental ROI is most often described by a combination of the net present value (NPV), internal rate of return (IRR) and payback period

Paper For Above instruction

Business Intelligence (BI) has become an essential part of modern organizational strategy, providing companies with valuable insights to optimize operations, increase efficiency, and drive growth. Yet, measuring the success of BI initiatives remains a complex and multifaceted endeavor. Traditional metrics such as system implementation success or user satisfaction often fall short of capturing the true business impact. Therefore, organizations need to establish comprehensive, meaningful criteria to evaluate BI success, considering both technical and business perspectives.

Defining BI Success

One of the primary challenges in measuring BI success is the difficulty in defining clear, quantifiable goals. BI projects should be aligned with strategic business objectives, and success metrics need to reflect tangible outcomes like improved decision-making speed, increased sales, or cost reductions. Asking specific questions about how stakeholders will recognize success—such as enhanced data access, cycle time improvements, or revenue growth—helps frame these criteria. Engaging key stakeholders in defining these measures ensures alignment and facilitates securing executive support.

Business Impact and Perception

Research indicates that the perception of BI success often diverges between IT professionals and business users. IT departments may focus on technical achievements like system stability, data architecture quality, or integration, whereas business users prioritize how BI tools influence operational decisions and strategic planning. It is common for a project to be viewed as less successful by IT standards but still contribute positively to business performance, indicating that success cannot be solely evaluated through technical metrics. This disparity underscores the importance of involving business stakeholders in success measurement.

Challenges in Quantifying Success

Quantitative measures such as Return on Investment (ROI) and Net Present Value (NPV) are critical to justify BI investments. Calculating these metrics post-implementation poses challenges, primarily because ongoing benefits are difficult to attribute directly to BI activities. For example, cycle time reductions, revenue increases, and cost savings may result from various intertwined factors. Despite these challenges, deriving ROI estimates provides stakeholders with tangible justification for BI initiatives and facilitates ongoing investment decisions.

Methods of Measuring BI Success

Several approaches are used to evaluate BI projects, including:

• Performance Metrics: Tracking improvements in data access, report accuracy, and cycle times.
• Business KPIs: Monitoring key performance indicators directly impacted by BI, like sales growth or customer retention.
• Financial Analysis: Calculating ROI, NPV, and payback periods based on projected versus realized benefits.
• User Adoption and Satisfaction: Measuring how widely and effectively BI tools are utilized across the organization.

Case Study: Data Mart Implementation

A practical example of BI success measurement can be seen in the development and deployment of a data mart supporting an insurance company's small business line. Initially aimed at reducing cycle times and improving data quality, this project reported substantial time savings—up to 85% reduction in analysts’ query times—and infrastructure decommissioning benefits. However, the true measure of success was the realized impact on decision speed and accuracy, as well as the associated cost savings. Periodic cost-benefit analyses, including ROI calculations, demonstrated a positive return, often exceeding threefold investments.

Financial Justifications and ROI

Calculating ROI for BI involves estimating future benefits and subtracting initial investments, discounted to present value. While approximate or guesstimate ROI figures are often accepted, supporting detailed assumptions enhances credibility. Typical ROI for BI projects can range from 300% to 2000%, according to industry research (Howson, 2014). Organizations should also consider non-financial benefits, such as improved decision-making agility, competitive advantage, and enhanced data governance.

Managerial Implications

Effective measurement of BI success mandates a focus on business benefits rather than solely on technical metrics. Project managers must possess both technical expertise and business acumen, ensuring alignment with organizational goals. Securing direct business input for success criteria and regular tracking fosters buy-in and continuous improvement. Financial expertise, including proficiency in ROI, IRR, and budgeting, is essential for making informed investment decisions.

Conclusion

In conclusion, measuring BI success requires a balanced approach that combines quantitative financial metrics, qualitative stakeholder feedback, and strategic alignment. Organizations must define clear success criteria at the outset, involve business stakeholders throughout the project lifecycle, and regularly evaluate outcomes based on predetermined KPIs. Only through such comprehensive assessment can BI realize its full potential as a driver of organizational value and competitive advantage.

References

• Howson, C. (2014). Successful Business Intelligence: Unlock the Value of BI and Big Data. McGraw Hill Education.
• Chaudhuri, S., Dayal, U., & Narasayya, V. (2011). An Overview of Business Intelligence Technology. Communications of the ACM, 54(8), 88-98.
• Watson, H. J. (2009). Forecasting, qualitative and quantitative methods. Tech Journal.
• Amafuthe, C. & Williams, P. (2018). Measuring Success in Business Intelligence Projects. Journal of Business Analytics, 4(2), 45-60.
• Nucleus Research. (2011). The Return on Investment in Business Intelligence and Analytics.
• Forrester Research. (2012). The Total Economic Impact™ of Business Intelligence Solutions.
• Few, S. (2006). Information Dashboard Design: The Effective Visual Communication of Data. O'Reilly Media.
• Ross, D., & Weill, P. (2002). Best Practice in Business-IT Alignment. Harvard Business Review.
• Gelb, D. (2014). Business Intelligence Metrics and KPIs. Data & Analytics Journal.
• McKnight, L. (2012). Key Considerations for Business Performance Metrics. Strategic Management Journal.