MAAIA：一种面向A/B实验效果精准评估的多维异动归因方法

MAAIA: A Multidimensional Anomaly Attribution Method for Accurate Evaluation of A/B Testing Effects

摘要: 在A/B实验中，有时会出现指标异常波动的问题，这里的异常波动是指无法从报表中直接分析出实验期指标显著上升或下降的原因，或者出现了无法从单一维度定位根原因的波动。这类异常波动会极大增加分析成本，数据分析师们通常需要手动拆解和组合大量维度的数据，过程繁琐且准确度低。本文提出了MAAIA方法(多维异动归因智能方法，Multi-dimensional Anomaly Attribution Intelligent Approach)，并将该方法应用到A/B实验中，该方法能够快速高效得识别出造成指标波动的根原因，并且能更深入地下钻到交叉维度对指标异常波动的影响。目前已通过已有的实验验证了这个方法的有效性以及效率，并成功将其应用到百度某业务的报表分析中，结果表明，MAAIA方法可以快速高效地定位根原因，节省了大量分析成本，增加了分析结果的准确率。

Abstract: In A/B experiments, there are sometimes issues with abnormal fluctuations in indicators. Such abnormal fluctuations refer to situations where the reasons for significant increases or decreases in indicators during the experimental period cannot be directly analyzed from the reports, or where the root cause cannot be identified from a single dimension. Such abnormal fluctuations can greatly increase the cost of analysis, as data analysts often need to manually disassemble and combine data from numerous dimensions, a process that is cumbersome and has low accuracy. This paper proposes the MAAIA method (Multi-dimensional Anomaly Attribution Intelligent Approach), and applies this method to A/B experiments. This method can quickly and efficiently identify the root cause of indicator fluctuations, and can delve deeper into the impact of cross-dimensional factors on abnormal indicator fluctuations. The effectiveness and efficiency of this method have been verified through existing experiments, and it has been successfully applied to report analysis for a certain business at Baidu. The results show that the MAAIA method can quickly and efficiently locate the root cause, save a significant amount of analysis cost, and increase the accuracy of analysis results.