实用基准测试：如何在嘈杂的结果中检测性能变化.pdf

1、Practical BenchmarkingHow to Detect Performance Changes in Noisy ResultsMatt Flemingfleming_mattPareidoliaThis is noiseThis Talk What is noise and how can we measure it?Where does noise come from and why cant you escape it?Detecting performance changes in noisy dataAbout Me Co-founder and CTO at Nyr

2、ki Former Linux Kernel maintainer Author of research papers on performance and testing*Not a mathematician or a statisticianNoise“I use noise to mean random patterns that might easily be mistaken for signals”Nate Silver-The Signal and The NoiseVarianceCoefficient OfVariationCoefficient OfVariationIn

3、terquartileRangeMedian AbsoluteDeviationWhere does noise come from?NondeterminismCPU FrequencyGarbage CollectionResourceBottlenecksProxy Metrics?Intel Cascade LakeIntel SkylakeMedian(nanoseconds)Detecting changesEyeball itThresholdsChange PointDetectionChange pointChange pointChange Point DetectionU

4、sed in systems atMongoDBNetflix,JetBrainsConfluentHazelcastDataStaxMany,many algorithmsE-divisivePELTED-PELTsearch()cmp()filter()search()cmp()filter()Find candidate change pointssearch()cmp()filter()Find candidate change pointsReturn statistically significant onessearch()Window-sliding Top-down(bise

5、ction)Bottom-up(agglomerative)cmp()parametric Maximum likelihood estimation Nonparametric Kolmogorov Smirnov test Euclidean distancesfilter()parametric Student T-test Nonparametric Permutation-based testCPD libraries signal-processing-algorithms Ruptures Perfolizer HunterDrawbacks Multiple data poin

6、ts are needed to detect a change Computational complexityTakeaways Your brain sees false positives Noise is inevitable but measurable!You can mitigate noise using the right statistical methodThank You!Matt Flemingfleming_mattResourcesAndrey