#| A/B test function in Racket/Gamble A simple function for calculation an A/B test. Example from https://towardsdatascience.com/the-statistical-significance-scam-db904be36714 """ A/B Test Results: Red vs Blue Button CTR: 18% CTR: 21% Test period: 1 week | Sample size: 2000 (1000 per variant) p-value: 0.060 (94% chance results are not random) """ variable : rateA 0.18174463809388475: 0.03126557828776792 0.1835382734811631: 0.030081966807475474 0.17534977954556905: 0.02929369273715369 0.18092493028883322: 0.028835408589135105 0.18412806952207386: 0.028006647121494036 ... 0.4142934665967692: 1.53e-322 0.1987855095842958: 1.53e-322 0.7166766099549209: 1.53e-322 0.7555224923039575: 1.53e-322 0.001201590898947778: 0.0 mean: 0.18088882465313738 HPD interval (0.9): 0.15938443478757253..0.199230320968516 HPD interval (0.95): 0.15914389471779433..0.2078599869649023 HPD interval (0.99): 0.15300818042339143..0.2132136371483797 HPD interval (0.99999): 0.1402579314275527..0.22352766362407364 variable : rateB 0.2117872207677055: 0.03126557828776792 0.2123429852667577: 0.030081966807475474 0.20811709129825867: 0.02929369273715369 0.21573614290454304: 0.028835408589135105 0.21494523548423633: 0.028006647121494036 ... 0.6446586703090207: 1.53e-322 0.7748823817288818: 1.53e-322 0.7273613624523891: 1.53e-322 0.5330358964215784: 1.53e-322 0.26830024641014183: 0.0 mean: 0.2092644418900223 HPD interval (0.9): 0.19100915886045022..0.23109022225729864 HPD interval (0.95): 0.19100915886045022..0.23869909527244085 HPD interval (0.99): 0.17910461255538312..0.24607472454245582 HPD interval (0.99999): 0.16720021946894065..0.25755551578291014 variable : diff 0.03004258267382076: 0.03126557828776792 0.0288047117855946: 0.030081966807475474 0.032767311752689626: 0.02929369273715369 0.034811212615709825: 0.028835408589135105 0.03081716596216247: 0.028006647121494036 ... -0.29552912938964737: 1.53e-322 -0.14058313605947925: 1.53e-322 -0.6378451391364894: 1.53e-322 0.6768385241898401: 1.53e-322 0.26709865551119405: 0.0 mean: 0.028375617236884948 HPD interval (0.9): -0.0016692296992353028..0.058378469741418915 HPD interval (0.95): -0.0016692296992353028..0.0667450072982321 HPD interval (0.99): -0.0156440908924681..0.07338716138161577 HPD interval (0.99999): -0.03266579164800951..0.08921228538930406 variable : diff > 0.0 #t: 0.9597193058082517 #f: 0.0402806941917485 mean: 0.9597193058082517 This program was created by Hakan Kjellerstrand, hakank@gmail.com See also my Racket page: http://www.hakank.org/racket/ |# #lang gamble ; (require gamble/viz) (require racket) (require "gamble_utils.rkt") ; (require "gamble_distributions.rkt") (ab-test 1000 1000 180 210 #:num-samples 30000) #| Here are some other tests from some of my other models. |# #| From gamble_ab_test_simple.rkt A: 20 tests, 1 successes B: 25 tests, 1 successes variable : rateA mean: 0.08872650806316244 HPD interval (0.9): 0.010101855609389096..0.1755239299168499 HPD interval (0.95): 0.0061922598801522144..0.20331380863861861 HPD interval (0.99): 0.0012346681529428016..0.2570634534923353 HPD interval (0.99999): 0.0002889608683644971..0.4006954920245589 variable : rateB mean: 0.07378573873683071 HPD interval (0.9): 0.0039719146138300345..0.14154502829439577 HPD interval (0.95): 0.001808827303138136..0.16467511529888068 HPD interval (0.99): 0.0011803255058658168..0.20785122324987754 HPD interval (0.99999): 0.00048122611985067194..0.33381937986783083 variable : diff mean: -0.014940769326331968 HPD interval (0.9): -0.14707331079725872..0.10552892941281267 HPD interval (0.95): -0.1652229040362851..0.13673354636255203 HPD interval (0.99): -0.23405346224755336..0.16797228382815715 HPD interval (0.99999): -0.3426311310626391..0.23481366287934474 variable : diff > 0.0 mean: 0.43383224664113573 |# ; (ab-test 20 25 1 1 #:num-samples 30000 #:skip-marginals? #t) #| From gamble_ab_test_simple2.rkt A: 25 tests, 16 successes B: 72 tests, 57 successes variable : rateA mean: 0.6286461327317804 HPD interval (0.9): 0.4979024498294399..0.7786982339199797 HPD interval (0.95): 0.4615994196335301..0.7878751556076633 HPD interval (0.99): 0.4063134341866887..0.8169168115820348 HPD interval (0.99999): 0.32046025960727215..0.8658522232308934 variable : rateB mean: 0.7816130989482887 HPD interval (0.9): 0.7171921152423366..0.857943476452716 HPD interval (0.95): 0.6916268123668377..0.8598606058193133 HPD interval (0.99): 0.6735812992906033..0.8838803733150713 HPD interval (0.99999): 0.6098137362252762..0.9124660233685982 variable : diff mean: 0.15296696621650496 HPD interval (0.9): 0.0038804201397388827..0.31718653923774853 HPD interval (0.95): -0.0315768362922183..0.3408714151671375 HPD interval (0.99): -0.07319042826742994..0.39236529162717254 HPD interval (0.99999): -0.19714772691078208..0.4605039177744961 variable : diff > 0.0 mean: 0.9420414018708284 |# ; (ab-test 25 72 16 57 #:skip-marginals? #t) #| From gamble_ab_testing.rkt http://rpubs.com/rasmusab/exercise_2_bayesian_ab_testing """ Exercise 2: Bayesian A/B testing for Swedish Fish Incorporated with Stan Rasmus Bååth """ A: number of sent mail: 16 number of signment 6 B: number of sent mail: 16 number of signment 10 variable : rateA mean: 0.39114279342361047 HPD interval (0.9): 0.22132125049977153..0.5482360972392316 HPD interval (0.95): 0.18743943467357782..0.5666567931773138 HPD interval (0.99): 0.15375867268611915..0.6165499452943881 HPD interval (0.99999): 0.053116229272345356..0.7429921971021347 variable : rateB mean: 0.6136123133096352 HPD interval (0.9): 0.43748180587646446..0.765915206871896 HPD interval (0.95): 0.41806743639442..0.7993647421683667 HPD interval (0.99): 0.37396426082624235..0.8349159380515837 HPD interval (0.99999): 0.29631818647066244..0.8997301103103542 variable : diff mean: 0.22246951988602234 HPD interval (0.9): -0.00013290649672392796..0.4624242117116174 HPD interval (0.95): -0.041723115428054736..0.4958650969324133 HPD interval (0.99): -0.10611910604759223..0.5529743128218958 HPD interval (0.99999): -0.2635740292534371..0.6849314026594231 variable : diff > 0.0 mean: 0.9166675321543789 |# ; (ab-test 16 16 6 10 #:skip-marginals? #t) #| From gamble_ab_testing2.rkt A: 750 tests 41 successes (generated by 750 (binomial 1 0.05)) B: 750 tests 24 successes (generated by 750 (binomial 1 0.04)) variable : rateA mean: 0.05536990820513191 HPD interval (0.9): 0.042230234461545786..0.06624831898513969 HPD interval (0.95): 0.03888261529501643..0.07078431939721452 HPD interval (0.99): 0.0348047525988147..0.078135040137676 HPD interval (0.99999): 0.028621244309666387..0.08991617359822249 variable : rateB mean: 0.03339299977812487 HPD interval (0.9): 0.024685191916936126..0.043145488207597275 HPD interval (0.95): 0.024685191916936126..0.04665668953604025 HPD interval (0.99): 0.020157734895208123..0.05116849103155309 HPD interval (0.99999): 0.015378464236833136..0.06447894362477809 variable : diff mean: -0.021976908427007034 HPD interval (0.9): -0.03389996304812218..-0.003492060482779495 HPD interval (0.95): -0.03975658655643038..-0.00208916455383118 HPD interval (0.99): -0.05211281939374903..0.0016034272513900175 HPD interval (0.99999): -0.06504145188476976..0.015277168642410056 variable : diff > 0.0 mean: 0.02438326717872654 |# ; (ab-test 750 750 41 24 #:num-samples 30000 #:skip-marginals? #t)