{"id":952,"date":"2012-02-07T23:35:37","date_gmt":"2012-02-08T06:35:37","guid":{"rendered":"http:\/\/www.talyarkoni.org\/blog\/?p=952"},"modified":"2012-02-07T23:35:37","modified_gmt":"2012-02-08T06:35:37","slug":"no-free-lunch-in-statistics","status":"publish","type":"post","link":"https:\/\/talyarkoni.org\/blog\/2012\/02\/07\/no-free-lunch-in-statistics\/","title":{"rendered":"no free lunch in statistics"},"content":{"rendered":"<p>Simon and Tibshirani recently posted <a href=\"http:\/\/www-stat.stanford.edu\/~tibs\/reshef\/comment.pdf\">a short comment<\/a> on the <a href=\"http:\/\/www.sciencemag.org\/content\/334\/6062\/1518.abstract\">Reshef et al<\/a> MIC data mining paper <a href=\"http:\/\/www.talyarkoni.org\/blog\/2011\/12\/17\/large-scale-data-exploration-mic-style\/\">I blogged about<\/a> a while back:<\/p>\n<blockquote><p>The proposal of Reshef et. al. (\u201cMIC\u201c\u009d) is an interesting new approach\u00c2\u00a0for discovering non-linear dependencies among pairs of measurements\u00c2\u00a0in exploratory data mining. However, it has a potentially serious drawback. The authors laud the fact that MIC has no preference for some\u00c2\u00a0alternatives over others, but as the authors know, there is no <em>free lunch\u00c2\u00a0in Statistics<\/em>: tests which strive to have high power against all alternatives can have low power in many important situations.<\/p><\/blockquote>\n<p>They then report some simulation results clearly demonstrating that MIC is (very) underpowered relative to Pearson correlation in most situations, and performs even worse relative to\u00c2\u00a0Sz\u00c3\u00a9kely &amp; Rizzo&#8217;s distance correlation (which I hadn&#8217;t heard about, but will have to look into now). I mentioned low power as a potential concern in my own post, but figured it would be an issue under relatively specific circumstances (i.e., only for certain kinds of associations in relatively small samples). Simon &amp; Tibshirani&#8217;s simulations pretty clearly demonstrate that isn&#8217;t so. Which, needless to say, rather dampens the enthusiasm for the MIC statistic.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Simon and Tibshirani recently posted a short comment on the Reshef et al MIC data mining paper I blogged about a while back: The proposal of Reshef et. al. (\u201cMIC\u201c\u009d) is an interesting new approach\u00c2\u00a0for discovering non-linear dependencies among pairs of measurements\u00c2\u00a0in exploratory data mining. However, it has a potentially serious drawback. The authors laud &hellip; <a href=\"https:\/\/talyarkoni.org\/blog\/2012\/02\/07\/no-free-lunch-in-statistics\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">no free lunch in statistics<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"footnotes":"","_jetpack_memberships_contains_paid_content":false,"jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[49],"tags":[327,721,576,575],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pEZxN-fm","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/posts\/952"}],"collection":[{"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/comments?post=952"}],"version-history":[{"count":1,"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/posts\/952\/revisions"}],"predecessor-version":[{"id":953,"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/posts\/952\/revisions\/953"}],"wp:attachment":[{"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/media?parent=952"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/categories?post=952"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/talyarkoni.org\/blog\/wp-json\/wp\/v2\/tags?post=952"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}