{"id":2097,"date":"2016-05-10T10:27:22","date_gmt":"2016-05-10T09:27:22","guid":{"rendered":"http:\/\/www.lookingatnothing.com\/?p=2097"},"modified":"2016-05-10T10:42:57","modified_gmt":"2016-05-10T09:42:57","slug":"what-does-rebinning-do","status":"publish","type":"post","link":"https:\/\/lookingatnothing.com\/index.php\/archives\/2097","title":{"rendered":"What does rebinning do?"},"content":{"rendered":"

Rebinning is an easy way to reduce the size of your datasets. But does it affect your results? <\/p>\n

The effect of (re-)binning your data<\/h3>\n

(Re-)binning, or averaging, is the practice of reducing the size of a dataset by bunching of datapoints. Of each bunch of datapoints, the mean and standard error on the mean can be determined with higher precision than the original datapoints. The arguments for rebinning are that:<\/p>\n

    \n
  1. It reduces the size of your dataset, allowing it to be fit (much) faster<\/li>\n
  2. It can add another estimator for the uncertainty of your data (through analysis of the datapoint spread in a bin)<\/li>\n
  3. A physical binning process is already done by the detector due to the finite pixel size. In rebinning, we effectively redefine the pixel dimensions.<\/li>\n
  4. It allows you to redefine the datapoint intervals, particularly useful for datasets spanning several decades.<\/li>\n<\/ol>\n

    So why aren’t we all doing this? There are those among us who are of the opinion that (re-)binning should not be done on data, as it might have unforeseen effects. Time to put a gumshoe on the case and figure out if it has an effect.<\/p>\n

    The dataset<\/h3>\n

    One practical way to find out if it has an effect is to take a dataset with many datapoints, and fit it within a particular range. We can then rebin the datapoints into fewer bins, and repeat the fit. Any practical effect will have to show up sooner or later due to the binning.<\/p>\n

    One such dataset is that used in the Round Robin experiment, measured on a synchrotron, and with a suitably large starting number of datapoints (829). The \"q\"-range of the dataset spans \"0.1.
    \nA second dataset a simulation of an extreme case: monodisperse scatterers. While, in practice, monodisperse scatterers are extremely rare, it is the case for which the rebinning should have the most drastic effect. This dataset has been simulated over the same $q$-range as specified above, but for 2000 datapoints in the starting set, with an uncertainty for all datapoints of 1%.<\/p>\n

    The binning<\/h3>\n

    The rebinning procedure sets a series of bin edges, grouping together the pixels that fall in between two edges. Of this group, the mean is calculated and the uncertainties are propagated through the averaging procedure. If the standard error on the mean (derived from the spread of the datapoint values) is larger than the propagated uncertainty, the uncertainty is reset to assume this larger value.<\/p>\n

    The evaluation<\/h3>\n

    The evaluation performed is very similar to that employed in the round robin experiment: we are interested in seeing the deviation of the resultant population parameters. By choosing an ever smaller number of bins, we can determine when it deviates.<\/p>\n

    \"Effect<\/a>
    Figure 1: Effect of binning on the population parameters<\/figcaption><\/figure>\n

    Figure 1 shows the relative deviation of the mean and variance of the distribution, as a function of the number of datapoints. As evident from that graph, meaningful deviations only start appearing when we bin the data to below 40 bins.<\/p>\n

    \"Deviations<\/a>
    Figure 2: Deviations of the population parameters resulting from re-binning of the original data.<\/figcaption><\/figure>\n

    For the monodisperse case, however, the situation is drastically different. As you can imagine (and as Figure 2 shows) the rebinning has drastic effects on the variance (so much so that I could not plot it in the original way). Indeed, the variance changes by several orders of magnitude. So at first glance it would seem that for monodisperse scatterers, we should avoid rebinning and measure without any smearing effects in place.<\/p>\n

    The discussion<\/h3>\n

    Practically, however, this is nonsense. When we translate the variance in the figure above to real units, we have distribution widths orders of magnitude below the size of atoms. Even for the more severe binning variants (i.e. 50 bins), the determined distribution width still remains well shy of an \u00c5ngstr\u00f6m.<\/p>\n

    Furthermore, there is no way to avoid any<\/em> smearing effects in the scattering pattern, which is at the very least limited to the pixel dimensions of your detector, but typically limited to the size and divergence of the beam, and the thickness of the sample. And while you can do desmearing of a scattering pattern in an attempt to retrieve the required features, doing so requires great care and great responsibility (and magnifies your uncertainties).<\/p>\n

    So while it is ostensibly feasible for (poly-)disperse, but bad for monodisperse patterns to (re-)bin, it makes little difference in reality!<\/p>\n","protected":false},"excerpt":{"rendered":"

    Rebinning is an easy way to reduce the size of your datasets. But does it affect your results?<\/p>\n<\/div>","protected":false},"author":2,"featured_media":2098,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":4,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-2097","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/lookingatnothing.com\/wp-content\/uploads\/2016\/05\/reBinDev.png","jetpack_shortlink":"https:\/\/wp.me\/p1gZ2v-xP","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts\/2097","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/comments?post=2097"}],"version-history":[{"count":1,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts\/2097\/revisions"}],"predecessor-version":[{"id":2100,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/posts\/2097\/revisions\/2100"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/media\/2098"}],"wp:attachment":[{"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/media?parent=2097"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/categories?post=2097"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lookingatnothing.com\/index.php\/wp-json\/wp\/v2\/tags?post=2097"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}