<DrumBeat: June 27, 2007 The Oil Drum This Week in Petroleum +>

139 comments on Finding Needles in a Haystack

Comments can no longer be added to this story.

| Show without comments | PDF version

Prof. Goose on June 27, 2007 - 1:19pm Permalink | Subthread | Parent | Comments top

Bayes rocks.

I know a decent amount of it, but not enough to express competence. I've done a couple of conference papers using Bayesian models, but that's because a buddy is much harder core Bayes than myself.

The frequentist world is a lot easier (central limit theorem, yadda, yadda), but Bayes makes a lot more intuitive sense. The math, however, is a lot harder.

I recommend Jeff Gill's Bayes book, if you're interested:

http://www.amazon.com/Bayesian-Methods-Behavioral-Sciences-Approach/dp/1...

(the estimation of priors, on the other hand, by this crowd, would be pretty good. *laugh*)

Comments can no longer be added to this story.
westexas on June 27, 2007 - 1:28pm Permalink | Subthread | Parent | Comments top

I'll probably get the book.

If you haven't read about the SS Central America, here is the link: http://www.amazon.com/Ship-Gold-Deep-Blue-Sea/dp/0349110999/ref=pd_bbs_s...

It's a remarkable story, from the sinking of the ship, and the rescue of some of the passengers, to the search for the wreck.

Comments can no longer be added to this story.
Don Sailorman on June 27, 2007 - 1:32pm Permalink | Subthread | Parent | Comments top

Prof. Goose,
IMO, the best introduction to Bayesian statistics is still the classic by Leonard J. Savage, "Foundations of Statistics." After sixty years, that text has stood the test of time. I remember struggling with the book back when I was fifteen years old and smarter than I am now, but after a few rereadings and working of problems I finally got it. (I may not be exceptionally bright, but I am exceptionally persistant.)

Comments can no longer be added to this story.
SonicTheHedgeFund on June 27, 2007 - 4:36pm Permalink | Subthread | Parent | Comments top

I find the combination of Bayes, diffuse priors, and the Kalman Filter to be very appealing & a useful linear model.

Neat stuff.

Comments can no longer be added to this story.
WebHubbleTelescope on June 27, 2007 - 7:26pm Permalink | Subthread | Parent | Comments top

The basic idea behind the Kalman filter, IMO at least, is to have a good model for the uncertainty in the data. In other application areas, this uncertainty can be related to noise or other fluctuations which is then used to for example extract a signal from noise. In the model presented here the uncertainty about the mean is really meant to represent fluctuations in the volume sampled, or also in terms of what we think the volume that we sampled. So in this regard we can try to extract the growth in discoveries from the underlying dispersion.

The latter uncertainty is also very critical as input to extraction models, because our estimation of, e.g., how much reserve we have, is crucial input to the amount of effort we expend on getting the stuff out.

Comments can no longer be added to this story.
auntiegrav on June 28, 2007 - 9:09am Permalink | Subthread | Parent | Comments top

Can you use a substitute model for comparison? Such as the number of dry holes per successful hole, with a slope adjusted for increasing data confidence due to better imaging and drill guidance?

"Dig your heels firm unto dirt; and where is the dirt going..?" -Frank Herbert, "The Jesus Incident"

Comments can no longer be added to this story.
SonicTheHedgeFund on June 28, 2007 - 1:03pm Permalink | Subthread | Parent | Comments top

Web -

To add to your KF description, I learned the KF (long ago) via recasting the standard OLS problem in the KF framework. I liked being able to 'see' the impact on the parameter estimates as data points were added to the time series: I found the explicit signal-noise decomposition the KF provides to be 'illuminating'.

Comments can no longer be added to this story.

139 comments on Finding Needles in a Haystack

Comments can no longer be added to this story.

| Show without comments | PDF version