In part two of this series I discussed Bayesian inference. Specifically, I discussed how Bayesian inference provided us with a mechanism for deciding in what we should place our confidence given all the information we possess and will yet obtain. This was all framed in the context of confidence. I’d like to discuss an alternative way of looking at Bayesian inference – namely optimization.
Optimization
Optimization largely rules our world. Virtually all of management, engineering, politics, and much of science is about optimization. Optimization, in this sense, is the process of determining the optimal solution given all the objectives and constraints. In management, the process may not be that formal. Perhaps there is a board of directors who gather around a table to discuss the optimal set of policies, the direction to go, etc. In politics it is likely similar. The President of the U.S. surrounds himself with experts on a particular topic, they then engage in discussion, and hope to land on the optimal answer given the objectives and constraints.
For engineers the process is much more formal and precise. Usually optimization takes the form of a cost function – a function incorporating, mathematically, all the objectives and constraints. An algorithm (and there are many) is then employed to “solve” the function resulting in the optimal solution. To demonstrate, here’s an example:
Suppose you are designing an aircraft. There are numerous design possibilities, a canard style, V-tail empennages, aspect ratio of the wings, length of fuselage, coating of the surfaces, wingtips, where to place the turbines, height of vertical stabilizer (if having one at all), etc. etc. We would like to find the optimal answer amongst all these parameters such that we maximize lift, maximize cargo space, maximize safety, minimize energy consumption, etc. Of course we also have constraints. We cannot physically manufacture a flexible fixed wing that is 800 ft long and thin as a toothpick. To solve the problem, we can write down a big, long, nasty equation that would mathematically characterize the physics, constraints, and objectives and then pick our favorite optimization algorithm and wait for it to churn out the answer (which may take a long time).
Bayesian inference is one algorithm that can be applied to such an optimization problem. Typically one would choose this algorithm amidst a cost function that was stochastic in nature, having noise and/or error in the system, that expressed our confidence.
Finding the Truth, Optimization Style
In some sense, the Bayesian inference mechanism I discussed in previous posts could be seen as an optimization method for finding the truth. If we assume that all the new information we regularly encounter has some (even if very little) truth therein, and we apply that information in the regular Bayesian inference sense, we could then reliably conclude that we have found the “truth,” with some probability (level of confidence), given all the information.
This is highly related to a comment FireTag made on my previous post. He asked
So there are routes to evolve our beliefs toward truth no matter where we start or whatever the order of our search algorithm?
In the context of this question, Bayesian inference can easily be seen as a search algorithm. And, in fact, if we used a Sequential Monte Carlo method, it really does feel like a search algorithm.
In expanding this notion, my response, in part, was:
Absolutely (at least in my book)! Though certainly some search algorithms are definitely worse than others and some starting places better than others! Otherwise what prayer in the world do we have (unless you’re absolutely certain that YOU’VE got it right, but I sure don’t)? I view my religion/spirituality as a compass that (I hope) points me in a good direction. My hope is that if/when the absolute truth is made manifest to me I will be humble enough (and my definition of humble is “openness to the truth”) to recognize it because/in spite of my current confidence distribution.
From this perspective, we might view the church (or whatever church you belong to), the Gospel, this life, and all our associated experience as tools to help us optimize for, and draw nearer to the truth given the objectives and constraints of our personal limitations and the limitations of this mortal existence. While I have encountered a very few number of Mormons who claim that we have ALL the truth, this is not the claim of the LDS church. Most of us, I believe, accept there are things we don’t yet know and don’t yet understand. The real challenge is to have an appropriate confidence distribution such that you will accept that truth when it is made known to you.
However, I finished my response to FireTag with the following:
However, this really opens another can of worms – namely, what is truth? My explanation thus far has been about our perception of truth which may or may not correlate with objective or absolute truth. To argue over whether or not our perception of truth is objective truth is to argue over what forms of evidence are acceptable and what weight we should apply to that evidence (which is the conclusion of this post and is an argument with no victor).
