These are some of the difficult questions Eliezer Yudkowsky tackles in a series of posts that I feel compelled to link to. So, here they are. Do read them in the same order.

The articles are long and the arguments presented in the form of dialogues. Thus the reading takes some effort, which, however is amply rewarded. I should add Yudkowsky's disclaimer that his own position on the matter is not represented by any of the parties depicted in these articles but will be elaborated in a follow-up post. 

It is probably fair to say that the linked articles contain no new revelations (at least none that I haven't myself derived). However their greatness lies in the way the central arguments and rebuttals have been crystallised, presented and clarified. Looking forward to the next one!

The difficult jump from classical to quantum is not thinking of an electron as an excitation of a field.  Then you could just think of a universe made up of "Excitation A in electron field over here" + "Excitation B in electron field over there" + etc.  You could factorize the universe into individual excitations of a field.  Your parietal cortex would have no trouble with that one - it doesn't care whether you call the little billiard balls "excitations of an electron field" so long as they still behave like little billiard balls.

The difficult jump is thinking of a configuration space that is the product of many positions in many fields, without individual identities for the positions.  A configuration space whose points are "a position here in this field, a position there in this field, a position here in that field, and a position there in that field".  Not, "A positioned here in this field, B positioned there in this field, C positioned here in that field" etc.

You have to reduce the appearance of individual particles to a regularity in something that is different from the appearance of particles, something that is not itself a little billiard ball.

Oh, sure, thinking of photons as individual objects will seem to work out, as long as the amplitude distribution happens t factorize.  But what happens when you've got your "individual" photon A and your "individual" photon B, and you're in a situation where, a la Feynman paths, it's possible for photon A to end up in position 1 and photon B to end up in position 2, or for A to end up in 2 and B to end up in 1?  Then the illusion of classicality breaks down, because the amplitude flows overlap:

In that triangular region where the distribution overlaps itself, no fact exists as to which particle is which, even in principle - and in the real world, we often get a lot more overlap than that.

I mean, imagine that I take a balloon full of photons, and shake it up.

Amplitude's gonna go all over the place.  If you label all the original apparent-photons, there's gonna be Feynman paths for photons A, B, C ending up at positions 1, 2, 3 via a zillion different paths and permutations.

The amplitude-factor that corresponds to the "balloon full of photons" subspace, which contains bulges of amplitude-subfactor at various different locations in the photon field, will undergo a continuously branching evolution that involves each of the original bulges ending up in many different places by all sorts of paths, and the final configuration will have amplitude contributed from many different permutations.

It's not that you don't know which photon went where.  It's that no fact of the matter exists. The illusion of individuality, the classical hallucination, has simply broken down.

And the same would hold true of a balloon full of quarks or a balloon full of electrons.  Or even a balloon full of helium. Helium atoms can end up in the same places, via different permutations, and have their amplitudes add just like photons.

Don't be tempted to look at the balloon, and think, "Well, helium atom A could have gone to 1, or it could have gone to 2; and helium atom B could have gone to 1 or 2; quantum physics says the atoms both sort of split, and each went both ways; and now the final helium atoms at 1 and 2 are a mixture of the identities of A and B."  Don't torture your poor parietal cortex so.  It wasn't built for such usage.

Just stop thinking in terms of little billiard balls, with or without confused identities.  Start thinking in terms of amplitude flows in configuration space.  That's all there ever is.

And then it will seem completely intuitive that a simple experiment can tell you whether two blobs of amplitude-factor are over the same quantum field.

Just perform any experiment where the two blobs end up in the same positions, via different permutations, and see if the amplitudes add.

(Overcoming Bias)

Yudkowsky is writing about politics and democracy at the moment and I encourage all readers, if you can spare the time, to follow these posts closely. I quite think I'd rather you read his posts than mine these days, and I'm not sure dropping another three or four blogs to follow his posts is at all a bad trade-off.

I am finishing up on Douglas Adams now, and I remembered the bit above very well when I reread it. The quote is from Yudkowsky's post Stop Voting for Nincompoops, and the rest of the post is very much quotable too. I will restrict myself to one more quote, from the conclusion:

If you vote for nincompoops, for whatever clever-sounding reason, don't be surprised that out of 300 million people you get nincompoops in office.

The arguments are long, but the voting strategy they imply is simple: Stop trying to be clever, just don't vote for nincompoops.

[...]

To boil it all down to an emotional argument that isn't necessarily wrong:

Why drive out to your polling place and stand in line for half an hour or more - when your vote isn't very likely to singlehandedly determine the Presidency - and then vote for someone you don't even want?

Enjoy the (un)holi(/y)days!

[...]

Not only did religion used to make claims about factual and scientific matters, religion used to make claims about everything. Religion laid down a code of law - before legislative bodies; religion laid down history - before historians and archaeologists; religion laid down the sexual morals - before Women's Lib; religion described the forms of government - before constitutions; and religion answered scientific questions from biological taxonomy to the formation of stars. The Old Testament doesn't talk about a sense of wonder at the complexity of the universe - it was busy laying down the death penalty for women who wore men's clothing, which was solid and satisfying religious content of that era. The modern concept of religion as purely ethical derives from every other area having been taken over by better institutions. Ethics is what's left.

Intrinsically, there's nothing small about the ethical problem with slaughtering thousands of innocent first-born male children to convince an unelected Pharaoh to release slaves who logically could have been teleported out of the country. It should be more glaring than the comparatively trivial scientific error of saying that grasshoppers have four legs. And yet, if you say the Earth is flat, people will look at you like you're crazy. But if you say the Bible is your source of ethics, women will not slap you.

[...]

The idea that religion is a separate magisterium which cannot be proven or disproven is a Big Lie - a lie which is repeated over and over again, so that people will say it without thinking; yet which is, on critical examination, simply false. It is a wild distortion of how religion happened historically, of how all scriptures present their beliefs, of what children are told to persuade them, and of what the majority of religious people on Earth still believe.

UPDATE: Today Eliezer elaborates:

Absence of proof is not proof of absence. In logic, A->B, "A implies B", is not equivalent to ~A->~B, "not-A implies not-B".

But in probability theory, absence of evidence is always evidence of absence. If E is a binary event and P(H|E) > P(H), "seeing E increases the probability of H"; then P(H|~E) < P(H), "failure to observe E decreases the probability of H". P(H) is a weighted mix of P(H|E) and P(H|~E), and necessarily lies between the two.

UPDATE #2: Even more here:

The rule that "absence of evidence is evidence of absence" is a special case of a more general law, which I would name Conservation of Expected Evidence: The expectation of the posterior probability, after viewing the evidence, must equal the prior probability.

If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction. If you're very confident in your theory, and therefore anticipate seeing an outcome that matches your hypothesis, this can only provide a very small increment to your belief (it is already close to 1); but the unexpected failure of your prediction would (and must) deal your confidence a huge blow. On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence - before you've actually seen it - should not shift your prior beliefs.

E.Y. says that "free will" has no sensible referent in the range between determinism and dice-rolling randomness. I'd add that appealing to "souls" only bumps the question up one recursion. Fair enough. However, in pondering the question, I think I have found a useful referent for "free will".

An example of a tool which achieves solutions in a search space for which all the rules are known is an arithmetic engine. There are relatively few ways of doing 1+1=2. With those few choices hard-coded (to best suit hardware, etc) the solver becomes fully deterministic. To any one search target it will have only one path.

An example of a tool which achieves solutions in a search space with unknown facts and rules is an intelligent mind. There are many ways of looking for solutions to the problems humans run across. Often there are red herrings and traps. The right path is uncertain and must be adaptively sought. We gather evidence and update our beliefs, but in the end we are forced to make stochastic choices. Those choices be predicted from any simpler theory than simulation. They are "incompressible decisions". This is free will.

Free will means the uncertainty in our knowledge of the search space. You could consider it as a sum over the possible paths in search space with non-negligible expected utility. (For future study: are paths in search space discrete from one another or continuous?)

Another way of putting it: free will is ignorance. When we know, we no longer have a choice of the right way to solve it. Extrapolating: total free will is total ignorance and undirected thrashing-around. Total knowledge means zero free will - we are back to our arithmetic solver. (This is not quite true. Total knowledge can still leave decision paths completely equivalent, never reducing down to one deterministic path.)

Corollary: as we gain knowledge (and later, as we gain intellect), our understanding will grow and our choices will narrow. We will have less free will -- unless our targets grow and our search space expands, in which case we will be back to being ignorant about larger and more complex things. But the trend is away from freedom and towards determinism.

A person who understands that mind is an optimization process will recognize that more of this sort of determinism is good. It's called "converging on a solution".

The two largest changes: I have come to understand a bit more of the Bayesian ideal of reasoning from E.Y.'s posts on Overcoming Bias blog, and I have reached a much better idea of artificial and natural intelligence (much credit to this paper), which has allowed me to unify my treatment of the two. Intelligence is that which hits a small goal in a large search space. General intelligence does this in the general case.

This has allowed me to understand that humans are not (as we arrogantly presume) great shakes in the generality department. You can see human generality in the understanding of maths, which is an easy problem from any objective standpoint. It's rare and mostly weak. We are almost specific intelligences, as tailored for being a monkey as a chess program is for its niche.

This kind of thing has led me to four, if not new then at least newly emphasized positions:

1. I am now a transhumanist. I want to do better than this meat mind. It is suited to apes, not to people. As E.Y. puts it: how strange we are, a mind with the form of evolution. The bridge between the era of evolution and the era of mind. "Artificial Intelligence" should be read as meaning "mind born of mind". The artificial is a badge of honor!
2. I am now a singularitarian. I understand the power of self-improving AI. Only two events are comparable: the big bang and the first ancestor of life. If we design an AI that is unsafe, we will be as chaff before the wind. Most kinds of unsafe AI are also boring (a variant of sorcerer's apprentice mode). Making interesting AI will be hard. Making safe AI will be very hard. Technical trends that make it easier to build unsafe AI are a danger, not a blessing.
3. I am now an atheist fully. I understand mind, and I can't harbor any romantic illusions about its nature. Mind is mathematical and computational. These are scientifically tractable concepts. Being info-computation is actually more interesting than being some undefined "soul". Information has properties, and they are interesting ones.
4. Not coincidentally, I really like maths!

Addendum to the below: E.Y. has convinced me that we can't dare to build a neural AI, because it's mathematically intractable and can't be proven safe. The same has to go for an instinct-simulating AI, except to the extent instinct approximates Bayesian reasoning. Although I can't yet follow the maths, I am convinced on good authority that any system of understanding either maps to Bayesian probability, or is inconsistent -- and that it only has value at all to the extent it approximates Bayes' law. Ergo, any good AI will either be Bayesian, or a hack. (Humans are a hack.)

And now for something completely different. Some random thoughts that interested me:

• Does conventional AI go about modeling natural language in the wrong way? Suppose: prose is-a poem. Poetry is the superset!
• E.Y. explains that evolution has a mathematical upper bound on the information it can sustain against mutation. Can we apply this to the quasi-evolution of institutional memory in the economy? (The kind of information I'm talking about here is embodied in structure, not personal knowledge inside people's heads. That's why I think it will behave differently to intelligence.) Is there an upper bound on the information that corporate capitalism can sustain against economic churn? Suppose there is a limit which is predicated on dumb humans. Can we act as smart humans by designing an information pump that re-inserts this lost information? Can we design company structures that would be amenable to this sort of on-the-fly rebuilding?