Unpredictability of actions

Followup to: The flow of reality.

Effects produced by optimization processes are much less predictable than other events happening in the environment. This happens because optimization processes can produce novel causal patterns, which break rules of environment that worked well in the past. When optimization process is set up with a known goal, some rules of the optimized environment may be known in advance. But specific path towards the goal is usually unknown. Leaving the details of the path unknown in advance may be the whole point of launching an optimization process with known goal: you describe the goal, perhaps only vaguely, and out comes a precise and efficient plan for achieving it. Along this unknown path the optimization process may produce all kinds of unknown causal patterns breaking the old rules.

Ordinary causal patterns appear in ordinary contexts. It is reflected both in their origin, where a pattern is produced through a usual kind of interaction between usual causal patterns, and in the form of rules of thumb that capture its operation, where the conditions of applicability include just a few surface properties. Zooming in on optimization process as an intelligent agent, the choice of actions depends on representation of environment that includes lots of contextual information. Unlike an ordinary causal pattern, a mind isn’t limited to any few of the configurations appearing in the environment; it absorbs as much of them as possible. As a result, any action that doesn’t take an obvious step towards the goal can depend on many contextual features, and so isn’t amenable to being captured by a simple rule.

Unpredictability of actions doesn’t necessarily refer to creation of novel configurations that break the semantics of old events. It can consist merely in the failure to predict or interpret specific actions, where they perform a choice between known causal patterns. You get in the taxi in unknown city, and you don’t know where driver will turn, even if you know the destination.

Forming cached interpretations for the actions of intelligent agent may be unreliable: they don’t work by the rules of natural causal patterns and may defy normal classification. An actual action is chosen by the agent for the specific context, which can place it in any of the context-insensitive bins an observer might have. It may be marked as “stupid”, “brilliant”, “careless” or “disastrous” and not actually be one.

Vague questions and precise answers

Followup to: Levels of structure.

There is a tremendous number of configurations allowed by the laws of physics. A perfect specification of a configuration would require all the tiny details, because it’s physically possible for each of them to be different. Specifying a configuration that actually exists is another matter. You often can just point and say “That thing over there!”, and it would be sufficient. An event doesn’t need to reflect all the facets of causal pattern to capture it precisely, it only needs to delineate the set of configurations that doesn’t contain any other causal pattern. An event is a way to disambiguate the choice between multiple alternatives, thus implicitly specifying the configuration without describing the details.

Even when you can’t precisely define a certain concept, but only have a vague intuition about what it is, there often is the precise and unique referent of that intuition. On the other hand, the change of the boundaries of the event, that may seem insignificant on the scale of the event with all the volume of configurations that it encompasses, may shift it from original causal pattern to a completely different one. This makes the business of defining something a dangerous one, and sometimes counterproductive where a less clear-cut method of identifying the target exists.

A collection of weak indicators that point to the same target works even better. It is possible to describe all the essential details about most of the situations is just a few words, so that the myriad of implied consequences become evident. Likewise, a configuration can be represented by several heterogeneous reusable categories, all of which contribute to the description.

Rules of thumb

People manage to successfully reason using the heuristic rules describing the behavior of macroscale things and events, not needing to derive their behavior from the fundamental laws of nature, and the world seems to behave in a way that allows it. How does this happen? This is not just a consequence of the objects following the simple laws of physics. A door consists of tremendous amount of atoms, yet when I close it, the door just closes. On the high level of description, it is a very simple object. It doesn’t occasionally grow legs and run away, it simply closes.

When trying to extinguish fire, one can try to pour gasoline on it. The fire gets extinguished by water, which is also a liquid. So the gasoline will work as well, won’t it? This is magical thinking, when one acts on surface similarities, and expects the similar consequences. But for objects consisting of a tremendous amount of underlying elements, what is the chance of surface similarities indicating similar behavior in other respects?

The rules of thumb work, because the objects that have the similar causal structure get repeated over and over, resulting from the common cause. The similarity in many properties doesn’t just follow from the presence of few superficial similarities. If I see a flying duck, it might be a robot assassin duck sent by aliens to get me, it just is improbable to the point of being ridiculous. The same cause, the physical process of development from the duck zygote, leads repeatedly to the same system appearing over and over again. By surface similarities I recognize not the other properties that follow from the properties I observed, but a certain causal pattern from the set of causal patterns that exist in my environment and which I learned to recognize in my experience. After the causal pattern is recognized, I can conclude that it has certain other properties.

The examples of repeated patterns in the environment come from many sources. The laws of physics tend to preserve the form of rigid objects, so that the same object will remain in the future, that was there in the past. The stars and other cosmic objects get formed over and over by the same laws of physics during the cosmological development of the universe. On out planet, biological organisms replicate, and so their forms get repeated many times. They mutate, and so slightly different versions of the same system may share many properties. Human culture propagates the ideas and technologies, and so many objects sharing the same properties get constructed for similar purposes. We are surrounded by objects resulting from the common causal processes, and so we are used to seeing the similar consequences from superficially similar events, even if it’s possible to construct systems with the same surface properties that act in an entirely different way.

Applying gasoline to extinguish fire confuses the recognition of the type of causal structure of an artifact with direct recognition of implied properties. Even though both gasoline and water obey the same underlying laws of physics, the causal processes happening during the chemical interactions involved are very different, and so the consequences of applying superficially similar actions to them often won’t result in sufficiently similar consequences. When I don’t expect the object that looks like a duck to be a robot assassin disguised as a duck, I rely on the identification of known causal pattern (a real duck) by few of its observable properties. If I had evidence (indirect experience) of there being in fact robot assassin ducks, I wouldn’t be so sure.

The rules of thumb can be formed about the relationship of many events occurring in sufficiently similar contexts. Groups of events may imply other events, both in the past and in the future, and due to the universality of laws of nature, rules of thumb tend to be reusable, at least where the expected repertoire of causal structures doesn’t change too much.