Continuous balancing of changing structure

Followup to: Restoring the structure, Balancing context with conceptual slippages.

Now that the operation of our algorithms is no longer monotonic, so that the scene is not just being extended, but balanced, with possible replacement and deactivation of patterns, it’s time to consider its continuous operation.

In continuous balancing, scene is never being reset, and the process of balancing never stops. Elements of the scene are being updated through external activation and deactivation of certain maps (change in their salience), concurrently with activity of structure waves. This is an equivalent of sensory input. For now, let’s assume that this input describes the scene on high level as well as on low level, activating maps corresponding to arbitrarily abstract properties and relations. To simplify the dynamics, let the scene change slowly relative to propagation of structure waves.

Known maps (long-term memory) are simply maps that were synthesized by structure waves at some point. When certain pattern loses support from sufficient number of other salient patterns, it gradually fades from the scene. Resulting maps with no salience can later be reactivated, if they fit a structure wave better than alternatives. The strength of parameters of a map depends on how it was constructed and changes with each reconstruction.

This setting threatens to leave too much debris, with elements of old scenes remaining active when current scene is updated to something else entirely. However, each active pattern interferes with other patterns, influencing global context. Forgotten elements of an old scene are influenced by the current scene, and vice versa. Old scene can’t change externally updated elements of the current scene, which to some extent gives the direction to dynamic. On the other hand, preserving context left from the old scenes is also a very important feature, allowing to model dynamics of environment and perform deliberative inference.

Representation can now be considered a dynamic inductive-predictive model of environment, responding to sensory input and improving itself by drawing inferences between its elements and learning new rules. This representation is a more technically elaborated substrate for holistic control framework, though far from being specified in enough detail to be implemented.

Interference of patterns

Followup to: Balance of context.

What makes a scene whole, as opposed to collection of unconnected parts? Parts (patterns) influence each other, so that some collections of patterns are balanced, and some are not. Balance is in stability of local reconstruction of structural contexts, and the process of balancing the scene in waves of local structure brings it together. There are many properties present in balanced scene and not in an unconnected collection of parts, but functionally the distinction comes down to local balance.

A balanced scene is consistent: different paths reconstruct the same pattern in approximately the same way. This notion is tricky, since one could say that if different paths create different reconstructions, these are reconstructions of different patterns, not of the same pattern. What is the identity of a pattern, when should we say that two slightly different structural contexts are approximations of the same, and when they should be considered separately?

Identity of a pattern is a functional property, it reflects the fact that all reconstructions of the same pattern are affected by changes in the pattern. Different reconstructions of the same pattern interfere with each other. In unbalanced scene, reconstructions constantly change each other, identity of patterns isn’t localized, they are mixed together. Changes accumulate to affect the global context, interfering with all kinds of patterns throughout the scene.

Pattern interference establishes structure on the implicit pattern graph. Different structure waves propagating through it interact by interference of enumerated patterns, even if they don’t share exact structural contexts along the way. There is a huge number of patterns and paths in the pattern graph, and interference allows them to interact with each other when the match isn’t exact.

As the scene becomes more balanced, each reconstructed pattern interferes with less and less other patterns. Change in a slightly unbalanced scene takes the form of a conceptual slippage, a localized interference that directly influences only reconstruction of the same pattern, with minimum influence on the global context. However, interference doesn’t refer to all changes initiated by variation in a pattern, only to changes in the first affected step of reconstruction in other structure waves, in immediate effect of changes in map salience on pattern matching. A local conceptual slippage could initiate a wave of reconstruction that eventually rewrites the whole scene, through local interference of reconstructions of the slipped pattern and global effects of structure waves that subsequently passed through it, changed by the interference.

Balancing context with conceptual slippages

Followup to: Balance of context.

In his book “Fluid Concepts and Creative Analogies”, Douglas Hofstadter, among other things, explored the notion of conceptual slippage, as a mechanism of fluid cognition. Conceptual slippage is a “context-induced dislodging of one concept by a closely related one, inside a mental representation of some situation”. It can be caught in action in slips of the tongue, when related words or phrases competing for expression of the same context get morphed together, resulting in combinations such as “I’ll chake a look” (“I’ll check is out” vs. “I’ll take a look”) or “I’m going to the begin — uhh, the entrance of the store”. Conceptual slippages perform steps of local search, gradually driving mental representation in a more balanced state.

When global context shifts, it changes the dynamic of structure waves. Some structural contexts get reconstructed in a slightly different way, and since different structural contexts match different patterns, the effects gets magnified along the path of structure waves. As a result, some new patterns get reconstructed, and some old patterns get forgotten. New patterns come with associations of their own, initiating restoration of whole new structures. The moment when a noticeably different pattern gets introduced in the scene corresponds best to the concept of conceptual slippage. It’s a change in representation that lies at the boundary between tiny changes, and a wave of integration of completely new structure.

Conceptual slippages can be used to describe the process of context balancing. Like patterns, they are a tool for describing the dynamic. They are above insignificant changes in structure waves, but still small enough to have structure of their own, to capture replacement of one pattern by another similar pattern. If global context is changed slowly enough, even significant reconfiguration of representation can be regarded as a sequence of conceptual slippages. As I’ll describe later, good representation learns to perform anticipated shifts of balance in as few conceptual slippages as possible, even if the change appears too big and distributed at first.

Balance of context

Followup to: Phantom hierarchies of structure, Structural representation of uncertainty.

Structure waves operate locally, looking at a single structural context and finding ways of improving it. All the knowledge about the world, and knowledge about the structure of a particular scene, gets expressed in this single procedure. The direction of improvement comes as a generalization from previous scenes, and from intermediate results obtained during transformation of these scenes.

Direction of improvement depends on the global context, on what is known and what is salient. Monotonic restoration of details is one modification that can improve structural contexts, filling in absent details. At the same time, global context gradually shifts, changing the preference relation, and some of the details become abandoned, so that they should now be excluded from structural contexts. Structural contexts don’t just grow and become more detailed, they change according to the global context, and in its turn global context is formed by this process, which closes the loop. Choice of specific structural contexts for enumeration is determined by global context and by other structural contexts. Representation is ever in the process of seeking balance between structural contexts and overall global context.

When balance is disturbed, representation starts changing, trying to find a new balance. Changes in a global context can provoke local changes in an area of representation that was previously well-balanced, which in turn create local disbalance that can initiate significant reconfiguration of that area, waves reconstructing whole new structures from previous scenes and integrating them in the context. When the dust settles, balance is restored again, but representation is different. It describes more of less the same state of knowledge about environment, but it now expresses it differently, making some of the inferred details explicit, shifting to different levels of representation, looking at the scene from a different point of view. It is the same change of perspective that happens on a smaller scale during enumeration of structural contexts, when hierarchies of structure turn upside down.

Structural representation of uncertainty

Followup to: Reduced representation of alternatives.

Scene graph represents information about environment, focusing on certain objects and relations, at a certain level of description. Each element of representation leaves fair amount of uncertainty, specifying only a coarse picture. At the lowest level, labels encode classes of properties without discriminating among the properties within represented class. Finer distinctions are expressed through the graph structure in which labeled nodes are organized. Common arrangements of local structure can later be summarized in maps.

Similar states of information about environment are represented by similar scene graphs, which allows to infer ways of transforming representation without significantly distorting represented information about environment. Many details and high-level properties of environment implicitly described by the scene are absent in the representation but known based on scenes encountered in the past. Scene can be transformed to replace a given description of environment with one that is expressed in terms of more familiar structural contexts, and that doesn’t leave out properties that are usually represented explicitly.

Details included in representation, especially restored ones, are of different levels of uncertainty. Furthermore, since patterns are represented implicitly, uncertainty of the same detail can vary between its different local reconstructions by structure waves. Essentially, stochastic process of modification of map salience through propagation of structure waves balances uncertainty of different elements of representation, by enumerating sufficiently certain structural contexts. Salience of a map is influenced through different routes, since the same map can be a part of many different patterns, and the same pattern can be suggested as sufficiently likely by many different structure waves.

A pattern that is used in many contexts in the scene, is of high certainty by itself, while each of its instantiation points can have different levels of uncertainty, and all of the details in the scene are very unlikely to be correct simultaneously. Similarly, contradicting properties generate alternative scenes of higher uncertainty than the scene without contradicting properties, but common areas of these alternative scenes are still pretty certain.

Reduced representation can be thought of as a state of knowledge that allows to answer questions. Structural contexts or patterns are specific answers, and structure waves are questions. Answers to different questions can contradict each other when considered by themselves. Different parts of the scene can have contradictory properties, just as alternative properties of the same part of the scene contradict each other. Which color is a zebra? Different colors at different points. What color is an animal? Depends on which animal it is. Encoding different structures of alternative states of the scene allows to represent the structure of uncertainty about the scene with the same expressive power as used for describing the fixed structure of a single scene.

Contradictions can be resolved either in the question, by specifying additional details that allow to give a simple answer, or in the answer, by making it conditional, by enumerating alternatives. Different alternatives in the answer can be recast as elaborations to the previous question, and this is one way of exploring an overly complex answer to an overly general question, such as the structure of the whole scene. Structure waves are iteratively refined questions shaped by state of knowledge about the scene, and structural contexts they restore are both local answers and iterations of further refinement. Scope of the question can include elements of alternative structures of the scene, or elements of different parts of the same structure of the scene expressed in common pattern, erasing the difference between these cases.

Reduced representation of alternatives

Followup to: Representation of structure, Phantom hierarchies of structure.

Structure waves only enter a property from instantiation point if it’s an unique match. As a result, only unambiguous properties get instantiated, and this leads to hierarchies of patterns. One property gets instantiated at multiple instantiation points, and property is usually considered to be a part of the scene, while instantiation points are embedded in the bulk of the scene. Even though such properties are represented in reduced form, the difference between explicit form of the scene and a form where all common properties get collapsed together is minor.

Structure waves don’t need to always start from the bulk of the scene, they can also start from a small property. In this case, they can only enter the scene if structure wave can unambiguously match a structural context of the scene in a context of the property. This works for rare or local properties that only apply at one point (with locality determined by the scene graph rather than spatially), or up to a certain depth for properties of big patterns in the scene (in which case a wave can exit the property and visit the whole pattern, even if it stops short of the whole scene). Also, given enough steps, structure wave could escape even patterns instantiated in multiple places.

Surprisingly, it also applies to small properties that can’t be visited from within the scene. These properties can’t be instantiated at any place in the scene, but structure waves starting from them can enter the scene. Considering such properties a part of the scene is equivalent to allowing multiple conflicting properties to be considered at one: a given interface in a scene will have multiple matching properties, so that choice between them is ambiguous, but each of the properties only matches this interface.

Following the terminology introduced for describing patterns, these ambiguous properties include instantiation points of their own that allow to instantiate the scene. Thus, implicitly, the whole scene gets instantiated multiple times, once for every alternative property.

This interpretation changes the perspective on what is represented. Where before the set of maps was supposed to only represent a single scene, restoring it with the knowledge of other scenes, now it can represent multiple alternative hypotheses about the structure of the scene. Reduced representation allows to represent not only a huge number of implicit details for a single scene, but also a huge number of implicit alternative scenes, by collapsing the repeated structures shared among alternative scenes as well as different areas of the scenes.

If alternative properties can only reach an intermediate pattern smaller than the whole scene, they specify alternative instances of that intermediate pattern. The intermediate pattern gets independently instantiated both by the scene and by the alternative properties. Context carried by structure waves can create biased versions of the intermediate pattern, specific to a certain property or instantiation point in the scene, and through these biased patterns some of the properties can find matching instantiation points in the scene, resolving ambiguity. For structure waves, there is no difference between alternative patterns that instantiate alternative structures of the same part of the scene, and different instantiation points in the scene itself.

Phantom hierarchies of structure

Followup to: Patterns of structure, Enumerating patterns.

As structure waves propagate through the scene, they bring up more and more details. Their progress is shaped by structural context in the wave-front, overall content of the current scene (salience of maps) and knowledge of maps from the past scenes. In every scene, it’s possible to infer huge number of specific facts, on different levels of representation, and that’s what structure waves do.

Representation keeps knowledge about the scene in a fluid format, converting elements of representation of different types into each other as the context demands. It can’t and doesn’t need to explicitly enumerate all the facts that could be inferred. Some irrelevant details uncovered by structure waves need to fade out of scene, if they can’t find enough support.

A pattern, as it exists in the scene, is embedded in its context. It’s connected by relations to other elements of the scene, and assigned properties specific to its place in the scene rather than to the pattern itself. On one hand, a pattern can be seen as a class, substructure containing elements repeating in most of its instances, and on the other, as an instance, form assumed in the scene, with all the context-specific details.

When structure wave starts reconstructing a pattern, it’s done with a certain level of detail, ranging from a closed atomic property like color from which there is no inference, to a scene root that leads to every element of the scene around the pattern. Depending on how specific origin of the wave is, it can lead to instantiation of different other patterns in the scene, with different levels of detail.

As wave propagates, there are two pressures on the level of detail present in reconstructed patterns. First, wave avoids ambiguity, which creates a hierarchy of patterns, running from instances to classes of their properties. But second, wave acquires details as it proceeds, being guided by salience of maps and knowledge of other maps. As a result, the hierarchy turns upside down, if structure wave is allowed to run long enough and doesn’t start from a complete dead end. Starting from a class of pattern, structure wave exits to the scene from the other side, where it can pick up a root of any other pattern.

This makes pattern hierarchy a fleeting description that applies only for the few steps, around a specific pattern instance in the scene. Keeping the hierarchy an adequate description of restoration process requires considering only structure waves that originate in the patterns of the hierarchy and that don’t run for too long. Scene graph is not hierarchical, and neither is its map representation. Hierarchy appears locally, and describes the way structure waves enumerate structural contexts of the scene. It also describes most of the salient maps in patterns instantiated by the waves, giving a bound on inferred details to be included in the scene. A significant reorganization of existing hierarchies can occur as a result of changes in map salience created by deviations of waves from exact patterns of a hierarchy.

Multiple hierarchies can coexist describing the reconstruction of the same scene, because they describe the same areas of scene graph with different levels of detail. Each hierarchy runs from instances to classes, abstracting out relations between the core of a pattern and its context in patterns further from the root. But the same patterns with context included in it can be at the root in another hierarchy. As this description was constructed, joining different hierarchies together complicates the picture, as patterns are implicit and different hierarchies will have different patterns included in them.

From this perspective, the scene can be seen as casting multiple hierarchical shadows wrapped around each other, each driving elaboration of detail in its specific direction, but stopping at some point and fading into other hierarchies. This is a way to stop combinatorial explosion by shifting to different aspects all the time, where search for a solution is simply conversion of the scene representation from one aspect to another, from a question to an answer, through many intermediate steps guided by unified scene representation.

Enumerating patterns

Followup to: Restoring the structure, Patterns of structure.

Patterns offer a limited perspective on structure of the scene, as supported on structure waves. From this perspective, scene can be considered a huge hierarchy of patterns, with the most rare patterns at the root, identifying the scene as a whole, and more and more frequent patterns placed towards the leaves of that tree, identifying objects, then their properties and relations, and finally the most basic features like color.

Let the root of a pattern be a minimal subgraph whose closure is equal to closure of that pattern. Hierarchy of patterns is stored in a reduced form, with groups of isomorphic subtrees collapsed, to be instantiated as needed. In this reduced form, scene can be seen as represented by a directed acyclic graph, with edges identifying which patterns instantiate which other patterns through interfaces, and nodes corresponding to roots of individual patterns (small groups of maps). Like a reduced binary decision diagram, this representation offers the potential to represent huge trees by relatively small graphs, and in the same way representation is never explicitly unpacked to perform operations on it. Every element of the scene can act as a window in a potentially bottomless well of contextually evoked detail, without proportionally cluttering the representation, especially considering that reachability is not limited to patterns included in the scene, but extends to all known patterns. Patterns show this side of wave-based representation, but the resulting picture is also significantly distorted for the reasons to be explained in this and the following posts.

The purpose of structure waves is not to explicate structure, but to visit various structural contexts, seeking opportunities to introduce new details in the scene. Structural contexts are local, and the only reasons global structure of the scene plays role in the order they get enumerated is that many structural contexts of the scene can only be restored in the context of surrounding structure, and that nearby structural contexts overlap, so changes to one context are also changes to neighboring contexts.

Thus, there is no reason for structure waves to reliably follow long paths in the scene, to proceed through many levels of instantiation in one pass. Structure wave only remembers its wave front, the structural contexts it’s currently visiting. Anything that doesn’t fit there gets forgotten, including the origin of the wave. Wave that starts at the root of global hierarchy of the scene and after several levels of instantiation finds itself inside a certain pattern, is equivalent to a wave that originated in that same pattern. The difference existed a step earlier, when the wave from outside was entering the pattern, keeping in its wave-front both instantiation point and part of the target pattern. At this point, wave can bring instance-specific changes to the pattern. Waves within a pattern solidify its structure, ignoring instance-specific variation, regarding pattern as a class rather than instance.

Related subtlety is that patterns themselves are also implicit. A slight variation in instantiation point (in the structural context at the wave front exiting the interface) may lead to assembly of a different pattern. This different pattern may be composed from mostly the same maps, but it will bear the trace of context in which it was instantiated, and so it won’t be identical to other patterns instantiated at similar interfaces, and it won’t self-assemble from within itself in this same form, without being found by a wave from outside. Along the trace, these differences retain the context of pattern roots that isn’t otherwise visible in the structural contexts of wave front. Structure waves starting from within those patterns don’t reconstruct the whole zoo, but focus on the common properties (choice between details otherwise brought by context from instantiation points being ambiguous). The reduced hierarchy, as discussed above, is based on the self-reconstructing patterns, but dynamic of structure waves running on it represents a more refined picture than one that can be produced by simply linking patterns from the hierarchy together.