Tactical Vision
By Jessica Lake
For months I have been trying to simplify vision. Every simplification removed something I had assumed was essential. Eventually I realized I had been asking the wrong question.
The question is not, “How do we build an internal picture of the world?”
The question is, “What is the minimum computation required to navigate it?”
That realization divided vision into two entirely different problems.
The first I call strategic vision. It is the rich, seamless experience we consciously enjoy: vivid colors, smooth surfaces, depth, shadows, texture, and all the beauty of the visible world. Whether this presentation exists primarily for communication, planning, or conscious thought no longer concerns me here.
The second I call tactical vision.
Tactical vision has no need for pictures.
It requires only enough information to recognize objects and act upon them.
That single decision simplified everything.
Objects became the fundamental unit of computation. An object might be a leaf, a chair, a crack in a wall, or an entire forest. There is no privileged scale because every object can be treated as a container of other objects. The hierarchy is recursive.
Likewise, a scene is simply another object.
Each object possesses a property sheet. It retains what matters to the identity being tested. Color might tell me that this is my bicycle even if it tells me little about whether the thing in front of me is a bicycle at all.
The system estimates which differences the conditions of observation can explain.
Lighting.
Shadows.
Perspective.
Distance.
Scale.
Orientation.
Color casts.
Even partial occlusion.
The object retains its identity through these changes in how it appears.
This inversion proved surprisingly powerful.
I had imagined transforming stored models until one matched what I observed.
I found it simpler to do precisely the opposite.
Estimate the transformations first.
Then normalize the observation back toward its canonical form.
Recognition can now compare normalized observations against the properties that define the identity being tested.
Constraint convergence performs the remainder of the work.
Recognition is the elimination of inadmissible answers.
A bicycle does not require two visible wheels, handlebars, and every other component before it becomes recognizable. Every observed invariant eliminates alternatives. Several explanations may survive, and they may all agree that what I’m looking at is a bicycle. For the action at hand, that may be all I need.
Exactitude is unnecessary.
Sufficiency is enough.
This same process repeats recursively at every level.
The environment constrains objects.
Objects constrain components.
Components constrain subcomponents.
The algorithm never changes.
Only the scale changes.
The first fixation therefore has a purpose very different from what I originally imagined. In this model, early fixations give priority to constraints that can help across the scene.
Where is the dominant light source?
Is the environment natural or artificial?
Where is vertical?
Where is the horizon?
What transformations are likely to govern everything that follows?
A recognizable object may help answer one of those questions. The point of looking there is what the answer can tell me about the rest of the scene.
Then the eye looks where the surviving hypotheses disagree most strongly. Each observation can rule out another set of explanations. The system carries forward what it has already learned about the conditions of observation.
The remarkable part is that constraint convergence is declarative. The constraints that can be evaluated with the evidence in hand can be tested independently and in any order. Thousands of those tests could run in parallel. The choice of where to look next depends on what those tests leave unresolved.
What began as an attempt to explain vision gradually became something much larger.
I think it is a computational architecture.
The same recursive declarative process appears capable of operating at every level of recognition.
Most surprisingly of all, it requires remarkably little that resembles what we ordinarily call intelligence.
Perhaps recognition is simply what happens when enough independent constraints leave an explanation sufficient for the action at hand.
Implementation Refinements
While constraint convergence is fundamentally declarative, practical implementation introduces an important consideration. Speed.
What the system learns about lighting, structural regularity, or vertical can help it interpret many objects at once. A broad constraint can earn an early fixation because of how much work it saves later. A particular object can earn that fixation when it supplies the evidence.
Another refinement emerges from the treatment of occlusion.
Objects generally continue when they are not seen.
A tree in front of a house does not cast doubt on the existence of the house. The system treats the house as continuing behind the tree. It can carry that assumption forward without filling in the hidden portion. If later evidence contradicts it, the assumption can be dropped.
Occlusion changes what is available to observe. The continuity assumption lets the system proceed without having to consider every possible hidden shape.
Statistical Fixations
One realization completed the architecture.
A fixation is a way to spend the next look.
Every surviving hypothesis predicts something about what can be seen. The next fixation goes where the expected observation would do the most to reduce the uncertainty that still matters. Early on, that might be a place where the system can estimate the color of the light or tell whether the scene is dominated by straight edges. Later it might be a single feature that distinguishes the remaining explanations.
There is no predetermined sequence of fixations. The available constraints can be tested in parallel, and the results guide the next look. As the scene becomes better understood, the system can revise its estimates and use them to normalize more of what it sees.
The same rule applies to an environment, an object, or a component. Look where the answer will do the most work. Stop when there is enough evidence to act.