Hand tools

Jakub builds sophisticated video recognition software that can interpret hand postures as input commands for a computer. I believe using the hand itself as a symbol is less exploitative of typical hand use than using the hand to manipulate a tool: few populations use hands for symbolic communication, and even those that do it constantly (brokers, hearing-impaired) do it as a result of constraints (too much sound; not enough). Even those individuals use their hands for tool and object manipulation as much or more than they do for symbolic communication. Hands evolved before symbols–they’re tuned for manipulation, not representation.

I think the universal means of symbolic hand interaction here could be pushed back a level to be the universal technology of interpreted vision. And the means should be different tools–tuned for each different task.

The same software can be used to recognize tools, as Jakub has beautifully done in his human/machine image above. I believe this use could be vastly more effective for certain tasks because:

• The hand would be used for what evolution “designed” it for
• The hand might use the same tool in different ways for different operation modes–as a pencil is used to draw, and upside-down to erase
• The physical tool will comfortably comport the hand, avoiding the need for uncomfortable hand-posture lock
• The feel of the tool may suggest how it can be used
• The look and feel of the tool can bring back memories and associations of interaction modes for specific tasks
• A tool might be more recognizable than hand postures for the system
• The hand posture may be more easily misinterpreted mid-action, “lost,” causing task flow interruption & restart.

Brad Paley