Stony Brook Experts

Imagine if you could know where a person is going to look before they move their eyes. Realizing that capacity is the domain of gaze-fixation prediction and it is a core problem studied by Gregory Zelinsky and his lab. The Eye Cog lab is a world leader in training models to predict where a person will look while performing a task, making attention in task-driven viewing a part of the signal, not the noise.

Recently, our lab has pivoted hard to a promising new capacity: visualizing the contents of memory. Imagine if you could know what information a person encoded when they viewed something — and to test that representation directly. We developed a model-in-the-loop behavioral paradigm for doing precisely this. Based on a person’s viewing behavior, we use generative AI to model and visualize the world that they represent. Then, we show the AI’s generation back to the person and ask them to judge whether it is what they just viewed. When they are fooled, we learn what they encoded into memory — and what was lost.