In my talk, I argue that literary narratives generally invite us to imagine creatively. Literary non-fiction such as an autobiography or historical narratives, however, put an additional constraint on the author as well as the reader to truthfully reconstruct the experience of a person. This constraint can be in conflict with creative imagining: on the one hand, the literariness of a narrative invites us to imagine creatively; on the other hand, we are asked to imagine, empathically, what a person’s experience really was. To empathize, as I wish to understand it, means to consciously take over the perspective of a target, while getting as close as possible to having an imaginative experience similar to the lived experience of the target. I will argue that this constraint comes not only with a responsibility of the author to use literary means such that empathic imagination is facilitated, it also comes with a responsibility of the reader to reconstruct rather than create the narrative.