Felicity Jones and Ralph Fiennes in “The Invisible Woman.”

The affair between British author Charles Dickens and his mistress, actress Nelly Ternan, lasted for 13 years, until the author’s death in 1870. In The Invisible Woman, the newest film by Ralph Fiennes, the actor-director paints a slow-burning tableau of their life together, crafting less of a story than a series of vignettes that continue to haunt Nelly after she marries and establishes a family of her own.

Felicity Jones lends strength to the film with her nuanced portrayal of Nelly, who was only 18 when she first met the 45-year-old Dickens as a bit actress in one of his plays. We first encounter Nelly not as Dickens’ mistress, but in her role after his death, as the wife to a gentle-natured headmaster in Margate, England. Though she lives a secure life with her own family, Nelly cannot escape her past. We see her lips tighten as her students rehearse a scene from one of Dickens’ plays, and we see her stiffen when her husband casually brags that his wife knew the great author as a child.

The story of their affair evolves, then, from a series of flashbacks that Nelly draws upon as she lives in Margate. Fiennes, for his part, plays an affable Charles Dickens — energetic, childlike, with a twinkle in his eye. But for all his playfulness, something cruel lurks beneath the surface. Consider, for example, a scene on Nelly’s birthday, when a jeweled necklace Dickens bought for her is carelessly delivered to his wife Catherine by mistake. Preoccupied with his writing partner, the author insists Catherine hand-deliver it to the house where he is currently boarding Nelly and her family, telling her to take it to its “rightful owner.” Later on, Dickens further humiliates his wife by announcing their separation in a newspaper editorial, leaving her unaware until the article is read aloud by her oldest son. Catherine’s uncontrollable sobbing after learning the news provides a rare moment of emotion in the usually muted film.

It is the women who provide the real heart of the movie, and the crux of the story, as we watch their methods of coping with their patriarchal Victorian world. It is Nelly’s mother, wearily played by Kristin Scott Thomas, who implicitly encourages her affair with Dickens, shrewdly recognizing that her daughter lacks the theatrical talent needed to survive without a wealthier benefactor. Nelly, too, is slowly forced to accept the reality of her situation. Early in the film, she is insulted by the insinuation when Dickens takes her to visit the gilded apartment a fellow writer has gifted to his own mistress. Less than a year later, she is gazing out the window of her own country house, a comfortable place for her to stay while Dickens shills his books around Europe. A view of an English castle is better than the cramped Manchester house she shared with her mother and sisters.

It’s exactly that logic, and the lack of true emotion behind characters’ decisions, that prevents The Invisible Woman from landing as a historical romance. Fiennes meticulously develops the relationship between Dickens and Nelly, so much so that their romance makes no sense. Society would destroy them, we’re told, if their affair was discovered, but there is no sense of urgency, no raw passion between the characters. Instead, we’re left with two talented actors, politely exchanging endearments against elaborate period sets.