Don't Wait for "The Right Time"

My Prototopia fantasy work is on its third rewrite, and I've received some interesting feedback.  Valuable, I think.

In the story, my two young main characters are very rapidly attracted to each other.  The woman, an escaped sexual slave, can only relate in the way she's known for all the past years.  So she doesn't even recognize the feelings.  The man has just lost someone very close to him, and is escaping the memory by crossing a desert called The Waste.  He doesn't want to lose again, so he's fighting it.

In my story up to now, their relationship deliberately stagnates as they cross the desert.  Without spoiling the story, something happens on the other side that triggers a new view of their future, and that's when I have the relationship progress.

But those who've read the story say that the story is flat through that progression.  They get my logic, but don't enjoy the story.  Defending a plot point as "It really happens that way!" doesn't work.  I've heard that over and over.  If the reader doesn't enjoy the story, then the realism is irrelevant.  They still won't read it.

So what I've discovered is that putting a plotline on hold for "the right time," doesn't seem to be working.  It's got to progress in some way.  Or I've got to find a better way to set it aside.

So I've got a bit of a rewrite ahead of me.  Excuse me while I get back to work.