Or, do the details matter?
Concede that sometimes they do, cue here whingeing from me and from others about historical inaccuracies anent the rules of succession, the laws on divorce, etc, which have completely undermined our belief in the narrative we were reading.
But exchange earlier today on bluesky about specific time/place cultural references, do they throw you out -
At which I was, have I not read books involving baseball, and, on reflection, elaborate gambling scams, and I do not understand these at all, but this does not interfere with my enjoyment of the story. Possibly we do need to feel that the author knows what they're writing about and is not commiting solecisms on the lines of 'All rowed fast, but none so fast as stroke' - though apparently this is apocryphal.
I also felt that when I was reading that Reacher novel the other day that perhaps we had a leeeetle more detail than we really required about his exact itinerary whenever he went anywhere - the street-by-street perambulations in NYC, for ex. I am sure one could trace them exactly on a map, and any one-way systems were correctly described, and the crossings in the right place.
Which is sort of the equivalent of where I got 'futtock-shroudery' from, which was reading Age of Sail novels with Alot of period nautical terminology. (On the whole I though O'Brian got the balance on this right.)
There has been a certain amount of querying expressed in the Dance to the Music of Time discussions about some of the significance of parts of London invoked by Nick Jenkins, which is not just geography but Class (there was at least one passage where I was getting strong Nancy Mitford's Lady Montdore dissing on Kensington vibes), connotations of bohemianism, etc.
Sometimes the detail is load-bearing. But often it's not, particularly.