Mostly, this was just poor writing.
Xander and Anya plan a wedding. The day arrives. Xander has a vision. He sees a future time in which he and Anya have turned into bitter people; he is bitter, stay-at-home guy; she is bitter go-out-to-work gal. He comes back to the present and breaks off the wedding.
It's so stupid, it makes you want to cry.
It is not unbelievable that Xander would run out on a wedding; it is unbelievable that he would do it in that way for that reason.
The thing about Xander is that he has no special powers. He's not superhuman. Or a (good) vampire. Or a witch. He's good at carpentry. That's all. His asset to the scoobies is one thing only: he sticks around. Even when he is so scared, he's going to wet his pants. Even when he is making bad jokes. Even when he is mocking his own freakingoutness, he sticks around. In a way, he's a bit of a lonely character, and it's to his credit that he recognizes in Buffy an equal degree of loneliness, a kind of remoteness. He fixes windows. He carries stuff. He waits to see if he can help.
The point being, that for Xander to leave Anya, the last thing that would work would be the threat of his own unhappiness or even the threat of things going wrong. Right up till the marriage, Xander is the optimist in the relationship. Considering how the season had run, I thought Xander would get back from the vision and say, "Yeah, well, life is hard, but I'm not so dumb or pessimistic that I'm going to buy into some demon's nightmare—even if the demon is me."
The point being: the vision is a total cliché. It is so unconnected to the actual personalities of Anya and Xander, it makes zero sense that level-headed Xander would buy into it.
What Xander would have bought into is a future where Anya has become completely dependent on him: the cute chidings of the present would have morphed into bullying, the kind of constant bullying that is completely unseen or acknowledged by the participants but easily noted by an outsider. Anya, for all her outspokenness, is prone to a "My man is everything" mentality, and Xander does have a streak of bully in his nature (which makes his refusal to act on it so much nicer). It would bother Xander more to see himself turning into a bully than to see himself turning into a bitter, old guy.
I think the writers knew this and tried to write this explanation back into the season later (the episode where Anya takes out an entire fraternity, for instance), but it was too late. They failed in the wedding episode and they failed big time. They didn't have enough set-up. Their payoff was a cliché. And the result was that Xander appeared a bigger jerk than he deserved.
Some of us simply excised the episode from our memory banks. I rewrote it in my head so it actually made sense.
It was a cop-out. Presumably, the writers felt that Xander and Anya just shouldn't get married. Maybe they felt Xander and Anya were too darn young (their characters, not the actors), which is probable considering that the latter half of the show seems to have been constructed by writers with serious 30-something angst. The point being: the writers used "marriage is hard and can suck" in the same way the Ballykissangel writers used Assumpta's death: it's SO BAD we don't have to prove that it actually makes sense.
And anytime a writer tells you, "Oh, yeah, but in real life there are no real answers and things don't make sense and bad stuff happens," you can be sure that what they are really saying is, "I don't know how to write a better ending so I'm going to be lazy and use the horribleness of life as a cover." I said the former myself as an undergrad in college, and the latter is exactly why I said it.