It's a bogus argument in any case. Zoe can marry a man, but I can't. Not that either of us would want to, but the point is that she's allowed to do something I'm not allowed to do for one reason only: her sex.
Apart from the fact that if I married anyone else it would be bigamy, of course.
On a serious note, there is one plausible argument I could see for why anti-miscegenation laws discriminate on the basis of race but opposite-sex-only marriage laws do not discriminate on the basis of sex. Although the racial laws were neutral on their face, they were universally understood to be part of a larger regime that held black people to be inferior. Opposite-sex-only marriage laws do not carry an analogous message that one sex is inferior.
Still, for conservatives, that's a pretty tough argument to push. If the law is formally neutral, do the proponents of "judicial restraint" (cough, cough) really want judges saying, "Yes, that law looks O.K., but I know what was really in the legislators' hearts, and it wasn't good?"
The fact is that, formally speaking, opposite-sex-only laws do discriminate on the basis of sex. The question is whether they're justified in doing so -- most people don't have a lot of problem with the police keeping men out of women's public bathrooms, after all, so the fact that a law allows one sex to do something but not the other isn't necessarily fatal.
Given the apocalyptic rhetoric around this issue, I'd say that most of the anti-equality crowd does think that "traditional marriage" is extremely important and that banning same-sex marriage is essential to protecting traditional marriage; if that were the case, then this would be one of those unusual instances where sex discrimination in the law was permissible.