Look on victims of abuse as if what happened to them had happened to you.
-Hebrews 13:4b, MSG
Adultery is abuse.
The Bible is clear that when we have sex with another that we become one with them. That means affair partners are becoming one with the faithful spouse as well. It is a deep, violation. Soul rape.
Now, which side does the Bible tell us to take in such situations?
The side of the abused.
It really is that simple.
People who do not support the faithful spouse are not behaving in biblical ways. They are choosing a path outside the one outlined by Scripture for followers of Christ. It is not godly.
We–faithful spouses–ought to expect support from fellow Christians. Sadly, we usually cannot as they are more carnal than godly on this point.
But it remains true:

