It's a matter of trust. If I can't trust that it's truly open source, then it erodes some of the confidence I have in them. I haven't personally tried running a functional Signal server without the spam filtering code, so I don't know if it truly works without it.