In recent years there has been a considerable interest in questions regarding approximate homomorphisms between groups, going under the name of group stability. For our setting, a group G is said to be HS-stable if any approximate finite dimensional unitary representation of G is close to a true unitary representation of G, where proximity is measured by the (normalized) Hilbert-Schmidt norm. In the situation of amenable groups, this question can be translated into a finite dimensional approximation property of the character space of G, an object originating in harmonic analysis. We will present an analysis of the character space of B. H. Neumann groups, an uncountable family of Z-by-locally finite groups, and as a result deduce they are HS-stable. The analysis involves both character bounds of finite symmetric groups, as well as character theory of infinite symmetric groups.