For a non-amenable group $G$, there may be many (exotic) group $C^*$-algebras that lie naturally between the universal and the reduced $C^*$-algebra of $G$. Let $G$ be a simple Lie group or an appropriate locally compact group acting on a tree. I will explain how the $L^p$-integrability properties of different spherical functions on $G$ (relative to a maximal compact subgroup) can be used to distinguish between different (exotic) group $C^*$-algebras. This recovers results of Samei and Wiersma. Additionally, I will explain that under certain natural assumptions, the aforementioned exotic group $C^*$-algebras are the only ones coming from $G$-invariant ideals in the Fourier-Stieltjes algebra of $G$. \\ \\ This is based on joint work with Dennis Heinig and Timo Siebenand.