Let $\mathrm{Mat}_{n \times n}(\mathbb{C})$ be the affine space of $n \times n$ complex matrices with coordinate ring $\mathbb{C}[{\mathbf x}_{n \times n}]$. We define graded quotients of $\mathbb{C}[{\mathbf x}_{n \times n}]$ where each quotient ring carries a group action. These quotient rings are obtained by applying the orbit harmonics method to matrix loci corresponding to the permutation matrix group $S_n$, the colored permutation matrix group $S_{n,r}$, the collection of all involutions in $S_n$, and the conjugacy classes of involutions in $S_n$ with a given number of fixed points. In each case, we explore how the algebraic properties of these quotient rings are governed by the combinatorial properties of the matrix loci. Based on joint work with Yichen Ma, Brendon Rhoades, and Hai Zhu.