解:$(1)$设$A$种花卉的单价为$x$元,$B$种花卉的单价为$y$元
根据题意,得$\begin {cases}2x + 3y = 21\\4x + 5y = 37\end {cases},$解得$\begin {cases} x = 3\\y = 5 \end {cases}$
∴$A$种花卉的单价为$3$元,$B$种花卉的单价为$5$元$.$
$(2)$设购买$A$种花卉$m {株},$则购买$B$种花卉$(10000 - m)$株,总费用为$W {元}$
根据题意,得$W = 3m + 5(10000 - m)=3m+50000 - 5m=-2m + 50000$
∵$-2<0,$∴$W {随}m $的增大而减小$.$
∵$A$种花卉的株数不超过$B$种花卉株数的$4$倍
∴$m\leqslant 4(10000 - m),$解得$m\leqslant 8000$
∴当$m = 8000$时,$W $取得最小值
此时$10000 - m=10000 - 8000 = 2000,$$W=-2×8000 + 50000 = 34000$
即当购买$A$种花卉$8000$株,$B$种花卉$2000$株时,总费用最少,最少总费用为$34000$元$.$