解:$(1)$设$A$种花卉的单价为$x$元$/$株,$B$种花卉的
单价为$y$元$/$株
由题意得$\begin {cases}2x + 3y = 21\\4x + 5y = 37\end {cases},$解得$\begin {cases}{x=3}\\{y = 5}\end {cases}$
∴$A$种花卉的单价为$3$元$/$株,$B$种花卉的单价为$5$元$/$株
$(2)$设$A$种花卉采购$a$株,总费用为$W $元,
则$B$种花卉采购$(10000 - a)$株
由题意,得$W = 3a + 5(10000 - a)=-2a + 50000$
又$a\leqslant 4(10000 - a),$∴$a\leqslant 8000$
∵$-2<0,$∴当$a = 8000$时,$W $取最小值
$W=-2×8000 + 50000 = 34000$
此时$10000 - a = 10000 - 8000 = 2000$
∴当$A$种花卉采购$8000$株,$B$种花卉采购$2000$株时,
总费用最少,且最少总费用为$34000$元