Let $N_k(m,n)$ denote the number of partitions of $n$ with Garvan $k$-rank $m$. It is well-known that Andrews--Garvan--Dyson's crank and Dyson's rank are the $k$-rank for $k=1$ and $k=2$, respectively. In this paper, we prove that the sequence $\{N_k(m,n)\}_{|m|\leq n-k-71}$ is log-concave for all sufficiently large $n$ and each integer $k$. In particular, we partially solve two conjectures on the log-concavity of Andrews--Garvan--Dyson's crank and Dyson's rank, which were independently proposed by Bringmann--Jennings-Shaffer--Mahlburg and Ji--Zang.