Quantum computing as a physics major — niche or growing field?

My advisor says quantum computing hires are limited and suggests I stick to condensed matter for better job prospects. But I find QC fascinating.

Anyone here heading into QC? What's the actual job market like for someone with a BS/MS in physics?