I'm not prejudice or racist by any means, but should it matter if both my coworkers are black?

I work at an agency of an insurance company, soon to be licensed as an agent. Both of the agents that I work with are black and I'm white. I've never worked so close-nit with black people before, I've always worked with a few white people at least. Well most of our clientelle are black also, so I rarely see any white people. It doesn't bother me, but I'm just used to working around my own race. I'd rather not switch agencies just because they are very motivated, hard working agents and we are actually number 2 in our region. So they are always out getting new leads, and making our office successful. But I'm just worried that if I stick with them, I'll never be able to meet people of my own race, or have clients of my own that are white. Is it wrong to feel like this? Please don't get me wrong, I'm not racist, I have some friends that are black. But it's just different fo me in a work setting. Do you think they hired me for a better market, because I'm white and attractive maybe?