I think me being male puts a lot of women off having photos taken wearing only their undies. Or even less! But there is really no need to let that affect you. I am a man for the same reason women are women. Genetics.
To me photography is a hobby, a passion and a career and I get just as excited about a baby, a wedding or a stunning sunset as I do a lady in lingerie. The fact that I'm a man and she's a woman is just a coincidence of chromosomes.
Don't get me wrong, I understand the concerns, but they are not necessary. Setting up as a photographer is a bit of a long-winded and expensive way to see what I can see for free on the internet! It's my job and I approach it with complete professionalism.
So if you were thinking of having some boudoir photos or a maternity shoot then please don't let that Y chromosome put you off!