Leadsom, who was appointed Environment Secretary yesterday by new Prime Minister Theresa May, made the comments in an interview with The Times last Friday while she was still running for the Conservative Party leadership.
During a discussion about the challenges faced by parents seeking childcare she said most people "do not employ men as nannies".
"Now you can call that sexist, I call that cautious and very sensible when you look at the stats, she added.
"Your odds are stacked against you if you employ a man. We know paedophiles are attracted to working with children. I'm sorry but they're the facts."
Neil Leitch, chief executive of the Pre-school Learning Alliance said he was "shocked and disappointed" by the comments.
"No one should feel that their career choices are limited by their gender, and yet such outdated prejudices are not only insulting to those dedicated male practitioners currently working within the sector, but also act as a barrier to more men entering the early years workforce.
"With the impending introduction of the 30 hours free entitlement offer, staff recruitment is a key challenge for the sector and encouraging more men to consider a career in early years is likely to play an important role in ensuring the long-term sustainability of the early years."
Voice, a union for education professionals that represents nursery staff and nannies, said it was "shocked and alarmed" by the "sexist and prejudiced" comments.
Tricia Pritchard, senior professional officer for early years and childcare, said: "Children need male as well as female role models. Both men and women are capable of providing excellent childcare, whether as nannies or in nurseries.
"There aren't enough men in childcare, and Mrs Leadsom's remarks have the potential both to damage the reputation of male childcare professionals already in the workforce and deter men considering childcare as a career.
"She should also remember that women can put children at risk too."
Department for Education statistics show that around two per cent of early education and childcare staff are men.
A 2011 DfE document titled Supporting Families in the Early Years, outlined the government's intention to tackle the "gender imbalance" in the sector and "make early education and childcare a viable career choice for all".
It added that the DfE was exploring with the sector how to promote careers in the foundation years, and how programmes such as the National Citizen Service and apprenticeships might give young men and women an insight into the sector.
However, research published by the London Early Years Foundation in 2012 found that 51 per cent of nursery workers believe men are discouraged from pursuing a career working in childcare because of "society's attitude".