A survey by Ofsted found that although there has been a 10 per cent increase in the number of two-year-olds from low-income families taking up the government's offer of free early education, around 80,000 are not.
Currently all three- and four-year-olds in England are entitled to 15 hours a week of free early education or childcare, as well as around 240,000 two-year-olds whose parents receive certain benefits.
This means that around one in three two-year-olds are missing out. The Ofsted report found that in some areas take-up among disadvantage two-year-olds was as low as 34 per cent.
Local authorities have a duty to ensure there is sufficient, high-quality early years provision in their local area. However, Ofsted chief inspector, Sir Michael Wilshaw said too few prioritise their role in helping disadvantaged children to benefit from free childcare.
"I am concerned that, as things stand, no one is being held to account for this scandalously poor performance," Wilshaw said.
"As a result, the opportunity to directly influence the future path of thousands of poorer children is being lost."
"Action is needed on a national and local level to address these variations and to ensure the weakest places learn from the best," he said.
Neil Leitch, chief executive of the Pre-school Learning Alliance, said providers are reluctant to offer two-year-old places because they are more expensive than those for three and four-year-olds.
He said two-year-old places, which often involve providing one-to-one care and spending a significant amount of time working with external agencies, are "not financially sustainable" for providers.
"With the impending roll-out of the 30-hour free childcare offer [for three and four-year-olds], there is a danger that access to these places will be reduced even further as providers have to make a choice between creating more places for three- and four-year-olds and retaining their two-year-old places."
Purnima Tanuku, chief executive of National Day Nurseries Association, said: "There is clearly more work to be done to ensure as many disadvantaged two-year-olds as possible are able to take up the places they are entitled to.
"The report highlights the need to make sufficient numbers of high-quality places available. Better funding is the key to ensuring this," she said.
"Chronic underfunding is causing some nurseries to limit the numbers of two-year-old funded places that they are able to offer."
Anne Longfield, children's commissioner for England, said: "I am concerned with reports that so many poor children are missing out on vital early years support that could improve life chances.
"Children who grow up with poverty and disadvantage are more likely to have poor health, will have poorer employment prospects and are more likely to have poor mental health."