Speaking at the annual Unite and Community Practitioners and Health Visitors Association conference in Bournemouth, Ashworth said analysis conducted by Labour has found that in the 12 months between June 2017 and June 2018, the total number of health visitors in England fell by 678 - from 8,588 health visitors to 7,912.
The total number had peaked at 10,309 in October 2015 following a five-year government drive to recruit an additional 4,200 health visitors, but it has since been falling.
Ashworth said numbers are now at their lowest level since October 2012, and highlighted a recent survey that found more than one in five health visitors are working with caseloads of more than 500 children as evidence that practitioners are under "intense pressure" with "potentially serious consequences".
He warned that government cuts are dismantling the country's public health system, failing some of the most vulnerable in our society and leaving children's services at risk.
"Unlike the Tories, I'm committed to giving every child the best possible start in life and central to our strategy will be delivering more health visitors in every community, ending the stark inequalities that continue to blight our children's future," he said.
"We know that high caseloads are bad for babies and bad for parents - mental health problems not dealt with, domestic abuse not spotted - sometimes with tragic consequences.
"Rising caseloads constitute a longer-term cost for the NHS because of the damage that arises from a lack of early intervention and prevention.
"We know the first 1001 days of a child's life, from conception to age two, are the most crucial for a baby's development and can have a serious effect on a child's health, education, brain development and behaviour as they grow up into adolescence and adulthood.
"In other words, by giving every child the best possible start in life, we create a strong foundation for almost every aspect of their future."
He added: "Investing in and supporting our health visitors will be a priority of mine in government."
Last month, it was claimed that health visitors are quitting the profession over fears they are unable to protect children.
A Department of Health and Social Care spokesman said: "Every child deserves the best start in life - that's why our Healthy Child Programme means they receive five mandatory health visitor checks before they are two-and-a-half.
"Improving the health of our children is a key priority for this government and will be pivotal in our long-term plan for the NHS, which we're backing with additional funding of an extra £20.5bn a year by 2023/24."