The institute said the move would help recognise mental health problems and offer support at a far earlier stage. It said health visitors are well placed to offer mental health advice to all family members and refer them to specialists if necessary.
The IHV said cuts to staffing numbers and local health provision in recent years have meant that health visitors are less able to identify mental health problems swiftly. It wants the new responsibility to come with additional funding.
The recommendation has been made in the IHV's response to an NHS England consultation on mental health in the NHS Long Term Plan.
"Health visitors, through their universal non-stigmatising service, were once the eyes and ears of the community, skilled at holistic assessment to identify families at risk of and/or suffering from mental health problems," states the IHV's response.
"However, due to the restructuring of services and the cuts to their numbers since they moved under local authority commissioning (at a time of cuts to the local authority public health budget), this function is now severely challenged."
Other measures put forward in the IHV's response are for more research on understanding infant mental health, including looking at the impact of domestic abuse and loneliness.
Support for fathers' as well as mothers' mental health is also needed, adds the response.
"Early intervention for families in the perinatal period will reduce the burden of mental illness across the life-course, said IHV executive director Cheryll Adams.
"We have to get that message over to politicians and policymakers, so that funding is allocated upstream and not disproportionately spent on secondary and tertiary care, fixing problems that could have been prevented from occurring in the first place."
NHS figures released last year showed that the number of health visitors fell by more than 10 per cent between October 2015 and January 2017.