A report published by Somerset Safeguarding Children Board (SSCB) found Somerset County Council's children's social care department, Avon and Somerset Constabulary, Somerset Partnership NHS Foundation Trust and more than four schools, among other services, missed opportunities to intervene since referrals about the children's welfare were first made to children's services in 2003.
It acknowledged that agencies at times faced difficulties in pursuing measures to support the children because their parents, especially the mother, was "hostile and aggressive towards professionals and did not want any interference in how she chose to raise her children".
However, the review criticised actions such as police and children's social care failing to investigate an allegation of sexual abuse reported by the mother in 2013.
The review also discovered the youngest child's school had kept daily notes on their behaviour for a period of four years, including a sexually explicit drawing by the child, but these details were not included in reports written by the school for core groups and child protection conferences.
The report states: "The failure to address the safeguarding needs of these children was systemic and all agencies share some responsibility for the failure to act."
As a result, SSCB has introduced changes across all agencies, including the development of a board-wide neglect strategy with a toolkit and training materials shared across organisations to help recognition of and response to signs of neglect at an early stage.
It has also asked all schools to sign up to a "team around the school" approach, which supports intervention where problems, including neglect, are identified at an early stage, and provides enhanced safeguarding support for schools.
Board independent chair Sally Halls said during the time the case had spanned, agencies had seen "significant change and improvements" in practice and "a significant amount of work on training around the recognition and response to child neglect".
"It's important not to forget that child protection is a very difficult job, particularly when faced with hostile families who forcefully resist the intervention of those who are there to safeguard children," said Halls.
"In this case professionals were struggling to challenge parental behaviour while maintaining a working relationship with the family that allowed access to the children.
"But that is not good enough. The system that is there to protect children did not respond adequately and three children have suffered significant harm as a consequence.
"A multi-agency action plan has been developed by the board in response to the findings. Actions have already been taken by individual organisations and the board will ensure that these are seen through."
Somerset Council's children's services was rated "inadequate" by Ofsted in 2015, but improved to "requires improvement" at its most recent inspection in January 2018.