The joint inspection team, from Ofsted and health watchdog the Care Quality Commission, found that council and health services are failing to offer children and their families their legal entitlement of support.
They found that leaders at North Somerset Council and the area's clinical commissioning group do not have a SEND strategy in place to improve support.
Health and council chiefs have also failed to meet the statutory deadline of April this year to convert SEND statements to education, health and care (EHC) plans, inspectors found following their visit in May.
When EHC plans have been put in place, often the quality is poor and too focused on education, with not enough detail about children and young people's health and care needs.
Monitoring and evaluating of SEND support is also weak, which means council and health leaders have been too slow to identify problems and put in place improvements.
"The provision that children and young people who have SEN and/or disabilities across the local area receive, and the outcomes they achieve, are not good enough," a letter outlining the findings states.
"Leaders acknowledges that there is no SEND strategy in place. As a result, not enough children and young people who have SEN and/or disabilities are being provided with the quality of support and services to which they are legally entitled as laid out in the 2014 SEND reforms."
Parents and carers interviewed by inspectors are also critical of their experiences with support services, and said they felt their views are being ignored.
"Parents and carers are overwhelmingly negative about their experiences of, and involvement with, the local area. The majority feel that they are not listened to by officers," adds the letter.
"They told inspectors that they have ‘been let down by the service' and typically have to ‘fight' to get their child's needs assessed to secure timely and appropriate support.
"Consequently, parents are largely dissatisfied and concerned about the services and provision that their children receive."
Inspectors also found that the local area's website to highlight local support "is not fit for purpose" as it is blighted by poor access and lack of promotion.
Joint commissioning across health, social care and education is also criticised for being "under developed".
In addition, professionals working with children "do not have an understanding of the importance of working together with parents, families and children and young people" to plan services.
"This lack of effective collaboration and co-production results in services being delivered that do not reflect or meet the needs of children and young people who have SEN and/or disabilities and their families.
"As a result, their needs are not being met effectively," adds the inspectors' letter.
Despite the concerns inspectors found that training of early years staff working with SEND children is effective. They also praised the development of specialist childminders to work with children.
Due to the seriousness of their concerns the two inspectorates have asked the CCG and council to send them a statement detailing how they plan to improve support.
"I'm pleased to receive this report - and while it shows that many services are working well to support young people with SEND, it also highlights that this is a long journey for us all in North Somerset and that we need to increase the pace," said Sheila Smith, director for people and communities at North Somerset Council.
"The findings of the inspection give us a helpful steer in how we can carry on improving things to ensure that all children benefit from a more joined-up system of support."