Torbay Council's children's services, which was rated by Ofsted as "inadequate" in 2015 and again in June 2018, was taken over by Plymouth City Council in April last year.
But inspectors found there has been failure to address "significant concerns" raised during their last inspection and the pace of change, under Plymouth's management, is "currently too slow" and " must accelerate", states a letter from Ofsted to Torbay.
Concerns include a lack of organisation around visits and note recording as well as problems in retaining social workers, which makes it difficult to assess whether children at risk are safe, says Ofsted.
"A briefing provided to inspectors described a ‘chaotic way of working', often with visits completed by different workers but not recorded or prioritised appropriately," states Ofsted's letter.
"This is exacerbated by the constant staff churn at social work and team manager level. This is extremely concerning. It means that it is not possible to track whether some children deemed to be at risk of significant harm are safe. The quality of child protection and children in need plans are too variable and do not address individual children's needs."
The quarterly monitoring visit found that assessments are also not considering children's wishes and feelings, and visits following a referral are not swift enough to ensure children are safe.
While inspectors acknowledge that Torbay has successfully recruited senior and middle manager posts it has struggled to attract permanent, experienced social workers and frontline managers.
Many social workers have caseloads in excess of 30 children, inspectors also found.
"The service currently relies on high numbers of agency staff, leading to some children experiencing numerous changes in social worker," adds Ofsted's letter.
Despite the concerns, inspectors praised the dedication of social workers who are described as "committed to improving the quality of services for children".
Staff also told inspectors that they welcomed a recent introduction of monthly performance reviews which helped improve communication between frontline managers and senior leaders in the department.
Torbay's and Plymouth's director of children's services, Alison Botham, said: "We are disappointed that Ofsted inspectors judged that we haven't made enough progress. However, we are very pleased that they recognised we have dedicated, skilled staff, who are committed to improving the quality of services to children.
"As a result of their monitoring visit we have carried out a review and taken decisive action to address the issues raised and move our improvement plan forward. We are committed to making the necessary changes as quickly as possible to ensure that we make consistent and sustainable improvements that make a real difference to the lives of children and young people we work with."
The decision to allow Plymouth to take charge of Torbay's children's services was made by the Department for Education in October 2017.
The move had been recommended by Hampshire County Council chief executive John Coughlan, who was appointed by the DfE as Torbay children's services commissioner.
Last month, Plymouth's children's services was rated "requires improvement" by Ofsted.
A DfE spokesperson said: "Torbay is a complex case, and so we are grateful for the support Plymouth City Council has provided to Torbay Council in helping it improve. Equally though, we recognise that the findings are concerning, and are working closely with the authority to consider the next steps to ensure improvements."