Cambridgeshire County Council was warned by the inspectorate in April 2018 that caseloads were "unacceptably high" following a focused visit.
While some improvements have followed extra financial investment, Ofsted downgraded the service after a visit in January 2019.
The Ofsted report stated: "The most significant challenge to the local authority's ability to provide consistently good services to children, young people and their families has been, and continues to be, the size of caseloads.
"These are too high for most social workers and unsustainable in some teams.
"The impact of this is that, too often, social workers and frontline managers have had to focus on the most urgent and important work to secure children's immediate safety, without sufficient capacity for the follow-up work needed to sustain change within families or to ensure that children in care have permanent homes as soon as possible."
Inspectors found cases where social worker visits were overdue or cancelled, and assessments had not been completed in timescales that matched children's needs.
While inspectors found examples of good practice, they also found caseload pressures made it "very difficult" for some social workers to do more than focus on completing statutory visits.
Lack of staffing was also affecting the health needs of children, which inspectors said were not being well met.
However, the report did note improvements in some areas, and stated that the council's efforts to improve services were relatively new, and had not yet had time to demonstrate effectiveness.
Inspectors rated one area of inspection, "the impact of leaders on social work practice with children and families", as "good", as a result of the authority increasing staffing and implementing "a more coherent team structure" during 2018.
This included the remodelling of the multi-agency safeguarding hub, which inspectors said had led to "faster and more consistent progression of referrals about children, and the replacement of "life-long" social work units and 14 to 25 teams with specialist teams.
Inspectors said this action had: "Improved the focus on the distinct needs of children in different circumstances, for example, the differing needs of a pre-school child living at home compared to those of an older young person preparing to leave care."
In October 2016, the authority appointed Peterborough City Council's executive director of people and communities to jointly take charge of children's services.
Wendi Ogle-Welbourn was confirmed in post as Cambridgeshire's executive director of people and communities in June 2017.
The Ofsted judgment notes: "Fruitful co-operation with neighbouring Peterborough Council has included some shared leadership and service delivery."
Chairman of the county council's children and young people's committee Simon Bywater, said the report confirmed the authority understood "what needs to be addressed, and that we have put the foundations in place that will deliver positive change."
His reaction to Ofsted's findings were backed by council leader Steve Count, who said the report "gives me great confidence that Ofsted have given us an independent vote of confidence that the changes and investment we have made will deliver the improvements needed."
In its last full inspection of children's services in Cambridgeshire in 2014, Ofsted rated the council as "good" and said social worker caseloads were appropriate and manageable.
Last February, the average number of cases children's social workers deal with at any one time was reported to have risen to nearly 18, according to the government's annual workforce statistics.