An inspection of Tinsley House immigration removal centre, near Gatwick, concluded that it is "one of the best-performing immigration removal centres", but calls for action on a number of areas of concern.
The Prisons Inspectorate report said some families with children could have simply been accommodated in a hotel prior to their deportation flight.
The report found that between May 2013 and November 2014, 123 families and 140 children had stayed in the unit.
Most families held there had been refused entry at Gatwick or other regional airports and were returned on the next available flight, with the average stay at the centre running to around 11.5 hours.
But inspectors said they found a number of cases where detention had not been necessary.
"We were not persuaded that sufficient consideration was always given to alternatives to detention for families and why many could not just be accommodated in an airport hotel," the report states.
Inspectors also said that escort arrangements for families were "unacceptable".
"In one case while we were there, a Brazilian family with a five-year-old and an eight-month-old baby came into the centre at 3.55am after arriving at Gatwick from Brazil at 2.20am," the report states.
"They reached their rooms at 4.30am and then had to leave the centre, with the children exhausted, at 10.30am to wait in an airport holding room for a flight at 5.40pm because female escort staff were not available to take them at a later time.
Inspectors also said they were not assured that management and risk assessments required by the different groups held in the family unit were given enough attention.
They also raised concerns that some detainees' cases had been progressed too slowly "resulting in prolonged, unnecessary and therefore possibly unlawful detention".
But overall, inspectors said they were happy with conditions.
They found that reception and early days arrangements were good, and the general atmosphere at the centre was "calm" with few recorded incidents of violence or bullying.
Meanwhile use of force was low, security was deemed to be proportionate, and relationships between staff and detainees were good overall.
And the centre was praised for picking up on two cases of suspected trafficking, where staff had carried out checks and detained children had subsequently been taken into the care of the local authority.
Nick Hardwick, chief inspector of prisons, said: "Tinsley House continues to be one of the best-performing immigration removal centres.
"However, that in no way detracts from the need to ensure that no one is held there unless absolutely necessary.
"That is particularly so with families with children and we were not persuaded that necessity to detain was always adequately considered for individuals or families.
"Tinsley House has many strengths on which to build, but it now has improvements to make."