Their study concludes the plans will produce inaccurate and unfair comparisons across children's primary school life, because a few months' difference in age can produce pronounced developmental differences in pre-school children.
The current plans also do not take into account wider factors, including family income and educational history, as well as whether the family uses English as an additional language.
The panel is also concerned that the tests do not help identify specific needs or develop the best teaching strategies to help children.
Under the proposals, testing will be piloted during the 2019 academic year and then rolled out in autumn 2020.
But the expert panel says more robust evaluation is needed and this cannot take place until 2027, which is when the first cohort of reception age children take their final Key Stage 2 test.
"In the panel's view the proposed baseline assessment will not lead to accurate comparisons being made between schools, as policymakers assume," states the study.
"Perhaps most importantly, they will not work in the best interests of children and their parents."
It adds: "The panel believe that the government's proposals for the
reception baseline assessment are flawed, unjustified, and wholly unfit for purpose. They would be detrimental to children, parents, teachers, and the wider education system in England."
The five-strong expert panel was established by the British Educational Research Association (BERA) and includes Bristol University social statistics professor Harvey Goldstein and Pamela Sammons, who is professor of education at the University of Oxford.
The study's findings have been backed by Neil Leitch, chief executive of the Pre-school Learning Alliance for highlighting "the fundamental flaws" in the government's plans.
"The assessment in its current proposed form is highly likely to produce unreliable and invalid results, as it fails to recognise that young children develop at different paces and doesn't adjust for key factors - such as differences in ages and family background - which are likely to have a significant impact on assessment outcomes," he said.
"Add to this the fact that teachers and practitioners are likely to become increasingly pressured to 'teach to the test', and it is difficult to see how baseline results will have any real value.
"Policymakers urgently need to ask themselves what it is they hope to achieve with baseline. The policy does nothing to support children's early learning, nor does it achieve its primary goal of assessing school effectiveness.
"Rather than pushing ahead with such a flawed policy, government should go back to the drawing board on this and look to implement an approach to early assessment that prioritises the child and ensures that the first few years of their educational journeys help them to reach their full potential."
More Than A Score, a coalition of parents, teachers and education experts campaigning against "over-testing" in schools, also backs the study.
In a statement it said: "This independent expert report demolishes the case for baseline assessment. It demonstrates that testing four- and five-year-olds as they start school cannot possibly produce valid and reliable results.
"The test is intended by the government to produce a score by which children can be measured, and schools held accountable. BERA has shown that the test cannot in fact do what it is intended to do: it is useless, grossly expensive and clearly not fit for purpose."
A DfE spokesperson said: "The reception baseline assessment has received support from the schools sector and is being designed and delivered by the National Foundation for Education Research, with pilots in schools across the country, to make sure it works for teachers and their pupils.
"The assessment is just the first half of a progress measure and an important step in making sure that schools are recognised for the education that they provide to all their pupils whatever their background, including those in reception classes, year 1 and year 2."