Speaking to CYP Now, the Conservative MP for East Worthing and Shoreham said there was little mention of children and young people's issues in his party's manifesto, conceding that Brexit will be the main preoccupation over the next few years.
He added that there is the additional challenge of having new faces in key policy areas after three ministers with briefs relating to the sector failed to retain their seats at the general election.
The Conservative manifesto features several pledges relating to children and young people, including introducing a new criminal offence for perpetrators of domestic abuse directed at a child, review of support for children in care, and to boost childcare provision.
There is also a pledge to look at ways to improve the family justice system. But Loughton said he does not believe the commitments amount to significant change for the sector.
"Brexit was always going to dominate things in terms of how much time is available for domestic issues," Loughton, who was children's minister between 2010 and 2012, said.
"[But] I'm afraid there is not a lot [in the manifesto] around children in care and or young people more widely which is a concern with me.
"It was a shame children and young people's issues didn't play such a prominent part in the campaign. There was such a good [voting] turnout among young people."
Loughton added that he believes reform for the youth sector will continue to be off the agenda, referring to the failure to implement reforms he put together on behalf of the government from his time in office in the form of the Positive for Youth policy statement.
"I fear that Positive for Youth has been largely on the shelf, gathering dust," he said.
He added that, in addition to a lack of firm plans going forward, the sector will also have to adjust to the loss of three "key" ministers who failed to secure re-election - children's minister Edward Timpson, youth minister Rob Wilson, and public health minister Nicola Blackwood.
"Three key ministers covering most of the [children and young people] remit lost their seats and there are completely new people coming into it," he said.
"That is a slight concern."
However, Loughton said he is more optimistic about children's mental health. The manifesto contains a pledge of better access to mental health care for children and young people, promising a green paper on young people's mental health before the end of the year.
"The Prime Minister flagged [mental health] as a big crusade," he said.
"I would hope that we are going to see progress on that."
Since the election, it has emerged that Robert Goodwill will take on Edward Timpson's role as children's minister, after he lost his seat in Crewe and Nantwich.
Meanwhile, sports minister Tracey Crouch has been handed the youth policy brief.
However, it is unclear who will take on the brief of childcare minister after Caroline Dinenage was moved to the Department for Work and Pensions as part of a wider reshuffle.
