The party plans to use some of the money raised by the move to fund an extra £4bn of education spending by 2022.
Currently, all children in the first three years of school receive a free lunch. The scheme was introduced in September 2014. Department for Education figures show that 1.9m children in English state-funded primary schools were eligible as of January 2016.
By scrapping the entitlement, the Conservatives aim to raise £650m.
Free school meals was a flagship policy for the Liberal Democrats during the coalition 2010 to 2015 coalition government.
But the Conservative manifesto published today states that the party does not believe that giving school lunches to all children free of charge for the first three years of primary school, regardless of the income of their parents, is a sensible use of public money.
"There is now good evidence that school breakfasts are at least as effective in helping children to make progress in school," the manifesto states.
"So under a new Conservative government, schools in England will offer a free school breakfast to every child in every year of primary school, while children from low-income families will continue to receive free school lunches throughout their years in primary and secondary education.
"The savings made from this change will be added to the core schools budget, meaning that every penny saved will go towards children's education."
Former deputy prime minister Nick Clegg pushed for the provision after an independent review of school food for the DfE found in pilots where all children received the free meal, pupils were academically months ahead of other peers and more likely to eat vegetables at lunchtime instead of less healthy food like crisps.
Clegg responded to the announcement by tweeting that abolishing the policy signalled "the sour, mean Conservative government to come".
Free infant lunches policy was saving millions of struggling families over £450. Breakfasts covers fewer children. Cheaper for govt. Cynical— Nick Clegg (@nick_clegg) May 18, 2017
Alison Garnham, chief executive of the Child Poverty Action Group, said the move would be a "big disappointment because the benefits are proven".
"When lunch is free for all primary school children, their school results are boosted, their diet is better and parents in food poverty have more money for nutritious family breakfasts and dinners," she said.
"And when hard-up families save on essential daily expenses, work incentives are strengthened and parents can get better off though work.
"If the current provision goes, the people who will suffer most will be the working poor. So surely it can have no place in an agenda for ordinary working families."
The Conservative Party's plans are in stark contrast to both the Labour Party and Liberal Democrats, which have both pledged to extend free school meals.
Labour's manifesto states it will introduce free school meals for all primary school children, paid for by removing a VAT exemption on private school fees.
The party also wants to overhaul education by introducing a National Education Service, which will deliver free education from "cradle to grave".
It plans to raise £19.4bn by reversing the Conservative Party's cuts to corporation tax to spend on extending free childcare entitlements, restoring education maintenance allowance for college students, and restoring student grants for university students.
Meanwhile, the Liberal Democrats' manifesto says it will: "Extend free school meals to all children in primary education and promote school breakfast clubs."
Its other plans for education include investing nearly £7bn more in schools and colleges over the next parliament and trebling the early years pupil premium.