The Babe Tots group, which is run at Eastover Youth Centre in Bridgwater, is based next door to a children's centre, although group leader Hannah Fear said she did not think this was the reason for the reduced numbers.
"The children's centre runs a proper nursery where parents can leave their children, whereas we are a parent and baby group, so it works on a different basis," she said. "We have lost a few people as some of the mums have gone back to work or their children have started school."
Fear said if more people were not found, the group would not be able to afford to continue to operate. The group has been running for three years.
Fear said: "We have about six people who use it regularly and a few more who come by occasionally. It would be a real shame if we had to close it down."