While having more than 36 grams of alcohol speeds memory loss, researchers found no change in memory and 'executive function' in teetotallers, former drinkers, and light or moderate drinkers.
The executive function involves attention and reasoning skills in achieving a goal, researchers said.
"Much of the research evidence about drinking and its relationship to memory and executive function is based on older populations," study author Severine Sabia said.
Our study focused on middle-aged participants and suggests that heavy drinking is associated with faster decline in all areas of cognitive function in men, she said.
The researchers assessed drinking habits of 5,054 men and 2,099 women over 10 years. The average participant took their first memory tests at the age of 56.
They then repeated the test twice over the next 10 years.
While there were no differences in memory and executive function decline between men who did not drink and those who were light or moderate drinkers, it said.
Researchers, however, found that executive function declines between one one-and-a-half to six years faster in heavy drinkers.