Croatia striker Mandzukic, who joined Atletico in the close season from Bayern Munich, fired into the corner of the net from the edge of the area after two minutes and Atletico then played the best football, although there were chances at both ends.
It was a fiery game at the Calderon in which Atletico coach Diego Simeone was sent off in the first half for protesting and Real's Luka Modric was dismissed in added time for a second yellow card.
Real forward Cristiano Ronaldo started on the bench after suffering a hamstring strain this week and failed to make much impact when he came on at the start of the second half.
The defeat ended Real's hopes of winning all six trophies this season at the second hurdle after they lifted the European Super Cup last week by beating Sevilla.
It was also an indication that Atletico have lost none of their desire ahead of the La Liga kickoff this weekend.
Despite fielding several new players, they have started as they finished last term, with the same tenacity that saw them win the domestic championship and reach the European Cup final where Real won 4-1 after extra time.
"I am very happy, the group showed that they still have a lot of fight. This was another final - obviously not as important as the Champions League but still against a strong rival. It gives us a boost for what is ahead," Simeone told reporters.
"In a contest with two games we can compete but it is much more difficult over a league season with 38 games. Barcelona had a poor season and yet still could have won the league and Real Madrid had a poor start but they could have done as well."
Real coach Carlo Ancelotti admitted that his side lost their way in the second half.
"We had no control, there were too many long balls and we needed more possession," he told a news conference.
"We did well in the first half but in the second we gave them more of a chance to defend as they wanted."
New signing Antoine Griezmann, having returned late to training after the World Cup, began on the bench for Atletico in the first leg but was now ready to start and he made an immediate impact.
The Frenchman headed on to Mandzukic to finish with a drive past keeper Iker Casillas.
Atletico were charged up and as the tackles flew in from both sides, Real struggled to find their feet but they gradually settled and James Rodriguez spurned the chance to equalise.
Receiving the ball back from Dani Carvajal, his shot was well blocked by keeper Miguel Moya who is looking a solid replacement for Thibaut Courtois.
Watching from the sidelines, Simeone lost his cool and was sent to the stands as Real pressed forward and Rodriguez and Gareth Bale both went close with shots.
At the other end, Raul Garcia failed to hit the target with a free header from a corner.
Ronaldo came on at half time but it was Atletico who went nearer to scoring again with a Garcia header that came back out after hitting the underside of the bar.