Williams drivers Valtteri Bottas and Felipe Massa, who had started on the front row, finished third and fourth respectively.
The victory was Rosberg's third of the season and his team's seventh. It was also their sixth one-two finish of the year and came after Massa had ended their run of seven successive pole positions.
"It was not the easiest of races, trying to manage certain things which were right on the limit on the car. But in the end I had a very, very fast car again," said Rosberg, interviewed on the podium by former Red Bull racer Mark Webber.
"It was fantastic to win today. It was great also to get a one-two here in Austria. It's been great to come back here," added the German after his eighth successive podium finish and sixth career win.
In the first Grand Prix held in Austria since 2003, champions Red Bull were eclipsed at their own track with quadruple world champion Sebastian Vettel retiring and Australian Daniel Ricciardo finishing eighth.
Rosberg now has 165 points to Hamilton's 136 in the drivers' world championship standings. Mercedes have 301 points to Red Bull's 143 ion the race for the constructors' title, with 11 races remaining but with double points on offer in the final one.
The race was decided in the pitstops, with Rosberg starting in third place with Massa leading away from pole while Hamilton made a sensational leap from ninth to fourth at the end of the opening lap.
Bottas, celebrating his first Formula One podium, led for a lap after Massa's first stop that saw the Brazilian drop down the order and was ecstatic with his team's best performance since 2012.
"It's been a long way for us since last year and many many years at Williams. Now it's so much better," said the Finn.
"The race was exactly what we needed at this point, clean, nice, everything went like planned. The car was good for a podium this time and I'm so happy."
Mexican Sergio Perez also led the field for Force India until Rosberg hit the front on lap 28 with Bottas caught in a Mercedes sandwich and making it hard for Hamilton to overtake.
Ferrari's Fernando Alonso finished fifth, after also briefly leading, with Perez sixth and Danish rookie Kevin Magnussen seventh for McLaren.
Germany's Nico Hulkenberg was ninth for Force India after being overtaken on ther final lap by Red Bull's Daniel Ricciardo and Kimi Raikkonen took the final point for Ferrari.