He also described as "ridiculous" the statement issued by the PMO attacking him on his book and said he was "amused".
Baru maintained that his book was a "balanced account" UPA-I and he had recorded "a lot of achievements" of Manmohan Singh and successes of the government that got them another term in 2009.
In his book 'The Accidental Prime Minister -- The Making and Unmaking of Manmohan Singh', he has written, "Pulok, who was inducted into the Manmohan Singh PMO at the behest of Sonia Gandhi, had regular, almost daily, meetings with Sonia at which he was said to brief her on the key policy issues of the day and seek her instructions on important files to be cleared by the PM.
"Indeed, Pulok was the single most point of regular contact between PM and Sonia. He was also the PMO's main point of contact with the NAC, a high profile advisory body chaired by Sonia Gandhi, with social activists as members. It was sometimes dubbed the shadow cabinet."
As the observations created a flutter, Baru said, "it was not a secret." He said it was well known that Chatterjee was Sonia Gandhi's secretary when she was the Leader of Opposition and had also worked with the Rajiv Gandhi Foundation chaired by her.
"He was very much part of the family...I was not witness to it, whether she physically saw the files. I knew she was consulted on issues and he was taking her concurrence," said Baru, who had resigned from the PMO in 2008 while Chatterjee joined it in 2011.
The PMO denied Baru's claim about Chatterjee taking files to Gandhi, saying it was "completely baseless and mischievous".
"It is categorically denied that any PMO file has ever been shown to Smt Sonia Gandhi," a PMO statement said.
Commenting on the book, the PMO said, "it is an attempt to misuse a privileged position and access to high office to gain credibility to apparently exploit it for commercial gain.
"The commentary smacks of fiction and coloured views of a former adviser."
PMO sources said Singh was very upset over the contents of the book and felt like he has been "stabbed in the back".