Judge Thokozile Masipa said she had been made aware that people watching Pistorius' trial on television feeds in a courtroom next to the main trial room were shouting and cheering at times during the proceedings.
"It is not an entertainment place," Masipa said. She said that if the spectators did not adhere to court protocol in the second room, "security will ensure that they leave."
Because of the huge interest in the trial, court authorities set aside a second, "overflow" room at the Pretoria courthouse for reporters and other spectators to watch the trial proceedings on a big screen.
Dozens of reporters are allowed in the main courtroom where the judge sits.
After Masipa's warning, chief prosecutor Gerrie Nel continued to question materials analyst and former policeman Roger Dixon on his findings regarding Pistorius' fatal shooting of girlfriend Reeva Steenkamp on February 14, 2013.
Dixon has offered a different sequence for the shots that killed Steenkamp, contradicting testimony by a police ballistics expert and the pathologist who did the autopsy on Steenkamp's body.
Nel, who also cross-examined Dixon on Wednesday, has criticized him for testifying about pathology, ballistics and other areas in which he acknowledged that he had no expertise.
Dixon testified that Steenkamp's wounds show she may have been in a different position than the prosecution says when she was shot multiple times through a toilet door by Pistorius.
The athlete is charged with premeditated murder and is trying to show that he mistook Steenkamp for an intruder behind a toilet cubicle door in his bathroom when he shot her in the pre-dawn hours.
The sequence of gunshots and positioning of Steenkamp when she was hit is critical to the cases of the prosecution and the defence.