"It's a worrying vote because it means that Switzerland wants to withdraw into itself," Fabius told RTL radio. "We're going to review our relations with Switzerland," he said, referring to Europe.
Swiss voters on Sunday narrowly backed proposals that curtail the relatively free movement of citizens to and from the EU and could unravel bilateral accords with the bloc.
Switzerland is not a member of the EU, but the bloc represents its biggest trading partner. The current pact on free movement of people supports Switzerland's globally oriented economy that employs large numbers of foreign professionals.
The yes vote for the "Stop mass immigration" initiative was also sharply criticised by the European Commission in Brussels, which said it would examine the implications for its relations with Switzerland.
"In my opinion it's bad news both for Europe and for the Swiss because Switzerland will be penalised (economically) if it withdraws," Fabius said.