A sale would mark the end of more than 40 years on the haute couture scene for Cavalli, known as "The Leopard King" for his animal prints colourfully cast in leather, silk and velvet.
And it would demonstrate once again the appeal of Italian family-owned fashion firms for investors after rival label Versace sold a 20 percent stake to U.S. private equity firm Blackstone for 210 million euros in February.
An agreement between Permira and Cavalli could be reached in the coming weeks after the 73-year-old designer, rarely seen without his large sunglasses and cigar, dropped his reluctance to ceding a controlling stake, the sources said.
One of the sources said Cavalli would retain between 20 and 40 percent of the company.
Several lenders are currently pitching to be part of a financing consortium that would include a pool of up to four Italian and international banks, the sources said.
Societe Generale is already working with Permira on the transaction, said one of the sources. BNP Paribas and UniCredit are also expected to be involved on the financing, given their historic ties with the Florence, Tuscany-based maison, said the sources, who asked not to be named because the talks are private.
BNP Paribas declined to comment. UniCredit and Societe Generale were not immediately available to comment.
Roberto Cavalli has been courted by buyout funds since 2006, a sector banker with knowledge of the company said. In 2009 it held talks with Italian private equity firm Clessidra to sell a minority stake.
The company's managerial structure has recently been weakened by the departure of Chief Executive Officer Gianluca Brozzetti and Chief Operating Officer Carlo Di Biagio in January.
Permira, headquartered on London's Pall Mall close to Buckingham Palace, has deep pockets after it raised 5 billion euros ($6.9 billion) for a new fund in February.
The buyout firm, whose investments stretch from Fish Fingers in Britain to insecticides in Japan, has a strong appetite for fashion brands, having backed Italian haute couture brand Valentino in the past. It still has a controlling stake in German fashion house Hugo Boss and earlier this year secured control of British footwear brand Dr. Martens for 300 million pounds ($500 million).
Bankers familiar with the situation have valued Cavalli at around 450 million euros, based on a multiple of 15 times earnings before interest, tax, depreciation and amortisation (EBITDA) of 25 million euros for 2013.
That would exceed Versace's valuation, which came in at 14.5 times EBITDA, but fall short of the 31.5 multiple that Qatar's royal family paid to buy Valentino from Permira in 2012.
A spokesperson for Roberto Cavalli called the talks with Permira market rumours, while Permira declined to comment.
Cavalli himself, who is in full charge of managerial duties following the management shake-up, is leading the talks with Permira, said the sources. Law firm Bonelli Erede Pappalardo in Milan provides additional advice, the sources said. Bonelli Erede Pappalardo declined to comment.
"Talks are exclusively being held among principals," said one of the sources, referring to private talks between the owner and the buyout firm without any banking intermediaries.
The same person said a rival bid from French fashion group LVMH was unlikely at this stage. He said LVMH, maker of Louis Vuitton leather goods and Bulgari jewellery, had been interested in Versace but had shown no interest in Cavalli, despite similar houses styles using extravagant garments and bright colours.
LVMH declined to comment.
Last year, LVMH bought an 80 percent stake in Italian cashmere clothing company Loro Piana for 2 billion euros ($2.8 billion) without hiring any financial adviser.
After any Cavalli stake sale, other family-owned brands are expected to be targeted more aggressively by deal-hungry financial sponsors.
Possible attractive targets include fashion house Etro, knitwear brand Missoni, clothing producer Blufin, known for its core brand Blumarine, and footwear manufacturer Fratelli Rossetti, several industry sources said.