The Securities and Exchange Commission said the NYSE, two of its exchanges and one affiliated brokerage "repeatedly engaged in business practices that either violated exchange rules or required a rule when the exchanges had none in effect."
The NYSE agreed to settle the matter and pay a $4.5 million penalty without admitting or denying the charges.
A spokesman for the NYSE declined to comment beyond the settlement.
The charges against NYSE mark the latest in a series of cases filed by the SEC since 2011 against exchanges, dark pools and other trading firms on a variety of market structure-related matters.
The SEC has been scrutinizing market structure issues more broadly in recent years, particularly in the wake of the May 2010 "flash crash."
Several of those cases, including Thursday's case, have focused on instances where an exchange's duty as a self-regulated organization (SRO) has clashed with its business practices.
Under federal securities laws, exchanges have very specific duties as SROs to police their own marketplace and provide transparency about their activities.
One such duty, for instance, requires exchanges to disclose material changes to their businesses by submitting proposed rule changes to the SEC for approval and public comment.
The SEC's case against NYSE, NYSE Arca, and NYSE MKT outlines a laundry list of instances between 2008 and 2012 in which the regulator says the exchange either failed to follow those rules, or engaged in practices without first getting a rule approved.
In one example, the SEC said that NYSE provided co-location services on "disparate contractual terms" without a proper rule in effect.
Co-location refers to a service in which exchanges allow trading firms to place computer servers inside their data centers so they may get to see the data faster.
The SEC also charged Archipelago Securities, an affiliated routing broker, with failing to have policies designed to prevent the misuse of material non-public information.
None of the charges rise to the level of fraud, hence the reason for a lower penalty.
Rather, the case is intended to send a message to the market that the SEC is serious about enforcing market structure rules.
"We will hold exchanges accountable if they fail to have rules governing their operations or fail to follow them," said Andrew Ceresney, the head of the SEC's enforcement division.
Thursday's case differs from the one filed by the SEC against NYSE in 2012.
In that matter, the SEC charged NYSE will giving certain customers "an improper head start" on trading information due to software issues.
The exchange paid $5 million to settle that case.