By Brian Todd and Steve Almasy
(CNN) -- There are as many as five bidders to buy the Los Angeles Clippers, with offers up to $2 billion, a source with knowledge of the situation told CNN on Wednesday night.
The team is owned by the Sterling family trust, with Donald and Shelly Sterling each owning 50% of the team.
Donald Sterling was banned by the NBA earlier this month and fined $2.5 million for racist remarks, and the league began proceedings to terminate the Sterlings' ownership rights.
The NBA has scheduled a special June 3 meeting of the Board of Governors for a vote on the matter.
The league said Wednesday it had received responses from Sterlings to the NBA's charge that Donald Sterling had violated the NBA constitution through comments made in an audio recording posted online and in an interview with CNN's Anderson Cooper.
if 75% of the 30 NBA ownership groups vote to terminate the Sterlings' franchise rights, the league will hire an investment banker to sell the team.
In his response, Donald Sterling accused the NBA of violating his constitutional rights in trying to end his ownership of the franchise based on what he says was an illegally recorded conversation during a lovers' quarrel, according to a document provided by Donald Sterling's attorney.
The document, Sterling's 26-page response to charges of violating the league's bylaws, adhered to the NBA's Tuesday deadline for the billionaire businessman's reply.
Sterling has been prejudged and won't get a fair hearing, the document says, citing comments from 10 other teams.
About one month ago, Silver banned Sterling from NBA facilities and functions and ordered him to pay a record fine after racist remarks by the Clippers' owner to former companion V. Stiviano surfaced online.
A source close to the situation said roughly two weeks ago that Donald Sterling -- who repeatedly denied he's a racist in a CNN interview -- wouldn't pay the fine, would reject the ban and might sue the NBA.
The document says Sterling faces a never-before-issued and unjust punishment.
"In reality, Mr. Sterling is being banned for life, fined $2.5 million, and stripped of his ownership rights for a purely private conversation with his lover that he did not know was being recorded and that he never intended to see the light of day," the document says.
Other NBA figures have made bigoted or racially insensitive remarks and weren't punished as harshly, Sterling argues.
His remarks were made in the heat of an argument, and while they were not justified, they were not meant to harm the NBA, the response says.
And while Sterling's remarks were viewed as racist, he says in his response that he was "instrumental in fostering the diverse body of players, coaches, general managers, employees and fans on which the NBA prides itself."
The document also says Sterling was active in the African-American community and that there was only one court case -- which he won -- of discrimination brought against him as owner of the Clippers in more than 30 years.
Sterling asks that the NBA's charges be denied and that his ownership rights not be terminated.
™ & © 2014 Cable News Network, Inc., a Time Warner Company. All rights reserved.