The Democrat used an anti-Asian slur in an email he sent from his official government address.
Brady is accused of using racist and sexist terms to discuss human trafficking.
The group Delaware Asian American Voice recently put up a billboard on Lancaster Pike in Wilmington calling for his resignation.
They protested in front of the billboard on Sunday to raise awareness.
Delaware group calls for resignation of Rep. Gerald Brady
Organizers say other lawmakers need to do more to hold Brady accountable.
"I am still urging lawmakers to say the right thing, do the right thing, condemn him publicly and ask him to step down publicly," Delaware Asian American Voice chairperson Yushu Fu told Action News.
Brady apologized for his comments and has gone through sensitivity training.
He vowed in a statement last week to not seek re-election after his current term expires.
"I cannot in good conscience ask the voters to put their faith in me again after I betrayed theirs," said Brady, who is executive director of the Delaware AFL-CIO and was elected to the House in 2006.
Brady made the racist comments in a June 27 email he inadvertently sent to an advocate for decriminalizing prostitution. Thinking he was forwarding an email from an advocate to another person from whom he was seeking input, Brady instead mistakenly hit "reply" and sent his comments to the advocate.
Democrat House leaders directed Brady to complete sensitivity training and reach out to members of the Asian-American community in an effort to regain their trust. But they indicated in their own statement Monday that they are not interested in initiating disciplinary proceedings that could lead to Brady's suspension or expulsion from the House.
"We want to be clear about something we have heard from residents this past week: As a duly elected official, only Rep. Brady can make a decision about his political future. House leadership cannot unilaterally take action," House Speaker Pete Schwartzkopf, Majority Leader Valerie Longhurst and Majority Whip Larry Mitchell said in a joint statement.
That's not true, however.
Schwartzkopf, Longhurst and Mitchell are members of the House Ethics Committee, which is chaired by Longhurst and includes Republican Minority Leader Danny Short and minority whip Tim Dukes. House rules authorize the committee to investigate complaints that a lawmaker has violated the rules of legislative conduct.
Among the rules of legislative conduct: "A member shall not engage in conduct which the House determines (i) brings the House into disrepute or (ii) reflects adversely on the member's fitness to hold legislative office."
Ethics Committee rules state that any House member, including any member of the committee, can file a complaint. If a majority of the committee decides that the complaint has been proven, the committee can then, again by majority vote, recommend that the House take "appropriate action," up to and including expulsion of the offending lawmaker.
Despite describing Brady's remarks as "reprehensible, racist, sexist and indefensible," Democratic leaders indicated that, instead of an Ethics Committee investigation, they would make sensitivity training available to all members of the House.
"While we do not believe our colleagues harbor such views, it would be beneficial for them to learn of any microaggressions or other attitudes or actions that negatively impact the Asian American community, and how we all can take steps to improve our relationships with the community," they said.
The Associated Press contributed to this report.