Donald Trump was expected to pack the Richmond Coliseum when he visited the city last week.

After all, he has packed arenas in other cities.

But in battleground Virginia, and this Democratic-friendly city, the presumptive Republican nominee ended up addressing a half-full building Friday evening.

A Richmond minister who led the rally’s opening prayer estimated that 6,000 to 7,000 people were on hand in a space that can hold up to 13,000 individuals.

Pastor Steven A. Parson Sr., founder and former pastor of the Richmond Christian Center, called the attendance impressive given the “one-day confirmed notice.”

A staunch Trump supporter, Pastor Parson believes more people would have come if much of the advance media coverage had not focused on the likelihood of a heavy police presence aimed at preventing disruption from Trump “protesters,” whom the minister described as “young people who are paid or go by the race card the Democrats have played for years.”

Despite all the talk, only about 200 protesters gathered and there was minimal disruption. Richmond Police reported detaining five people; one for disorderly conduct.

However, for others, the relatively small attendance for the Trump rally could be a signal that interest in the GOP candidate has peaked and could be on the decline, at least in Virginia, a state seen as a must-win for any contender for the presidency.

The Richmond rally took place after Mr. Trump came under fire for bashing the U.S.-born federal judge who is presiding over a fraud case involving the defunct Trump University. Former students are seeking damages, claiming that the for-profit university that Mr. Trump led ripped them off.

Mr. Trump surprised and dismayed a wide swatch of Republicans when he attacked U.S. District Judge Gonzalo Curiel. Mr. Trump’s claim: The judge, the son of Mexican immigrants who was born in East Chicago, Ind., is biased against him because Mr. Trump is promising to build a wall on the southern U.S. border to keep Mexicans from illegally entering the country.

The attack on the judge appeared to be the last straw for a number of GOP party members, who have been not been enthralled by many of the remarks from Mr. Trump, who launched his campaign last year stating Mexicans who cross the border without permission are “criminals and rapists.”

During his free-wheeling speech in Richmond, Mr. Trump doubled down on attacks that others view as racist.

For example, he expressed tongue-in-cheek remorse for calling U.S. Sen. Elizabeth Warren of Massachusetts “Pocahontas,” a jab at a past claim she has made of being part American Indian.

“I’m doing such a disservice to Pocahontas,” Mr. Trump told the crowd.

Sen. Warren, a Democrat whose heritage claim was questioned during her run for the senate in 2012, has been an outspoken Trump critic and is often mentioned as a potential running mate for presumptive Democratic presidential nominee Hillary Clinton.

The crowd rewarded his remarks with chants of “Build the wall!” and “USA! USA!”

Outside the Richmond Coliseum, protesters chanted, “No KKK, no fascist USA, no Trump.”

The demonstrators were generally peaceful until some Trump backers with “Wall” signs ventured into their ranks, leading to shoving and the throwing of fists.

Richmond police officers quickly moved to end the confrontation, pulling at least two Trump backers over a barricade and leading them away with their hands bound.

During his run for the nation’s top office, Mr. Trump has attracted some hard-core conservatives while repelling others.

His brazen shots at people like the judge seems to have helped push staunch Bernie Sanders supporters into the arms of Mrs. Clinton while splintering his own party.

That’s how a man wearing a Ted Cruz shirt and hat found himself in the unusual position of protesting outside the Coliseum alongside people shouting to protect abortion rights.

Once a dyed-in-the-wool Republican, the man said that a Clinton presidency would be better for the conservative movement. In his view, the billionaire businessman would cut deals with Democrats and sully the name of the conservatism.

Still, Mr. Trump won a hearty on-stage endorsement from Republican state Sen. Richard H. Black of Loudoun, one of the General Assembly’s most strident Christian conservatives. Sen. Black, who made headlines at home and abroad by meeting recently with Syrian President Bashar al-Assad, told the crowd that Mrs. Clinton had to be defeated to prevent her from stacking “the United States Supreme Court with Marxists.”

Alice Martin, 21, is one of the many young Bernie Sanders backers coming to terms with the Vermont senator’s defeat at the hands of Mrs. Clinton. Ms. Martin described the former secretary of state “wishy-washy” and unprincipled.

Still, the Virginia Commonwealth University senior said that Mrs. Clinton would likely get her vote. “I would never vote for Donald Trump. He’s going to regress this country.”

Since the rally, Mr. Trump has continued to offend. Witness his remarks this week in the wake of the massacre in Orlando, Fla., in which a gunman mowed down 49 patrons at a nightclub catering to gay people and injured 53 others.

Within hours of the horrific attack, Mr. Trump sharpened his vow to ban temporarily the admission of Muslim immigrants if he becomes president and to suspend immigration from areas of the world where “there is a proven history of terrorism.”

He did so in pointing to the Muslim ties of the gunman, who was killed by police.

Asked his response to Mr. Trump’s remarks, Pastor Parson said that Mr. Trump wants a “temporary” ban of Muslims and only wants to ban radical Muslims, not the “good ones like Muhammad Ali.”