But the former Foreign Secretary is refusing to back down, claiming his observations were innocuous and adding it was "ridiculous" to attack his comments.
Mr Mitchell, who was chief whip in 2012, told BBC Two's Newsnight: "I don't think he should apologise. One day, I am sure, they will go".
The party has received complaints that the comments breach its code of conduct, and under Conservative rules they must be investigated by an independent panel. He said he opposed ban- ning burqas and other face-cov- ering garments worn by some Muslim women, but referred to a woman wearing the attire as "looking like a bank robber".
That provoked a backlash from the party's right, with many MPs repeating similar arguments.
But with anger from the Brexit wing of the party over even starting the process, party headquarters confirmed that Mr Lewis - who has already criticised Mr Johnson's comments - would not pick the panel.More news: Emery names five Arsenal captains
"This is quite an important issue about free speech and it's got nothing to do with the awful events that take place over Enoch Powell and the Rivers of Blood speech".
But Muslim organisations said there had been "a real impact" from Johnson's remarks.
The Tory peer said he had received a wave of abuse since speaking out against Mr Johnson.
A spokesman said only: "The code of conduct process is strictly confidential". "But we don't immediately shut those people down if we take offence".More news: How blue light accelerates blindness
The actor, known for his comedy performances in Mr Bean and Blackadder, said the remarks were amusing.
The actor, who plays funnyman Mr Bean, said that you should only apologise for a "bad joke" and that on that basis - "no apology is required".
Senior figures such as Chairman Lewis and Prime Minister Theresa May have called on Johnson to apologize, and Muslim advocacy groups such as the Muslim Council of Britain have called for more action to be taken against Islamophobia in the Conservative Party.
'Full bandwagon jumping territory'Conor Burns, Tory MP for Bournemouth West'It's bash Boris'Andrew Bridgen, Tory MP for North West Leicestershire " This [inquiry is] being use to try and discredit Boris Johnson from people who are anxious about him in inside the Conservative party and outside the Conservative party.More news: Trump condemns 'all types of racism' on Charlottesville anniversary; critics slam wording