Turnbull also confirmed he offered Dutton the opportunity to stay on in the position but Dutton felt he could not stay on after challenging for the leadership.
Treasurer Scott Morrison will assume his job until a replacement is appointed.
If just seven more Liberal MPs switched their support to Mr Dutton, he would have the numbers to seize the leadership.
Cabinet minister Christopher Pyne said earlier Mr Dutton had told him the prime minister has his absolute support.
Seeking to safeguard his leadership, Turnbull called on his party to back him or risk losing the next election to the opposition Labor party.More news: US President Trump takes aim at Federal Reserve for rates hike
John Hewson, a former leader of the Liberal party who is now with the Australian National University's school of public policy, said Turnbull was wounded and another challenge was likely within weeks.
Dutton then refused to rule out a second leadership challenge. However, a state election in the state of Victoria scheduled for late November is seen limiting Turnbull to dates before the end of October if he meant to hold a national poll this year.
"That was not to be today and I understand and I respect the outcome and I fully support the prime minister and the cabinet".
But with the Turnbull's government at war with itself as a conservative faction has leveraged a slim parliamentary majority to challenge policy it disagrees with, voters may set aside questions over Shorten's leadership.
"The decline. may reflect the impact of the messy political debate locally and the associated slump in support for the current Turnbull government", said ANZ Head of Australian Economics David Plank.
Australia is due to hold a general election sometime before May next year.More news: Asia Argento Denies Sexual Assault, Says Payoff Was Anthony Bourdain’s Idea
Had just seven votes gone the other way, he would have lost office, and so the talk in Canberra has been when, not if, a second challenge would come. "I worked very closely with Malcolm Turnbull to make sure that we could achieve that".
Turnbull declared the leadership open earlier on Tuesday amid a backbench uprising as opinion polls showed the government on course for a heavy election defeat.
"It is really important that we put these differences behind us and get on with our job of looking after the 25 million Australians who have put us here", he said afterwards.
Senator McGrath was one of the key backers behind Mr Turnbull's rolling of Tony Abbott in 2015.
Australia's Immigration Minister Peter Dutton.
Liberals said the claims were certain to spark claims that Mr Dutton's political enemies were behind the story, but Professor Twomey told Fairfax Media she had been asked by Ten News about the legal issue on Monday of last week.More news: South Koreans gather on eve of rare reunion