Australian Muslim leaders on Wednesday said they would boycott a meeting with Prime Minister Scott Morrison after he said they needed to do more to halt terror attacks in the country.

Morrison singled out Muslim community leaders as having a "special responsibility" to counter "the radical and dangerous ideology of extremist Islam" following a terror attack in Melbourne earlier this month that left two dead.

"They must be proactive, they must be alert and they must call this out," Morrison said, adding that he would hold a roundtable meeting with Muslim leaders this week to discuss the problem.

But Australian Grand Mufti Ibrahim Abu Mohamed and a group of other senior Muslim figures rejected the invitation in an open letter to Morrison on Wednesday.

The men said they were "deeply concerned and disappointed" with comments by Morrison and other ministers "which infer that the community is collectively culpable for the criminal actions of individuals and should be doing more to prevent such acts of violence".

"These statements have achieved nothing to address underlying issues, but rather, have alienated large segments of the Muslim community," they said in the letter, which was published by Australian media.

The letter prompted a tweetstorm from Morrison, who accused those behind the boycott of "continuing down a path of denial" and making their communities "less safe and more vulnerable".

"We all have responsibilities to make Australia safe, and that means making sure Muslim communities do not become infiltrated with this dangerous ideology," he tweeted.

In the November 9 attack in Melbourne, a Somali-born man who police said was inspired by the so-called Islamic State, stabbed and killed one man and wounded two others before being fatally shot by police.

And this week police arrested three more men in Melbourne for allegedly plotting another Islamic State-inspired attack on the city — the 15th terror attack authorities say they have thwarted since 2014.

The terror threat has fuelled calls from some conservatives for a further toughening of Australia's already tight immigration laws, raising concerns among the Muslim community.

In their letter the Islamic leaders, who head community groups in several Australian states, said they would attend a later meeting with Morrison if their "views and concerns will be genuinely respected".