Fast bowler James Pattinson has left the Australia camp in Brisbane and will return to his home state Victoria in the wake of his one-Test ban for personal abuse of an opponent.

Pattinson was handed the ban when he admitted abusing a Queensland player during a domestic match last week.

He had been vying with Mitchell Starc for the third pace-bowling spot in the Australia side for the first Test against Pakistan, which starts in Brisbane on Thursday.

Cricket Australia (CA) said Pattinson had been given a couple of days' leave to rest with family and would then return to his Victoria base to prepare for the second and final test against Pakistan in Adelaide from November 29.

The decision had been made during a regular meeting of team management late on Sunday and was considered the best course of preparation for the 29-year-old bowler, a CA spokesman said.

The spokesman denied local media reports that said the decision was made for "player welfare" reasons.

Australia have cracked down on bad behaviour by their players since the ball-tampering scandal in South Africa in early 2018 that led to one-year bans for then captain Steve Smith and vice-captain David Warner.

Current captain Tim Paine told state radio on Sunday that Pattinson knew he had "let himself down (and) let the group down". His Australia and Victoria team mate Peter Siddle backed Pattinson to regain his place in the Test side.

"It's an emotional game and sometimes you go a bit too far [...] but its because you care," the veteran seamer told local media.

"He's not trying to do anything stupid, he just cares about the team and wants to get results. He's got to miss this Test and that will hurt him."