Sky News, citing the agency, reports that although the government had discussed the issue, it had resolved the challenges posed by Covid would be too great.
The games will be held from 23 July to 8 August and the Paralympic games from 25 August to 5 September. Both had been due to take place in 2020.
The International Olympic Committee is yet to formally confirm the decision.
A spokesperson for InsideJapan Tours, which had nearly 300 people due to travel to the rescheduled games, said that while the news game as a big blow, the operator understood why it was necessary.
