It certainly can be.
As it is a summer sport, you are consistently in the heat...sometimes for the majority of the game.
When batting, especially if you spend a long time at the crease, it can be exhausting dealing with the heat and concentrating on every ball you face - some coming at 150km/h+ at your face, some moving sideways in the air, others moving sideways off the pitch and yet others turning at the last moment. You have to deal with hostile crowds, sledging from players on the field, the weather, the scoreboard and having to run quite often. Elite players get hit by the ball a lot as well and if really unlucky, they can lose their lives ala Phillip Hughes.
When bowling you are hyper aware that you will get very few chances to get batsmen out, therefore you are concentrating on every ball for hour after hour, day after day in oppressive conditions. The bowlers themselves (fast) cover many kilometers of sprinting as they run in to deliver each ball. The spinners bowl a huge amount of overs that are very taxing on the body.
As the games go for 5 days, you also have to deal with managing your preparation between days and innings. Net sessions for batting and bowling are still held, meetings are still held and you have to make sure you sleep well - not an easy task if you are 90+ not out over night or you need to get really quick wickets against the worlds best players the following morning. This is made even harder if you are not playing well and the media is laying into you as well. Quite often, you are dealing with all this stress whilst away from your family on the other side of the world.
Its a pretty taxing sport and I can see why young players are opting to specialise in the 20/20 format.