Dave this a frequent point of debate amongst referees, match observers, referee coaches and the administration layer. The way the game has changed in the last 5 years at L5 and 6 for sure, has made it much quicker and more intense,with longer breaks for technical scrums.
To make it at panel level you have to have all components - fitness, speed, management (control and communication), law knowledge, empathy, consistency, rugby awareness, and the right image/culture/ethos.
Often on Federation and on Group we as Match Observers collect the evidence of who has them, who are missing some bits but can be developed, and those who have plateaued at L5 or L6 (which is no disgrace at all as these are high standards of refereeing achievement).
But you have to be quick enough physically and mentally to be at the breakdown to see the first offences, or better still manage them proactively, and fit enough to maintain high performance throughout the 80 minutes, and often need to raise intensity of both physical and mental aspects in the final 10 minutes