For me it depends on the game.
On FPS games I mostly stick with normal because in a lot of them the AI has perfect targeting and does not share the same visual obstructions in some cases that the player does. For example, in Medal of Honor (I think, could have been another warfighter FPS like Battlefield or CoD) there was a building I was going through where there were curtains in the doorway, while I could not see through them, the AI did not have t - and would shoot me. A lot. In games like that on harder settings shots kill you faster so normal is about the good spot to not have to worry about those kinds of AI/Game Dev issues.
RPGs and most RTS I will go with a higher difficulty just for the challenge of it. For Witcher 3 I went with the hardest difficulty right out because I wanted the challenge and surprisingly, playing it on lesser difficulties now feels too easy. I did Divinity Original Sin and Pillars of Eternity on their hardest settings.