What you describe is entirely normal, and your way of describing it is accurate, if a bit casual! The ego is indeed fighting against being “downsized”. Longer, more discursive prayers give the ego a certain amount of elbow room, as it were; we can “sleep” between sentences, and “coast” through the memorized passages. But this isn’t possible, or at least it’s much more difficult, when we’re invoking the Name: its brevity, and the force of its repetition, keeps “squeezing” the ego, and the ego doesn’t like it!
What to do? A certain balance is important. It’s important, in other words, to combine rigor with gentleness. You mustn’t give up, or give in, in order to flee to something less demanding. On the contrary, it’s crucial, especially at times like those you describe, to take yourself firmly in hand and say, “This is my Way forward toward an increasing freedom and peace and happiness. I mustn’t allow myself to fall back into my old habits.” On the other hand, you must avoid all sense of “punishing” or “getting back at” the ego; this itself is just an egoic trick. After all, we’re trying to get past the illusion that everything somehow depends on us; it’s the Name that carries us, protects us, sustains us, feeds us—if only we let it.
So by all means don’t stop invoking: keep yourself focused on that practice, as much as you are able, but as you do so think of the “space” of the Name—the matrix of those precious syllables—as a little garden of peace, in which you need only rest and “melt”.