Whether or not you 'hold Napoleon responsible' is irrelevant.
I would think that if Napoleon didn't support the idea of the Egyptian expedition and was not in command, you'd still blame him.