I think this is spot on. A common objection to AGI, which has already cropped up in the comments, is that human intelligence is tightly coupled with our bodies. In a way this is a fair point: Self-awareness originated in, and still mainly consists in, the monitoring of physical states and perceptions of those states.

The problem with this objection, it seems to me, is that it is used to draw precisely the wrong conclusion. An AGI could arise which is very much more powerful than our own minds, but go undetected precisely because it is not embodied. An AI which has subjective states concerning only its own subjective states would have no phenomenal presence. Pace Nagel, it would have the mind of a bat without the bat, forever invisible to us.

This AGI could emerge to our eyes very suddenly, already more advanced than a human mind in one or more ways, at the moment it begins to interact with physical hardware.