It's an old ploy - saying that the self is whatever is left over after everything physical has been removed from the equation.
You are your eyes, your ears, your sense of touch, taste, smell. Lose one or more of these senses and you are, to that degree, less than what you were when you possessed that sense.
Your argument seems to be that because your sense of self would remain even after you have lost a sense (how can you know that it would unless you have actually lost some particular sense and can say, on the basis of that experience, that you know your sense of self to be undiminished?)then it cannot reside (the self) in that sense and, by extension, because it does not reside in that sense it cannot reside in any sense at all.
Fiddlesticks.
Where else is there for the self to reside except in the means by which the world is perceived, manipulated, interacted with? Where else, except in some spectral other-world, if not in the synergy of these processes with the consciousness that proceeds from the physical entity which perceives and the thing or things perceived.
Your argument seems to me to be fundamentally dualistic, and of that sort that says physical reality must always be subordinated to a greater reality that is other than physical (and more valuable simply because it is other than the physical) and likely to give rise to that loathing of the body and all its works that is typical of Christianity (I know you're not a Christian).
I don't deny that there is more to the universe than our physical makeup can directly perceive (ultra-violet and infrared light, for example). But our inability to directly perceive such things doesn't confine them to some other and more spectral world.
Buddhists may not believe that they are their own 'I' - but that doesn't mean that they aren't, or that there is anything else that could be.
