When I wear my five-fingers in the gym, I often see people funny and surprised looks. They look an extra time with a very surprised face. But they never ask anything... which perhaps a Swedish thing :-)
Anyway - do you have the same experiences? That people think that you are a bit weird/funny wearing strange shoes?