scifi from 50s or 60s? about man who becomes invisible to his coworkers

I read this story some time ago. Man wakes up and can't get anyone on the street or in his office to see him or react to him. Over time he realizes there are a few other people who also are not noticed but some of them are sadistic and torment those who can't see them. When they realize that he is also invisible to most people, they chase him because they get more satisfaction out of tormenting people who can see them. It was not necessarily science fiction but maybe speculative fiction? Sort of like "The Unpleasant Profession of Jonathan Hoag" about modern society not working like we think it does.

I hope this jogs some memories. I believe this was written in the middle part of last century

Maybe The Sinful Ones by Fritz Leiber? Most people are entirely confined to predestined actions, with no free will. A few "wake up" and can detach from their appointed motions. The non-woke keep reacting to the scheduled actions of the woke, to the point of having conversations with someone who's not present. So the woke are essentially invisible to the non-woke. Some of the woke torment some of their fellows.

I haven't actually read this book, but just flipped through the first few pages of the copy I've had since (apparently) 1980, so I can't be sure. Original publication is 1950, about right for your dates.

I read it so long ago that I can't remember whether the details are the same, but is there a chance it could be The Glamour?

It is The Sinful Ones! Thanks so much. A thought provoking book and really scared me when I first read it. It is the type of book Lieber would write. When I looked at Amazon for the book it says it was originally published with the title YOU ARE NOT ALONE. I think that is the one I remember.

I'm glad someone was able to find the title, because now I want to read it too...

>4 DevaRae: Glad I could help.