I'd like to second (third, fourth?) about giving him more time. I know it's hard, you just want him to integrate and become a part of the family. But like you said, you have no idea what he's been through prior. Cats are territorial, and not only was one disrupted by fire (I'm assuming), but he's also been shuffled through three different shelters. That's a lot for him to take on. I also second establishing "basecamp" as SierraAutumnGirl mentioned. Super important. You can even use a blanket or towel that he's willing to lay on in his deemed "safe space" and once he has his smell on it, place it somewhere else in the room - somewhere he can still feel safe, but in another part of the room - slowly expanding his world. Also, if you can use a wand toy to get him interested, this can also help builds his confidence. Moments of catnip, pets, treats, his smell in various places, and whatever he finds "Interesting" to play with, will all help him come out of his shell. You can also try Bach's Remedy for Pets (the pet version, without alcohol) that may help ease anxieties. It works on some, and others it doesn't. It all depends on their chemistry/what's causing their behavior. I know it might be a little hard, but you adopted him to give a home he could feel safe in - provide that for him, on his terms as needed, in order to heal. It's much easier to adjust your expectations than to adjust his healing response. You can never rush recovery. I hope, in time, you will both get what you need if not want.
#PetBehaviorandTraining