That's funny, because he also remarked about how he didn't appreciate being cited as religious.

The difference between Einstein and atheism is the difference between the cultural environment of his era and ours. If Albert Einstein had to suffer the religion-on-your-sleeve nonsense that goes on today, as well as the pervasiveness of anti-intellectualism, I'm certain the polarity of his disposition towards atheism and theism would reverse.

I think it is woefully inappropriate to say that Einstein believed in a "God" considering the wealth of information we have about him ridiculing the concept as it is defined by the vast majority of people.

Einstein had profound admiration for the mystery of the universe. But it sided with science more than arguments from ignorance (arguments for deism/theism):

It was, of course, a lie what you read about my religious convictions, a lie which is being systematically repeated. I do not believe in a personal God and I have never denied this but have expressed it clearly. If something is in me which can be called religious then it is the unbounded admiration for the structure of the world so far as our science can reveal it.

The man wasn't interested in intellectual dead ends; he abhorred anti-intellectualism. His criticism of atheism was specific to behavior of people who identified as atheists, not the philosophical position of atheism, for which his beliefs seem entirely compatible. He was interested in knowing how the universe worked and couldn't seem to help wonder if there was any form of consciousness out there that actually did understand how the universe worked.

Above all else, Einstein was probably smarter than most people and he didn't like other people pretending that they could explain or synthesize his views. He didn't trust anyone else to do it, so he didn't like using these labels.