Well, the issue is that the benzoyl's drying effect forces your pores into overdrive, overproducing sebum/oil in an attempt to compensate for the dryness of your skin. If it continues for a long time, acne can come back, the skin will be more fragile/stressed due to the harshness of benzoyl, and more prone to damage and premature aging.

A lot of people with oily/acne-prone skin make the mistake of assuming dry skin = no acne, when what is actually preventing your acne is the chemical exfoliation of the benzoyl. SO many of my acne-prone clients are absolutely terrified to use a strong enough moisturizer because of the whole dry skin = no acne idea and instead just use the strongest anti-acne creams they can find (and most of this stuff isn't intended for the whole face, and should really only be a spot treatment at most), and its actually making their skin much worse off than it could be.

You can replace harsher stuff like benzoyl with other chemical exfoliators like lactic acid peels (makeup artist's choice makes a good one) or Beta Hydroxy or Alpha Hydroxy Acid creams/gels, along with a water-based moisturizer. Save the benzoyl for spot treatments. Skin that is simulataneously oily and dry is usually suffering from a lack of hydration (hence the oil overproduction to compensate), which can be solved by a water-based moisturizer. The traditional moisturizers people think of when they equate them with breakouts are usually oil-based for people with low oil production and true dry skin, rather than simply dehydrated skin.

Sorry for the novel, this is just a common misconception that is of great interest to me. :P