Mandelbrot is credited for practically inventing the subject of Fractals, coining the name, working with other researchers and with IBM to crank out some of the earliest examples of computer-generated fractal imagery, and writing a series of books, which, although their price tags meant that they weren't exactly “popular” as far as retail sales were concerned, were actually fairly readable in parts by people without a background in advanced math background. And if you didn't like the text at all, you could always look at the pictures.
Most people know him from the Mandelbrot Set fractal that's named after him.
Some mathematicians probably get frustrated that Mandelbrot never seemed to give a conventional ironclad definition of what actually counted as a “fractal”. This wasn't sloppiness on Mandelbrot's part, it was quite deliberate. Fractals were “broken” geometry in several ways: they'd broken the rules and definitions of existing classical geometry, and to immediately replace those definitions rules with another similarly restrictive set would have been to fail to learn an important lesson. Unlike some in the mathematical community, Mandelbrot understood the occasional importance of appropriate imprecision. He could have produced a strict definition and classification system for fractals, but he chose not to. Instead, he looked at applying fractals to cosmology, to art, to the stock market and to social behaviour in general, and in the process he expanded our knowledge and appreciation of everything that fractal arguments were applied to. Knowing the fractal characteristics of a seismic tremor or a graph of stock market prices didn't let us predict when the next earthquake or financial crash (or tornado) was going to happen, but it let us understand more about the limits of our ability to predict.
To some other mathematicians, Mandelbrot's existence (and success) was probably a bit frustrating. Here was a high-profile guy popularising his work through books rather than just peer-reviewed papers, and making great strides without necessarily doing all the careful derivational work and constructed proofs that “normal” mathematicians considered essential. Where other mathematicians took great care to make sure that their works fitted flawlessly into the existing body of previous research, and with the existing rules and conventions, Mandelbrot simply declared the existing rules to be inadequate and unfit for purpose for the sort of problems that he wanted to study. Classical geometry was based on clean and simple geometrical entities such as straight lines and regular curves … and Mandelbrot essentially tore up the rulebook.
“Clouds are not spheres, mountains are not cones, coastlines are not circles, and bark is not smooth, nor does lightning travel in a straight line.”Like Stephen Hawking with Hawking radiation, and Albert Einstein with gravitational time dilation, Mandelbrot took an existing breakdown in mathematical procedure and instead of accepting the conventional wisdom that these breakdowns were “bad” and not worthy of study, he dove in and treated them as legitimate. He listened to the mathematics rather than to his fellow mathematicians, and if his results sometimes seemed like gibberish according to the conventional classification systems, well, that just meant that those classification systems were inadequate.
The math was what it was, and if the community didn't like it, then their opinions were based on sociology rather than mathematics, and could be ignored.
Einstein and Hawking and Mandelbrot all saw the distinction between the imperfect social operation of science as a community endeavour, and the idea of a deeper perfect underlying truth, and if the two failed to mesh, then they sided with the math.
An expert opinion was a transient fleeting thing, but an equation was forever. If you wrote an equation in an attempt to solve a problem, and instead of settling down to give you a single stable solution, it spiralled madly out of control, then many mathematicians considered your example to be a case of "bad" or "pathological" math, something to be noted only so that experts could add the example to their list of dysfunctional equations that we knew ought to be ignored and avoided. Mandelbrot's approach was different. An equation wasn't "bad" simply because it didn't do what we hoped of it. If a function didn't break any of the fundamental laws that spawned it, then its behaviour might be inconvenient, or even perplexing, but it was still mathematics.
Einstein wasn't a great physicist … in the opinion of his early college lecturers, he probably wasn't even shaping up to be a particularly //good// physicist. But Einstein broke through the mental blocks that had imprisoned some sections of physics theory and fractured parts of the existing definitional systems that we should probably have outgrown a century earlier. According to the prevailing social norms, Einstein arguably wasn't a "great physicist", but he was a physicist, and he //was// great at what he did. This wasn't necessarily what //other// physiicists did, but that's what made him different, and valuable. Similarly, Stephen Hawking still provokes grumbles from some math folk that he's not as good as the public think he is, and as a mathematician he rushes in and builds on unproven hypotheses in a way that a mathematician really shouldn't. Hawking's work arguably didn't demonstrate that he was a "great mathematician", but he's nevertheless one of the greats -- he was able to throw away the socially-ingrained preconceptions of how a subject "ought" to behave, and see how it really functioned. Both men "saw behind the curtain", and were able to communicate what they saw, and in the process they both expanded our conceptual vocabulary.
I'd place Mandelbrot in the same category.