This is my answer to the same question posed on the Mathematics Stack Exchange. It is therefore licenced under CC-BY-SA 3.0.

Question

It often happens in mathematics that the answer to a problem is “known” long before anybody knows how to prove it. (Some examples of contemporary interest are among the Millennium Prize problems: E.g. Yang-Mills existence is widely believed to be true based on ideas from physics, and the Riemann hypothesis is widely believed to be true because it would be an awful shame if it wasn’t. Another good example is Schramm–Loewner evolution, where again the answer was anticipated by ideas from physics.)

More rare are the instances where an abstract mathematical “idea” floats around for many years before even a rigorous definition or interpretation can be developed to describe the idea. An example of this is umbral calculus, where a mysterious technique for proving properties of certain sequences existed for over a century before anybody understood why the technique worked, in a rigorous way.

I find these instances of mathematical ideas without rigorous interpretation fascinating, because they seem to often lead to the development of radically new branches of mathematics. What are further examples of this type?

Answer

Following from the continuity example, in which the epsilon-delta formulation eventually became ubiquitous, I submit the notion of the infinitesimal. It took until Robinson in the 1950s and early 60s before we had “the right construction” of infinitesimals via ultrapowers, in a way that made infinitesimal manipulation fully rigorous as a way of dealing with the reals. They were a very useful tool for centuries before then, with (e.g.) Cauchy using them regularly, attempting to formalise them but not succeeding, and with Leibniz’s calculus being defined entirely in terms of infinitesimals.

Of course, there are other systems which contain infinitesimals - for example, the field of formal Laurent series, in which the variable may be viewed as an infinitesimal - but e.g. the infinitesimal \(x\) doesn’t have a square root in this system, so it’s not ideal as a place in which to do analysis.