Amateur scientist

There have been a few more stories that have followed the amateur scientist route. These days, people will tell you that it is difficult to do scientific research on your own. This is not always true. To be sure, there are branches of scientific research where it would be difficult to conduct without having a lot of collaborators around. But there are still a lot of areas, like pure mathematics and theoretical physics, where a large deal of the work centers around locking yourself up in a room, and cranking things out with a pencil and paper. Recently I have read three stories which revolve around this theme.

The big story nowadays is Zhang Yitang, a University of New Hampshire lecturer who hadn’t published anything for 10 years, and later on stunning the world with a proof of a weaker form of the twin primes conjecture. It turned out that he had been working on and off that problem for years, in secret, until one day, the final piece came when he was at a friend’s house.

There was another story of a mathematician who claimed to have proven the abc conjecture. He wrote a 500+ page manuscript, and it was so dense and abstruse that nobody could read it. Since he refused to go on a lecture tour and present his results to the mathematical community, many have argued that he hadn’t actually proven anything. It does say something about knowledge that if you cannot communicate it, it’s not really knowledge.

Then there was an outlandish claim by a mathematician turned hedge fund analyst that he managed to construct a way to make quantum mechanics square with general relativity, which is one of the biggest unsolved problems of physics. Apparently his lecture just took place a few hours ago, so I can’t be sure what the reaction was like.

In any case, when things like that happen, you have to ask yourself, why do these big and bold ideas take place outside of academia, by people working outside of the system? Isn’t the whole purpose of a tenure system so that professors can be let alone to do their own work in peace without people bothering them? But no, these days it would be very difficult for that to happen. Department chairs would be extremely eager to push their universities up the rankings, and then they would get put all sorts of outrageous burdens on their professors. They wouldn’t really have the time and space to do a lot of tranquil thinking and reflecting that people need. As a result, a lot of progress gets made. But it tends to be the incremental improvements, rather than the quantum leaps that scientific revolutions are made of.

Still, a lot of work gets done, but I get the feeling that peer reviewed science does tend to hamper progress. It demands a more linear progression of science, one where the old ideas are piecewise modification of the new ideas, rather than something that blasts the old edifice of the new ideas to smithereens and starts something anew. In fact, we don’t really have a lot of honest research going on on economics. There was the Kenneth Rogoff incident about a few dodgy statistical methods that exaggerate the impact of debt on the economy.

Perhaps one of the more interesting cases of amateur research has been Judith Rich Harris’ great idea about development psychology: she struck a big blow against the traditional idea that peoples’ personalities are shaped by their parents, and argued that peer influenced plays a more important role. It is a point of view which I am increasingly leaning towards, and it explains a great deal of things, such as why siblings turn out so different even when parented by the same people, why the country you grew up in has such a great impact on you, and why people have this tribal instinct about them.

To be sure, amateur research has contributed only a relatively small amount to the body of knowledge. Although when they contribute, these contributions tend to be huge. Even an example like Andrew Wiles solving Fermat’s Last Theorem, is the work of a hermit. Obtaining the Fields medal apparently scared off Grigory Perelman, who solved one of the seven great open problems in mathematics, into leaving mathematics for good (although some speculate that he was working on another of the millennium problems.