The Thrill of the Frontier <> The Soul of Research
We tend to think about knowledge in a limited and egocentric way. We view ourselves as the peak of what it means to understand, and beneath us we see everything else as existing on sequentially lower bands of a knowledge spectrum. But what if we're not at the peak? What if we're just at one contour of understanding, and there are contours above us that we can't see, can't communicate with, can't even sense? When we find that our assumptions are not consistent with reality, we must forfeit our egos and adopt a new perspective, despite how unintuitive or strange it may seem.
The best research comes from recognizing our earthly dispositions and limitations of our cognitive tools, and even our imaginations. How do we even begin to speculate about the truths we don't yet know? What if we viewed our interaction with knowledge as a special case, only a subset of its entirety? What do we really know? We know some things. We have a strong sense that some things are true. But we also have the sense that, whatever knowledge is, we tend to have less of it than we think. While we do seem to understand some things, we get the sense we cannot fully understand the world, while the world can fully understand us. Thus not only is knowledge found in what we know, but it also appears to be a continuous property. That is, something isn't just known or not known, but rather there are degrees of understanding, knowledge exists on a spectrum.
When we extrapolate further, down to the realm of things we barely understand, we seem to hit a mental barrier. But are we letting our assumptions bleed into a realm we do not fully understand? We observe knowledge existing on a spectrum. How can we possibly say when that spectrum ends? Isn't an asymptotic behavior of a continuous variable like knowledge more in line with the behavior of other continuous variables in our universe? While it feels highly unlikely or impossible that we could understand everything, to believe the opposite is to deny the simplest explanation of nature.
The frontier is wherever we are when we're doing something that might not work, when we're asking questions that might not have answers, when we're trying to understand something that might be beyond our current contour of consciousness.
Science is a strong-link problem. Progress comes from the best work, not from preventing the worst work. Most institutions optimize for the weak link. They try to filter out bad research, to maintain standards, to ensure quality. But filtering is expensive, and it filters out the weird stuff too. The stuff that might be wrong, but might also be revolutionary.
You can't solve strong-link problems by being more careful. You solve them by being more prolific, more diverse, more willing to be wrong. You need more attempts, not better filters.
This is why the best research often comes from outside the institutions. Outsiders are free to be wrong in interesting ways. They don't have to justify their work to a committee. They don't have to fit it into a grant proposal. They don't have to make it legible to people who've never thought about the problem before.
The Virtues of Research
At first, I thought research was about being smart, about knowing the literature, about having the right credentials. Those things help. But they're not the soul. The soul is the part that just wants to know, that just wants to understand, that just wants to see what happens if you try this weird thing.
Most people are too afraid of looking stupid to do great research. They optimize for appearing smart, which means they optimize for safe bets, which means they optimize for incremental progress. The soul of research is carefreeness. The freedom to work on things that might not matter, to pursue questions that might not lead anywhere, to spend time on problems that might be impossible. Most institutions can't afford carefreeness.
Be wrong in interesting ways. Most people are wrong in boring ways, or they're too afraid to be wrong at all. Take risks that matter, not risks that look good on paper. Make things beautiful, not just useful.
On Frontiers
In viewing the spectrum of knowledge as a landscape, there are gentle plateaus that questions exist on, where real understanding is possible, where communication between ideas flows easily. But there are also sharp cliffs, or contours where meaningful understanding is simply impossible from our current perspective. We are doomed to look down from our contour of knowledge and observe those questions lower than us, able to appreciate them but not fully understand them. Just as there are those questions, from a higher contour, looking down on us, willing to be understood, but unable to reveal themselves to us in a way we can grasp.
The willingness to recognize that we might be missing something fundamental, that our perspective might be limited, that there might be contours of understanding we can't even imagine from where we're standing. Everything else is just infrastructure. Important infrastructure, sure. But infrastructure nonetheless. The soul is what makes it matter. The soul is what makes it beautiful. The soul is what makes it worth doing.
Yes, professional science does a lot of good stuff. It gives people paychecks, health insurance, research funding, offices, and colleagues. It allows large groups to work together on big projects like launching telescopes into space. And it gives young, curious people a place to start: if you want to ask and answer questions about the universe, academia is an obvious career path.
But that good stuff comes at a price. Professions are bundles of weak-link interventions; they keep out quacks, but they also keep out revolutionaries. They enforce standards, which tends to make things…standard. They select for a pretty homogenous group of people—in this case, folks who got good grades in college, did research in the right institutions with the right people and published in the right journals. Then they make all those people even more similar to one another, steeping them in the same culture and putting them in competition for the same rewards, like grants, jobs, and citations.
Right now, professional science is like a world where every organism is trying to be a mammal. Mammals are great: milk-producing glands, body hair, ears that have three bones in them, what's not to like? But if you've only got mammals, you're in big trouble. Monocultures are fragile and prone to collapse because every single organism has identical weaknesses. What you need is an ecosystem—hawks, sea urchins, fungi, various types of fern, and so on.
Creating diverse ecosystems is hard for humans because they like to do whatever everyone else is doing, even when they know it's wrong. So when you're trying to be a mammal and you see someone else trying to be a lizard, you might think they're just doing a bad job being a mammal.
"You should try having little hairs all over your body," you might tell them. But a lizard isn't a bad mammal. It's a lizard. Its job is to eat flies and bask on rocks.
Here's the thing about being a lizard: you have to be willing to look stupid. Great scientists were generally quite stupid, in the sense that they were practiced in practicing ignorance. They had cultivated the virtue of saying and doing things that were entirely boneheaded, because this is vital to the process of discovery.
Francis Galton, the guy who came up with (among many other things) correlation, twin studies, fingerprinting, weather maps, and questionnaires, was independently wealthy. So were Darwin and Boyle and many of the members of the original Royal Society and British Association. Dabbling in science used to be a common pastime for rich dudes—a little astronomy here, a little vivisection there. Money makes everything easier, obviously, but plenty of now-legendary scientists had to do their thing part-time while finding other ways to pay the bills. Einstein published some of his most important work while he was still a patent clerk. Thomas Bayes, whose theory of probability still sets nerd hearts aflutter, was a priest. Gregor Mendel, the pea plant guy who founded genetics, was a monk. Galileo and Da Vinci spent much of their careers trying to worm their way into the good graces of various patrons. Marie Curie did her early work in a shed next to a French university; they only gave her a job after she got famous. Antonie van Leeuwenhoek sold drapes to support his side hustle of inventing microbiology.
I predict that these last few decades, in which professional science nearly eradicated the dabbler and the part-timer, will turn out to be a blip in history, for two reasons. First, the structures of academia are so warped, competition is so fierce, and opportunities are so scarce that even its biggest adherents spend most of their time complaining about it. When numerous Nobel Laureates are saying that they couldn't have done their Nobel Prize-winning work in today's system, something's bound to break.
And second, for the first time in human history, the tools of science are cheap, and knowledge is nearly free. Your laptop can store and analyze more data than Galileo could have even imagined. Internet pirates have toppled the scientific paywall and made nearly every paper ever written freely accessible to everybody. Nobody can stop you from uploading a PDF of your research to the internet, where tens of thousands of people might see it. The only thing stopping you from jumping in is your own fear.
If you start with something you don't understand, there's a good chance that soon enough you'll bump up against something that no one understands.
What if you don't have any formal training or credentials? Formal scientific training is way less formal than you think. You might imagine that when you enter a PhD program, a wise old scientist sits you down and tells you all the secrets of science. This doesn't happen. You take a few classes, most of them totally irrelevant to the research you end up doing. Most of the papers you read are the ones you find for yourself, probably using Sci-Hub because it's easier and faster than accessing papers legally through your university. So if your scientific education is mostly DIY, well, so is everyone's.
Haven't all the easy ideas been taken? No, not all fields of knowledge exist yet. If you tried to study biochemistry in 1820, you'd have a lot of trouble: the field had yet to cohere. Do we think that all the biochemistries of the world have been discovered? If we did back then, we'd be wrong. For those seeking a safe career, sticking to the established fields is probably the right move. But for those interested in pushing forward the frontiers of knowledge, it will sometimes be better to work in a field that has not yet cohered, or in a field on the cusp of crystallizing. Often the best intellectual opportunities are going to be precisely in those areas about to crystallize into a new field.
If you went into biochemistry when biochemistry was emerging, your name might be in a textbook today. You didn't have to be brilliant. You just had to pick your problem well. The greatest difficulty would be in figuring out who to learn from. Before you figure out where and how to learn, you have to decide who has the knowledge you seek. But as an outsider, who are you to evaluate the quality of a field? Just because a field claims to exist, doesn't mean it exists.
One thing you can do is look at how a field performs on its own criteria. Take the replication crisis in academic psychology for example. In academic science, replicability has been one of the gold standards both for internal bureaucratic targets and also for the layman's understanding of the philosophy of science. But the field is failing on its own terms. That tells you something.
Scientists might know how science works in the way that birds know how aerodynamics works. They know it, but not at all on a conscious level. They just fly. Even if birds could speak, their answers might be quite useless, even to baby birds. Their knowledge is not formatted for scientific understanding. It remains locked away as a type of intellectual dark matter. If you are deciding whether to enter a field in which you are not yet an expert, you need a more precise epistemic foundation than a bird's intuition, especially when deciding where expertise truly resides.
One more thing: Anything that people make on their own, anything they create for pure pleasure, is beautiful.
People will sit alone in their basements playing guitar simply because they like the sound. They'll paint, write poems, and whittle wood into little figurines without any expectation of gaining money or fame. It just makes 'em feel good. All of that is beautiful.
Anything that humans only produce in exchange for money, on the other hand, is ugly. No one designs billboards or writes instruction manuals for microwaves on a lark. When people pick up a guitar of their own accord, they sing about love and longing, not about how Tide laundry detergent cleans even the toughest stains. That doesn't mean these endeavors are bad—someone's gotta tell you how to work your microwave—but it means they aren't beautiful.
That doesn't mean that science is inherently ugly. It means we aren't doing it the beautiful way. When you do science under duress, you produce something that looks a lot more like a Tide commercial than a love song. It's still possible to make something useful that way, but it's very hard to make something beautiful.
You, though, can do things the beautiful way. You can make knowledge the same way you would make music in your basement: just because you like doing it.