I expect scientists to view the title of this article with scorn. "The Magic of Science" indeed. How about the drudgery?

Well, I know about the drudgery. But what use would it be without the magic?

For it is not the laws of physics that make science possible but the unprovable proposition that there exists a grand design underlying the physical world. And not just any old "grand design" but one that is accessible to the limited senses and modest reasoning powers of the species to which we belong. Scientists subscribe with such conviction to this article of faith that they are willing to commit a lifetime to the pursuit of scientific discovery.

It is hardly surprising that an activity so magical is also undefinable. Science is what scientists do. And what they do is look around themselves for messages written in the sky, the earth, the oceans and all living things – messages that tell of the unity of creation.

These messages have been there – unseen, though at times written in letters miles high – since the dawn of history. But we have just passed through an epoch in which, quite suddenly, scientists seem to have learnt speed reading. Discoveries have been coming at an unprecedented pace.

In the wake of such a period it is common to consider that we may be approaching the point where all that is readable in nature will have been read. We should be skeptical of such claims. Success in reading some messages brings with it a temporary blindness to others. We forget that between the words written in black in nature's book there are likely to be messages of equal importance written in white.

It is a truism that success in science comes to the individuals who ask the right questions. At a more fundamental level the questing eye acknowledges as messages only the symmetries that it recognizes, in the same way that a word is distinguished from a scrawl. If we teach our minds to recognize new symmetries, we can see messages where previously we saw none. The empty page that the day before we would have consigned to the wastepaper basket will become a treasured communiqué.

A number of such events are in process at any time in the history of science. Let me mention one in which an illegible scrawl is becoming a decipherable message.

This new development is the archetypal discovery, seen at the grotesque but wondrous instant of its birth; the most magical moment in science, as in life. The emerging field is known as the science of "chaos." It stems from the dawning perception that a great number of seemingly irregular events in nature have an underlying rhythm. Since most of nature is irregular (the shape of a cloud, the fall of a leaf), by our failure to incorporate the irregular into the subject matter of science we have been relegating the greatest part of nature's messages to the trash heap. Or, rather, we did until a couple of decades ago, when scientific thinking began to change.

How did this change of view occur? First let me stress how it did not occur. It did not come from planning bodies charged with the dispensation of public funds or from the collective decision of the scientific community. The new flowering of science came, as is usual, from the lonely work of a few eccentrics who found their way to one another through the network of global scientific gossip. Edward Lorenz, Stephen Smale, James Yorke, Robert May, Benoit Mandelbrot, Mitchell Feigenbaum, Albert Libchaber, Michael Barnsley and John Hubbard are the names of some of these explorers. The majority work, as is natural, in the United States, since America remains the world capital for science – a title it has held, however, only for a brief half-century.

While counting our blessings, we Canadians should rejoice in the fact that we share this continent with the boldest and best scientists alive. If we want to encourage these people to venture north of the border – and we surely should – then we must take note of the conditions that make possible scientific achievement. I shall not dwell on the obvious, which has to do with the physical resources at one's command. A less obvious one is equally important. It is necessary to belong to a community that encourages daring and does not begrudge the individual the freedom to dare.

In considering the birth of this new field we encounter a heterodox group of mathematicians, physicists, chemists and biologists. For it is the unorthodox who, by definition, will be the ones to surprise us. And discoveries of note, also by definition, are surprising.

The fact that these people took as their subject matter the incoherent stammerings of nature, which others had rejected as meaningless, should occasion no surprise. Until they are deciphered, the jottings of the Creator are invariably gibberish, and the spectacle of grown individuals poring over them has through the ages been a subject for ridicule; in our own century it has become additionally a focus of indignation. "Why, with so many vital problems crying out for solution, should we tolerate these meanderings down the byways of science?" it is asked.

Charles II of England, who more than three centuries ago was among the first rulers to sponsor science, was also prominent among those who mocked it. When he heard that the Royal Society of London was studying vacuum, he asked caustically why these gentlemen should be encouraged to waste their time, quite literally, on nothing. It seemed axiomatic to him that so long as one did not understand vessels that were full, one should not devote time to studying those that were empty.

But he was wrong. The answer to those who attempt to block the progress of science down what appears to their untutored eyes to be byways is that no explorer in history has ever discovered a new land that was already served by a highway.

Before I set aside this important question of the geography of the continent of science, let me say that I am not arguing for a wholesale diversion of science down byways. That, too, would be folly, since most byways lead nowhere. In science there are no such simple guidelines. All I suggest is that we should recognize the vital role that the freedom to roam plays in maintaining the vitality of science.

When I say "to roam," I do not mean to roam at will, but with purpose. The difference is evident to the trained observer as it may be to the layperson in what I am about to describe.

I have arbitrarily chosen to begin this story, taken from the history of contemporary science, in 1960, when Benoit Mandelbrot, a mathematician at IBM's fundamental research laboratory in New York State, chanced upon some interesting data concerning fluctuations in the price of cotton over time. (A highly readable account of these events is to be found in James Gleick's book Chaos, published in 1978.) Stories in science have, in truth, no clear starting point, since the seeds of new ideas are everywhere. They frequently grow as far as one discipline or epoch will allow and then remain in suspended animation until the development of some other branch of science provides the soil in which the can take root once more.

Myriad treasures, only crudely labelled as to worth, lie upon the shelves of scientific history until such time as they are required. The analogy with the propagation of plant life is appropriate. The growth of science must be organic. Those who suppose otherwise end by despoiling the intellectual environment.

Benoit Mandelbrot first glimpsed the pattern of fluctuating cotton prices by chance on a blackboard in the office of Hendrik Houthakker, a professor of economics at Harvard University – but not entirely by chance. Houthakker had, after all, invited Mandelbrot, a mathematician to give a seminar. And, again, it was not by chance that Mandelbrot paid more attention to the intricate curve of cotton prices than to other items in Houthakker's office. Mandelbrot's mind has been prepared by his studies of recurrent complex patterns; he also held the conviction that the time was ripe for a broad new synthesis and an equal conviction that Benoit Mandelbrot was the man for the job.

The field in which he was destined to play a major role is breathtakingly broad in its scope. Had Mandelbrot's eyes happened to lock on other items in Houthakker's office – an early map of Cambridge perhaps, a coat rack or a potted plant – he could also have considered applying his theories to any one of them. Wisely, however, he chose the fluctuation of cotton prices noted on the blackboard. Though random to the majority of observers, they conveyed hints of a message to Mandelbrot.

In the conventional analysis of random fluctuations, small changes are unconnected with large ones. Chaos reigns. But Mandelbrot and a group of pioneers, some of whom I have just referred to, had developed the ability to recognize a new type of order in which large, complicated fluctuations precisely mimicked the shape of small, equally complex ones.

This "self­replication" of complex patterns on ever smaller scales is now thought to be a property of a multitude of nature's most important doodlings. The doodler, of course, works in the reverse direction, starting with the elemental shape and then elaborating it through endless repetition. But the shape of the whole is the same whether you read it from large to small or small to large.

Shapes that exhibit this feature of self-replication Mandelbrot called "fractals"; break them up and you are left with smaller replicas of the original. This type of symmetry has long had an appeal; what it lacked until recently was a sufficient basis in mathematics. The aesthetic attraction is evidenced, for example, in the notion, dating from before the advent of modern microscopes, that the male spermatozoa carries within itself a replica of the individual who gave rise to it. Open up a human, it was proposed, and you will find within a further human. Of course the human sperm does not contain a recognizable representation of the entire individual but carries it in the form of a coded molecular message.

If doodles have so much to teach us, there was never a better time to doodle. Computers, which endlessly repeat boring tasks, are consummate doodlers. Computers have opened the door to mathematical operations that in earlier times would have reduced the whole of mankind to slavery. Mandelbrot, from his office at IBM, was in an especially favourable position to instruct his computer to repeat a very simple operation a vast number of times. The operation he programmed into the computer was "square a number and add it to itself." The number was simply the result of the previous identical operation. Of course the cycle had to be set in motion by choosing an initial number, corresponding to the first squiggle that the bored doodler makes on the back of an envelope.

But what is the point of an exercise such as mathematical doodling? It has a simple point, namely that it is an intellectual adventure into a region of the physical world as real as any other. Since, however, it takes us into a part of the world into which we would not normally penetrate, it extends our ability to picture and therefore to think.

Our much vaunted intuition, which seems at times to guide our thinking in an almost magical fashion, operates by recognizing from among the confused jumble of sensations those data that fall into patterns that we have previously encountered. Mathematics is a vital means to recognize new patterns and hence to the further education of our intuition. That, too, is what science is about. It is an economical and comprehensive description of the symmetries that go to make up our world.

The beautiful fern-like shape that accompanies this article was generated by a computer following a simple prescription for doodling. The shape, as you would expect from the fact that it was traced by endless repetition of a simple form, is a fractal – if you break off a piece it will, on a small scale, resemble the whole. The resemblance to a fern is so marked that one is left wondering whether the seed of a fern encodes the genetic instructions for forming its fronds in the same manner as does this fractal equation. We do not know. Surely nature must put a premium on compact codes, and it is unlikely that one will find a more compact rendition of the shape of a leaf than is to be found in this simple set of instructions.

This new field of mathematics is tantalizing, not merely because fractals constitute an entrancing mathematical toy but because they may address profound questions concerning the structure of our world. The most sweeping name for the new field is "chaos theory," an apt name since the field hinges on the discovery that simple mathematical procedures can have vastly intricate and at times chaotic outcomes, or, conversely, that seemingly chaotic behaviour may have simple origins.

Joseph Ford of Georgia Institute of Technology, a 30­year veteran of this young field of research, has claimed (in a 1989 issue of the magazine Science) that this new math constitutes " the beginning of a major revolution. The whole way we see nature will be changed." His belief (and he is one of a number who share this view) is that the vast range of natural phenomena that we have virtually ignored in the past, phenomena that neither are smooth in outline nor evolve smoothly with the passage of time, will become amenable to systematic study.

It is interesting to notice how this shift in attention on the part of a substantial segment of the scientific community has occurred concurrently with a major change in aesthetics and in world view in society at large. I mention it in order to make the point that science does not have a "magic" that is all its own, but takes its place within the broad context of cultural history.

In the early part of this century, painting, architecture and music went in search of the daringly simple. A canvas, a building or a musical composition could be based on a smooth surface, a straight edge or at times a single note. In this, art was paying homage to the enormous power of such simple scientific concepts as linear extrapolations from the known into the hitherto unimagined. Art was deliberately distancing itself from nature and celebrating the perfection of the machine.

There followed a cultural counter-revolution in which the machine came to be associated with the exploitation of science for hazardous and, at times, evil ends. Orwellian tyranny, to cite an important example, was based on high tech, as was the rape of the environment. Humanity and nature were seen to be suffering at the hands of science.

The machine was dethroned, along with many oppressive rulers who bore an uncanny resemblance to Big Brother. Architecture moved, in response, from rectilinear purity in the direction of the timidly postmodern. Trees and even human beings occasionally strayed into paintings. Composers listened, albeit somewhat critically, to the music of birds. The physical sciences witnessed an exodus to biology. And in the midst of this, not entirely coincidentally, mathematicians discovered the equation for a leaf.

Let me approach this same point from a slightly different angle. I think that the illustration of the scientific method as it applies to chaos theory properly stresses the important element of play in creative activity. But any other growing point of science or the arts would exhibit this same vital element. We need to play because logic is such a feeble and restrictive tool. Play liberates the imagination, throwing truly surprising elements into the air. The trick – and it is a very difficult trick – is to pick the significant from among the trivial in the grab bag of novelties brought to light by play. This requires that the creative individual – scientist or artist – be alert to subtle symmetries. We can find only what we are looking for.

What we are looking for, if it is to have importance, should be based on insights that extend beyond the corner of the laboratory or studio where we engage in our work. We are guided in our assessment of what is or is not significant by all that has been revealed to us in every branch of contemporary culture. Whether we are physicists or painters, mathematicians or musicians, we are children of our times.

It is commonplace for the mathematicians to scale a height that proffers a remarkable panoramic view but that does not face the contemporary battlefield. It even happens that the mathematicians get bored and move on, to be followed by other mathematicians who, coming from other directions, unknowingly scale the same heights. Finally, the first physicist or chemist trudges up the by now well-worn path exclaiming gleefully at his discovery, under the impression that he is making the first ascent. The mathematicians lose no time in disabusing him of that idea.

This criss-crossing of the broad swath of territory situated along the forward path of science sounds more disorganized than it is. Certainly it is self­organized, like the confluence of tributaries in search of the main stream. Each scientist senses the contours of the physical world she or he is exploring. Each is at the same time vitally concerned to conserve effort, seeking to follow the easiest path while always checking to see that he or she is in fact moving forward. Each is, therefore, alert to what others may be doing, since all wish to be first at the finishing post, there being little to be gained by being second or fifth.

No one but an all­seeing deity could improve on the organic pattern of the explorers' interactions with the terrain and with one another. Governments nonetheless continually attempt to direct the traffic in this most subtle of operations, a voyage across unknown territory to an as yet undiscovered destination. And governments continually fail, since, although they feel themselves to be all­seeing, they in fact suffer from a crippling form of myopia, a result of their distance from the scene of action.

Of course, there is waste in the process of discovery as pursued by free agents whose only guideline is that they must succeed, cheaply and rapidly. The dismay that this waste causes in high places is nicely captured by the story of an interview that Josef Stalin is reported to granted to the Soviet Union's leading film director. "How many films did you make in the past year?" Stalin enquired. "Eighteen," the reply. "And how many of those would you judge to have been truly successful?" Stalin went on. "Three, perhaps," hazarded the eminent director. "Well, next year," said Stalin firmly, "just make three."

The marketplace is also wasteful, and nature is quite notoriously so, but, as yet, governments have failed to come up with an acceptable substitute for either. This is not, unfortunately, for want of trying.

Governments should manage, on behalf of the governed, those things that are manageable. The proper use of science is such a thing. The sorcerer's apprentice must be prevented by all means possible from opening sluice gates that he cannot close. But the reservoir of ideas that he taps into is supplied by a process – the magic of science, if you like – that governments can only impede.

Occasionally the magic of science is palpable. Not long ago I sat in my laboratory between two students at the computer screen of a scanning tunnelling microscope, an instrument capable of sensing atom-sized bumps on a surface, which they had just finished constructing. It was the culmination of a lengthy effort. Though seven years have passed since Gerd Binnig and Heinrich Rohrer were awarded the Nobel Prize for developing the first such device, tunnelling microscopes that work remain sufficiently rare that their owners constitute a select club.

Following instructions keyed in by the more senior student, a needle of atomic sharpness moved its way to and fro across the surface of a semiconductor crystal mounted at the centre of the ultrahigh vacuum chamber. Throughout this operation the needle hovered at a distance comparable to the diameter of a molecule above the crystal. To ensure that the tip was sufficiently steady in its motion, it was necessary to isolate it from the outside world. The needle was therefore suspended from a set of springs at the centre of its housing. The housing, a steel container weighing some 300 kilograms, floated on air.

Despite these precautions, the three of us, crowded together in front of the computer screen, were obliged to sit very still. We communicated in hushed tones, so that the sound of our voices would not disturb the needle. Our reverent demeanour was rewarded by the slow emergence on the screen of some freckles and ridges. The freckles were the individual atoms on the semiconductor surface, and the ridges were atomic scale imperfections resulting from the fact that when a crystal is formed, one row of atoms often fails to fit precisely into the adjacent row.

Even the most perfect crystal, it turns out, carries what at first may be dismissed as blemishes but later comes to be recognized as the fingerprints of creation.

As for the magic, it is not in the fingerprints, since nature does not partake of the supernatural. It is in our perception and hence in the pursuit we call science.