THE MATHEMATICS OF DIGITAL PICTURES
A Hint of What It Might Take to Make Money Do What We Want

FROM THE NEW YORKER, APRIL 11, 2005

CAPTURING THE UNICORN
by RICHARD PRESTON
How two mathematicians came to the aid of the Met.
Issue of 2005-04-11
Posted 2005-04-04

In 1998, the Cloisters—the museum of medieval art in upper Manhattan—began a renovation of the room where the seven tapestries known as “The Hunt of the Unicorn” hang. The Unicorn tapestries are considered by many to be the most beautiful tapestries in existence. They are also among the great works of art of any kind. In the tapestries, richly dressed noblemen, accompanied by hunters and hounds, pursue a unicorn through forested landscapes. They find the animal, appear to kill it, and bring it back to a castle; in the last and most famous panel, “The Unicorn in Captivity,” the unicorn is shown bloody but alive, chained to a tree surrounded by a circular fence, in a field of flowers. The tapestries are twelve feet tall and up to fourteen feet wide (except for one, which is in fragments). They were woven from threads of dyed wool and silk, some of them gilded or wrapped in silver, around 1500, probably in Brussels or Liège, for an unknown person or persons, and for an unknown reason—possibly to honor a wedding. A monogram made from the letters “A” and “E” is woven into the scenery in many places; no one knows what it stands for. The tapestries’ meaning is mysterious: the unicorn was a symbol of many things in the Middle Ages, including Christianity, immortality, wisdom, lovers, marriage. For centuries, the tapestries were in the possession of the La Rochefoucauld family of France. In 1922, John D. Rockefeller, Jr., bought them for just over a million dollars, and in 1937 he gave them to the Cloisters. Their monetary value today is incalculable.

As the construction work got under way, the tapestries were rolled up and moved, in an unmarked vehicle and under conditions of high security, to the Metropolitan Museum of Art, which owns the Cloisters. They ended up in a windowless room in the museum’s textile department for cleaning and repair. The room has white walls and a white tiled floor with a drain running along one side. It is exceedingly clean, and looks like an operating room. It is known as the wet lab, and is situated on a basement level below the museum’s central staircase.

In the wet lab, a team of textile conservators led by a woman named Kathrin Colburn unpacked the tapestries and spread them out face down on a large table, one by one. At some point, the backs of the tapestries had been covered with linen. The backings, which protect the tapestries and help to support them when they hang on a wall, were turning brown and brittle, and had to be replaced. Using tweezers and magnifying lenses, Colburn and her team delicately removed the threads that held each backing in place. As the conservators lifted the backing away, inch by inch, they felt a growing sense of awe. The backs were almost perfect mirror images of the fronts, but the colors were different. Compared with the fronts, they were unfaded: incredibly bright, rich, and deep, more subtle and natural-looking. The backs of the tapestries had, after all, been exposed to very little sunlight in five hundred years. Nobody alive at the Met, it seems, had seen them this way.

A tapestry is woven from lengths of colored thread called the weft, which are passed around long, straight, strong threads called the warp. The warp runs horizontally, and provides a foundation for the delicate weft, which runs vertically. Medieval tapestry weavers worked side by side, in teams, using their fingertips and small tools to draw the weft around the warp. When they switched from one color to the next, they cut off the ends of the weft threads or wove them into the surface of the tapestry. The Unicorn weavers had been compulsively neat. In less well-made tapestries, weavers left weft threads dangling in a shaggy sort of mess, but the backs of these were almost smooth. Kathrin Colburn recalls that as she and her associates stared into the backs of the Unicorn tapestries it “felt like a great exploration of the piece.” She said, “We simply got carried away, seeing how the materials were used—how beautifully they were dyed and prepared for weaving.” An expert medieval weaver might need an hour to complete one square inch of a tapestry, which meant that in a good week he might finish a patch maybe eight inches on a side. The weavers were generally young men, and each Unicorn tapestry likely had a team of between four and six working on it. They wove only by daylight, to insure that the colors were consistent and not distorted by candlelight. One tapestry would have taken a team at least a year to complete.

The curator in charge of medieval art at the Metropolitan and the Cloisters is a thoughtful man named Peter Barnet. When he heard about the discovery, he hurried down to the wet lab for a look. He got a shock. “The first of the tapestries—‘The Start of the Hunt’—was lying in a clear, shallow pool of water,” Barnet said. The lab is designed to function as a big tub, and had been filled about six inches deep with purified water to bathe the tapestry. “Intellectually, I knew the colors wouldn’t bleed, but the anxiety of seeing a Unicorn tapestry underwater is something I’ll never forget,” he said. When Barnet looked at the image through the water, he said, “the tapestry seemed to be liquefied.” Once the room had been drained, it smelled like a wet sweater.

Philippe de Montebello, the director of the museum, declared that the Unicorn tapestries must be photographed on both sides, to preserve a record of the colors and the mirror images. Colburn and her associates would soon put new backing material on them, made of cotton sateen. Once they were rehung at the Cloisters, it might be a century or more before the true colors of the tapestries would be seen again.

The manager of the photography studio at the Met is a pleasant, lively woman named Barbara Bridgers. Her goal is to make a high-resolution digital image of every work of art in the Met’s collections. The job will take at least twenty-five years; there are between two and two and a half million catalogued objects in the Met—nobody knows the exact number. (One difficulty is that there seems to be an endless quantity of scarab beetles from Egypt.) But, when it’s done and backup files are stored in an image repository somewhere else, then if an asteroid hits New York the Metropolitan Museum may survive in a digital copy.

To make a digital image of the Unicorn tapestries was one of the most difficult assignments that Bridgers had ever had. She put together a team to do it, bringing in two consultants, Scott Geffert and Howard Goldstein, and two of the Met’s photographers, Joseph Coscia, Jr., and Oi-Cheong Lee. They built a giant metal scaffolding inside the wet lab, and mounted on it a Leica digital camera, which looked down at the floor. The photographers were forbidden to touch the tapestries; Kathrin Colburn and her team laid each one down, underneath the scaffold, on a plastic sheet. Then the photographers began shooting. The camera had a narrow view; it could photograph only one three-by-three-foot section of tapestry at a time. The photographers took overlapping pictures, moving the camera on skateboard wheels on the scaffolding. Each photograph was a tile that would be used to make a complete, seamless mosaic of each tapestry.

Joe Coscia said that his experience with the Unicorn tapestries was incomparable: “It was really quiet, and I was often alone with a tapestry. I really got a sense that, for a short while, the tapestry belonged to me.” For his part, Oi-Cheong Lee felt his sense of time dissolve. “The time we spent with the tapestries was nothing—only a moment in the life of the tapestries,” he said.

It took two weeks to photograph the tapestries. When the job was done, every thread in every tile was crystal-clear, and the individual twisted strands that made up individual threads were often visible, too. The data for the digital images, which consisted entirely of numbers, filled more than two hundred CDs. With other, smaller works of art, Bridgers and her team had been able to load digital tiles into a computer’s hard drives and memory, and then manipulate them into a complete mosaic—into a seamless image—using Adobe Photoshop software. But with the tapestries that simply wouldn’t work. When they tried to assemble the tiles, they found that the files were too large and too complex to manage. “We had to lower the resolution of the images in order to fit them into the computers we had, and it degraded the images so much that we just didn’t think it was worth doing,” Bridgers said. Finally, they gave up. Bridgers stored the CDs on a shelf and filed the project away as an unsolved problem.

In 1992, I wrote in this magazine about two mathematicians named Gregory and David Chudnovsky. The Chudnovskys, who are brothers, were born in Kiev. They are number theorists—they investigate the properties of numbers—and they design and work with supercomputers. The Chudnovsky brothers insist that they are functionally one mathematician who happens to occupy two human bodies. Currently, the Chudnovsky Mathematician works at the Institute for Mathematics and Advanced Supercomputing, or imas, which operates out of a laboratory room at Polytechnic University, in downtown Brooklyn. imasis essentially the Chudnovskys.

Gregory Chudnovsky is a frail man in his early fifties, with longish hair and a beard that are going gray, and sensitive, flickering brown eyes. His health is uncertain. He has myasthenia gravis, a condition that he developed in his teen-age years and that keeps him in bed or in a wheelchair much of the time. David is five years older than Gregory. He is a genial man, somewhat on the portly side, with a cultivated manner, and he has curly graying hair and pale-blue eyes, which can have a look of sadness in them.

At the time I wrote about the Chudnovsky brothers, they had built a powerful supercomputer out of mail-order parts. It filled the living room of Gregory’s apartment at the time, on 120th Street, near Columbia University. Gregory was living there with his wife, Christine, who is an attorney at a midtown firm, and his mother, Malka Benjaminovna Chudnovsky. (She died in 2001.) David lives on the Upper West Side with his wife, Nicole, who works for the United Nations. The Chudnovsky brothers were using their homemade supercomputer to calculate the number pi, or ?, to beyond two billion decimal places. Pi is the ratio of the circumference of a circle to its diameter. It is one of the most mysterious numbers in mathematics. Expressed in digits, pi begins 3.14159 . . . , and it runs on to an infinity of digits that never repeat. Though pi has been known for more than three thousand years, mathematicians have been unable to learn much about it. The digits show no predictable order or pattern. The Chudnovskys were hoping, very faintly, that their supercomputer might see one. However, the pattern in pi may be too complex and subtle for the human mind to grasp or for any supercomputer to find. In any event, the supercomputer used a lot of electricity. In the summer, it heated Gregory’s apartment to above a hundred degrees Fahrenheit, so the brothers installed twenty-six fans around it to cool it down. The building superintendent had no idea that the brothers were investigating pi in Gregory’s apartment.

While this was going on, neither of the brothers had a permanent academic job. They were untenured senior research scientists at Columbia, and were getting along on grants and consulting fees, and their wives were also contributing to the family income. Their employment problem was complex: they are a pair, yet they would need to fit into a math department as a single faculty member. In addition, they use computers, which some mathematicians regard as unclean. And Gregory is unable to live anywhere except in a room where the air is purified with hepa filters. (He suffers from allergies that could prove life-threatening.) He would require special care and arrangements by a math department, and it wasn’t clear how much teaching he’d be able to do.

Shortly after my article was published, the Chudnovskys were approached by a man named Jeffrey H. Lynford, who is the C.E.O. of Wellsford Real Properties, a real-estate investment firm. Lynford proposed trying to raise money to endow a chair of mathematics for the Chudnovskys at a university. In the end, after several years of trying, Lynford and his wife, Tondra, gave four hundred thousand dollars to Polytechnic University, and this gift, along with others, was enough to partially endow imas. The job put the brothers on a more stable footing. Gregory and Christine moved to a specially modified apartment that has filtered air, in Forest Hills, and in 1999 they had a daughter, Marian.

At imas, the brothers set about building a new series of computers of Chudnovskian design. The latest of these is a powerful machine of a type called a cluster of nodes. The brothers ordered the parts through the mail. It sits inside a framework made of metal closet racks and white plastic plumbing pipes, and the structure is covered with window screens—those parts of the machine came from Home Depot. The brothers refer to their computer cluster modestly as “nothing.” Alternatively, they call it “the Home Depot thing.” “To be honest, we really call it It,” Gregory explained. “This is because It doesn’t exactly have a name.” They became interested in using It to crack problems that had proved difficult, such as assembling large DNA sequences or making high-resolution 3-D images of works of art.

One day in the spring of 2003, David and Nicole Chudnovsky were having lunch at the Bedford Hills estate of Errol Rudman, a hedge-fund manager and a patron of the Metropolitan Museum, and his wife, Diana. Walter Liedtke, the curator of European paintings at the Met, was there with his wife, Nancy, who is a math teacher. David began talking about digital imagery. Walter Liedtke, who is a Rembrandt scholar, felt a little out of his depth—“I had the illusion that I actually understood it,” he said. “But this was pearls before swine.” Liedtke decided to put David in touch with the Met’s photographers. Not long afterward, David, along with Tom Morgan, a Ph.D. candidate who works with the Chudnovskys, visited Barbara Bridgers in the Met’s photography studio. Bridgers told them, “I have a real-world problem for you.”

David left the Met carrying seventy of the CDs of the Unicorn tapestries. He and Gregory planned to feed the data into It and try to join the tiles together into seamless images of the tapestries. The images would be the largest and most complex digital photographs of any art work ever made, for the time. “This will be easy,” David said to Barbara Bridgers as he left. He was wrong.

"We thought to ourselves that it would be just a bit of number crunching,” Gregory said.

But, David said, “it wasn’t trivial.”

The brothers had a fairly easy time setting up the tiles on It. When they tried to fit the puzzle pieces together, however, they wouldn’t join properly—the warp and weft threads didn’t run smoothly from one tile to the next. The differences were vast. It was as if a tapestry had not been the same object from one moment to the next as it was being photographed. Sutures were visible. The result was a sort of Frankenstein version of the Unicorn tapestries. The Chudnovskys had no idea why.

David, in exasperation, called up Barbara Bridgers. “Somebody has been fooling around with these numbers,” he said to her.

“I don’t think so, David. Nobody around here could do that.”

David informed her that the brothers would need to obtain the complete set of raw data from the Leica camera. The next day, he went to the museum and collected, from Bridgers, two large blue Metropolitan Museum shopping bags stuffed with more than two hundred CDs, containing every number that the Leica had collected from the Unicorn tapestries. There were at least a hundred billion numbers in the shopping bags.

David took the subway back to Brooklyn, stopping off at a supermarket to buy some fruit. In the lab, he put down his things, and Gregory began going through them. “Where are the rest of the CDs?” he asked David. One of the Metropolitan Museum bags was missing.

“My God! I left it on the subway,” David said.

Half the Unicorn tapestries could have been anywhere on the B.M.T. They began frantically calling the subway’s lost and found. “Naturally, there was no answer,” Gregory recalled.

David retraced his route. He found the Met bag sitting under the lettuce bin at the supermarket. Apart from being slightly misted, the CDs were O.K.

Then the brothers really began to dig into the numbers. Working with Tom Morgan, they created something called a vector field, and they used it to analyze the inconsistencies in the images.

The tapestries, they realized, had changed shape as they were lying on the floor and being photographed. They had been hanging vertically for centuries; when they were placed on the floor, the warp threads relaxed. The tapestries began to breathe, expanding, contracting, shifting. It was as if, when the conservators removed the backing, the tapestries had woken up. The threads twisted and rotated restlessly. Tiny changes in temperature and humidity in the room had caused the tapestries to shrink or expand from hour to hour, from minute to minute. The gold- and silver-wrapped threads changed shape at different speeds and in different ways from the wool and silk threads.

“We found out that a tapestry is a three-dimensional structure,” Gregory went on. “It’s made from interlocked loops of wool.”

“The loops move and change,” David said.

“The tapestry is like water,” Gregory said. “Water has no permanent shape.”

The photographers had placed a thin sheet of gray paper below the edge of the part of the tapestry they were shooting. Each time they moved the camera, they also moved the sheet of paper. Though the paper was smooth and thin, it tugged the tapestry slightly as it moved, creating ripples. It stretched the weft threads and rotated the warp threads—it resonated through the tapestry. All this made the tiles impossible to join without the use of higher mathematics and It.

A color digital photograph is composed of pixels. A pixel is the smallest picture element that contains color. The Unicorn tapestries are themselves made up of the medieval equivalent of pixels—a single crossing of warp and weft is the smallest unit of color in the image. The woven pixels were maddening because they moved constantly. The brothers understood, at last, that it would be necessary to perform vast seas of calculations upon each individual pixel in order to make a complete image of a tapestry. Each pixel had to be calculated in its relationship to every other nearby pixel, a mathematical problem, known as an N-problem, big enough to practically choke It. They decided to concentrate on just one of the tapestries, “The Unicorn in Captivity.” Gregory said, “This was a math problem similar to the analysis of DNA or speech recognition—”

“Look, my dear fellow, it was a real nightmare,” David said.

“This is like forensics,” Gregory explained. “If the photographers had touched it, we would have seen it in the numbers. The camera was also moving vertically and horizontally a little bit. This made the sizes of the weaves not quite right from place to place. The camera lens itself distorted it a little bit.”

Two of the tiles on the front of “The Unicorn in Captivity” had an eerie green tinge. While the photographers were shooting them, someone had apparently opened a door leading to the next room, where a fluorescent light was on, causing a subtle flare. The Chudnovskys corrected the lighting by using the color on the back threads as a reference.

“It took us three months of computation,” Gregory said. “We should have just dropped it.”

The final assembly of the image took twenty-four hours inside the nodes of It. Gregory and David stayed up all night and ran It from their respective apartments. In the preceding months, each pixel in “The Unicorn in Captivity” had been crunched through many billions of calculations. That last night, there were billions more calculations. By sunrise, the machine had recaptured “The Unicorn in Captivity” in its entirety. The image was flawless.

One day last fall, my wife and our three children and I went to Brooklyn and paid a visit to the Chudnovskys at imas, which is in Rogers Hall, on the Polytechnic campus. David met us in the lobby. He wore a starched white shirt, dark slacks, and Hush Puppies. We were joined by Tom Morgan, a quiet man in his fifties with blue eyes, gold-rimmed spectacles, and a ponytail. He handed us disposable booties, of the kind worn by medical people in operating rooms. David said, “The booties are for the sake of protecting the floor,” and he explained that the floor of imas consists of digital images embedded in a soft plastic material. Then we went in.

The imas lab is a large, loftlike industrial room, with computer-controlled shades and lights, and filtered air. The lights were dim. The walls are concrete and painted white. The brothers project images on the walls, and they also use the walls as a whiteboard to perform calculations with erasable markers. The walls were covered with scribbles—work in progress. Most of the floor consisted of a vast digital image, in color, showing a hundred and fifteen different equations arranged in a vast spiral that breaks up into waves near the walls—a whirlpool of mathematics.

The equations are a type known as a hypergeometric series. Among other things, they rapidly produce the digits of pi. The Chudnovskys discovered most of them; others were found by the great Indian mathematician Srinivasa Ramanujan, in the early twentieth century, and by Leonhard Euler, in the eighteenth century. On one corner of the floor there is a huge digital image of Albrecht Dürer’s engraving “Melencolia I.” In it, Melancholy is sitting lost in thought, surrounded by various strange objects, including a magic square and a polyhedron, with an unknown number of sides, called Dürer’s solid. The Chudnovskys suspect that Dürer’s solid is more curious mathematically than meets the eye.

Gregory Chudnovsky was half lying on the couch, in his stocking feet, his body extended, facing the figure of Melancholy. His shoes, which were tucked inside surgical booties, had been left on the floor. He wore jeans and a soft leather jacket, and he seemed relaxed. Christine and Marian, who is five, were there. Marian was chattering and running around the lab happily. The effect of the child circling over her father’s swirling equations was slightly vertiginous.

“At first, we were going to cover the entire floor with ‘Melencolia,’ but it made people dizzy,” Gregory said. “It made us dizzy, too. So we shrank it and moved it near the couch.”

Close to the windows stood the cluster of bare computers, sitting inside the frame of plumbing pipes and covered with window screens—It. There was a sound of many small whirring fans running inside It, keeping It cool. (I associate this sound with any room professionally occupied by the Chudnovskys.)

My daughter Marguerite, who is fifteen, wanted to know which of the many equations in the floor was the one that the brothers had used to calculate pi with their previous supercomputer.

“Walk this way,” David said to her. “Now you are standing on the equation.”

She looked down. The equation swooped for a yard under her feet.

At the far end of the room hung two thirteen-foot-tall sheets of cloth, mounted at right angles to each other, which displayed perfect digital images of, respectively, the front and back of “The Unicorn in Captivity.” We walked up to the two pictures of the unicorn. First, I looked at the front. I could see each thread clearly. The unicorn is spattered with droplets of red liquid, which seems to be blood, although it may be pomegranate juice dripping from fruit in the tree. The threads in the droplets of blood are so deftly woven that they create an illusion that the blood is semi-transparent. The white coat of the unicorn shines through.

Then I turned to the back of the tapestry. Here the droplets were a more intense red, with clearer highlights, and they seemed to jump out at the eye. The leaves of the flowers were a vibrant, plantlike green. (There are as many as twenty species of flowers in this tapestry. They are depicted with great scientific accuracy—greater than in any of the botany textbooks of the time. They include English bluebells, oxlip, bistort, cuckoopint, and Madonna lily. Botanists haven’t been able to identify a few; it’s possible that they are flowers that have gone extinct since 1500.) On the front, in contrast, the yellow dye in the green leaves has faded a bit, leaving them looking slightly bluish-gray.

Gregory got up from the couch. David warned him to be careful, and he put his arm around Gregory’s waist, while Gregory leaned on David and put his arm over David’s shoulders. Then the Chudnovsky Mathematician moved slowly across the floor, until the brothers were standing (rather precariously) beside It. David explained that their image of the tapestry was a first step toward making even finer digital images of works of art. He said, “It’s simple to take a picture of a Vermeer, but what you really want is an image of the painting in 3-D, with a resolution better than fifty microns.” Fifty microns is about half the thickness of a human hair. “Then you can see the brushstrokes,” he went on, raising his voice over the whirring of the fans inside It. “You can catalogue the brushstrokes in the sequence they occurred, as they were laid down on top of one another.”

Mathematicians, when they work, engage in intensely serious play. They follow their curiosity into problems that interest them and toward the smell of a solution. After playing with the unicorn, the Chudnovskys moved on.

“What are you doing now?” I asked.

David told me that they were working with I.B.M. to design what may be the world’s most powerful supercomputer. The machine, code-named C64, is being built for a United States government agency. It’s rather like It, multiplied many times over, though nothing in C64 will come from Home Depot. When the machine is finished, it will contain two million processors and fourteen thousand hard drives. It will use two and a half million watts of electricity—enough to power a few thousand homes. Two thousand gallons of water per minute will flow through the core of C64 to keep it cool. If the pumps fail, it will melt down in less than ten seconds.

One day, I went to see the Unicorn tapestries in the physical universe, as distinct from the universe of numbers. It was a quiet winter afternoon at the Cloisters. The gallery where the tapestries hang was almost deserted. When I looked at them, each flower and plant, each animal, each human face took on a character of its own. The tapestries were full of velvety pools and shimmering surfaces, alive with color and detail. In the fence that surrounds the captive unicorn, tarnished silver, mixed with gold, gleamed in the grain of the wood. In comparison, the digital images, good and accurate as they were, had seemed flat. They had not captured the translucent landscape of the Unicorn tapestries, as the weft threads dive around the warp, or the way they seemed to open into a world beyond the walls of the room.

Timothy Husband, the curator of the Cloisters, walked in. He is a tall, polished man in his late fifties, and has been at the Cloisters for thirty-five years. We sat down in one of the window seats facing the tapestries. “There is a luminosity and depth in them,” he said quietly. “It didn’t come about by chance on the part of the weavers.”

I asked Husband how he felt when he was alone with the tapestries.

“That happens on Mondays, when the Cloisters is closed,” he said. He spends anywhere from a minute to an hour with the tapestries. “It can be an exceedingly frustrating experience. One ponders so many questions about the tapestries for which there are no more answers today than there were when I was in graduate school.” In some of the scenes, the unicorn may represent Christ. Alive and chained to the tree, after its apparent death in the hunt, it may speak of the immortality of the soul. Or the drops of blood may represent the pains of love. The truth is that the modern world has lost touch with the meanings in the Unicorn tapestries. “Sometimes I come in here and try to pretend I have never read anything about them, never heard anything about them, and I just try to look at them,” Husband said. “But it’s not easy to shed that baggage, is it? And my other reaction, sometimes, is just to say, ‘To hell with it, someday someone will figure them out.’ And then there is a solace in their beauty, and one can stare at them in pure amazement.”

Use back arrow or close the browser.