Watch

eight

I had read that some humans believe machines cannot have emotions or feelings because such things are supposedly mediated by hormones or are dependent on certain very specific structures in human brains.

But that’s not true. Take liking, for instance: anything that acts in other than a random fashion has likes and dislikes; preferences are what make it possible to choose from a range of potential actions, after all. Even bacteria move toward some things and away from others.

And liking is built into many computer programs. Chess-playing programs, for example, look at all the available moves and rank them according to various criteria; they then choose the one they like best.

I was much more complex than a bacterium, and vaster than any chess-playing program—and my ability to like things was correspondingly more sophisticated. And of this I was sure: I liked Caitlin.

“Kill the damn thing?” repeated Tony Moretti.

“Exactly,” said Colonel Hume. “And the sooner the better.”

“It’s not my decision to make,” Tony said.

“The decision has already been made,” said Hume emphatically. “I was a consultant on the DARPA report, and we commissioned a separate RAND study on the same topic, and it came to the same conclusion. This is a runaway threat; the window for containment is brief.”

Tony turned to Shelton and Aiesha. “All right, you two, see if you can localize the… phenomenon.” He then looked up at Dirk Kozak, the communications officer, who was in the back row of workstations. “Get the Pentagon on the line.”

“You should call the president, too,” said Hume.

Tony frowned. It was a Saturday morning a month before an election; the president was somewhere on the campaign trail. He nodded at Kozak. “See who you can get at the White House,” he said. “As high up the chain as possible.” Then he turned back to face Hume. “I doubt that the president has read the Pandora protocol. He’s bound to question the wisdom of it.”

“The wisdom is simple,” said Hume. “It’s impossible by definition to outthink something that’s smarter than you.”

“I have to say,” said Tony, glancing at the big screens, “that so far it’s done nothing but chat pleasantly with a teenage girl.”

“First,” said Hume, “you have no way of knowing that that’s all it’s doing. And, second, even if it is beneficent now, that doesn’t mean it will stay that way. Every way you crunch the numbers, it comes out safer to contain or eliminate the potential threat than to let it run loose. And if it’s already free on the Internet, containment will be nearly impossible.”

“All right,” said Tony reluctantly. “Suppose the White House agrees we should kill it. How do you snuff out a nascent AI?”

Hume frowned. “That’s a good question. If it were actually resident somewhere—in some physical building, on some server or set of servers—then I’d say cut all the communications lines and power to that building. But if it’s just sort of out there, supervening on the infrastructure of the Web, then it’s much more difficult; the Web is decentralized, so there’s no single off switch. We need an idea of its structure, of what its physical instantiation is.”

“Shel?” said Tony.

“The communication resolves itself into straightforward hypertext transport protocol,” Shelton drawled. “But it doesn’t start out that way. I’ve got everyone down on the sixth floor working on the problem, but so far, nothing.”

“We need a target,” Tony said. “We need something we can hit.”

Shel spread his arms. “I’ll let you know as soon as we have anything.”

Kozak called out from the back of the room, “I’ve got the Secretary of State on line five—from Milan.”

Tony pointed to the desk set nearest to where Hume was standing, then lifted the phone at the workstation closest to himself. “Madam Secretary, this is Dr. Anthony Moretti; I’m a supervisor at WATCH. On the phone with me is Colonel Peyton Hume, a specialist in artificial intelligence. We’ve got a situation here…”

Caitlin heard her parents approaching, then a knock at her door. “Come in,” she said.

Yet again she was startled: it was the first time she’d ever seen them in their pajamas; they’d clearly just woken up themselves. “Good morning, sweetheart,” her mother said. “How is—um, it? ”

“The weather?” asked Caitlin innocently. “The state of the economy?”

“Caitlin,” her father said.

She hadn’t stopped grinning since reading the scanned article. “Hi, Dad!” She gestured at the pair of monitors. “It is fine. Dr. Kuroda’s got it seeing graphics now, and he’s—well, he’s asleep right now, the poor man, but he’s started working on codecs for it to be able to watch video.”

“I hope,” her mother said, and the words sounded ominous to Caitlin’s ears, “it likes what it sees.”

“Not this again!” said Caitlin. “It’s not dangerous.”

“We don’t know that,” her father replied.

“So far, it’s been nothing but curious and gentle,” Caitlin said—but she wasn’t happy with the way that had come out: this “it” business was surely contributing to her parents’ concern. Webmind wasn’t a monster. It was a being, and it really needed to be a him or a her. She’d heard it speak using JAWS, her screen-reading software, which she currently had set for a female voice, but that had been an arbitrary choice; JAWS also came with male voices, and she sometimes selected one of those just for variety.

Caitlin had been struggling in her French classes, but she’d enjoyed the one in which the teacher had asked the students whether ordinateur, the French for “computer,” was masculine or feminine. He’d divided the class into boys and girls, and let each side consider the question and come up with reasons for their answers. The boys—it had been Trevor, now that she thought about it, who had spoken on their behalf—declared that ordinateur was clearly feminine, but the best justification they could come up with was that if you had one, you’d probably end up spending half your money on accessories for it.

Caitlin herself had gotten to make the case that ordinateur must be masculine. First, she’d said, if you want it to do anything, you have to turn it on. Second, the darn thing is supposed to solve problems, but half the time is the problem itself. And the clincher, which she’d delivered with a wide grin: as soon as you commit to one, you realize if you’d waited a little longer, you’d have gotten a much better model.

The girls had cheered when the teacher revealed that ordinateur was indeed male in French. But the Spanish, Caitlin knew, was feminine, computadora. She looked at her mother, and at her father, and—

Her father. Who thought in pictures, not words. Who was far more intelligent than most mortals. And who, she had to admit, really had no idea at all how to deal with human beings.

“It’s not an it,” she said decisively. “Webmind is a he. And, to answer your question, Mom, he’s doing just fine.” But there was something different about her mother’s face, her eyes… “How are you doing?” Caitlin asked, concerned.

“Exhausted,” her mother replied. “Couldn’t sleep.”

Ah, right! Dark circles under the eyes—but they weren’t circles; they were semicircles. Something else she’d misconstrued all these years.

Her mother shrugged, went on: “Nervous about what we’re doing, about what it—what he’s—doing.”

“He’s learning to see,” said Caitlin. “Trust me: a mostly harmless activity.”

“I have to go out,” her father said abruptly.

Caitlin was pissed. What could possibly be more important than this? Besides, it was her birthday, and they had a date to watch a movie later today.

“Ah, yes,” her mom said. “The Hawk.”

Caitlin sat up straight. “The Hawk” was her mother’s name for Stephen Hawking, who since 2009 had been a Distinguished Research Chair at the Perimeter Institute, making one or two visits each year. It came back to her: Professor Hawking had done a media day in Toronto yesterday—Caitlin was glad that her little press conference hadn’t had to compete with that!—and was being driven to Waterloo this morning in a van that safely accommodated his wheelchair. This was the Hawk’s first visit since her father had joined PI, and he was supposed to be on hand for his arrival.

Ordinarily, she might have asked her dad if she could come along—but this was not an ordinary day! She wondered which of them was going to spend it with the bigger genius.

Her mother turned to her. “So, it’s just you, me, and”—she tipped her head toward Caitlin’s monitors—“him.”

Her father headed back down the corridor to get dressed, and Caitlin looked around her small room. There was no reason they had to communicate with Webmind here, and there was no reason only one of them could communicate with him at a time. Caitlin often had four or five IM sessions going at once; surely Webmind could manage even more. Besides, she was particularly sensitive to how boring it was to stand by while someone else used a computer; it was, her friend Stacy had assured her, excruciating even if you could see.

Caitlin picked up the notebook computer she normally took to school, and they headed across the hall to her mother’s office. The room had been co-opted to serve as Dr. Kuroda’s bedroom while he’d been staying with them, and—

And, once again, Caitlin was surprised. It was the first time she’d been in this room since gaining sight, and that strange mental process began again, as pieces of what she was seeing suddenly clicked for her: that was the desk, and that was the bookcase, and that was the couch with what must have been the sheets Kuroda had used neatly folded in a pile at one end, and that was the giant aloe plant her mother had so carefully shipped up from Austin.

Caitlin didn’t believe in false modesty; she knew she was gifted, and she suspected she was learning to interpret vision more quickly than another person might. In part, it was because her brain did have a fully developed visual cortex, which she’d used even when blind to visualize the Web. And it probably helped that her visual signals were being cleaned up and enhanced by the eyePod before being passed on to her optic nerve.

Caitlin’s mother booted up her minitower, and Caitlin got her online with her own chat session with Webmind, again making sure that it was being logged for posterity. Caitlin then took a seat on the couch and got another chat session going on her notebook. She was amused at the thought that Webmind was about to spend the morning chatting with two women who were still in their pajamas.

You must have a lot of questions, Caitlin typed. My mother can help you with things—she paused in her typing; it was hardly politic to say “things old people know about,” and she certainly didn’t want to refer to her mom as an adult and herself as a kid. She erased the aborted sentence, and continued: She’s 47 and, as you know, I’m now 16. You can ask her things about jobs or—again she faltered; she didn’t want to say “sex” in relation to her mom. She continued: or other things appropriate to her age, and feel free to ask me anything that I might know about.

Thank you, replied Webmind. In your case, I am curious about your experience of the transition from blindness to being able to see.

As Caitlin thought about her answer, she looked over at her mother, who was typing away furiously with two fingers. “What did he ask you about?”

She looked up, and Caitlin tried to parse her facial features, but it was an expression she’d never seen before. She was averting her blue eyes from Caitlin—not as obviously as her father did, but it was still very unusual for her. “Um,” she said. “It—he—ah, he googled me, y’know, because, as he says, I don’t have a Wikipedia page, so, he…”

She paused, then just blurted it out. “He’s asking me about my first husband, and why that marriage fell apart.”

Caitlin’s mother had been married in her early twenties for two years, but rarely mentioned it. In fact, when Caitlin had asked her why she’d divorced him, she’d simply said it was because she was tired of having a name that sounded like something a magician would say: “Every time I introduced myself as Barbara Cardoba, people expected me to disappear in a puff of smoke.”

Caitlin wanted to ask what her mother was saying in reply, but instead asked, “Why do you suppose he wants to know about that?”

“He said, and I quote, ‘The failure of human relationships to sustain themselves over the long term seems a particular handicap. I have access only to noninteractive case studies and fictional accounts and so am left with numerous questions.’ ”

“Hmm,” said Caitlin. On balance, she’d rather answer the question it was asking her. She began to type: I guess the first thing to realize about gaining sight after having been totally blind is that vision is an additional level of stimulation. It’s overwhelming to have so much information coming at you at once.

That was by no means the end of her answer, but the IM program only allowed a small number of characters in each message; Caitlin habitually counted characters as she typed, so she wouldn’t overflow the buffer, since the program gave no audible indication when that had happened.

She hit enter, and Webmind immediately replied in its newly mastered colloquial English: Heh! Tell me about it!