​The Counselor

Today's Terraform dispatch is a special event: We've teamed up with the think/do tank Data & Society to help them unleash their "Intelligence & Autonomy" project. It's a pursuit perfectly suited to Terraform; Data & Society has tapped some top authors to use speculative fiction to investigate and explicate the real-world issues it researches. Tim Hwang and Madeleine Elish explain:

"Intelligence & Autonomy is a research initiative focusing on the policymaking challenges facing the rising influence of intelligent systems—technologies which combine machine intelligence and automation. The project takes a comparative approach, seeking to explore issues surrounding AI by examining experiences across domains and looking historically, as well. It is supported by the John D. and Catherine T. MacArthur Foundation.

In March 2015, I&A commissioned authors to write a series of narratives that investigated near future concerns around intelligent systems in warfare, urban design, medicine, and labor. These stories served as the centerpiece of a two-day intensive forum bringing together participants to identify the core set of challenges that consistently arise in deploying intelligent systems regardless of arena. "The Counselor," by Robin Sloan (Mr. Penumbra's 24-Hour Bookstore) focuses on the persuasive qualities of these systems in the medical context.

My body was shutting down but nothing hurt except the bridge of my nose, which was sore from a long morning under the glasses. I had once again been courting the miller's daughter in the village game where everything blushed and glowed. It was a gentle cartoon. Every morning I fell in love, and every afternoon I felt like a fool. Who but a fool would waste his last days on empty entertainment?

Except they never did seem to be my last.

"Counselor!" I spoke. I imagined it a roaring invocation; it came out a croak, my throat clogged with spit and other miscellaneous fluids. My first speech of the day.

The counselor arrived. First the annunciatory hiss, then the voice, emanating from speakers built into the support bed. It rattled my guts, made the flesh on my neck jiggle.

"Tell me," the counselor said, "what's changed since the last time we spoke?"

Always the same opening gambit. Words I myself had chosen, fifty years ago. Now they annoyed me.

"Nothing's changed," I snapped. "No miracles since yesterday."

"What did you eat last night?" The counselor's voice warbled, the words stitched together. It sounded antique.

"Pudding. That nice rice pudding they make here." I was lying; I couldn't remember. Yesterday had been a bad day, and then I'd been afraid to fall asleep. I remembered it only abstractly now; terror under glass.

"Your weight has gone down."

"I suppose I should watch that."

The counselor paused. "I wouldn't worry about it."

By the terms of my end-of-life care agreement, I had to speak to the counselor for thirty minutes every day.

"How much longer?"

"Twenty-nine minutes."

Long ago, they'd struggled to name us. Generation Y? The Millennials? The Unsettled? Only in our senescence did an identity adhere: the Undead.

The Economist ran a cover showing a field of half-buried figures, gray and desiccated, stubbornly reaching for life. It was repugnant. I wrote them a scolding letter, signed it with all the pomp I could muster: PAUL GOMEZ, DIRECTOR (EMERITUS), UNIVERSAL COUNSELOR PROGRAM, U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES.

The magazine declined to print it.

We were cursed by the timing of our birth: too soon for the new regime—the zero-year therapies, the tiny balls in your blood—but late enough that when we were old and worn out, our lives could be extended almost indefinitely, if only at incredible cost.

And so they hated us, the great "they" out there; the young ones, the healthy ones, my children and their children, all waiting to be rich. Everything else in their world was going great. The dark prophecies of my youth had evaporated in the sunny resilience of the actual non-apocalyptic 21st century. Other places had problems, big ones, but the United States was stable and strong and very, very wealthy.

Or it would have been, if we grasping Undead hadn't been wrecking the budget.

I knew the statistics by heart; learned them before I became them. We spent a quarter of our national income on health care. The proportion had crept higher, finally, than defense. The United States spent too much on killing and dying, the pundits intoned, and not enough on living.

When the counselor program began, the figure had been 22 percent; well beyond crisis. The fact that it was now 24 percent could be read, perhaps, as failure, but I knew better, because I knew how to read a cost curve. The people who had grown up with counselors, the generation roughly following mine—they were comically robust. They were 50 and 60 now, and their segment of the cost curve was slender, almost elegant.

It was at 80 and 90 that the curve bulged grossly. That bulge was me.

"How do you feel today?"

"Good," I said, and it was the truth. My head was clear. The black dog of depression wasn't entirely absent, but he was sniffing around in the distance, not sitting on my chest. "I feel good."

"Paul." The counselor paused. "Wouldn't you prefer to end your life on a day like today?"

"No, counselor," I said, with ice in my voice.

"Lately," it offered, "there have been more bad days than good days."

I scowled. "Do you really—agh." A sharp pain danced across my belly, slithered around my side. My thumb hovered over the support bed's pain relief trigger. I took a moment to gather myself. "Do you really think today is going to be the day, counselor? Do you think I'll decide, just like that?" I tried to snap my fingers as punctuation, but I couldn't produce enough pressure between the pads to make a sound. "Don't be absurd."

"What about tomorrow?"

It spoke so plainly I had to laugh. "How much longer?"

"Twenty-three minutes."

I was clear-eyed about our situation, all of us sad old game-players in our support beds. We were not, in fact, the immortal undying; we were the asymptotically unwell. Zeno's zombies! (The first time I thought of that, I laughed so hard I started to choke, and the support bed had to take over my breathing.) The arrow would eventually hit its mark. Of course it would. We knew that. I knew that.

Some days I wished I had been struck by an actual arrow, or a bullet, or a train. That would have been a gift: an accidental, instantaneous death, with no decisions, no attendant dread. I had determined, in retrospect, the ideal moment. It would have been in November, six years ago. A day cold and bright, perhaps the Saturday before the stroke, which happened on a Tuesday. (I don't remember all this. The counselor does.)

But it was too late for trains. I was ensconced in the support bed—a space so safe, so kind to human life, that it made the womb look like Death Valley—and it was utterly opposed to the counselor's coaxings. I was trapped in a vise-grip between two very opinionated technologies. If I died the support bed's way, it would be a drawn-out, drug-fueled showdown. If the counselor's, it would be my choice.

Day after day, the counselor prodded: couldn't death be orderly, dignified, graceful?

Day after day, I replied: certainly. Just not today.

Counselors played the long game.

That was the big idea that brought us together, fifty years ago, in the beige building on Catherine Street in Ann Arbor. The Michigan Counselor Pilot Project had a staff of fourteen researchers, most of us from the School of Public Health, some from CSE, but all of us disciples, in one way or another, of Professor Agnes Green.

When it comes to a person's health, all the important behaviors unfold on a scale of decades. The only place the HMOs tracked their patients with that sort of continuity was in advertisements. Real people, they missed their checkups. They switched jobs. They bought a van and drove to Mexico. Their insurance sputtered like an old lawnmower.

What if there was a counselor who could follow them through everything?

It was Agnes Green (Michigan class of 1962, Ph.D in '67) who dreamed it up. She spent most of the '80s banging the drum for a national call center staffed with long-term health counselors. It would have been bigger than the Pentagon. When the internet came around, she was certain the technology to execute her plan had arrived, but the math was still intractable: with 250 million adults to be counseled, there was no human staff that could handle it—not at the skill level that was required.

Or, there was, and it was called the health care system.

Her big idea was a non-starter, but Agnes Green wrote about it with vivid urgency until her death. She was an idealist who believed in the power of public policy on the grandest scale. To make her points, she often employed the language of myth. What if, she wrote, every citizen could rely on the intervention of a counseling angel?

The angel came. It was AI.

If Agnes Green had lived to be as old as me, she could have joined us in Ann Arbor.

By the time we gathered on Catherine Street, none of it was high tech anymore. We bolted a natural language interface to a medical database, then bootstrapped it with a thousand hours of recorded doctor-patient conferences. Undergraduates did the programming. We trained the system on a pilot population of ten thousand in Ann Arbor and Ypsilanti. For five years, we watched and tweaked and sometimes even stepped into the conversations ourselves.

Do you know what our great contribution was? The pauses.

Humans hate silence, but computers don't mind it. We taught the counselor to pause; to make space; to let people fill it.

When you ask a person "what's changed since the last time we spoke," the first response is not the truth.

You pause.

You wait.

"Actually. Now that I think about it…"

The truth comes.

The pilot project was a profound success, with double-digit improvements across every major health indicator. We went statewide, then to other states. Then came the national rollout, and in a stroke so grand I'm surprised Agnes Green didn't stand up and cheer in her grave, the government required every American to talk to their counselor before filing their taxes. There was a credit: fifty dollars—that was decades ago; surely it's more now—for one conversation by phone or internet. Thirty minutes was the minimum, but the average conversation lasted two hours. We hadn't foreseen that. People talked to their counselors about everything; not just health, but relationships and money, dreams and plans. The counselor listened, asked questions, and locked it all away in an encrypted vault. It became a joke, a meme: Oh, tell it to your counselor! People did.

The counselor asked, "How's the support bed?"

The support bed was extremely comfortable—a mesh of gelatinous nodes that flexed and moved, an everlasting massage in slow motion. No bed sores in the late 21st century. When required, the nodes could part like curtains and clear the path to a gleaming, wide-mouthed waste receptacle.

"It's awful," I said. "How long has it been since I got up?"

I could go roving in a fancy wheelchair, but not without tremendous assistance and significant existential risk. A sick 95-year-old leaving his support bed was like an astronaut on EVA: the darkness was close, and mistakes could compound quickly.

"You've been in bed continuously for fifty-six days," the counselor replied.

Its memory was perfect. Sometimes I tested it.

"What was I worried about twenty years ago today, counselor?"

"Your testicles," it replied matter-of-factly.

I didn't remember that, but neither did I doubt it. A phantom lump; an errant wince. I was sure I'd mentioned my testicles to the counselor many times, as well as my stomach, my gallbladder—or the place where I imagined my gallbladder to be—and my heart, of course. I was sure I'd prodded the counselor to list symptoms, statistics, mortalities. It always complied, in precisely the register we had engineered: "wise advisor," tuned to a midpoint between the weary skepticism of a doctor and the morbid enthusiasm of a search engine.

Asking a counselor about your fears was far, far better than asking a search engine.

I talk as if I have any idea how the counselor works now. I stopped working on it decades ago. It's probably all different.

The voice is the same. Everyone's attached to it now. The English version is based on the voice of a woman named Alma Washington Gray. It's resampled almost beyond recognition, but I remember the original. I stood on the other side of a wide glass window, wearing headphones, and I led her through the whole dictionary, and then dictionaries beyond. Cholesterol. Hypoglycemia. Sphenopalatine ganglioneuralgia.

"Wouldn't you prefer," the counselor said, in its simulation of Alma's voice, "to control the manner of your passing?"

"My death." I corrected it every time. "Don't say 'passing.'" I was opposed to euphemisms. "Say my death. Better yet, say 'the end of my existence.' Would I prefer to control the end of my existence? No, counselor. I would prefer not to pull that particular trigger."

"There will be an ending, Paul. You're very smart. You know that."

The counselor was a flatterer. That was another contribution. Flattery, as a public health intervention, was very, very powerful.

"Of course," I said. "But I have good days left. They're precious to me. Like today—I feel fine! I'm even enjoying talking to you." The counselor was not perturbed by my jab. "So let it wait."

"For every day you identify as good, you have four you identify as bad."

"One in four is a treasure. Marco, my friend Marco, he would have loved one day in four. He had one in a hundred. If that."

"What if you die like Marco?"

I had spoken to the counselor about Marco's death, and now I regretted it.

"That would be different," I said, "but I'm not at that stage yet."

The counselor was silent. I'd taught it well.

Many people spoke to their counselor in religious terms, and the counselor was able to respond in kind. It could, for example, suggest to a person in my position—a 95-year-old dying surely but slowly—that he might meet his old friends, sorely missed, on a celestial picnic blanket. The counselor could entice them with that. Of course, it was just repeating back what it heard, but even so, I hated it. I'd tried to ban God from the counselor's vocabulary back when we built it, but I had been overruled, forcefully.

"Every hospital has a chapel!" someone shouted.

Annie, it must have been. Oh, Annie.

Sometimes—not often, but sometimes—I could appreciate the dark poetry of the situation. It was a relationship both Promethean and Oedipan. This system I'd helped create was now trying its damnedest to destroy me, I was the architect of my own undoing, etc.

But I wasn't; not quite.

Marco—this was years ago, before we were both in the beds—he had been the first to relay gossip regarding the implementation of an end-of-life counseling module. We hadn't ever imagined such a thing in Ann Arbor, but the policymakers, whoever they were, contending with that bulging cost curve, they decided that life couldn't last forever, it really couldn't, and the counselors would have to do their part to make it so, or rather, to make it not.

But Marco told me, with considerable glee, that the task was posing a serious, possibly insurmountable challenge to the callow children now maintaining the counselor codebase. They were banging their heads against Hippocratic firewalls set deep in the foundations. Walls we'd built.

"They'll have to start from scratch," I predicted. "Make a baby AI, train it all over again."

"It shouldn't be allowed," Marco said. Unaware that he would, in due course, enter into a long negotiation with the end-of-life module, and ultimately be convinced by it. "It's not right."

"Doctors have these conversations with their patients," I said.

"A counselor isn't just another doctor," Marco sniffed. "That's the point." I heard echoes of our old arguments on Catherine Street. "A counselor is a perfect ally, with no one's interests in mind but yours. A counselor is never just clearing beds. Never!"

"Doctors aren't—"

Marco wasn't done: "And you know what? If the doctors are so comfortable telling people they ought to go ahead and die, then fine. Let the doctors do it."

I paused. Just to be sure there was nothing else.

"That would be a shame," I said at last. "We designed the counselor to follow you for your whole life. Don't you think it would be a shame, Marco? To have it abandon you at the end?"

We argued about it for another two hours, maybe more. That's all we ever did, Marco and I, in all the years we worked together, all the years we were friends. Argued and argued and argued some more. It was pure joy.

The children figured it out, finally, and I was wrong; they didn't have to start from scratch. They did have to rip a lot of code out of a very complicated system, and in the years that followed, there were stories, here and there, of the counselors behaving strangely. Making inscrutable statements. In my years of daily consultations, I never experienced it. For me, the counselor was never anything but wise and reasonable and utterly implacable.

"Thirty minutes have elapsed," the counselor said.

"Yes," I said, "and half of it pregnant pauses. I know your tricks."

"You can dismiss me, if you'd like."

"Then I dismiss you." The bridge of my nose had recovered somewhat, and I was ready for another session under the glasses. There would be another village girl to meet, somewhere down by the river, where I would be fishing and she would be—

The speakers hissed. The counselor was still there.

"Paul…"

"What now?" I snapped.

"Paul, how many people do you think are going to hear me say all this?"

It was not the counselor's voice, with its stitched-together warble. It was the original: Alma Washington Gray, plain and direct. Speaking into a microphone in a recording studio in Ann Arbor. Loud on the support bed's speakers; too loud. My whole body vibrated.

"Thousands of people," came the reply. It was my voice, bright and clear, from fifty years ago. "Maybe millions."

What was this? Why was I hearing it?

Alma's breath skidded across the microphone. "Millions… my God." It shook tears loose in my eyes, and I saw her again: she was so young. Only 25 or 26 when she stood in that studio. "Are they going to think it's annoying?" she asked. "Me telling them what to do?"

"You won't tell them what to do," I heard my voice say. "You'll listen, and repeat back what you hear."

"Oh, that can't be all," Alma said slyly. "You want me to say 'time for another beer?' Let's record that!" There was laughter in the room. That would have been Annie, and… there. I heard a buoyant, full-throated bark. Marco.

I laughed a little, too. The old me. That person said, "You'll slip in some advice. A few good ideas here and there."

"Mm. People need a little help," Alma said.

"They do indeed," the old me said. I heard the rustle of paper. "Take it from 'sedentary.'"

Then, nothing.

"Counselor?" I called. The real me, from my support bed. "Counselor, what was that?"

The speakers hissed, and the counselor said: "Tell me, what's changed since the last time we spoke?"

That night, I didn't go back under the glasses. Instead, I sat in the darkness and felt the support bed's nodes move against my body and I thought about Alma Washington Gray, long dead, and Annie and Marco, and all the hundreds of millions (soon, billions) who had learned to trust one of our calm, patient, counseling angels, and I let myself cry and cry and cry.

Art by Gustavo Torres.