From the Annals of Cyber Monday

When it's too late to take the stitch in time that would have saved nine ...

I've had better Mondays. Coming off the Thanksgiving weekend I spent my re-entry day dealing with two critical problems where I work. I started thinking that instead of "Cyber Monday" it should be called "Blue Monday."

The first issue was that one of our printers, an expensive laser photo-imaging device, was down. Indications were that one or more of its lasers were not working (it has red, green, and blue ones). There are also three computers driving this beast. The first prepares images to be printed, the second is dedicated to feeding them to the printer because it's a delicate operation that requires tight synchronization. Neither of those systems had any problems, but the one that sends the images lit up with a warning screen about the bad lasers.

A call to tech support for the printer reminded me to check the third computer, which is tucked away inside the machine and normally not used by the operator. Unlike the first two computers, which run HP Unix, this shy unit runs a command-line-only version of ... you'll never guess ... Windows 98! Which kind of shows how old the whole thing is. Anyway, this computer has a job to do in controlling the lasers, and it was simply powered off.

But why? It's supposed to come on when you start the machine. Turned out the little button battery that keeps the CMOS memory alive when it's powered down had finally run out of juice. (That's where all the BIOS settings are saved.) After all, it was at least 10 years old. Apparently when that computer failed to boot up properly the system just shut it down. Tech support coached me on how to put the BIOS settings in manually so we could get up and running, and we sent out for a replacement battery. For want of a $6 part the whole thing was dead. Problem solved.

In the middle of all this the second problem came up. One of our Mac users couldn't get on the network. I couldn't find any obvious reason. Then a Windows user came to report that he had the same problem -- in fact "the whole network" was down. Well, I knew that wasn't true because I'd just been on the network at my desk. And the Mac user confirmed that at least some of the other Macs were not having any problem. While I was looking into the network settings on the Windows machine it suddenly started working again. So, maybe a temporary glitch? But no, the other computers were still having problems.

We now had a select group of Windows PCs, Macs, and even one Unix machine exhibiting the connection problem, and scattered between our two buildings. It couldn't be a network switch, because most likely that would have affected all the computers connected to it, but I restarted the switches anyway. No help. We tried our wireless access point and could connect to it, but could not access the Internet, so it seemed to be having the same problem -- it couldn't connect to the network gateway to the Internet. It seemed like it had to be something on the server, but the server appeared to be working normally and to have Internet access itself. Time to call in our friendly IT support company which does all the heavy lifting on our server issues.

Meanwhile I noticed that all the machines having the problem had similarly misconfigured IP addresses. Our network uses 10.1.1.x addresses, and these all had 192.168.3.x addresses. I tried manually configuring one of them with a correct address. It still didn't work, but now we were thinking about DHCP.

[If you want to know, that stands for Dynamic Host Control Protocol, which is simply a procedure by which a server can control the addresses of the computers on its network. When each computer starts up it shouts down the hallway, "Hey, I'd like to join the network -- what address can I use?" Everyone else on the network ignores this request, but the server shouts back something like "10.1.1.203."]

The first thing the support tech told me was that the DHCP service was not running on the server. Aha! that made sense. Only those computers that had recently rebooted, or whose addresses had to be renewed, were having connection issues, because they couldn't get an address on the network. All those that had been left on over the weekend were still running normally. So, just as simple as starting the DHCP service up again, right?

Not really. Because we didn't know what had caused it to shut down. And we didn't know where those bogus IP addresses were coming from. Normally if the computer asks for an address and can't get one, then it simply has no address; it doesn't substitute a different one. The network engineer deduced that something else on the network -- either an unauthorized device or a piece of malware -- was acting as a DHCP server. If that was the case, then it would cause the Windows server to stop its own DHCP service to avoid conflicts. And it would explain where all those similar IP addresses were coming from.

We had not added any new equipment, so the only thing to do was try to find out which machine was the culprit. Time to play detective. The engineer began unplugging one system at a time from the patch panel in the server room. After each one he tried to connect to the network with his iPad. As soon as he was unable to get an address, he knew he had unplugged the offending device. The answer was port 18 on the patch panel. Using our sketchy documentation of our wiring plan we located the guilty party.

If your prejudices are like mine, you probably think that either someone had plugged in their own router somewhere, or else one of the Windows machines had gotten a virus. But surprise -- it was a Mac Pro workstation.

So now we knew where and what, but not how. Why would one of the Macs suddenly decide to start acting as a DHCP server? Consulted by phone, a Mac guru identified the problem by that "3" in the IP address, which is not commonly used. It's an address range used by the Internet connection sharing service on the Mac, by means of which one computer can share its network access with a group of other computers, and which works by acting as a DHCP server.

But why would this service suddenly be switched on? We questioned the users -- had they made any changes, had network problems that they tried to solve? No, but, now that we mentioned it, the only unusual thing about that Mac was that over the weekend it had been moved from one desk to another, so it was plugged into a different port on the wall. But why should that change anything?

Of course, it shouldn't. But the Mac Pro has two ethernet ports on the back which can be configured differently. Sure enough, the other port did NOT have Internet connection sharing enabled. All that had happened is that when the computer was moved the network cable had been switched to the other port, with all the resulting complications. Normally it would not have been a problem, and everything would have "just worked" as Macs are famous for. But somewhere in the past someone must have configured that other port for some long-forgotten reason, and it had emerged to bite us.

They say you're always supposed to learn from your mistakes, but that implies that you can figure out what your mistakes were. What we learned from this is that the littlest things -- like that bad CMOS battery -- can have far reaching implications. In an organization of any size, where the network must run smoothly, even a small glitch can bring everything to a standstill. In this case, we lost several hours of impaired production time while we tried to find something that should have been apparent as soon as it happened.

If an IT support person -- me, for example -- had moved the Mac instead of one of the users, then the problem still might have come up. After all, there was a 50/50 chance that I might plug into the "wrong" ethernet port, and even if I were careful about testing the connection the problem might not have been noticable on that computer because it was the one causing the problem. But at least I would have known that this was the last thing that had changed on the network, and I could have started looking there. I might have just disconnected it, and when that solved the problem we could have left it disconnected until we had a chance to get down to the source of the problem. Instead, since I was working in the dark, we had to take the long way around to the solution.

What else could we have done? Well, if time travel was an option, I would suggest going back to when that second ethernet port was configured to support some external device. Then we could document what was done, and why. We might even label the port as being dedicated to a specific purpose in the hopes this might prevent someone in the far future from trying to use it to connect to the network.

The moral, I guess, is that the future arrives sooner than we think, and the past -- in which all mistakes happen -- is right now.

In His Own Words

Yes, war is hell. Take it from someone who knew ...

On some previous Veterans Days I've taken the opportunity to remember my father, who, among other things, fought and was wounded in the largest and deadliest war in history. This year I thought I'd let him speak for himself. The following piece from his posthumous collection of poetry contains everything he had to say on the subject of war.

Like many veterans of World War II he seldom spoke about what had happened to him. But I remember him reading this poem to my mother when he wrote it, probably ten years after the war had ended. It was as if everything he had bottled up inside spilled out.

In it you can see the cynicism about organized religion that let him to question his faith and to abandon the Church. (He had been born and raised a Catholic.) But there is also, I think, an undercurrent of an abiding faith in the power of the natural world to heal and restore us if we will only allow it.

      Your Wars

Go on and fight your wars
Pseudo-Christians, cowards all,
Professing meek cheeks,
Afraid to sheath your swords.

I've seen every war since your God forbade them,
Astonished at your unstrained compromise,
And I've seen enough:
The distraught of men before the firing,
Of other men before our firing,
And I mourn these young men
Dying before their works are done.

One I hardly knew fell across my path,
Unwilling mutant, an extra mouth
Newly dug above his ear.
What words from that eloquent mouth!
But never mentioned little cakes or chasubles.
I still see his questioning eyes.

One sought cover beneath a tank
Which, seeing the fire it drew,
Backed without warning, its steel teeth
Chewing his brain to pulp on icy ground.
Farmers no doubt wondered
At that especially fertile spot.

One oblivious to the fighting
Sat clutching himself to himself,
Cupping the ragged remains of genitals
In bloody hands. Impassioned bullet
To engage in such sadistic intercourse.

A woman sprawled broken among the rubble,
Breast suckling the air, legs undecorous.
We forgave her impropriety,
In unknowing embrace with fornicating death.

And I've seen the children.
Have you seen them, never young?
Eyes wondering?
Legs wandering?
Grimy claws sifting garbage,
Scraping maggot off?
Don't worry, belly-bloated child,
There's plenty of food in heaven --
By the way, have you been baptized?
In the name of the Father, the Son, and Holy Ghost.
Amen. There. Now you can die.

What are your wars for?
I see only one good in them --
A speedier evolution to weed out the unfit --
The cowards yearning for death
Who fight and die for make-beliefs.
It's living requires courage -- living never tried.

So go on and fight your wars.
Your countries will go, and your rich kings,
Leaving only the unturmoiled world.
Marked only with natural boundaries
Her hair grows green and luxuriant,
Her tears wash over ancient festers, healing them,
Leaving only the brave, holding hard to nebulous dreams.

So go on and fight your wars,
Expunge yourselves, become vague specimens
In future museums: "Here is Acquisitive Man
From the Age of Veneers, offshoot of Homo Sapiens.
Note the small braincase and large grasping hands."

So go on and fight your wars,
Churchmen send them out on new crusades,
Pray of the Peaceful Prince luck in the kill,
Satraps continue doing foolish things in duty's name,
Drape the hallowed bunting on the deluded slob's eternal bed,
Safe old men, cheer them on,
tear in eye, drink in hand.

-- "Scott" Donachie, 1922-1973

See Also:

My Father the Veteran

Another Veterans Day

The Day After Veterans Day

On the Joys of Reading Long Books

There was a time in my life when I found books of a certain size intimidating. Maybe it goes back to my attempt, at the age of twelve, to absorb my father's copy of War and Peace when the only adult literature I had read until then was science fiction. I remember getting to the end of chapter one and feeling like I'd just eaten way too much chocolate cake. I knew it was good stuff, on a level above anything I'd seen, but I also knew I wasn't ready for it.

By the time I reached high school I was able to plow my way through not only Tolstoy but Dostoyevsky, and not only Russians but Americans like Steinbeck and Faulkner -- even Absolom, Absolom with its pages-long sentences. In my college years Hermann Hesse came into vogue, so I consumed most of his works including the monumental Magister Ludi (or The Glass Bead Game) which is supplemented at the end by a sort of appendix of poetry and three short stories  purported to have been written by its main character, a wonderful example of a book that contains other books. I also developed my fondness for philosophical works by digesting William James' Varieties of Religious Experience and Aldous Huxley's The Perrenial Philosophy. Both of those are heavy wading, but they made an interesting pair since they shared the idea that all the world's religions have much in common, an idea that appealed to me as a Unitarian who was destined to become a Quaker.

Then, as happens to many of us, I became so caught up in the world and the need to make a living that I found less time for reading. I returned to my first love of science fiction for recreation, and gravitated toward short stories that were quicker to consume. I lost my taste for big thick books that revealed from their sheer bulk the amount of time and attention that would have to be give to them. It was so much easier to take two hours to absorb a movie or a magazine.

I suppose it has just been reaching a later stage of my life that has drawn me back to those substantial works I had avoided for so long. But the pleasure of discovery -- or rediscovery -- has been well worth the wait. Armed only with the willingness, I launched into such projects as reading the twenty volumes of Patrick O'Brian's seafaring novels, and Neal Stephenson's trilogy of 700-pagers set in the Baroque period. I also got an ebook reader and have been using it to catch up on many of the classics that I had somehow managed to overlook, things like Moby Dick, The Magic Mountain, and the other wonderful novels of Joseph Conrad which are not called Heart of Darkness.

Having acquired the taste, it is now one of the most satisfying feelings I can imagine to sit down and crack the covers on something that measures a couple of inches in thickness. I'm finally reading Shadow Country, Peter Matthieson's final version (900 pp.) of his legend about Mr. Watson, the backwoods killer who was shot down by his neighbors in the Florida Everglades.

And, just to keep things interesting, I've also started in on the three volumes of Shelby Foote's definitive account of the Civil War. Something wonderful happens when you realize the author will take as much time and space as it takes to give you the full picture. Thus we begin with two capsule biographies of Lincoln and Davis, the rival Presidents, and proceed at leisure up to the first conflict at Harper's Ferry.

The reader can sit back with a sigh, confident in the hundreds of pages remaining to unfold the whole tale, and comforted by the other two volumes still waiting on the shelf. When you're having this good a time, the best news you can hear are the words, "To be continued ..."