One night in Kuwait, Dana Beausoleil, a technician for defense contractor Raytheon, sat in a broken-down white truck that was supposed to serve as a mobile command-and-control platform for the occupying government in postwar Iraq. The truck had been delivered from the United States that day, and the 58 radios, its omnidirectional and directional wireless network access technology, and its satellite uplinks would not power up.
As a thunderstorm raged overhead, driving wind and rain against the truck, Lt. General Jay Garner, the U.S. Army officer who initially headed the reconstruction and postwar governance of Iraq, came outside to address Beausoleil and his colleagues.
"We were fairly panicked because this truck came in dead on arrival…" Beausoleil said. "I was out there in this van troubleshooting the electrical system. Jay Garner came out saying, 'We've got to get this thing going. We're going to Baghdad the day after tomorrow.' "
It was mid-April, 2003, and this was the beginning of Beausoleil's three and one-half months assignment to help build and maintain the network that would serve as the heart of the reconstruction and government of occupied Iraq.
"[Coalition forces] were looking for people that could take over the banking system, the secretary of state, everything from the president to his cabinet, and to do that they had to network with the entire world," he said. "And seeing as we had just blown them into the Middle Ages, taking out
Beausoleil got the truck running, and his team accompanied a 180-truck convoy filled with equipment to Baghdad. They set up shop in Saddam Hussein's Ramadan Presidential Palace, which would become the seat of the Coalition Provisional Authority and today serves as part of the United States Embassy.
The mobile command-and-control truck would serve as the network hub for the palace for the next month and a half as Beausoleil and his colleagues set about building a wired network inside its marble walls. General Garner and his staff would initially use a handful of encrypted, rugged laptops that had a secure connection to the truck's wireless network. They would rely on these laptops to communicate with occupying forces and with the White House.
"When we pulled into Baghdad, it was still burning," Beausoleil said. "We would go out at night and there would be firefights. There would be machinegun fire coming in, and there would be attacks on the palace. We used to watch Baghdad burn. It was like watching Rome burn while we were there."
The palace was a nightmare for network engineers. The walls were made of thick marble. All the wiring would have to be exposed.
"Initially, there were no windows in the palace," Beausoleil said. "We had bombed its nearby power center and that had taken out all the windows. We got in there and there were two inches of sand everywhere. We called it moon dust. It was amazingly fine. We hung sheets in the doorways and windows to keep it out. But we had to go in every day and clean the equipment. We would have to shut down the equipment and blow it out with air."
Working 17-hour days, Beausoleil and a team of 17 other engineers laid out an ad hoc network in these harsh conditions. It consisted of a Promina core switch from Network Equipment Technologies, dozens of Cisco Catalyst switches, and Dell 2650 servers.
"The Cisco switches did great," he said. "We had lots of port failures. It didn't matter what kind of switch we had. The digital thermometer in my truck hit 148 degrees one day. The devices would not totally fail, but it would tend to burn out a port or two. It would get clogged with dirt and sand."
The elements were harsh, and the palace was anything but ideal. Because it was built out of solid marble, Beausoleil was limited in where he could pull cable for the network. There were fiber cable, CAT5 cable, electrical cables, switches and generators scattered everywhere. He described the palace's courtyard as resembling a giant pile of spaghetti, with cables running in and out of windows, connecting one wing of the palace to another.
All this exposed equipment was vulnerable -- users would move things around and spill food on the sensitive gear. And the scalability of the network quickly became an issue.
"The original task was to get a network up and running that could support 200, but by the time I left in August, there were 2,000 people in the network," Beausoleil said. "We were pretty much working 17 hours a day trying to keep the network going. We just kind of went into a casualty mode where you basically kept it going no matter what. And there was always a problem. When people came in, they would go in and change things, and we would drop an entire wing of the palace.
"It was one long catastrophe because we were constantly troubleshooting everything," he said. "They finally called a meeting and told everyone on the networking and telecommunications team to get some sleep because we couldn't keep it up."
Eventually the Promina core switch went bad and took down the whole network for three days before engineers determined that it was the cause of the outage and installed a replacement. Given the fragility of this young network and the lack of visibility the engineers had, Beausoleil's team decided they needed to install some sort of network management technology.
The team settled on using WhatsUp Gold from IPswitch. Everyone on the team had worked with it in the past and liked it.
"[WhatsUp Gold] did discovery," he said. "And it gave us real-time information. We tested it out, and it popped up a display with all the devices out there. We said, 'OK, this machine is alive.' Then we started setting SNMP traps for all the different services and links that weren't automatically configured. So when we had troublesome parts of the network that failed, it would show a failure."
"That was a turning point -- from catastrophe management to being able to follow the network and figure out what we have working and don't have working, and to get alerted if we had a link go down," Beausoleil said. "We put in a big plasma display in this monitoring room we built. And when the rats weren't crawling on it, we had WhatsUp Gold on the plasma. Whenever any link went down, we would get a red alert and we could go take care of it. We got to the point where we had a very reliable network."
Beausoleil would eventually leave Baghdad to return to work as an analyst for the FBI. A reservist, he was called to duty again in 2004 and spent another year in Iraq, helping build and maintain other networks in Tikrit. He said he used some of the same network topology there, but for security reasons he was unable to use WhatsUp Gold to monitor the network. While he was in Tikrit, the Army never installed a network management platform.
"The big difference was we had more downtime and troubleshooting because we didn't have any automated tools to work with," Beausoleil said. "Sometimes we would have days where the computers would just go down."