Experience that Matters
Welcome to thisseries of short papers on the topic of cybersecurity lessons-learned. I am approaching 40 years in my career, first in analog communications, then digital computer systems, and transitioning to cybersecurity. I have played roles in some of the most interesting jobs, like senior executive at the FBI in IT and from 2002 to early 2005 serving as CIO for the White House.Hopefully these lessons can be useful to those deep in, or just starting a career in cybersecurity. That is the purpose of these short papers, for me to speak to some of the lessons that come from a 40-year base of experience.
I make the claim, intending to a bit of provocation, that there is little that is new in regard to types of attacks and types of defense. A study of military history or of the biology of species are good analogs for these attacks and defense. We, in cybersecurity, even adopt many of the same terms, like the kill chain and defense-in-depth. If this is true, that little is new, then the old stories I will tell here should still resonate, still apply to a world that seems to be spinning faster than we can understand. The lessons remain relevant even today. Which makes this a good segue to start with an introduction.
About this 40+ year career and where it all started for me…
Starting with my name, Carlos Solari. I was born in Colombia, in a coastal town on the Pacific. My family immigrated in 1962 and we grew up in Long Island, NY. I got a BSin Biology from Washington and Lee University in the heart of Virginia. This is my biology connection. On graduation I was commissioned a US Army Officer, Regular Army, for those of you who know what that means. This is my military connection. My Army time was in the Signal Corps where I learned the technology and the operations of communications. I earned my Airborne Wings and Ranger Tabs, also for those who know what that means. The Army sent me to get a MS degree from the Naval Postgraduate School in 1988 and in 1992 I changed jobs and went to work for the FBI. This is my law enforcement connection where the responsibilities came with positions that eventually reached SESLevel 4 during a 6-year tenure with the Bureau. In 2002 I got a call from a former colleague who asked if I would be willing to serve as his deputy CIO. On the day of arrival, after signing all the papers, he told me that he was being promoted and asked if I would consider taking the CIO role. So, first in a temporary role, where the bosses in the administration wanted to try before buying, I got to serve in one of the most challenging and interesting roles for those of us in IT. It lasted for 2 years, 5 months and 16 days, not that I was counting. This is how I earned my stripes, as is said. And this is where the stories come from; from the experiences serving in very senior roles facing down the many challenges, incidents that came with those years in government and what followed after.
Before proceeding, a few words about my White House time. I will always be grateful to the people of this administration, the bosses up the chain and for the very special people who worked with me during this special time in my career. A boy from Colombia got to serve in this special trusted role. This does not happen except in this very special country of ours. From then to now I have been in the private sector. The lessons I will write about in this series are drawn from this period serving in the White House (2002-2005). In full disclosure, there were “issues” or if you prefer, “controversy” but I will stay away from these topics as I mean to stick to the topic of cybersecurity lessons learned and leave politics to others. The stories relayed herehave context to these other periods with the Army and with the Bureau. And they also have context to what happened after I left the White House where I decided that cybersecurity was the next wave. It has been quite a ride.
Experience is always useful in our space. Here, listed below for introduction, is a list of these stories of experience, all as true as I can make them. Not every detail is provided. I need not say why. This is not a tell-all. The stories, many of them, can only be corroborated by others who served in the same place and time. Once again, I tell these stories for the lessons, that they can be used to help others see their way forward in the present time.
Like the distributed-denial-of-service (DDOS) attack that was motivated by geo-political passions. We take the idea that cyber-attacks are often motivated by geo-political events for granted today. It is in the news daily; the fodder for much of the political battlesof the day. But back in 2003, this was a fairly new connection. In this case the target was the web sitewww. whitehouse.gov. The attribution was from South Korea, no doubt about it. More on this story in one of the upcoming segments in this series.
There was a PenTest of the White House network. The plan was that it would last for two weeks; the “penetration tests” escalating in sophistication. We (my team) had the full expectation that the testers would eventually “capture a flag,” in the vernacular of PenTests. We just did not expect the test to stop on the morning of the second day. The PenTest team had gained access to something that should not have happened and needed to be addressed immediately.
A worm called Sasser entered our White House network through a laptop that had been connected to other networks, possibly when that laptop had connected to a hotel, and it, the worm, started replicating, eating the data on the hard drives for breakfast. By the time we had figured out why we were in the midst of a crisis / incident – the help-desk phones would not stop ringing - the only remaining option was for me to authorize disconnecting the entire White House network to stop the spread. 600 computers were already dead. There was no time to ask for permission. My call – my career - probably ending in this one decision. But I never had to make that call because one of our young stars, all of 19-years-old,saved the day.
The social events database back in 2004 served the Office of the First Lady. It may still be in existence today. Back then it was the central place for all the historical and future event planning repository for all the White House social events, like State Dinners. It crashed hard – data corrupted and so was the backup.
These are four of the stories. I plan on writing about these and maybe a few more depending on how things go. And one last point emphasizing again the purpose. I have a lot of “I,” “me” and “my” in recounting these stories. It sounds like a lot of bragging, but I know better. Credit belongs to the teams and sometimes to heroic individuals who always do the right thing and who do their jobs every day. Where I have had success, it is because of them, these teams and these individuals who are behind the stories. The lessons we can draw from the telling of them, and how they can serve us today is the purpose. Until the next one, I offer all the best wishes.