When I started in datacenters, we had places where we used network addressing and layer 3/layer 4 firewalls as the primary way to manage service identity for internal services where we needed it. Even at that time, there were some annoying attack scenarios you had to deal with in multitenant environments and compromise scenarios attacking your layer 2 and 3 infrastructure (MAC spoofing, ARP poisoning, etc.) that meant even then it wasn’t a totally reliable source for identity information. Luckily, for those reasons and other trends, people have largely moved away from source IP as a source of workload identity.

Network Service

With the advent of high automated infrastructure systems and the birth of “cloud” infrastructure we saw other approaches to providing identity to compute workloads emerge. The most well known this is probably the AWS EC2 metadata service. Fundamentally this method boils down to using a link-local address to allow the workload to access an API to get information about itself over the network. This include operational characteristics and user data used to power cloud-config. Cloud-config powers a lot of the run time first boot features like executing arbitrary scripts, or configuring the system based on data provided by the metadata service.

However, if configured with an IAM role and instance profile, this service also provides a set of AWS IAM credentials that can be used to call permitted AWS services (based on the access configuration of that role) that are effectively accessible to any process that can source network traffic from the system by default. Another endpoint this metadata service provides is an identity document. This is a cryptographically signed document containing configuration information about the instance. Because this identity document is made available only to the workload itself this link local address people also use possession of this identity document as proof of workload identity.

This approach has some short comings, many of which these service providers have worked on strategies to mitigate. AWS on its documentation page for the metadata service (at the time of writing) calls out one of the largest ones. Any workload that can communicate over the network to that link local address can see all that data and those credentials. Outside of obvious attacks resulting in system compromise, there are a surprising number of attack vectors that allow you to make a vulnerable workload or systems perform a web request on behalf of the attacker – and with this network metadata approach all of them potentially expose your service identity/credential data.

It is easy to build broad API support for this credential discovery approach. Both the service provider themselves can do this in their libraries and SDKs as AWS does and others can do this using presigned URLs with instance role credentials or validating the instance identity document.

Application Environment

Another approach we see is folks providing credentials to applications, particularly beyond the IaaS level, is injecting data in the processes running environment. This broadly comes in two flavors, actual process environment variables or a file/filesystem provided within the filesystem presented process. I feel like the dangers of storing credentials and secrets in your processes environment variables has been documented extensively, but in summary both infrastructure systems and often times applications themselves (or their dependent libraries or frameworks) don’t protect the process environment as if they contain secret data in many cases. It is still common to find logs that freely log environment variables or send them as part of exception reports or debug error messages/screens. It is more rare to find applications that allow attackers to read arbitrary files from the file system, but they do exist – generally by honest developer mistake or misconfiguration rather than by design as many of the environment variable exposures appear to be.

These patterns are commonly used with Kubernetes secrets or config maps, and container PaaS/function as a service platform like Lambda. Particularly with the filesystem approach this feels like the most prudent path at this point as a service provider to me.

It is noteworthy that these Application Environment methods require more awareness and integration between the scheduler and the workload. In high level cases like function or container as a service, this probably feels natural. However, it gets more complex if you want to support many operating systems and depends heavily on the features available in your scheduler/hypervisor layer.

Why bother?

Very few workloads we deploy today are an island. Most communicate with service providers we leverage consuming PaaS services or other services we write and deploy separately in a modern microservice environment. Traditionally we dealt with long lived credentials that we stored treated as secrets used in the production configuration management or deployment processes.

There are other solutions to machine identity out here. Enterprise organizations have been dealing with Active Directory and their related machine account dynamics with Windows for decades. Even earlier than that systems like Kerberos had solutions to this problem. Most of the solutions that solve this do not fare well when applied to the Service Provider use case. I do not know any service providers today that operate with multiple customers sharing an Active Directory environment for instance. I think many customers would be uncomfortable with this scenario if they did.

As technology and security practices have evolved, the benefits of moving to short lived credentials have been wildly discussed. It caps the duration of risk related to someone getting access to credentials that have been used in your production environment. This includes attack vectors like someone compromising a production backup, snapshot, or set of log files that inadvertently included credentials for your environment.

Many of the advancements people think of as cloud are highly automated infrastructure and modern deployment practices. Giving customers a secure root of identity solution can encourage them to adopt other useful infrastructure automation patterns, like treating servers as short lived and replaceable rather than long lived and maintained. This sort of identity can be foundational to first boot scripts interacting with other services from your provider or reaching out to your centralized identity provider to get other credentials it needs to access other systems. Getting people to think about their infrastructure differently, rather than any technical capability is the most impactful improvement most organizations make during a cloud transformation.

Today’s cloud landscape has no shortage of options when it comes to deploying HTTP-based applications. For small ideas and side projects I still have a hard time justifying running an entire container cluster, so I find myself reaching for options like Google Cloud Run or AWS Fargate because it provides me with a  clear path towards something like Knative if my small idea starts to grow into something more substantial. At some point if you have enough of these small workloads running it will start to create cost drivers toward building a cluster multiple workloads could share.

The minimum footprint for a side project is a small amount of CI/CD configuration in Gitlab, GitHub, Google Cloud Build or AWS CodeBuild, a deployment target (or several if you run a proper multi stage deployment process including dev, staging, and production). This makes it quick and inexpensive to get things up and running quickly so you can iterate on your idea. 

Several years ago, I was living just outside New York City. This particular morning I happened to be at my girlfriend's place in central Pennsylvania and we weren't even completely awake when her brother in law came to the door almost panicked. I didn't know him very well and didn't know what to make of it at first when he came running in. We turned on the television just in time to see the second plane crash into the towers.

I'll never forget the feeling I had trying to take a mental inventory of where my friends and colleagues were likely to be that morning. I started calling, texting, emailing (like everyone else at that moment) trying to make sure my friends were safe. Only a few were in Manhattan that morning and luckily they were all OK.

I will never forget

I was raised by my mother and my grandmother. Throughout my life they have provided me with the love, caring and support to make it through even my toughest days. Anything good that I have or will accomplish is in no small part because of them. They both broke their backs to provide their children and grandchildren with opportunities and experiences that surpassed the ones they had growing up. They taught me right from wrong, strength during hardship, compassion and empathy, how to improve yourself, and countless other things that one could easily overlook. I carry these things with me for all the days of my life and even in times I lose my way it is these tools that I use to find it again. There is no gift, no card and no flowers that will ever truly express how much I appreciate what they’ve done and continue to do for both me and my son.

Tonight I found out that another co-worker is taking the next step in their career and leaving our company. Selfishly I’m sad I won’t be working with them anymore, but honestly I’m happy for them and know they have a very bright future ahead.

One of the Operations Directors at work developed an application we have started to use to gather 360 feedback from the other people in the organization you interact with frequently. This tool basically asks for a number of objective criteria to be evaluated on a 1 to 10 scale and then provides the ability to give you some structured (but largely free form) data on things you are doing well and things you should consider changing.

I hope most people have watched this amazing speach, but it is worth watching today. Steve Jobs had an incredible impact on the world in his years.

This was originally a pearl of wisdom I sent to a co-worker who told me I'd have something "by the end of the day" on a Sunday:

After 10 years in different roles I've stopped wearing "workaholic" like it is a badge of honor. I work weekends if I can't avoid it - but it leaves me regretting the time I missed with my son and my friends. I work nights if I have to, because I need to carry my weight and not create undue stress on my colleagues (most of whom I also count among my friends).

When I started traveling all the time, my son gave me one of his stuffed animals to bring with me. I carry it in my bag and wherever I go I try to take a few pictures of this little stuffed penguin in various locations. I was touched because it is one of his favorite stuffed animals and I smile every time I see it in my bag.

Today, I had a rare insight into the customer experience of someone trying to purchase a moderately sized managed hosting installation. Somehow in the many years that have passed since I first entered this space I've never stopped to think about how grueling the sales experience must be to have to do that shopping. Most every organization I've worked at that had a sales staff incentives them on the amount of hosting they sell. This makes sense because you want to reward sales people that generate revenue. However many of the sales compensation structures I've seen place little or no emphasis on the long term customer satisfaction and deep relationship developed by selling a customer the RIGHT solution.

People purchasing managed hosting often do so because they want to offload some of the responsibility of building, maintaining or managing their infrastructure. This is the way that many hosting vendors try to position themselves to fill for customers. So why does the initial experience so often begin with someone who doesn't have your long-term best interest as his or her primary motivator?

In many times where I've filled a pre-sales capacity for a customer I've built a deep relationship with that customer that spanned several roles in the company for me and the entire lifetime of their account. As an industry, we need to figure out how we can make that the normal experience, rather than a situation where people sometimes suffer through the sales process in the hopes that the operations experience and the technology they come out with are worth the headache.

we spend just advertising to get people to fill out the census. If the 3 post cards including the actual form containing the census and TV ads were not enough.... BOOM! Facebook Ads :)

I've accepted a position at work that has me spending the majority of my time acting as Product Owner in our Software Engineering organization (which has adopted a mostly agile development process). The primary team I serve as product owner for (our customer portal team) has some of our most seasoned developers and is more or less self sufficient. My day to day oversight of their efforts is minimal if at all. I set priorities every two weeks, check is periodically and answer questions as they come up - but that is pretty much it.