Thoughts on service provider rooted identities

When I started in datacenters, we had places where we used network addressing and layer 3/layer 4 firewalls as the primary way to manage service identity for internal services where we needed it. Even at that time, there were some annoying attack scenarios you had to deal with in multitenant environments and compromise scenarios attacking your layer 2 and 3 infrastructure (MAC spoofing, ARP poisoning, etc.) that meant even then it wasn’t a totally reliable source for identity information. Luckily, for those reasons and other trends, people have largely moved away from source IP as a source of workload identity.

Network Service

With the advent of high automated infrastructure systems and the birth of “cloud” infrastructure we saw other approaches to providing identity to compute workloads emerge. The most well known this is probably the AWS EC2 metadata service. Fundamentally this method boils down to using a link-local address to allow the workload to access an API to get information about itself over the network. This include operational characteristics and user data used to power cloud-config. Cloud-config powers a lot of the run time first boot features like executing arbitrary scripts, or configuring the system based on data provided by the metadata service.

However, if configured with an IAM role and instance profile, this service also provides a set of AWS IAM credentials that can be used to call permitted AWS services (based on the access configuration of that role) that are effectively accessible to any process that can source network traffic from the system by default. Another endpoint this metadata service provides is an identity document. This is a cryptographically signed document containing configuration information about the instance. Because this identity document is made available only to the workload itself this link local address people also use possession of this identity document as proof of workload identity.

This approach has some short comings, many of which these service providers have worked on strategies to mitigate. AWS on its documentation page for the metadata service (at the time of writing) calls out one of the largest ones. Any workload that can communicate over the network to that link local address can see all that data and those credentials. Outside of obvious attacks resulting in system compromise, there are a surprising number of attack vectors that allow you to make a vulnerable workload or systems perform a web request on behalf of the attacker – and with this network metadata approach all of them potentially expose your service identity/credential data.

It is easy to build broad API support for this credential discovery approach. Both the service provider themselves can do this in their libraries and SDKs as AWS does and others can do this using presigned URLs with instance role credentials or validating the instance identity document.

Application Environment

Another approach we see is folks providing credentials to applications, particularly beyond the IaaS level, is injecting data in the processes running environment. This broadly comes in two flavors, actual process environment variables or a file/filesystem provided within the filesystem presented process. I feel like the dangers of storing credentials and secrets in your processes environment variables has been documented extensively, but in summary both infrastructure systems and often times applications themselves (or their dependent libraries or frameworks) don’t protect the process environment as if they contain secret data in many cases. It is still common to find logs that freely log environment variables or send them as part of exception reports or debug error messages/screens. It is more rare to find applications that allow attackers to read arbitrary files from the file system, but they do exist – generally by honest developer mistake or misconfiguration rather than by design as many of the environment variable exposures appear to be.

These patterns are commonly used with Kubernetes secrets or config maps, and container PaaS/function as a service platform like Lambda. Particularly with the filesystem approach this feels like the most prudent path at this point as a service provider to me.

It is noteworthy that these Application Environment methods require more awareness and integration between the scheduler and the workload. In high level cases like function or container as a service, this probably feels natural. However, it gets more complex if you want to support many operating systems and depends heavily on the features available in your scheduler/hypervisor layer.

Why bother?

Very few workloads we deploy today are an island. Most communicate with service providers we leverage consuming PaaS services or other services we write and deploy separately in a modern microservice environment. Traditionally we dealt with long lived credentials that we stored treated as secrets used in the production configuration management or deployment processes.

There are other solutions to machine identity out here. Enterprise organizations have been dealing with Active Directory and their related machine account dynamics with Windows for decades. Even earlier than that systems like Kerberos had solutions to this problem. Most of the solutions that solve this do not fare well when applied to the Service Provider use case. I do not know any service providers today that operate with multiple customers sharing an Active Directory environment for instance. I think many customers would be uncomfortable with this scenario if they did.

As technology and security practices have evolved, the benefits of moving to short lived credentials have been wildly discussed. It caps the duration of risk related to someone getting access to credentials that have been used in your production environment. This includes attack vectors like someone compromising a production backup, snapshot, or set of log files that inadvertently included credentials for your environment.

Many of the advancements people think of as cloud are highly automated infrastructure and modern deployment practices. Giving customers a secure root of identity solution can encourage them to adopt other useful infrastructure automation patterns, like treating servers as short lived and replaceable rather than long lived and maintained. This sort of identity can be foundational to first boot scripts interacting with other services from your provider or reaching out to your centralized identity provider to get other credentials it needs to access other systems. Getting people to think about their infrastructure differently, rather than any technical capability is the most impactful improvement most organizations make during a cloud transformation.

Infrastructure for side projects

Today’s cloud landscape has no shortage of options when it comes to deploying HTTP-based applications. For small ideas and side projects I still have a hard time justifying running an entire container cluster, so I find myself reaching for options like Google Cloud Run or AWS Fargate because it provides me with a  clear path towards something like Knative if my small idea starts to grow into something more substantial. At some point if you have enough of these small workloads running it will start to create cost drivers toward building a cluster multiple workloads could share.

The minimum footprint for a side project is a small amount of CI/CD configuration in Gitlab, GitHub, Google Cloud Build or AWS CodeBuild, a deployment target (or several if you run a proper multi stage deployment process including dev, staging, and production). This makes it quick and inexpensive to get things up and running quickly so you can iterate on your idea. 

My toolchain of choice continues to change often because the technology landscape around PaaS and Serverless deployment options is evolving rapidly. AWS Lambda, Google Cloud Functions and similar offerings do not feel like the right fit and the tradeoff between ease of use and vendor gravity doesn’t seem worth it to me at this moment. I have built many solutions using these services and for many use cases they are a better fit than something container based, but I have committed technology sins by using serverless functions in places I should not have. That scar tissue is still fresh.

Close to home

Several years ago, I was living just outside New York City. This particular morning I happened to be at my girlfriend's place in central Pennsylvania and we weren't even completely awake when her brother in law came to the door almost panicked. I didn't know him very well and didn't know what to make of it at first when he came running in. We turned on the television just in time to see the second plane crash into the towers.

I'll never forget the feeling I had trying to take a mental inventory of where my friends and colleagues were likely to be that morning. I started calling, texting, emailing (like everyone else at that moment) trying to make sure my friends were safe. Only a few were in Manhattan that morning and luckily they were all OK.

I will never forget

Reflections on Mother's Day

I was raised by my mother and my grandmother. Throughout my life they have provided me with the love, caring and support to make it through even my toughest days. Anything good that I have or will accomplish is in no small part because of them. They both broke their backs to provide their children and grandchildren with opportunities and experiences that surpassed the ones they had growing up. They taught me right from wrong, strength during hardship, compassion and empathy, how to improve yourself, and countless other things that one could easily overlook. I carry these things with me for all the days of my life and even in times I lose my way it is these tools that I use to find it again. There is no gift, no card and no flowers that will ever truly express how much I appreciate what they’ve done and continue to do for both me and my son.

Forks in the road

Tonight I found out that another co-worker is taking the next step in their career and leaving our company. Selfishly I’m sad I won’t be working with them anymore, but honestly I’m happy for them and know they have a very bright future ahead.

What really got me thinking was the conversation we had after covered the basics (e.g. sorry to see you go, where are you heading, etc.). The reality is that most careers don’t follow the clean-cut path they have in the past. Not very long ago it was common to work for the same company for decades, leave with a gold watch, benefits and a pension. That definitely isn’t the reality I live in, and I often think about the going on five years I’ve invested in my current organization. Granted my career path has been pretty interesting over those years – but I’ve been at the same place.

Today you are responsible more than ever before for your own retirement, your own career and your success. My friend made the comment that loyalty doesn’t exist anymore, I responded that I don’t think it is gone but I think it has changed. Today’s loyalty is a bond between humans, not between a human and an organization. I think I like it that way

When giving me advice at one point a few months ago a senior executive whom I’ve grown to trust said to me that the only way to build the kind of bond he and I were talking about at the time, where near frictionless disagreement and collaboration are possible, was to go through “battle” with someone. He didn’t mean this strictly in the military sense however I can only imagine the result is even stronger in that case. He was referring to the absolute certainty to the guy standing next to you will still be there, fighting right along with you, even when things get ugly. In some jobs, that might be an everyday event or an extraordinary circumstance. Either way the result is the same, you learn who you can trust. You learn who you can count on and you learn whom you can’t.

I’ve lost another one of the people who fit solidly in former category. The good news for me is that you never really lose those people; you just might not know how or when your paths will cross again. In fact, I got a call from another one of the people I’d put in that category earlier today asking me to speak at a User Group his organization is hosting next month.

360 Feedback

One of the Operations Directors at work developed an application we have started to use to gather 360 feedback from the other people in the organization you interact with frequently. This tool basically asks for a number of objective criteria to be evaluated on a 1 to 10 scale and then provides the ability to give you some structured (but largely free form) data on things you are doing well and things you should consider changing.

I like this application and the idea behind it. When it is presented to you (and your manager) it shows you all the feedback but keeps the people providing it anonymous. I will say that I wish there was a way for me to ask questions (anonymously would be fine) about some of the feedback so I can really understand of it but most of it is pretty interesting.

One of the outputs it gives you is the graphic I'm showing here, it maps your certain traits based on the objective feedback. The other most interesting part is looking at the delta between your average rating from peers and how you rate yourself.

Steve Jobs on Death

I hope most people have watched this amazing speach, but it is worth watching today. Steve Jobs had an incredible impact on the world in his years.