tag:adamgreenfield.com,2013:/posts AdamGreenfield.com 2020-11-26T07:27:03Z tag:adamgreenfield.com,2013:Post/1621306 2020-11-25T23:09:57Z 2020-11-26T07:27:03Z Thoughts on service provider rooted identities

When I started in datacenters, we had places where we used network addressing and layer 3/layer 4 firewalls as the primary way to manage service identity for internal services where we needed it. Even at that time, there were some annoying attack scenarios you had to deal with in multitenant environments and compromise scenarios attacking your layer 2 and 3 infrastructure (MAC spoofing, ARP poisoning, etc.) that meant even then it wasn’t a totally reliable source for identity information. Luckily, for those reasons and other trends, people have largely moved away from source IP as a source of workload identity.

Network Service

With the advent of high automated infrastructure systems and the birth of “cloud” infrastructure we saw other approaches to providing identity to compute workloads emerge. The most well known this is probably the AWS EC2 metadata service. Fundamentally this method boils down to using a link-local address to allow the workload to access an API to get information about itself over the network. This include operational characteristics and user data used to power cloud-config. Cloud-config powers a lot of the run time first boot features like executing arbitrary scripts, or configuring the system based on data provided by the metadata service.

However, if configured with an IAM role and instance profile, this service also provides a set of AWS IAM credentials that can be used to call permitted AWS services (based on the access configuration of that role) that are effectively accessible to any process that can source network traffic from the system by default. Another endpoint this metadata service provides is an identity document. This is a cryptographically signed document containing configuration information about the instance. Because this identity document is made available only to the workload itself this link local address people also use possession of this identity document as proof of workload identity.

This approach has some short comings, many of which these service providers have worked on strategies to mitigate. AWS on its documentation page for the metadata service (at the time of writing) calls out one of the largest ones. Any workload that can communicate over the network to that link local address can see all that data and those credentials. Outside of obvious attacks resulting in system compromise, there are a surprising number of attack vectors that allow you to make a vulnerable workload or systems perform a web request on behalf of the attacker – and with this network metadata approach all of them potentially expose your service identity/credential data.

It is easy to build broad API support for this credential discovery approach. Both the service provider themselves can do this in their libraries and SDKs as AWS does and others can do this using presigned URLs with instance role credentials or validating the instance identity document.

Application Environment

Another approach we see is folks providing credentials to applications, particularly beyond the IaaS level, is injecting data in the processes running environment. This broadly comes in two flavors, actual process environment variables or a file/filesystem provided within the filesystem presented process. I feel like the dangers of storing credentials and secrets in your processes environment variables has been documented extensively, but in summary both infrastructure systems and often times applications themselves (or their dependent libraries or frameworks) don’t protect the process environment as if they contain secret data in many cases. It is still common to find logs that freely log environment variables or send them as part of exception reports or debug error messages/screens. It is more rare to find applications that allow attackers to read arbitrary files from the file system, but they do exist – generally by honest developer mistake or misconfiguration rather than by design as many of the environment variable exposures appear to be.

These patterns are commonly used with Kubernetes secrets or config maps, and container PaaS/function as a service platform like Lambda. Particularly with the filesystem approach this feels like the most prudent path at this point as a service provider to me.

It is noteworthy that these Application Environment methods require more awareness and integration between the scheduler and the workload. In high level cases like function or container as a service, this probably feels natural. However, it gets more complex if you want to support many operating systems and depends heavily on the features available in your scheduler/hypervisor layer.

Why bother?

Very few workloads we deploy today are an island. Most communicate with service providers we leverage consuming PaaS services or other services we write and deploy separately in a modern microservice environment. Traditionally we dealt with long lived credentials that we stored treated as secrets used in the production configuration management or deployment processes.

There are other solutions to machine identity out here. Enterprise organizations have been dealing with Active Directory and their related machine account dynamics with Windows for decades. Even earlier than that systems like Kerberos had solutions to this problem. Most of the solutions that solve this do not fare well when applied to the Service Provider use case. I do not know any service providers today that operate with multiple customers sharing an Active Directory environment for instance. I think many customers would be uncomfortable with this scenario if they did.

As technology and security practices have evolved, the benefits of moving to short lived credentials have been wildly discussed. It caps the duration of risk related to someone getting access to credentials that have been used in your production environment. This includes attack vectors like someone compromising a production backup, snapshot, or set of log files that inadvertently included credentials for your environment.

Many of the advancements people think of as cloud are highly automated infrastructure and modern deployment practices. Giving customers a secure root of identity solution can encourage them to adopt other useful infrastructure automation patterns, like treating servers as short lived and replaceable rather than long lived and maintained. This sort of identity can be foundational to first boot scripts interacting with other services from your provider or reaching out to your centralized identity provider to get other credentials it needs to access other systems. Getting people to think about their infrastructure differently, rather than any technical capability is the most impactful improvement most organizations make during a cloud transformation.

tag:adamgreenfield.com,2013:Post/1589927 2020-09-02T15:55:20Z 2020-09-02T15:55:20Z Infrastructure for side projects

Today’s cloud landscape has no shortage of options when it comes to deploying HTTP-based applications. For small ideas and side projects I still have a hard time justifying running an entire container cluster, so I find myself reaching for options like Google Cloud Run or AWS Fargate because it provides me with a  clear path towards something like Knative if my small idea starts to grow into something more substantial. At some point if you have enough of these small workloads running it will start to create cost drivers toward building a cluster multiple workloads could share.

The minimum footprint for a side project is a small amount of CI/CD configuration in Gitlab, GitHub, Google Cloud Build or AWS CodeBuild, a deployment target (or several if you run a proper multi stage deployment process including dev, staging, and production). This makes it quick and inexpensive to get things up and running quickly so you can iterate on your idea. 

My toolchain of choice continues to change often because the technology landscape around PaaS and Serverless deployment options is evolving rapidly. AWS Lambda, Google Cloud Functions and similar offerings do not feel like the right fit and the tradeoff between ease of use and vendor gravity doesn’t seem worth it to me at this moment. I have built many solutions using these services and for many use cases they are a better fit than something container based, but I have committed technology sins by using serverless functions in places I should not have. That scar tissue is still fresh.

tag:adamgreenfield.com,2013:Post/600829 2013-09-11T18:15:04Z 2013-10-08T17:29:49Z Close to home

Several years ago, I was living just outside New York City. This particular morning I happened to be at my girlfriend's place in central Pennsylvania and we weren't even completely awake when her brother in law came to the door almost panicked. I didn't know him very well and didn't know what to make of it at first when he came running in. We turned on the television just in time to see the second plane crash into the towers.

I'll never forget the feeling I had trying to take a mental inventory of where my friends and colleagues were likely to be that morning. I started calling, texting, emailing (like everyone else at that moment) trying to make sure my friends were safe. Only a few were in Manhattan that morning and luckily they were all OK.

I will never forget

tag:adamgreenfield.com,2013:Post/578658 2013-05-12T22:17:22Z 2013-10-08T17:25:18Z Reflections on Mother's Day

I was raised by my mother and my grandmother. Throughout my life they have provided me with the love, caring and support to make it through even my toughest days. Anything good that I have or will accomplish is in no small part because of them. They both broke their backs to provide their children and grandchildren with opportunities and experiences that surpassed the ones they had growing up. They taught me right from wrong, strength during hardship, compassion and empathy, how to improve yourself, and countless other things that one could easily overlook. I carry these things with me for all the days of my life and even in times I lose my way it is these tools that I use to find it again. There is no gift, no card and no flowers that will ever truly express how much I appreciate what they’ve done and continue to do for both me and my son.

tag:adamgreenfield.com,2013:Post/344277 2011-12-06T15:42:02Z 2013-10-08T16:35:20Z Your words have power use them wisely ]]> tag:adamgreenfield.com,2013:Post/344281 2011-10-17T03:45:19Z 2020-09-02T16:45:37Z The Good Life The Good Life]]> tag:adamgreenfield.com,2013:Post/344282 2011-10-11T04:50:00Z 2020-09-02T16:01:48Z Forks in the road

Tonight I found out that another co-worker is taking the next step in their career and leaving our company. Selfishly I’m sad I won’t be working with them anymore, but honestly I’m happy for them and know they have a very bright future ahead.

What really got me thinking was the conversation we had after covered the basics (e.g. sorry to see you go, where are you heading, etc.). The reality is that most careers don’t follow the clean-cut path they have in the past. Not very long ago it was common to work for the same company for decades, leave with a gold watch, benefits and a pension. That definitely isn’t the reality I live in, and I often think about the going on five years I’ve invested in my current organization. Granted my career path has been pretty interesting over those years – but I’ve been at the same place.

Today you are responsible more than ever before for your own retirement, your own career and your success. My friend made the comment that loyalty doesn’t exist anymore, I responded that I don’t think it is gone but I think it has changed. Today’s loyalty is a bond between humans, not between a human and an organization. I think I like it that way

When giving me advice at one point a few months ago a senior executive whom I’ve grown to trust said to me that the only way to build the kind of bond he and I were talking about at the time, where near frictionless disagreement and collaboration are possible, was to go through “battle” with someone. He didn’t mean this strictly in the military sense however I can only imagine the result is even stronger in that case. He was referring to the absolute certainty to the guy standing next to you will still be there, fighting right along with you, even when things get ugly. In some jobs, that might be an everyday event or an extraordinary circumstance. Either way the result is the same, you learn who you can trust. You learn who you can count on and you learn whom you can’t.

I’ve lost another one of the people who fit solidly in former category. The good news for me is that you never really lose those people; you just might not know how or when your paths will cross again. In fact, I got a call from another one of the people I’d put in that category earlier today asking me to speak at a User Group his organization is hosting next month.

tag:adamgreenfield.com,2013:Post/344283 2011-10-08T20:37:00Z 2013-10-08T16:35:20Z 360 Feedback

One of the Operations Directors at work developed an application we have started to use to gather 360 feedback from the other people in the organization you interact with frequently. This tool basically asks for a number of objective criteria to be evaluated on a 1 to 10 scale and then provides the ability to give you some structured (but largely free form) data on things you are doing well and things you should consider changing.

I like this application and the idea behind it. When it is presented to you (and your manager) it shows you all the feedback but keeps the people providing it anonymous. I will say that I wish there was a way for me to ask questions (anonymously would be fine) about some of the feedback so I can really understand of it but most of it is pretty interesting.

One of the outputs it gives you is the graphic I'm showing here, it maps your certain traits based on the objective feedback. The other most interesting part is looking at the delta between your average rating from peers and how you rate yourself.
tag:adamgreenfield.com,2013:Post/344284 2011-10-06T03:11:37Z 2020-09-02T16:26:06Z Steve Jobs on Death

I hope most people have watched this amazing speach, but it is worth watching today. Steve Jobs had an incredible impact on the world in his years.

tag:adamgreenfield.com,2013:Post/344285 2011-08-10T18:01:29Z 2013-10-08T16:35:20Z The only reason people listen to me is because of my f-ing sweet hat

tag:adamgreenfield.com,2013:Post/344286 2011-04-17T18:58:39Z 2013-10-08T16:35:20Z Achievement Unlocked: Workaholic

This was originally a pearl of wisdom I sent to a co-worker who told me I'd have something "by the end of the day" on a Sunday:

After 10 years in different roles I've stopped wearing "workaholic" like it is a badge of honor. I work weekends if I can't avoid it - but it leaves me regretting the time I missed with my son and my friends. I work nights if I have to, because I need to carry my weight and not create undue stress on my colleagues (most of whom I also count among my friends).

tag:adamgreenfield.com,2013:Post/344288 2011-01-12T17:06:00Z 2020-09-02T16:32:26Z A little stuffed penguin

When I started traveling all the time, my son gave me one of his stuffed animals to bring with me. I carry it in my bag and wherever I go I try to take a few pictures of this little stuffed penguin in various locations. I was touched because it is one of his favorite stuffed animals and I smile every time I see it in my bag.

tag:adamgreenfield.com,2013:Post/344289 2010-12-17T22:28:06Z 2013-10-08T16:35:20Z Angrybirds: Dean's Favorite Game! ]]> tag:adamgreenfield.com,2013:Post/344293 2010-06-08T02:42:37Z 2013-10-08T16:35:20Z Nostalgia ]]> tag:adamgreenfield.com,2013:Post/344297 2010-04-13T15:04:28Z 2013-10-08T16:35:20Z Stop selling used cars

Today, I had a rare insight into the customer experience of someone trying to purchase a moderately sized managed hosting installation. Somehow in the many years that have passed since I first entered this space I've never stopped to think about how grueling the sales experience must be to have to do that shopping. Most every organization I've worked at that had a sales staff incentives them on the amount of hosting they sell. This makes sense because you want to reward sales people that generate revenue. However many of the sales compensation structures I've seen place little or no emphasis on the long term customer satisfaction and deep relationship developed by selling a customer the RIGHT solution.

People purchasing managed hosting often do so because they want to offload some of the responsibility of building, maintaining or managing their infrastructure. This is the way that many hosting vendors try to position themselves to fill for customers. So why does the initial experience so often begin with someone who doesn't have your long-term best interest as his or her primary motivator?

In many times where I've filled a pre-sales capacity for a customer I've built a deep relationship with that customer that spanned several roles in the company for me and the entire lifetime of their account. As an industry, we need to figure out how we can make that the normal experience, rather than a situation where people sometimes suffer through the sales process in the hopes that the operations experience and the technology they come out with are worth the headache.


tag:adamgreenfield.com,2013:Post/344273 2010-04-12T07:16:57Z 2013-10-08T16:35:20Z I wonder how much money...

we spend just advertising to get people to fill out the census. If the 3 post cards including the actual form containing the census and TV ads were not enough.... BOOM! Facebook Ads :)

tag:adamgreenfield.com,2013:Post/344275 2010-04-06T02:54:01Z 2013-10-08T16:35:20Z What it is all about ]]> tag:adamgreenfield.com,2013:Post/344278 2010-04-04T19:05:04Z 2013-10-08T16:35:20Z Improving my password security We all have *that* password. The one you try first when you are presented with a login form that you don't know your login for off the top of your head. Maybe if your a little more security minded you have a few of those passwords.

In the last few weeks I've been trying to force myself to adopt 1Password for improved security. So far so good. The idea is that you store your randomly generated unique passwords in a system that you trust and don't use the same password everywhere.

I'll let you know how the experiment ends up, but it is worth trying yourself as well. If you can get use to it - you'll be in a better situation when a malicious person discovers your password to some forum you registered for to make one post three years ago.
tag:adamgreenfield.com,2013:Post/344280 2010-04-04T08:56:00Z 2020-09-02T16:08:57Z To learn a new language

I've accepted a position at work that has me spending the majority of my time acting as Product Owner in our Software Engineering organization (which has adopted a mostly agile development process). The primary team I serve as product owner for (our customer portal team) has some of our most seasoned developers and is more or less self sufficient. My day to day oversight of their efforts is minimal if at all. I set priorities every two weeks, check is periodically and answer questions as they come up - but that is pretty much it.

Having a background in software engineering is it my inclination to learn the language we develop in (C#). I'm confident I could do it in short span of time, but I am worried the result will end up in me becoming too involved in our project for the team's comfort. I'm happy to let them drive technical direction and don't want to meddle in a team that is clearly producing, but I'm also someone who enjoys discussing software architecture.

I recently spoke with our CTO about my progress in my new role - and one of the things he left me with was that too often I go out of my way to "seem like the smartest person in the room." While this isn't something I realized outright, I do understand where he is coming from. He suggested in places where I might be inclined to make a declarative statement, I instead restructure my thought as a question to drive discussion. My gut tells me his advice here is solid, but I've always thrived in highly technical environments where challenging one another on ideas is the norm.

I wonder how his advice would be applied to my current situation, where learning the language might result in my inclination to start throwing out ideas for the team to challenge.]]>