The Dangers of Implicit Communication

The online chatroom has become a pretty standard fixture for many software development teams: distributed, remote, and colocated. They’re really fantastic tools that allow a persistent conversation to take place even as people’s attention switches in and out of the chatroom throughout the day.

However, a source of frustration I have experienced is a reliance on implicit communication. For example, if I ask a question in the chatroom, “Does anyone know why the cache server is down?” and nobody answers, I’m left with little information to act on. Did anybody see my question? Did someone see my question and just not know? Did someone see my question and just not care?

It’s made even worse when you ask a question and people are talking about something else. If the reader doesn’t know the answer, they may choose to remain silent so someone else can respond with more helpful information. The reader may believe this silence is communicating “I don’t know the answer”, but the reader has no way to differentiate between them not seeing the question, know knowing the question, or not caring about your problem.

In the real world, you can ask a question, and get a shrug or see a “whaa?” look on someone’s face, and get a lot of information just by being in that same room. In a chat room, forum, email, or instant message, you don’t have all those cues, so it’s really hard to know how to react.

Take a look at this pull request. The maintainer closed the PR without merge and without comment. What does that mean? Do they feel the PR is not in the spirit of the project? Do they think the quality is insufficient? How can the OP interpret the maintainer’s response or learn from it at all?

Implicit communication has a real potential to make people feel disenfranchised and ignored, and it’s incredibly easy to avoid. Sometimes a simple “I don’t know.” in response to a question is enough to spur a conversation.

If you’re asking a question, it can be effective to address it to a single person (“Tom, do you know why the server is down?”) in order to avoid the Bystander effect. You don’t necessarily need to expect that person to know the answer, but it will often ensure you get at least one response.

Role Based Configuration Management with SaltStack

When I first started using SaltStack, I started with a top.sls that looked like this:

But I didn’t like having the roles of a machine hard bound to the hostname of the machine. That seemed arbitrary. Also, we ended up wanting to split and merge roles and the machines they run on (for instance, it became reasonable to merge the cache and worker roles) and so we outgrew that.

To solve this problem, we introduced role grains. An example worker minion might have the following `/etc/grains` file:

With this, our topfile turned into this:

Which was nice. For each role grain, it included the necessary role state file, and all was well… for a while.

Our environment grew to the point where we had many more roles than we at first anticipated. For each role we added, we had to add another 4 lines to our top.sls, and it grew pretty ugly and duplicate-y. To solve this, we leveraged the jinja templating system to create roles.sls:

This allowed us to condense our top file to the following:

Now each minion tells the master what kind of machine it dreams of being via the roles grain, and the salt master tells each minion how to become those things. If you want to add a role to a machine, you simply add it to the role grain on the minion and run highstate.

Managing a SaltStack Salt Master with Git

UPDATE Sept. 18, 2013 – I no longer use the method described here for managing my saltmaster. I simply use gitfs to manage a salt-states git repo. Maybe one day I’ll describe that setup in a blog post.

When I first started out with SaltStack at rover.com, I was a little confused as to how to manage the configuration and state files in source control. Stuff is strewn all about the system on a salt master, so there wasn’t an obvious “just check stuff out here” approach.

I wanted to keep everything in one place, and by everything I mean:

/etc/salt/master – salt master config
/etc/salt/master.d – more salt master config
/srv/salt – salt state files go here
/srv/pillar – salt pillar stuffs go here
/srv/reactor – salt reactor stuffs go here

There were a lot of gotchas, and I bet someone could golf this for me, but the solution we came up with has been working really nicely so far.

Please note my salt master is running on Ubuntu 12.04 on as an EC2 instance. This may or may not matter.

Now let’s talk about this, cuz it’s kinda weird.

Line 7 – This is a local directory where we’ll check stuff out of the repository.
Line 9 – This is a bare repository that we’ll be pushing into. Create it with `git init –bare`
Line 11 – Since this is a post receive hook. After you’ve pushed into your bare repository defined on line 9, it will do a checkout FROM that repository into the directory specified on Line 7.
Line 14 – Copy the salt master configuration into the /etc directory
Line 15 – Copy the other config files into master.d. Note: It doesn’t delete files that aren’t there anymore. It probably should.
Line 18:19 – Probably more paranoid than I need to be, but I make sure the config files have the proper permissions and file ownership.
Line 22 – Restart the salt master to load up all the new configurations
Line 25-34 – Detect if there has been a push into the highstate branch, if so, merge highstate into master, and trigger a highstate on all minions.

Also, for this to work, you have to create 3 symlinks from /srv/(salt|pillar|reactor) into the directory specified in Line 7 where those files will reside after a checkout. You also have to copy the post-commit hook into the bare repository specified on Line 9.

Now, on your local dev box, you’ll have a git repo you’re working in that contains all these state files, config files, etc. You’ll need to add your salt master as a remote to that repo.

james@LocalDev:~/Development/production (master)$ git remote add salt ubuntu@salt.example.com:salt.git

With that, you can now push directly to your salt master.

git push salt master – To update the saltmaster with the new files.
git push salt master:highstate – To update the saltmaster and trigger a highstate on all minions.

So that’s it! Good luck. Let me know if you have any problems.

Stay Salty! Stay Fresh!

Personal Kanban – Why Do We Pay Bills?

My partner and I have been using a personal kanban board to manage certain aspects of our lives. He tracks his career stuff, I track my career stuff, and we collectively track household tasks like paying bills, planning vacations, and cleaning the garage.

Yesterday, we did an exercise where we took all the items in our backlog and clustered them around their associated goals. We wanted to make sure that the things we were doing were contributing towards our long term ambitions. Alignment between words and actions can be incredibly difficult to maintain, and it turned out to be a very valuable exercise. There were several tasks we couldn’t come up with a reason for doing, so we decided to either not do them, or to delegate them.

The household stuff turned out to be kind of tricky though. Why do we pay bills? Why do we clean garages? Why do we send address update notices? There’s a specific answer to each of these questions (e.g., so we have electricity, so we don’t get rats, and so I continue to receive my model train magazines), but they really felt like they all belonged together.

After some discussion, my SO and I decided the goal for these tasks was, “Maintain Quality of Life”. There’s a certain level of maintenance work that is required to enjoy the lifestyle we prefer (an exciting lifestyle of being in bed by 9PM and having electricity). It was interesting to consider it in those terms, because I’ve never consciously made decisions about what “quality of life” I would prefer. It’s completely subjective, but it gave us a chance to talk about things that we implicitly wanted with respect to the more mundane (yet still very important) aspects of our lives. It felt revelatory in a very small way.

How else could we have categorized these tasks? What other goals might they fit under?

Unemployment Protips: Calculating Your Runway

An important question for those without jobs is “When will I run out of money?” The prospect of moving into your parent’s basement is terrifying if effective motivation. The clichéd metaphor for this is “your runway”, the idea is that a plane taking off (or landing) doesn’t have enough of a runway, it runs off the end and crashes into flames. Cheery!

The general idea is to calculate your burn rate, the rate at when you’re spending money per day/week/month. You can whip out a spreadsheet and put in all your information and do a bunch of math and blah blah blah. You can do that, and it works, if you want to pretend it’s 1994.

We live in the future! There are tools for everything, and there’s a tool that makes this calculation ridiculously simple: Claratii.

Claratii does a lot of stuff, but you can explore that on your own. We’re on a mission.

After you create an account, there are 3 steps to getting an estimate of your runway.

Step 1: Enter Current Account Balance

Thankfully, the lack of further income makes the data entry simpler. That’s a bonus, right? Simply enter the amount of cash you have on hand as your balance.

Claratii Accounts

Step 2: Enter Expenses

Now just enter in all your monthly, weekly, and upcoming one-off expenses. This is an estimate, so for things like food and entertainment, make an educated guess or review your last month’s bills for a good idea of what they will be next.

Claratii - Expenses

Step 3: Project!

That’s pretty much it. Now you can check to Projections tab and see when you’ll be destitute, homeless, and/or sharing your parent’s basement with their terrible cat, Felix.

Claratii - Projection

One thing to keep in mind is, like all estimates, it will only be as accurate as you are. Also, it’s just an estimate, I wouldn’t recommend cutting things too close.

So head over to Claratii, create an account, and find out when you get to snuggle with Felix.