I've hit the limit of "my" AWS Graphite single server install (i2.xl) and I needed to come up with a scaled out design. I'm really looking forward to Jason Dixon Graphite's book, but can't wait for it so I went down the interwebs rabbit hole (1, 2, 3, 4, 5, 6, 7). This is what I've come up with for my needs, maybe this helps somebody else out there, also hoping for feedback from the community.
About a year ago, I was interviewing quite a bit and many of the jobs had "DevOps" somewhere either in the title or the description, unfortunately several times I felt like the companies that were trying to hire me hadn't done their homework in terms of researching what DevOps is about. Luckily around that time Gene Kim was presenting at an SVDevOps meetup and gave an excellent tip about gaining fast insight about a company status in terms of DevOps practices: "Cycle Time" (aka "Lead Time") and "Deployment Rate".
tl;dr: grab my GIST for VIM and bash prompt setup.
In these days of highly automated systems, I work with several languages/DSLs and environments, Ruby, Python,Puppet,Ansible, YAML, JSON, Ubuntu,CentOS. In order to optimize my workflow, I've customized my VIM and prompt setup quite a bit (all with open source code) and so I thought to share it. I usually work from an Os X laptop (as my host for Linux VMs managed via Vagrant)thus I'm including a couple of tricks for iTerm2, finally some bash prompt goodness.
Create a Pingdom http check that points to the /health end point ( you'll need to enter your sensu dashboard auth in user name/password in optional settings) AND select in optional settings/request headers "Accept" and enter this in the empty field
took me a couple of hours and some tcpdumping to find this out so hopefully it saves some time to other folks out there!
Today I wanted to install the MCollective Process Agent plugin and so I searched to no luck for a native OS package for ubuntu (ended up filling a ticket with puppetlabs). I was on my way to do some effing package making but some guy on IRC ("rc" you know who you are ^^ ) saved me a lot of time with a single mco command. It turns out MCollective can create native os packages for its plugins, and since I haven't found any doc on the topic I thought I document it here.
Today while exploring another revolutionary-ultra-scalable-and-easy-to-setup tool I decided the amazing-and-incredible amount of gobbledygook I found on that page justified a post, albeit a short one.
As a "systems guy" I spend a lot of my professional time deploying and/or reusing code/tools somebody else wrote, my point being reading technical documentation is a second nature. It then seems logic to think I'm a target of such docs and so here is for the feedback loop...
I'm currently implementing a monitoring pipeline based on Sensu for the monitoring data routing and Graphite for the actual storage and calculations, it wouldn't be complete without a notification systems to it so I thought I would document using PagerDuty at the end of the pipe.
There isn't much to it but the information is spread and not always straight forward (at least it wasn't to me) so hopefully this help other adepts of monitoring love!
If like me you're in charge of the vagrant base boxes supporting a team with folks in different time zones you might find this hack useful. I use it to bring the time zone of a RHEL 6 or cousins (tested on CentOS 6.4) guest VM to the same as an Os X host (tested on 10.7.5) in Vagrant, VirtualBox provider environment. I'm leveraging an Os X CLI utility with a shell provisioner but it shouldn't be too much of a deal to refactor for Ansible/Puppet/Chef provisioner. I'm pretty sure there are better ways of doing this, so if you have a more elegant solution please describe it in the comments!
Ansible, the configuration management and command orchestration tool I use currently, has several modules and ways one can use to push plain text based configuration. I got a lot of help from the Ansible community, the guys on IRC are quite a model when it comes to the S of CAMS and I thought I would share back the knowledge I've gained.
While thinking about Devops culture metaphors (I like them a lot and have used and abused them through out my career) I enjoyed David Lutz post "Running IT like a rock band" but there was something about that metaphor that bothered me. I also remember a tweet stating something along these lines "Show me your rockstar developer and I'll show you your bottleneck" and the analogy of a soccer team as opposed to the Rock band started to materialize. When John Willis presented at the Silicon Valley Devops last month his 100% culture focused "State Of The Union" presentation added another layer. In particular his emphasis on systems thinking (the fact that I've always had "systems" in my job title is most likely not a coincidence).
It then was clear to me why I was having this second thoughts about "Running IT like a rock band", that is, I like to think of the IT Industry as a science based industry, and in that regard I think a soccer team has much more in common with IT than a Rock Band, here is why.