This was originally posted in June of 2007 and I have not updated it since, however this list is still pretty applicable. I'll update it soon...
(85) Operations Rules to Live By
1) Capacity First - Optimizations Later - this rule broken will guarantee downtime. Do not optimize under the stress of downtime - focus on capacity first.
2) Make sure you have every net available to catch you – Postgres examples - WAL Files, Slony Replication, Snapshot technologies, Disk based DB versioning (spinoff of Snapshots)
3) Do not ‘Optimize’ problems into your architecture. Often things created to solve problems turn into operational albatrosses to maintain later. Make sure that tools developed in engineering for operations are handed off completely. It usually doesn't work to go back to eng for further development. More importantly, the change requests can be disruptive to well scheduled engineering plans.
4) Keep it simple. Keep it simple, because you are smart. Do not make it overly complex because you can.
5) Caching should be used very sparingly and really in order to protect resources that are hard to horizontally scale. If however, you can scale something horizontally, rarely is it wise or prudent to add a caching layer. If used, it should be to gain performance for the end user not to gain capacity for a web site; otherwise, you have just created another bottleneck that has very unclear limits of capacity. They can have potentially negative impacts to the system as a whole. Cascading failures often come at the result of a failed caching layer.
6) Don’t code everything in house; do not buy everything from vendors - use the right tool for the business at the right time
7) Negotiate - the only way to negotiate with real strength is to have done the homework of creating options that are viable so that you would be willing to walk from your preferred vendor if necessary. Do not bluff.
8) Always N+1 and if N=1, do not under any circumstances use +1 for anything but waiting for N to fail. When using the redundant server to 'share' the load, the management of 49% capacity or rarely managed. There are many times when opportunities to have N+2 arise – take advantage of them.
9) Data loss is never a risk the company can take - this has been universally true. The cost of lost data far exceeds the cost to ensure data cannot be lost
10) Parallelize whenever and wherever you can - this is an important way of thinking when it comes to multiple locations. For example, If MogileFS were setup to be location aware and needed to replicate its data in real time, it should work so that each MogileFS server can replicate its data to the entire load balanced farm of MogileFS servers at the other end. Implement many to many wherever possible.
11) RTFM - To this day, I will read the entire manual of a pair of RAID cards to see what the subtle differences are. The devil is in the details. Do your homework
12) Know your bottlenecks and know how to spot them - every layer - know if you are blocking on disk, RAM, or CPU. It is usually that simple.
13) Have a regular capacity management process - Do not react. Be proactive. Knowing where your soft spots are is critical to staying ahead of the capacity curve.
14) Don’t promote failures and do not fear change
15) Do not breathe your own exhaust. Do not think that the output of your work should be the motivator for how you do things in the future.
16) Ops people that code should write ops tools, not application software
17) The value of a project manager, tech writer, and financial analyst in the ops organization should not be underestimated. They will more than pay for themselves.
18) Monitor EVERYTHING - alert on actionable only, record other for trend information
19) Have a regular process to look at trend data everywhere
20) Do not make the monitoring so noisy that it becomes useless
21) Ensure your monitoring system is so simple EVERYONE in the company can easily use it. It is surprising how often ops metrics turn into business metrics, marketing metrics, sales metrics, etc.
22) Do postmortems only if the people that can make changes are there. Otherwise, it is a waste of time.
23) Publish your postmortems. Attach the event data to the postmortem so people can easily look at a Keynote for example and connect the incident to the data.
24) Assign people to be point people for every bit of technology.
25) Assign backup people to those primary people.
26) Hire constantly - even when you do not have headcount - always be hiring.
27) Be your own harshest critic. You can always improve no matter how smart you are or think you are.
28) Compare yourself to as many companies as possible. Look outside of the company.
29) Pick one Tradeshow/Conference, only one, per year and go. If the one meets more than once a year fine, but pick one and only one.
30) Buy what you need, not what you want. Never ever, take off your corporate hat and leave the “what’s easiest and safest for me” hat on.
31) Do what is best for the business, always, even if that means you should not be there.
32) Formalize accountability - record commitments, circle back on those, expose commits not made.
33) You should not get more than one or two times to fail. There is goodness in fear. There is goodness is some fear, but manage the difference between chronic mistakes and honest mistakes.
34) Be ruthless - your competitors are.
35) Think of the work you do as something you want to sign your name with when done. That also means finish the job.
36) Be available for others.
37) Partner with startups - give them your expertise and scale and you will be rewarded with free product, sometimes for life.
38) Capacity is a business/product issue. This means the net cost per page/post/login, etc. must be visible to make the right business/product decisions.
39) Always beat the budget. The Operations group is usually the largest spender of discretionary money. Revenue targets are often missed, but the Operations group has many ways usually to defer spending, etc.
40) Because something sucked in the past, does not mean it will suck today or in the future - try things and have the tools to test
41) Documentation - document everything - well. Make it so new people should not have to talk to anyone to learn everything
42) Create gigantic plotter size drawings of the physical layouts of your data center
43) Create gigantic plotter size drawings of the logical flows of each of your products
44) Faq-O-Matic, Wiki, something where people can easily post “this is how you fix this” and make it very findable. This is where a tech writer comes in handy, but the idea is to make incremental, even if informal, documentation easier along the way.
45) Make sure everyone, truly everyone is totally replaceable
46) Most people get more done at home, than at the office, some people do not.
47) Bundle your orders - you get to ask for the most discounts, contract terms, etc when you have batched your hardware orders. Ask for everything - price locks, spares kits, lease terms, everything before they get the PO.
48) Develop long term relationships with your vendors - make sure you can call them at your next job
49) Give everyone in ops everything they could ever use to make them useful remotely - Treo’s, EVDO card, dual 24” LCD Panels, ANYTHING and EVERYTHING. With talented people, the costs will be more than paid back by effective remote employees. Remember Ops (and Eng) are power users, understand what a pixel is and will make the most of screen real estate.
50) Be a complete stick in the mud with IT standards. Until the Mac runs office 2k7 and outlook, you have to run some form of windows. Period. Unless it is *all* Macs - it destroys office productivity for meeting/calendaring, contact management, mail lists, etc. If an employee is willing to stake their job on running an instance of XP on parallels, fine. This is very rare This rule is now dated/moot and stick in the mud isn't necessarily the best approach. This list is very '2007'
51) Have a streamlined purchasing process - know your budget, make sure you get to manage to it. Get the 'actuals' report from Finance. There is a gap between technical driven budgets/reports and financially driven budgets/reports. A good Ops manager can create models that bridge those different views into COGS. A CFO who understands is a supportive CFO and helps drive business decisions.
52) The weekly meetings must have continuity. Items from previous meetings committed to then must have accountability.
53) Create a separate escalation system to involve engineering into the problems with code that effects operations negatively. The idea is ops issues that escalate into engineering, will either get lost in any ops ticketing system or any engineering ticketing system. Create a separate ticket queue and manage to a cleaner and simpler list.
54) Incorporate Operations staff in every stage of a feature or product development from the design stage so that scalability, monitoring, and reliability are baked in to production. This also makes sure operations is responsible for hardware procurement, monitoring systems in place, run books written, etc and that it launches on time and according to operational standards.
55) Practice for being a real company - Sarbanes, WebTrust, SAS 70, Visa, Banking, etc. Do not forget that if you are successful you are going to have to deal with this stuff. It is easier earlier in the process, even if it is nothing more than awareness and knowledge vs. any real change early on. Deploy a ticketing/task tracking tool. USE IT. Put change control/change management into the same system. USE IT. Keep putting info in there. It helps to figure what “what changed in the last week” as an example.
56) Do not make it hard to have redundancy or multiple locations. Things are hard in the beginning, but do not slow your success down with bad architecture that does not allow true scale and reliability.
57) Oracle Standard Edition (MS SQL Server Std.) is affordable. If you can constrain your use of Oracle to Standard Edition, there is no reason that a viable business even a small startup cannot use it.
58) Postgres and MySQL are free for a reason. If you do not really care about transactional integrity, MySQL is fine. Until the mandatory chaining of the words ‘Vacuum’ and ‘Postgres’ is broken, Postgres represents an unpredictable, usually negatively surprising database.
59) Capacity should be measured at the peak second of the day +20-30%. Unless you are a vmotion guru that is the way it goes.
60) Read as many trade magazines as possible. They are usually free; the only ‘price’ to be paid is to answer their somewhat exhaustive surveys. The value of what is new is huge. Have them delivered to your home. The chances of them being read at work are next to nil.
61) Care about security. Engineers should not be in production. Engineers should do code review. There should be separation of duties in ops. There should be people in ops that can easily revoke access to other ops staff. Create an employee handbook that states there will be serious consequences for breaches of security. Build everything from the ground up with the security and privacy of your customer’s data in mind (physical, logical and functional). If you had your customer in a courtroom and you were on the stand, would you feel good about what you had done to make sure the company’s and customers’ data was protected with vigor and diligence?
62) Control your access points. First, people must be able to do their jobs; secondly, make sure you know exactly where they are coming in. Implement two factor authentication methods.
63) Keystroke logging is critical for bastion/gateway hosts that people must go through to get to production hosts. This is harder with Windows, however TS gateways helps and hopefully there are auto screen recording options soon.
64) Make sure you have many redundant ways to get to production if things go badly. Do not assume the corporate VPN solution is going to get you to production when the corporate network breaks. Put VPN devices directly in production.
65) Use LDAP for auth. Even if you have just 10 or more machines and you are copying passwd and shadow files around, you need LDAP (AD).
66) Do not discount how helpful a Windows Server 2003(2008) R2 machine can be in a UNIX environment. Just because you do not know it, does not make it bad. Learn it.
67) Do not waste people’s time with ineffective wireless solutions. People are mobile. They want to work on couches, in meetings, un-docked, what have you. Make sure wireless is solid.
68) There will always be people that put in extra effort and time - make sure you tear up their vacation request forms. There will always be people that do not put in the extra time, be diligent about getting their forms. Operations is asked to make a huge sacrifice of personal time on a regular basis. Manage ‘comp’ time by gut and reward the people that are woken up at 3am, are online quickly and happy to fix something.
69) Implement an RDBMS for tracking all your production assets. Have a centralized, then replicated source of truth for everything. Assets, Staff, Networks, Contracts, everything. Replicate this database offsite. This means a working copy somewhere, not daily backups to tape.
70) Automate as many processes as safe to do so (e.g. OS or product releasing, pushing of files, grep of logs, etc.)
71) Automation should always be using the Operations’ RDBMS (Source of Truth).
72) There are usually three states of a machine - Out of Service, In Service, and In Production. In Service is the state where it is being brought up to speed with cfengine, rsync, what have you. In Production is when it is taking traffic. There needs to be a state where a machine can have information collected from or tests run against without taking production load.
73) Respect your log data. Have methods to harvest logs off machines you need to take out of service/rebuild, etc.
74) If you are growing fast and don’t have a lot of time to ‘optimize’, lock stuff down as you go - when you have a process or procedure that works, don’t change or modify it until absolutely necessary at a later stage. Essentially - default to locked down, grow, and then revisit only when necessary.
75) You will never avoid an Ops engineer blowing away a critical part of your infrastructure by mistake - it happens - take some of the most critical parts of your infrastructure and run a “what if Joe Random does a ‘rm -rf /’ on our XYZ box?”
76) Try to keep stuff fun and interesting for your team - if they are not enjoying what they are doing, they will go find that buzz elsewhere. Let the team know they are also responsible for this and this is not just a management task. Boss’s need the buzz too.
77) The true value in having the ability to deliver on 99.999% uptime is being able to stay nimble. It means you have the ability to utilize redundancy whenever you need to. This allows physical changes, location changes of equipment, code changes, rollback changes, etc. The value of this is huge to a company, more than the value to the customers.
78) If you can deliver 99.999% - Give the customers a 100% SLA
79) Never drown the ability to release software in process. Drown yourself in the ability to rollback or failover to an older version of code. You will never ‘process out’ failures. It is futile. You can however do tremendous things to cover your butt when things go bad. CYA = Staying Nimble = Successful Company.
80) Keep in mind the why’s and what-for’s of what you’re building for the customer every step of the way - whatever you set out to deploy for the end users, keep that at the forefront throughout and design everything (infrastructure, processes and staffing) for the purpose of delivering the best of breed service or product.
81) Do it right the first time. Rarely do you get the chance to go back and redo things. If you do, it comes at a very big cost to the company. Take the hit on work, the first go round.
82) Reach out to partners, allies, and similar companies in the industry and see how they run Operations. Chances are they will have the same challenges as your organization and maybe have better solutions too. Do not be afraid to share your experiences or processes, as they will probably reciprocate.
83) Hire people that are so good that they make you nervous about holding onto your current job. This goes beyond hiring ‘A’ players. Hire people that you admire and want to learn from. Hire people that you would be willing to work for someday.
84) IT and Operations are two completely different beasts. A solid Operations manager/exec can manage corporate IT, but rarely does a traditional IT person have the abilities to manage Internet Operations functions.
85) When starting a new job, or each year at your current job, fight for budget. That doesn’t mean utilize the squeaky wheel approach, it means have a good documented case based on historical information. If you are evaluating a new job, make sure you know *exactly* what your budget is and what it is based on. Also, ensure that you have the ability to change the budget should the landscape change.
10/20/2010
85 Operations Rules to Live By
Posted at 11:18 PM | Permalink | Comments (0)
01/13/2009
12/16/2008
03/22/2008
Rock Band - Will it ever stop?
Rock Band and Guitar Hero I, 80's, II, and III have completely sucked me in. I like to bounce back and forth between the two games, but I have a secret love for GH III as a better game. There are a lot of variables in being a good game. Rock Band is really fun, and awesome with lots of people, but GH III still just put together really well. Rock Band has one thing going for though and that's new content. As I write it's a 129 songs of which I have 125 taken care of on expert. Great new song called Shockware by Black Tide is currently holding my attention. Here are some shots. First one is Green Grass - a song that for a very long time now has kept me at the same exact percentage number of being complete. It's really dirving me nuts. I feel like if I figure this song out, no other song will be hard. Second shot - uh, yeah, my living room. No real explanation needed here, I have issues. Last is just a little proof of me putting my time into this and taking it seriously. ;-) Oh yeah, my avatar - I look just like that.
Posted at 08:29 PM | Permalink | Comments (4)
03/17/2008
Oh yeah, I remember doing stuff like this...
So I happen to be a big fan of Heroes after my wife gave me the now defunct HD-DVD Season One box set. Season 2 is not out yet on (HD)DVD. But, I found out that netflix was simply streaming season two and I got sucked in for 11 episodes, otherwise known as "My Sunday". Aimee was actually being productive and taking care of things which left the kids to either be asked to do some chores with mom or hang out with me being a slug. Well you know how kids are. They don't do 'slug'. So this first shot is Jennifer, my youngest, telling me that she is in a boat, fishing. By the time I got my camera to take a shot the fishing pole had turned into a mustache. Pretty classic.
Of course having 2 laundry baskets meant Jennifer had options. She told me she was caught in a net this time (somehow turned into the fish she was trying to catch earlier). I told her, "Sorry Honey, you look like a hermit crab to me." She thought that was pretty funny and ran with that idea for a few.
It was nice to remember some of the silly stuff like this I/we did as kids. I used to play fort a lot with couch pillows, tables, chairs, blankets, and anything else I could grab. Dad always enjoyed that mess. ;-)
Posted at 09:26 AM | Permalink | Comments (0)
03/13/2008
If it quacks like a router...
I've been struggling lately with what is supposed to be quite a simple task:
1) Build a Windows (in this case 2008) Primary Domain Controller
2) Build a Windows (in this case 2008) Routing and Remote Access Server
Starting from scratch with fresh installs, no problem. I tackle the PDC with ease. It's happy and letting people and things join the domain at will.
The PDC was supposed to be the harder of these two tasks.
NOT
4 Days - beginning to end I tried to make a working RRAS Server. I tried 2008 with many configurations, everything from vanilla configurations, like it should be, to outlandish ones in hopes of something working.
What's the problem? Well, simply put - When one is working with a router, one should be safe to assume said router will actually route packets between two seperate networks. Silly me! All these years working with normal routers and I have to stumble on a router that doesn't forward packets.
Here's the rub - I back down to Windows 2003 and set up RRAS, with of course, the same problem. The crazy thing is there are two screens (which I'll post sometime) for configuration of the RRAS server. The first one asks quite simply "Router?" with a checkbox. I click the box. Then inside the IP Protocol there is another screen with another question of "IP Router". Since I'm not playing with banyan vines I click the check box next to IP Router.
That's two out two check boxes that quite simply ask if it should be a router, with a unanimous vote by me for Yes! Be a Router!
It told me to pound sand, just as the 2008 server did and just as my Windows 2003 Server at home did. What the hey?
So I finally cave. I'm a registered microsoft partner and have been for many years. I've never used the support that comes with the program and I figure - ok, why the hell not use it for once. So I call them. I happened to be at home sick and working on my 2003 server at home. They asked who's server I was working on, said my own. They said "Sorry we can't help you because it is your server." I scrambled and said I had two other servers at 'my customer's' site and they have the same problem. Nice Save. Got me through.
I worked with a pretty cool tech from MS - normally they charge 260 a call so that'll pay for some educated support staff.
We work through it for a while, we do all the normal diagnosis, and he gets to the point where he sounds like he's a little out of ideas. He asks me if he can put me on hold for a while. I said no problem take your time.
He comes back and says "Please open regedit" I said "No Problem" because I simply had no more will power to fight anything at this point.
He then says to go to HKEY_LOCAL_MACHINE then SYSTEM then CurrentControlSet then Services then Tcpip then Parameters.
Yeah, we're pretty 'in there'
He asks "Do you see a ipenablerouter entry?" I do indeed see a DWORD for ipenablerouter. I then look over to the right and in utter disbelief I see a big fat zero instead of a one. He tells me to change it to one (I had already figured that out) and to restart the Routing and Remote Access Service. Yup - works like magic, of course.
I could have been more grumpy but I was just so happy to have this problem behind me. Next to see if that fixes it on 2008 as well, but I have 2003 as a viable backup.
Just a little FYI if someone else is struggling with Windows Server 2003 Routing and Remote Access Server or RRAS and needs help, use regedit to enable ipenablerouter=1
Posted at 05:16 PM | Permalink | Comments (1)
01/21/2008
12/27/2007
The little network that could...
When I first arrived at Media Machines, they were renting cubes from a company called Coghead in Redwood City. They had a wireless access point and that was connected into Coghead's network. All basic, except for one important aspect - Network performance was so bad it was quite literally useless. Coghead had a couple admins but they didn't know what to do and it was clear they weren't going to figure it out anytime soon. Since Media Machines was only about 8-10 people, I figured I would give my Junxion Box a shot.
The Junxion Box had a EVDO Rev A card from Verizon. Performance was great and people were really happy to have something that works. Remember this is 8-10 people sharing at most a couple mbits and now being happy.
Media Machines grew and moved. We couldn't get any type of Internet connection except long range wireless which is supposed to be turned up tomorrow. As we grew I was asked to get 2 more Junxion Boxes so people could associate to a specific one and get some connectivity as the company was maxing out a single EVDO card.
I was working with Les, Ops Mgr, on a thought around "Why can't put all these together somehow to have one network that utilizes as many Junxion Boxes that we have?"
So with that in mind we did exactly that. We dusted off the original Cisco wireless access point and made it a DHCP server and router. We set the IP addresses of the Junxion Boxes and created a separate network for clients to use internally. Then we made sure Cisco Express Forwarding and destination-based load balancing were enabled on the Cisco AP and simply set up 3 default routes to each of the Junxion Boxes.
It's not been 'great', but we have 25 people operating off this network and have been for weeks. I still think it's pretty cool to have that many people off of what is usually meant for a single client's Internet connection.
We did have to switch from Verizon to Sprint however. Verizon's "Unlimited" data plan is not unlimited at all. They used to disconnect accounts but got into legal trouble and had to cough up a million bucks to those disconnected, etc. They now define Unlimited as 5GB per month. After 5GB they will not stop your card from working nor disconnect you, they will slow it down to the point of it being useless. Anyway, turns out that Sprint is truly Unlimited, so word to the wise. Just a point of reference, as an individual, I have never come close to 5GB. When I had the Junxion Box setup the very first day, the 8-10 people did 1GB in 4 hours. To add insult to injury, we have a corporate account with Verizon and they permit even less data transfer and charge more (tried to anyway) than consumer cards.
Just a story for startups in a pinch or even for bigger companies who need a "In case of emergency break glass" setup. Our Junxion Boxes aren't going anywhere, just our default routes.
Posted at 09:28 PM | Permalink | Comments (0)
10/16/2007
I would like to break this record, Cessna pilot needed.
On 10/15/07, david benett wrote:
They even had a Cesna flying ahead of them
http://jalopnik.com/cars/speed-record/alex-roy-reveals-transcontinental-run-claims-record-310735.php
-----------------------------------------------------------------------------------------------------------------
On 10/16/07, Ben Lutch wrote:
I still claim my 3 hrs 21 min (verified by you Tongue) to be the fastest UCLA-to-MTV run practically achievable. These dudes' average speed cross-country was ~90mph, and mine was closer to 100mph (forget the exact mileage). That was also the run where I hit 175+mph on I-5
Now driving a 525i with 3 baby seats,
b
-----------------------------------------------------------------------------------------------------------------
On 10/15/07, david benett wrote:
I can attest to the time.
I called him on his land line in Mt. View 3 hours and 21 minutes after leaving Westwood.
-----------------------------------------------------------------------------------------------------------------
On 10/15/07, Jon Prall wrote:
The Cessna is officially cool. That plane is refueling a lot as well though.
I've done this exact trip (Napa to northern Massachuesttes and Napa to Sebring Florida (Central FL)). Both trips took me less than 48 hours. Both with cars in tow though.
There are two alternative vehicles that would be interesting. My truck, which has 120 gallons, but could be changed to have 300 gallons. Top speed is 100MPH but it can store food, has a bathroom and with two people there wouldn't need to be any stops whatsoever.
With 300 gallons and the truck getting 9 miles to the gallon, you still get 2,700 miles before you have to refuel. Having more than 300 is also easy.
So that trip is simply a set the speed at 100 and don't stop. Downside, there are lots of times where you simply can't go 100 with the truck and you can't make up the time by driving faster than 100.
Next Vehicle: A motorcycle. Currently a BMW motorcycle with a top speed of over 173 miles per hour, has a 7.1 gallon tank and get 40mpg (when going really fast) netting 284 miles. Even if ugly, a sensor goes into the main tank, when it's is low, it turns on the pump of the auxiliary tank which then fills the main tank. A standard accessory is a 'top case' from bmw that sits high above, but behind you. It holds 13 gallons. A bladder situation could easily make this a total of 20.1 gallons netting 804 miles between fill ups. You would need to fill up 4 times and there would be plenty to spare. I give the m5 8 fill ups.
The motorcycle has some serious advantages if it has the range of 800 miles. Coast to coast = always hit traffic in some city. Motorcycle = lane split.
That motorcycle does 173 like it is nothing: http://www.youtube.com/watch?v=tqvkNDx6JwM
I've driven the coast to coast thing enough times to know that it is always a two lane road and *constantly* you are are stuck between an asshole on the left hand lane and semi-truck in the right. It's a little sketchy, but you get used to lane splitting a semi and a car.
Motorcycles are small on radar too. That said, they are more than capable of carrying a ridiculous array of gadgets and communication devices.
Downside, gotta do it solo. 31 Hours is a long time for one but doable.
Also, the m5 is fast, but the ability for the K1200S to just jump to 173 and back down is far easier than in the m5.
Totally tempting. Would be fun as hell. Who's flying the plane for me?
jp
----------------------------------------------------------------------------------------------------------------------------------------
On 10/16/07, Ben Lutch wrote:
Had you asked me 10 years ago, I would have been headed to the Palo Alto airport to get a license.
Now, I am headed into the kitchen for a cheese sandwich before I crash out.I am a lesser man
b
-----------------------------------------------------------------------------------------------------------------
The brutal reality of this endevour can be found here:
http://www.wired.com/cars/coolwheels/magazine/15-11/ff_cannonballrun?currentPage=1
Posted at 06:23 PM | Permalink | Comments (0)
Recent Comments