Originally posted internally 12 Jan 2016:
What Would Google Do? It’s a good generic question when considering any problem in the IT space.
Often the answer is pretty obvious, where Google’s already doing something (and better still if it’s published the hows, whats and whys). Other times there’s a shape of an answer, where Google can be seen to be doing something, but it’s less clear how they’re doing it.
There is also a generic answer. Google is a data driven organisation (arguably often to the point of damaging itself and its users), so the answer to all questions is driven by data. If there’s no data then the first job is to get the data – making the mechanism to source the data if need be.
The alternative to WWGD is the HiPPO – the Highest Paid Person’s Opinion. There are a few problems with HiPPOs, which is why they’re best consigned to 60’s fictional characters like Don Draper rather than the decision making processes of modern organisations:
- HiPPOs are expensive
- HiPPOs are a bottleneck to decision making
- HiPPOs are subject to all kinds of human frailties that might misalign their opinions with the realities of the world around them, not least the ‘tyranny of expertise’
Google have a saying for dealing with this, ‘don’t bring an opinion to a data fight’.
Retrospective
As we build out operational data mining (ODM) and built the operations engineering team (OE, now OE&E) to support that there was a palpable shift in the culture of the organisation from being opinion driven to data driven. This has been empowering for front line staff, and generally made DXC Technology less political and hence a nicer place to work. As we set about building OE there were two aspects of Google practice that we borrowed from heavily. The first was Site Reliability Engineering (SRE) and the second was Google’s ‘data bazaar’ Goods.
Original comments
NB:
HiPPO is the road to irrelevance.
‘The best ideas win, independent of titles: In a social business, ideas and information flow horizontally, vertically, from the bottom and from the top; throughout the business. Ideas are like sounds, and they should be heard through the seams of the social fabric. In the absence of sound, ideas die. The most damaging syndrome is the HIPPO (highest paid person’s opinion) syndrome, whereby all the decisions are ultimately dictated by the biggest title. The best ideas must win. That’s the biggest benefit of being social.’
Recognizing Good Ideas (link broken by demise of C3, referenced HBR’s ‘Innovation Isn’t an Idea Problem‘)
Re-examine how you tackle tough problems, and make important decisions. “Decision Making By Hippo” that is, following the lead of the most highly paid person simply because they are in that position, is a very bad idea. Instead, the intelligence and capability of all the organization members can, and should, be tapped.- Andrew McAfee
If only CSC knew what CSC knows (link broken by demise of C3, referenced HP’s former CEO Lew Platt, “If only HP knew what HP knows, we would be three times more productive.”)
LEF paper ‘Energizing and Engaging Employees – Social media as a source of management innovation‘ (page 35)
Filed under: DXC blogs | 1 Comment
Tags: data science, google, HiPPO
At the recent DockerCon event in Austin Docker Inc announced two significant open source projects, Moby and LinuxKit. Moby essentially marks the split of Docker the open source project from Docker Inc the company, with the docker/docker GitHub repo moved to moby/moby. LinuxKit provides a set of tools to build ‘custom Linux subsystems that only include exactly the components the runtime platform requires’.
Continue reading the full story at InfoQ.
Filed under: Docker, InfoQ news | Leave a Comment
Tags: Docker, LinuxKit, Moby, open source, Unikernels
Originally posted internally 6 Jan 2016:
I’d meant to post this before the Christmas break as a guide to things to tinker with over the break, but then I hit the point where pretty much everybody seemed to already be on leave, and it was clearly too late….
So Happy New Year, if you’re not already on top of these things then here’s a taste of what’s coming in 2016.
1) Infrastructure as code
‘Infrastructure as code’ means that infrastructure people now need to worry about source code in the same way that application developers have.
The good news is that distributed source control and collaboration is now pretty much a solved problem[1]. History might well remember Linus Torvalds more as the creator of the Git source control system than just the ‘just a hobby’ Linux kernel project it was built to support.
GitHub
Git is built around the idea of ‘local’ and ‘remote’ repositories, where local is generally a development environment. GitHub provides the remote part where source can be pulled from to avoid starting from scratch, and pushed to once changes are made.
There are alternatives to GitHub out there such as BitBucket and GitLab (there are even alternatives to Git such as Mercurial and Perforce), but we went with GitHub at DXC Technology in order to have a consistent feature set and user experience with the most popular open source projects (that are hosted on GitHub). DXC Technology has an organisation on public GitHub where we can collaborate with customers and partners, and also an enterprise GitHub (for internal ‘inner source’ code)
Fork and Pull
Like the source control systems that came before it Git supports the idea of creating code branches for developing a given feature (or fixing a given bug), with merging back into the ‘trunk’ once a branch has served its purpose. Where things get a little different, and much more powerful, is the concept of ‘fork and pull’.
A fork is a copy of a given project so that somebody can make changes without having to ask permission of the original creator. In the past forks have often been considered to be a bad thing as they lead to potentially exploding complexity as a project splinters away from its original point. This led to a point of view that open source projects that forked had governance issues that might be detrimental to the project (and its users).
Pull requests are where somebody who’s made a fork asks for their change to be incorporated back into the main project. If a fork is ‘beg forgiveness’ then a pull is ‘ask permission’. Using fork and pull together means that a project can very quickly explore the space around it, and bring back what’s of value into the main body; and it’s a process that now lies at the heart of nearly every successful open source project.
Some enterprises have struggled with fork and pull, as it doesn’t fit into traditional governance processes – it is after all a governance process (or the beginnings of one) in its own right. The good news for infrastructure people is that infrastructure as code is generally a new enough thing that there’s no need for demolition work before erecting the new building.
Learn Git/GitHub
Pro Git Book (it’s worth paying particular attention to the chapter on branching and merging, a topic that’s also well covered at learn Git branching)
2) Configuration Management
The whole point of infrastructure as code is to reach a desired configuration for the infrastructure elements in an automated, programmatic, and repeatable fashion to support infrastructure at scale. That’s normally achieved using some sort of configuration management tool, with the code being what drives it. It’s worth noting that the use of such tools move administration away from people logging into individual machines and carrying out manual tasks.
There are essentially two approaches to configuration management:
- Imperative – ‘do this, then do that, then do this other thing’. This is what scripts have been doing since the dawn of operating systems, and arguably the mantra of ‘every good systems administrator will replace themselves with a script’ means that we’ve always at least had intent to have infrastructure as code. The problem with imperative systems is that they can be very brittle. All it takes is one unexpected change and the script doesn’t do what’s expected, and potentially everything breaks.
- Declarative – ‘achieve this outcome’. Modern configuration management tools at least aspire to being declarative, which hopefully makes them less fragile than older script based systems[2].
Ansible
There are lots of popular configuration management tools in the marketplace such as Puppet, Chef, Salt and even the original CFEngine. At DXC Technology we’ve chosen Ansible (and Ansible Tower) largely because of its SSH based agentless operating model, which eliminates the need to install stuff onto things just to begin the configuration process, and also allows it to be used with infrastructure such as networking equipment where it might be impossible to install an agent for a config management tool.
Learn Ansible
3) Containers
The use of virtual machines (VMs) to provide resource isolation and workload management has been popular now for well over a decade, and has found its way into the most conservative late adopters. Containers achieve resource isolation and workload management within an operating system rather than by using an additional hypervisor. This has multiple advantages, including quick startup time (fractions of a second) and low memory overhead (due to sharing a common kernel and libraries). The containers approach has been popular in various niches ranging from small virtual private server (VPS) hosting companies to how Google manages the million plus servers spread across its global data centres. Containers are now escaping from those niches, primarily because of the Docker project.
Docker
Docker is a set of management tools for containers that brings the mantra of ‘build, ship and run’
- Build – a container from a ‘bill of materials’ (known as a Dockerfile) that describe what goes inside the container in a simple script.
- Ship – a container (or just the Dockerfile that describes it) anywhere that you can copy a file
- Run – on any environment that can support containers, ranging from a laptop to a cloud data centre.
The ‘run’ part is in many ways pretty much ubiquitous already – any Linux machine or virtual machine with a relatively new kernel can run containers, and soon the same will apply to Windows. This makes ‘build’ and ‘ship’ the more interesting part of the story, and DockerHub provides a central (GitHub like) place where things can be built and shipped from. To complement the public DockerHub DXC Technology will be deploying its own Docker Trusted Registry (which will be equivalent to our Enterprise GitHub).
Learn Docker
I already wrote about how Docker gives you installation superpowers, and put out the plea to Install Docker.
4) Cloud Services
There are all kinds of definitions of ‘cloud’ out there[3], and our industry has suffered from a great deal of ‘cloudwashing’.
The modern pillar is the collection of services that expose an application programmer interfaces (APIs) that can be used to manage the life cycle and configuration of resources (on demand). The API allows the human interface, whether that’s a command line interface (CLI) or web interface or some other bizarre tool to be supplanted; though of course once an API is in place it’s easy to build a CLI, web interface (or even something bizarre) in front of it. It’s possible to automate without APIs (by screenscraping older interfaces), but it’s much easier to automate once APIs are in place.
Infrastructure driven by APIs, better known as Infrastructure as a Service (IaaS) was misunderstood in its early days as a cost play (despite never being *that* cheap). It’s now pretty well understood that the main point is time to market (for whatever it is that depends on the infrastructure) rather than cost per se; though many organisations have the numbers to show that they can get cost savings too.
AWS
Amazon Web Services (AWS) is the elephant in the cloud room. It’s 15x larger than its nearest competitor and 5x larger than the rest of the providers put together. Modern management techniques such as ‘two pizza’ have allowed Amazon to release new features at an exponential rate, which has been combined with huge ($Bn/qtr) infrastructure investment to force every other provider out of business or into an expensive game of catch up[4].
Learn AWS
DXC Technology is part of the Amazon Partner Network (APN), so there’s a structure programme for training and certification. Better still there are cash bonuses on offer for getting professional certifications
Where do I start to get Amazon Web Services (AWS) Training (DXC Technology staff only)
Wrapping Up
These four pillars are tied together by the use of code and the APIs that they drive. I’ll write more on how these come together with techniques like continuous integration (CI) and continuous deployment (CD), and how this fits in to the concept of ‘DevOps’.
Notes
[1] The one remaining caveat here is that GitHub has (re)introduced centralised management to a distributed system, and with it a single point of failure. GitHub (and by extension its users) have been victim to numerous distributed denial of service (DDOS) attacks over the past few years (often attributed to Chinese action against projects that allow circumvention of the ‘great firewall’). This is one reason for having a separate enterprise GitHub installation (as we have at CSC). There are some other answers – check out distributed hash tables (DHT) and interplanetary file system (IPFS) for a glimpse of how things will once again become distributed.
[2] It’s worth noting that declarative systems should easily deal with idempotence, which means that if a change has already been made then it shouldn’t be duplicated.
[3] Though NIST did everybody a favour by publishing a set of standard definitions.
[4] As we see Rackspace and CenturyLink fall by the wayside it leaves Microsoft, Google and EMC pretty much alone in terms of having the resources to compete. Google still outspends Amazon on infrastructure, which is entirely fungible between Google services and Google Cloud Services – so it’s able to do interesting things with scale and pricing. Microsoft has done a great job of upselling its existing customers into its cloud offerings, so for many it’s the cloud they already bought and paid for. It should be noted that both Apple and Facebook operate hyperscale infrastructures along similar lines to the IaaS providers, they just haven’t chosen to go into that business (yet).
Retrospective
This post very much became my manifesto for my time in Global Infrastructure Services. I presented talks based on this at TechCom 2016, and it also formed the basis of the ‘infrastructure as code boot camp’ workshops in the global delivery centres (which has now become a set of online Katacoda scenarios).
We ended up not going with Ansible Tower, for reasons I’m not going to get into here and now – that’s a story for another day.
The incentive for doing AWS Certified Solution Architect training has now been withdrawn, but it helped propel us to having the second largest community of certified people.
Filed under: cloud, code, Docker, DXC blogs | 2 Comments
Tags: Ansible, aws, cloud, code, config management, containers, Docker, Fork and Pull, git, github, infrastructure
The DXC Blogs – Install Docker
Originally published internally 24 Dec 2015:
Install Docker, because it gives you installation superpowers for many other things.
Install Docker, so that you don’t have to install Ansible, or Python, or Golang, or Greylog, or pretty much whatever else you were needing to install.
Install Docker, so that the next Yak you shave will be a pre shaved Yak for somebody else.
Retrospective
This internal post was mostly just a signpost to the public post I wrote that day. We subsequently used Docker as the basis for the ‘infrastructure as code boot camp’ and spent some of the workshops explaining how much time it saved.
Filed under: Docker, DXC blogs | 1 Comment
Tags: Docker
Originally published internally 15 Dec 2015:
I recently came to a realisation that every hire I’ve made has been for an aptitude to change rather than a given set of skills. This makes my most important interview question ‘how do you keep up to date with tech?’.
Tech is changing all the time. I constantly hear statements that the rate of change is accelerating – something that I don’t happen to believe, but even with a steady rate of change like Moore’s law there’s a whole lot to keep track of. The fire hose of tech news can be hard to manage, so it’s important to have good filters. My friend JP Rangaswami often quotes Clay Shirky’s ‘filter failure‘ when talking about the danger of information overload.
My coarse filters are Feedly [1] and Twitter, but there’s a much reduced output from that, which is stuff that I find interesting enough to add to Pinboard in case I want to go back to it later. Even that’s probably too much – I bookmark a ton of stuff on politics, religion, law and order and all manner of other subjects that have nothing to do with tech. As a new iteration of my earlier experiment into directed social bookmarking I’ve created a feed that’s being pulled into the technical design council (TDC) Slack team, simply by tagging things tdc. Pinboard very usefully allows me to easily turn that into an RSS feed for consumption by your favourite aggregator (which brings us back to Feedly if you don’t already use one, and the sad demise of Google Reader).
[1] I could probably share my Feedly feed list if anybody is interested – though beware that it’s large and very specifically tuned to things I’m interested in.
Retrospective
I keep on tagging things with ‘tdc’ and occasionally I bump into somebody who’s following along and finding some use. This post also unearthed a bunch of Feedly fans.
Original comments
I’m not going to post the original comments, as they’re mostly self referential to the people who made them, and hence impossible to psuedonymise.
I did myself however discover some Feedly sharing options, and posted this great ‘2 Kinds of People‘ cartoon:
Filed under: DXC blogs | Leave a Comment
Tags: Feedly, news, Pinboard, rss, twitter
This was originally published internally on 4 Dec 2015, my 4th day at CSC:
My first few days have made me think carefully about where I see value.
This is what’s emerging:
- I value code with documentation, samples and examples more than just code, because code with documentation, samples and examples gives us a repeatable offering that we can take to market.
- I value code more than slides, because code gives us a prototype that we can take to a brave customer who’s willing to share risk.
- I value slides more than ideas, because slides give us a way to share concepts and engage with customers about what might be.
- I value ideas on their own least of all, which is not to say that ideas have no value. Ideas are what drive us forward, and DXC Technology is full of brilliant people with great ideas – I just want to see the best ones evolve as quickly as possible into code with documentation, samples and example.
BTW code, and it’s documentation, samples and examples lives in a repository such as GitHub – if you’re not already using Git (and GitHub) then learn about it here – Learn Git | Codecademy
Retrospective
After just a few days it was already clear to me that far too much ‘solutioning’ was being done in PowerPoint rather than tools that could touch a production environment. Those slides were worthless compared to code. Not completely worthless, but definitely worth less. Of course code on its own isn’t great either, as successful projects need great documentation, samples and examples; a theme that I return to later.
Original comments
CN:
+1 Chris :-)
There was a great session from EMC and Puppet at VMworld which said think of Github/Version-Controlled code as “Live Documentation”.
The presenter made everyone promise never to log in to a server again before we left!
Whilst it’s not possible to stick to that when you have legacy customers on systems that can’t, yet, accept any change to them to be pushed out centrally it’s a great goal to have in the back of your mind when doing anything. “If this change works it need to go back into the build code/docs”
Filed under: DXC blogs | 2 Comments
Tags: documentation, git, github
The DXC Blogs – an intro
Over the course of the next few weeks I’m planning to publish a bunch of posts that originally went onto my internal CSC blog hosted on ‘C3’ (Jive). My reasons for doing this are:
- C3 is being shut down as part of the merger of CSC and HPE Enterprise Services (ES) to create DXC.technology, and I see little point in moving them to a SharePoint site.
- My new colleagues who’ve come from ES haven’t been able to see what I posted on C3.
- Our CTO Dan Hushon is very keen that we communicate ‘outside in’ rather than relying on internal communications systems.
- James Governor’s recent ‘Further thoughts on 10x engineers‘ explains quite clearly that communication is an essential part of (tech) leadership. It’s great to see so many people who are friends, colleagues and that I’ve hosted at conferences mentioned in the post, and I obviously need to up my game to join them.
I’m going to try to publish one post per day, hopefully along with psuedonyminised comments. There may be some minor edits to make up for the shift from private to public, and I’ll also s/CSC/DXC Technology/. I’m planning to add a retrospective to each post.
Filed under: DXC blogs | 18 Comments
Tags: DXC
A colleague asked me the other day how to get started with GitHub on a Windows machine, and I ended up doing a quick screen share to show him my usual setup. Thinking that it’s likely a common question I’ve put together a quick screencast of installing Git Bash and Atom on Windows, and using them with GitHub.
Filed under: howto | 2 Comments
Tags: ATOM, bash, git, github, howto, Windows
Wilful ignorance
I’m not much of a podcast fan, but I came across Sam Harris interviewing Michael Hayden and set aside some time to listen to it. I wasn’t expecting much common ground between the interviewer and interviewee, but a mutual dislike of Glenn Greenwald seemed to get their rapport clicked into place, and I very much enjoyed their discussion.
The purpose of this post is that there’s a massive logical discrepancy in what Hayden said. If it’s true then the (US) Intelligence Agencies are being wilfully ignorant, something that I find unlikely.
The President’s BlackBerry
In one part of the interview Hayden talks about how Obama had to be weaned off his BlackBerry after taking office at the White House.
The assertion: Every embassy in Washington would be able to tap in to the President’s email traffic.
Why this seems unlikely: BlackBerries had a reputation for strong end to end encryption, so much so that there has been controversy over whether they could be permitted in India. For sure any (foreign) agency with proximity to the cell tower near the White House would be able to intercept the encrypted traffic, but that should be pretty much useless to them. At worst traffic pattern analysis would reveal when the device was sending and receiving.
Tipping their hand: This could reveal that NSA already knew how to intercept BlackBerry traffic and that it was simple enough that they also expected less friendly agencies to be able to do the same and/or their was a TEMPEST issue with the handsets that presented a vulnerability to anybody with sufficient proximity (‘anybody on Pennsylvania Avenue’).
Trump Tower Wire Taps
Elsewhere in the interview Hayden marvelled at how the discourse around Trump Tower wiretapping allegations had taken place so publicly, with key agency representatives going on the record to say that Obama had not authorised spying on the Trump campaign/President Elect.
The assertion: Trump Tower wasn’t under surveillance.
Why this seems unlikely: Hayden had talked earlier in the interview about domestic metadata collection for back tracing, and how even his own call records would be amongst that horde. So it would be exceptional for Trump Tower not to be included. Perhaps the point here is the oft argued content versus metadata; but in a world of ‘we kill people based on metadata’ it matters not; and the metadata would be plenty to build the graph of Trumps relationships and influence.
Putting it together
Trump is well known for using an Android phone, which is likely a lot less secure than Obama’s BlackBerry, so if the earlier assertions are true, then any (foreign) agency in New York would be all over his comms and collecting their ‘Kompromat‘. If the latter assertion is also true then the US are basically saying that they left the field open for foreigners, but didn’t take a look themselves; and that would be wilful ignorance. At the very least I’d expect a ‘this is what we (and everybody else) got on you before you were sworn in, and that’s why things need to change’ type conversation and the business as usual activities behind it. Maybe I’m wrong; maybe wilful ignorance is part of how the game is played.
Filed under: wibble | Leave a Comment
Tags: privacy, SIGINT
Lovely
My wife took me out to the ‘Supper Club‘ at the newly opened Waitrose in Haywards Heath earlier this week. I wouldn’t be writing about it here if the Daily Mail hadn’t done a hatchet job review that totally misaligns with own experience.
That’s me in the blue shirt with my back to the camera
The title of this post comes from the head chef’s introduction to the menu. He must have said ‘lovely’ over a dozen times.
Let’s examine some of the headline quotes..
‘I could make a better dessert at home’
Well, that was kind of the point. Diners were given a pack as they left with recipe cards so that they could use (Waitrose sourced ingredients) to make everything at home.
‘1970s style’ food
This seems to be a standard attack for anywhere with a prawn cocktail on the menu, since that’s probably the pinnacle of 70s retro chic; though I expect the same rock could be thrown at the chicken fricassee. I’m a child of the 70s, and I happen to like a good prawn cocktail. As fate would have it this was my second prawn cocktail in the week, and the second time where the menu had offered ‘brown prawns’. It wasn’t as good as the ‘half pint of shell on brown prawns with Marie rose sauce’ I got on Monday at the Lockhart Tavern, which came with many prawns that had roe attached, but that was the best prawn cocktail I’ve ever had. The Waitrose one was a lot less fiddly and time consuming.
Small portions and long waits
I’m not a small chap, and the quantity of food was more than enough for me (and I didn’t take any potatoes or eat half of the biscuits that came with my cheese). They were also pretty generous with the wine, with bottles left on the table for us to help ourselves. Mineral water was also free flowing at no extra expense for those choosing not to drink.
I never felt like I was waiting, which could have been just because the wine and conversation was flowing. There was also no wait to pay at the end (my most hated part of the usual restaurant experience) – just say goodbyes and walk away.
My own view
The communal seating wouldn’t have been my choice, but I’m an introvert at heart so I wouldn’t generally sit and chat to strangers. But these were nice middle class Waitrose buying strangers, and the conversation was in the end what made it an entertaining night out.
£70 for dinner for two would have also bought us dinner and a few craft beers up the road at the recently opened Lockhart Tavern, my new favourite in Haywards Heath. It wouldn’t have bought us 3 courses with bubbly, still wine, mineral water etc.
I enjoyed the food, I enjoyed the format, I’ll be back again for a new menu if they keep doing it. I might even try the chicken fricassee recipe at home (and if I have one ruined expectation it’s that I expected more emphasis on how to cook the meals). It was lovely.
It’s also worth noting that the #waitrosesupperclub Tweets belie the Mail’s version of events, and I’m told it’s the same over on Facebook.
Filed under: cooking, wibble | Leave a Comment
Tags: dining, news
Chris Swan's Weblog 