Things are not normal, so I’ve been trying to mix things up a bit with cooking at home to make up for the lack of going out.

Also one of the people I met at OSHcamp last summer retweeted this, which led to me ordering a bunch of meat from Llais Aderyn

When the box came it had:

2 packs of mince
2 packs of diced stewing steak
1 pack of braising steak
1 pork loin joint
1 lamb joint

It got to me frozen, but not cold enough to go back into the freezer, so I had lots of cooking to do over the next few days. The meat went into making:

A giant slow cooked Bolognese sauce (nine portions), which used the mince and one pack of stewing steak (as I like chunky sauces).

Slow roast pork and fries (four portions)

didn’t last long enough for a photo

Slow roast lamb (with the usual trimmings – four portions)

Shepherd’s pie (four portions) from the leftover lamb

Slow cooked casserole (six portions), which used the remaining stewing/braising steak

and that was the end of that box of meat. It’s a shame that Ben now seems to be mostly sold out, as all the meat was really tasty.

Tonight will be kimchi jjigae, inspired by @bodil’s recipe

Taking it slow

The observant amongst you will have noticed a theme. Everything was slow cooked, either in my Crock-Pot SC7500[1] or simply in a fan oven overnight at 75C. As I wrote before in The Boiling Conspiracy there is really no need for high temperatures.

My Bolognese recipe

Here’s what went into the Bolognese pictured above:

2 medium brown onions and 1 red onion, lightly sauteed before adding
2 packs mince and 1 pack stewing steak (each ~950g) which were browned off
2 tbsp plain flour
2 cans chopped tomatoes
~250g passata (left over from pizza making)
~3/4 tube of double stength tomato puree
handful of mixed herbs
ground black pepper
a glug of red wine
1 tbsp gravy granules
4 chopped mushrooms
and then left bubbling away in the slow cooker all afternoon (about 6hrs)

Note

[1] The SC7500 is great because the cooking vessel can be used on direct heat like a gas ring, and then transferred into the slow cooker base to bubble away for hours – so less fuss, and less washing up.


Background

I use Enterprise GitHub at work, and public GitHub for my own projects and contributing to open source. As the different systems use different identities some care is needed to ensure that the right identities are attached to commits.

Directory structure

I use a three level structure under a ‘git’ directory in my home directory:

  • ~/git
    • which github
      • user/org
        • repo

 

  • ~/git – is just so that all my git stuff is in one place and not sprawling across my home directory
    • which github – speaks for itself
      • user/org – will depend on projects where I’m the creator, or I need to fork then pull versus stuff where I can contribute directly into various orgs without having to fork and pull
        • repo – again speaks for itself

So in practice I end up with something like this:

  • ~/git
    • github.com
      • cpswan
        • FaaSonK8s
        • dockerfiles
      • fitecluborg
        • website
    • github.mycorp.com
      • chrisswan
        • VirtualRoundTable
        • HowDeliveryWorks
        • Anthos
      • OCTO
        • CTOs

Config files

For commits to be properly recognised as coming from a given user GitHub needs to be able to match against the email address encoded into a commit. This isn’t a problem if you’re just using one GitHub – simply set a global email address (to the privacy preserving noreply address) and you’re good to go.

If you’re using multiple GitHubs then all is well if you’re happy using the same email across them, but there are lots of reasons why this often won’t be the case, and so it becomes necessary to use Git config conditional includes (as helpfully explained by Kevin Kuszyk in ‘Using Git with Multiple Email Addresses‘).

The top level ~/.gitconfig looks like this:

[user]
  name = Your Name
[includeIf "gitdir:github.com/"]
  path = .gitconfig-github
[includeIf "gitdir:github.mycorp.com/"]
  path = .gitconfig-mycorp

Then there’s a ~/.gitconfig-github:

[user]
  email = [email protected]

and a ~/.gitconfig-mycorp:

[user]
  email = [email protected]

With those files in place Git will use the right email address depending upon where you are in the directory structure, which means that the respective GitHub should be able to then match up the commits pushed to it to the appropriate user that matches those email addresses.


TL;DR

Best practice gets encoded into industry leading software (and that happens more quickly with SaaS applications). So if you’re not using the latest software, or if you’re customising it, then you’re almost certainly divergent from best practices, which slows things down, makes it harder to hire and train people, and creates technology debt.

Background

Marc Andreessen told us that software is eating the world, and this post is mostly about how that has played out.

As computers replaced paper based processes it was natural for mature organisations with well established processes to imprint their process onto the new technology. In the early days everything was genesis and then custom build (per Wardley map stages of evolution), but over time the software for many things have become products and then utilities.

Standard products and utilities need to serve many different customers, so they naturally become the embodiment of best practice processes. If you’re doing something different (customisation) then you’re moving backwards in terms of evolution, and just plain doing it wrong in the eyes of anybody who’s been exposed to the latest and greatest.

An example

I first saw this in the mid noughts when I was working at a large bank. We had a highly customised version of Peoplesoft, which had all of our special processes for doing stuff. The cost of moving those customisations with each version upgrade had been deemed to be too much; and then after many years we found ourselves on a burning platform – our Peoplesoft was about to run out of support, and a forced change was upon us.

It’s at this stage that the HR leadership did something very brave and smart. We moved to the latest version of Peoplesoft and didn’t customise it at all. Instead we changed every HR process in the whole company to be the vanilla way that Peoplesoft did things. It was a bit of a wrench going through the transition, but on the other side we had much better processes. And HR could hire anybody that knew Peoplesoft and they didn’t have to spend months learning our customisations. And new employees didn’t have to get used to our wonky ways of doing things, stuff worked much the same as wherever they’d been before.

Best of all Peoplesoft upgrades henceforth were going to be a breeze – no customisations meant no costly work to re-implement them each time.

SaaS turbocharges the phenomenon

Because SaaS vendors iterate more quickly than traditional enterprise software vendors. The SaaS release cycle is every 3-6 months rather than every 1-2 years, so they’re learning from their customers faster, and responding to regulatory and other forms of environment change more quickly.

SaaS consumption also tends to be a forced march. Customers might be able to choose when they upgrade (within a given window), but not if they upgrade – there’s no option to get stranded on an old version that ultimately goes out of support.

Kubernetes for those who want to stay on premises

SaaS is great, but many companies have fears and legitimate concerns over the security, privacy and compliance issues around running a part of their business on other people’s computers. That looks like being solved by having Independent Software Vendors (ISVs) package their stuff for deployment on Kubernetes, which potentially provides many of the same operational benefits of SaaS, just without the other people’s computers bit.

I first saw this happening with the IT Operations Management (ITOM) suite from Microfocus, where (under the leadership of Tom Goguen) they were able to switch from a traditional enterprise software release cycle that brought with it substantial upheaval for upgrades, to a much more agile process internally, running into quarterly released onto a Kubernetes substrate (and Kubernetes then took care of much of the upgrade process, along with other operational imperatives like scaling and capacity management).

Conclusion

Wardley is way ahead of me here, if you customise then all you can expect is (old) emerging practices, which will be behind the best practices that come from utility services. I did however want to specifically call this out, as there’s an implication that everybody gets lifted by a rising tide, and the evolution from custom to utility. But that only happens if you’re not anchored to the past – you need to let yourself go with that tide.


Turning a Twitter thread into a post.

I wrote about the performance of AWS’s Graviton2 Arm based systems on InfoQ

The last 40 years have been a Red Queen race against Moore’s law, and Intel wasn’t a passenger, they were making it happen. I used to like Pat Gelsinger’s standard reply to ‘when will VMware run on ARM?’

It boiled down to ‘we’re sticking with x86, not just because I used to design that stuff at Intel, but because even if there was a zero watt ARM CPU it wouldn’t sufficiently move the needle on overall system performance/watt’. And then VMware started doing ARM in 2018 ;)

Graviton 2 seems to be the breakthrough system, and it didn’t happen overnight – they’ve been working on it since (before?) the Annapurna acquisition. AWS have also been very brisk in bringing the latest ARM design to market (10 months from IP availability to early release).

Even then, I’m not sure that ARM is winning so much as x86 is losing (on ability to keep throwing $ at Moore’s law’s evil twin, which is that foundry costs rise exponentially). ARM (and RISC-V) have the advantage of carrying less legacy baggage with them in the core design.

I know we’ve heard this song before for things like HP’s Moonshot, but the difference this time seems to be core for core the performance (not just performance/watt) is better. So people aren’t being asked to smear their workload across lots of tiny low powered cores.

So now it’s just recompile (if necessary) and go…

Update 20 Mar 2020

Honeycomb have written about their experiences testing Graviton instances.


AnandTech has published Amazon’s Arm-based Graviton2 against AMD and Intel: Comparing Cloud Compute which includes comprehensive benchmarks across Amazon’s general purpose instance types. The cost analysis section describes ‘An x86 Massacre’, as while the pure performance of the Arm chip is generally in the same region as the x86 competitors, its lower price means the price/performance is substantially better.

Continue reading the full story at InfoQ.


TornadoVM was definitely the coolest thing I learned about at QCon London last week, which is why I wrote up the presentation on InfoQ.

It seems that people on the Orange web site are also interested in the intersection of Java, GPUs and FPGA, as the piece was #1 there last night as I went to bed, and as I write this there are 60 comments in the thread.

I’ve been interested in this stuff for a very long time

and wrote about FPGA almost 6 years ago, so no need to go over the same old ground here.

What I didn’t mention then was that FPGA was the end of a whole spectrum of solutions I’d looked at during my banking days as ways to build smaller grids (or even not build grids at all and solve problems on a single box). If we take multi core CPUs as the starting point, the spectrum included GPUs, the Cell Processor, QuickSilver and ultimately FPGAs; and notionally there was a trade off between developer complexity and pure performance potential at each step along that spectrum.

Just In Time (JIT) Compilers are perfectly positioned to reason about performance

Which means that virtual machine (VM) based languages like Java (and Go[1]) might be best positioned to exploit the acceleration on offer.

After all the whole point of JIT is to optimise the trade-off between language interpretation and compilation, so why not add extra dimensionality to that and optimise the trade-off between different hardware targets.

TornadoVM can speculatively execute code on different accelerators to see which might offer the best speedup. Of course that sort of testing and profiling is only useful in a dev environment, once the right accelerator is found for a given workload it can be locked into place for production.

It’s just OpenCL under the hood

Yep. But the whole point is that developers don’t have to learn OpenCL and associated tool chains. That work has been done for them.

Again, this is powerful in addressing trade-offs, this time between developer productivity and system cost (ultimately energy). My FPGA experiment at the bank crashed on the rocks of it would have cost $1.5m in quants to save $1m in data centre costs, and nobody invests in negative payouts. If the quants could have kept on going with C++ (or Haskell or anything else they liked) rather than needing to learn VHDL or Verilog then it becomes a very different picture.

Which means it’s not real Java

So what, arithmetic doesn’t need fancy objects.

There’s some residual cognitive load here for developers. They firstly need to reason about suitable code blocks, and apply annotations, and then they must ensure that those code blocks are simple enough to work on the accelerator.

If I had greater faith in compilers I’d maybe suggest that they could figure this stuff out for themselves, and save developers from that effort; but I don’t the compiler will not save you.

Conclusion

My FPGA experiment some 15 years ago taught me a hard lesson about system boundaries when thinking about performance, and the trade-off between developer productivity and system productivity turns out to matter a lot. TornadoVM looks to me like something that’s addressing that trade-off, which is why I look forward to watching how it develops and what it might get used for.

Updates

10 Mar 2020 I’d originally written ‘TornadoVM doesn’t (yet) speculatively execute code on different accelerators to see which might offer the best speedup, but that’s the sort of thing that could come along down the line‘, but Dr Juan Fumero corrected me on Twitter and pointed to a pre-print of the paper explaining how it works.

Note

[1] Rob Taylor at Reconfigure.io worked on running Golang on FPGA


TL;DR

I prefer working from home over the grind of a daily commute. After many years of doing it I’ve been able to refine my working environment to make it comfortable and productive.

Background

As the COVID-19 pandemic bites a bunch of people are working from home who might not be so used to it. Guides like this one from Justin Garrison are showing up, so I thought I’d take a pass at my own.

I’ve more or less worked from home for the past 7 years, which covers my time at Cohesive Networks then DXC. Before that I generally had a work from home day (or two) in a typical week. Now there’s no such thing as a typical week; I work from home whenever I don’t need to be somewhere else – whether that’s the London office, or the Bangalore office (which is where I was supposed to be today).

A (semi) dedicated room

I’ve used 4 different rooms over the years, and in each case it’s been my ‘office’.

  • Bedroom 2 had plenty of room for my desk and a guest bed, but that was when $son0 was still a toddler, and when he grew up he needed the space.
  • Tiny front room had just enough space, but was an acoustic disaster when the family were home. I was kind of glad when it became our utility room and I was kicked out in favour of the washing machine and tumble dryer (and dog).
  • Bedroom 3 was good enough, but didn’t still serve as a family or guest bedroom, which was kind of a waste.
  • The loft conversion is perfect – dedicated space at the top of the house away from family noise and other distractions. There’s a futon in there that can be pressed into service for guests.

Desk

Prior to the loft conversion I used a computer unit from Ikea, which was entirely satisfactory in terms of layout and storage. The loft conversion provided an opportunity to get a custom desk made (from kitchen worktop).

Chair

If you’re spending 45 hours a week in a chair then it needs to be a good one that’s not going to cause back trouble. My tool of choice is a Herman Miller Aeron (used from eBay).

Light

The loft conversion has large Velux windows front and back, which provide plenty of natural light during the day (and in the summer I’m glad for the awnings that keep down the heat and glare). During the winter months the ceiling mounted LED down lights see service at the beginning and end of days.

Quiet please

My silent PC means I’m not dealing with fans whirring all day, which really helps with concentration. There’s a bit of background coil whine from power supplies; but then there’s also wind, and rain, and traffic going by.

Big screen(s)

I run a Dell U2715H 27″ 2560×1440 as my main display, which gives me space for two browser sessions docked to each side of the screen (and most of my stuff is done in a browser).

A Samsung 24″ 1920×1080 screen provides extra real estate for full screen apps, including presenting on web conferences etc.

Webcam

I have a Logitech device that dates back to the dark ages before laptops had their own webcams built in, but it’s still perfectly functional. It attaches to mounting points made from Lego and held down with Sugru.

Headset

I can use the webcam mic and monitor speakers for a speakerphone type experience, and that’s great if I’m expecting to be mostly on mute, but can cause echo issues.

When I’m presenting or expecting to talk a lot I have a Jabra 930 that I covered in my previous Headsets (mini review).

Staying healthy in body

I try to stand every hour to keep streaks going on my Apple Watch (and it will sometimes remind me if I’m remiss on that).

The futon provides a useful anchor for sit-ups to keep core strength.

The loft conversion means plenty of flights of stairs to run up and down for glasses of water, cups of tea, and answering the door to the postman and delivery drivers.

My lunchtime routine is to walk the dog to the local shop, whilst listening to a podcast, so that’s exercise, learning and food all taken care of.

Staying healthy in mind

I don’t miss the ‘water cooler’. It’s nice to bump into colleagues when I’m in the office, but I don’t feel bereft of human contact whilst at home. That might be different if I spent less of my days talking to colleagues on various calls.

Time management

Not commuting means not wasting 3hrs each day on train rides and getting to and from stations, but it’s easy for that time to get soaked up by work stuff. I try to force a stop when the family gets home from their days. It’s good to be flexible at times though, and taking a late call from home is a lot less bother than staying late at the office.

Colleagues

Most of the people I work with at DXC also work from home with occasional trips elsewhere to be in the office, visit customers and partners, and attend events; so the company has a decent amount of remote working baked into the culture. That includes time zone sensitivity and management that don’t expect people to be in a particular place.

Conclusion

I’d hate to go back to the daily grind of a commuting job. I think it would make me less productive, and also a lot less happy. Working from home is definitely something I see as a positive, but to make the best of it takes a little work to get the right environment.

Update 18 Mar 2020 – Power and UPS

This post I just saw from Gautam Jayaraman on LinkedIn reminded me of something I’d left out – making sure the lights stay on, or in the case of home working your PC, monitor, router and anything vital connecting them together. I’ve ended up with 4 UPSs dotted around the house, which are vital given the poor quality of my utility electricity supply.


Dr Juan Fumero presented at QCon London on TornadoVM, a plug-in to OpenJDK and GraalVM that runs Java on heterogeneous hardware including Graphical Processing Units (GPUs) and Field Programmable Gate Arrays (FPGAs). Demos during the presentation showed code being speeded up by hundreds of times when running on a GPU vs a CPU.

Continue reading the full story at InfoQ.


Background

A friend sent me a link to an ‘AI Ops’ company over the weekend asking me if I’d come across them. The product claims to do ‘continuous optimisation’, and it got me wondering why somebody would want such a thing?

Let’s explore the Rumsfeld Matrix

Known knowns

This is where we should be with monitoring. We know how the system is meant to perform, and we have monitors set up to alert us when performance strays outside of those expected parameters (at that point giving us a known unknown – we know that something is wrong, but the cause is yet to be determined).

Such known knowns are predicated on the notion that the performance of our system is deterministic. Expected things happen at the inputs, leading to expected things happening at the outputs, with a bunch of expected transformations happening in the middle.

Computers are superficially very amenable to determinism – that’s why they call it computer science. Modern systems might encompass vast complexity, but we can notionally reason about state at pretty much every layer of abstraction from the transistors all the way up to applications written in high level languages. We have at least the illusion of determinism.

Unknown unknowns

In ‘why do we need observability‘:

When environments are as complex as they are today, simply monitoring for known problems doesn’t address the growing number of new issues that arise. These new issues are “unknown unknowns,” meaning that without an observable system, you don’t know what is causing the problem and you don’t have a standard starting point/graph to find out.

This implies that we have determinism within a bounded context – that which we do understand; but that complexity keeps pushing us outside of those bounds. Our grip on determinism is slippery, and there will be times when we need some help to regain our grasp.

Unknown known

This is the intentional ignorance quadrant, and I think the target for the ‘AI Ops’ tool.

Yes, the dev team could spend a bunch of time figuring out how their app actually works. But that seems like a lot of hard work. And there’s a backlog of features to write. Much easier to buy a box of magic beans neural network to figure out where the constraints lie and come up with fixes.

This confronts the very real issue that getting a deterministic view of performance is hard, so of course people will be tempted to not bother or take shortcuts.

It’s easy to see why this is the case – I’ll provide a couple of examples:

Compilers work in mysterious ways

And so do modern CPUs.

In ‘The compiler will not save you‘ I’ve previously written about how a few different expressions of a completely trivial program that flashes Morse code on an LED result in entirely different object code. If our understanding of the compiler isn’t deterministic for a few dozen lines of C then we’re likely way out of luck (and our depth) with million line code bases.

Also for an embedded system like MSP430 (with its roots in TMS9900) we can reason about how op codes will flip bits in memory. But with modern architectures with their speculative execution and layered caches it gets much harder to know how long an instruction will take (and that’s the thing that lies at the heart of understanding performance).

Alex Blewitt’s QCon presentation Understanding CPU Microarchitecture for Performance[1] lays out some of the challenges to getting data in and out of a CPU for processing, and how things like cache misses (and cache line alignment) can impact performance.

Little changes can make big differences

Back when I ran the application server engineering team at a bank I was constantly being asked to ‘tune’ apps. Ultimately we set up a sub-team that did pretty much nothing but performance optimisation.

Most memorable to me was a customer facing reporting application that was sped up by 10,000x in the space of a few hours. I used Borland’s ServerTrace tool to profile the app, which quickly showed that a lot of time was spent connecting to the backend database. The developers had written their own connection pool class rather than using the connection pools built into the app server. Only that class didn’t actually pool connections. Amateurs 0, App Server Engineers 1. A single line of code changed, one additional config parameter to set up the connection pool, and the app was 30x faster, but it was still horribly slow – no wonder customers were complaining.

With the connection pooling resolved the next issue to become obvious was that the app was making a database call for every row of the report, with a typical report being hundreds to thousands of rows. This of course is what recordsets are for, and another trivial code change meant one database query per report, rather than one for every line of the report, which is MUCH faster.

The journey to understanding starts with the first step

If developers don’t look at the performance of their app(s) then they’re not going to be able to run small experiments to improve them. We have tools like profilers and tracers, we have practices like A/B testing, canary deployments and chaos engineering. But the existence of those tools and practices doesn’t fix anything, they have to be put to work.

Conclusion

Software is hard. Hardware is hard too. Understanding how a given piece of software works on a particular configuration of hardware is really hard. We can use tools to aid that understanding, but only if we care enough to pick them up and start trying. The market is now offering us a fresh batch of ‘AI’ tools that offer the promise of solving performance problems without the hard work of understanding, because the AI does the understanding for us. It might even work. It might just be a better use of our valuable time. But it’s unlikely to lead to an optimal outcome; and it seems in general that developer ignorance is the largest performance issue – so using tools as a crutch to not learning might be bad in the longer term.

Note

[1] The video of his talk to complement the slides is already available to QCon attendees, and will subsequently be opened to public viewing.


TL;DR

Amazon Web Services Certified Solution Architect Professional (AWS CSA Pro) took me a lot more time to study for than Google Cloud Platform Professional Cloud Architect (GCP PCA). They fundamentally test the same skills in terms of matching appropriate services to customer needs, but there’s just more of AWS, and greater fractal detail (that’s often changed over time).

Background

In my previous post on certification I mentioned that DXC’s CTO Dan Hushon asked all his CTOs to get a cloud or DevOps certification prior to the company launch in April 2017. I went for AWS CSA Associate, and after 3 years it was about to expire. So with a half price exam voucher at my disposal I decided to try levelling up to CSA Pro.

I knew it wouldn’t be easy. CSA Pro has a formidable reputation. But I’d already done the Google equivalent with PCA, and by some accounts that was supposed to be harder.

Study time

It took me something like 12 hours to get from Google Associate Cloud Engineer to Professional Cloud Architect using a combination of Coursera, Qwiklabs and Udemy practice tests.

The journey from associate to pro with Amazon was somewhat longer – I’d estimate that I spent something like 42 hours running through Scott Pletcher’s excellent A Cloud Guru course[1], White Papers and Udemy practice tests. Part of that is there are just more services in AWS, and part of that is the practice questions (and answers) just have more detail. The whole thing was more of a grind.

The foreign language comparison

At the end of my 3rd year at high school we’d completed the French curriculum with two weeks left to spare, so our teacher spent those two weeks teaching us Spanish. I learned as much Spanish in two weeks as I’d learned French in three years[2]. Spanish was just easier to get to grips with due to fewer irregularities.

In this case GCP is like Spanish – mostly pretty regular with a small number of exceptions that can easily be picked up, and AWS is like French – lots of irregularity all over the place, which just needs to be learned[3].

And the ground keeps moving under our feet

All the clouds keep growing and changing (which is why they demand re-certification every 2-3 years). But it seems that AWS is perhaps more problematic in terms of things that were true (usually limitations) being invalidated by the constant march forward. In this respect GCP’s progress seems more orderly (and perhaps better thought through), and hence less disruptive to the understanding of practitioners; but maybe that comes at a cost of feature velocity. Cloud services are after all a confusopoly rather than a commodity, and certifications are ultimately a test of how understandable the service portfolio is in relation to typical problems an architect might encounter.

Conclusion

In terms of intellectual challenge I’d say that working with either platform as an architect is roughly the same. But AWS has more to learn, and more irregularity, which means it takes longer, so if asked which certification is ‘harder’ I’d have to say AWS.

Notes

[1] I know that I’ve previously stated that I’m not a fan of learning from videos, but Scott’s course might be the exception to that. It was wonderfully dense, and I also appreciated being able to learn on the move with my iPad.
[2] Subsequently I found myself dearly wishing I’d been just a bit worse at French, which would have led to me doing Spanish ‘O’ Level, which I might just have passed – I failed French (not helped by a combination of dyslexia and negative marking for spelling mistakes).
[3] Of course I come at this with the benefit of being a native English speaker, and so the even worse irregularity of English is largely hidden to me, because I never had to learn it that way.