Metaprogramming

26Sep16

I spent part of my weekend absorbing Rod Johnson’sSoftware That Writes And Evolves Software‘, which introduces what he’s been doing at his new company Atomist, and particularly the concept of ‘Editors’, which are essentially configuration templates for programs. The combination of Atomist and its Editors is a powerful new means of metaprogramming.

I’ll repeat Rod’s suggestion that it’s well worth watching Jessica Kerr’s demo of using Atomist and Editors with Elm:

Why this could be huge

Firstly Rod has form for making the lives of developers easier. His Spring Framework for Java transformed the landscape of Enterprise Java programming, and largely supplanted the ‘Enterprise Edition’ parts of Java Enterprise Edition (JEE [or more commonly J2EE]).

The war against boilerplate

One of the trickier things about using the EE parts of JEE was the sheer volume of boilerplate code needed to get something like Enterprise Java Beans (EJB) working. This is a sickness that still plagues Java to this day – witness Trisha Gee’s Java 8 in Anger (and particularly the parts about Lambdas). Spring fixed this by stripping out the boilerplate and putting the essence into a config file for dependency injection – this got even better when Rod brought Adrian Colyer on board to integrate aspect oriented programming, as it became possible to do really powerful stuff with just a few lines of code.

Jess’s Elm demo shows that the war against boilerplate rumbles on. Even modern programming languages that are supposed to be simple and expressive make developers do grunt work to get things done, so there’s a natural tendency towards scripting away the boring stuff – something that Atomist provides a framework for.

For infrastructure (as code) too…

Atomist’s web site shouts ‘BUILD APPLICATIONS, NOT INFRASTRUCTURE’, but there’s clearly a need for this stuff in the realm of infrastructure as code. Gareth Rushgrove asked yesterday ‘Does anyone have a tool for black box testing AWS AMIs?’ the discussion rapidly descends into ‘everybody starts from scratch’ with a side order of ‘there should be a better way’. The issue here is that for any particular use case it’s easier to hack something together with single use scripts than it might be to learn the framework that does it properly. Metaprogramming is potentially the answer here, but it also raises an important issue…

This stuff is hard

If programming is hard then metaprogramming is hard squared – you need to be able to elevate the thought process to reasoning about reasoning.

Jessica’s demo is impressive, and she makes it look easy, but I take it with the pinch of salt that Jessica is a programming goddess, and she can do stuff with functors that makes my brain melt.

Documentation, samples and examples to the rescue

Perhaps the whole point here isn’t to *do* metaprogramming, but to use metaprogramming. Spring didn’t have to be easy to write, but it was easy to use. Likewise if the hard work is done for us by Rod, and Jessica, and Gareth then that provides a boost for everybody else as we stand on the shoulders of giants.

It’s about making the right way to do things also the easy way to do things – from a learning perspective, and Rod, Jessica and Gareth all have great form here with their books and presentations. If Atomist succeeds then it will be because the documentation, samples and examples that come with it make it easier to get things done with Atomist – the value isn’t just in the tool (or the approach that underlies it), but in the learning ecosystem around the tool.

I have great hopes that metaprogramming (and particularly the Atomist implementation of metaprogramming) will help us win the war against boilerplate (and hacked together scripts) – because it will be easier to start with their documentation, samples and examples.


TL;DR

Meat can be cooked safely at well below 100C, and comes out better for it, so why do cook books and TV chefs never suggest it?

Background

I love to eat and I love to cook, which is one of the reasons that I made my own temperature controlled sous vide water bath – so that I could experiment with new cooking styles.

Today’s Sunday Roast was beef brisket that I cooked for 16 hours at 88C (190F). The gravy that I made with the juices was so nice that my wife finished off what was left over with a spoon.

brisket

Low and Slow

Using the water bath has encouraged me to try other low temperature cooking methods. I frequently roast leg of lamb or pork belly in a fan oven overnight at around 75C, which brings me to the point – for me the low in ‘low and slow’ is generally below 100C – I don’t boil my food for hours on end.

But I’ve never, ever seen a TV chef or cook book (apart from the excellent Cooking for Geeks) suggest roasting at anything less than 100C (and generally a good margin above that), and with higher temperatures come quicker cooking times, so ‘slow and low’ becomes neither.

It’s an artefact of human history that there were two ways to make water safe to drink:

  1. Boiling
  2. Alcohol

Of course neither come into play in the modern first world, where we get safe drinking water at the twist of a tap that doesn’t need boiling. It seems however that 100C has become a magical number for food safety that nobody in the public eye is willing to go below.

If I look at official guidance (such as this from the Canadian government) it ranges from 63C for beef steaks to 82C for whole poultry – all comfortably below 100C. These are of course internal temperatures, so with normal high temperature roasting we might go for an hour or two at 180C or more externally to get to the required internal temperature; but with ‘low and slow’ then (given enough time) the internal temperature becomes the same as the external temperature. This happens quicker with a water bath (which uses conduction) than an over (which uses convection), but given sufficient time (which is the whole point) the outcome is the same.

I strongly suspect that the ‘low and slow’ of earlier generations, before the advent of modern ovens, was generally below 100C – so I expect that we’re just rediscovering the delights of how my grandparents used to get their Sunday roasts (when they were lucky enough to have meat).

There’s an important point here – water turns to steam at 100C and meat dehydrates, so proper ‘low and slow’ results in moist and tender meat, which is why I find it such a mystery that all the modern recipes advise above 100C. My guess is that the TV chefs use proper low temperatures for their own cooking, but are too scared to bring the general public along with them in case some fool messes it up and poisons themselves (and their family).


Let the 80s and 90s computer nostalgia continue…

Between writing about how I learned to code, and watching the latest season of Halt and Catch Fire, I’ve been thinking about how the online services I’ve used over the years have shaped my view of the IT landscape.

WarGames

Like so many others my journey started with the 1983 classic WarGames. I came away from that movie desperately wanting an acoustic coupler (and free phone calls).

Prestel

I didn’t (yet) have a modem at home, but there was one in the school computer room, and we used it to connect to Prestel – British Telecom’s (BT) videotex service. Videotex was a big deal in the UK with BBC’s Ceefax and ITV’s Oracle, but they were broadcast only – Prestel was bi-directional. Prestel was also a big deal in the IT press due to the case against Steve Gold and Robert Schifreen for hacking Prince Philip’s account. They were charged with forgery because there were no hacking laws then.

Compunet

I never did get an acoustic coupler, but I finally got my hands on a modem with a promotion by Compunet where they gave away their C64 modem if you signed up to their service for a year. The modem supported the v.23 standard common with videotex services of 1200/75 baud – 1200 bits per second download and 75 upload (and yes – you probably can type faster than that). It also did v.21 300/300, which was generally better for uploading stuff to bulletin board systems (BBSs).

modem

My main problem then was paying the phone bill. I’d get home from school and wait until 6pm for ‘off-peak’ calls, though that put me at odds with the rest of the household wanting to use our single landline for making and receiving calls, and it was still far from cheap. I think eventually we subscribed to BT’s ‘midnight line’ where it was possible to pay a fixed fee for unlimited calls between midnight and 6am – like many teenagers I became semi nocturnal – though the noise from my typing would sometimes result in angry shouts from my mum to pack it in and go to sleep.

Compunet had a great community, and I remember being able to find people who’d help me out with some of the crazy homebrew hardware projects I used to engage in at the time.

Kermit

Some of the companies and organisations I worked with on my evening/weekend jobs found that they needed to send files from office to office, so I created scripts that made the modems connect then transfer files with Kermit.

JANET/CIX

First year at University meant living in halls of residence, which in turn meant no access to a telephone point to use a modem. It didn’t matter much as I had my Amiga and PPC640 on hand. The fact that the University network was connected to JANET, and in turn the entire Internet eluded me at that time.

That all changed in second year. Project work meant burning the midnight oil, and a dialup connection to the University’s VAX cluster gave me a jumping off point into the Unix boxes of both the Electronics Department and Computer Science department, and from there I had worldwide connectivity. The World Wide Web hadn’t been invented yet, so I gorged on a diet of Telnet, FTP and Usenet guided by Zen and the Art of the Internet. One of the amazing things at the time was that people would give you Telnet access to their computers if you just asked. It was also a time when almost everything was connected to the Internet without any firewalls.

At roughly the same time I signed up to CIX[1], a service that I still use to this day. CIX was the place to be amongst the UK’s IT savvy during the early 90s, and it gave me the chance to electronically rub shoulders with IT journalists whose stuff I’d been reading in magazines like PCW for years.

WWW and ISPs

The World Wide Web (WWW) was born just before I left University, but I don’t recall using it then. My first memory of browsing was downloading some pictures from the site for Pulp Fiction using the text browser in CIX’s ‘go internet’ portal. The Lynx based text browser wasn’t the best way to view the web, but at this stage I didn’t have a proper Internet Service Provider (ISP).

My first try of a proper web browser was Netscape on OS/2 WARP, which came with a trial of IBM’s dial up Internet service (which I also managed to get going with Windows 95). By that time I’d ditched the built in modem on my PPC640 for a 14.4kbps Pace Pocket Modem (originally bought to go with a Sharp PC3100, but by then used with a homebrew 486 PC). Shortly afterwards CIX launched a dial up Internet service that I could combine with my usual subscription, so that was an easy switch to make.

Since then it’s been a succession of better browsers with Internet Explorer, Firefox and Chrome, better dial up speed wit 56k modems, then better ISPs/bearers with Nildram ADSL and now PlusNet VDSL. What a shame the UK government haven’t been doing more to encourage fibre to the premises (FTTP) in new build homes, as I’d love a gigabit service.

Note

[1] I still subscribe to CIX, which means I’ve had the same email address for 24 years. If you know that address (or my University email) then you can go back and see my (now sometimes super embarrassing) early Usenet posts.


A quick overview of WebVR based on Ada Rose Edwards’ awesome ‘getting started with WebVR‘ presentation that I saw at Nineworlds Geekfest

The demos that I showed off can be seen from Ada’s GitHub pages:

Sadly the odd colour basic demo and the T-Rex thing haven’t (yet) made it from the dev container on my laptop to anywhere in the outside world.

If you want to try developing this stuff yourself then it’s Dockerised:


sudo docker run -dp 4000:4000 --name webvr cpswan/webvr

The title page comment about no conspiracy is because I was the 3rd former Royal Navy person in a line up of 6 speakers at an event that has no leaning towards military/defence stuff – what are the chances of that?


In my last post ‘The Surveillance Party‘ I wrote about how the UK Labour Party used their ersatz SIGINT operation to exclude me from their leadership election process. I was told ‘You posted inappropriate comments on social media on 5 July 2016’, so let’s take a look at my tweets and see what might have scored as ‘racist, abusive or foul language’.

Before proceeding it’s worth noting for any readers who don’t know me personally that I don’t use Facebook, which is why I’m just looking at Twitter.

Racist

I don’t think any of my tweets that day (or any other day) were racist or could be interpreted to be racist. I do however realise that as a reasonable well off, middle class, middle aged, white, CIS guy I get to play life on the easy settings, and may not notice the hidden bias in my language; so comments welcome on where I’ve transgressed so that I can correct my behaviour going forward.

Foul language

This one is perhaps a slam dunk:

IOS_TPlink_RT

I’ve RT’d an account with a naughty word in its name, which would have been an easy hit for an analysis system with a ‘foul language’ blacklist. It may not have mattered that it wasn’t me using that word – guilty by association.

Abusive

This is where we venture into the political correctness twilight zone, so I’ll tune for the most hysterical interpretations of what might cause offense and thus be interpreted as abuse…

Immigrants

Here’s a reply to Dick Morrell venting his anger about Brexit:

immigrants

In this tweet I imply that some of the Brexit voters don’t like immigrants. I think from the media coverage that it’s a fact that many Brexit voters voted that way because they don’t like immigrants, so I’ll take that as a fact; but I recognise that people can still take offense from statements of fact, especially when they’ve aligned themselves with a group associated with the fact (whether it’s a view they hold themselves or not).

Rail unions

Here’s a complaint about the atrocious state of the South Coast main line for the last few months:

train_jam

This could be taken as a jab to the Rail Maritime and Transport workers Union (RMT) and their ongoing industrial action, though that day it was just run of the mill signalling problems.

Disabled/old

Here’s an RT of a wonderful article showing how technology (in the shape of the Apple Watch) has helped a deaf/blind person. Based on the excellent CloudCamp London presentation by Chris Lewis I repeated his point that old age will make us all disabled to some degree:

disabled

Could this somehow be interpreted as abuse of disabled people and/or old people?

Blairites

This is definitely my most political tweet of the day, but it’s quite subtle in that it doesn’t directly mention Blair or Labour. My friend Justin Cormack had tweeted an FT article about a potential split in Labour along the lines of when the ‘gang of four’ split to create the Social Democratic Party (SDP). I replied:

heels

So this is a pretty clear statement of fact. The ‘gang of four’ weren’t any of those things; but by implication I’m pretty clearly accusing the potential ‘SDP mark 2’ splitters of being:

  • neoliberal (a label which no politician ever seems to have aligned themselves with – there are no self describing neoliberals).
  • war criminal aligned – because Blair and his cronies lied to the British public to engage in an illegal war in Iraq that’s had terrible consequences here in the UK, but even more terrible consequences for Iraqis. Frankie Boyle summed it up perfectly a few weeks earlier here, and let’s not forget that the entire purpose of the #chickencoup seemed to be to derail Corbyn ahead of the publication of the Chilcot enquiry. As Blair hasn’t been found guilty of anything (and I expect he never will – at least outside of the court of public opinion) I was perhaps missing an ‘alegedly’, but 140 characters!
  • heels – according to the Oxford dictionary ‘an inconsiderate or untrustworthy man’ (informal, dated). Potentially a term of abuse, but hardly a very strong one.
  • clutching for power at any cost – because they’d rather split from the party that got them voted in than face deselection for not toeing the line from the leader elected by their party.

It takes what I expect to me more than £22 worth of analysis to pick this one apart, and anybody thinking that they’re the target of my ‘abuse’ here is almost certainly a more genuine enemy of the party than I ever will be – because they’re the ones who would split away to satisfy their own political careers.

#ChilcotReport #ChickenCoup

This RT doesn’t show up in the advanced search I linked to above, but I did RT it on 5 Jul:

chickencoup

It’s clearly a political tweet, but does calling some MPs ‘chickens’ count as ‘abuse’?

Conclusion

Along the lines of ‘rather be hanged for a sheep than a lamb’ I’m hoping that my offending tweet was the ‘SDP mark 2’ one where I call potentially splitting Blairites ‘neoliberal war criminal aligned heels clutching for power at any cost’, but knowing how this type of sausage is made I’d bet that it was the naught word in name RT :(


TL;DR

The UK Labour Party has been running an ersatz signals intelligence (SIGINT) operation to identify and exclude members and supporters that they don’t want voting in their leadership election; people who under some sort of criteria are identified as enemies of the party. This should be terrifying, as the difference between enemy of the party and enemy of the state is an election away (and if that party gets its hands on the levers of state power then it levels up to a full capability SIGINT operation).

I don’t think it’s overstating things to say that this is the most significant post Snowden example of the impact of mass surveillance on democracy.

Background

A vote of no confidence in leader Jeremy Corbyn from the members of parliament (MPs) in the Labour Party has triggered a leadership election. I won’t go into the details here, as it’s covered in its own Wikipedia entry. The party has been crawling through social media to identify unwanted members and supporters and exclude them from the party or leadership election process. The Twitter hashtag for this process is #LabourPurge2.

I applied to be a ‘supporter’ of the Labour Party as I feel it’s important that the country has an effective political opposition and I wanted to have a say in the process. Here’s the letter I got:

LabourPurge2

I’ll use another post to pick apart my 5 Jul 2016 Tweets for what might be considered ‘racist, abusive or foul language’ (and whether any of that could be construed as being aimed at Labour Party members) as I don’t want to get sidetracked on this post.

The point here is that they went a few weeks back before finding something that matched the criteria to exclude me. The interesting questions I’ll spend the rest of the post on are: who are ‘they’, what are ‘the criteria’, and how is this all being done?

‘They’ are watching, in secret

Somewhere at the core of the Labour Party there’s a bunch of people who decided that this was necessary, and somewhere else there’s a bunch of people actually doing the spade work. I’m not going to waste time here speculating about the motivations of the former group. The interesting part is that even with open source collection (like public tweets or Facebook posts) and sophisticated analysis tools there needs to be some degree of human assessment. The humans doing that are (like in the case of grown up SIGINT operations) having to do their work in secret, because the reveal the details would be political dynamite. The last thing that the group directing this will want is their criteria being spilled out (like say the ATOS fit-to-work checks), so they can’t just outsource it to the lowest bidder – a high quality circle of trust needs to be established here.

This is where I necessarily stray into speculation (because the facts are hidden):

  1. I expect that the work is being done by a law firm, because they provide clerical/analytical skills for hire, and understand (and most importantly respect) non disclosure agreements. Another possibility would be a ‘big 4’ type consulting firm.
  2. The £25 paid by ‘supporters’ is being used to pay the professional services fees associated with this whole exercise.
  3. Somebody had already figured out the cost of this exercise hence the steep jump in the ‘supporter’ fee (it was £3 last time).

What are the criteria?

The precise criteria of what makes an ‘enemy of the party’ are secret, and that’s a big part of the problem, but we can get some feel for it by observing the responses to and reactions of those being excluded.

Isn’t this just about enforcing the rules?

I’ll take a brief diversion here on the topic of party rules and how they’re interpreted. One might argue that the criteria are publicly and transparently embodied in the party’s rules, and this whole exercise is simply about ensuring that the rules are properly enforced. It’s quite fair and proper that a party doesn’t want people who are racist, abusive or use foul language; but each of those things is open to some degree of interpretation, and it’s in that interpretation that the true criteria for this exercise lie. When I pick through my tweets for 5 Jul the only thing I can be sure of is that I didn’t press the racist button – the criteria are subtle enough that it’s not clear to me which tweet got me found out as an enemy of the party. What I am pretty sure of is that if it hadn’t been 5 Jul then there would have been something incriminating further back in my timeline.

It’s harder on members

I was only trying to be a ‘supporter’ before being found out as an enemy of the party and excluded. People who’ve switched allegiances from other parties to Labour have found themselves excluded for their past support of those previous parties. Here’s an example from Ben Crawford (picked from the top of the pile on #LabourPurge2):

LabourPurge2_Ben

Here lies the point about the dangers of using SIGINT like this – people are being found guilty for their sins of the past. It creates a world where nobody can engage in politics and change their mind (because they think something changed or they were presented with a better argument, reasoning or set of data). It means that only the purist ideologues can engage in the political process. It is exclusionary.

Conclusion

I’ve used the word ‘exclude’ a lot in this post, because that’s what this is about – exclusion. Rules against people who are racist, abusive or use foul language are there to create an inclusive environment, but they’ve been twisted into a set of SIGINT filters to identify ideological enemies and exclude them. We’re getting a front row seat here to what happens when modern SIGINT is used for political purposes, and I’m writing this because I don’t like its impact on democracy.


TL;DR

Many SSDs are also Self Encrypting Drives (SEDs) they just need a few bits flipped to make them work. As the SSDs use encryption under the hood anyway there’s no performance overhead.

Background

This is something of an almanac post after a couple of days of prodding around the topic of PC device encryption. I wanted to make sure that the PCs I use for work stuff were properly protected, but I also wanted to minimise the impact on performance.

Bitlocker

As my laptop runs Windows 8.1 it seemed obvious to check out BitLocker, but a quick search revealed that software based BitLocker has some degree of performance overhead.

In the end I actually went with BitLocker on my laptop, as the SanDisk X300 SSD I have isn’t a SED (as it doesn’t support Opal or Microsoft eDrive), which is a shame as the article I found on the X300s gives a pretty good review of what’s out there.

Even if I did have an X300s rather than a plain old X300 the eDrive/BitLocker combination wouldn’t have been easy, as it requires doing a clean install of Windows rather than letting you keep your existing setup.

SEDs

SSDs use encryption internally anyway so that the blocks written to flash memory don’t have long runs of 1s or 0s, so it’s almost trivial for an SSD to also be a SED – all that’s needed is a means to manage the keys that are used to unlock that encryption. Out of the box SEDs are like safes with the door open and no combination set – they just need some tools to set the lock.

Class 0

With my desktop machine (a NUC) I’ve got a Samsung SSD that supports three different modes of encryption:

Magician_Security

  • Encrypted Drive is eDrive/BitLocker – too much trouble to configure
  • More on Trusted Computing Group (TCG) Opal below
  • Class 0 just uses a BIOS boot password. After reading this piece on Class 0 I decided it was probably worse than useless.

Opal and sedutil

The X300s article had run through the basics of Opal and use of the Wave Embassy app to enable it. Sadly as I have just a plain X300 I wasn’t getting a free license for that. There are a bunch of commercial offerings for Opal, from the usual suspects, and frankly they all look awful.

Open Source to the rescue… the Drive Trust Alliance offers sedutil for Windows and Linux. It’s a combination of a command line tool to configure Opal, and a Linux based pre boot application (PBA) to ask for the password that unlocks your drive.

After a bit of downloading and testing I confirmed that I was good to go, and following the encrypting your drive instructions worked perfectly.

The user experience

Most of the time the encryption is totally seamless in terms of performance and use experience. The only change is at boot (or resume from hibernation) when the PBA is launched first and asks for a password – the system then unlocks the SSD and reboots into the normal OS.

No Sleep

The one issue seems to be that the system will no longer make use of sleep mode, instead dropping into hibernate (to force a request for the password for resume). I can see why that’s more secure, but for my own use case I’d be happy to have sleep/wake without being asked for a drive password.

Conclusion

I wish the drive in my laptop was a SED. The BitLocker performance overhead isn’t too annoying, and it didn’t even take too long to encrypt the whole SSD, but it’s still sub optimal.

Using open source tools with the SED in my desktop was quick and easy. So if I’m even unlucky enough to be burgled I won’t have to worry about the data on that device.


I got an email proclaiming:

AmazonFresh – now in your area

If that wasn’t exciting enough in its own right (which would probably be the case) it went on to say:

Get a £20 Amazon.co.uk gift card when you spend £60 or more on an AmazonFresh order and have it delivered between 16 – 30 June, 2016

I’m a sucker for a deal, so I thought I’d give it a try (in place of my usual Waitrose Deliver order).

The first problem was when I clicked on the link in the email https://www.amazon.co.uk/b/?ie=UTF8&node=10407261031&bbn=6723205031

Amazon_Amp

Ah – the old & to & gag. A few backspaces later and I’m at this:

Fresh_Postcode

Alarm bells are starting to ring now. You know where I live Amazon. You know the postcode. Rule #1 of UX is don’t ask a human to answer a question where the computer already knows.

So then I enter my postcode:

Fresh_postcode2

And I look back at the email:

… The Offer is limited to selected London postcodes where AmazonFresh is available …

So AmazonFresh isn’t now in my area, because my area isn’t London, that’s not where I live (and Amazon knows that). So the whole thing is a ham-fisted waste of time.

I look forward to my London friends telling me how great Fresh is. If they can get past the old & gag.