TL;DR

The Administrator setup for Google Apps Migration guide makes things look pretty straightforward, but it’s much, much more complicated. What should be just a couple of check boxes turned out to be a twisty turny journey through hidden menus littered across distant parts of the administrators console.

Background

The move from CohesiveFT to Cohesive Networks meant I needed to move all of my email out of one service and into another. Last time I did this it was easy – suck email down from old account using an IMAP client (Outlook), then push email up to the new account via IMAP. Obviously this was too much of a good thing, and was hurting Google’s poor, tiny and fragile infrastructure[1].

It all started out fine

I actually had no problem whatsoever pulling down all of my emails from the old account, even though at 3.1GB of data it should have bust my bandwidth limit. The trouble began when I tried to upload to the new account. About 30 items (of about 35,000) made it over, and then it choked.

Google Apps Migration for Microsoft Outlook

Next I tried the official tool. But that didn’t get me very far:

GAppsMigration

I didn’t have admin access to the new account, but I was assured that the Email Migration API was enabled. If you were an admin, and you saw this then you’d probably think everything was fine:

GAppsAPI

Further down the same page there’s a section about the Email Migration API. It doesn’t actually let you do anything – it just links to this (not very helpful) web page:

GAppsEmailAPI

Show More

To actually get headed in the right direction you first have to click on the little ‘Show More’ at the bottom of the Security page[2]:

GAppsShowMore

This brings up the ‘Advanced settings’ option. It will remain a mystery of the universe why Google choose to hide a single extra item with a ‘Show more’.

GAppsAdvanced

At this point you might jump straight at ‘Manage API client access’ but don’t. It’s ‘Manage OAuth domain key’ that you want first:

GAppsDomainKey

Now check the box to ‘Enable this consumer key':

GAppsConsumerKey

It takes a few minutes for this to take effect. So grab a coffee or check email or something before returning to the ‘Manage API client access’ part:

GAppsManageAPI

Now paste in your domain name and the URL for the email API, which is https://www.googleapis.com/auth/email.migration

GAppsMailAPI

If you’ve waited long enough after enabling the consumer key for your domain then Authorize should work.

We’re not done yet

At this stage I managed to upload about 70 emails from the tool before it failed complaining about network issues. Subsequent attempts didn’t get any further.

A visit to Apps > Google Apps > Setting for Gmail > Advanced settings revealed some additional boxes to be checked:

GAppsGmailAdvanced

GAppsEmailUploads

 

Got there in the end

The migration tool still didn’t work, but I was now able to upload via IMAP (just as I’d planned to do in the first place). It took a whole day, but it got there in the end.

Conclusion

It’s quite possible that I could have made my Outlook IMAP upload work just by doing the last bit (in the Google Apps menu).

Enabling mail API access, which is what the migration tool seems to want, is much harder than it should be (or is made out to be). It’s also pointless, as the migration tool doesn’t seem to work properly.

I can’t end here without saying

The only time I ever use Outlook (which I despise) is for doing this sort of thing. Well… it ought to be useful for something.

Notes

[1] Since Google’s infrastructure is basically the largest in the world I’m struggling to imagine what sort of abuse let to them clamping down on email uploads, but I’d bet it has something to do with spammers.
[2] The Google Apps Migration for Microsoft Exchange Administration Guide (pdf) got me pointed in the right direction here.


I fell into a trap with my new Gen 8 Microservers like this:

  1. Install 60 day trial license for iLO Advanced
  2. Update BIOS date/time
  3. Find that trial license has now expired :(

There really should be some sort of warning on the license page (and maybe also the serial/password tag) to say update your clock before applying a trial license. Here’s how I got things back to factory defaults:

Firstly press F8 at the appropriate part of the boot sequence:

Gen8_iLO_F8

The config tool opens on the option to set defaults:

Gen8_iLO_defaults

So just hit enter and then F10 to confirm:

Gen8_iLO_F10

That’s it – the trial license will now work again. If like me you set a more memorable password than the one on the factory tag then that will have to be reconfigured.


TL;DR

I’ve been a fan of HP Microservers since the original NL36 model. When the newer Gen8 servers came to market they were a bit pricey, but the cost has come down, and cash back deals have returned. Faster CPUs, larger official memory capacity, dual NICs and remote console capabilities makes these ideal for a home lab.

Background

I’ve been working on our new vns3:turret platform a lot recently. It’s designed to run on enterprise networks rather than in the public cloud, which means that I needed some VMware hosts to play with. My older NL36s and NL40 Microservers were pressed into action, but the need for more capacity pushed me towards the latest model (which isn’t all that new any more, and might well be replaced by a Gen9 offering any day[1]).

Price

A bare bones model with G1610T CPU, 2GB RAM and no disk is presently £149.95 (£179.94 in VAT) at ServersPlus. HP are offering £35 cashback so that’s an out of pocket cost of £144.94 – not quite as amazing as when the original Microservers came with £100 cash back, but not far off.

I went for the 16GB ESXi 5.5 Test Bed Bundle, and ServersPlus did an excellent job of getting me the machines quickly and efficiently.

Construction

The Gen8 looks a lot prettier than the earlier model, and it’s much easier to get the motherboard out (though that’s only necessary for a CPU upgrade as the RAM is now easily accessible).

Unfortunately the 5.25″ drive bay has been sacrificed for a laptop style optical drive slot, which limits additional storage options. The eSATA port has also disappeared.

The newer drive caddies don’t feel as robust as the older ones, not that it matters once a disk is screwed in.

Remote insight

Probably the best feature of the Gen8 is the inclusion of HP Integrated Lights-Out (iLO), which can be used to provide a remote keyboard/video/mouse (KVM) capability. Out of the box the remote console only works until the OS boots, but an iLO advanced license provides the ability to use KVM after boot. Those licenses are hideously expensive at full sticker price, but there’s a healthy secondary market, and I found one on Amazon for less than $20. A 60 day free trial license can also be obtained.

Since I keep the servers out in my garage (which is presently very cold) I’m glad that I don’t have to go out there.

Upgradability

16GB of ECC RAM is officially supported and very easy to install. It’s a shame it’s not 32GB, but with the standard CPU offerings the balance is probably right.

One of the things that put me off the Gen8 when it launched was the weedy CPU range. The Celeron G1610T and Pentium G2020T on offer are both a bit weak (though notably better than the AMD CPUs in earlier Microservers). Fortunately the CPUs are upgradable. I was able to find a couple of E3 1220L V2 parts on eBay for £129 each[2], which at 17W power rating are an ideal upgrade option. Others have had success with 45W CPUs such as the E3 1265L V2, and many have even got away with running full power 69W parts such as the E3 1230 V2 (even though the heat sink is only rated at 35W)[3].

Besides the extra speed on offer my main reason for doing a CPU upgrade was to get VT-d, though my attempt to pass through the B120i storage controller to a VM failed.

We’re going to need a bigger boat switch

The Gen8 has two integrated Broadcom GigE ports (which is great for VMware) plus the iLO has its own port (though it can share one of the main ports if required). Along with buying secondary GigE NICs for the other servers in my garage this has quickly pushed me from 5 ports to 8 ports to 16 ports

Running ESXi

The supplied USB drive with the HP customised ESXi 5.5 install just worked, and I was immediately able to start installing VMs onto iSCSI and NFS storage without even putting any drives into the bays. I’ve yet to load up these machines, but I’m tempted to migrate over a bunch of VMs from my present Hyper-V setup on a Dell T110 II as potentially both Microservers will have a lower power budget than the single larger server (and provide better tolerance to a single machine hardware failure).

NAS potential?

I had a go at installing NAS4Free on ESXi using raw device mappings (RDM) to 4x 2TB HDDs. Everything seemed to work pretty well, and I was able to get a nice big RAID-Z volume. That’s a setup I’d probably only use for warm storage or media files as I’d want SSD for anything else.

Conclusion

I really like the Gen8 Microserver. It’s proper server engineering in a small, cheap and elegant package. The best bit is the iLO capability, but there are plenty of other things to like about it.

Notes

[1] I’m not too concerned about the possibility of newer Microservers, as the Gen8 is very capable, and the Gen9 is unlikely to be offered at such a bargain price.
[2] In some places the Gen8 is available with the E3 1220L V2, though I’ve never seen it on sale in the UK.
[3] There are so many CPU choices that there’s a FAQ about them.


The announcement of Rocket by CoreOS was perceived by many to be a direct challenge to Docker, particularly as it came on the eve of DockerCon Europe and threatened to overshadow news coming out at the event. Docker, Inc. CEO Ben Golub was quick to fire back with his ‘initial thoughts on the Rocket announcement’. This piece isn’t about the politics of ecosystems and VC funded startups, which I’ll leave to Colin Humphreys (and note an excellent response from Docker Founder and CTO Solomon Hykes). It also isn’t about managing open source community, which I’ll leave to Matt Asay. Here I want to look at systemd, which lies at the heart of the technical arguments.

continue reading the full story at The New Stack


At their re:invent 2014 show Amazon launched AWS Key Management Service (KMS), “a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data, and uses Hardware Security Modules (HSMs) to protect the security of your keys”. At launch the service supported EBS, S3 and Redshift. Additional support for Elastic Transcoder was added in late November.

continue reading the full story at InfoQ


When I first saw the LinkIt ONE at Mediatek’s stand at the Web Summit it was being shown of with a bunch of neat modular peripherals. The person on the stand had no idea about them, and couldn’t explain why the board had its own connectors for these modules. It didn’t take much sleuthing to discover that the modular system was Seeed Studio’s Grove system[1], and I ordered myself a Grove Starter Kit (for Arduino[2])

The kit comes with a bunch of sensors and actuators. It’s much like what can be found in an Arduino starter kit like the ARDX, but the key is that everything is connected together with the same 4 wire leads rather than having to plug individual components and wires into breadboard.

The simplicity of connection makes prototyping much quicker and easier, so I think the Grove kit does a good job of making Arduino development more interesting and accessible. I’m also pleased to say that the hardware is backed up by decent libraries and demos (licensed with LGPL).

Notes

[1] Mediatek partnered with Seeed to create and distribute the LinkIt ONE.
[2] They also do starter kits for the LinkIt ONE, TI Launchpad and a variety of other dev boards.


TL;DR – The LinkIt ONE is an awesome Arduino, with a ton of great integrated peripherals, but I fear there’s too much of a gap between Arduino style development and building the next generation of connected things – though hopefully that gets covered by the forthcoming Eclipse based SDK.

Introduction

I was at the Web Summit in Dublin a month ago, and one of the key themes that stood out for me was cellular connectivity for devices. I saw a ton of things that were taking advantage of low end, low cost and low power data services – mainly 1G/2G stuff like SMS and GPRS (not power hungry high bandwidth 3G). The coolest thing I saw at the event was the Brewbot (which doesn’t [yet] have any cellular), but not far away was a stand showing off Mediatek’s LinkIt ONE.

The LinkIt ONE development board is an open source, high performance board for prototyping Wearables and IoT devices

The board

The board has the familiar Arduino form factor, and can be programmed from the Arduino IDE, so it ends up feeling like an Arduino bolted on to a mobile phone. It comes with antennae for cellular (SMS/GPRS), bluetooth/wifi and GPS. There’s also a LiPoly battery in the box meaning that you can straight away make projects that could be put in a tupperware box, left in a field and send data back to base – all for $79 from Seeed Studios (who Mediatek partnered with for distribution).

The system on chip (SoC) used by the LinkIt ONE is Mediatek’s MT2502A. It isn’t actually a cellphone SoC, and (according to the pdf documentation) is intended for wearables.

Installation

After a brief fight with the Mediatek registration wall I found the SDK on github. I then moved on to a fight with Windows 8 to install the drivers[1]. With a fresh install of the IDE in place it was easy to get the blinking LED ‘hello world’ demo going.

Huge memory

One thing that stands out straight away is that the capacity of the LinkIt ONE is huge compared to a more typical ATmega328 based Arduino like the Uno. There’s little risk of running out of memory – my simple projects never reported more that 0% usage!

I build a pager

The project I ended up putting together at the ThingMonk hack day was a pager. It receives SMS messages and displays them on a dot matrix screen. I used a Grove LCD RGB Backlit display, which is part of the excellent Grove Starter Kit (something I’ll write more about another day), so I was able to make the display change colour to red/green/blue/white according to the first letter of the received message.

LinkItONEpager

The code is on GitHub, and makes use of the Seeed Studios Grove LCD RGB libraries.

Great but…

The Arduino form factor and development environment have a very easy on ramp for quick and simple development, but I fear it wouldn’t be long before I’d feel held back by its limitations and want to do something more native. I also wonder how I could take the next step and turn something I developed with this board into a (mass produced) product. Mediatek obviously want to sell more SoCs, but this approach means that I don’t get close enough to the SoC to do much with it. I’d like the SDK better if it also had libraries, demos etc. for C (and maybe a real time OS).

It’s relatively simple to take a project developed on a regular Arduino and move it to an ATmega328 kit (like a Shrimp), and it’s then another easy step to port things onto an ATtiny or similar. That progression is presently missing in the Mediatek environment.

Update 5 Dec 2014 – maybe MediaTek have updated their site, or maybe I just wasn’t playing close enough attention in my rush to get started, but I see a promise of ‘an Eclipse version planned for later this year’, which I think will be pretty much exactly what I’m looking for.

The LinkIt OS is based on an RTOS kernel. On top of this kernel is a set of drivers, middleware, and protocol stacks that expose the features of the chipsets to a Framework.

I can hear the open source zealots bleating from here about proprietary binary blobs, but embedded development sadly remains about a decade behind the mainstream. If the APIs are as easy to consume as the Arduino examples then I’ll be pretty happy.

Conclusion

I’m looking forward to doing more projects with the LinkIt ONE. It’s the best Arduino clone I’ve ever tried, and seems amazing value for money compared to buying a whole bunch of shields for a regular Arduino. Sadly there seems to be a gap between the Arduino style of development and unleashing the full capabilities of the hardware, but hopefully the Eclipse SDK fixes that.

Note

[1] The SDK claims to support Windows 8, but you need to jump through hoops to disable enforcement of driver signing to get the drivers installed. This is sadly all too typical even though Windows 8 has now been out for 2 years.


TL;DR – it runs – now I need to put together some GPIO nodes.

Updated 5 Dec 2014 – In my original post Node-RED was so slow it was unusable. Using Michal Vondráček’s node-ws package fixed that.

Background

The WRTnode website features a screenshot of Node-RED, so I thought it would be fun to make it come true.

Node.js on WRTnode

My first hurdle was to get Node.js running on the WRTnode. Node has previously been run on OpenWRT, but that implementation was old and specifically targeted to a big endian MIPS architecture. Luckily Michal Vondráček published a working WRTnode implementation of Node.js the day before the ThinkMonk hack day.

Installation

I next struggled with installation, as ‘npm install –production’ was first running out of memory and then complaining about a lack of filesystem locks. Thankfully Node-RED creator Nick O’Leary was on hand to point out that I could simply copy an installed Node-RED from another system (like a VM on my laptop).

It runs

With Node.js installed (mostly onto a USB stick) and Node-RED also copied onto the USB stick I was able to start Node-RED, see it coming up on port 1880 and browse to it.

NodeRED_WRTnode

Having installed Michal’s node-ws and deleted the ws package from the Node-RED node_modules directory everything works:

NodeRED_WRTnode_working

Todo

I now need to get the GPIO mapped so that I can get Node-RED to blink some lights etc.

Too slow

Sadly running interpreted JavaScript on a low end MIPS CPU doesn’t result in a development environment that anybody would want to use. After half an hour or so of waiting I didn’t even see the tool palette load. I squashed any ideas of moving on to GPIO mapping for the WRTnode.

Can it be made to work?

Possibly – more of the code needs to be compiled to native MIPS. I don’t really know whether this is an issue that can only be addressed in the core V8 runtime, or whether some smart cross compilation of modules might help. Since Nick told me that most of the modules are now pure JavaScript, implying that the clever optimisation has moved from the edge to the core.


Netflix have announced the release of the Message Security Layer protocol (MSL), which they describe as ‘A Modern Take on Securing Communication’. The project is available on github under the Apache 2.0 license, with implementations in Java and JavaScript. The high level goals of the protocol are to improve performance, be cross language, flexible and extensible, standards compatible and offer automatic error recovery.

continue reading the full story at InfoQ

MSLperformance


The TP-Link WR-703N is a nice, hackable pocket WiFi router. I’ve seen them put to some interesting purposes, but I didn’t own one until yesterday when the kind folk at QCon Shanghai gave me one. I’m not a fan of factory TP-Link firmware at the best of times, but more so when it defaults to Chinese, so it was time for another OpenWRT install (like I’ve done in the past with my TL-WR2543ND and TL-WR841N [1]).

First read the official OpenWRT Wiki article. The point of this post is to cover stuff that’s not presently covered in the Wiki.

I downloaded the stable Barrier Breaker release 14.07, r42625 factory update image and blundered through the Chinese language web UI to upload the new firmware and flash it.

By default the OpenWRT image comes up with no WiFi, and the ethernet port configured as LAN. This makes perfect sense from a security perspective, but not much sense for this particular device.

Turning on WiFi

Having connected to the ethernet port from my laptop I was able to access the web UI (and SSH). From there the WiFi can be set up by clicking on Network > WiFi.

Enable WiFi:

TL-WR703N-WiFi-enable

Provide a name for the network and make it part of the LAN:

TL-WR703N-WiFi-name

Then turn on security and provide a key:

TL-WR703N-WiFi-key

At this stage it’s time to connect to the WiFi interface – using the SSID and security key that were just configured. Unplug the ethernet cable to confirm that connectivity is still OK.

Ethernet == WAN

Next go to Network > Interfaces and Add new interface…

TL-WR703N-WAN

At this point the ethernet port (eth0) is on both the lan and wan, which isn’t good, so go back to the lan interface and edit out the eth0:

TL-WR703N-LAN

It should now be OK to plug the ethernet into a hotel outlet, your home network or whatever else you’re trying to WiFi enable with the mini router.

WiFi bridging (not working)

One of the things I love about WRTNode is the ability for it to route on from another WiFi access point, so with hotel WiFi you can sign in many devices but only need one registration. I’m told that it should be possible with the TL-WR703N, but I haven’t figured out how to do it. Putting in a similar block of config to what I use on WRTNode doesn’t seem to have the desired effect of creating an apcli0 device:

config wifi-iface
    ....
    option ApCliEnable '1'
    option ApCliSsid 'theSSID'
    option ApCliAuthMode 'WPA2PSK'
    option ApCliEncrypType 'AES'
    option ApCliPassWord 'theWiFiKey'

Note

[1] Both of these routers support the Barrier Breaker release 14.07, r42625 firmware, though the relevant Wiki articles haven’t been updated to reflect that.




Follow

Get every new post delivered to your Inbox.

Join 105 other followers