Our New Cross-Platform Build, Test, Store and Deploy Architecture

First, let me preface this by saying no, I’ve not migrated any Compact Framework application to Visual Studio 2013. We’re still using Visual studio 2008 for CF apps, so don’t get too excited. That said, we’ve done some pretty interesting work over the last week that was interesting so please, read further.

Microsoft recently announced the availability of not just Visual Studio 2013, but also Visual Studio Online, which is effectively a hosted version of Team Foundation Server 2013.  We use the older TFS 2010 internally as our source control provider as well as for running unit tests, but it’s got some significant limitations for our use case.

The biggest problem is that our flagship Solution Engine product runs on a lot of platforms – Windows CE, Windows Desktop and several Linux variants.  For Linux we’re using Mono as the runtime, which means we’re using XBuild to compile and Xamarin Studio for code editing and debugging.  Well Mono, XBuild and Xamarin Studio don’t really play well with TFS 2010.  To put it bluntly, it’s a pain in the ass using them together.  You have to train yourself to have Visual Studio and Xamarin Studio open side by side and to absolutely always do code edits in Visual Studio so the file gets checked out, but do the debugging in Xamarin Studio.  Needless to say, we lose a lot of time dealing with conflicts, missing files, missing edits and the like when we go to do builds.

TFS 2013 supports not just the original TFS SCC, it also supports Git as an SCC, which is huge, since Xamarin Studio also supports Git. The thinking was that this would solve this cross-platform source control problem, so even if everything else stayed the same, we’d end up net positive.

I decided that if we were going to move to TFS 2013, we might as well look at having is hosted by Microsoft at the same time.  The server we’re running TFS 2010 on is pretty old, and to be honest I hate doing server maintenance.  I loathe it.  I don’t want to deal with getting networking set up.  I don’t like doing Hyper-V machines.  I don’t like dealing with updates, potential outages and all the other crap associated with having a physical server.  Even worse, that server isn’t physically where I am (all of the other servers we have are) so I have to deal with all of that remotely.  So I figured I’d solve problem #2 at the same time by moving to the hosted version of TFS 2013.

Of course I like challenges, and Solution Engine is a mission-critical product for us.  We have to be able to deliver updates routinely. It’s effectively a continuously updated application – features are constantly rolling into the product instead of defined periodic releases with a set number of features.  We’ll add bits and pieces of a feature incrementally over weeks to allow us to get feedback from end users and to allow feature depth to grow organically based on actual usage.  What this means is that the move had to happen pretty seamlessly – we can’t tolerate more than probably 2 or 3 days of down time.  So how did I handle that?  Well, by adding more requirements to my plate, of course!

If I was going to stop putting my attention toward architecting and developing features and shift to our build and deployment system, I decided it was an excellent opportunity to implement some other things I’ve wanted to do.  So my list of goals started to grow:

  1. Move all projects (that are migratable) to Visual Studio 2013
  2. Move source control to Visual Studio Online
  3. Abandon our current InstallAware installer and move to NSIS, which meant:
    1. Learn more about NSIS than just how to spell it
    2. Split each product into separate installers with selectable features
  4. Automate the generation of a tarball for installation on Linux
  5. Automate FTPing all of the installers to a public FTP
  6. Setting up that FTP server
  7. Setting up a nightly build for each product on each platform that would also do the installers and the FTP actions
  8. Setting up a Continuous Integration build for each product on each platform with build break notifications

Once I had my list, I started looking at the hosted TFS features and what it could do to help me get some of the rest of the items on my list done.  Well it turns out that it does have a Build service and a Test service, so it could do the CI builds for me – well the non Mono CI builds anyway.  The nightly builds could be done, but no installer and FTP actions would be happening.  And it looked like I was only going to get 60 minutes of build time per month for free.  Considering that a build of just Engine and Builder for Windows takes roughly 6 minutes, and I wanted to do it nightly meant that I probably needed to think outside the box.

I did a little research and ended up installing Jenkins on a local server here in my office (yes, I was trying to get away from a server and ended up just trading our SCC server for a Build server).  The benefit here is that I’ve now got it configured to pull code for each product as check-ins happen and then to do CI builds to check for regression.  If a check-in breaks any platform, everyone gets an email.  So if a Mono change breaks a CF change, we know.  If a CF change breaks the desktop build we know.  That’s a powerful feature that we didn’t have before.

Jenkins also does out nightly builds, compiles the NSIS installers and builds the Linux tarballs.  It FTPs them to our web site so a new installation is available to us or customers every morning just like clockwork, and it emails us if there’s a problem.

It was not simple or straightforward to set all of this up – it was actually a substantial learning curve for me on a whole lot of disparate technologies.  But it’s working and working well, and only took about 6 days to get going.  We had a manual workaround for being able to generate builds after only 2 days, so there was no customer impact.  The system isn’t yet “complete” – I still have some more Jobs I want to put into Jenkins, and I need to do some other housekeeping like getting build numbers auto-incrementing and showing up in the installers but it mostly detail work that’s left.  All of the infrastructure is set up and rolling.  I plan to document some of the Jenkins work here in the next few days, since it’s not straightforward, especially if you’re not familiar with Git or Jenkins, plus I found a couple bugs along the path that you have to work around.

In the end, though, what we ended up with for an extremely versatile cross-platform infrastructure.  I’m really liking the power and flexibility it has already brought to our process, and I’ve already got a lot of ideas for additions to it.  If you’re looking to set up something similar, here’s the checklist of what I ended up with (hopefully I’m not missing anything).

Developer workstations with:

  • Visual Studio 2013 for developing Windows Desktop and Server editions of apps
  • Xamarin Studio for developing Linux, Android and iOS editions
  • Visual Studio 2008 for developing Compact Framework editions

A server with:

  • Visual Studio 2013 and 2008 (trying to get msbuild and mstest running without Studio proved too frustrating inside my time constraints)
  • Mono 3.3
  • 7-Zip (for making the Linux tarballs)
  • NSIS (for making the desktop installers)
  • Jenkins with the following plug-ins
    • msbuild
    • mstest
    • git
    • ftp

Responsible M2M

About a year ago, maybe two years now, we had a large manufacturing customer that we were working with to implement MTConnect on their production floor. Basically they had 20 five-axis machine tools running creating aircraft parts and they wanted to be able to get data off of those machines and “put it in The Cloud.” Well, first off I’ve talked about how much I dislike the term “The Cloud” so we had to clarify that. Turns out they meant “in a SQL Server database on a local server.”

MTConnect is a machine tool (hence the “MT” part) standard that we leveraged and heavily extended for use in our Solution Family products. Painting it with a broad brush, what it means is that all data from each machine tool – axis positions, part program information, door switches, coolant temperature, run hours, basically the kitchen sink – can be made available through a REST service running either on the machine tool or on a device connected to it.

They wanted to take that data and put it into SQL Server so their engineering group could run analytics on the data. Maybe they wanted to look at part times, energy consumption, tool path length, whatever. They actually weren’t fully sure what they wanted to do with the data, they just knew that “in The Cloud” is where everyone said it should be, so the commandment came down that that’s where the data would go.

Ugh. The conversation went something like this.

“So you want all of the data from each machine tool to go to the server?”

“Yes. Absolutely.”

“You know that there are 6 continually moving axes on those machines, right? And a constantly changing part program.”

“Of course. That’s the data we want.”

“You are aware that that’s *a lot* of data, right?”

“Yes. We want it.”

“You’re sure about this?”

“Yes, we’re sure. Send the data to The Cloud.”

So we set up a mesh of Solution Engines to publish *all* of the data from *all* of the machines to their local server. We turned on the shop floor. And roughly 20 seconds later the network crashed. This was a large, well built, very fast, hard-wired network. There was a lot of available bandwidth. But we were generating more than a lot of data, and the thing puked, and puked fast.

So what’s the lesson here? That you can always generate more data out at the edge of your system than the infrastructure is capable of carrying. If you’re implementing the system for yourself, trying to transfer all of the data is a problem, but if you’re implementing it for a customer, trying to transfer all of it is irresponsible. We did it in a closed system that was just for test, knowing what the result would be and that it would be non-critical (they simply turned off data broadcasting and everything went back to normal), but we had to show the customer the problem. They simply wouldn’t be told.

We need to do this thing, this M2M, IoT, Intelligent Device Systems or whatever you want to call it responsibly. Responsible M2M means understanding the system. It means using Edge Analytics, or rules running out at the data collection nodes, to do data collection, aggregation and filtering. You cannot push all of the data into remote storage, no matter how badly you or your customer might think it’s what needs to happen.

But that’s fine. Most of the time you don’t need all of the data anyway, and if, somehow, you do there are still ways you can have your cake and eat it too.

Let’s look at a real-world example. Let’s say we have a fleet of municipal busses. These busses drive around all day long on fixed routes, pickup up and dropping off people. These busses are nodes that can collect a lot of data. They have engine controller data on CAN or J1708. They have on-board peripherals like fare boxes, head signs and passenger counters. The have constantly changing positional data coming from GPS and/or dead-reckoning systems. They’re also moving, so they can’t be wired into a network.

Well we could send all of that data to “The Cloud”, or at least try it, but not only would it likely cause network problems, think of the cost. Yes, if you’re AT&T, Verizon or one of the mobile carriers, you’ve just hit pay dirt, but if you’re the municipality the cost would be astronomical. Hello $20 bus fares.

What’s the solution here? Well, first of all there’s a load of data that we have that’s near useless. The engine temperature, RPMs or oil pressure (or any of the other of the thousands of data points available from the engine controller) might fluctuate, but generally we don’t care about that data. We care about it only when it’s outside of a “normal” range. So we need Edge Analytics to be able to watch the local data, measure it, and react when some conditions are met. This means we can’t just use a “dumb” device that grabs data from the controller and forwards it on. Instead we need an Intelligent Device – maybe an Intelligent Gateway (a device with a modem) – that is capable of running logic.

Now when we’re out of the “normal” range, what do we do? Maybe we want to just store that data locally on the vehicle in a database and we can download it at the end of the shift when the vehicle returns to the barn. Maybe we want to send just a notification back to the maintenance team to let them know there’s a problem. Maybe we want to send a capture of a lot of a specific set of data immediately off to some enterprise storage system for further analysis so the maintenance team can order a repair part or send out a replacement vehicle. It depends on the scenario, and that scenario may need to change dynamically based on conditions or the maintenance team’s desires.

Positional data is also ever-changing, but do we need *all* of it? Maybe we can send it periodically and it can provide enough information to meet to data consumer’s needs. Maybe once a minute to update a web service allowing passengers to see where the bus is and how long it will be until it arrives at a particular spot. Or the device could match positional data against a known path and only send data when it’s off-route.

And remember, you’re in a moving vehicle with a network that may or may not be available at any given time. So the device has to be able to handle transient connectivity.

The device also needs to be able to affect change itself. For a vehicle maybe it puts the system into “limp mode” to allow the vehicle to get back to the barn and not be towed. For a building maybe it needs to be able to turn on a boiler.

The point here is that when you’re developing your Intelligent Systems you have to do it with thought. I’d say that it’s rare that you can get away with a simple data-forwarding device. You need a device that can:

– Run local Edge Analytics
– Store data locally
– Filter and aggregate data
– Run rules based on the data
– Function with a transient or missing network
– Effect change locally

Intelligent Systems are great, but they still need to be cost-effective and stable. They also should be extensible and maintainable. You owe it to yourself and your customer to do M2M responsibly.

Of course if you want help building a robust Intelligent System, we have both products and services to help you get there and would be happy to help. Just contact us.

Intel’s New Quark Core

This week Intel announced their new processor core named Quark.  It is smaller than their current embedded-focused Atom core (hence the name – smaller than an atom is a quark) and more importantly it uses about 10% of the power of an Atom.  We can probably assume it also produces a lot less heat, so your embedded devices will no longer double as a Panini press.

Intel has, unfortunately, been pretty vague about the Quark, so I’ll have to remain equally vague about some things.  We don’t know exactly when we’ll be able to actually buy processors using the Quark core (Q4 for general availability of evaluation systems?).  We don’t know what price point they are targeting (I’ve seen guesses at the $5 range).  We can be pretty sure that with the market we’re in that the definitive answers will be “soon” and “low.”

So what do we know, then?  Well, it’s x86 (so 32-bit) and probably single core. What?!  I can hear the screams now.  In fact reading the comments on several other tech sites, there seems to be a lot of furor about how it can’t compete with ARM processors shipping in phones and tablets today and how 32-bit, single-core architecture is so 1990’s and useless in today’s landscape.

I think those people are totally missing the point.  This isn’t a processor for a phone or tablet.  Intel has even said it isn’t.  Quit trying to place it into the devices you think of.  This baby is designed for the Internet of Things (IoT) and M2M, and I think it’s going to be a game changer.

M2M – and I’m just going to call it that for now, instead of IoT or the acronym I saw this week IoE (internet of everything. seriously)  – is growing.  It looks like it’s the next wave of “things to do” and happily, we’ve been doing it for a decade.

Quark is going to enable all of those applications using 16- and 32-bit microcontrollers to run full-blown OSes.  That means they’ll have access to connectivity.  That means they’ll be able to do local analytics and run local rules. It means they’ll be able to push data upstream to clouds.  It means they’ll start participating in overall solutions. That also means they’ll need security, but they’ll have the capacity to implement it.

The core itself is also synthesizable, meaning it’s “open”.  No, it’s not that anyone can go in and change the actual processor core, it’s not that open, but they can change the fabric, meaning they can build their own SoC with the Quark core directly wired to peripheral components like radios and crypto devices to further reduce cost and footprint.

I’m confident that we’ll have Solution Engine running on a Quark system very soon and it will be interesting to see how it performs compared to the Atom and ARM systems we’re already running on.

What I’d really love to see is someone building a Windows CE OS for it to give us low-latency, real-time capabilities coupled with the familiar Win32 API.  Since it’s still x86, that’s not a big stretch.

Intelligent Devices and the Internet of Things

Over the past couple years, I’ve seen the proliferation of some new terms. The ideas don’t seem new to me – hell, we’ve been doing the tasks these definitions describe for a decade or more and have a full product suite based on the ideas – but evidently some corporate management teams seem to have gotten interested, and so they needed catchy names, with alphabet soup acronyms. Here’s my take on what some of them mean.

Machine to Machine (M2M)

M2M seems to be all the new buzz. Oracle thinks it will be huge (a $25B market by 2015). Intel thinks it will be even bigger (15 billion connected devices by 2015). But really, the concept is pretty simple, and it’s really not new at all. M2M is simply two machines talking together. Typically it’s low-powered (resource wise) things. So it’s not your PC talking to a server, but it might be a sensor talking to an embedded device. Or two embedded devices sharing data.  What’s new is that we’re finally at the cusp of technology and cost where we can actually start fielding widespread M2M capabilities.

Intelligent Devices

Moore’s Law is has finally gotten us to where pretty small devices, using small amounts of energy and costing very little can do a lot of things. Not so long ago a municipal bus couldn’t tell you anything about itself. Then it could tell you about fault codes if you brought it back to the bus barn. Then it could tell you where it was. But now, you can put a small appliance right on the vehicle that can tell the driver if a problem requires returning to maintenance. It can even call maintenance and give them diagnostic info or even tell them what part to have ready. Basically any device out at the “edge” of an overall solution that is capably of doing more than just collecting sensor data is an Intelligent Device. Obviously there’s a pretty wide range of what can be done out there, depending on your solution’s ability to absorb size, power and cost.

Maybe the device does nothing but monitor a temperature and then send a notification when it exceeds a specific set point. That’s intelligence when compared to a device that just sends the temperature to a server where the comparison and notification is generated off-device. It means that the device is capable of using Analytics to make a decision like when the temperature exceeds 205F, send an alert.  It’s capable of using Aggregation to summarize, filter and/or generate new data like the alert condition itself which the temperature probe doesn’t have. It may be able to run Rules such as when the temperature is over 200F, start storing the oil pressure and passenger count.  It may be able to Publish any of that data to another device or to a server somewhere.

It’s this intelligence, and really the ability to distribute different parts of this intelligence across a many devices and machines in a solution that has started changing the landscape of what can be done. Before we had a machine shop where operators would report part counts at the end of a shift, and a manager would create a spreadsheet that they would send off to the customer at the end of the week or month. Now, the machines themselves can report exact part counts in real time right to the end customer. That is enabling a lot of cool things.

The Internet of Things (IoT)

The Internet of Things, or IoT, in my mind anyway, is the collection of all of these Intelligent Devices and how they use M2M to actually send useful information (as opposed to raw data) between one another.

Let’s look at a concrete example.

Let’s say I have some sensors in an apartment building that give current temperatures. These report back to an Intelligent Device. This communication may be M2M (or it might just be a thermocouple wire – let’s not get too hung up on it).

That Intelligent Device can record that data. It can run rules on that data to determine if it needs to turn on or off the boiler. It can send a signal to another device to actually power up the boiler – again this might be M2M. The Intelligent Device can do aggregation of data, providing hourly rollups of data and even calculating heating and cooling curves for the building it’s in. It can also publish that data to another device or to the cloud (we’ll talk more about this magical “cloud” word in another post). Yet again, here’s more M2M.

Now let’s say the Property Manager has 10 buildings. They can connect to the cloud and look at a history of the temperatures in an apartment. They can send commands from their office to the Intelligent device telling it to change a set point, or just override the current rules on the device and turn on the boiler. All of this interaction is an “Internet of Things.”