Eclipse Hono – Connecting large numbers of IoT devices

The value of an IoT solution often correlates directly to its ability to connect many different devices and integrate them into higher-level solutions. Although all solutions combine devices using a unique integration strategy, the reliable, secure, and scalable connection of devices is always crucial.

Connectivity is therefore best handled as a separate service, independent of a solution – and this is exactly what Eclipse Hono aims to accomplish.

Overview

Eclipse Hono is written entirely in Java. It also provides a uniform messaging infrastructure for IoT solutions. Eclipse Hono offers:

Dejan Bosanac

  • Senior Software Engineer @ Red Hat
  • Committer @ Eclipse Foundation / Apache Software Foundation

Follow Dejan on Twitter

  • a horizontally scalable microservice architecture
  • a design for container-based cloud environments (based on AMQP 1.0)
  • load protection for all microservices using AMQP 1.0 flow control
  • protocol adapters for different IoT protocols
  • handling of different communication patterns
  • a tenant-based security model, including authentication of device

Solutions integrated with Eclipse Hono are freed from the details of device connectivity, and can instead concentrate on providing business value.

IoT communication patterns and QoS

Most messages in IoT setups are usually telemetry messages: data flows from devices to solutions. Messages of this kind usually need not be guaranteed of reaching their destinations and can be handled especially efficiently because there are no acknowledgements involved (QoS “at most once”). A typical type of telemetry data is sensor data that is sent within short time frames.

Paolo Patierno

  • Principal Software Engineer @ Red Hat
  • Committer/Lead @ Eclipse Foundation

Follow Paolo on Twitter

Telemetry messages that absolutely must be delivered (QoS “at least once”) are called events, which are always acknowledged to the client and occur infrequently by comparison. Typical events include alarms that need a guarantee to reach the solution and must never be dropped.

The other direction is called command and control: data flows from solutions to devices. This addresses the control of actors, or in general messages initiated by the solution, and will be provided in future versions of Eclipse Hono (versions > 0.5).

Hono in production setups

Eclipse Hono defines APIs for all aspects of device connectivity:

Karsten Frank

  • Senior Software Developer @ Bosch Software Innovations
  • Committer @ Eclipse Foundation
  • Spring Core 4.2 Certified Developer

Take a look at Karsten's GitHub page

  • sending and consuming message payloads for different communication patterns
  • handling of device identities and device credentials for secure communication
  • authentication of devices and microservices

Because the handling of device identities is assumed to be specific to a production setup, it is provided by a simple file-based implementation. Due to the use of Spring beans and the Vert.x event bus, this implementation can easily be substituted by a database implementation.

The same applies for the (file-based) authentication server, which may be replaced with other implementations.

Bird’s-eye architecture view

There are several different categories of Hono components:

Infographic showing the different sections the components of Eclipse Hono are split into.

Protocol adapters

Protocol adapters connect devices to Hono. Each covers a specific protocol and is responsible for the authentication of devices by querying the device registry.

Hono provides the following protocol adapters:

  • HTTP
  • MQTT
  • Eclipse Kura-based gateways

Custom protocol adapters can be added by using the APIs of Hono Core.

Hono core

Hono core consists of:

  • the Device Registry, implementing
    • the registration and activation of devices belonging to a tenant
    • the provisioning of credentials for devices
  • the Auth Server, implementing
    • the construction of security tokens to support the authentication of a client
  • Hono Messaging, implementing
    • the messaging APIs for telemetry and event messages
    • verification of compliance with the APIs
    • the forwarding of messages to the AMQP 1.0 Network

AMQP 1.0 Network

The AMQP 1.0 Network decouples Hono’s core microservices from solution applications. It provides a scalable messaging infrastructure that supports different communication patterns with varying quality of service levels.

A few properties of the AMQP 1.0 messaging protocol make it an ideal candidate for this purpose.

First, this is one of very few messaging protocols that is truly symmetrical. It does not require a broker as an intermediary in the message exchange process. It supports both “store and forward” (brokered) and direct messaging communication patterns. The symmetrical nature of AMQP 1.0 allows Hono to use different communication semantics and different messaging components to handle various flows of messages.

The entry gateway into Hono’s AMQP messaging system is the Apache QPid Dispatch Router. It provides direct communication between producing and consuming endpoints. Unlike a broker, the router never takes ownership of messages; it instead simply passes AMQP packets between different endpoints. This means that, from a messaging perspective, the router is basically stateless. As such, it is much more vertically scalable than a message broker. In addition, it is far easier to scale it horizontally in a cloud environment – as we will see in a moment.

Another important feature of AMQP 1.0 is its flow control mechanism, which is implemented directly in the protocol. This means that consumers will, at every point, declare their capacity, or how many messages should be delivered to them. This information is propagated by the router, and producers of messages can be throttled to the speed of consumption.

Infographic showing the telemtry flow in regard to Eclipse Hono.

It is now easy to see why Hono uses only routers to deliver telemetry messages to the solution tier. These messages typically exhibit a large volume, which requires scaling – but at the same time, we can afford to lose them if something goes wrong.

In the example deployment, Hono comes with a single router for simplicity. In real-world deployments, routers are usually deployed in networks. Often, full-mesh networks are used. With such a topology, we achieve two things. First, we improve the scalability of the system as clients are distributed over different routers. Second, we improve the reliability of the system, as it can easily survive crashes of one or more routers while remaining functional.

We have covered messaging needs for telemetry traffic. But what about events and command messages that require a genuine guarantee of delivery? Dispatch routers can be configured to “auto link” certain addresses to external AMQP 1.0 containers. In this way, we can route messages through the broker queue and provide “at least once” delivery guarantees for events and commands.

In the default deployment, Hono uses Apache ActiveMQ Artemis Broker. It provides a scalable AMQP 1.0 message broker; its queuing capabilities are used to store event messages.

Infographic showing how Eclipse Hono employs the Apache ActiveMQ Artemis Broker.

Of course, you can use any AMQP 1.0-compatible message broker for this. Or provide more scalable and more reliable solutions by using broker clusters and high-availability configurations. In addition, the default deployment of a single router and broker can be replaced with a scalable AMQP 1.0 cloud messaging platform – such as the one described later in this article.

Solution tier

All solutions connect to the AMQP 1.0 network and receive messages from the messaging endpoints.

/telemetry/<tenant>/<device>

or

/event/<tenant>/<device>

They do not themselves need to integrate specific IoT protocols and benefit from the flow control of the AMQP 1.0 network.

The User Guide portrays examples of how to write solutions.

Monitoring infrastructure

Hono Messaging and the protocol adapters report different kinds of metrics to an InfluxDB, which can be viewed using the Grafana dashboards provided. Please refer to Getting Started for details.

APIs and their endpoints

Hono’s internal APIs are defined as AMQP 1.0 endpoints:

Infographic showing the components of the Eclipse Hono API.

Protocol adapters use Hono’s internal APIs to:

  • authenticate the device using the Credentials API
  • check the status of the device using the Registration API
  • forward device messages to the AMQP network using the Telemetry or Event API

As with the AMQP 1.0 network, flow control also enables the load protection regarding devices.

The Device Registry also implements RESTful variants of the Registration APIs and the Credentials APIs in order to support the administration of devices. This leverages the use of CLI tools like curl and simplifies implementation of a complex Device Operations Console (which is not part of Hono itself).

Deployment options

All microservices are built as Docker images and are available on Docker Hub. The following container management systems are supported based on the Docker images:

  • Kubernetes
  • OpenShift
  • Docker Swarm

The supplied deployment scripts start a preconfigured orchestration of the microservices for a fully functional installation. They are contained in Hono’s example module.

Deployments can be executed either on premise or hosted in a public cloud. On premise deployments work well for a small number of devices or test setups; a public cloud deployment is better for large installations that can scale dynamically per number of connected devices.

You can set up Hono yourself or use the sandbox deployment, which is accessible at hono.eclipse.org. The following steps assume you have Hono running locally:

  1. Register a device by using the REST API of the device registry

curl -X POST -i -H ‘Content-Type: application/json’ -d ‘{"device-id": "4711"}’ http://localhost:28080/registration/DEFAULT_TENANT

  1. Start a consumer; refer to Getting Started for details.
  2. Upload data using the HTTP adapter.

curl -X POST -i -u sensor1@DEFAULT_TENANT:hono-secret -H ‘Content-Type: application/json’ –data-binary ‘{"temp": 5}’ http://localhost:8080/telemetry

  1. Observe the consumer receiving the data.

Scale it up!

Hono is fundamentally designed for horizontal scalability. This is reflected in its microservices architecture:

  • protocol adapters can be scaled independently from the messaging infrastructure itself.
  • solutions are decoupled from Hono by the AMQP 1.0 network layer, which can be scaled independently from Hono itself.
  • the device registry is separated from the messaging components and, consequently, can be scaled independently as well.

You can use the scaling strategy of your container management system to scale the microservices you need. The internally used AMQP 1.0 protocol is designed for scalability and very large networks.

EnMasse – the elastic messaging infrastructure

Because Eclipse Hono is not itself a messaging system, it needs an underlying messaging infrastructure to work. Messages are exchanged between all internal components and sent from external devices to and from business applications. We refer to this type of infrastructure as the “AMQP 1.0 Network” in the overall Eclipse Hono architecture. As noted elsewhere, the off-the-shelf solution consists of the Qpid Dispatch Router and the ActiveMQ Artemis Broker.

The Eclipse Hono architecture was designed and developed in such a way that the underlying messaging infrastructure can be modified – provided that it is based on AMQP 1.0. This means you can use RabbitMQ with the AMQP 1.0 plugin, for example, or any other messaging broker implementing the AMQP 1.0 protocol.

Big numbers? We need elasticity!

When it comes to supporting a large number of connected devices and high throughput in terms of exchanged messages, the “single router, single broker” solution has its limits.

The messaging infrastructure needs to grow, and the scalability becomes the most important criterion for allowing Eclipse Hono to handle devices and consumers as well as the tremendous traffic associated with them.

A major problem in this case is deployment. How can we deploy the messaging infrastructure simply, without having to address the related complexity? Besides scalability, you also need high availability and resiliency.

In short, your messaging infrastructure needs to be elastic so it can support spikes in the numbers of connected devices and traffic generated, so that computational resources are then used productively. It would be a waste to run too many AMQP 1.0 brokers for a low number of devices and for low traffic. At the same time, such an infrastructure needs to grow as numbers increase. As that can change frequently, you need elasticity.

The answer? The EnMasse project!

How to address all the challenges mentioned above? The EnMasse project comes to the rescue!

EnMasse is an open source “messaging as a service” platform which simplifies the deployment of a messaging infrastructure both “on premise” and in the cloud. It provides the scalability and elasticity needed to support the messaging requirements for IoT use cases.

Moreover, EnMasse supports all common messaging patterns (request/reply, publish/subscribe, and competing consumers) and two main protocols: AMQP 1.0 and MQTT. Adding HTTP support is on the roadmap.

EnMasse provides multi-tenancy, meaning that different tenants can share the same infrastructure – but are isolated from each other. Finally, it provides security with respect to securing connections using TLS, but also authenticating clients using Keycloak as the identity management system.

Infographic showing how EnMasse works.

Another aspect that makes EnMasse appealing to use with Eclipse Hono: it is completely containerized and runs on key container orchestration platforms such as Kubernetes, Docker Swarm, and OpenShift. It is an excellent complement to Hono’s microservice architecture and deployment models portrayed in preceding paragraphs.

Infographic showing how EnMasse fits into Eclipse Hono.

As you can see, EnMasse offers all the features needed to be the messaging infrastructure that supports Eclipse Hono. It is easy to deploy both together in OpenShift; you can find more information on that in the step-by-step guide.

Conclusion and outlook

This article covers the basics of the Eclipse Hono project as well as describes its APIs, architecture, and deployment options. There are many things still to come in this field. The Telemetry and Event APIs we described are only the first that have been implemented. Next on our to-do list: the Command and Control and Tenant APIs.

The Command and Control API will provide a way for business applications to reliably send commands to the devices. The Tenant API will provide platform operators with the resources necessary for managing tenants (accounts).

We will also improve the EnMasse integration, particularly as regards shared security models and scalability testing.

We hope that we have increased your interest in IoT connectivity in general and the Eclipse Hono project in particular. We have a lot planned, and hope to see you in the community!

The post Industry 4.0 in 2017 – a quick look at the powerful 7 appeared first on Bosch ConnectedWorld Blog.

Bosch ConnectedWorld Blog

4 Public Companies Investing Large Sums in IoT

investing in iot

The advent of fast and reliable Internet paved the way for a more interconnected world. People now have the ability to interact with one another, no matter where they are in the globe. But after “perfecting” interpersonal communications through the web, we’ve begun to experiment on interactions with common household and commercial products. By embedding electronic, software, and sensor technology in these “things,” we have made them “smarter” and are now able to do more than they’re original purpose. This is what we now call the Internet of Things (IoT,) and this is a growing trend that we can’t, and shouldn’t, ignore. Rather, we should make sure we are alongside the large public companies with investing in our own future.

IoT aims to improve our lives by leveling up some of the most common items that we use. By adding these devices into a network of interconnectivity, we are able to use them in ways we’ve only imagined a couple of years ago. Smart TVs, automated cleaning devices, and health-tracking apparel are just some of the things that IoT has given us. There’s a lot more coming up, and it’s only a matter of time before everything in our homes are “smart” and connected.

What this means is that there’s a boom in spending for anything related to IoT, whether it’s for smart home appliances or more automated tools for work. It is time to consider investing in IoT and taking advantage of its growth and success for financial gain. There is a huge potential for growth, so putting your money in an IoT-related company is a sound decision. There are publicly-traded companies related to IoT, choose one that you think has the best potential and buy stocks. Here’s a list of such companies:

The Companies Investing in IoT

  1. Alphabet Inc. (GOOGL) – Google’s parent company is definitely the leader in all of IoT. Aside from being the search engine authority, Google has its own home monitoring products in the form of Nest. Acquired in 2014, Nest is the leading name in home automation products like smart smoke detectors, thermostat, and surveillance cameras. Google also recently launched Google Home, a home automation device that “assists” you with all your queries and tasks for devices connected in your home network.
  2. Amazon (AMZN) – Right behind Google is Amazon, an undeniable IoT market leader offering quite a few home improvement and automation products. Amazon’s Echo speaker is actually what prompted Google to produce Home, because the latter has cornered the market for “always listening” devices that’s ready to answer your queries with the help of its Alexa home connectivity technology.
  3. Apple (AAPL) – The Cupertino-based company is best known for its sleek and powerful devices, and has a large following solely for its line of smartphones. In terms of IoT, the first thing that comes to mind is its smartwatch, the Apple Watch. They’ve taken wearable tech to new heights, cramming a multitude of features in a sleek wristwatch profile. Aside from this, Apple also owns patents related to IoT. One of this is a patent called “local device awareness,” which is about the automated relationship between close-range devices. This is definitely IoT, opening up possibilities for device position tracking, virtual reality, and more.
  4. Skyworks Solutions (SWKS) – Unlike the previous entries which have actual IoT products and devices, Skyworks is more into semiconductor manufacturing, and is primarily partnered with Apple. And even though they’re more known to produce a vital part of the iPhone, the semiconductors they produce are the backbone of IoT in general. It is a vital component of smart devices and interconnected home appliances.
  5. Cloud and telecoms companies – Telecommunications companies like AT&T and Verizon are now improving their communications infrastructure, to ensure reliable connectivity for their subscribers. These subscribers require the latest, fastest network, be it 5G or fiber, for their interconnected devices to work seamlessly. Cloud-based companies like RingCentral (RNG) plays a vital role in providing robust communications options to your network, which becomes more vital the more you get interconnected with people and “things” (Editor Note: author works for RingCentral).

The possibilities are endless when it comes to the Internet of Things. It is definitely one of the most exciting tech trends today, and it created one of the most lucrative markets in the business world. Don’t just settle with trying out and becoming part of IoT. Instead, use the opportunity to gain financially by investing in IoT. There’s a lot of untapped potential with IoT, and we can only expect it to grow further in the coming years.

About the Author

Francis has been writing for more than a decade now, focusing on Digital Marketing in the last couple of years. He is currently in charge of writing web-optimized content for RingCentral, an industry-leading cloud phone systems provider. Francis is also a voracious reader, spending most of his free time immersed on fictional worlds. You can reach him through Twitter and LinkedIn.

The post 4 Public Companies Investing Large Sums in IoT appeared first on ReadWrite.

ReadWrite

Vodafone research reveals number of large scale IoT projects doubled in the last year

Vodafone research reveals number of large scale IoT projects doubled in the last year

Vodafone published today its fifth annual IoT Barometer Report1 – the leading global survey of business sentiment regarding investment and innovation in the Internet of Things.

The Barometer report found that:

  • The percentage of companies with more than 50,000 connected devices active has doubled in the last 12 months
  • 84% of IoT adopters say that their use of IoT has grown in the last year
  • 51% of IoT adopters say that the technology is increasing revenues or opening up new revenue streams
  • 66% of all companies agree that digital transformation is impossible without IoT

Businesses in the Americas have led the way in embracing large scale IoT projects, where 19% of companies using IoT have more than 10,000 connected devices, compared to 13% in Europe and 7% in Asia Pacific. The large scale users also report some of the biggest business gains with 67% of them highlighting significant returns from the use of IoT.

Energy and utility companies are at the forefront of the largest IoT projects worldwide, with applications such as smart meters and pipeline monitoring.

The range of benefits that users are getting from IoT is also widening as adoption increases – greater business insights, reduced costs and improved employee productivity top the list globally. In Asia Pacific 53 % of respondents cited increased market competiveness as the top benefit compared to 35% in the Americas and 33% in Europe. In the automotive sector, 51% of companies say that IoT is helping to improve brand differentiation.

Security in IoT is still the biggest barrier for organisations regarding deployment. However, in companies with 10,000 or more connected devices in operation only 7% say security is their top worry. Organisations are taking more steps to tackle security concerns including an increase in security training for existing staff, working with specialist security providers and recruiting more IT security specialists.

As the scale of IoT projects increases the report also notes a rise in connectivity requirements. Companies are looking to use a mix of technologies from fixed line to low power wide area networks (LP-WAN) depending on the application. Typically, large scale projects use four different connectivity options with mobile and Wi-Fi the two most popular. There is increasing interest in the newer technologies such as Narrowband IoT, with 28% of all companies now considering it and other LP-WAN options, for new IoT projects.

Vodafone director of IoT Erik Brenneis said:

“Over the five years of this report we have seen the number of companies that have adopted IoT double, and projects have grown from small pilots to global rollouts of tens of thousands of connected devices. IoT is clearly here to stay and the future looks exciting as 79% of adopters are saying that IoT will have an enormous impact on the whole economy in the next five years.”

“I believe we can now say that IoT has come of age and is proving itself across all industries and geographies.”

1 The IoT Barometer Report surveyed 1,278 respondents in 13 countries, US, Brazil, Ireland, UK, Germany, Italy, Spain, South Africa, China, India, Japan, Australia and New Zealand.

The post Vodafone research reveals number of large scale IoT projects doubled in the last year appeared first on IoT Business News.

IoT Business News

Vodafone IoT Barometer: Large global IoT projects doubled last year

Vodafone IoT Barometer: Large global IoT projects doubled last year

2017 could be a milestone year in terms of IoT adoption, as a surge in connected devices opens up new revenue streams for companies, according to the Vodafone IoT Barometer Report.

The mobile phone operator’s fifth annual IoT Barometer Report finds that, over the last 12 months, the number of companies worldwide with more than 50,000 connected devices active has doubled. Meanwhile, more than eight of ten (84 percent) of the IoT adopters polled say that their use of IoT has grown in the last year.

Just over half (51 percent) of IoT adopters say that the technology is increasing revenues or opening up new revenue streams, while two thirds (66 percent) agree that digital transformation is “impossible” without IoT.

The IoT Barometer research surveyed 1,278 respondents in 13 countries: the US, Brazil, Ireland, the UK, Germany, Italy, Spain, South Africa, China, India, Japan, Australia and New Zealand.

Read more: Vodafone report suggests IoT budgets are on the rise

Regional differences

The report says that companies in the Americas have led the way in embracing large-scale IoT projects, where 19 percent of companies using IoT have more than 10,000 connected devices. That’s compared to 13 percent in Europe and seven per cent in Asia-Pacific. Large-scale users of IoT also report some of the biggest business gains, with 67 percent of them boasting of “significant returns” from the use of IoT. Energy and utility companies are behind some of the largest IoT projects worldwide, with applications such as smart meters and pipeline monitoring.

The report claims that the range of benefits users get from IoT is also widening as adoption increases – greater business insights, reduced costs and improved employee productivity top the list globally. In Asia-Pacific, 53 percent of respondents cited increased market competitiveness as the top benefit compared to 35 percent in the Americas and 33 percent in Europe. In the automotive sector, 51 percent of companies said that IoT is helping to improve brand differentiation.

Read more: Vodafone launches NB-IoT network in Ireland

Security remains a big barrier

But the report warned that IoT security is still the biggest barrier for organisations when it comes to deployment. However, in companies with 10,000 or more connected devices in operation, only seven percent say security is their top concern.

As the scale of IoT projects increases, so are connectivity requirements. Companies are looking to use a mix of technologies from fixed-line to low-power wide area networks (LPWAN) depending on the application.

According to the report’s authors, large-scale projects tend to use four different connectivity options, with mobile and Wi-Fi the two most popular. There is increasing interest in newer technologies such as NB-IoT, with 28 percent of all companies now considering this and other LPWAN options for new IoT projects.

Vodafone director of IoT Erik Brenneis said, “Over the five years of this report, we have seen the number of companies that have adopted IoT double, and projects have grown from small pilots to global rollouts of tens of thousands of connected devices.”

“IoT is clearly here to stay and the future looks exciting as 79 percent of adopters are saying that IoT will have an enormous impact on the whole economy in the next five years.  I believe we can now say that IoT has come of age and is proving itself across all industries and geographies,” he added.

Read more: How will Vodafone’s NB-IoT network affect IoT device manufacturers?

The post Vodafone IoT Barometer: Large global IoT projects doubled last year appeared first on Internet of Business.

Internet of Business

Five Rules for Managing Large, Complex Projects

“Megaprojects” — defined as projects with budgets exceeding $ 1 billion — are important contributors to numerous sectors, including health care, defense, mining, telecommunications, transport, energy and water infrastructure, sporting events, science, and manufacturing. They represent a significant proportion of many nations’ economic activity and profoundly affect productivity, social cohesion, and the environment.1 Yet megaprojects have proved notoriously difficult to deliver on time and on budget; one estimate suggests 90% of them end up over budget.2 Two prominent examples are the Sydney Opera House in Australia, which was 10 years late and a staggering 1,400% over budget when it opened in 1973,3 and the “Big Dig” Central Artery/Tunnel Project in Boston, Massachusetts (original estimate $ 2.6 billion, actual cost $ 14.8 billion).

Why are megaprojects so difficult to manage? The reasons include technical challenges, changes in design and operational requirements, increases in costs, disputes over responsibility, and new regulations. Complexity usually increases with project scale, and complexity can give rise to uncertainty and an inability to foresee the difficulties, changing conditions, and unanticipated opportunities that will be encountered once the project is underway. In this article, we argue that one way to manage the uncertainties is to innovate throughout the course of the project. What’s more, we believe our suggestions are applicable to all large-scale, long-term projects — not just projects with billon-dollar budgets.

Specifically, we’ll distill five rules for innovation in large, high-risk projects, providing managers with guidance on how to modify their plans and processes when opportunities arise or conditions change. Our findings are based on more than 10 years of research into megaprojects. (See “About the Research.”) The projects we studied included:

  • High-Speed 1 (1998-2007), a high-speed, 109-kilometer railway from London to the Channel Tunnel, which cost £5.8 billion (roughly $ 7.5 billion at today’s exchange rates)4.
  • Heathrow Terminal 5 (2002-2008), a new airport terminal, hotel, car park, subway line, and air traffic control tower, which cost £4.3 billion.
  • Infrastructure for the London 2012 Olympics (2006-2012), which cost £6.8 billion.
  • Crossrail (started in 2007, scheduled to open in 2018), an 18-kilometer railway across London that has a budget of £14.8 billion and includes 42 kilometers of new railway tunnels and 10 new and 30 upgraded stations.
  • Heathrow Terminal 2 (2009-2014), an airport terminal now serving 29 airlines that replaced an existing terminal and cost £2.5 billion.
  • The Thames Tideway Tunnel project, begun in 2016, a 25-kilometer tunnel and an upgrade to London’s sewer system; the project is expected to take up to seven years to complete, at a cost of £4.2 billion.

Five Rules for Innovation

We follow the view that Donald Sull of the MIT Sloan School of Management and Kathleen M. Eisenhardt of Stanford’s School of Engineering express in their book Simple Rules: How to Thrive in a Complex World: Management strategy in complex, uncertain circumstances is often best articulated as a series of simple rules.5 Based on our experiences working on and studying megaprojects, here are our five rules for such projects. (See “Five Innovation Rules for Large, High-Risk Projects.”)

1. Assess what’s worked before. When Heathrow’s Terminal 5 was being planned, the project team systematically studied every international airport opened in the previous 15 years and every U.K. construction project that cost more than £1 billion built during the previous 10 years.6 One of the chief problems discovered was the use of fixed-price contracts to transfer risks to suppliers, thereby creating adversarial relationships with contractors and — worse — freezing designs at an early stage of the project, limiting innovation. The planners of Terminal 5 calculated that if they used a fixed-price contract approach, the project would end up over budget and one year late. Endeavoring to learn from these accumulated lessons, the planners created a new delivery model based not on rigid fixed-price contracts but on a collaborative, innovative, and flexible process. Unfortunately, the public remembers the Terminal 5 project for its disastrous opening days, when more than 20,000 pieces of luggage were misplaced and several hundred flights were canceled.7 What’s often overlooked, however, is that the Terminal 5 project was delivered on time and on budget, and in several subsequent years was voted the world’s best airport terminal by passengers.8 Furthermore, the problems with the Terminal 5 opening provided important lessons guiding key improvements in the megaprojects that would follow — improvements we’ll outline in greater detail below.

2. Organize for the unforeseen. While fixed-price contracts may be adequate for dealing with predictable and stable conditions, more flexible contracts are required to deal with unexpected and rapidly changing circumstances. By using cost-plus or cost-reimbursable contracts, for example, the client and contractor enter into a relational agreement where there are incentives to build trust, form a collaborative culture, and share risks and opportunities.

True, flexible contracts can appear undesirable at first because of their higher up-front costs. But such contracts support coordination by mutual adjustment when project activities and schedules are modified in real time to address unforeseen circumstances. The client creates incentives encouraging contractors working on the project to reveal problems, recover costs, achieve agreed-upon profit margins, exploit innovative possibilities, and build solutions. The “T5 Agreement” between the client and major contractors in the Terminal 5 megaproject formed the basis of a new flexible process for dealing with uncertainty — and was subsequently used as a model for the London 2012 Olympics and Crossrail megaprojects.

Another key to managing megaprojects is staffing project teams with innovative thinkers — and encouraging teams to remain flexible. After all, a megaproject comprises numerous smaller projects, each executed by a project team. When organized and incentivized effectively, people with different knowledge and skills can adapt and respond flexibly to rapidly changing conditions, unforeseen problems, and emergent opportunities. These teams treat existing knowledge and skills as bases from which to modify old routines and build new ones.

When it came to planning the London 2012 Olympics, the Olympic Delivery Authority exemplified the approach of using flexible project teams. One senior Olympic Delivery Authority manager told us that team dynamics depended on “having enough excellent people with a real attitude of rapid assessment and decision-making to be able to see issues, discuss them, make decisions, and move on.” In addition to providing flexibility, well-organized team structures can also forge collaboration and overcome tensions that arise when companies with differing interests are expected to work together.

3. Rehearse first. The risks of cost and time overruns associated with the adoption of new technology and practices were minimized on the Terminal 5, Olympics, Crossrail, and Terminal 2 projects by reliance on established technologies and practices. Where new technologies and practices were introduced, they were first tested and proven in off-site trials, dry runs, and other operational environments such as smaller airport terminals.

For example, the “roof project” for the main terminal building was considered one of the most uncertain parts of the Terminal 5 project. There were concerns about erecting roof abutment structures with spans of more than 150 meters. The solution was erecting these structures in advance, at an off-site location. Through the off-site pilot, project leaders identified 140 lessons, each with a preemptive risk mitigation plan, enabling contractors to work more rapidly on-site.9 As a result, the roof project was delivered three months earlier than planned.

In contrast, the first few days of operation of Heathrow’s Terminal 5 infamously suffered from a lack of rehearsal. Having learned lessons from the problems associated with the opening of Terminal 5, the leaders of the Terminal 2 project established a new rehearsal-style process for Terminal 2: a “soft” opening. Importantly, the soft opening occurred two years prior to the official opening of Terminal 2 in June 2014. Terminal 2’s soft opening was led by a dedicated “operational readiness” team. That team managed a gradual handover to operations, including 180 trials with 14,000 volunteers and 1,700 training sessions. The soft opening also included the creation of a “model terminal” mock-up to test check-in software. In addition, there was a test with a live flight and a staged gateway process to move each airline into a live terminal building.

Terminal 2 wasn’t the only project that learned lessons from the problem-plagued opening of Terminal 5. In its planning for the London 2012 Olympics, the Olympic Delivery Authority made a point of completing its construction program by July 2011 — providing one whole year of testing on live events in the run-up to the Olympic Games.

4. Calibrate and apportion risks appropriately. A megaproject contains a large proportion of predictable, standardized, and repetitive tasks that have been performed many times on previous projects — as well as novel and innovative procedures being applied for the first time. This combination requires a balancing act, and the concept of “targeted flexibility” provides a solution to it.10 The idea is to break down a megaproject into distinct projects, structures, and processes, each of which addresses a different piece of the uncertainty.

A targeted flexibility approach creates different contracts and collaborative arrangements to address the varying challenges of individual projects within the program. A cost-plus contract, as we mentioned earlier, can be used when uncertainty is high; a fixed-price contract may be more appropriate when there’s less uncertainty. The London 2012 Olympics megaproject used a targeted flexibility approach to great success, relying on fixed-price contracts to deal with known conditions and risk-sharing, and target-cost contracts (including contracts based on a suite of what have been called “New Engineering Contracts”11) to deal with less predictable projects, such as the construction of the London Aquatics Centre, the Velodrome, and the Olympic Stadium (now London Stadium).

Thanks in part to this approach, ISG plc, the contractor that built the Velodrome, came forward with the suggestion to switch from a steel roof to a cable-net roof, resulting in significant reductions in time and cost.12

5. Harness innovation from start to finish. Formulating a coherent statement about innovation can help project leaders plan, coordinate, and communicate with research partners and other collaborators from start to finish.13 We saw proof of this during the Crossrail megaproject, which introduced the idea of establishing an innovation strategy for the construction phase of the project.

Established in 2012, the Crossrail Innovation Strategy created a formal process for encouraging members of the project supply chain to submit ideas for innovation. In collaboration with Imperial College London, Crossrail’s leaders created a small team for the express purpose of identifying, evaluating, and developing new ideas — ideas developed internally or originating with members of the project supply chain. Crossrail also developed an online digital platform called “Innovate18” to provide both insiders and outsiders with a mechanism to submit ideas, including an “Innovation Management System” to manage, track, and report on the progress of ideas. Innovations likely to benefit Crossrail thus had the chance to gain relevant sponsorship and commitment from interested parties well in advance of the actual construction. By summer 2015, the program had attracted more than 800 ideas ranging from the use of high-definition drone-mounted cameras for site inspections to the repurposing of grout shafts to cool the train tunnels by transferring heat to adjacent buildings.

The success of the program made Crossrail’s leaders recognize that there would be additional advantages in starting even earlier. So when Andy Mitchell, Crossrail’s program director, became CEO of the Thames Tideway Tunnel megaproject in 2014, he decided to build upon the Innovate18 digital platform. He also recognized that it was important to involve the leaders who would manage the tunnel after construction was complete. The cost of operating a rail system, airport, or tunnel over a lifespan of several decades is much higher than the cost of designing and building it; those who will eventually maintain the asset can often identify and implement innovations during the front-end planning and design phase that will improve performance and reduce costs later on. Participants in Crossrail and Thames Tideway have developed this concept, creating an industry-wide program called i3P, which stands for Infrastructure Industry Innovation Platform. The i3P program has been rolled out to support innovation in a number of new megaprojects.14

A More Flexible Approach

Despite the diversity of large, high-risk projects, there are some simple rules that can help improve their performance. The five rules we have described encourage innovation to deal with uncertainty. They confer the flexibility to change while maintaining the stability required to deliver projects efficiently. And they help coordinate innovative action across multiple parties. These simple rules challenge traditional project management, which has pushed too far toward control and prescription and been characterized by complicated, highly rigid contracts that stifle flexibility and innovation. These five rules might seem like common sense, but the marked failures of past megaprojects show the value of making such sense much more common.


MIT Sloan Management Review