Lessons from the
Vancouver 2010 Olympic Winter Games

White paper
July 2010

By Justin Webb
VP of Olympic Services, Bell Business Markets

1.0 The customer experience

Hosting an Olympic Games is a singular opportunity for the host nation to present itself to the world. It's also a massive undertaking: the local organizing committee must devote years of planning and coordinate tens of thousands of individuals. As high as the expectations are for the competing athletes, the bar is set even higher for the performance of the Games themselves. There is no room for error.

When Bell was selected to deliver every aspect of the communications infrastructure for the 2010 Olympic and Paralympic Winter Games, it was a once-in-a-lifetime opportunity to employ all of our skills, knowledge and technology toward a single goal. Above all else, the Olympic Games demand flawless execution–no downtime, no delays, no glitches.

And when the eyes of the world were on Canada, Bell did it.

This white paper will take you behind the scenes of what it took to deliver a complex, large-scale communications solution for the Vancouver 2010 Olympic and Paralympic Winter Games Organizing Committee (VANOC)–a high-profile client with some of the most demanding service-level requirements anywhere. It shares Bell's experiences, and the best practices we developed and refined while working with VANOC.

In addition to an overview of the advanced Internet-Protocol (IP) based technologies and innovations that were necessary to deliver an unparalleled customer experience for VANOC and the 3.5 billion viewers around the world, this white paper will also detail the wide array of best practices that Bell developed and employed in the following areas:

• Governance of a fully managed outsourced service organization
• Project planning and management
• Solutions engineering

It will also reveal important insights into building cultures of peak performance and transparency.

1.1 One provider, one network

A critical part of the planning for any Olympic Games focuses on the communications infrastructure that allows the entire operation to run smoothly–everything from the phones that staff use to talk with each other at and between venues to the real-time transmission of official times and results–as well as ensuring that the Olympic Games are presented to the rest of the world, whether that is via television broadcasts, journalists filing newspaper stories or on the official Web portal.

Early on, VANOC made the unprecedented decision to entrust its complete communication requirements–voice, data, Internet, Web services and more–to a single managed outsourced service provider: Bell.

In essence, Bell became VANOC's communications department, charged with planning, deploying, and implementing the day-to-day operations and security of its networks and services.

During those few weeks in February and March, every image on TV, every story published around the world, every real-time score transmitted, and every visit to the official vancouver2010.com Web portal traversed a communications solution designed and delivered by Bell.

The dedicated network for the Olympic Winter Games was completed by a skilled team of Bell network architects and engineers 14 months before the Games, faster than any other communications partner in Olympic Games history. And by the time the 2010 Olympic and Paralympic Winter Games came to a close, the network had provided both higher connectivity and higher levels of reliability than ever before at an Olympic Games.

How did Bell do it?

1.2 Familiar challenges, massive scale

In many ways, the business and operational challenges that VANOC faced relative to its network are similar to what large businesses and public sector organizations in Canada must manage on a daily basis–multi-site operations, mobile workforce, and mission-critical communications infrastructure.

What makes a client like VANOC unique is the size, scope and prominence of the assignment.

The Bell Vancouver 2010 Olympic Winter Games experience offers a window into the benefits of utilizing a holistic approach for planning and operating communication infrastructure, including the services, staffing, and associated cost models. In order to succeed in any capacity at the Olympic Games, one must execute flawlessly.

This is how Bell contributed to ensuring VANOC's Olympic Winter Games performance was extraordinary.

2.0 High-performance technology

Bell was responsible for the delivery of an end-to-end communications network and managed services to VANOC. As its outsourced communications department, Bell provided everything from voice, data, Internet, cabling, Web services, network security and broadcast support, as well as communications support for HR, budgeting, service delivery, procurement, supply chain and more.

VANOC required more than simply a long list of technological solutions, however. It required professional services from Bell that could put it all together, and find innovative solutions to its unique problems.

Most fundamentally, Bell committed to delivering an always-on, always-secure all-IP network architecture, which converged three networks into one, reducing its build requirements and resulting in a more cost effective and sustainable solution with twothirds the copper footprint, lower operating costs, and faster response times.

In this section we provide an overview of the technology solutions that Bell delivered.

2.1 Network

The Carrier Ethernet Olympic Games core network provided 10Gbps connections between 130 venues at 15 geographically dispersed venues in Vancouver and Whistler, spanning 120 km across mountains. A 285-km fibre-optic network included a broadband access corridor that crossed eight municipalities between Vancouver and Whistler. In all, there was 700 km of fibre-optic cable with up to 144 strands in redundant rings linking the main Olympic Games sites in Vancouver and Whistler.

The core network provided Layer 2 infrastructure for all IP/Ethernet services, including:

• VoIP network–more than 14,000 connections across Games venues, including 6,000 VoIP phones, all interworked with the public network via the VANOC datacenter
• 32,000-port Ethernet across all venues
• 5000 km of CAT5 cabling at venues, with a massive in-building overlay of 24,000 drops
• Internet (Netzone and private IP services)
• VANOC administration network
• Games network–large, specialized data network supporting the competitioncritical applications of scoring and timing systems, as well as core Games systems applications such as accreditation, sports entries and qualifications, medical systems and workforce management
• Rate Card Transparent LAN Venue Connect, which supported the purchase of Ethernet bandwidth by the slice for sports, media, broadcasters and other Olympic family officials

2.2 Wireless

Integrated with wireline access in Vancouver and Whistler, the High Speed Packet Access (HSPA) mobile broadband-enabled wireless network provided seamless coverage in and around all Olympic Games venues. Some of this used to be very inaccessible terrain. In Howe Sound, Bell deployed a solar-powered cell site–the first of its kind in Western Canada–on the steep cliffs of Anvil Island. Forty-one other cell sites were built by Bell throughout the B.C. Lower Mainland.

In order to ensure that the 9,000 mobile phones provided to Olympic Games organizers would have wireless connectivity regardless of how much traffic was on the Bell Mobility network at any given moment, a dedicated slice of spectrum was parsed just for them.

A further enhancement of the Olympic Games experience for visitors to Vancouver and Whistler came courtesy of the Bell Mobility HSPA+ network, which gave them access to video highlights of events at other venues.

Other parts of the wireless solution included:

• Two-way Radio–32-site private wireless network and 7,000 radios for athletes and Games officials to access real-time information and constant communications where there is no wired infrastructure
• WiFi–hundreds of access points supporting remote Internet access services at 54Mbps on each of three radio channels

2.3 Security

Bell needed to provide the highest level of security possible for both the 2010 Winter Games network, and the vancouver2010.com web portal.

Managed and Professional Security Solutions from Bell included:

• Managed Firewall Services
• Managed Intrusion Detection and Protection
• Managed Network Security for Content
• Vulnerability Assessment and Penetration Testing
• Business Continuity Consulting Service

The entire VANOC network infrastructure provided secure logical partitioning for up to 20 classes of traffic across multiple networks and application types. It also featured a secure network access (SNA) platform, designed to inspect, assess, ensure compliance to security policy and remediate connecting devices prior to granting network access. The SNA was also used for dynamic VLAN provisioning with secure access via network admission control, so Olympic Games officials were able to plug into any LAN port in any venue and get assigned the appropriate network resources based on credentials on their device.

In addition, WiFi wireless security switches located in the datacenter allowed for the centralized management of all venue wireless LANs, including user authentication, dynamic radio channel and power management, and automated load balancing of wireless users across WiFi access points.

2.4 International Broadcast Centre

Every moment of broadcasting traversed the Bell Olympic Games network prior to transmission to the rest of the world.

The nerve centre for more than 24,000 hours of Olympic Games coverage was the International Broadcast Centre (IBC). Located in the Vancouver Convention Centre, a 10-story building with more than a million square feet (about four city blocks), the IBC housed 7,000 people working with 50 broadcasters from around the globe.

Bell fibre connected every single sporting venue to the IBC, and terminated in a single room with patch panel racks designed and managed by Bell. Inside the IBC, Bell provided more than 2,000 circuits to broadcasters for video, data, phones, ISDN, analogue lines, and audio.

In order to quickly connect broadcasters with the bandwidth they needed, Bell built a 188-km fibre-optic duct system with break-out boxes–literally a series of tubes– throughout the IBC. By blowing the fibre strands into the interconnect system with compressed nitrogen, Bell was able to provision broadcasters with additional bandwidth in 10 minutes, rather than the days or weeks that they are accustomed to.

In all, there were 432 strands of fibre that fed the high-definition video streams to the IBC, including redundant architecture for two cable entrances to the building. Inside the IBC, there was 357 km of copper communications cabling, providing 23% more bandwidth than at the Beijing Olympic Games.

A team of 50 Bell experts worked on the IBC's telecommunication needs. Noteworthy aspects of the network include:

• 8,000 jacks for network access
• 350 WiFI access points for mobile devices and Internet connection bandwidths of up to 1 gigabit per second
• Dedicated transmission lines from Vancouver to New York, and from Vancouver to Seattle that extended bandwidth to L.A. to connect with international carriers and undersea cable
• Six HD-capable Bell Electronic News Gathering boxes located at IBC to allow camera crews to connect to a digital video network and send video around the world

2.5 Venue connectivity

Bell ensured that at each of the 15 key Olympic Games venue sites, wired and wireless Ethernet connectivity was available wherever it was required. More than 400 fibre-optic closets were installed at the sites, fed by dual fibre connections and built out with 100 m of copper within close range of Ethernet users and to WiFi access points.

The venues presented some unique challenges. First, there was broadcast connectivity: at Whistler, for instance, three competition venues each required 15-30 camera positions for each event, and each camera required a single strand of dedicated fibre to the main broadcast centre in Vancouver.

Providing IP connectivity up the side of Whistler required more technological ingenuity. For example, in order to give photographers access to a wired connection during the alpine ski events, Bell developed something it called a “gigasled”–a mini wiring closet that included an Ethernet switch and an uninterrupted power supply put on a sled that could be towed up the hill to photographer stations. Four gigasleds were deployed at the venue during the Games.

Bell had to adapt to a different challenge at the Cypress Mountain venue. When a lack of snow forced VANOC to change significant aspects of the venue's layout, Bell responded to minute-by-minute changes so that broadcast, timing and scoring cabling was in place. Also, with just four hours notice, Bell moved all of the technology supporting concession stand operations (data transfer, connectivity, etc) to the opposite end of the venue, including reconfiguring switches and deploying new conduit and access points so that this VANOC retail operation would be open for business on that day's competition.

2.6 Web

Bell collaborated with VANOC to design, develop and host the vancouver2010.com portal.

The project delivery included:

• Architecture design
• Creative and branding strategy
• CRM integration
• Deployment of social media technology

The portal was built with an architecture that leveraged the cloud where possible, with two back-up data centres located thousands of kilometers apart, and a Content Delivery Network (CDN) that cached frequently accessed content close to its audience. It all cost about 50% less than previous Olympic Games Web portals.

3.0 Building an effective partnership: aligning people at all levels

Whenever an organization chooses a managed outsourced model for its IT department, a lot rests on the working relationship that is forged between the client and the service provider. To some extent, the terms and conditions of the formal contract provide structure as to how the teams and its leaders function together, but the reality is that contracts only go so far.

A critical factor in whether a partnership is ultimately a success is how the business relationship develops between the individuals on both sides–client and service provider–who interact on a regularly basis. To be truly effective, a strong but flexible governance framework must be in place to ensure that everyone involved is aligned in their priorities, goals, standards and general work culture.

This is never more true than in a high-pressure, dynamic and complex project like the 2010 Olympic Winter Games. All parties had to be working in concert. Success would not only be determined by what technology was deployed, but by the way that people worked together to deliver an exceptional communications solution, and ultimately, an exceptional experience at the Games themselves. In other words, for Bell to build the technology VANOC required for a flawless Games, it had to infuse its own organization and its VANOC partnership with cultures of transparency, trust and peak performance.

There were three dimensions to the operational relationship between VANOC and Bell

• Staffing – Bell selected employees to be integrated into roles in VANOC
• IT organization partnership – Bell worked as one of four key partners in the overall IT organization to align activities
• Olympic Technology Board – Bell established this governing body in conjunction with VANOC

Bell identified three tenets to building a superior partnership with VANOC:

1. Embedding working-level employees early in the relationship
2. Setting common objectives with all partners
3. Operating with an open and transparent governance model

In this section we will explore each tenet.

3.1 Integrated staffing

In order for Bell to gain a clear understanding of the client's most basic but everchanging communication needs, Bell placed its own employees in the central IT organization of VANOC from the earliest days of the engagement. What began as two people grew to nearly 100 at VANOC headquarters, with some 300 Bell employees stationed at Olympic Games venues.

These were working-level people in operational roles, serving full-time on behalf of VANOC, and in some cases performing tasks that were as simple as deploying telephone handsets and lines for new hires. Entrenching individuals as necessary was more effective than deploying a large group without a clear understanding of client needs.

3.1.1 A culture of collaboration: one team

Over time, integrated employees start to feel less a part of a managed service provider, and more a member of the client organization. They build important working relationships daily with their new colleagues, gradually adopt the overall culture, and in the end, accumulate better insights into the operations of the client and its unique challenges and requirements.

By the time the Games began, it was difficult to distinguish between a VANOC employee and a Bell employee–and it was irrelevant to the operations of the Games.

3.1.2 Staffing flexibility

It was a flexible arrangement. When VANOC requested that one of the first embedded employees from Bell change roles to determine requirements of the IT organization more broadly than telecom, Bell agreed, and in so doing served its client better, and gathered new information that enabled them to deliver their mandate.

3.1.3 Selecting employees

Placing the correct Bell employees inside VANOC was key to delivering the service the project demanded. Recruits were drawn from an internal pool of applicants, and tested for the unique requirements of the Olympic Games organization. The testing had three parts:

1. Technical knowledge
   Test: Verbally provide a series of telecom acronyms, which progressively become more arcane, and applicants must fully explain the technical term
   Purpose: To test applicants' depth of telecommunications knowledge and ability to explain complex technology
   
2. Team management
   Test: With no information, applicants were asked to draw a diagram of how they would structure a team
   Purpose: to test how applicants reacted to the challenge, how they thought on their feet and their problem-solving thought processes
3. Communication skills
   Test: Applicants asked to write a short essay on an extracurricular activity
   Purpose: To test communication skills, and to identify candidates with deeper sense of commitment outside of work who understand value of team work and collaboration

3.2 Collaborative leadership

In some managed outsourcing engagements, a service provider needs to integrate its role with other partners to the larger organization. Typically this is coordinated at the leadership level, where decisions can be made quickly and effectively.

During the final 16-month push to the Games, the various components of VANOC's IT infrastructure started to come together. Telecommunications played a major role. In order for Bell to function as VANOC's communications department, and not merely its telecom vendor, it needed to work collaboratively at a leadership level with the other technical partners, most importantly the IT system integrator and the provider of timing and scoring systems.

This was accomplished in a weekly three-hour meeting between the five key leaders who were core to the VANOC IT organization–two from VANOC itself, and one each from Bell, the systems integrator, and the timing and scoring provider. They met with no set agenda, and mapped out whatever issues had to be addressed.

In order for these leadership meetings to be effective, they required a single, common view of the partnership. Three steps were taken:

1. Establish a common goal

No collaborative group can work effectively without a common goal. VANOC's mission for its technical solution was to deliver a “flawless games”, but each service provider had its own separate definition of what would meet that goal. For example, Bell defined “flawless execution” as 99.999% network up time over 17 days of the Olympic Games, with a 30-minute mean-time for repairs.

To avoid inevitable conflicts, the leaders agreed to a common objective: no Severity One or Severity Two trouble tickets during the Games. This was the definition of “flawless” that guided their decisions.

2. Rank priorities

In order for each partner to understand what its role is in the overall set of priorities for the Games, the leadership team categorized and ranked six broad priorities:

1. Safety
2. Fair competition
3. Broadcast
4. Reputation
5. Customer services
6. Full operations

The leaders then mapped what technology solutions supported those priorities–for example, two-way radio and timing and scoring systems were essential for Sport Operations, while the phones deployed to VANOC employees' desks, were ranked lower, given the other communication channels available (email, mobile phone, etc.).

Together, the leaders agreed to a common prioritized list of all the technologies that would be deployed for the Games, providing a common understanding of what issues had to be addressed before others.

3. Set top 20 dates for key deliverables

Each IT partner had to be aligned to the larger purpose of delivering “flawless” Games.

Based on the prioritized list of technologies, the leaders agreed upon the top 20 dates and deliverables that the IT organization as a whole needed to meet. Regardless of how many “telecom” items were in the top 20, each partner had some role in each priority, and now understood what had to be delivered, and by when. Trouble, prioritized

The list was then published for each IT team, and project timelines were scored on a monthly basis, and labeled red if they missed a date (see Project management section for more).

This process cemented the partnership at a leadership level, so as the components were integrated to bring a single solution to market–the IT infrastructure for the Olympic and Paralymic Winter Games–all the players knew the components, the priorities and the timing for the key deliverables.

The leaders were able to focus on those items that mattered most. The partnership model established an aligned set of work activities; when they struggled with issues, it was a touchstone that guided decisions.

3.3 Transparent governance

As a service provider engaged in a complex, long-term project, Bell needed an internal governance structure that coalesced its own resources effectively and paved a path to success.

Instead of striking a steering committee, which is the typical approach, Bell established a governing body called the Olympic Technology Board with the expectation that its members would participate in a meaningful way, and govern as they would on a board of directors.

The board included senior Bell leaders from IT, operations, network, human resources and others–people with the ability to affect change or find solutions to issues.

The reporting structure was rigid:

• A single monthly report with a standard template:
  • Updates on all 29 program streams
  • Matters of interest, including major decisions that were upcoming
  • finances
• The report was issued three days prior to the board meeting
• The report was not socialized, pre-empting any attempts to address shortcomings, and potentially cover up areas that needed assistance

3.4 All the dirty laundry

Most importantly, the CIO of VANOC was invited to sit on the governing board, with full access to the monthly meetings. This was done for two reasons:

1. To expose the client to the unembroidered facts of how Bell was meeting its objectives in project management, finances, etc.
2. To forge a closer partnership with the client that allowed for more collaborative problem solving, particularly in regards to issues that Bell faced with VANOC itself

The result was an unprecedented level of transparency and trust among the leadership group. As that sense of trust deepened, both Bell and VANOC were able to push each other for higher standards of delivery speed, faster decision making and innovation.

3.5 Lessons learned: Developing a strong partnership through transparency and trust

The success or failure of any new business program often hinges on how well a team coalesces and works together toward a common goal. When that team is based on a managed outsourced model, representatives from both the service provider and the client organization must find ways to integrate their efforts at all levels.

The Bell-VANOC partnership demonstrated several best practices in aligning people and building a mutual culture of trust collaboration and peak performance:

1. Embed working-level employees early in the engagement – being part of the clients' organization from the beginning, or nearly so, helps to gain a clear understanding of the service requirements and in forging an effective day-to-day working relationship
2. Establish common priorities and goals with partners – to alleviate the complexity and potential conflicts inherent with multiple service providers, leaders must mutually determine a framework for addressing how decisions should be made and challenges addressed
3. Employ a governance model of transparency – approaching the managed outsourced engagement with an inclusive and open attitude toward the client will build trust and allow for problems to be solved more efficiently and effectively

4.0 Managing the project: marshalling resources to achieve high standards

From start to finish, the VANOC program required some six years of project planning and management from Bell. In that time, every single business unit at Bell played a role at some point.

Marshalling and organizing the resources required to meet and exceed the requirements of designing, deploying and operating the communication infrastructure for the Olympic Games was a critical success factor for Bell.

In this section, we detail the model by which Bell structured its own operations to ensure success.

4.1 The Olympic Games Program Management Office

Established in the fall of 2005, this department oversaw all of the 29 different program streams that Bell managed as part of its VANOC engagement. In addition to tracking every penny that Bell spent on the Olympic Games program, the office carried the responsibility of making it easy for executives to quickly know the status of its work on the Games.

The PMO consolidated all of the finance functions into a single Olympic Games finance unit.

Staff at the PMO was purposely kept to just 8 people to reduce overhead, and avoid the tendency to build out a needless layer of administration. There were three aspects to its oversight role:

1. Strategic Planning, which was comprised of:
   1. Development of the strategic portfolio of projects necessary to hit its deliverables and meet its commitments
   2. Development and review of tactical plans and change strategies
   3. Facilitation of mission and mandate sessions that created charters to specify what each program stream was in charge of managing
2. Business portfolio management, consisting of:
   1. Portfolio delivery, including assessments, launches and facilitation
   2. Training of project managers in best practices
3. Program governance, which included:
   1. Ongoing risk assessments
   2. Risk mitigation strategy sessions
   3. Guidance provided on project management standards and methodologies

Overall, the PMO focused on providing value to the project managers, not a burdensome layer of administration. The goal was to set high standards and help the project managers achieve it.

4.2 Balanced matrix

Each of the 29 program streams had multiple projects–most had more than 10, while some had nearly 40; some programs had a single project manager overseeing multiple projects, and others had multiple project managers. The wireless program stream, for example, had 12 project managers, as well as a separate program manager and program director.

For much of the engagement with VANOC, Bell did not need to dedicate many full-time resources to managing programs or projects. Instead, Bell utilized a “balanced matrix” reporting structure, whereby employees assigned to VANOC projects also retained the leadership and direction from their own business units.

Similarly at the vice-president level, no single VP had authority for the entire VANOC program, in part because no one would have the experience to effectively manage rollouts of solutions that span the full spectrum of Bell services. Instead, each program stream had a vice-president who was directly responsible for it, as well for other clients.

4.3 Reporting structure

How a PMO establishes its method of oversight is a crucial test of the culture it hopes to define for the duration of the engagement. Given the scope and complexity of the VANOC program, a careful balance needed to be struck between too much control, and not enough.

The PMO put in place a governance structure whereby project managers “on the ground” were given clear direction from vice-presidents and other leaders, but also granted the autonomy to use whatever tools they typically use to manage projects. The PMO did not attempt to impose methods on project managers, recognizing that those were the experts in their fields.

4.4 The charter

An important tool by which project managers and their teams were given direction was a charter for each project, revised annually. Developed in collaboration between the PMO and the project managers, these documents laid out in detail the key deliverables and timelines, which were then incorporated into the reporting system for monthly tracking.

Project teams recognized they needed to set realistic dates given the close monitoring, and everyone signed off on the charter: the project managers, the Director of the Olympic Program Management Office, Finance, and the project or program stream sponsor, typically a vice-president in the related operating division.

4.5 Done, Won and One

The PMO used several tools to provide focus and purpose for each project. One of these was a series of three questions that defined a project's critical facets, which were then incorporated into a project's charter.

Known as “Done, Won and One,” the questions the PMO and the project manager answered were:

1. When are we done? In other words, what are the key deliverables and dates that need to be met?
2. When have we won? It's not just about crossing the finishing line on time. What else defines success for the project? For instance, did we find savings? Were we able to leverage buying power for better prices? Telling the project team what specific priorities they should have in mind–above and beyond the completion of the project on time and on budget.
3. Who is the one person responsible for deciding what's “done” and “won”? A key reason why projects fail is because a committee oversees it and provides conflicting direction. Assigning an individual, typically a vice-president, to be a project's sponsor at the outset gives him or her sole responsibility to set expectations and provide clear direction on deliverables–the “done” and “won”–to the team that is managing the project. If there were disagreements over a project's direction, they could be elevated to the sponsor. For the PMO, it also determined who had these performance objectives in his or her purview.

In the later stages of the engagement, many of the remaining projects were operational in nature, and were therefore sponsored by the Vice President of Olympic Services. In some cases, particularly with projects related to marketing and company profile or reputation, the sponsor was the Chief Executive Officer.

This framework provided clarity for the project management at all levels, eliminating any grey areas or second-guessing. Directors and vice-presidents in particular appreciated having the material at their fingertips so they could refer back to the specific directions established at the outset of the project.

4.6 Green, yellow, red: Monthly tracking

Project managers were given explicit requirements about reporting progress back to the PMO, especially in regards to budget. Closely tracking the amount of budget consumed by a project while underway is an effective means of knowing whether it is on track or not.

Project teams regularly updated information into a project management system that detailed what needed to be done, by when, and cost. On a monthly basis, this information was gathered by the PMO to track it against its charter, and each project was given one of three status colours:

• Green – on track, and within budget
• Yellow – haven't missed on either deliverable date, scope of project, or budget, but are in danger of doing so and require assistance
• Red – one component (schedule, quality or budget) has been missed, and the project needs to reset its baseline for how it will be completed

If targets were being missed, the PMO openly discussed what could be learned and how to get back on track.

In addition, the PMO would review every program's budget at a granular level every month. Following where the pennies were being spent allowed leaders to quickly assess how projects were going.

All of this information was extracted into a master tool that the PMO and the VP of Olympic Services would look at to identify areas of concerns. This information was also the basis of the monthly report presented to the Olympic Technology Board.

4.7 The project priority triangle

There's a saying in project management: You can have it good, you can have it fast, or you can have it cheap–which two do you want? Balancing the naturally conflicting agendas of quality, schedule and budget is a central part of managing projects.

For deploying the hundreds of projects for VANOC, the Bell PMO used a project management tool called the project priority triangle. A visual representation of the three constraints that any project faces, it helps to determine for each project which constraints should be given greater emphasis than others, and which could be sacrificed to the benefit of the others. In the case of most VANOC projects, the “dot” was placed such that the following held true:

1. Quality: Do not skimp on quality or performance – project managers could spend additional money to ensure it met a high standard
2. Budget: Be within budget at all times, and where possible, find savings
3. Schedule: Given the long lead time of many of the projects, the schedule could be sacrificed should it be necessary for benefits in regards to quality, or to save costs

The benefit of this approach was that it provided clear direction to how project managers made decisions about their individual projects, with minimal added oversight from their sponsors or the PMO. Each project was given a baseline set of expectations in terms of schedule, cost, and scope. Then, when something changed in the process of completing the project or issues arose, as they inevitably do, project managers were aware of their variances and knew how to react. The same held true for the PMO and the sponsors if a project manager came to them with suggested changes.

4.8 Lessons learned: Empowering success

Without careful attention to its processes, project management can become a burdensome layer of administration that generates many reports but few results. For project management to deliver effective governance and oversight, it must determine unobtrusive ways of tracking progress and providing support to project managers.

In a complex program like the 2010 Winter Games, with thousands of deliverables and milestones that numbered in the tens of thousands, Bell employed world-class project management best practices. Along the way, it demonstrated the importance of several critical lessons:

1. Clarify goals – the entire organization must have a clear understanding of goals and priorities about what the client is trying to achieve, with a complete view of what risks, assumptions and dependencies exist for each project
2. Focus on what is necessary to win, not simply complete – for defining success at a higher standard than delivering projects on time, on budget and to scope
3. Trust you have the right people to deliver, and empower them – allow project managers to use their own expertise to decide how best to reach the standards set for them, with clear guidance as to the framework that will be used to judge success; when requested, provide assistance as necessary
4. Establish a reporting structure that ensures transparency – helps create a culture in which project managers understand that success relies upon them acknowledging mistakes or setbacks immediately, and seeking assistance

5.0 Designing the ideal Olympic Games network

Unlike prior Olympic Games, the Vancouver 2010 Winter Games would have a communication network designed and built from scratch. Such a “Greenfield” implementation presented a large blank canvas upon which Bell could create and deploy the ideal world-class communication network for these Olympic Games.

But what is that ideal?

Clearly there were many technologies available that Bell could have deployed, but technology is only a means to an end – in this case, delivering a flawless Olympic Games experience.

How did Bell engineer a world-class all-IP communications infrastructure from the ground up?

This section will detail the processes by which Bell made decisions about the architecture of the VANOC technology solutions, and the “how” and “why” of the technology used to address VANOC's communication requirements.

5.1 Defining requirements

A preliminary step that Bell performs at the beginning of an engagement with any large organization is to evaluate its business and communication requirements.

In planning any new deployment a managed outsourced provider needs to focus on establishing the types of communication that will be used, the sources of data, and the various usage scenarios, not the individual technological components.

With a complete picture of the business requirements, those requirements can then be mapped to the right products, services, and the professional services that will stitch them all together.

5.1.1 Information gathering

The technical solutions deployed at previous Olympic Games, even as recently as the 2008 Beijing Olympic Games, were of little consequence to the design of the 2010 Winter Games networks, for several reasons:

1. Communications technology advances at an increasingly fast pace
2. There were many subtle differences in terms of the geography and venues
3. Because VANOC had made the unprecedented decision to have a single provider of all communication services–in contrast to the separate telephone, mobile, and data service providers of prior Olympic Games–Bell was able to take a holistic approach to its planning

However, Bell did study the 2006 Olympic Winter Games in Torino, Italy, for example, and watched how communication networks performed in Beijing in order to gain insights into how technology was used. By observing and interviewing Olympic Games officials and volunteers on the ground, Bell understood what kinds of telecom services they used and when–for instance, how often they used wired phones instead of mobiles. Bell created personas of what certain classifications of individuals need in order to accomplish their business needs. In general, Bell built up a body of knowledge of how it might deliver solutions.

One important insight Bell gained from shadowing previous Games organizers was that Olympic Games-related wireless usage has generally tripled every two years, with instant fourfold increases in demand during pivotal events such as the opening and closing ceremonies. That information lead Bell to overbuild wireless network coverage throughout venues by 25%.

5.1.2 Establishing design criteria

At a high level, it was critical that Bell have some high-level criteria for building the technical architecture. Although the Olympic Games is, at its foundation, a sporting event, it has other broad requirements, including:

• Safety of individuals
• Security of information
• Accuracy of record for timing, scoring

Bell took the business requirements for usage, and incorporated design criteria. The challenge, however, was that there some unknowable components including the requirements for bandwidth and capacity. In all, Bell defined five key requirements for the converged network infrastructure and to address the unknown levels of demand its services would face during the Games:

• Capacity
• Reliability
• Redundancy
• Survivability
• Security

These five “design mantras” were entrenched in the contractual relationship between Bell and VANOC as the top five considerations in every aspect of the services they delivered.

5.1.3 Technical architecture

Bell then developed technical architecture that supported the services and all of the different delivery mechanisms for everything forecasted for users. This included how to deliver each one of those products or service components, and what types of vendors to partner with.

In the final architecture, this is how those design criteria were met:

• Capacity–must support bandwidth-intensive applications delivered on excess of 10,000 VoIP phones and up to 40,000 wired and wireless Ethernet ports on a 10Gbps core network
• Reliability–carrier-class core network switches include redundant power, common control, and switching fabrics to guard against single points of failure
• Redundancy–redundant Ethernet switch deployed at each venue must support the entire capacity with sub-second failover in case of catastrophic failure of one of the switches. Non-disruptive operation must be maintained 7/24 for staff, athletes and media
• Survivability–provided through a physical diversified access and core network
• Security–across multiple networks, application types, and devices via logical partitioning

5.2 Flow-through provisioning

One challenge that Bell encountered in its design of the technical architecture was how to ensure that services could be provisioned very quickly. In most cases, Bell designed architecture without any pre-existing infrastructure in place, and in some cases, no access to buildings until just days before the Games.

For Bell, that meant it had to modify how it typically provisions new services such as a phone line to a desk. It purpose-built an entirely new business process and IT system for flow-through provisioning.

Typically the provisioning of phone service to a desk entails many steps: assigning a telephone number, assigning the equipment, deploying a technician to patch or crossconnect the cable centrally, on the street, at the business location, and then at the desk.

With the new flow-through provisioning system, everything was automated, from the end-user's request right through until the phone was provisioned. Bell severed the interdependencies of the inventory from the services:

• Cabling: where will people sit? When the location of end-users was determined through space planning, the ubiquitous Ethernet access made questions about what those users would use irrelevant. Bell could cable based on the seating plan, not on the orders for phone service, Internet access, etc.
• Service infrastructure: this worked under the assumption that the cables would reach the desks, stripping out as much complexity in the process as possible by automating it. Based on requests from users, the network could automatically configure service, so no one had to manually verify.

The all-IP network made the delivery mechanism for phones and LAN access alike flexible and with a lower cost for implementation, moves and changes. From the perspective of the IT department, no custom implementations were necessary–it was all identical. The network detected the phones and provisioned them accordingly, while users authenticated with their credentials to access the services assigned to them.

The result was that Bell was able to provision up to 4,000 VoIP connections in less than an hour, with just one IT employee watching the system to make sure there were no errors.

In total, the business process applied to 9 distinct product categories that included more than 55 individual products–73% of which could be ordered and provisioned without any direct human intervention.

The flow-through provisioning proved its flexibility on the day of the Opening Ceremonies, when VANOC requested an additional 1,000 NetZone internet access accounts for the press centre–something that Bell delivered within minutes.

One key facet of the flow-through provisioning system was a security mechanism that restricted computer access to any resources until the network authenticated it. The network-based security protected the ports at each desk, alleviating the need for endpoint connections to be made manually.

5.3 Security

Securing the network was one of the top design criteria of the technical architecture. This was a critical decision made at the earliest stages. The benefit was that rather than try to add in security or address potential vulnerabilities after the infrastructure was in place, security was from the start a primary consideration for every decision made about the technical architecture.

One key was to build into the core infrastructure areas Bell knew it could change without disrupting the rest of the network. The network also had monitoring points that looked for suspect data traffic patterns and the heuristics of data patterns similar to known suspect patterns.

The Bell professional services security experts were also employed extensively. Not only did they watch trends about potential threats, but they conducted internal audits for each stage of the network build and validated the solutions engineering team's assumptions. In what was likely the most comprehensive security assessment ever done in Canada, they conducted vulnerability testing every six months to scrutinize the network in the event of either internal or external attacks.

In the end, no security issues significantly affected the network–confirmation that the intensive security design process successfully addressed potential issues in advance.

5.4 Vancouver 2010.com: Delivering content instantly to billions of people

Two key components influenced the technical design for Vancouver2010.com:

1. Capacity to meet the demands of a global audience and a large variability in traffic
2. Ensuring results were updated in sub-second time frames

First, Bell anticipated an exponential increase in traffic even compared to the Beijing Olympic Games only two years earlier. Bell Web Solutions also needed to scale-up the site for a global audience with a huge variability in traffic. In order to meet its capacity requirements, Bell used a Content Delivery Network (CDN) to move content from its core to the edge of the network, in data centres located around the globe. Not only do CDNs keep large amounts of traffic from crashing a site, they also provided security in that content is distributed and no single location is vulnerable to a Denial of Service attack.

Secondly, it delivered real-time results updating using solutions that were compatible and optimized for CDN. Forecasting that real-time data would be a key draw to Vancouver2010.com, Bell Web Solutions analyzed how content protocols were evolving, how CDNs could index content nearly instantaneously and how to design the site to leverage that.

Ultimately, the Vancouver2010.com website received 1.2 billion page views and was the first official Olympic Games site to suffer no network outages during the Games.

5.5 Lessons learned: No standards except what's right for the customer

In designing a technical solution for a customer, the focus must remain on its needs, scope and scale. In its work for VANOC, Bell had a large client with a unique set of needs that required professional services, married with managed services in an outsourced model.

In delivering a holistic solution custom-built to match VANOC's comprehensive communications needs, Bell demonstrated three critical best practices:

• Define business requirements, then map solutions to it – employing a structured process that first determines the complete set of needs allows for the selection of technologies and services that better serve the client and end-users
• Set design criteria for common requirements – establishing clear standards guides the process toward the most effective solutions
• Leverage IP infrastructure – provisioning services quickly and cost effectively is now easier than ever before thanks to advancements in IP-based networking technologies

6.0 The Olympic Winter Games legacy

Olympic Games are a singular opportunity for athletes to prove to the world what they are truly capable of achieving. For years, they prepare on a daily basis, dedicating their lives toward attaining peak performance in that one, brief moment, when the stakes are at their highest, and when nothing short of flawless execution will guarantee success.

For Bell, it was no different. In the six years leading up to the Vancouver 2010 Winter Games, Bell applied resources from every corner of its organization to deliver for VANOC a holistic communications solution worthy of the Olympic name. It needed to be scalable, flexible, and secure–and it needed to perform flawlessly when it counted most, when nearly the whole world was watching.

All the pieces had to fall in place at just the right time.

They certainly did – and then some.

Bell's Vancouver 2010 Olympic Winter Games experience showcased what is possible when it combines its full range of products, services and professional services expertise and applies all of it to a common goal.

The unprecedented success of what Bell accomplished, particularly at the massive scope and scale of the Olympic Games, sets a new standard for managed outsourced service engagements in Canada. It represents an evolutionary step in the depth of best practices that can be applied in large, complex communication implementations, particularly in regards to program governance, project management, and solutions engineering.

The Bell Vancouver 2010 Olympic Winter Games experience demonstrates the significant value of these best practices:

• Developing transparency and trust between client and service provider
• Building a culture of peak performance among employees
• Managing projects to encourage innovative approaches that improve the end deliverable
• Designing an effective end-to-end solution that exceeds client requirements
• Deployment of all-IP networks
• Rapid provisioning of services over IP

For all Canadians involved with the Vancouver 2010 Olympic Winter Games, no matter how small their role, it was an event that will remain with them all their lives. The experience has had an equally lasting impact on Bell as it learns from those successes and applies them throughout its organization.

7.0 Bell and your organization

Bell is a global leader in delivering innovative managed outsourced services that span the full spectrum of communication solutions, including IP-based voice and data networks, mobile and LAN wireless, Web services and security, as well as the professional services to support it all. We have the vision, people, technology and incomparable credentials to help you at all stages of addressing your business requirements, from planning through to execution and the ongoing provisioning of equipment, services, applications and expertise.

To find out more about how Bell can help you, to explore your options, or to inquire about a specific technology, contact your Bell representative or click here to have a Bell representative contact you.

About the author

Justin Webb is Vice President of Bell Olympic Services. He led a team of over 400 Bell specialists from across the country with a mandate of helping Vancouver 2010 deliver flawless Games to the world.

Every image seen on TV, every news story filed to the world and every real-time score transmitted during the Games traversed a network and communications solutions designed by Bell. It was Justin's team that delivered these solutions: everything from voice, data, Internet, cabling, portal, two-way radio and mobility services for the world's first all-IP Olympic Games. Justin also managed the Bell 2010 Sustainability Program that ensured the benefits of Bell's involvement remained long after 2010.

Previously, he provided executive leadership for Bell West, Bell Mobility and Bell International as well as consultation to a burgeoning telecommunications firm in South Korea.