Architect Roles in Digital TV Projects

The subject of IT architecture is vast and multi-faceted; often open to interpretation by organisations in the same and different businesses alike; especially the definition and expectations of the "architect" role in projects in supporting the technical needs, administration and day-to-day operational management. A number of people attempt to explain what architecture means, from professional institutions like IEEE & TOGAF that provide a framework for describing the various architect roles for IT projects to attempts by wikipedia.

The rationale behind my post is to highlight how the role of architect manifests in companies specialising in digital TV systems, be it on the broadcast systems engineering front, or product developer offering systems and software services. It has been my experience that even with companies involved in the same industry, the use and understanding of the role "architect" varies, and is open to interpretation, often leading to confusion when you're mixed up in customer-vendor service relationships in managing projects. Some companies do attempt at having an organisational structure supporting technical architects, whilst other companies try to hire anyone with a technical background and assign the role "architect". But the aim of this post is NOT to describe the traits and skill-sets of architect VS coder; but to focus on the functional roles that an increasing number of DTV projects are now demanding from architects.

If you're a PayTV operator like BSkyB, DirecTV, Multichoice, etc - you need to think about the level of technical expertise required for your business:

• how much technical systems knowledge do you need to own?
• how much do you depend on external vendors to manage this on your behalf (e.g. the design of your end-to-end system)?
• do you have to produce & own your very own technical specifications for the entire value chain; or just key parts (e.g. EPG Spec, Middleware Spec, UI Spec)?
• or are you comfortable relying on your preferred system partners?
• do you need to have technical people on your side to prevent vendors from pulling the wool over your eyes?
• do you need technical owners for each product/sub-system of your deployment?
• do you want to drive or your suppliers/vendors or be driven (locked-in) by your suppliers solutions?
• do you opt for generic products to configure & customise yourself, or do you impose a custom requirements to serve your purpose, to meet your business needs, according to your environment?

This list goes on & on... There are an increasing number of PayTV operators that are moving away from the traditional vendor-supplier-user model, to taking more ownership and control over their products. Some operators have set-up internal development houses to manage Headend & STB UI/Middleware development; whilst other operators are taking over STB manufacturing in an attempt to control the end-to-end value chain.  In such cases, it therefore becomes even more relevant to have a solid technical product team, connected from end-to-end, with a thorough understanding of complete system, delivery and value chain of products, from high level features, requirements and design to detailed technical operations of the product --- this is not just my opinion, some companies have already embarked on this path (BSkyB, DirecTV, Foxtel).

If you're in the business of providing systems & software services to the above-mentioned operators, the likes of NDS, OpenTV, Irdeto and other Professional Services companies, then you probably have well defined roles for architects, and probably have an organisational structure of an architects group, supporting your many projects, with a group of chief architects as overseers for your product deployments.  If not, then please contact me as I'm interested in learning about what others in the industry are doing here :-)

I've been mulling over writing about DTV architects for a while now, but what really triggered this humble attempt stems from a recent experience on one of the projects I'm managing: I'm sure most of you are familiar with "It's not my problem, I'm an application architect not a systems architect" - you look for a systems architect only to find that one doesn't exist yet!

Anyway, to take this discussion forward - I consider the end-to-end layout of a typical Digital TV system. I'll present the different flavours of architect roles, and conclude laying out the minimum expectation of the roles that must exist to fulfill your typical DTV product deployment, with a bias towards the STB-side than the Headend.

This is NOT a theoretical or academic exercise. I'm describing real world experiences from past projects I've worked on with NDS/BSkyB/DirecTV/Foxtel/OpenTV, although the major influence is from NDS who, in my opinion, do an excellent job at executing complex DTV projects; and have an org structure not so dissimilar to what I'm presenting here.

I am really interested in your comments and feedback, what is the rest of the industry doing in this area?? Feel free to comment directly or drop me an email.

Consider this picture of a traditional end-to-end Digital TV system:

Figure 1: Components of DTV System Value Chain (Download the PDF)

Disclaimer: I'm no architect, this is Version 1, just my understanding and interpretation only. Architects do the best they can under the circumstances....there is a level of pragmatism involved and no one should be under the pretense that the solutions created are perfect, they are what they are i.e. a solution out of a continuum of many possible solutions for the problem which solves a business need at that point in time for a price the organisation can afford under the assumptions of the perceived known knowns and anticipating the future usage for that solution at that time.

A typical end-to-end DTV system is complicated; there's certainly nothing simple to it. At the highest macro level, one could summarise a DTV system as being split fundamentally into three areas (or in architect-speak, subsystems):

• Transport/Signalling: This is basically the physical layer, the profile of the underlying delivery mechanism. It can be abstracted and treated as a separate system as the products that sit on top of this layer still provide the same application behaviour, regardless of medium. I've separated it out because here you'll find physical hardware/infrastructure systems that prepare the data for the transmission across the appropriate delivery mediums (Satellite, Cable, IP, Terrestrial, Mobile)
• Producer: Headend/Backend: Systems that generate, secures & distributes content and information over the delivery medium (satellite, cable, terrestrial, mobile, IP)
• Consumer: Devices - typically the STBs receiving the transmission, decode and presenting the info and a suitable user experience to the subscriber at home

Within each area, the system is broken down further and further down to individual components performing a specific task. In reality, you will find that this end-to-end system is a carefully orchestrated integration of a number of different subsystems; offering a mix of different hardware and software components that are supplied by different companies. This is especially true for the back-end, integrating third party components such as the Conditional Access (CA) system, Billing & Subscriber Management, Scheduling, Information Data Generators, Content Catalogue Servers, Play-out systems, Encoders & Multiplexers.    At the consumer device end, the STB is a system in itself, also broken up into components or subsystems; and also not provided by one single company: Typically the STB consists of an Application Engine / User Interface component (the EPG), Middleware (the TV operating system) and Platform Components (Drivers, Hardware, Kernel). And these core STB components themselves are broken down into sub-components.

So your end-to-end DTV system is complicated & highly component-based. These sub-components, components, sub-systems, systems and ultimately the end-to-end solution must be co-ordinated and controlled -usually all these systems obey strict rules and contracts as defined by Interface Control Documents - Architecture 101...

So where do the different architect roles fit in then?

At the macro level, an end-to-end system could be managed by a team of architects specialising in these broad roles, be it at the Headend or STB domain:

• End-to-End (E2E) Systems Solutions Architect
• This is the top of the hierarchy, the solutions architect has a birds-eye view of the end-to-end system. Generally working directly with the the business (marketing/sales) in understanding the business and market driving factors in creating a solution for a specific need or product. 
• The E2E solutions architect is someone who can work with any technology domain, has a deep appreciation for all the subsystems including third-party external components, and will work with the relevant system architects in defining the high-level solution. This results in producing a grand, high-level design of the system components or building blocks, specifies the requirements and interfaces expected from each subsystem, laying the ground rules for system architects to take forward, adapt as necessary to apply the design in a real-world deployment.  
• The solutions architect is a domain expert, expected to take existing deployments and adapt, expand and enhance as necessary to deliver new product requirements.  This is generally the first stage of the projects design and scoping phase.  
• In some projects the solutions architect goes right into the detail of enhancing the MPEG/DVB signalling specification; and some times even creating a new end-to-end signalling protocol altogether. 
• This is a highly collaborative role, involving a lot of communication back-and-forth with sales/marketing, product development and integrators.  This could be one person, or a team of solutions architects taking responsibility for certain slices of the solution, depending on the scale of the project and problem being solved.
• Some people use the traditional IT term "Enterprise Architect" - but in all the projects I've worked with, the DTV industry tends to use "Solutions Architect". 
• In a nutshell, the solutions architect uses existing subsystems to deliver a solution to a problem and identifying shortcomings/gaps for attention Systems/Product Architects

• End-to-End Product Systems Architect
• Depending on the project and product being delivered, the solutions architect usually takes an existing solution and adapts in to the needs of the new product. In doing so, the solutions architect will touch on many components or subsystems that are impacted by this change, thus highlighting the areas that make up the new product.
• One might therefore assign specific Product Architects to take responsibility and own the components making up the product's value chain. 
• Primarily concerned with interactions between subsystems where Headend and STB is subsystem.
• The Product Architect assumes responsibility for the product solution and works hand-in-hand with the solutions architect in defining the solution that best meets the needs of the business.
• Solutions architects typically get involved at the initiation stage of the project and leave to solve the next challenge - they rarely see a project through to completion.  Product Architects however are expected to stay till the very end, supporting and making all architectural decisions to get the product from development through launch.
• This is still a high level responsibility, product architects work closely together with respective system products architects: Headend & STB.
• The product architect drives through the respective product specifications, covering functional and non-functional requirements (performance, reliability, resilience, etc).
• This role should not be confused with Product Owner or Product Manager; which are more management roles focussed on product features, specifications and high level requirements, which feed the work of the Product Architect.  Depending on the size of the organization and skill-sets available, these roles could be merged.
• Some people tend to refer to Product Architects as Technical Owners.

• System Architect - The system architect spans Headend & STB subsystems:
• STB System Architect / STB Product Architect
• Responsible for the end-to-end stack of the STB covering platform hardware, kernel, driver layer, middleware and EPG - Responsible for the overall STB product, taking ownership for the full product.
• Has a deep understanding of the key areas of the STB to the point of specifying the interfaces for all the above components
• Collaborates with respective EPG & Middleware architects in designing the overall STB solution
• Is responsible for defining the Functional & Non-Functional Requirements of the system: e.g. monitor/stipulate memory usage and own performance
• Helping the product team with requirements, and that includes backlog, supporting the initial planning phases and producing high level estimate
• The gate-keeper for analysing change requests
• Assisting in analysing, reviewing & prioritizing defects

• Headend System Architect / Headend Product Architect
• Whist the STB is a fairly closed system, the Headend on the other hand is an  integration of distributed systems. These systems are all interconnected, the Headend product architect is concerned with the interactions, interoperability and reliability of the integration.
• Works closely with other architects (subsystem/component architects) in defining the product specification. Since the Headend is meant to provide generic services and is a high-risk operation, Headend systems are generally well-thought out in advance, go live months before the STB is ready for deployment. The product architect plays a big part in driving the integration of these systems at the high level.
• Product architects generally own slices of the system, e.g. VOD
• All these slices, i.e. product architects work closely with the Solutions Architect for Headend, sometimes referred to as Chief Architect

• Component Architect - STB
• Application Architect - UI / EPG
• Essentially designs the UI / EPG as a component. Responsible for the internal architecture of this component
• Collaborates with Middleware Architects as necessary e.g. influencing APIs and interfaces
• Works with the STB Product Owner to understand requirements from high-level to detail
• Works closely with user experience specification team to ensure the UI demands are realistic, respecting the boundaries of the architecture of the UI's internal components
• Supports the product team in defect analysis, review and prioritization
• Implement STB system architect's Functional & Non-Functional requirements
• Assist the programme management team in high-level planning and estimates

• Middleware Architect
• Middleware is a system in itself, consisting of multiple components. Without drilling down into too much detail, the Middleware Architect ensures the component satisfies all requirements by providing services to applications using it
• Generally, the STB Architect is in fact the Middleware architect; using Middleware sub-component architects for detail. I worked on a Middleware product that had a centralised STB architect driving a team of component architects, where the Middleware was split into 80+ components...
• Defines APIs both internal and external to the Middleware
• Works with the STB Product Owner to understand requirements from high-level to detail
• Supports the product team in defect analysis, review and prioritization
• Implement STB system architect's Functional & Non-Functional requirements
• Assist the programme management team in high-level planning and estimates

• Platform Architect - Kernel/Drivers
• Responsible for the design and specification of the low-level drivers and kernel interfaces, typically the industry refer to this as Hardware Abstraction Layer (HAL) or Common Driver Interface (CDI)
• In general STB hardware specifications, once a platform launches, undergoes minor updates over time - the interfaces are generally future-proof, and new interfaces try to be backwards compatible, so the Platform Architect is usually required at the start of the project to define the major hardware features and enhancements...

Example scenarios of where this hierarchy is applicable?

Of course, projects don't always have to adopt the complete structure. Real world example projects applicable to this model include: Introducing Hybrid-Satellite/IP features offering Progressive Download (PDL), PushVOD, Broadcast Downlod, PullVOD/IP, Recommendations, Integrated Search, Application Stores, Catch-up, Network PVR, DRM - all these products impact the end-to-end value chain and needs a carefully orchestrated architecture and project execution team.

Why I think projects need to have clearly assigned architect roles?

Projects without a well defined technical team are bound to face difficulty and will most likely fail.

Projects need to have access to a technical team, there must be ownership and accountability for the different areas of the system. More importantly, the product must be connected end-to-end, the value-chain must offer a coherent design solution. Some broadcasters operate separate business units that not well connected. Just like a project team is set-up to manage the project delivery, it is vital to set-up your technical solutions team, assigning people to support the product and project from start to completion.

Take for example, a common project for DTV companies nowadays of changing your STB supplier, be it changing EPG UI or the STB Middleware. For simplicity, assume there are no changes required in the HeadEnd, we're just want to swap out old for new.  At the very least, the project should have the following identified as key roles:

• End-to-End Systems Architect
• STB Architect (doubling as Product Architect)
• EPG UI Architect

At the end of the day, we need to deliver a product. We need to go live and make some money. Having clearly defined roles for architects on your project team will:

• provide clarity, removes confusion over roles, responsibilities & expectations
• highlight gaps in your organisation - for e.g. you might have an EPG architect, but missing a STB architect - this is only going to cause problems down the line for integration.
• provides interfaces and eases communication channels
• promotes a coherent system - without this, chaos looms on the horizon

If you're a broadcaster and lack the skill-set and competencies for these roles, then you should demand this from your vendors. 

I'm an operator, do I really need to go into that level of detail?

Again it depends on how you answered the opening questions, fundamentally how much do you want to be in control?? BSkyB & DirecTV for example, have  STB Middleware & EPG architects; as well as STB & Headend System architects.  BSkyB produce their own Middleware design specification, in addition to requirements. BSkyB are very involved with end-to-end solution design, often taking weeks to review and comment on design proposals. As an operator, BSkyB have a strong technical team who understand the technology implementation of their entire value chain and can influence architectural discussions and decisions from both Headend & STB-side.  It is your product after-all, so why not control it as much as you can? At the macro level, it's down to trusting your suppliers and risk mitigation. Suppose your vendors disappear, do you know enough to support, enhance and maintain your products in their absence??

Useful Reference Material on the Subject

Who needs an architect anyway?

Architecture Definitions

Software Engineering Institute on Architecture definitions

How should an IT Architecture team be organized within a large company?

The Role and Task of the System Architect

The Role of the architect

Difference between coder and architect

The Role of the Agile Architect

Principles of the Agile Architect

Useful book on System Architecting

MIT Coursework on Architecture

MIT Coursework on Role of Architect

On E2E architect role in projects

I have shared my thoughts in the past on the subject of architects in the domain of digital TV software systems which can be found here. In this post I will share further thoughts about the specific role of an E2E (End-to-End) architect from a program/project management perspective, including my expectations of the role as well as the associated challenges I've faced with this concept, especially where the role either does not exist, or is not clearly defined in cases where companies don't have a clear enough understanding or have not yet set up a formal architecture competency.

My Background & My possible Anchor Bias

In my first decade of working experience with software and systems engineering projects, I was fortunate to work with world class technology service providers (S3 in Ireland & NDS/Cisco in UK). This experience helped shaped the way I would view future technology projects, including product & project management, software engineering, integration and delivery functions. It's the kind of experience that never leaves you, especially when you've witnessed first hand, one successful delivery after another both as an engineer and manager. So I'd come to see the world in a particular light, taking with me the style of project execution where ever I went, and assumed that all companies naturally followed similar concepts. And the cool thing about this experience was a well grounded appreciation for software & systems engineering design principles - principles which should not be confused with implementation methods, as what is a hot topic of debate these days, that of Agile V Waterfall, that has been the bulk of my recent challenges in projects in the last six years.

On returning back home to South Africa* still working in Digital Media TV domain, I soon realised that things are a little different to say the least (absence of architect roles, working with business analysts where I'd been used to analysis being a function of architecture, no defined role of systems integration, no method of continuous delivery & lack of solid post-launch monitoring & operations support systems). My first major program, involved a large-scale overhaul of the end-to-end broadcast system, including a video-on-demand component and a new set top box software stack. In order for the program to have a decent chance of success, I not only had to restructure & reshape the program, but also had to introduce functional roles that for me, were always obvious: an architecture team with a specific role of E2E architect, E2E systems integration, test and delivery teams. With a lot of hard work of convincing senior management these roles were necessary for project delivery, we managed to implement the structures I proposed that ultimately helped in the successful delivery of the project.

Strangely enough, as I took on other engagements in large programs in the same group of companies, I found myself repeating the same. It could be because I have a bias that is so strong and deep that does not allow me to see any other way, even though I always try to adapt my style to the culture and teams currently at my disposal, yet I still find myself coming back to these principles because its been proven and tested, that I can't really find any real fault in - possibly it is because these are indeed engineering principles that have stood the test of time, especially this concept of an End-to-End Solution Architect role. As the saying goes, never leave home without it, if I'm running a large program of work, that involves many systems & component suppliers, I never run a program without having an E2E architect attached to my programs, as step one. Once that is established (often as a result of canvassing support which takes up some time and energy), I move on to structuring the role of E2E systems integration, which in turn drives the processes and behaviours expected from component engineering and test teams. The program timeline then shows a simple picture of the high-level milestones involved, where the bulk of the program management and execution is a function of coordinating delivery plans with respective project delivery teams (who maintain the detailed project plans) as well as managing stakeholder engagement (which is a vital component of running large programs). This hasn't been smooth sailing all the time, as I'll try to share some of the challenges that frequently come up, later on in this post.

*Disclaimer: Although my writing is about an experience with primarily one industry domain in South Africa (Digital PayTV), I'm mindful about not passing a broad value judgement against all large corporates with a software competency. At first I thought these challenges of projects and roles like BAs for example was limited to just this environment, but the more I researched by skimming through job specs, attending networking events, technical conferences, meetups, training courses and meeting people from other corporates, I later realised the patterns run across all the major corporates like financial services, banking and the big telcos, that more often than not, the structures in place mirror more traditional-IT governance than hardcore technical software development structures that I was used to, in working in a pure technology services environment. So I'm reasonably confident there is no broad value judgement being applied here, that these experiences are likely to resonate with large companies operating outside PayTV.

Typical Landscape

The picture below aims to illustrate a typical landscape of this ecosystem. This could be separate entities run independently, or even group under a technology division, with different functional expertise, that in reality are still perceived as separate silos of responsibility:

This picture speaks to a typical reality of technology service providers in a digital TV value chain. Each silo represents a set of core technology expertise that exist in separate functional lines either under a technology division, or could exist as separate lines of business altogether. 

Each silo may have its own set of speciality with architects and development teams. Each unit may also likely to have a very unique culture: people, skill-set and ways of working. Each silo may also have its own operating model: how work gets prioritised and injected, build, test and release procedures. These units might also have their own terminology, a lexicon / vocabulary for their specific domains. They may also not be entirely self-sufficient, relying on external third-party component suppliers that form part of their system. These third-party suppliers are just like another silo, each with their own operating model, from architects to development teams, to ways of working as stipulated by their underlying contractual agreements.

These systems & teams usually come together in delivering a product or service that directly impacts a customer (end user). Typically launching a major new product (or platform, say for example a new internet product like a streaming app for video on demand) will impact at least all these systems. To do so, a project is set up as a program delivery initiative that will call for the co-ordination of all systems coming together in delivering the new product or feature to market. So a program team would usually be setup to manage this execution and delivery. Typically a program manager would be appointed to manage the full end-to-end delivery, which is not only the technology arm, but also the wider business streams for delivering product to market (marketing, customer care, sales, training, retention, communications, legal & regulatory). Sometimes, the overall program manager is supported by an end-to-end technical program manager, but this is not always the case. More often than not, the overall program manager assumes co-ordination of the technology ecosystem as well. For now though, I use program manager to either mean end-to-end technical program manager or overall program manager interchangeably.

The need for an End-to-End Solutions Architect

The picture below illustrates the positioning for a dedicated role of E2E Solution Architect to a project or program of work:

Just as you would appoint an end-to-end project / program manager to the role of maintaining the coherency of the overall delivery project, so too should you assign an end-to-end solution architect role, to maintain the integrity and coherency of the overall technical solution design. This is the crux of my proposition. Related to architecture is a close relative called end-to-end systems integrator role, I've touched on systems integration topic on a previous post called Worlds of System Integration, so will not discuss integration in this post. Related to end-to-end integration is testing, in my past experience, integration implied testing or QA, but as I later learnt on coming to back to South Africa, most companies still see QA as a separate activity, so I have another post that talks to the Worlds of QA testing.

Why?

Because it makes running a project so much simpler! 

When I work with an E2E architect, I use it as a vital supporting pillar of the project. The E2E architect will take care to understand the business or product requirements from the business owner, works with all the respective technology domain experts (usually the assigned system architects) on mapping out a solution architecture, defining the new technology dictionary and model (if needed), highlights the system building blocks and supporting components that would be impacted, agrees on high level technical requirements from such systems, as well as important integration points, technology risk assessment, along with unearthing required non-functional requirements (performance, stability, redundancy, etc.) that would be needed for a complete solution. All of this is kept high-level, with just enough information to allow the project team to shape and scope the project, as well as prepare the technical teams of design constraints to think about. All of this is done in the early stages of shaping and scoping a project, has proven in my experience to be absolutely vital in a successful project outcome.

To be assigned an E2E Solution Architect to a project is no small feat. The expectations are quite high indeed. The role not only requires a sound grasp of technical concepts, but also mandates a personality that is comfortable with extreme forms of collaboration, managing and dealing with diverse stakeholders, personalities and egos (don't forget egos - tech guys can be a difficult bunch of people with huge egos), a strong suite of leadership skills, self-discipline, self motivation and the ability to work with ambiguity, a responsible attitude to self-management of tasks (without depending on a project manager to co-ordinate every meeting for example). Added to that, the E2E Architect must have excellent communication skills, demonstrating both in written and verbal form, the ability not only to handle technical communications, but also be the person to translate to business stakeholders in a simple, non-technical manner.

I rely on this architect to also grasp technical and architectural issues that might come up in delivery, and be able to provide solutions and advise on ideas and proposals to manage any impediments on the overall solution provisioning.

Why can't you just feed the business requirements to each division?

The short answer is the risk of lack of coherency of an overarching, end-to-end solution design. The absence of an E2E architect acting as the gatekeeper for maintaining overall integrity and coherency of the business requirements and related solution design, runs the risk of each division designing and implementing fairly independent solutions, leading to fragmentation, increased integration and last-minute rework to get a coherent solution delivered. This also leads to a system implemented without foresight, lacking the vision or with the end in mind. An E2E architect would fill this void, save a lot of time and energy in putting all the pieces of the puzzle together. 

Though not impossible to complete a project without the E2E architect, it is often through a lot of last minute co-ordination efforts of a project management team, or some efforts of a few technical heroes coming together, taking collective ownership of the problem. Generally these units have other work and projects going on concurrently, where focus on a single project is difficult to achieve. If the environment of very open collaboration and team work, where there aren't really any silo-mentality, then yeah, it's possible. But the reality is that in most organisations, silos exist, collaboration is not the norm, and the overhead of a project management team coordinating these activities is too high, let alone the implied technical knowledge of these project managers to make this a worthwhile activity.

This reminds me of the story about Christopher Wren, the architect responsible for rebuilding St. Paul's Cathedral after the Great Fire of London. Wren was walking the length of the partially rebuilt cathedral when he asked three bricklayers what they were doing. The first bricklayer responded, "I'm working." The second said, "I'm building a wall.". The third paused, looked up, and then said, "I'm building a cathedral to the Almighty."**

How many teams fully appreciate the work they do in the context of the end-to-end ecosystem? It is not expected every team have this kind of vision, it would be great if they could all see the big picture and be visionary, but in reality they're not - and there's nothing wrong with that. The instructive part of Wren's story is that he didn't come up with a sense of purpose himself and pound the vision into everyone's head. Each bricklayer cared about something different, even though all three were working on the same thing. Wren's role was to listen, to recognise the significance of what he heard, and to create working conditions that allowed everyone to find meaning in their own way.

In the same way that Wren appreciated the vision, the E2E architect plays the same, working with, collaborating, listening and negotiating with individual teams. Feeding a business requirements spec to each technical division independently, without a golden thread joining everything together, is not the most effective way of maintaining coherency.

[Next section: Why a solution architect and not a technical project manager instead?...]

Architecture Interview Questions

I've written about architecture topics in the past, especially relating to the role of "Architect" in Digital TV Systems & Software projects. Though I'm no architect myself, although I could very well have become an architect if I chose to maintain on the purely technical path; I still have to be across the technical disciplines at the high level, when it comes to program or project managing large-scale technical projects. 

When I start or join a new project, one of the first things (apart from familiarizing myself with the product spec) I do is to seek out the architects, looking for any technical information in the form of design,

(document/wiki/etc) that scopes the high level system design or architecture that we're trying to implement & deliver. 

Failing that, having found no clearly defined architect role, I would go about setting up this role as a matter of urgency. No architect means no technical ownership, means flaky interfaces, chaotic system integration points, and a very bumpy project ride lays ahead!

In the past I have worked very closely with many great architects, both as an engineer as well as manager, that I've learnt from this experience enough to know what to look for when seeking out architects. If a project requires it, I do get involved with the interviewing architects, system integration managers, project managers and other roles. 

I thought I'd share a list of some of the questions I generally tend to ask when interviewing for the role of Architect in a Software/Systems project - hope you find it useful. If you read my other posts on architecture, you will get to understand I have a more holistic expectations from architects, rather than just answering this list of ~100 questions!

Career Ladders: Avoiding chaos and anarchy in Software / Systems Projects

This post is part rant & part structured discussion around the topic of career development in the domain of Software & Systems Engineering. I have been in the business of core software engineering for fifteen years (15), and was fortunate to experience working with highly rated professional companies (from Silicon Valley, internationally renowned) whose sole purpose was producing software systems from scratch, including as providing software design services to other software companies - these companies maintained true to engineering as a discipline.

A graduate engineer would enter the company with a fairly good idea of the where he/she is starting, and the various options & opportunities that lay ahead in terms of career growth, and, depending on the company - this graduate could spend

the next 10-15 years with just one company alone and never get bored, traversing many job disciplines as he/she so desired. 

This is the baggage (rightly or wrongly) that I come with, and hence my surprise to learn that some companies in South Africa are still making a serious mistake of not having a simple template that maps roughly the career ladders for the team, covering Development/Test/Integration Engineers, Architects and Managers. I firmly believe this is a recipe for chaos & anarchy that must be avoided as far as possible, and the solution to this problem is to map out a simple Career Jobs Ladder for your technical department.

Don't get me wrong: I am all for meritocracy, flexibility and not bureaucracy - but it really irks me to see things happen in the workplace that just don't make sense, especially when promotions happen when there is no real truth, track record or backed-up peer recognition that warrants a role / title change. 

I have seen the following happen as an example of chaos:

• An integration engineer with no prior design or architecture experience is promoted to Architect
• An software engineer with no prior architecture accolades is promoted to Architect
• A software engineer with no prior team leading, people management or facilitation experience is promoted to Scrum Master
• A team lead with no prior project management or scrum mastering track record becomes an Agile Manager
• An integration engineer with no product management experience becomes a Product Manager
• A new recruit with no prior experience in the technology domain joins as a Senior Solutions Specialist
• A new architect who has never architected any product before enters as a Solutions Architect
• An enterprise architect who has never delivered to market any real enterprise-class systems product that has a user base of more than fifty is made Enterprise Delivery Architect
• A component architect who's only worked on a single software module / component becomes Enterprise-wide Solutions Architect
• An integration engineer who's only experience in embedded devices joins an enterprise systems integration team as a Senior Integration Specialist
• An automation specialist or tech lead who is misunderstood as the Head of Automation

To an observer, the above scenarios are candidates for chaos (What do all these roles mean? What's the job spec? Is there a clear map that shows the progression of one role to another?, etc. etc.), although these cited migrations would not be that far fetched if there was a career ladder to hand, that facilitates the growth path - that can be used to take the individual on the journey to reaching his/her desired goal....

And this is where anarchy comes in (again, a little rant on my part):

Appreciating the Role of System Integration

Last year I started my series of posts around the different aspects of DTV system projects. Starting out by describing the a typical end-to-end system architecture and the importance of clearly identifying the roles and responsibilities of the architect team (see post: Architect Roles), followed by a write-up on considerations for setting up a programme organisation structure (see post: Managing a DTV Programme) and more recently, a brief overview of auditing the STB System Integration process (see post: SI/Dev Process Audits).

I still have a few more topics in the pipeline, however the aim of this post is to highlight the worlds of System Integration, as this can often be overlooked and open to interpretation, introduces dangerous assumptions and automatically puts the programme at risk.  With a large-scale DTV deployment, the programme must at the outset, clearly define the boundaries of system testing, and establish clear roles for the system integration effort.

This post is especially relevant to TV Operators deciding to become more involved with technical ownership of their value chain, as previously highlighted, more and more companies are taking the plunge into doing their own Software Development, and System Integration. This post is another humble attempt at raising awareness that such operators should proceed with caution, ensuring the full extent and gravity of the undertaking is understood, because System Integration is a significant investment in people (highly experienced technical experts required), equipment (investment in costly infrastructure) and time (starting from scratch, building this competency from the ground up is at least a 5 year maturity period).

Recall the end-to-end architecture originally present a few posts back:

As highlighted in previous posts, the world of Digital TV consists of Broadcasters / PayTV operators employing one of more component vendors or Technology Service Providers offering specialist services of Software development for Headend/STB components, Conditional Access Providers, Billing, Scheduling  & Play-out Systems and System Integration services.  This is made possible by basing the various technologies on open standards (MPEG/DVB/IET/IEEE) and thus allows for creating an end-to-end system consisting of a variety of components from different, independent vendors.  However, there are some TV operators who opt to use one supplier, often referred to as the Primary System Integrator, who are tasked with not only ensuring the end-to-end system is architected correctly, but quite often, the Primary System Integrator provides the bulk of the system components end-to-end (i.e. dual role of component developer supplier and integrator).

System Integration and Architecture are linked, they go hand-in-hand. They are not separate entities, although in theory the act of architecting is different to the act of integrating, but integration drives the architecture.  Looking at the above figure showing the different architectural streams, the following becomes clear:

• Solutions Architect is replaced with Solutions Integrator
• Headend Architect is replaced with Headend Integrator
• STB Architect is replaced with STB Integrator

This now exposes us to the world and levels of System Integration.  A Programme needs to define its System Integration strategy at the outset, success of the programme depends on a solid System Integration strategy.

If you're a broadcaster taking ownership of SI, you need to decide in what capacity you will be involved in:

• STB SI
• This is generally taking responsibility for integrating the various components of the STB stack (typically Platform Drivers, CA Component, Middleware, UI & Interactive Engines) providing a stable build (STB image).
• Interfacing with a broadcast/backend is part of this responsibility but STB SI assumes the necessary backend/headend infrastructure is in place
• Responsible for managing and driving through component deliveries in terms of quality expectations and to some extent project time-lines
• It is always useful to have access to component source code to aid in investigating and characterising bugs, but this isn't always possible as it has commercial implications with IPR, etc.

• Headend SI
• Within the Headend system itself, there are a number of component systems that must come together in implementing a specific macro-component features, like Conditional Access, Scheduling, On-Demand Catalogue/Playout, IPTV streaming.
• Generally though, headend SI is about integrating these headend components either individually or collectively, proving the interfaces are working as designed (i.e. verifying the input/output is as specified by the protocols, essentially obeying the Interface Control Definitions ICDs)
• Test tools might be used as test harnesses, or the STB itself can be used depending on the situation
• There is a fine line between Headend SI and End-to-End SI or Solution SI, which can often lead to confusion and false assumptions

• End-to-End SI or Solution Integration
• This is about testing the solution in a representative environment, as close to real life operations as possible
• It brings together proven systems at the component level, i.e. STB, Headend and other dependent infrastructure systems like Transport (Encoders, multiplexors, IP networking), Billing & Subscriber Management
• There is often a dependency or interface with systems that are currently live (live components are considered crown jewels, usually managed by a separate delivery/operations team).
• Failover, Recovery and Upgrade scenarios are performed as part of this exercise
• Requires investment in one or two end-to-end labs (Some broadcasters impose this on their major vendors to ensure appropriate end-to-end testing is setup locally at the vendor premises; and some vendors have end-to-end labs set-up on the operator's site)
• Often confusion around how this activity defers from System Delivery/Operations - think of this as pre-delivery, the last mile of the programme's development phase.
• Becomes really important to define an end-to-end SI strategy where the overall DTV value change is a conglomerate of components provided by different business units offering different services, e.g. a Broadcaster might have separate departments, sometimes independent units each offering a slice of the system (Broadcast, Interactive, VOD, Guide, Operations, R&D) - all these business units must come into alignment at the outset of planning the programme...

In a nutshell, the following diagram tries to illustrate the worlds of SI, the intersections shaded show the typical integration points required. The worlds are interconnected, and can't be expected to be tested and qualified in isolation of other systems (during development phase, unit/component testing is expected, but as soon as we have stable elements to make up the E2E system, then its time to kick-off your End-to-End SI activity). Typically, E2E testing must start well in advance of launch/go-live, typically 6-8 months of E2E testing is not unheard of...

Worlds of SI

The world of Delivery/Operations

The absolute last mile in completing a DTV Programme, is the actual launch - go live!  This is usually under the control of an Operational entity who's sole purpose is to ensure the smooth operational deployment of the systems on live - 99.9999% of reliability is expected, the transition from End-to-End SI to Live Deployment must have the highest probability of success.  In parallel to the E2E SI activity, there is usually a Delivery/Operations team that are mirroring the E2E SI effort, making preparations for launch.  This delivery team must be included in the Programme Schedule from the start, so they can make the necessary equipment procurements, network configurations and deployment schedules in advance to ensure the smooth transition.  Practically though, the delivery/operations team might re-use the same people from E2E SI, but it is important to ensure that everyone understands the milestones and phases of this activity.   Forward planning is crucial to avoid ambiguity and confusion down the line, which goes back to my point earlier that the boundaries and scope for each Testing/Integration activity are clearly understood.

As always, I'm interested in your comments and feedback. 

Managing Large-Scale Projects using Agile, Part 2 - Organisational Challenges

Organisational Challenges of Large-Scale SDPs

In this post I will describe the organisational challenges with supporting large-scale software projects, especially when the management philosophy is one that promotes agility, or has its roots in adopting Agile/Scrum early with a small team and then ramps up to using a globally distributed project team. Specifically describing the approach we took in managing the example project as described in the previous post. As with any organisation adopting Agile, there must be management support. In some cases organisations have transformed through bottom-up approach, but in an LSSDP project, some initial strategic planning is expected, it's imperative that management support is won at the outset (more about this later...).

Whether you're running a classic, relatively small project (say a team of 15) locally or multiple teams (250+ people), the essential elements of team/project management remain the same, the only difference is the scaling factor as the team expands. In my opinion, the following factors are integral to any software development initiative:

• Communications: Classic example is the communication model that Brooks provides, with n being the size of your team, the different communication paths is formulated as: n*(n-1)/2  - so a team of 10 would have 45 communication paths, so a team of at least 200 would create 19 900 paths!  With this in mind, pains must be taken to ensure that organisationally, a suitable structure must be put in place to sustain manageable communications.
• Organisational Team Structures: Flowing from the above problem of communications, each team needs to have an identity - roles, responsibilities needs to be understood. This is a trait common to both small and large teams alike. Defining the structure for the project/organisation through the classic org chart, does help in clarifying the structure.  Even though Agile promotes a flat team, collaborative decision making and participation, it is still useful to ensure the roles are identified and understood.  With a large-scale project though, collaborative discussions and decision making can still sometimes happen, but is extremely challenging. Large scale projects call for a really strong management structure (which involves technical people as well) to ensure momentum in decision making. 
• Worskpace Environments:  Common challenges to both large and small teams alike, one has to ensure the team's physical environment at which they work, is not only comfortable from a personal space perspective, but also to ensure the workspace can support the needs. For example, Set-Top-Box development/testing requires a desk-space large enough to cater for a few screens (monitors / TVs), ample power supplies, space for mobile stream players, etc.  Agile/Scrum promotes the use of whiteboards for tracking work for example: So do you have mobile whiteboards, fixed boards or what? How do you solve this problem when your team is distributed?? How do you ensure that all your people in all geographic locations have a similar setup as the rest of the world?
• People Skills / Training needs: Whether you have a small or large team, you will experience the same people challenges: Does the team have the knowledge/skill-set to do the job or task at hand, effectively, efficiently?? How do you build competencies across your team base?  What are the basic fundamental prerequisites of skills/knowledge you need to have? How do you promote transfer of knowledge & training?
• Peopleware Issues & Cross-Cultural Challenges: Building software is more about people dynamics than implementing technology. If you don't have a well-formed team (Agile/Scrum promotes self-organsing, well-formed teams), which is true for both large and small teams alike, you will experience problems that inhibit the smooth flow of development, impacting your project's delivery.  In a small team, the Team Leader, Scrum Master or Development Manager has more control over this, and has the time to see his/her team through the various stages of Forming, Storming, Norming & Performing. With small teams this process takes time.  With a large contingent of 200+ people, this process is exacerbated, being very difficult to manage and assess due to the spatial and temporal differences in managing distributed teams.  In our project we had people from UK (two locations), India, Israel (Russian, Polish, American, S. African), France & Korea, with third parties from Poland, UK, Ireland & India.   How can you achieve a well jelled team of international players? How do you solve the cultural issues? How do you avoid the frequent Lost In Translation moments?
• Maintaining Integrity of Software Architecture: All teams need a custodian of the software architecture, typically this is your architect, in our context, the STB architect. Where the team is small, you can have just one STB architect. On a team as large as 200+ people, all implementing a complex architecture, how do you ensure the technical integrity of the architecture is maintained?
• Tools/Infrastructure/Configuration Management: Like any other trade, a software team need the right tools for the job. Managing your local team it's easy to notice gaps and provide tools; it's far more complicated with distributed teams, since the challenge is in maintaining consistent adoption of tools, i.e. tools should proliferate and be different from team to team (Defect tracking tools spring to mind) so establishing a common infrastructure plan does help in creating consistency for the programme, and maintaining a sense of harmony within the distributed team itself.

In the remaining sections I'll touch on how we addressed each of the above areas referring to the example case study mentioned in my starting post. This post is arranged as follows:

• Workspace Environment
• Peopleware
• Organisational Structure
• Communications
• Maintaining Architectural Integrity
• Consistent Tools, Infrastructure & Configuration Management

Managing Large-Scale Projects using Agile, Part 3 - Product Management Methodology

Agile Product & Project Management Model - Large-Scale Set-Top-Box Development Project

A while ago I started writing about my experiences with managing large scale software development projects using the Agile philosophy of very Disciplined Agile Delivery (DAD). In order to do this, I had to first introduce some context & background in the three previous posts leading up to this one, the last post dealing with overall organizational challenges. Now the scene is all set to discuss the actual Agile model we eventually implemented, that continues to stand the test of time: 4 years and is still running quite strong to this day.

Some Background Material
Quick introduction to the world of digital TV
What is a Set-Top-Box?
What is an EPG?
What in the world is Sky TV?


Disclaimer
The experiences I'm writing about is about past projects that I personally worked on or touched in my time a Middleware Development Owner & Middleware Products Manager, that are very much in the public domain:
BSkyB replacing OpenTV HDPVR with NDS technology
Fusion is the new NDS Mediahighway to displace OpenTV
BSkyB Launches new HDPVR
Sky Italia replacing OpenTV HDPVR with NDS Mediahighway
Sky Deutchland delivers enhanced VOD powered by NDS Mediahighway
UPC being powered by NDS MediaHighway


The Project
Take a look at this video clip:





Yes, this is the background project I was fortunate to work on, which forms the basis for my writing on a large-scale software development project. We were responsible for delivering the core software component (Middleware / Operating System) that enabled the EPG (UI Application) to be developed to offer the Sky Anytime+ User Experience.


On the surface, when interacting with an EPG, the user experience might suggest this was a simple application, compared to powerful PC software products like Excel/Word or Adobe Photoshop; surely it can't be a complicated endeavour to create this system? That's not entirely true: Beneath the seemingly simple UI is a complicated piece of software and electronics, so much so, that typical projects such as this one takes at least three years to complete (if all goes well)...


The project was greenfields development on all fronts for the software vendors employed, especially on our side for the Middleware component. But for the broadcaster it was a massively complicated migration project, because essentially we were replacing the full software stack of a legacy deployed based of 4 million+ STBs, with the option for rollback!! 

The project, though massive in size, relied on a core product & architecture team to manage the roadmap and project planning; supported by a very large technical team (software engineers, integrators & testers) that were geographically distributed throughout the world.  Having joined the project a year-and-half into its development cycle, then spent the next two years executing and driving through the implementation and delivery to Launch Candidate builds, I played primarily the role as Middleware Development Owner. I inherited the Agile founding principles of the Programme, which after two full years of practical exposure, resulted in a deep appreciation for Agile, such that all my subsequent projects going forward re-used and evolved the model based on those principles. I later joined the Product Management team that championed the processes through roadshows, training and ramping up new project managers on the "Project Toolbox" required to deliver all future customer projects...

This post is split into the following sections to make for easy reading:

• Recap Software Architecture
• Agile Philosophy - Call it Structured / Disciplined Agile Delivery
• Concept of Product Backlog and Work Packages
• Digging further into the Product Backlog
• Work Package Software Development Lifecycle
• Hybrid / Structured / Disciplined Agile Delivery
• Large Scale equivalents of Agile / Scrum Meetings
• Iteration Timeline Template (Blueprint for Heartbeat)
• Overall Development Process Model

Pressure versus Urgency - is there a difference?

Last week I experienced an interesting interaction with the lead architect on my team, who'd been away for several days on a conference; and come back to find things not to his liking - the quality of the deliveries was not up to standard, people were not adhering to the architecture -- his take was that the team are under enormous pressure from me to get things done and delivered at the end of the sprint.

This caught me by surprise at first - how odd, I've not even started applying any pressure...But I've grown used to the different kinds of people on the project team; and hence didn't react negatively or emotionally.  My reaction:

"I appreciate your concern and understand where you're coming from. I think there's been a misunderstanding, perhaps due to my not making myself clear. There is a difference between Pressure and Urgency.  Yes, there's a sense of urgency around us finishing work according to plan because people are depending on work being created, but at no point should we compromise quality  - there was no pressure to take short-cuts; the architecture rules must be adhered to, no working off a branch, everything must be done on the main product, adhering to all quality standards..."

Having satisfied this architect, who left away happy that we were on the same page, I was still left with a sense of uncertainty myself. The team have probably got the concepts mixed up, although to be fair, there is rather a vague line between the two terms Pressure & Urgency -- but big difference with how these terms manifest themselves through action.  In the spirit of collaboration I then fired off an email to the team highlighting the difference in the context of our project and business needs, and then settled the matter at the retrospective. It turned out, as so often is the case with these things, the architect misunderstood what the team had been attempting, working on a branch only to prove concepts and theories that necessitate rapid development at the expense of some the architectural bottlenecks, i.e. we willingly broke the architecture to prove a few points....

Nevertheless, I still wonder if I may have unconsciously contributed to applying artificial pressure. I will explain the nature of my current project in a future post - Suffice to say my role is blurred, call it "Pseudo-Scum-Master-trying-to-balance-Hard-Delivery-Project-Manager-Development Manager-with-a-slight-touch-of-Product-and-Programme-Manager". I am seriously not blowing my own trumpet here, we are an evolving project team where the (unwritten) roles&responsibilities that were supposedly assumed well defined is in reality quite blurry - no surprise to software development projects, and especially not surprising when its an organisation's first attempt at truly in-house software development...

Definition of Pressure according to Free Dictionary

The ones that stand out for me are:

• the act of pressing
• the condition of being pressed
• the application of continuous force by one body on another that it is touching; compression
• a compelling or constraining influence, such as a moral force, on the mind or will: pressure to conform
• urgent claim or demand: under the pressure of business
• to force, as by overpowering influence or persuasion.
  an urgent claim or demand or series of urgent claims or demands to work under pressure

Definition of Urgency

• the quality or condition of being urgent; pressing importance: the urgency of the call for help; pleading with urgency.
• a pressing necessity
• all hands and the cook A state of emergency which of necessity becomes everyone’s top priority. This early American cowboy expression described the precarious state of affairs in which the herds were wild and all available persons were needed to bring the situation under control. Under normal circumstances, cowboys tended herds and cooks fed the cowhands; however, an emergency required that everyone chip in, temporarily ignoring differences of rank or task.
• the state of being urgent; an earnest and insistent necessity

Agile lends itself well to Urgency

The very nature of Agile/Scrum with its fixed sprint cycles, the two-week time-box for example can be in itself quite pressurising. There is a natural sense of urgency to ensure that enough detailed planning is done up-front, the work is well understood and the team hit the ground running on day one of the sprint. In Ken Schwaber's Agile Project management with Scrum, he talks about how important it is to do detailed planning, that agile doesn't ignore sensible process; and the importance of delivering something at the end of the sprint.

Incidentally, the same architect who talked to me about pressure, at my interview took pride in saying that the team would do whatever it takes to finish the sprint  according to plan - work overtime, come in on weekends, etc - what gives? According to Dave@PracticalAgility, projects with urgent goals are prime agile candidates, but he also cautions not rushing along to do things quickly, and the importance of taking a pause - which coincidentally is what I've proposed to senior management just yesterday (only saw his post today as I googled on this subject)...

Is there really a difference between Urgency & Pressure?

Yes, there is. It's not just my take on it - Others outside of software face similar challenges. Take for example the Pressure VS Urgency from Real Estate, although software projects are a little more peculiar.

People Under Pressure Don't Think Faster!

Tom DeMarco dedicates chapter 15 "Think Fast" of The Deadline to the effects of pressure. The Deadline is an interesting tale as it covers almost every experience one encounters in software projects, especially the unrealistic and nonsensical demands from senior management placed on their underlings, with the poor project manager acting as the buffer. "Think Fast" is about the main stakeholder changing direction and enforcing his might just because he can, and this sets the team down the road of discussing & modelling the effects of pressure. It concludes by asking the ever-knowing Oracle:

Why does the effect of pressure on programmers max out after only 6% productivity gain?

Oracle replies:

PEOPLE UNDER PRESSURE DON'T THINK FASTER -- Tim Lister

DeMarco then summarises the effects of pressure (Page 199, The Deadline):

• People under pressure don't think any faster
• Extended overtime is a productivity-reduction tactic
• Short bursts of pressure and even overtime may be a useful tactic as they focus people and increase the sense that the work is important, but extended pressure is always a mistake
• Perhaps managers make so much use of pressure because they don't know what else to do, or are daunted by how difficult the alternatives are
• Terrible suspicion: The real reason for use of pressure and overtime may be to make everyone look better when the project fails

The same message is discussed in the more serious book, outside of the novel context, in DeMarco's "Slack", Chapter 7, titled "The Cost of Pressure". Hopefully you can see this snapshot, please get a copy of this book.

Essentially confirming what is now well known: Pressure has limited capacity to reduce delivery time, maybe 10 or 15 percent at the most. Excessive pressure weakens performance. The model is divided into 3 regions:

• Region 1: workers respond to increasing pressure, eliminating waste, staying late, focus on critical path - falsely suggesting improved delivery
• Region 2: workers are getting tired, feeling pressure from home, doing other stuff "undertime" - no motivation, unwilling to sustain, resigning to ignorance
• Region 3: workers are polishing up CVs and beginning to look elsewhere

Closing Summary - Software projects can't hide from Pressure/Urgency

I have in the past worked on many a demanding project and I've always challenged management's decisions of increasing the pressure points, always quick to point out that adding pressure doesn't make people work any faster, it'll take as long as it takes, people will leave the project, etc. I must admit though, that I've come to learn there is indeed a difference between urgency and pressure; but it takes some tact to get it right. When people are motivated by an urgent need - urgent needs can be good motivators - people surprise themselves - problems are solved in novel ways, there is more focus and concentration. There is a natural pressure of course, but not the kind of pressure from a demanding boss expecting results now now now - if the urgency is communicated well, it brings out the best in people. If it's just imposed pressure, people turn to acid and can't wait to leave the project.

Applied in bursts, pressure/urgency can be a great tool - ultimately though, you're working through people - so ensure people are rewarded and appreciated for their efforts!

I have personally been on projects that may have seemed like Death Marches, but appreciated the sense of urgency to get things done, the reward was self-gratification is seeing the product deployed in tens of millions of people's homes, or providing crucial services to improving the company's bottom-line. Yes, some  projects could definitely been managed differently to avoid burnout, but sometimes, in the world of business this is an unfortunate reality; and especially in today's globalised economy - the hard truth of the matter is, engineers are a dime a dozen..

It is still important though to make sure your team understands the context of the business, the phase of the project and have a sense of awareness of the demands being placed on them, giving them the right and freedom to express concerns of pressure impacting the deliveries.