4


Technical Review | January-March 2012


and certainly one that needs to be carefully explained and illustrated, in particular to ensure that expectations about what can be achieved and in what timescale, are managed. From these documents the important test scripts can be produced against which the systems will be exercised prior to service. Additional staff may be bought in to perform some of this process, however to ensure that future changes can be delivered the knowledge needs to be retained in the business by training of existing staff or engaging a business analyst who will continue to be associated with the business. Experience shows that demands of the business can create a need for continuous change and development. Such demand shows no sign of abating as media markets continue to evolve and we continue to need to satisfy the requirements of emerging platforms.


Whereas some years ago broadcasters may have written and operated their own software, most of us now buy products from the industry. One challenge of operating commercially produced software is managing the version control cycles. The inherent conflict or race between the supplier releasing new software versions to provide new functions and take account of different client requirements and the broadcaster’s need to test and regression test a new version to ensure it meets their own specific requirements. It is a fact of life for us, something that has to be mitigated but can never be resolved. I would suggest that even if scheduling products could be acquired as Cloud Services, there would be no less demand on the Engineers to manage version control to ensure reliability and performance are maintained.


We will see later how the need to maintain main and reserve, or online and offline systems, is critical to achieving the required levels of operational service. The reserve system and ideally a smaller but similarly configured test system provide the environment in which developments can be exercised first and shown to the operations team. Access to the test environment allows ‘super-users’ to experience developments first hand and conduct thorough testing of the system prior to being loaded onto the offline server environment. Overall this process can take many months and it is at this stage where confidence is established in the performance of new developments, safely isolated from the day to day business environment, in readiness for being taken online. We operate a structured change control process modelled on the ITIL1


methodology, this ensures


that all parties are familiar with the proposed change and have consented to its implementation prior to service.


Service Delivery


With good design, good implementation and support reliable database and server systems can deliver 99.999% (five 9’s) availability. However faults are likely to take longer to fix than with non IT systems and hence it is essential to have resilient systems to minimise the downtime of the channel.


Achieving high levels of availability requires both a well engineered infrastructure and appropriate maintenance. Good maintenance programmes for the databases and server systems are at the heart of these operations. Two important elements of this are well trained staff performing processes to a defined maintenance schedule and good monitoring of systems performance to address any short comings before they affect the service.


The way in which operations teams respond to issues is equally important. Very reliable systems can encourage complacency and it is important that the operational and support teams are well practised on how to respond when incidents do occur. Do they know of planned maintenance by partners, or how to ensure that the standby systems can be brought into service? Do they test the alarms by failing key components? Are they aware of the power supply arrangements for their building? We have found that it is not unusual for control room based operational staff to have little knowledge of the power systems on which they are so dependent. It is important to not just define processes for dealing with incidents but to test them to ensure that all staff can respond confidently to incidents.


Such regimes do not require large teams to support them. A key factor in sizing the team is the rate of change and development. Staying at the leading edge requires experienced staff to be available to support this change alongside ‘business as usual’. With good system design and configuration, the operational teams (working a shift pattern) can maintain a 24/7 service with the system support specialists working normal days. In this way it is much easier for the support team to accommodate project- led demands of training and supplier meetings. Even the smallest teams should always try to have individuals allocated to either project delivery or operational support, the response required and the approach needed in each area being quite different.


We are all likely to have service partners supporting our business either upstream or downstream (between us and the audience). The relationship with these partners and their performance is critical to ensuring that our services reach the audience reliably and consistently. Good customer service skills and a good understanding of broadcasting are essential. This together with defined and agreed protocols for managing incidents help to ensure that our own operations teams are immediately informed of outages, kept informed of resolution process and then receive a full incident report within 24hrs. A good example is provided by one of our suppliers that has monitoring systems which very rapidly confirm issues that have beendetected by our own equipment. Typically they will be calling us with an issue at the same time as we are cancelling our own alarms. One challenge we face is to ensure that those supplying data or satellite comms. services understand the broadcast market and the expectations of the audience. Taking circuits down in the


1


ITIL – Information Technology Infrastructure Library – A UK Govt backed methodology for the structured management of IT based environments.


Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32  |  Page 33  |  Page 34  |  Page 35  |  Page 36  |  Page 37  |  Page 38  |  Page 39  |  Page 40  |  Page 41  |  Page 42  |  Page 43  |  Page 44  |  Page 45  |  Page 46  |  Page 47  |  Page 48