Saturday, 11 March 2017

Technology Leaders Be Brave and Focus On Company Culture

Image result for leaderAny technology business has to work out from a tech standpoint if they want to be an innovator or follower in the IT industry. It isn't always a simple choice for technology leaders, some companies that become followers will argue they have to do due diligence, they are hindered by internal bureaucracy, have the pressure of daily deliverables to worry about, so they can't afford to be too "bleeding edge" all the time. The more pertinent question is can they afford not to be leading the way? 

Technology leaders may believe that it is in their interest to play it safe, not rock the boat, but in reality there is no such thing as safe. The IT industry is so fast moving, that cracks will begin to appear in existing technology stacks over time, if they aren't invested in, cared for, upgraded, or continuously improved and automated. Standing still in IT will mean the tech that they are using will become dated compared to competitors.
Image result for rock the boat 
One point that becomes apparent when I attend technology conferences and speak to some companies is that they evidently don't have people passionate about technology directing the company. If you look at the most successful companies organisation structures they have a great mix of technical and business people at the helm. So why do so many technology companies in particular have this flaw and why are getting it so wrong? 

To innovate and be successful in the technology industry you need strong leadership at CTO, CIO, Director and Head Of level, that really understand technology and are passionate about continuously improving it, driving it forward, towards common and agreed goals. The worst thing any company can do is not have a clear and concise set of goals at senior management level, as it sends mixed messages to the engineers on the ground that need to deliver on the vision. Engineers can often become caught up in middle management politics which is exhausting for all involved. One joined up message is important, organisations should avoid having departments that act like shadow governments that will pick fault in other teams processes and ways of working. This is a collaboration anti-pattern that will just cause conflict.
Image result for passion 
These technology leadership positions in some companies are all too commonly made up of career managers that may not necessarily have the best understanding of the technology, who believe they have made it, hold a title and power so they are above being challenged on the decisions they make or questioned. There are good ones though, which I have had the pleasure to work with over the years. 

Image result for listenGreat technology leaders listen to their staff they don't treat them as disposable commodities. This is where the most common technology leadership faux pas lies, the power of a technology leader lies in their people and engineering talent, a good leader will know when to get out the way and let the team get on with it. Good technology leaders see their job as giving the engineers an environment to prosper and do their job, shielding them from internal politics and blockers.

Image result for devops cultureCTO's, Directors and Heads Of job is not about making low level technology decisions anymore, the role of a technology leader is now more about inspiring a successful culture in the business that allows engineers to innovate and deliver great products to market. Technology leaders need to be open to being challenged, not taking it as a slight, and improving ways of working based on constructive feedback. 

This of course means they need to have a low ego, they will put the company’s interests at the forefront, and not abuse the power they have or be easily offended. Command and control models led by technology leaders is not the way modern companies should be run, it is proven to fail. Tech leaders are in reality too detached from the day to day solution to make accurate technology decisions so this needs to be delegated. 

Image result for empowerThe engineers the tech leaders have hired should be making the low level technology decisions, tech leaders instead need to focus on the culture, steering the company objectives in the right direction and trusting the engineers to deliver the results. This may be hard for some technology leaders who have come through the engineering ranks, but they need to let go, trust those they have hired to do their jobs and not micro-manage their staff.  

For some technology leaders this means taking a step back, looking at what they are doing and evolving too to adopt this new way of working. Like engineers, technology leaders can learn too by looking at what others are doing in industry, so they too can improve. However, this means actually being open to change and listening, it shouldn't mean looking at what others are doing and using conformational bias to validate what they are already doing. It isn't too late to change and improve as no-one is perfect.

Image result for agile or waterfallMoving towards an agile approach is fundamentally important too. Milestone based approaches, dictated by managers going in an offsite meeting is anti-agile and doesn't make sense. 10 years ago that may have been acceptable but not today. The software industry is so fast moving that the road map that is documented this month will be out of date next month. This milestone approach should simply make way for agile prototyping using spikes in sprints to evaluate tools and processes, where engineers are trusted to try and test new solutions and show the business value.

Image result for agile spikeNot all prototyped solutions will work, but some will and bring huge business value or speed up ways of working. If an idea doesn't work as expected the engineering team and business will have at least learned a valuable lesson and the engineering team will have improved as a result. The ask of technology leadership from engineers is that their managers trust and support them to make the correct technology decisions. This by no means doesn't mean their decisions can't be questioned or challenged using data. 

The engineers on the ground are normally the most qualified people to make this assessment, so they should be empowered to do so. Management should know when to step aside, this takes bravery though. Including engineers in the decision making processes or even better delegating decision making is very hard for some technology leaders as they see this as power. My argument is that empowering people, is an even greater power.

It is vitally important that any technology decision made by technology leaders or engineers needs to be backed up by statistics, metrics, KPIs and data driven analysis. This is a common theme in DevOps culture, using feedback loops and measuring results to dictate the best ways to continuously improve the solution. 

Image result for grafana metricsIn the past technology leaders would traditionally make technology decisions based on gut feelings, instinct or even a whim. This really isn't acceptable today, as your competitor will likely be using big data analytics or machine learning, which should be shaping our business decisions and enabling us to become better decision makers, everything must be data driven now.  

If I was a betting man I would bet on data over gut instinct every time. The value of data, stats and measurement cannot be underestimated. In any industry there are the leaders and the followers. The followers will wait until someone like Gartner says that a technology is ready to use before even considering it, by then it is too late as half the industry will have already adopted it. 

Taking weeks or months to deliberate over putting a solution in is frankly not acceptable, if decision making is handed to engineering teams they can evaluate solutions within a few weeks. Managers will take weeks, months or even years evaluating a product or new technology, scared to put it into production encase it breaks. 

Image result for paceBureaucratic IT processes will outline rigid phases that need to be completed, such as a requirements phase, analysis, with multiple word documents or PowerPoint slides written before finally allowing an engineer to actually evaluate a new tool and see if it proves its worth with real business use cases. Instead this could have been prototyped in a week by an engineer in an agile spike and tested on a subset of use cases using a real world examples to see if the solution was viable. 

My view on this is that these bureaucratic processes are implemented by managers, to quell the pace at which the world around them is moving. So they put in these processes to make it more stagnant environment so they can control everything that is going on, this is done out of fear. Put simply IT management don't need to understand every fine grained detail, that is what they hire and should be empowering and trusting their staff to do. 

Image result for no micro-managementEngineers that feel empowered are more likely to be happy and will do their best work, happy staff will also encourage other good people to join the business, therefore raising the benchmark and technical level of the engineers in a company. When you hire good people they will bring good people with them. Poor managers or technology leaders instead will micro-manage staff, hire people less clever than themselves, create an unhappy toxic culture and ultimately the best people will leave the business and move on to companies that empower and trust them.

In the fast paced technology space it is imperative for technology leaders to be brave and encourage innovation and not suppress it. If there are tools or new processes are available that will improve a solution it is imperative that they are implemented straight away. If you wait too long, then the value add or competitive advantage of implementing the solution depreciates and your competitors may implement it first.

Image result for devopsThe most successful projects are normally when a collection of engineers are given creative freedom to collaborate, share ideas and build something great together, without having to deal with middle management in fighting and corporate politics. This is DevOps in its truest form. 

This is what the DevOps movement is all about, engineers in reality want to work on interesting projects, with cutting edge technologies that allow them to continually improve and couldn't care less about company politics. It is probably true that what most engineers want in life is to be left alone to get on with their job. Gartner calls this mode 2, but they should simply call it the future.


Sunday, 1 January 2017

Lessons Learned From Running OpenStack In Production For Over A Year

Why we chose OpenStack and SDN?

When we first embarked on building a new private cloud in 2015, we needed technology which would last at least 5 years so we chose to use the Red Hat distribution of OpenStack and software defined networking with Nuage Networks.
Image result for paddy power betfairThe reference architecture for our Paddy Power Betfair OpenStack implementation can be found here, which was created to advise anyone interested in embarking on a similar private cloud journey, as we believed we would have benefited from a similar document at the start of this project and wished to share our experiences:
https://betsandbits.com/2016/10/25/openstack-reference-architecture/

OpenStack was chosen as we wanted an API middle-ware that we could connect networking, compute and storage to and control programmatically. This would make the private cloud solution vendor agnostic, so at a later date if we wanted to introduce a new compute, storage or network vendor we could easily substitute them in without having to write new automation orchestration each time.

Image result for openstack new logoIt also gave us the option that we could even switch out our OpenStack vendor at a later date if we desired or use the upstream OpenStack distribution once we had built the necessary in house skills. So as long as there is a Nova (Compute), Neutron (Networking) or Cinder (Block Storage) vendor driver for the compute, networking or storage then it can easily be swapped in and used in our private cloud.

Using OpenStack as our automation middle-ware meant that we could build our continuous delivery pipeline orchestration using the OpenStack APIs and they would be compatible with any OpenStack distribution. This was instead of using vendor API's directly that could suddenly change in the next major release meaning rework, or worse still, becoming stuck on older versions of software that would eventually go out of support.

For our automation pipeline orchestration we use the OpenStack Shade library heavily https://github.com/openstack-infra/shade/tree/master/shade and the Ansible OpenStack modules http://docs.ansible.com/ansible/list_of_cloud_modules.html#openstack

Image result for openstack shade libraryAll the OpenStack Ansible modules are written with the Shade client library, so any OpenStack Ansible module that is written will tested to make sure it is inter-operable with any OpenStack distribution be it Red Hat, Mirantis, Suse etc. This compatibility is maintained by the OpenStack community and tested as part of the openstack-infra project and Shade releases and we have also added features to the Shade library when required to extend or create new modules to serve our needs.

Software defined networking was also deemed a must on the project as it would help us simplify network operations and allow us to mutate the network in a completely programmatic fashion. At the time the OpenStack Neutron component, was not nearly as mature as it is today, as of the Kilo release struggled to support OpenStack clouds at the scale of 650 compute nodes in one OpenStack cloud.

A good article on scaling the latest Newton release of the OpenStack Neutron (Networking) service can be found here and it seems to have come a long way since the Kilo release and now works at massive scale https://t.co/ywe9cK4upC

Image result for nuage vsp openstackUsing the Nuage VSP SDN platform meant that we could scale OpenStack Kilo to 650 compute nodes (hypervisors) we required per datacenter. Another benefit of using Nuage was that Neutron to date still does not have support for multicast and a lot of our applications needed this for cluster discovery in the overlay network. Details on how the Nuage SDN solution deals with multicast can be found in this video:
https://www.youtube.com/watch?v=oqF6ezq3eWE

The Nuage VSP solution also meant that we could easily bridge back to our legacy network for application dependencies that had not yet been migrated into OpenStack. This was achieved by making the legacy network routable using the Nuage VSG hardware gateway, which meant we didn't have to fiddle with individual VLANs each time an application needed to talk to the native network. Application access permissions are locked down using the Nuage ACL ingress and egress firewall policy on least privilege basis meaning each application is set-up with minimum amount of ACL policies it needs to operate.

What Challenges Have We Faced With OpenStack or Nuage VSP SDN?

One of the initial challenges we had was updating the OpenStack installer, as Red Hat had recently went from using the Foreman installer in the Juno release to using the new Red Hat OpenStack Installer (OSP Director) in the Kilo release which is a hardened version of the  RDO/ Triple O upstream project we needed to work out how we would integrate Nuage into this installer.
Image result for redhat rdo

The Red Hat OpenStack Installer uses Heat (OpenStacks orchestration service) templates to install OpenStack but at the time these only worked for installing pure OpenStack services, not SDN plug-ins, so initially a lot of development work had to go into integrating Nuage into the installer.


Nuage have since integrated these features https://github.com/dttocs/nuage-ospdirector/wiki which will save users from having to do the custom work we had to do when we started our OpenStack implementation. It is important to bear in mind any service needs to have a Heat template before it can be installed with OpenStack as the installer is a full life-cycle management tool. If as service isn't integrated in a Heat template it will be overwritten the next time the installer is run. There is no room for manual tweaks in this world or you will come unstuck as the infrastructure is scaled out.

Another feature we required was making the solution support our CLOS leaf-spine architecture provided by our Arista switches so the solution is rack aware. This customization is now going into the upstream OpenStack project https://review.openstack.org/#/c/377088/
This will make our OpenStack upgrades far easier as Nuage plug-in and leaf-spine support are integrated in the installer without the need for bespoke customisations every time a new Red Hat distribution is released.

A real game changer has been our partnership with Red Hat, who offered us to be a part of their high touch programme, this has allowed us to collaborate with Red Hat on a monthly basis on features we would like to see included in OpenStack. The CLOS leaf-spine support is one such feature that has been developed as a result of our feedback to Red Hat and they have created a blueprint to implement the features we required in the Triple O/ RDO/Red Hat OpenStack Installer to allow it to support modern networking needs.

Image result for timeoutsAside from the challenges with a brand new installer we have also had some teething problems when we have scaled out the infrastructure beyond a certain point, as some of the OpenStack default settings were not really set-up for scale we required, which caused some disruption when we hit some timeouts as we scaled out OpenStack but these were quickly found and increased. Although this is a superficial problem some timeout settings were initially quite hard to track down as you had to understand if the API call was controller or compute node side. When upping timeouts it is important that each timeout is updated across all the required OpenStack services as operations can cascade between different OpenStack API's.
So I would advise users to understand how the API calls work and how they traverse each service so when an issue is encountered they can track via the OpenStack request id in the logs.


When increasing timeouts on Nova and Neutron it was important to make sure that Nova, Neutron and HAProxy configuration are upped sufficiently to sensible values. Another default value to look out for is the file descriptor settings on RabbitMQ, in OpenStack these were exceptionally low meaning that once multiple concurrent API calls it hit the limit and maxed out. Upping these settings were really very easy to do once you found out where to look. I do think OpenStack would really benefit from having more sensible default settings and I know it is something that is being discussed in the OpenStack community.

We have also hit some minor bugs with Pacemaker and RabbitMQ versions in the Kilo release which were fixed by going to the latest versions that were already available. But these are simply all minor niggles which are natural when dealing with any new technology so really there was nothing out of the ordinary on this front or any show stoppers.


Image result for openvswitchOn the Nuage VSP SDN front one of the main challenges we hit was that we allowed developers to set-up their ACL rules themselves and the Nuage VRS (Open Vswitch) in Nuage VSP version 3.2 has a limit of 100 flows at ingress and 100 flows at egress per VM vport. So we have had a few development teams hit this limit initially as the ACL rules were configured in a sub-optimal way, but with a little help these were consolidated down to just the flows that were required per application so even our most complex application contain on average 50 egress and 50 ingress flows which is 50% below the limit. Nuage have since upped  the limit to 500 ingress/egress flows on the Nuage 4.x release so it would take a pretty insane application to require 500 ACL rules all to itself on ingress or egress to breach that limit.

The other main challenge we encountered with Nuage VSP was the amount we were calling the Nuage APIs meant that we had to move to the Nuage 3.2 R10 release as we were spinning up over 1000 virtual machines a day with a huge amount of ACLs which was being very CPU intensive on the Nuage VSDs.  But Nuage like Red Hat through our partnership programme, analysed how we were using the platform, replicated our set-up in their lab and then made some great optimizations for us to ensure as we scaled out the platform we could continue to make sure all requests for new virtual machines became active in a few seconds, as opposed to having to wait 30 seconds during busy periods. It really does pay to choose your partners with care as I haven't seen this degree of support from many vendors I have worked with over the years,

Quantifiable Benefits of OpenStack and Nuage VSP SDN?

So what are the quantifiable benefits of using OpenStack and Nuage VSP SDN together? So in our 1st year of running OpenStack in production here are some highlights when taking the approach we have achieved the following:

    Image result for benefits
  1.      Developers can now can self-serve on-boarding of applications and receive compute, networking and storage on demand
  2. ·        82 production microservice applications have been migrated onto the OpenStack platform so far using our automated continuous delivery pipelines and are live in production
  3. ·        We do on average 500 deploys a day to test and production environments on OpenStack 
  4.      We provision over 1100+ virtual machines each day on the 2 OpenStack clouds we have in each of our datacenters as all our virtual machines are immutable
  5. ·        We deployed 50 hypervisors using Red Hat OpenStack Installer across the 2 OpenStack clouds we have in each of our datacenters in one business day
  6. ·        We peaked at 650 deployments and 2000+ virtual machines deployed in the 1 day prior to Christmas
  7. ·        We do on average 220 production releases a week now on OpenStack
  8. ·        We currently have 2207 deployed active virtual machines in OpenStack which is just % of our end estate for our newly merged Paddy Power Betfair company
  9. ·        We now run 120 KVM hypervisors per datacentre (240 hypervisors in total) with the end state being 1300 KVM hypervisors.
  10. ·        We are currently running 17280 cores on OpenStack with 384 Terabytes of storage (end state is 100000 cores and 2.08 Petabytes of storage)

All in not a bad first year running OpenStack in production with some pretty impressive landmarks. In the new year we will move to the Ocata release of OpenStack so we are pretty excited about implementing the Manila and Ironic projects in OpenStack to offer bare metal and manage NFS.


Shameless Book Plug

I have also written a book called DevOps for Networking which looks at some of the techniques that can be applied to automate the data center and networking in particular. It also shows how to approach building a DevOps model at a company and trying to encourage network teams to automate their job. It focuses on multiple topics such as continuous integration and continuous deployment and covers OpenStack, Nuage and AWS as well as Ansible. 
It can be purchased at Packts website:
https://www.packtpub.com/networking-and-servers/devops-networking
or alternately on Amazon:
https://www.amazon.co.uk/DevOps-Networking-Steven-Armstrong/dp/1786464853/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=&sr=:




Image result for devops for networking steven armstrongI