When joining a new company it is always slightly intimidating, as a newbie at a company you always question; Am I really smart enough to work here? What can I possibly bring to this organisation?Betfair was engineering led, when engineers on the ground had an idea, they were allowed the freedom to implement it and they were in charge.
My personal opinion was that the developers were really delivering fantastic products to market in spite of the internal IT processes that were hindering them. My main cause for surprise was on my 1st day I asked if I could have some virtual machines to start working on some proof of concepts and I was told that one of my colleagues had some servers I could use underneath his desk.
In early 2015 we started our i2 (infrastructure 2) project initiative to apply development principles to infrastructure. We would look at doing a major hardware refresh, which was long overdue. So we would use this opportunity to choose a new technology stack to create a new infrastructure platform that respected an everything as code mandate, nothing would be done manually, and would look to apply continuous integration and continuous delivery processes to all infrastructure.
At the same time we would provide a self-service framework for developers to consume that infrastructure so we had a consistent set of delivery tooling and workflow actions to create VMs, networks, storage in the same way, using Ansible playbooks, so we had complete consistency in our deployment methodology that we had previously lacked. This would make the software release process more predictable, manageable and whisper it boring.
This brought us to our i2 reference architecture, with OpenStack at the centre of it and Nuage networks used for software defined networking. We would use OpenStack as the API middleware that would programmatically control our infrastructure: OpenStack White Paper
At the time, during the selection process we heard from many detractors "OpenStack was dead", "it wouldn't scale", "it wouldn't be able to support the performance we required". "We needed proprietary software". This was all of course nonsense, fake news if you will, and we knew it. However, convincing everyone in the business this was the case was a monumental battle and I am proud to say we won by making data based arguments. In an argument it is hard to argue against hard facts. Always fight for what you believe is right and never give up, winning hearts and minds is the biggest challenge. For a company like Betfair that was engineering led, innovative, loved open source, linux based, OpenStack was a perfect fit and I still believe that.
However, implementing great technology such as OpenStack and Nuage are not enough to be successful, we were very lucky at Betfair as we had top down sponsorship from CTO level to make necessary culture changes to make it successful and the trust and support of management. If you do not couple the two together you will not create the desired business benefits.
On the infrastructure side when I joined there were cross-country silos, so we set-up DevOps roadshows to break down those boundaries. Infrastructure teams would present to each other what they were all doing and knowledge share. We sought to encourage collaboration across countries, improved team spirit and shared ideas to create cross-county initiatives. We empowered engineers, we didn't dictate to them.
Now at Paddy Power Betfair today we have teams from 4 counties (Ireland was added post-merger) peer reviewing each others merge requests daily on our i2 framework and collaborating on new features. This really wouldn't have happened in 2014 with each country deploying their applications in a completely different way using different tooling and processes. Teams were not aligned at all and we had silos everywhere.
These new cultural initiatives meant when we started i2 project in earnest, we had sorted the majority of the cultural issues and managed to create a cross functional team made up of people that were bought into these new ways of working who would create the core automation. This took people from a development, operations and networking backgrounds and put them in a single team to create self-service processes to help automate all the pain points on the incumbent infrastructure. This team was made up of people from the United Kingdom, Porto and Romania.
This created T shaped teams that brought deep dive knowledge from a particular discipline and then through working in the team that created automation for:
- Base images using Packer to create Centos 6, Centos 7 and Windows 2012 R2
- OpenStack VM provisioning
- OpenStack/ Nuage Network provisioning and ACL rules
- Load Balancer configuration
- Storage Provisioning
- Bare Metal provisioning
- Switch provisioning
If we had not instigated the necessary cultural changes and had the backing from the organisation, or trust from our managers to implement the necessary changes, we would have never have built a successful i2 project. We would likely have just another failed OpenStack or private cloud initiative.
To date our automation initiatives have won the i2 project a RedHat innovation award, an HP innovation award and most recently picked up the OpenStack Super User Award, at the latest OpenStack summit in Boston, which we are immensely proud of.
Allowing engineers to be creative, innovate and try new things continually will bring huge business benefits. We now do around 1000 deployments on any given working day, when a development team checks in code it triggers the continuous delivery cycle for one of the 200 applications that have been onboarded onto the platform so far. Each application uses the self-service automation and we provision about 3000 VMs a day over our two data centres.
Our deployments are completely immutable, every deployment will create new flavours (CPU, RAM and Disk), host aggregates, organises hypervisors, create networks, load balancing and virtual machines then install software on them based on the YAML files developers have filled in and added to GIT.
But through the continuous improvement model we implement we are incrementally improving the self-service framework and adding new features. When adding new features it is important to not compromise the integrity of the platform so we won't say yes to everything as scope creep is one of the reasons successful projects die.
Company culture allowed us to achieve the i2 project, sometimes I think people don't understand what an important factor this was. We were brave, innovative and we achieved great things. So if you have great people at your company, I plead with every manager, empower them. Steve Jobs once said:
“It doesn’t make sense to hire smart people and then tell them what to do; we hire smart people so they can tell us what to do.”
I think every manager can learn from this statement, the people you hire will make your company successful if you hire better than what you have.
Always strive to raise the bar with every hire, hire people that may not agree with you all the time. The bravest thing a manager can do is empower their staff, those that are fearful will micro-manage their staff and they will ultimately fail in initiatives as they can't know everything and are a level of abstraction too far away from the detail to make the correct decisions.
In conclusion, it is manager's job to remove blockers and impediments that hinder their engineers from doing their jobs and help create a company culture that allows them to be successful. OpenStack and Nuage has been a huge technology enabler but without sorting the necessary company culture before building the i2 platform we wouldn't have gotten anywhere. Don't underestimate how important company culture is at your organisation, the majority of cloud initiatives fail because of the culture not technology.
So anyone wanting to go on a similar journey, you will encounter detractors, non-believers, critics, but don't give up, it isn't impossible, you need to disrupt the notion that anything is impossible. Tear up the script, throw away the rulebook and create a new one. You only push technology forward by doing something that hasn't been done before.
So prototype new solutions, be brave, be unapologetic, not everything will work and treat failures as learning experiences. OpenStack has been a massive success for us and could be for others, I for one believe in open source technology, community and continual learning and so should you.
Videos of Paddy Power Betfair's OpenStack Summit talks from the most recent Summit in Boston are below:
Lessons learned from running 1000 Application Deployments a Day on OpenStack:
Immutable OpenStack Infrastructure:
How Paddy Power Betfair uses OpenStack Manila to manage stateful data in the DevOps process
Continuous Delivery Of Stateful applications using Cinder at Paddy Power Betfair