Lessons Learned from a Large-Scale Telco OSP+SDN Deployment Guil - - PowerPoint PPT Presentation

Lessons Learned from a Large-Scale Telco OSP+SDN Deployment

Guil Barros, Sr. Pr. Product Manager OpenStack, Red Hat Vicken Krissian, Sr. Technical Project Manager, Red Hat Cyril Lopez, Sr. OpenStack Consultant, Red Hat

Agenda

• Who we are
• The Timeline of an OSP engagement
• Challenges
• New Functionality & How to get there
• DCI & Testing Automation
• The future of 3rd party deployments - Composable Services/Roles

Who We Are

Cyril Lopez

• Red Hater from eNovance acquisition
• Cloud consultant and devops
• Committing in tripleo project

Vicken Krissian

• Red Hater from eNovance acquisition
• Senior Technical Project Manager
• Lead and contribute to the delivery of Cloud projects using Iterative methodology

Guil Barros

• Long-time Red Hater, Consulting, Support, OSP BU
• Handles partner engagement for OSP partners

A Timeline of an OSP Engagement

RFP published RFP Answer SDN Integration for OSP 7/ upstream POC Delivery Deal signed RH Services started Integration Hackfest Fully working RH OSP 8 and SDN platform User Acceptance Tests Pre-Production platform Go Live

2015 2016 July Dec June Nov March Sep July Feb

Project Challenges

• Customer

○ Going to production (Q4 ‘16.) ○ Expect mature product with full documentation on integrated platform. ○ Familiar working with providers not with communities.

• Red Hat

○ Full integration scheduled on RHT OSP 10 (Q4 ‘16.) ○ Focusing on product rather than services. ○ Upstream first for all code (opensource model.)

• SDN Partner

○ Certification on RHT OSP 10 (Q1 ‘17.) ○ Proprietary solution.

Integration Challenges

• The first version of Red Hat OpenStack director (based on TripleO)

delivered in summer 2015 (no SDN integration at the time.) ○ Initial release pains.

• Partner familiar with OSP packstack, needs to learn OSP director.
• Initial certification process not using OSP director + SDN integration.
• Plan for complete and official SDN integration with RHT OSP 10 but had to

deal with RHT OSP 8 and 9 with limited resources and process in the meantime

• Certification delays (up to 6 months)
• Difficulties reproducing the joint platform for testing and certification

People Challenges

• The Customer wants to be autonomous (learning from doing) from the

beginning with little support in terms of RHT services : ○ Only one dedicated engineer onsite. ○ No workshops to define all together the design and deployment phases.

• Customer team was new to RHT OpenStack, RHT OSP director and all

related technologies. ○ Slow deployment and learning pace.

• RHT and SDN partner not having deep knowledge and skills on other

partners technology and difficulties to find people with both expertise on the project.

Implementation Challenges

• Telco sensitive to change.
• Build a solution with Telco security requirements.

○ Making OSP Secure, not making a secure OSP.

• Build an easy to deploy solution.
• Integrate SDN controller with a pattern.
• Define a complete lifecycle.
• Certification timeline with new partners.

○ Line up with Customer deadlines. ○ SDN release timeline - OSP is a fast moving target.

Integration Hackfest

• Objectives

○ Having Partner and Red Hat engineering/consultants working together face to face and sharing Knowledge. ○ Verify and validate the technical feasibility and supportability of Integration Use Cases.

• Deliverables

○ Documentation and details procedure to reproduce the implementation

• n the field.

○ New bugs found with documented workaround.

What Red Hat brings to the table

Our capability to drive changes.

• Upstream development
• Project management
• Integrated product

Many features are new or require development. Lots of investment from all sides to deliver requested functionality. Not a product, an ecosystem.

Feature requests

• Upstream development
• Existing or entirely new?
• Can you get an upstream sponsor to help?
• Will it be accepted by the community?
• Timelines & Roadmaps
• When can we bring the new feature into the GA product?
• Does it require special handling from Support?
• Do the timelines from all partners line up feature-wise?
• What about lifecycle-wise?

(i.e. Partner-A LTS release w/ Partner-B LTS release)

Distributed CI - The Long-term Vision

• The single largest time saver available.
• Automated testing with each GA and pre-release.
• Test with RHT-internal OSP puddles.
• Takes time to set up and create test harnesses.
• But worth every second!
• Loop it into your certification chain!

• Allows Partners and Customers to work on the latest Red Hat OpenStack builds

available, as soon as possible, in their own environment.

• Shortens the time and effort for RHOSP Certification since it can run the certification

test suite on every change.

• Simplifies the upgrade progress by testing incremental changes.
• Provides automated feedback to Red Hat with helpful logs.
• Focus debugging efforts on the subset of the code that has changed.
• Include RHOSP tests as well as Partner's and customer's specific tests.

Why Distributed CI?

DCI aims to change the way you interact with Red Hat OpenStack.

Communication

• Have a good long-term idea of where you need to be and when.
• Which functionality is due when?
• Specific version or testing requirements?
• Three-way communication is critical. There are many subgroups in each

company, all with different touch-points. Having official communication points of contact makes life easier.

• Consistent sync-up calls with all partners and customers. It’s surprising how

fast misinformation spreads.

Composable Roles?

• Current OSP Nodes have fixed roles: Controller, Compute, Storage.
• Composable roles allows you to define the base OSP roles.

○ Neutron services on Controller 1,3,5 ○ Keystone + Others on Controller 2,4,6

• 3rd party services inside their own roles

○ SDN WebUI + Cassandra on Node 1,3 ○ SDN Manager on Node 2,4

The roles_data.yaml definitions look like this:

• name: Controller

CountDefault: 1 ServicesDefault:

• OS::TripleO::Services::CACerts
• OS::TripleO::Services::CephMon
• OS::TripleO::Services::CinderApi

Test, Test, Test

• Hackfests are your friend
• Include all stakeholders IN PERSON!
• Engineering also, not just frontline!
• File and follow up on bugs while fresh.
• Have goals and create a document as the output.
• For each major version
• Re-implement with prior hackfest document
• Note changes, regressions, etc
• Create a new document

For next time...

• Discovery/Design Workshop at the beginning of the project in order to :
• Have all teams involved with the same level of information.
• Share detailed Architecture Review.
• List requested functionality and implementation targets.
• Create feature roadmaps and define clear ownership.
• Define communication plans.
• Align Upstream, RHT, SDN, and Customer roadmaps early.
• Lock in features, architecture, and timelines for all four workstreams.
• Put in place a shared integration platform in order to :
• Test Early and Often.
• Implement CI as soon as possible.
• Make more Hackfests !

Questions?

THANK YOU