Figure 8: Nuage developers build features as SDK that platform developers are able to then use.
Auto UI generation
As mentioned above, Rest.li and the Pegasus data model are readable by both human and machine. This means that we were able to leverage both characteristics to communicate more clearly and innovate more quickly by developing a rendering engine.
The purpose of this engine is to ingest the Pegasus data model file and Rest.li spec, and auto-generate UI elements. Preliminary use of this engine has shown it to dramatically reduce our UI efforts to integrate with a platform.
Nuage SDK features
In this section, we will cover a few features in Nuage SDK to showcase the interaction between SDK logic and platform-specific logic. For each scenario, we identify what SDK provides and what platform developers need to do.
For most platforms, the basic API operations are: Create, Read(Get), Update, and Delete (CRUD). These are exposed as standard Rest.li methods.
For example, Kafka topic creation has 2 phases: provision topic in Kafka cluster (create in platform) and create metadata entry (create in Nuage). SDK will create the required metadata (ownership, ACL, alert configurations, etc.) out of the box. LinkedIn Kafka developers implement a SDK API to provision the topic in the Kafka cluster. It is also easy to extend above this 2-step execution pipeline to include extra steps, such as preCreate, create in platform, postCreate, and create in Nuage.
Async Execution Management
Short Running Async Job: ParSeq is an open source Java async library used extensively within LinkedIn. Our SDK adds a thin layer over ParSeq and provides platform developers a verb: “submit async task.” Anyone with previous ParSeq experience is able to use it the same way. Behind the scenes, the Nuage SDK persists execution status periodically until task completion or failure. We also built an application to support query and visualize execution status with key words. As you can see, the only thing platform developers need to do is to submit vanilla ParSeq task to our wrapped engine.
Approval Workflow By Humans: Certain requests (increasing quota, for example) require approval from resource owners. Such cases are asynchronous by nature. The SDK again provides the verb: “createApprovalWorkflow(requester id, payload, target resource id).” Platform developers can implement business logic (e.g., trigger an approval workflow if the requester is not authorized). Behind the scenes, the Nuage SDK persists the original request. We built an application to support “approval/ rejection/ query queued tasks.” Upon approval, the persisted tasks are executed on behalf of the original requester.
There’s an economic principle that says gains in efficiency come from better division of labor and specialization. This is why we created a framework built around self-service and empowerment of our teams to focus on the code logic they know best.
We found that the cost of integrating more infrastructure platforms is not linear to the number of platforms we support. One of the key observations in solving for this was that natural language discussion (human-to-human) can be an inefficient channel to transfer domain knowledge and requirements. Therefore, Rest.li spec and Pegasus data models are used as standard protocol to communicate API requirements among Nuage and platform developers, reducing costs. The Rest.li spec + Pegasus files are also used to auto-generate UI.
Such effort has brought down the cost of integrating infrastructure platforms to Nuage and enabled our team to deliver on our vision at a much faster pace.
Thank you to Changran Wei, Terry Fu, Yifang Liu, Nishant Lakshmikanth, Hunter Perrin, Tyler Corley, Darby Perez, and Micah Stubbs for contributing to the success of this project over the past year and a hlaf. It would have been impossible to turn this idea into reality without everyone’s perseverance and effort. Thank you Mohamed Battisha and Eric Kim for your strong support as managers. Thank you Ke Wu and the ISB team for being the first team to pilot our SDK early on. Your success boosted our confidence in our own work. Finally, thank you Yun Sun, Hai Lu, and Lei Xia for taking the time to review this article.