Mlops on Letters From The Wild Side

Flutter and Microservices

Sun, 29 Aug 2021 00:00:00 +0000

Last modified: May-29-2022, 02:40PM +08

Flutter Web and Microservices

If one ever needs to deploy neural-based services that can scale to zero, one of the best options include a client-server architecture that is based on microservices. This is not a comparison between software architecture design patterns, but rather an evaluation on the feasibility of flutter and microservices.

In most web applications, there is a common process which involves data aggregation and processing for downstream tasks. Some of those tasks include analytics, error reporting and threat monitoring. These tasks often come from different domains of data science, cybersecurity, cryptography, database, networking and distributed computing. It would be infeasible and impractical to expect any single programming language to cater to each and every domain. Not forgetting users of every level, ranging from novices to experts and public to enterprise.

The natural and intuitive solution to this hard requirement of cross-domain resources quickly points to a modularity of recomposing units with service granularity. Individual services can be improved incrementally, agnostic of language and platform. Communication can be independent or inter-dependent between separate components hosted on the same server. Seamless integration allows monolithic legacy and modern subsystems to be fully integrated.

Flexible integration with third-party resources such as autocomplete search engines, databases and object-stores are fully swappable for new frameworks and technologies in the future to prevent vendor lock-in.

Most managed backend solutions on the market today fully supports encrypted traffic with HTTPS. There is little reason not to secure and safeguard the user’s data and privacy against malicious actors looking to exploit zero-day vulnerabilities. There is a global community effort in developing and maintaining security standards and protocols which serve the needs of billions of users on the internet. These open-source cryptographic libraries form a critical part of the arsenal available to developers, without resource-stricken proprietary libraries exposing the attack surface.

The microservice developer also has access to open-source and efficient media codecs for both transmission and storage. Attributes of a microservice architecture are well suited for a rapid evolving technology that is seeking regional and global reach.

Development

Common message formats include HTTP, gRPC, GraphQL and WebSocket providing uni/bidirectional communication channel depending on the application. Maturity of these standards and protocols allow them to be widely implemented in languages popular with web development. With the advent of microservices in the early 2010s, Docker remains a popular tool for local development of containers.

There has been ongoing work on porting docker to Internet-of-Things(IoT), mobile and GPU platforms. The same codebase for any particular microservice can be easily adapted to target a diversity of platforms during expansion into surrounding markets. This is evident in the growing number of cross-platform web applications. The targeting platform can be expanded to include x86, ARM, RISC-V, NVPTX or AMD ISA, not excluding FPGAs and ASICs.

Service endpoints comprise of IoT, dedicated servers and public cloud for both internal and public facing APIs. To efficiently scale to millions of users, a container orchestration tool such as Kubernetes should be utilized to manage network traffic and computing resources.

Driven by open-source momentum, a healthy and growing ecosystem of developer tools is the end result of that collective effort. There is no lack of options targeting different niches of the technology stack required to bring a minimal viable product(MVP) to satisfy early adopters, which in turn provide valuable feedback for the next product iteration.

The adoption of continuous integration and continuous deployment(CI/CD) best practices simplifies testing and deployment of microservices. This strategy enables the timely delivery of new features on budget.

Deployment

Once a microservice is deployed, its API will accept authenticated requests for identity authentication and access-control of resources. Further restrictions on CPU, memory or GPU are enforced through configurations flags. Resource requirements can be forecasted based on usage patterns during off-peak and peak periods. Service granularity allows real time procurement of additional resources before spikes in traffic and also scaling to zero for deprecated services.

Containers by default have restricted access to host resources and they run in isolation with regards to other processes on the host system. A microservice being self-contained and coupled with its dependencies can be deployed in parallel with version control. As user base grows, vertical and horizontal scaling of these containers is what allow microservice to be scalable across different regions via cloud.

The high development cost is compensated by low cost of entry and low running costs with generous credits from public clouds. Network latency is offset with load-balancing proxies, CDNs and regional deployment with redundancies to ensure high availability to users.

Fine-grained handling and securing of network traffic is done using a service mesh such as Istio, Traefik or in-house solution. API security is enforced through threat and error monitoring with logging/tracing. To ensure high availability, periodic health checks are conducted on endpoints. In the event of a disruption, the operations team will be notified within seconds while an automated rollback or spinning of new instances during surges will take place simultaneously.

This composition of behaviors ensure the microservice stays fault-tolerant.

Security

OAuth2
SAML
JWT

Future Applications

Self-driving vehicles and autopilot UI
Self-planning residential and industrial robots with remote human intervention
Self-organizing behavior in drone swarms at scale with mining and agriculture industries
Implementation of the first distributed superintelligence with a brain-computer interface
A bundle of artificial intelligence(AI) tools with the analogy of the Swiss army knife

What’s Next?

Experimentation of running neural-based eBPF programs as microservices with flutter.

Summary

Attributes of a microservice architecture are well suited for a rapid evolving technology that is seeking regional and global reach.

healthy, growing ecosystem of developer tools and CI/CD best practices ensures timely delivery of new features on budget.
composition of service granularity, identity and access control of resources ensures the microservice to remain fault-tolerant and highly available.
API security
future applications

ML Engineering

Tue, 08 Jun 2021 00:00:00 +0000

Last modified: Dec-14-2021, 10:35PM +08

From Manual To Semi-Automatic

Before the advent of the concept “MLOps”, getting a single machine learning(ML) model to production was tedious and belaboring. Every single detail pertaining to the inputs, model server, training and inference have to be defined explicitly. This is to ensure the input tensors follow a strict requirement for them to be processed by user defined functions.

To serve a single model, these predefined configurations have to be under version control as the ML field and software ecosystem is accelerating at near exponential speeds. In addition to the model, version control has to be applied to the training data as well as the software infrastructure that is used to host the model. A working production pipeline is like a moving train loading and offloading compartments to keep up with cutting-edge development.

After the release of ImageNet dataset, there was tremendous effort poured into surpassing the human baseline. In the early 2010s, that baseline was exceeded with the combination of readily available data, open-source frameworks and modern computing resources that can be bought off the shelf. However, being proficient in these resources was restricted to experts and those within the technical community. Developer sanity was largely dependant on up-to-date documentation or comments within the source where documentation was absent.

In the mid 2010s, a number of Deep Learning(DL) frameworks were designed to unify the common primitives in building these DL models. These include TensorFlow, Keras, Pytorch, Apache MXNet and many others.

To tackle the problem of productionizing models, one of the solutions explored was the usage of Docker containers, to package both the dependencies and the actual model as lightweight components that can be easily shared through a public repository. This approach greatly democratize the deployment of DL models to common hosting providers like the public clouds or in-house servers.

The natural progression in using Docker containers meant the inclusion of shell scripts, cron jobs and triggers that allow the automation of the entire ML pipeline. Docker-based workflows gave developers access to version controlled resources locally on their laptops and globally across different time zones.

Components-Based Workflow

For organizations that need to scale to millions of containers in production, the de facto solution include container orchestration platforms such as Kubernetes. The platform allows hundreds and thousands of engineers to collaborate on different levels of a complex ML system. This ranges from low level implementation of hardware drivers to the high level design of user-interfaces such as click-and-drag block diagrams.

The low-code or no-code approach is an industry effort to lower the cognitive strain in designing complex ML models. The design and implementation of mission-critical models requires non-trivial engineering efforts, so why should their deployment be unnecessarily complex?

Behind the scenes of the components-based workflow lies Kubernetes applications such as Argo Workflows, Tekton as well as many others. These applications specify steps in a ML pipeline as containers that can spun up sequentially or in parallel. These steps can be expressed as a directed acyclic graph(DAG), which can be version controlled and compiled for export to different hardware architectures.

Initially, we had manual design, hand-tuned and hand-crafted models without A/B testing because deployment of new models simply could not keep up with the development of a core application(4~6 weeks cycle). Now we can churn dozens of models daily in parallel, set to trigger on arrival of new data or based on adjacent/over-lapping time windows. The models that passed evaluation are then uploaded to a model repository for further downstream processes.

Cautionary Tales

A majority of kubernetes applications are rather new to the scene, many more are emerging to solve critical issues pertaining to storage, security, networking and other peripherals. Choosing the right software stack requires an in-depth technical review of existing solutions with respect to dimensions of correctness, latency and costs.

At the SME scale, one single competent ML engineer is the bare requirement for a sufficiently complex ML system, serving requests up to the number of CPU cores procured with default settings.

At the enterprise scale, ML engineering is not well suited to be an one-man job, but rather spread across different teams with each being a subject matter expert on their domains.

What’s Next

Currently working on a regression pipeline, targeting TensorFlow.js models to be deployed in a Flutter application, hosted by Firebase. The pipeline is designed to be agnostic to regression problem domains. Future regression tasks include cryptocurrency market size, health monitoring, renewable energy forecasts and EV tank-to-wheel efficiency(70~90%).

Other pipelines include tasks under the pillars of ML:

classification
density-estimation
dimensionality-reduction

Pipelines for generative models are in the roadmap as well.

Incorporating accelerators such as GPUs or TPUs into pipeline to further parallelize existing workflows.